The New Tools Framework

Johnny C. Lam

jlam@NetBSD.org

The NetBSD Project

Outline

• What are we trying to solve?
• Why does tools.mk suck?
• How is the new tools framework used?
• Future work

Problem domain

• Not all platforms are created equal for pkgsrc.
• Some platforms are missing a usable sed (Solaris).
• Some platforms are missing a usable patch (Interix, Darwin).
• Some platforms are missing everything (IRIX 5, AIX).
• Q: Where can we find the missing bits?
• A: pkgsrc (duh)

tools.mk sucks

• tools.mk is great big hack to provide GNU tools where native tools were not powerful enough for pkgsrc, e.g. gawk, gsed, etc.
• I looked at it recently and didn't understand how it worked.
  • Bad news: I wrote most of it
• Others have piled more hacks on top of the existing hacks in tools.mk, e.g. AUTOMAKE_OVERRIDE, texinfo, bison/yacc.
• No longer has a clear purpose anymore

USE_TOOLS

• list of tools that we require to build the package, e.g. awk, cp, mv
• pkgsrc itself requires certain tools to function
• Package Makefiles can append their own list of tools, e.g. gmake, gawk
• Framework will cause BUILD_DEPENDS to be added if required tools is missing

USE_TOOLS (cont.)

• Documents which tools are needed by pkgsrc itself
  • each target uses some tools, e.g. ${CP}, ${ECHO}, ${MKDIR}, etc.
  • make(1) variables are set using ${AWK}, ${SED}, ${TEST}, etc.
  • different phases of build process require different sets of tools
• pkgsrc requires a surprising number of tools!

TOOLS_PLATFORM.*

• Each system has a set of TOOLS_PLATFORM.* definitions marking system-supplied tools that pkgsrc can use.
• Tools must be powerful enough to be used by pkgsrc, e.g. can't use /usr/bin/sed on Solaris.
• pkgsrc/bootstrap should test for usable system tools and create an appropriate set of TOOLS_PLATFORM.* definitions.

"bootstrap" issues

• Q: How to install sysutils/coreutils if we don't have a powerful enough cp, mkdir, mv, etc.?
• A: Split out targets not needed by "make clean && make package" into bsd.pkg.helper.mk file.
  • Isolates the tools that are critical to core pkgsrc functionality
  • "Do something" to provide fallback tools to avoid circular dependencies
• awk is used very heavily, especially by builtin.mk files and to create PLISTs.
  • should probably add this to pkgsrc/bootstrap

TOOLS_CREATE, TOOLS_CMD.*

• TOOLS_CREATE is the actual list of tools that will appear in ${TOOLS_DIR}.
• TOOLS_CMD.* is the path of the tool in ${TOOLS_DIR}
  • used to specify the name that the tool can be invoked with, e.g. TOOLS_CMD.gsed is ${TOOLS_DIR}/bin/sed.

TOOLS_REAL_*

• TOOLS_REAL_CMDLINE.* is the full statement list to use in the wrapper script.
• TOOLS_REAL_CMD.* is the command itself (no arguments)
  • may be full path or shell builtin function
• TOOLS_REAL_ARGS.* is the argument list that ${TOOLS_REAL_CMD.*} should be invoked with, .e.g. mkdir needs "-p".

Wrapper or Symlink?

• Depending values of TOOLS_REAL_*, the framework creates either a wrapper or a symlink.
• Make a wrapper if:
  • TOOLS_REAL_CMDLINE.* is defined
  • TOOLS_REAL_CMD.* isn't a full path
  • TOOLS_REAL_ARGS.* is non-empty
• Make a symlink otherwise
  • purely for speed issues

Future work

• Implement compiler framework on top of the tools framework.
• Test on operating systems that are the bane of my existence, e.g. IRIX 5, Interix, Mac OS X
• Better cooperation from pkgsrc/bootstrap to probe system and create correct TOOLS_PLATFORM.* definitions.
• Solve the "bootstrapping" issue with sysutils/coreutils.