ankit.im

The self-hosted Swift Package Manager

Feb 4, 2016
4 minute read

Swift Package Manager is itself a swift package containing 6 targets :

  1. libc : Generates a module called libc which is Glibc on linux and Darwin.C on OSX.

  2. POSIX : This module Swiftifies the POSIX APIs so that they can be used without all the C like code inside swift.

  3. sys : Contains APIs to interact with file system (in swift-like way).

  4. PackageDescription : The module we import in our swift packages containing package description, target, dependencies, versions.

  5. dep : This is the main module which clones git repos and invokes llbuild process to build the packages.

  6. swift-build : And finally, the executable module which parses the input and drives the build process.

llbuild is “A low-level build system, used by the Swift Package Manager”. The swift-build-tool product of llbuild is used by swift package manager to compile the swift packages.

Swiftpm has a python bootstrap script which initially compiles all of swiftpm targets and generates the swiftpm executable. The bootstrapped exectuable is then used to compile the swiftpm source and generate the final swift-build product.

Lets explore a bit of bootstrap script :

Target : A python class which contains name, dependencies, flags etc for a target. As soon as it is initialized, it figures out if a target is a library or an executable by checking for main.swift

class Target(object):
    
	...
	
    def __init__(self, name, dependencies=[], swiftflags=[], extra_libs=[],
                 subpath=None, is_test=False):
        self.name = name
        self.dependencies = list(dependencies)
        self.swiftflags = list(swiftflags)
        self.extra_libs = list(extra_libs)
        self.is_test = is_test

        # Discover the source files, and whether or not this is a library.
        self.is_library = True
        self.swift_sources = []
        for (dirpath, dirnames, filenames) in os.walk(
                os.path.join(g_source_root, subpath or self.name)):
            for name in filenames:
                path = os.path.join(dirpath, name)
                _,ext = os.path.splitext(name)
                if ext == '.swift':
                    if name == 'main.swift':
                        self.is_library = False
                    self.swift_sources.append(path)

            # Exclude tests and fixtures, for now.
            dirnames[:] = [
                name for name in dirnames
                if name != " " and not name.lower().startswith("fixtures")]
        self.swift_sources.sort()
		
	...

It contains a method which writes the .llbuild commands needed for this target to be built to output string buffer.

def write_swift_compile_commands(self, opts, target_build_dir,
                                     module_dir, output, objects,
                                     link_input_nodes, predecessor_node):

The array of Target objects are defined in a targets global variable :

targets = [
    Target('PackageDescription'),
    Target('libc'),
    Target('POSIX', dependencies=["libc"]),
    Target('sys', dependencies=["POSIX", "libc"]),
    Target('dep', dependencies=["sys", "PackageDescription"]),
    Target('swift-build', dependencies=["dep", "sys", "PackageDescription",
                                        "POSIX", "libc"]),
										
   ...

When the script is run, the following options are pretty important to it but they’re all filled automatically if toolchain is installed and exported correctly.

  • swiftc_path : Swift compiler path

  • sbt_path : Swift built tool path (llbuild)

  • sysroot : needed on OSX (macosx sdk)

  • build_path : $pwd/.build dir

    parser.add_option("", "--swiftc", dest="swiftc_path",
                      help="path to the swift compiler [%default]",
                      default=os.getenv("SWIFT_EXEC") or "swiftc", metavar="PATH")
    parser.add_option("", "--sbt", dest="sbt_path",
                      help="path to the 'swift-build-tool' tool [%default]",
                      metavar="PATH")
    parser.add_option("", "--sysroot", dest="sysroot",
                      help="compiler sysroot to pass to Swift [%default]",
                      default=g_default_sysroot, metavar="PATH")
    parser.add_option("", "--build", dest="build_path",
                      help="create build products at PATH [%default]",
                      default=".build", metavar="PATH")
    parser.add_option("", "--prefix", dest="install_prefix",
                      help="use PATH as the prefix for installing [%default]",
                      default="/usr/local", metavar="PATH")
    parser.add_option("-v", "--verbose", dest="verbose", action="store_true",
                      help="use verbose output")
    parser.add_option("--xctest", dest="xctest_path", action="store",
                      help="Path to XCTest build directory")
    parser.add_option("", "--build-tests", dest="build_tests",
                      action="store_true", help="enable building tests")
    opts,args = parser.parse_args()

the build_path (where swiftpm will be finally built to) is set to .build dir and sandbox_path (where bootstrap binary will be built) is set to .build/.bootstrap

    # Compute the build paths.
    build_path = os.path.join(g_project_root, opts.build_path)
    sandbox_path = os.path.join(build_path, ".bootstrap")

The get_swift_build_tool_path() method is used to look for the swift-build-tool (llbuild) if it is not supplied at input

    # Determine the swift-build-tool to use.
    opts.sbt_path = os.path.abspath(
        opts.sbt_path or get_swift_build_tool_path())

create_bootstrap_files() method creates the .llbuild file which will build all targets and the executable.

It uses the write_swift_compile_commands method on target objects.

The file is created at .build/.bootstrap/build.swift-build.

    # Create or update the bootstrap files.
    create_bootstrap_files(sandbox_path, opts)

Now the bootstrap script is ready to spawn swift-build-tool to compile swiftpm

    # Run the stage1 build.
    cmd = [opts.sbt_path, "-f", os.path.join(sandbox_path, "build.swift-build")]
    if opts.verbose:
        cmd.append("-v")
    note("building stage1: %s" % ' '.join(cmd))
    result = subprocess.call(cmd)
    if result != 0:
        error("build failed with exit status %d" % (result,))

process_runtime_libraries method is called on the generated PackageDescription module to check if they’re being loaded correctly or not.

runtime_module_path,runtime_lib_path = process_runtime_libraries(
        sandbox_path, opts, bootstrap=True)
def process_runtime_libraries(build_path, opts, bootstrap=False):

And finally, the swiftpm builds itself :

    # Build the package manager with itself.
    env_cmd = ["env",
               "SWIFT_EXEC=" + opts.swiftc_path,
               "SWIFT_BUILD_TOOL=" + opts.sbt_path,
               "SWIFT_BUILD_PATH=" + build_path]
    if opts.sysroot:
        env_cmd.append("SYSROOT=" + opts.sysroot)
    # On Linux, we need to embed an RPATH so swift-{build,get} can find the core
    # libraries.
    if platform.system() == 'Linux':
        env_cmd.append("SWIFTPM_EMBED_RPATH=$ORIGIN/../lib/swift/linux")
    else:
        env_cmd.append("SWIFTPM_EMBED_RPATH=@executable_path/../lib/swift/macosx")
    cmd = env_cmd + [os.path.join(sandbox_path, "bin", "swift-build")]
    note("building self-hosted 'swift-build': %s" % (
        ' '.join(cmd),))
    result = subprocess.call(cmd, cwd=g_project_root)

    if result != 0:
        error("build failed with exit status %d" % (result,))