40ants-builder - A reusable builder of standalone binaries for Common Lisp projects

40ANTS-BUILDER ASDF System Details

Features

Dumping a standalone binary of a Common Lisp application requires more than calling asdf:make or ros build. A production build needs a prepared environment: local checkouts registered in ASDF, implementation specific quirks patched, Slynk contrib modules loaded before the dump (a Slynk client cannot load them at run time on the production host, where neither Quicklisp nor the sources are reachable), and the build identification captured while the sources and .git are still around. This recipe usually lives in a per-project build.lisp script, copied between projects, untested and undocumented. 40ants-builder extracts it into one reusable, tested function: (40ants-builder:build :my-app).

The library does not replace Deploy: it complements it. Your system still declares deploy-op as its :build-operation, and Deploy bundles foreign libraries and dumps the image. 40ants-builder takes care of everything around that dump and finishes by calling asdf:make, which performs deploy-op.

Installation

You can install this library from Quicklisp, but you want to receive updates quickly, then install it from Ultralisp.org:

(ql-dist:install-dist "http://dist.ultralisp.org/"
                      :prompt nil)
(ql:quickload :40ants-builder)

Usage

Call the build function with the name of the ASDF system which describes your program. For a project which defines a pingvito system with a deploy-op build operation, run from the project's root:

qlot exec ros run --eval '(ql:quickload :40ants-builder)' --eval '(40ants-builder:build :pingvito)'

The pipeline, essentially a common build.lisp script extracted into a reusable library:

  1. installs a debugger hook which fails the process with exit code 1 instead of hanging in the interactive debugger;

  2. extends the ASDF registry from the local-deps.lisp file in the system's source directory, when it is present;

  3. installs implementation workarounds: patches CFFI's foreign library loader and conduit-packages's DEFPACKAGE clause computation on CCL, and creates an empty NIL package on ECL;

  4. quickloads Deploy, the system being built and Slynk contrib modules, which otherwise would be missing in production when a Slynk client connects;

  5. pins the build identification: the git commit goes into 40ants-builder:*git-commit* and the system's ASDF version becomes VERSION+GIT.COMMIT;

  6. prepares the image for the dump: renames the Clack handler package to its slash nickname, pushes the :deploy-console feature and removes the build debugger hook, so it does not get dumped into the image;

  7. dumps the binary: asdf:make on SBCL and CCL, and asdf:program-op on ECL, which does not support uiop:dump-image.

On success the process exits with code 0; on failure the condition is printed with a backtrace and the process exits with code 1.

Pass slynk-modules nil to skip the Slynk contribs and exit nil to keep the process alive, returning the pinned commit instead of quitting:

(40ants-builder:build :pingvito :slynk-modules nil)

Note that a binary built on one operating system or architecture cannot run on another one. Build on the target host or in an equivalent environment, and keep the bin/ directory with native libraries next to the binary in the release.

For AI Agents

This library is written with AI coding agents, and it ships an agent skill so that agents can adopt it correctly. The skill teaches how to convert a project to build through 40ants-builder: the program system layout, the optional local-deps.lisp registry file, the direct command and the 40ants-lake builder slot integration, build verification, and the retirement of an old build.lisp script.

Point your coding agent to the SKILL.md file of this repository, or copy it into your project's .agents/skills/setup-40ants-builder/ directory. The AI badge in the README marks the code as LLM generated.

API

40ANTS-BUILDER/CCL

Functions

Return only those names which are not in the SEEN hash-table yet, registering every returned name.

function
&key extends extends/including extends/excluding

Return DEFPACKAGE clauses for a conduit package, skipping symbol names already imported by a previous extension.

This is a copy of the compute-conduit-clauses function of the org.tfeb.conduit-packages library which elides such names: importing the same external symbol via two :import-from clauses is tolerated by SBCL, but is a hard error on CCL ("Name ... is already used in a conflicting option"). The Lisp-Stat project defines its packages with the :extends option of the conduit-packages DEFPACKAGE macro, and several of its extended packages export the same symbols.

Patch CFFI and conduit-packages to make Lisp-Stat and its dependencies compile and load on CCL.

Quickloads CFFI and conduit-packages and installs the patches. Call before the target system is quickloaded, because the patches are used during compilation of its dependencies. Symbols are resolved at run time, so this module compiles on implementations where CFFI and conduit-packages are absent. On other implementations this is a no-op.

40ANTS-BUILDER/CORE

Classes

BUILD-ERROR

Readers

Functions

function
system &key (slynk-modules \*default-slynk-modules\*) (exit t)

Build a standalone binary of SYSTEM and return the pinned git commit.

SYSTEM is an ASDF system designator of the program system, like :pingvito. The pipeline is essentially a build.lisp script extracted into a reusable library:

  1. installs a debugger hook which fails the process with exit code 1 instead of hanging in the interactive debugger;

  2. extends the ASDF registry from the local-deps.lisp file in SYSTEM's source directory, when it is present;

  3. installs implementation workarounds (the CFFI and conduit-packages patches on CCL, the empty "NIL" package on ECL);

  4. quickloads Deploy, SYSTEM and SLYNK-MODULES (defaults to default-slynk-modules; pass NIL to skip them);

  5. pins the build identification: the git commit goes into git-commit and the commit is appended to the ASDF version of SYSTEM;

  6. prepares the image for the dump: renames the Clack handler package to its slash nickname, pushes the :deploy-console feature and removes the build debugger hook;

  7. dumps the binary: asdf:make on SBCL and CCL, asdf:program-op on ECL.

On success the process exits with code 0. On failure the condition is printed with a backtrace and the process exits with code 1. When EXIT is NIL, build does not quit: it returns the commit on success and signals build-error, wrapping the original condition, on failure.

Capture build-time identification of SYSTEM into the image.

Stores the git commit into git-commit and appends the commit to the system's version, updating its component-version. The version is taken from the system's component-version or, when it is unset, from the topmost entry of its docs/changelog.lisp.

Called before the image dump, so that the binary's version flag reports the state of the sources it was built from, and not of whatever is on disk at run time. Returns the commit.

Variables

variable
(:slynk/indentation :slynk/stickers :slynk/trace-dialog :slynk/package-fu :slynk/fancy-inspector :slynk/arglists)

Slynk contrib modules loaded into the image before the dump.

The Slynk server tries to load these modules when a client connects and cannot do that in production, where neither Quicklisp nor the sources are reachable. Pass NIL as SLYNK-MODULES argument of the build function to skip them.

Git commit captured by pin-build-version before the image dump.

The binary usually runs on a host without .git and sources (ProtectHome=read-only), so the commit has to be captured at build time. Projects use this variable in their version flag output.

40ANTS-BUILDER/DEPENDENCIES

Functions

function
system &key (slynk-modules \*default-slynk-modules\*)

Load everything which has to be present in the dumped image.

This is the Deploy library, the SYSTEM being built and the SLYNK-MODULES.

Variables

variable
(:slynk/indentation :slynk/stickers :slynk/trace-dialog :slynk/package-fu :slynk/fancy-inspector :slynk/arglists)

Slynk contrib modules loaded into the image before the dump.

The Slynk server tries to load these modules when a client connects and cannot do that in production, where neither Quicklisp nor the sources are reachable. Pass NIL as SLYNK-MODULES argument of the build function to skip them.

40ANTS-BUILDER/ECL

Functions

Prepare the image so the dependencies' DEFPACKAGE forms can be read on ECL.

Some dependencies (metacopy, in particular) read

.(if load-with-contextl '#:contextl (values)) inside DEFPACKAGE.

On SBCL, the reader inserts nothing into the list for zero values of sharp-dot, while the ECL reader inserts NIL, which DEFPACKAGE turns into :use of a package named "NIL", and ECL fails with "There exists no package with name NIL". Creating an empty package with this name makes such DEFPACKAGE forms valid. On other implementations this is a no-op.

40ANTS-BUILDER/IMAGE

Functions

Dump the standalone binary of SYSTEM.

On SBCL and CCL this is asdf:make, which performs the system's :build-operation (deploy-op for Pingvito). ECL does not support uiop:dump-image, and deploy-op relies on it, so under ECL the standard asdf:program-op is used instead: the system is compiled to C and the object files are linked into an executable with c:build-program, using the :entry-point and :build-pathname from the system's definition file.

Add NICKNAME to the package named PACKAGE-NAME.

Does nothing when the package does not exist or when a package named NICKNAME is already present, because in these cases rename-package either has nothing to do or signals a correctable error. Returns T when the package was renamed.

Install a debugger hook which fails the build process immediately.

Any error during the build must exit the process with a non-zero code, instead of leaving it hanging in the interactive debugger of Roswell/CCL.

Prepare the build image for the dump.

Gives the Clack Hunchentoot handler package its slash nickname: Clack looks handlers up by a name with slashes, does not find it in the current dist and calls ql:quickload at run time, which fails under ProtectHome. The nickname removes the runtime load.

Then pushes the :deploy-console feature to silence Deploy's status output at service startup ("Performing warm boot", "Loading foreign library ..." etc.): with this feature the warm boot sets status-output to NIL, the same way deploy-console-op does for console applications. Build identification stays available through the project's version flag.

Finally, removes the build-only debugger hook, so it does not get dumped into the image: otherwise any runtime error would kill the service with a build failure message instead of the project's own error handling.

40ANTS-BUILDER/REGISTRY

Functions

Extend the ASDF registry with systems local to the project being built.

Projects keep a local-deps.lisp file next to their system definition file; it defines an extend-registry function which pushes local checkout directories onto asdf:*central-registry*. When the file is missing, nothing is registered and the call is a no-op, because a project may have all its dependencies on Quicklisp.

40ANTS-BUILDER/VERSION

Functions

Capture build-time identification of SYSTEM into the image.

Stores the git commit into git-commit and appends the commit to the system's version, updating its component-version. The version is taken from the system's component-version or, when it is unset, from the topmost entry of its docs/changelog.lisp.

Called before the image dump, so that the binary's version flag reports the state of the sources it was built from, and not of whatever is on disk at run time. Returns the commit.

Return the topmost version from SYSTEM's docs/changelog.lisp or NIL.

This mirrors what the 40ants-asdf-system class does on an explicit load-system: the first changelog entry defines the version. Forms are read into a temporary package, because the changelog macro and version symbols are not interned anywhere in the build image.

Variables

Git commit captured by pin-build-version before the image dump.

The binary usually runs on a host without .git and sources (ProtectHome=read-only), so the commit has to be captured at build time. Projects use this variable in their version flag output.