40ants-builder - A reusable builder of standalone binaries for Common Lisp projects
40ANTS-BUILDER ASDF System Details
Description: A reusable builder of standalone binaries for Common Lisp projects.
Licence: Unlicense
Author: Alexander Artemenko <svetlyak.40wt@gmail.com>
Homepage: https://40ants.com/builder/
Bug tracker: https://github.com/40ants/builder/issues
Source control: GIT
Depends on: asdf
Features
Dumping a standalone binary of a Common Lisp application requires more
than calling asdf:make or ros build. A production build needs a
prepared environment: local checkouts registered in ASDF,
implementation specific quirks patched, Slynk contrib modules loaded
before the dump (a Slynk client cannot load them at run time on the
production host, where neither Quicklisp nor the sources are
reachable), and the build identification captured while the sources
and .git are still around. This recipe usually lives in a
per-project build.lisp script, copied between projects, untested and
undocumented. 40ants-builder extracts it into one reusable, tested
function: (40ants-builder:build :my-app).
The library does not replace Deploy: it complements it. Your system
still declares deploy-op as its :build-operation, and Deploy
bundles foreign libraries and dumps the image. 40ants-builder takes
care of everything around that dump and finishes by calling
asdf:make, which performs deploy-op.
One call builds the binary.
(40ants-builder:build :my-app)runs the whole pipeline: registry setup, workarounds, dependency loading, version pinning, image preparation and the dump.Local systems registered. A
local-deps.lispfile in the project root, when present, is loaded to push local checkout directories onto theASDFregistry.Implementation quirks patched. On
CCLthe builder fixesCFFI's foreign library loader, which chokes on logical pathnames with a:newestversion suffix, and teaches conduit-packages to elide duplicate defpackage imports, which Lisp-Stat requires; onECLit creates an emptyNILpackage so the dependencies' defpackage forms stay readable.Build identification pinned. The git commit and the version from the system's
docs/changelog.lispare stored into the image, so--versionreports what the binary was built from, even on a host without sources.Production ready image. Slynk contrib modules are loaded before the dump, the
:deploy-consolefeature silences Deploy's status output at service startup, and the build debugger hook is removed, so it does not ship inside the binary.Multiple implementations.
SBCLandCCLbuild viaasdf:make(usuallydeploy-op),ECLbuilds viaasdf:program-op, which does not rely on the unsupporteduiop:dump-image.Fail fast behavior. Any build error prints the condition with a backtrace and exits with a non-zero code instead of hanging in the interactive debugger; pass
exit nilto embed the pipeline into another tool and handle errors yourself.
Installation
You can install this library from Quicklisp, but you want to receive updates quickly, then install it from Ultralisp.org:
(ql-dist:install-dist "http://dist.ultralisp.org/"
:prompt nil)
(ql:quickload :40ants-builder)Usage
Call the build function with the name of the ASDF system which
describes your program. For a project which defines a pingvito
system with a deploy-op build operation, run from the project's
root:
qlot exec ros run --eval '(ql:quickload :40ants-builder)' --eval '(40ants-builder:build :pingvito)'The pipeline, essentially a common build.lisp script extracted into
a reusable library:
installs a debugger hook which fails the process with exit code 1 instead of hanging in the interactive debugger;
extends the
ASDFregistry from thelocal-deps.lispfile in the system's source directory, when it is present;installs implementation workarounds: patches
CFFI's foreign library loader and conduit-packages'sDEFPACKAGEclause computation onCCL, and creates an emptyNILpackage onECL;quickloads Deploy, the system being built and Slynk contrib modules, which otherwise would be missing in production when a Slynk client connects;
pins the build identification: the git commit goes into
40ants-builder:*git-commit*and the system'sASDFversion becomesVERSION+GIT.COMMIT;prepares the image for the dump: renames the Clack handler package to its slash nickname, pushes the
:deploy-consolefeature and removes the build debugger hook, so it does not get dumped into the image;dumps the binary: asdf:make on
SBCLandCCL, and asdf:program-op onECL, which does not support uiop:dump-image.
On success the process exits with code 0; on failure the condition is printed with a backtrace and the process exits with code 1.
Pass slynk-modules nil to skip the Slynk contribs and exit nil to
keep the process alive, returning the pinned commit instead of
quitting:
(40ants-builder:build :pingvito :slynk-modules nil)Note that a binary built on one operating system or architecture cannot run on
another one. Build on the target host or in an equivalent
environment, and keep the bin/ directory with native libraries next
to the binary in the release.
For AI Agents
This library is written with AI coding agents, and it ships an agent
skill so that agents can adopt it correctly. The skill teaches how to
convert a project to build through 40ants-builder: the program system
layout, the optional local-deps.lisp registry file, the direct
command and the 40ants-lake builder slot integration, build
verification, and the retirement of an old build.lisp script.
Point your coding agent to the
SKILL.md
file of this repository, or copy it into your project's
.agents/skills/setup-40ants-builder/ directory. The
AI badge in the README marks the
code as LLM generated.
API
40ANTS-BUILDER/CCL
Functions
Return only those names which are not in the SEEN hash-table yet,
registering every returned name.
Return DEFPACKAGE clauses for a conduit package, skipping symbol
names already imported by a previous extension.
This is a copy of the compute-conduit-clauses function of the
org.tfeb.conduit-packages library which elides such names: importing
the same external symbol via two :import-from clauses is tolerated by
SBCL, but is a hard error on CCL ("Name ... is already used in a
conflicting option"). The Lisp-Stat project defines its packages with
the :extends option of the conduit-packages DEFPACKAGE macro, and
several of its extended packages export the same symbols.
Patch CFFI and conduit-packages to make Lisp-Stat and its
dependencies compile and load on CCL.
Quickloads CFFI and conduit-packages and installs the patches. Call
before the target system is quickloaded, because the patches are used
during compilation of its dependencies. Symbols are resolved at run
time, so this module compiles on implementations where CFFI and
conduit-packages are absent. On other implementations this is a
no-op.
40ANTS-BUILDER/CORE
Classes
BUILD-ERROR
Readers
Functions
Build a standalone binary of SYSTEM and return the pinned git commit.
SYSTEM is an ASDF system designator of the program system, like
:pingvito. The pipeline is essentially a build.lisp script extracted
into a reusable library:
installs a debugger hook which fails the process with exit code 1 instead of hanging in the interactive debugger;
extends the
ASDFregistry from the local-deps.lisp file inSYSTEM's source directory, when it is present;installs implementation workarounds (the
CFFIand conduit-packages patches onCCL, the empty "NIL" package onECL);quickloads Deploy,
SYSTEMandSLYNK-MODULES(defaults to default-slynk-modules; passNILto skip them);pins the build identification: the git commit goes into git-commit and the commit is appended to the
ASDFversion ofSYSTEM;prepares the image for the dump: renames the Clack handler package to its slash nickname, pushes the :deploy-console feature and removes the build debugger hook;
dumps the binary: asdf:make on
SBCLandCCL, asdf:program-op onECL.
On success the process exits with code 0. On failure the condition is
printed with a backtrace and the process exits with code 1. When EXIT
is NIL, build does not quit: it returns the commit on success and
signals build-error, wrapping the original condition, on failure.
Capture build-time identification of SYSTEM into the image.
Stores the git commit into git-commit and appends the commit to the system's version, updating its component-version. The version is taken from the system's component-version or, when it is unset, from the topmost entry of its docs/changelog.lisp.
Called before the image dump, so that the binary's version flag reports the state of the sources it was built from, and not of whatever is on disk at run time. Returns the commit.
Variables
Slynk contrib modules loaded into the image before the dump.
The Slynk server tries to load these modules when a client connects
and cannot do that in production, where neither Quicklisp nor the
sources are reachable. Pass NIL as SLYNK-MODULES argument of the
build function to skip them.
Git commit captured by pin-build-version before the image dump.
The binary usually runs on a host without .git and sources (ProtectHome=read-only), so the commit has to be captured at build time. Projects use this variable in their version flag output.
40ANTS-BUILDER/DEPENDENCIES
Functions
Load everything which has to be present in the dumped image.
This is the Deploy library, the SYSTEM being built and the SLYNK-MODULES.
Variables
Slynk contrib modules loaded into the image before the dump.
The Slynk server tries to load these modules when a client connects
and cannot do that in production, where neither Quicklisp nor the
sources are reachable. Pass NIL as SLYNK-MODULES argument of the
build function to skip them.
40ANTS-BUILDER/ECL
Functions
Prepare the image so the dependencies' DEFPACKAGE forms can be read
on ECL.
Some dependencies (metacopy, in particular) read
.(if load-with-contextl '#:contextl (values)) inside DEFPACKAGE.
On SBCL, the reader inserts nothing into the list for zero values of
sharp-dot, while the ECL reader inserts NIL, which DEFPACKAGE turns
into :use of a package named "NIL", and ECL fails with "There
exists no package with name NIL". Creating an empty package with
this name makes such DEFPACKAGE forms valid. On other implementations
this is a no-op.
40ANTS-BUILDER/IMAGE
Functions
Dump the standalone binary of SYSTEM.
On SBCL and CCL this is asdf:make, which performs the system's
:build-operation (deploy-op for Pingvito). ECL does not support
uiop:dump-image, and deploy-op relies on it, so under ECL the standard
asdf:program-op is used instead: the system is compiled to C and the
object files are linked into an executable with c:build-program,
using the :entry-point and :build-pathname from the system's
definition file.
Add NICKNAME to the package named PACKAGE-NAME.
Does nothing when the package does not exist or when a package named
NICKNAME is already present, because in these cases rename-package
either has nothing to do or signals a correctable error. Returns T
when the package was renamed.
Install a debugger hook which fails the build process immediately.
Any error during the build must exit the process with a non-zero
code, instead of leaving it hanging in the interactive debugger of
Roswell/CCL.
Prepare the build image for the dump.
Gives the Clack Hunchentoot handler package its slash nickname: Clack looks handlers up by a name with slashes, does not find it in the current dist and calls ql:quickload at run time, which fails under ProtectHome. The nickname removes the runtime load.
Then pushes the :deploy-console feature to silence Deploy's status
output at service startup ("Performing warm boot", "Loading foreign
library ..." etc.): with this feature the warm boot sets
status-output to NIL, the same way deploy-console-op does for
console applications. Build identification stays available through
the project's version flag.
Finally, removes the build-only debugger hook, so it does not get dumped into the image: otherwise any runtime error would kill the service with a build failure message instead of the project's own error handling.
40ANTS-BUILDER/REGISTRY
Functions
Extend the ASDF registry with systems local to the project being built.
Projects keep a local-deps.lisp file next to their system definition file; it defines an extend-registry function which pushes local checkout directories onto asdf:*central-registry*. When the file is missing, nothing is registered and the call is a no-op, because a project may have all its dependencies on Quicklisp.
40ANTS-BUILDER/VERSION
Functions
Capture build-time identification of SYSTEM into the image.
Stores the git commit into git-commit and appends the commit to the system's version, updating its component-version. The version is taken from the system's component-version or, when it is unset, from the topmost entry of its docs/changelog.lisp.
Called before the image dump, so that the binary's version flag reports the state of the sources it was built from, and not of whatever is on disk at run time. Returns the commit.
Return the topmost version from SYSTEM's docs/changelog.lisp or NIL.
This mirrors what the 40ants-asdf-system class does on an explicit load-system: the first changelog entry defines the version. Forms are read into a temporary package, because the changelog macro and version symbols are not interned anywhere in the build image.
Variables
Git commit captured by pin-build-version before the image dump.
The binary usually runs on a host without .git and sources (ProtectHome=read-only), so the commit has to be captured at build time. Projects use this variable in their version flag output.