Common Lisp Docs Builder

DOCS-BUILDER ASDF System Details

This system is a generic documentation builder for Common Lisp Systems. It able to generate HTML documentation for specified ASDF system.

The idea is to use docs-builder as an universal HTML documentation builders which can be used in a continuous integration pipeline. For example, it is used inside build-docs GitHub action, which can be used to build docs update gh-pages for any Common Lisp library (if it is uses documentation generator supported by docs-builder).

Currently Docs Builder supports only 40ants-doc, MGL-PAX and Geneva, but it can be extended to support other documentation builders, covered by examples in here: cl-doc-systems.github.io.

Usage

Documentation can be built in a few ways: from the lisp REPL, command-line and using the GitHub action.

REPL

From the REPL, you need first to call a docs-builder:build function:

function
docs-builder:build
system &rest rest &key (error-on-warnings t) &allow-other-keys

Builds HTML documentation for ASDF system and returns absolute path to the dir with docs.

Inside, it will try to guess which documentation builder should be used:

generic-function
docs-builder/guesser:guess-builder
system

Returns a builder object which can be passed to the docs-builder/builder:build generic-function along with system.

The builder type is guessed using different euristics which depends on a documentation system.

If you want to add support for a new documentation generator, use defguesser macro.

Then it will pass the builder object and ASDF system to the docs-builder/builder:build generic-function:

generic-function
docs-builder/builder:build
builder system &key local root-sections &allow-other-keys

Builds HTML documentation for ASDF system and returns absolute path to the dir with docs.

Here is an example how to build documentation for :docs-builder ASDF system:

CL-USER> (docs-builder:build :docs-builder)
 <INFO> [02:12:00] docs-builder/core core.lisp (build :before system) -
  Building docs for system #<PACKAGE-INFERRED-SYSTEM "docs-builder"> found at /Users/art/projects/docs-builder/
 <INFO> [02:12:00] docs-builder/builders/mgl-pax/builder builder.lisp (build builder system) -
  Building docs in "/Users/art/projects/docs-builder/docs/build/" dir
#P"/Users/art/projects/docs-builder/docs/build/"

Command-line

You can use builder from command-line. To do this, first install it using Roswell:

# Note, we need to install this patched mgl-pax
# first, to be able to load docs-builder.
# This step in necessary until this pull
# will be merged:
# https://github.com/melisgl/mgl-pax/pull/8

$ ros install 40ants/doc

$ ros install 40ants/docs-builder

Here we call it to build documentation for "docs-builder" ASDF system:

$ build-docs docs-builder
 <INFO> [02:26:32] docs-builder/main main.lisp (main) -
  Quickloading system "docs-builder"
 <INFO> [02:26:34] docs-builder/core core.lisp (build :before system) -
  Building docs for system #<PACKAGE-INFERRED-SYSTEM "docs-builder"> found at /Users/art/projects/docs-builder/
 <INFO> [02:26:34] docs-builder/builders/mgl-pax/builder builder.lisp (build builder system) -
  Building docs in "/Users/art/projects/docs-builder/docs/build/" dir
Scan was called 2146 times.

GitHub Action

If you host your project on the GitHub, then the most easy way to build and host documentation would be to use Github Pages.

To build docs and update the site, create a file .github/workflows/docs.yml with a content like this:

name: 'Docs'

on:
  # This will run tests on pushes
  # to master branch and every monday:
  push:
    branches:
      - 'main'
      - 'master'
  schedule:
    - cron:  '0 10 * * 1'

jobs:
  build-docs:
    runs-on: ubuntu-latest
    
    env:
      LISP: sbcl-bin

    steps:
      - uses: actions/checkout@v1
      - uses: 40ants/setup-lisp@v1
      - uses: 40ants/build-docs@v1
        with:
          asdf-system: cl-info

You'll find more info in the action's documentation.

Additional Params

You can customize a builder by defining a method for this generic function:

generic-function
docs-config:docs-config
asdf-system

Should return a plist which will be passed as keyword arguments to the documentation builder when building docs for a given asdf-system.

Implement a method, EQL specialized on a concrete ASDF system.

Here is a typical method I use for my own libraries to set a custom theme for 40ants-doc system:

(defmethod docs-config ((system (eql (asdf:registered-system "cl-info"))))
  ;; 40ANTS-DOC-THEME-40ANTS system will bring
  ;; as dependency a full 40ANTS-DOC but we don't want
  ;; unnecessary dependencies here:
  (uiop:symbol-call :ql :quickload :40ants-doc-theme-40ants)
  (list :theme
        (find-symbol "40ANTS-THEME"
                     (find-package "40ANTS-DOC-THEME-40ANTS"))))

Try to load additional dependencies inside the method. This users of your library will not download dependencies needed only for building documentation.

For some special cases it might be useful to return a special key DYNAMIC-BINDINGS. This could be useful, for configuring some custom extensions like it did with interactive demos for the Weblocks. Here is how a method looks like when I configure Weblocks documentation builder:

(defmethod docs-config ((system (eql (asdf:registered-system "weblocks"))))
  ;; ...
  (list :theme
        (find-symbol "40ANTS-THEME"
                     (find-package "40ANTS-DOC-THEME-40ANTS"))
        :dynamic-bindings (list (cons 'weblocks/doc/example:*server-url*
                                      ;; When local examples server is running,
                                      ;; we'll be using it instead of production:
                                      (unless weblocks/doc/example::*port*
                                        "http://examples.40ants.com/"))))

Extending

Docs builder was made to be extensible and here we see how to add support for a new documentation generator. A documentation builder we'll use is a Geneva.

Some time ago I've wrote a template system to demonstrate how to document ASDF system using Geneva. Lets try to automate docs building and add Geneva support to the docs-builder!

First, we need a way to detect if the system uses the Geneva. This is done using "guesser". Let's define a guesser which will consider we are working with Geneva, if there is a docs/source/index.mk2 file. Files with *.mk2 extensions are special markup used by Geneva.

Add a Builder Class

First thing we need to do is to create a builder class.

Create a src/builders/geneva/builder.lisp file with following content:

(defclass builder ()
  ())

Guessing a Doc Generator

DOCS-BUILDER uses a number of euristics to determine the CLOS class to be used for documentation generation. Euristics are incapulated in "guessers". Let's create a src/builders/geneva/guesser.lisp file and define a simple guesser, which will return the builder class defined in the previous section if a file docs/sources/index.mk2 exists.

To define a guesser, we'll be using docs-builder/guesser:defguesser macro:

macro
docs-builder/guesser:defguesser
name (system) &body body
(docs-builder/guesser:defguesser geneva (system)
  (when (probe-file
         (asdf:system-relative-pathname system
                                        "docs/source/index.mk2"))
    (uiop:symbol-call :ql :quickload :docs-builder/builders/geneva/builder
                          :silent t)
    (make-instance (intern "BUILDER" "DOCS-BUILDER/BUILDERS/GENEVA/BUILDER"))))

When all rules matched, guesser loads the builder package and all its dependencies. In our case, Geneva system will be loaded.

Add this guesser file to the docs-builder.asd file:

(defsystem "docs-builder" 
  :class :package-inferred-system
  :author "Alexander Artemenko"
  :license "Unlicense"
  :pathname "src"
  :description ""
  :defsystem-depends-on ("40ants-doc")
  :depends-on ("docs-builder/core"
               "docs-builder/builders/40ants-doc/guesser"
               "docs-builder/builders/mgl-pax/guesser"
               "docs-builder/builders/geneva/guesser"))

This way, it will be loaded along with the primary system while geneva/builder and it's dependencies will be loaded only if the system we are building documentation for is using Geneva.

Now we can call docs-builder/guesser:guess-builder generic-function to create a builder for example system:

CL-USER> (docs-builder/guesser:guess-builder :example)
#<DOCS-BUILDER/BUILDERS/GENEVA/BUILDER::BUILDER {1003D89313}>

Add a Build Method

Now open a src/builders/geneva/builder.lisp file again and add a method to the docs-builder/builder:build generic-function. The method should build HTML documentation and return a path to the folder.

(defmethod docs-builder/builder:build ((builder builder) (system asdf:system))
  (let* ((docs-source-dir
           (asdf:system-relative-pathname system
                                          "docs/source/"))
         (docs-output-dir
           (asdf:system-relative-pathname system
                                          "docs/build/"))
         (index-input-filename
           (uiop:merge-pathnames* docs-source-dir
                                  "index.mk2"))
         (index-output-filename
           (uiop:merge-pathnames* docs-output-dir
                                  "index.html"))
         (document (with-open-file (s index-input-filename)
                     (geneva.mk2:read-mk2 s))))
    (ensure-directories-exist docs-output-dir)
    
    (uiop:with-output-file (s index-output-filename
                              :if-exists :supersede)
      (geneva.html:render-html document :stream s)
      ;; Return a directory with resulting docs:
      docs-output-dir)))

Finally, we can build our documentation:

CL-USER> (docs-builder:build :example)
 <INFO> [23:53:48] docs-builder/builder builder.lisp (build :around system) -
  Building docs for system #<PACKAGE-INFERRED-SYSTEM "example"> found at /Users/art/cl-doc-systems/geneva/
#P"/Users/art/cl-doc-systems/geneva/docs/build/"

Of cause, in reality this method could be a more complex. It should process all *.mk2 files and build API reference for the primary system and all package inferred subsystems.

Supported Docs Generators

40ANTS-DOC

This guesser tries to find if your system depends on 40ants-doc system. If it is, then the 40ANTS-DOC will be used to build documentation.

During the build phase, the builder will try to find documentation sections not refereced from any other sections. For each root section, builder will create a separate HTML page. If there are few root sections, make sure one of them is having 40ants-doc name. Otherwise index.html page will not be created.

Algorithm searches section amongh all exported symbols. If you don't want it to find some root section, just pass :export nil to the 40ants-doc:defsection macro.

If you want your documentation link back to the GitHub sources, make sure you have either :homepage or :source-control in your ASDF definition:

(asdf:defsystem #:example
  :licence "MIT"
  :version "0.0.3"
  :author "John Doe"
  :mailto "john@doe.me"
  :homepage "https://github.com/john-doe/example"
  :source-control (:git "https://github.com/john-doe/example")
  ...)

Note, that this builder not only renders HTML documentation, but also updates README files in the system's root directory.

What is next

MGL-PAX

This guesser tries to find if your system depends on MGL-PAX system. If it is, then the MGL-PAX will be used to build documentation.

During the build phase, the builder will try to find MGL-PAX sections not refereced from any other sections. For each root section, builder will create a separate HTML page. If there are few root sections, make sure one of them is having \mgl-pax name. Otherwise index.html page will not be created.

Algorithm searches section amongh all exported symbols. If you don't want it to find some root section, just pass :export nil to the MGL-PAX:DEFSECTION macro.

If you want your documentation link back to the GitHub sources, make sure you have either :homepage or :source-control in your ASDF definition:

(asdf:defsystem #:example
  :licence "MIT"
  :version "0.0.3"
  :author "John Doe"
  :mailto "john@doe.me"
  :homepage "https://github.com/john-doe/example"
  :source-control (:git "https://github.com/john-doe/example")
  ...)

Note, that this builder not only renders HTML documentation, but also updates README files in the system's root directory.

Geneva

This guesser tries to find a file docs/sources/index.mk2 and if it exists then Geneva documentation generator will be used.

What is next

API

condition
docs-builder:documentation-has-problems
reader
docs-builder:num-of-warnings
(:num-of-warnings)

Utilities

function
docs-builder/utils:external-dependencies
system &key all

Returns a list of string with names of external dependencies of the system.

It works with package-inferred systems too, recursively collecting external-dependencies of all subsystems.

Warning! By default, this function does not return dependencies of dependencies. To get them all, add :ALL T option.

generic-function
docs-builder/utils:system-packages
system

Returns a list of packages created by ASDF system.

Default implementation returns a package having the same name as a system and all packages matched to package-inferred subsystems:

CL-USER> (docs-builder/utils:system-packages :docs-builder)
(#<PACKAGE "DOCS-BUILDER">
 #<PACKAGE "DOCS-BUILDER/UTILS">
 #<PACKAGE "DOCS-BUILDER/GUESSER">
 #<PACKAGE "DOCS-BUILDER/BUILDERS/GENEVA/GUESSER">
 #<PACKAGE "DOCS-BUILDER/BUILDER">
 #<PACKAGE "DOCS-BUILDER/BUILDERS/MGL-PAX/GUESSER">
 #<PACKAGE "DOCS-BUILDER/DOCS">
 #<PACKAGE "DOCS-BUILDER/BUILDERS/MGL-PAX/BUILDER">)

This function can be used by builder to find pieces of documentation. For example, mgl-pax builder uses it to find documentation sections.

Roadmap