Common Lisp Project of the Day

litterae

Fri, 26 Mar 2021 00:55:49 +0300

This is yet another Common Lisp documentation builder.

It renders beautiful one-page docs. Sadly, Litterae itself has no documentation. I've used it to document Teddy and also created a template project for you:

https://github.com/cl-doc-systems/litterae

Why Litterae can be interesting for you:

It uses README.md as documentation source.
Litterae builds API reference from docstrings.
It provides four color themes.
It uses LSX for templating and probably there is a way to create a custom HTML.

However, there are many problems with Litterae. It seems not mature enough to use in my projects. To name a single feature I miss is cross-referencing. That is the main reason, why I'll not use it for my libraries.

declt

Sun, 17 Jan 2021 16:58:46 +0300

This is the documentation builder behind Quickref site. It is good for generating API references for third party libraries.

Most interesting features of Declt are:

Declt uses Texinfo file format for intermediate document store. This makes it possible to generate not only HTML but also PDF and other output formats.
It can automatically include license text into the documentation. But this works only for a number of popular licenses like MIT, BSD, GPL, LGPL and BOOST.

As always, I've created a template project, ready to be used:

https://github.com/cl-doc-systems/declt

Here is how it is rendered in HTML:

https://cl-doc-systems.github.io/declt/

And in PDF:

https://cl-doc-systems.github.io/declt/index.pdf

Sadly, Declt does not support markup in docstrings and cross-referencing does not work there.

Some other pros and cons are listed on example site.

Remember, all example projects from https://github.com/cl-doc-systems include a build script and GitHub Action to update documentation on every commit!

atdoc

Thu, 31 Dec 2020 22:45:28 +0300

This is yet another documentation builder for CL.

Its unique features are a special markup and ability to render not only HTML, but also PDF and Info files.

As always, I've created an example project which can be used as a template for your own library. Here how it is rendered in HTML:

https://cl-doc-systems.github.io/atdoc/

This is how it is rendered into PDF:

Beautiful. Isn't it?

More pros and cons of ATDOC are listed in the repository's README:

https://github.com/cl-doc-systems/atdoc

Remember, all example projects from https://github.com/cl-doc-systems include a build script and GitHub Action to update documentation on every commit!

cl-api

Sat, 26 Dec 2020 22:04:59 +0300

This is a small and simple documentation builder. It was removed from Quicklisp in 2014 because this project is SBCL only, but I've added it to the Ultralisp and you can test it after upgrade to the latest version.

CL-API is suitable for building a reference for third-party libraries if they don't have their own documentation. But lack of ability to process handwritten chapters and work with package inferred systems, make it unusable for 40ants projects.

As always, I've created a template repository for you.

Here is an example project's documentation built with CL-API:

https://cl-doc-systems.github.io/cl-api/

Use this template if you are making a small library which needs autogenerated API reference.

Also, you'll find a "Pros & Cons" section in the README:

https://github.com/cl-doc-systems/cl-api

Here you will find template projects for other documentation systems.

Choose what is more suite your needs:

https://github.com/cl-doc-systems

eazy-documentation

Sun, 20 Dec 2020 00:59:54 +0300

This is yet another documentation generator for Common Lisp, built by Masataro Asai.

Its unique feature is the documentation processor which is able to extract docstring from nonstandard Lisp forms. Also, it supports all markups, supported by Pandoc and can be used to generate documentation from any folder.

You'll find more pros and cons in the template repository I've prepared for you.

Despite many cool features, I have these stoppers for using Eazy Documentation for my own projects:

it is hard to control sections ordering;
there is no helper for cross-referencing symbols.

MGL-PAX, reviewed recently is still my favourite.

But Eazy Documentation still can be useful when:

the system is small;
you just have a number of RST/Markdown/other files and want to make a site;
you want to build a doc for a third-party library. It can build a doc for any ASDF system.

cl-gendoc

Sat, 12 Dec 2020 23:59:43 +0300

This is yet another CL documentation generator by Ryan Pavlik. It's interesting features are:

Markdown support out of the box.
New markups can be easily added.
Code from snippets can be linked to CLHS.

As always, I've prepared an example project for you:

https://cl-doc-systems.github.io/cl-gendoc/

You'll find there a full list of cl-gendoc's pros and cons.

Here is a short example of library documentation builder. It includes a few markdown sections and an autogenerated API spec for two packages:

(defun build ()
  (let ((output-filename "docs/build/index.html"))
    (ensure-directories-exist output-filename)
    (gendoc:gendoc (:output-filename output-filename
                    :css "simple.css")
      (:markdown-file "docs/source/index.md")
      (:markdown-file "docs/source/pros-and-cons.md")
      (:markdown-file "docs/source/handwritten.md")
      (:markdown-file "docs/source/reference.md")
      (:apiref :example/app
               :example/utils))))

The cl-gendoc has an interesting macro define-gendoc-load-op. It forces documentation build after the asdf:load-system call. Here is how it can be used:

(defsystem example-docs
  :class :package-inferred-system
  :defsystem-depends-on ("cl-gendoc")
  :pathname "docs/scripts/"
  :depends-on ("example-docs/builder"))

(gendoc:define-gendoc-load-op :example-docs
                              :example-docs/builder 'build)

The macro call will expand into:

(progn
 (defmethod asdf/action:perform :after ((o asdf/lisp-action:load-op) 
                                        (c (eql (asdf/system:find-system
                                                 :example-docs))))
   (let ((fn
          (find-symbol (symbol-name 'build)
                       (find-package :example-docs/builder))))
     (funcall gendoc::fn)))

 ;; This method makes ASDF think that load-system wasn't successful
 ;; and subsequent call will build documentation again.
 (defmethod asdf/action:operation-done-p ((o asdf/lisp-action:load-op) 
                                          (c (eql (asdf/system:find-system 
                                                   :example-docs))))
   nil))

Personally, I'm don't like this hack. But I'm sure there should be the more correct way to use asdf:make to build docs. If you know how to do this, please, let me know in the comments.

codex

Thu, 03 Dec 2020 22:02:34 +0300

This is a documentation generator by Fernando Boretti.

It is built for extensibility and uses CommonDoc as an internal format for document representation and manipulation.

Another interesting feature is a custom markup language Scriba (reviewed in post #0178). Actually, Codex supports any number of markup languages, you just need to write a parser from this markup into CommonDoc.

The sad part is that there is no support for cross-referencing and there are a number of bugs :(

As always, I've created a template project for you, demonstrating how to use Codex to document a simple Common Lisp library:

https://github.com/cl-doc-systems/codex

Read about other pros and cons on this page:

https://cl-doc-systems.github.io/codex/pros-&-cons-of-codex.html

mgl-pax

Mon, 30 Nov 2020 00:43:06 +0300

This is a very cool documentation generator for Common Lisp projects.

The most interesting features are:

Emacs/SLIME integration.
Ability to generate Markdown.
API to add new entity types.
Linking to the sources on the GitHub.
Docstrings deindentation.
Generation docs for multiple ASDF systems with cross-referencing.
Auto-export of documented symbols.

Some cons:

The recommended way to mix documentation section with code leads to the runtime dependency from PAX and all it's dependencies. But you might define documentation as a separate ASDF system.
It is inconvenient to write Markdown in docstrings. Is there any way to teach Emacs to use markdown minor mode for documentation strings?

There are more pros and cons. All of them are listed in the example project. It is up and ready to be cloned. Use it as a template for your own Common Lisp library with great documentation!

You'll find all such templates in this GitHub organization:

https://github.com/cl-doc-systems

I have plans to review a few other documentation builders, but MGL-PAX is my favourite so far.

staple

Tue, 10 Nov 2020 01:10:25 +0300

Today we'll look at @Shinmera's documentation system called Staple.

As always, I've created the skeleton project for you.

But I must warn you. This skeleton is almost unusable, because of a few problems.

The first problem is that I wasn't able to make markdown work for the docstrings. When I do setup like suggests Staple documentation, it does not see all package of my system.

The second problem is that for some reason it works badly with my simple example system. It shows on the index page multiple equal links to the Example system.

Probably both problems are related to the ASDF system type. I'm using package inferred systems.

Also, the Staple's error messages a complete mess. It is not possible to figure out the origin of a warning. The output looks like that:

Scan was called 76 times.
Warning: could not find hyperspec map file. Adjust the path at the top of clhs-lookup.lisp to get links to the HyperSpec.
Scan was called 384 times.
Scan was called 116 times.
Scan was called 287 times.
Scan was called 94 times.
Scan was called 2005 times.
Scan was called 162 times.
Scan was called 162 times.
Scan was called 186 times.
Scan was called 397 times.
Scan was called 221 times.
Scan was called 502 times.
Scan was called 134 times.
Scan was called 130 times.
Scan was called 395 times.
Scan was called 1453 times.
WARN: Error during code markup: Invalid function name: STR:DOWNCASE
WARN: Error during code markup: Symbol name must not end with a package marker (the : character).
Scan was called 2782 times.
WARN: Error during code markup: Unquote not inside backquote.
Scan was called 95 times.
Scan was called 2019 times.
Scan was called 4518 times.
Scan was called 2670 times.
Scan was called 2209 times.
WARN: Error during code markup: A token consisting solely of multiple dots is illegal.
WARN: Error during code markup: Invalid function name: DOCPARSER:PARSE
WARN: Error during code markup: The character < is not a valid sub-character for the # dispatch macro.
WARN: Error during code markup: The character < is not a valid sub-character for the # dispatch macro.
WARN: Error during code markup: Symbol name must not end with a package marker (the : character).
WARN: Error during code markup: Invalid function name: 4
WARN: Error during code markup: Invalid function name: STAPLE-SERVER:START
WARN: Error during code markup: Invalid function name: STAPLE-SERVER:STOP
WARN: Error during code markup: Invalid function name: LQUERY-TEST:RUN
WARN: Error during code markup: Invalid function name: 1
WARN: Error during code markup: Symbol name without any escapes must not consist solely of package markers (: characters).
WARN: Error during code markup: The value
                                  NIL
                                is not of type
                                  CHARACTER
WARN: Error during code markup: The value
                                  NIL
                                is not of type
                                  CHARACTER
WARN: Error during code markup: While reading backquote, expected an object when input ended.
WARN: Error during code markup: While reading backquote, expected an object when input ended.
WARN: Error during code markup: The value
                                  NIL
                                is not of type
                                  CHARACTER
WARN: Error during code markup: Symbol name without any escapes must not consist solely of package markers (: characters).
WARN: Error during code markup: If a symbol token contains two package markers, they must be adjacent as in package::symbol.
WARN: Error during code markup: The value
                                  NIL
                                is not of type
                                  CHARACTER
WARN: Error during code markup: While reading symbol, expected the character | when input ended.

Another hard thing is customization. Seems Staple was built for maximum flexibility and sometimes it is not obvious how to do such simple thing like adding yet another markdown page to the output.

Now to the good news :)

As I said Staple is very flexible. It uses CLOS and generic functions almost everywhere and you can extend it in a number ways. For example, you can add support for your own markup language.

There is also a unique feature: Staple automatically links every function documentation to its source on the GitHub:

Documentation for @Shinmera's libraries looks really good. Take look to the Radiance, for example:

https://shirakumo.github.io/radiance/

Probably Staple just needs more polishing to become a great documentation system for Common Lisp libraries?

For now Coo is my favourite. But in the next week or two, we'll look at a few other Common Lisp documentation systems.

Seems documentation building works quite unstable and sometimes does not build some pages where is running on the CI :(

coo

Fri, 06 Nov 2020 01:14:56 +0300

This system is not in the Quicklisp yet. You can install it from the GitHub or Ultralisp.org.

I think to make it a tradition – creating a ready to use project skeletons.

Here is one which uses Coo:

https://github.com/cl-doc-systems/coo/

Here is how it's rendered documentation looks like.

Read this doc to learn more about the pros and cons of this documentation system:

https://cl-doc-systems.github.io/coo/

Here I want to highlight only a few moments.

The system has a huge potential for improvement. There are some GitHub issues to work on. I've already sent my pull-request to add support of ASDF package inferred systems:

https://github.com/fisxoj/coo/pull/24

The two coolest features are:

Well-known markup format – Coo uses reStructured text.
It can be extended in Common Lisp.

Right now Coo is good for documenting small libraries which don't need for abstract chapters of documentation, but I see how it can be extended to render handwritten docs along with generated from docstrings!

sphinxcontrib-cldomain

Sun, 01 Nov 2020 03:43:43 +0300

This is an add-on to the Sphinx documentation system which allows using of information about Common Lisp packages in the documentation.

Initially, Sphinx was created for Python's documentation and now it is widely used not only for python libraries but also for many other languages.

Sphinx uses reStructured text markup language which is extensible. You can write your own extensions in Python to introduce new building blocks, called "roles".

sphinxcontrib-cldomain consists of two parts. The first part is a python extension to the Sphinx which adds an ability to render documentation for CL functions, methods and classes. The second - a command-line docstring extractor, written in CL.

Initially, cldomain was created by Russell Sim, but at some moment I've forked the repository to port it to the newer Sphinx, Python3 and Roswell.

The coolest feature of the cldomain is its ability to mix handwritten documentation with docstring. The second - cross-referencing. You can link between different docstrings and chapters of the documentation.

Today I will not show you any code snippets. Instead, I've created an example repository with a simple Common Lisp system and documentation:

https://cl-doc-systems.github.io/sphinxcontrib-cldomain/

This example includes a GitHub workflow to update the documentation on a push to the main branch and can be used as a skeleton for you own libraries.

The main thing I dislike in Sphinx and cldomain is the Python :( Other cons are the complexity of the markup and toolchain setup.

In the next few posts, I'll review a few other documentation tools for Common Lisp and try to figure out if they can replace Sphinx for me.

I think we as CL community must concentrate our efforts to improve the documentation level of our software and choosing the best setup which can be recommended for everybody is the key.

cl-pdf

Wed, 28 Oct 2020 23:49:29 +0300

This is the library for PDF generation and parsing.

Today I'm too lazy to provided step by step examples, but I have a real task to do with this library.

Some time ago I've read the article about productivity which recommended to print a "life calendar". This calendar should remind you: "The life is limited and the time's price is high."

The calendar is a grid where every box is one week of you life. The article suggested to buy a poster with the calendar, but I don't want to wait for a parcel with the poster! I want to print it now!

And here is where cl-pdf comes on the scene!

I wrote this simple function to generate the poster of A1 format:

(defun render (&optional (filename "life.pdf"))
  (flet ((to-ppt (size-in-mm)
           (/ size-in-mm 1/72 25.4)))
    (let* ((width (to-ppt 594))       ;; This is A1 page size in mm
           (height (to-ppt 841))
           (margin-top (to-ppt 70))
           (margin-bottom (to-ppt 30))
           (span  (to-ppt 2))
           (year-weeks 52)
           (years 90)
           (box-size (/ (- (- height (+ margin-top margin-bottom))
                            (* span (1- years)))
                        years))
           (boxes-width (+ (* box-size year-weeks)
                            (* span (1- year-weeks))))
           (boxes-height (+ (* box-size years)
                             (* span (1- years))))
           ;; horizontal margin depends on box size,
           ;; because we need to center them
           (margin-h (/ (- width boxes-width)
                        2))
           (box-radius (/ box-size 3))
           (helvetica (pdf:get-font "Helvetica")))
      (pdf:with-document ()
        (pdf:with-page (:bounds (rutils:vec 0 0 width height))
          ;; For debug
          ;; (pdf:rectangle margin-h margin-bottom
          ;;                boxes-width
          ;;                boxes-height
          ;;                :radius box-radius)
          (loop for year from 0 below years
                do (loop for week from 0 below year-weeks
                         for x = (+ margin-h (* week (+ box-size span)))
                         for y = (+ margin-bottom (* year (+ box-size span)))
                         do (pdf:rectangle x y box-size box-size :radius box-radius)))
          
          ;; The title
          (pdf:draw-centered-text
           (/ width 2)
           (+ margin-bottom
               boxes-height
               ;; space between text and boxes in mm
               (to-ppt 15))
           "LIFE CALENDAR"
           helvetica
           ;; font-size in mm
           (to-ppt 30))

          ;; Labels for weeks
          (let ((font-size
                  ;; We want labels to be slightly smaller than boxes
                  (* box-size 2/3)))
            (pdf:draw-right-text
             (+ margin-h
                 (/ box-size 4))
             (+ margin-bottom
                 boxes-height
                 ;; space between text and boxes in mm
                 (to-ppt 10))
             "Weeks of the year"
             helvetica
             font-size)
            
            (loop for week below year-weeks
                  do (pdf:draw-centered-text
                      (+ margin-h
                          (/ box-size 2)
                          (* week (+ box-size span)))
                      (+ margin-bottom
                          boxes-height
                          ;; space between text and boxes in mm
                          (to-ppt 3))
                      (rutils:fmt "~A" (1+ week))
                      helvetica
                      font-size))

            ;; Labels for years
            (pdf:with-saved-state
              (pdf:translate
               (- margin-h
                   (to-ppt 10))
               (- (+ margin-bottom
                      boxes-height)
                   (/ box-size 4)))
              (pdf:rotate 90)
              (pdf:draw-left-text
               0 0
               "Years of your life"
               helvetica
               font-size))
            
            (loop for year below years
                  do (pdf:draw-left-text
                      (- margin-h
                          ;; space between text and boxes in mm
                          (to-ppt 3))
                      (+ margin-bottom
                          (/ box-size 4)
                          (* year (+ box-size span)))
                      (rutils:fmt "~A" (- years year))
                      helvetica
                      font-size))

            ;; The Question.
            (pdf:draw-left-text
             (- width margin-h)
             (- margin-bottom
                 (to-ppt 10))
             "Is this the End?"
             helvetica
             (* font-size 2))
            
            (pdf:close-and-stroke)))
        (pdf:write-document filename)))))

Here is how result will look like:

The PDF can be downloaded here.

This program demonstrates a few features of cl-pdf:

ability to set page size;
text drawing and rotation;
font manipulation.

There are a lot more features but all of them aren't documented, only several examples :(

GitHub shows 4 forks with some patches. And some of them are turned into a pull-request, but maintainer is inactive on the GitHub since 2019 :(

cl-async-await

Mon, 26 Oct 2020 23:48:09 +0300

This library implements the async/await abstraction to make it easier to write parallel programs.

Now we'll turn "dexador" http library calls into async and will see if we can parallel 50 requests to the site which response in 5 seconds.

To create a function which can return a delayed result, a "promise", we have to use cl-async-await:defun-async:

POFTHEDAY> (cl-async-await:defun-async http-get (url &rest args)
             (apply #'dexador:get url args))

Now let's call this function. When called it returns a "promise" object not the real response from the site:

POFTHEDAY> (http-get "https://httpbin.org/delay/5")
#<CL-ASYNC-AWAIT:PROMISE Not awaited>

Now we can retrieve the real result, using cl-async-await:await function:

POFTHEDAY> (cl-async-await:await *)
"{
  \"args\": {}, 
  \"data\": \"\", 
  \"files\": {}, 
  \"form\": {}, 
  \"headers\": {
    \"Accept\": \"*/*\", 
    \"Content-Length\": \"0\", 
    \"Host\": \"httpbin.org\", 
    \"User-Agent\": \"Dexador/0.9.14 (SBCL 2.0.8); Darwin; 19.5.0\", 
    \"X-Amzn-Trace-Id\": \"Root=1-5f9732d6-148ee9a305fab66c26a2dbfd\"
  }, 
  \"origin\": \"188.170.77.131\", 
  \"url\": \"https://httpbin.org/delay/5\"
}
"
200 (8 bits, #xC8, #o310, #b11001000)
#<HASH-TABLE :TEST EQUAL :COUNT 7 {1002987DE3}>
#<QURI.URI.HTTP:URI-HTTPS https://httpbin.org/delay/5>
#<CL+SSL::SSL-STREAM for #<FD-STREAM for "socket 192.168.43.216:49762, peer: 35.170.225.136:443" {10085B0BF3}>>

If we look a the promise object again, we'll see it has a state now:

POFTHEDAY> **
#<CL-ASYNC-AWAIT:PROMISE (VALUES {
  "args": {}, 
  "data": "", 
  "files": {}, 
  "form": {}, 
  "headers": {
    "Accept": "*/*", 
    "Content-Length": "0", 
    "Host": "httpbin.org", 
    "User-Agent": "Dexador/0.9.14 (SBCL 2.0.8); Darwin; 19.5.0", 
    "X-Amzn-Trace-Id": "Root=1-5f9732d6-148ee9a305fab66c26a2dbfd"
  }, 
  "origin": "188.170.77.131", 
  "url": "https://httpbin.org/delay/5"
}

  200 #<HASH-TABLE :TEST EQUAL :COUNT 7 {1002987DE3}>
  https://httpbin.org/delay/5
  #<SSL-STREAM for #<FD-STREAM for "socket 192.168.43.216:49762, peer: 35.170.225.136:443" {10085B0BF3}>>) >

Ok, it is time to see if we can retrieve results from this site in parallel. To make it easier to test speed, I'll wrap all code into the separate function.

The function returns the total number of bytes in all 50 responses:

POFTHEDAY> (defun do-the-test ()
             (let ((promises
                     (loop repeat 50
                           collect (http-get "https://httpbin.org/delay/5"
                                             :use-connection-pool nil
                                             :keep-alive nil))))
               ;; Now we have to fetch results from our promises.
               (loop for promise in promises
                     for response = (cl-async-await:await
                                     promise)
                     summing (length response))))

POFTHEDAY> (time (do-the-test))
Evaluation took:
  6.509 seconds of real time
  2.496912 seconds of total run time (1.672766 user, 0.824146 system)
  38.36% CPU
  14,372,854,434 processor cycles
  1,519,664 bytes consed
  
18300

As you can see, the function returns in 6.5 seconds instead of 250 seconds! This means cl-async-await works!

The only problem I found was this concurrency issue:

https://github.com/j3pic/cl-async-await/issues/3

But probably it is only related to Dexador.

parseq

Fri, 23 Oct 2020 23:47:49 +0300

With this library, you can write parsers to process strings, lists and binary data!

Let's take a look at one of the examples. It is a parser for the dates from RFC 5322. This format is used in email messages:

Thu, 13 Jul 2017 13:28:03 +0200

Parser consist of rules, combined in different ways. We'll go through the parser's parts one by one.

This simple rule matches one space character:

POFTHEDAY> (parseq:defrule FWS ()
               #\space)

;; It matches if string contains one space
POFTHEDAY> (parseq:parseq 'FWS
                          " ")
#\ 
T

;; But not on string from many spaces:
POFTHEDAY> (parseq:parseq 'FWS
                          "   ")
NIL
NIL

;; And of cause not on some other string
POFTHEDAY> (parseq:parseq 'FWS
                          "foo")
NIL
NIL

The next rule we need is the rule to parse hours, minutes and seconds. These parts have two digits and we'll use rep expression to specify how many digits matches the rule:

POFTHEDAY> (parseq:defrule hour ()
               (rep 2 digit))

POFTHEDAY> (parseq:parseq 'hour
                          "15")
(#\1 #\5)
T

See, this rule returns digits as the list! But to make it useful, we need the integer. Parseq rules support different kinds of transformations. They can are optional and can be specified like this:

;; This transformation will return as the string instead of list:
POFTHEDAY> (parseq:defrule hour ()
               (rep 2 digit)
             (:string))

POFTHEDAY> (parseq:parseq 'hour
                          "15")
"15"
T

;; Now we'll add a transformation from string to integer:
POFTHEDAY> (parseq:defrule hour ()
               (rep 2 digit)
             (:string)
             (:function #'parse-integer))

POFTHEDAY> (parseq:parseq 'hour
                          "15")
15 (4 bits, #xF, #o17, #b1111)
T

We'll define the minute and second rules the same way.

The next rule matches the abbreviated day of the week. It combines other rules or terms using or expression:

POFTHEDAY> (parseq:defrule day-of-week ()
               (or "Mon" "Tue" "Wed"
                   "Thu" "Fri" "Sat"
                   "Sun"))

POFTHEDAY> (parseq:parseq 'day-of-week
                          "Friday")
NIL
NIL

POFTHEDAY> (parseq:parseq 'day-of-week
                          "Fri")
"Fri"
T

;; The same way we define a rule for month abbrefiation
POFTHEDAY> (parseq:defrule month ()
               (or "Jan" "Feb" "Mar" "Apr"
                   "May" "Jun" "Jul" "Aug"
                   "Sep" "Oct" "Nov" "Dec"))

A little bit complex rule is used for matching timezone. Timezone is a string from 4 digits prefixed by plus or minus sign. We'll combine this knowledge using or/and expressions and will use option :string to get results as a single string:

POFTHEDAY> (parseq:defrule zone ()
               (and (or "+" "-")
                    (rep 4 digit))
             (:string))

POFTHEDAY> (parseq:parseq 'zone
                          "0300")
NIL
NIL

POFTHEDAY> (parseq:parseq 'zone
                          "+0300")
"+0300"
T

POFTHEDAY> (parseq:parseq 'zone
                          "-0300")
"-0300"
T

Now let's return to the time of day parsing. According to the RFC, seconds part is optional. Parseq has an expression ? to match optional rules.

Here is how a rule matching the time of day will look like:

POFTHEDAY> (parseq:defrule time-of-day ()
               (and hour
                    ":"
                    minute
                    (? (and ":" second))))

POFTHEDAY> (parseq:parseq 'time-of-day
                          "10:31:05")
(10 ":" 31 (":" 5))
T

To make the rule return only digits we have to use :choose transform. Choose extracts from results by index. You can specify index as an integer or as a list if you need to extract the value from the nested list:

POFTHEDAY> (parseq:defrule time-of-day ()
               (and hour
                    ":"
                    minute
                    (? (and ":" second)))
             (:choose 0 2 '(3 1)))

POFTHEDAY> (parseq:parseq 'time-of-day
                          "10:31:05")
(10 31 5)

;; Seconds are optional because of ? expression:
POFTHEDAY> (parseq:parseq 'time-of-day
                          "10:31")
(10 31 NIL)
T

;; This (:choose 0 2 '(3 1)) is equivalent to:
POFTHEDAY> (let ((r '(10 ":" 31 (":" 5))))
             (list (elt r 0)
                   (elt r 2)
                   (elt (elt r 3)
                        1)))
(10 31 5)

Another interesting transformation rule is :flatten. It is used to "streamline" result having nested structure and used in this rule which matches both time of day and timezone:

;; Without flatten we'll get nested lists:
POFTHEDAY> (parseq:defrule time ()
               (and time-of-day FWS zone)
             (:choose 0 2))

POFTHEDAY> (parseq:parseq 'time
                          "10:31 +0300")
((10 31 NIL) "+0300")

POFTHEDAY> (parseq:defrule time ()
               (and time-of-day FWS zone)
             (:choose 0 2)
             (:flatten))

;; Pay attention, :flatten removes nils:
POFTHEDAY> (parseq:parseq 'time
                          "10:31 +0300")
(10 31 "+0300")
T

Now, knowing how rules are combined and data is transformed, you will be able to read rest rules yourself:

POFTHEDAY> (parseq:defrule day ()
               (and (? FWS)
                    (rep (1 2) digit)
                    FWS)
             (:choose 1)
             (:string)
             (:function #'parse-integer))

POFTHEDAY> (parseq:defrule year ()
               (and FWS
                    (rep 4 digit)
                    FWS)
             (:choose 1)
             (:string)
             (:function #'parse-integer))

POFTHEDAY> (parseq:defrule date ()
               (and day month year))

(parseq:defrule date-time ()
    (and (? (and day-of-week ","))
         date
         time)
  (:choose '(0 0) 1 2)
  (:flatten))

Another cool Parseq's feature is an ability to debug parser execution. Now I'll turn on this debug mode and parse a string:

POFTHEDAY> (parseq:trace-rule 'date-time :recursive t)

POFTHEDAY> (parseq:parseq 'date-time
                          "Thu, 13 Jul 2017 13:28:03 +0200")
1: DATE-TIME 0?
 2: DAY-OF-WEEK 0?
 2: DAY-OF-WEEK 0-3 -> "Thu"
 2: DATE 4?
  3: DAY 4?
   4: FWS 4?
   4: FWS 4-5 -> #\ 
   4: FWS 7?
   4: FWS 7-8 -> #\ 
  3: DAY 4-8 -> 13
  3: MONTH 8?
  3: MONTH 8-11 -> "Jul"
  3: YEAR 11?
   4: FWS 11?
   4: FWS 11-12 -> #\ 
   4: FWS 16?
   4: FWS 16-17 -> #\ 
  3: YEAR 11-17 -> 2017
 2: DATE 4-17 -> (13 "Jul" 2017)
 2: TIME 17?
  3: TIME-OF-DAY 17?
   4: HOUR 17?
   4: HOUR 17-19 -> 13
   4: MINUTE 20?
   4: MINUTE 20-22 -> 28
   4: SECOND 23?
   4: SECOND 23-25 -> 3
  3: TIME-OF-DAY 17-25 -> (13 28 3)
  3: FWS 25?
  3: FWS 25-26 -> #\ 
  3: ZONE 26?
  3: ZONE 26-31 -> "+0200"
 2: TIME 17-31 -> (13 28 3 "+0200")
1: DATE-TIME 0-31 -> ("Thu" 13 "Jul" 2017 13 28 3 "+0200")

("Thu" 13 "Jul" 2017 13 28 3 "+0200")
T

We can improve this parser by using :function transformation to return a local-time:timestamp. First, let's redefine rule for matching the month and make it return the month number:

POFTHEDAY> (parseq:defrule january  () "Jan" (:constant 1))
POFTHEDAY> (parseq:defrule february () "Feb" (:constant 2))
POFTHEDAY> (parseq:defrule march    () "Mar" (:constant 3))
POFTHEDAY> (parseq:defrule april    () "Apr" (:constant 4))
POFTHEDAY> (parseq:defrule may      () "May" (:constant 5))
POFTHEDAY> (parseq:defrule june     () "Jun" (:constant 6))
POFTHEDAY> (parseq:defrule july     () "Jul" (:constant 7))
POFTHEDAY> (parseq:defrule august   () "Aug" (:constant 8))
POFTHEDAY> (parseq:defrule september () "Sep" (:constant 9))
POFTHEDAY> (parseq:defrule october  () "Oct" (:constant 10))
POFTHEDAY> (parseq:defrule november () "Nov" (:constant 11))
POFTHEDAY> (parseq:defrule december () "Dec" (:constant 12))

POFTHEDAY> (parseq:defrule month ()
               (or january february march april
                   may june july august
                   september october november december))

POFTHEDAY> (parseq:parseq 'month "Sep")
9 (4 bits, #x9, #o11, #b1001)
T

Next, we need to reimplement the rule matching a timezone to make it return local-time:timezone.

We'll be using an advanced technique of binding variables to pass value from one rule to another, because I want to store the timezone as a string and to parse it's hour and minute parts simultaneously.

To accomplish this task, we have to divide or timezone matching rule into two. The first rule will match it as a string of sign and four digits. Then it will save the result into an external variable and exit with a nil result to give a chance to execute the second rule:

POFTHEDAY> (parseq:defrule zone-as-str ()
               (and (or #\+ #\-)
                    (rep 4 digit))
             (:string)
             (:external zone-as-str)
             ;; Save the value into a variable:
             (:lambda (z)
               (setf zone-as-str z))
             ;; and just exit:
             (:test (z)
               (declare (ignore z))
               nil))

Now we'll redefine our zone rule to call zone-as-str first and then to parse the same text again, this time as hours and minutes. As the final step, it creates a local-time:timezone object:

POFTHEDAY> (parseq:defrule zone ()
               (or zone-as-str
                   (and (or #\+ #\-)
                        hour
                        minute))
             (:let zone-as-str)
             (:lambda (sign hour minute)
               (local-time::%make-simple-timezone
                zone-as-str
                zone-as-str
                ;; This is an offset in seconds:
                (+ (* (ecase sign
                        (#\+ 1)
                        (#\- -1))
                      hour
                      3600)
                   (* minute 60)))))

;; Here is the execution trace:
POFTHEDAY> (parseq:parseq 'zone
                          "+0300")
1: ZONE 0?
 2: ZONE-AS-STR 0?
 2: ZONE-AS-STR -|
 2: HOUR 1?
 2: HOUR 1-3 -> 3
 2: MINUTE 3?
 2: MINUTE 3-5 -> 0
1: ZONE 0-5 -> #<LOCAL-TIME::TIMEZONE +0300>
#<LOCAL-TIME::TIMEZONE +0300>
T

Now we need to redefine the original date-time rule, to create local-time:timestamp as the result:

POFTHEDAY> (parseq:parseq 'date-time
                          "Thu, 13 Jul 2017 13:28:03 +0200")
("Thu" 13 7 2017 13 28 3 #<LOCAL-TIME::TIMEZONE +0200>)
T

POFTHEDAY> (parseq:defrule date-time ()
               (and (? (and day-of-week ","))
                    date
                    time)
             (:choose '(1 2)                       ; year
                      '(1 1)                       ; month
                      '(1 0)                       ; day
                      '(2 0)                       ; hour
                      '(2 1)                       ; minute
                      '(2 2)                       ; second
                      '(2 3))                      ; timezone
             
             (:lambda (year month day hour minute second timezone)
               (local-time:encode-timestamp
                0             ; nanoseconds
                (or second 0) ; secs are optional
                minute
                hour
                day
                month
                year
                :timezone (or timezone
                              local-time:*default-timezone*))))

POFTHEDAY> (parseq:parseq 'date-time
                          "Thu, 13 Jul 2017 13:28:03 +0200")
@2017-07-13T14:28:03.000000+03:00
T

I've got a different value for the time because local-time prints timestamp in my timezone which is UTC+3.

The cool feature of the Parseq is its ability to work with any data, including binary. This way it can be used to parse binary formats.

As an example of parsing binary data, Parseq includes this parser rules for working with PNG image format:

https://github.com/mrossini-ethz/parseq/blob/master/examples/png.lisp

There are other interesting features. Please, read the docs to learn more.

If you are aware of other parsing libraries which worth to be written about, let me know in the comments.

pzmq

Mon, 19 Oct 2020 21:47:16 +0300

ZeroMQ is a networking library. It is not a message broker and it will not run tasks for you. Instead, it provides simple primitives for different network patterns.

With ZeroMQ you can easily implement these patterns: Request-Response, Pub-Sub, Push-Pull.

I found 3 CL systems implementing bindings to ZeroMQ:

pzmq - https://github.com/orivej/pzmq (ZeroMQ 4.0 bindings) active
zeromq https://github.com/freiksenet/cl-zmq ZeroMQ 3 (7 years ago)
zmq https://github.com/galdor/lisp-zmq Up to ZeroMQ 3.2 (7 year ago)

I know, names of the repositories, CL systems and packages are all different. That is the HELL :(

There is also at least two different versions of the zmq:

First one is referred by https://www.cliki.net/cl-zmq and included into Quicklisp. But examples from the ZeroMQ Guide not work with this zmq because msg-data-as-is function is absent.
The second one is https://github.com/tsbattman/cl-zmq and seems it is the version, used in ZeroMQ Guide. But it is not in the Quicklisp (yet or anymore).

Anyway, both of them are stale and didn't get updates 7-8 years. They are using the old 3.2 version of ZeroMQ. Today we'll talk about pzmq.

PZMQ has some activity in the repository and uses ZeroMQ 4. It does not have docs but it has some examples, ported from the ZeroMQ Guide.

I slightly modified the examples code, to make the output more readable when client and server are running from one REPL.

This snippet shows the server's code. It listens on the 5555 port and blocks until a message received, then responds and waits for another message:

POFTHEDAY> (defun hwserver (&optional (listen-address "tcp://*:5555"))
             (pzmq:with-context nil ; use *default-context*
               (pzmq:with-socket responder :rep
                 (pzmq:bind responder listen-address)
                 (loop
                   (write-line "SERVER: Waiting for a request... ")
                   (format t "SERVER: Received ~A~%"
                           (pzmq:recv-string responder))
                   (sleep 1)
                   (pzmq:send responder "World")))))

The client does the opposite - it sends some data and waits for the response. Depending on the pattern you use, you have to set socket types. For the server, we used :rep (reply) and for client we are using :req (request).

POFTHEDAY> (defun hwclient (&optional (server-address "tcp://localhost:5555"))
             (pzmq:with-context (ctx :max-sockets 10)
               (pzmq:with-socket (requester ctx) (:req :affinity 3 :linger 100)
                 ;; linger is important in case of (keyboard) interrupt;
                 ;; see http://api.zeromq.org/3-3:zmq-ctx-destroy
                 (write-line "CLIENT: Connecting to hello world server...")
                 (pzmq:connect requester server-address)
                 (dotimes (i 3)
                   (format t "CLIENT: Sending Hello ~d...~%" i)
                   (pzmq:send requester "Hello")
                   (write-string "CLIENT: Receiving... ")
                   (write-line (pzmq:recv-string requester))))))

Here is what we'll see when running the server in the background and starting the client in the REPL:

POFTHEDAY> (defparameter *server-thread*
             (bt:make-thread #'hwserver))

SERVER: Waiting for a request... 

POFTHEDAY> (hwclient)
CLIENT: Connecting to hello world server...
CLIENT: Sending Hello 0...
CLIENT: Receiving... Hello
SERVER: Waiting for a request... World
CLIENT: Sending Hello 1...
CLIENT: Receiving... Hello
SERVER: Waiting for a request... World
CLIENT: Sending Hello 2...
CLIENT: Receiving... Hello
SERVER: Waiting for a request... World
NIL

What is next?

Read about Pub-Sub and Push-Pull patterns at the ZeroMQ Guide and try to port them on pzmq.

Also, it would be cool to port all Common Lisp examples from the unsupported library to the pzmq and to send a pull-request.

By the way, there is at least one cool project, which already uses pzmq to connect parts written in Common Lisp and Python. It is recently reviewed common-lisp-jupyter library.

To conclude, this library definitely should be tried if you are going to implement a distributed application! Especially if it will interop with parts written in other languages than Common Lisp.

quickfork

Fri, 16 Oct 2020 22:32:51 +0300

This is an interesting system which provides information about other systems sources. Also, it is able to show commands, necessary to clone libraries into the local-projects dir.

This system is not in Quicklisp yet, but it can be installed from Ultralisp or by clone into some directory like /quicklisp/local-projects~.

Also, to make it work, you have to clone quicklisp-projects repository somewhere. This repository contains metadata about all projects in the Quicklisp:

POFTHEDAY> (uiop:run-program
            "git clone https://github.com/quicklisp/quicklisp-projects /tmp/projects")
POFTHEDAY> (setf quickfork::*projects-directory* "/tmp/projects/projects")

An interesting thing happens right after you load quickfork system. It installs a few hooks into Quicklisp and ASDF and begins tracking the systems which are installed during the ql:quickload:

POFTHEDAY> (ql:quickload :dexador)
To load "dexador":
  Load 14 ASDF systems:
    alexandria asdf babel bordeaux-threads cffi cffi-grovel
    cl-ppcre cl-utilities flexi-streams local-time
    split-sequence trivial-features trivial-gray-streams
    uiop
  Install 17 Quicklisp releases:
    chipz chunga cl+ssl cl-base64 cl-cookie cl-reexport
    dexador fast-http fast-io proc-parse quri smart-buffer
    static-vectors trivial-garbage trivial-mimes usocket
    xsubseq
; Fetching #<URL "http://beta.quicklisp.org/archive/usocket/2019-12-27/usocket-0.8.3.tgz">
; 83.84KB
...

; Loading "dexador"
...
[package cl+ssl]..................................
[package dexador].

Systems compiled by QL: 
 ("proc-parse"
  #P"/Users/art/poftheday/.qlot/dists/quicklisp/software/proc-parse-20190813-git/")
 ("xsubseq"
  #P"/Users/art/poftheday/.qlot/dists/quicklisp/software/xsubseq-20170830-git/")
  ...
 ("dexador"
  #P"/Users/art/poftheday/.qlot/dists/quicklisp/software/dexador-20200427-git/")
Systems loaded by QL: 
 ("proc-parse"
  #P"/Users/art/poftheday/.qlot/dists/quicklisp/software/proc-parse-20190813-git/")
 ("xsubseq"
  #P"/Users/art/poftheday/.qlot/dists/quicklisp/software/xsubseq-20170830-git/")
 ...
 ("dexador"
  #P"/Users/art/poftheday/.qlot/dists/quicklisp/software/dexador-20200427-git/")
Systems installed by QL: 
 "usocket"
 "trivial-mimes"
 ...
 "chipz"
 "dexador"
Inspect ql:*compiled-systems*, ql:*loaded-systems*, and ql:*installed-systems* for more info.
(:DEXADOR)

Also, there is a function quickfork::make-clone-commands which prints which commands should be executed in command-line to clone given system and all its dependencies.

Sadly, quickfork::make-clone-commands fails on dexador with some strange errors. You will need my fix, to make it work like this:

CL-USER> (quickfork::make-clone-commands :dexador)

git clone  "https://github.com/sharplispers/split-sequence.git"
git clone  "https://github.com/sionescu/static-vectors.git"
git clone  "https://github.com/sionescu/bordeaux-threads.git"
git clone  "https://github.com/fukamachi/dexador.git"
git clone  "https://github.com/fukamachi/fast-http.git"
git clone  "https://gitlab.common-lisp.net/alexandria/alexandria.git"
git clone  "https://github.com/fukamachi/proc-parse.git"
git clone  "https://github.com/cl-babel/babel.git"
git clone  "https://github.com/trivial-features/trivial-features.git"
git clone  "https://github.com/fukamachi/xsubseq.git"
git clone  "https://github.com/fukamachi/smart-buffer.git"
git clone  "https://github.com/trivial-gray-streams/trivial-gray-streams.git"
git clone  "https://github.com/fukamachi/quri.git"
git clone  "https://github.com/rpav/fast-io.git"
git clone  "https://github.com/fukamachi/cl-cookie.git"
git clone  "https://github.com/dlowe-net/local-time.git"
git clone  "https://github.com/Shinmera/trivial-mimes.git"
git clone  "https://github.com/sharplispers/chipz.git"
git clone  "https://github.com/takagi/cl-reexport.git"
git clone  "https://github.com/cl-plus-ssl/cl-plus-ssl.git"
git clone  "https://github.com/lmj/global-vars.git"
git clone  "https://github.com/trivial-garbage/trivial-garbage.git"

Non-git dependencies:
("cl-utilities" :HTTPS
 "https://common-lisp.net/project/cl-utilities/cl-utilities-latest.tar.gz")
NIL
("flexi-streams" :EDIWARE-HTTP "flexi-streams")
("uiop" :HTTPS "https://common-lisp.net/project/asdf/archives/uiop.tar.gz")
("cffi" :HTTPS
 "https://common-lisp.net/project/cffi/releases/cffi_latest.tar.gz")
("chunga" :EDIWARE-HTTP "chunga")
("cl-ppcre" :EDIWARE-HTTP "cl-ppcre")
("cl-base64" :KMR-GIT "cl-base64")
("usocket" :HTTPS
 "https://common-lisp.net/project/usocket/releases/usocket-latest.tar.gz")

Suddenly, I've remembered another similar project: ql-checkout.

Probably, yesterday we'll see how it works!

font-discovery

Wed, 14 Oct 2020 23:52:55 +0300

This is a library by @Shinmera to find out which fonts are known to OS and where their files are located.

Here is how you can list all "Arial" fonts and find where the "bold" version is located:

POFTHEDAY> (org.shirakumo.font-discovery:list-fonts :family "Arial")
(#<ORG.SHIRAKUMO.FONT-DISCOVERY:FONT "Arial" ROMAN REGULAR NORMAL>
 #<ORG.SHIRAKUMO.FONT-DISCOVERY:FONT "Arial" ITALIC REGULAR NORMAL>
 #<ORG.SHIRAKUMO.FONT-DISCOVERY:FONT "Arial" ROMAN BOLD NORMAL>
 #<ORG.SHIRAKUMO.FONT-DISCOVERY:FONT "Arial" ITALIC BOLD NORMAL>)

POFTHEDAY> (third *)
#<ORG.SHIRAKUMO.FONT-DISCOVERY:FONT "Arial" ROMAN BOLD NORMAL>

POFTHEDAY> (org.shirakumo.font-discovery:file *)
#P"/System/Library/Fonts/Supplemental/Arial Bold.ttf"

It is also possible to find a single font filtering it by family, slant and other parameters:

POFTHEDAY> (org.shirakumo.font-discovery:find-font :family "PragmataPro")
#<ORG.SHIRAKUMO.FONT-DISCOVERY:FONT "PragmataPro" ROMAN REGULAR NORMAL>

POFTHEDAY> (describe *)
#<ORG.SHIRAKUMO.FONT-DISCOVERY:FONT "PragmataPro" ROMAN REGULAR NORMAL>
  [standard-object]

Slots with :INSTANCE allocation:
  FILE                           = #P"/Users/art/Library/Fonts/PragmataProR_0828.ttf"
  FAMILY                         = "PragmataPro"
  SLANT                          = :ROMAN
  WEIGHT                         = :REGULAR
  SPACING                        = NIL
  STRETCH                        = :NORMAL

However, I found this library is still unstable on OSX and sometimes crashes somewhere in the CFFI code. @Shinmera has fixed some of these errors but some of them are still uncaught.

Read the full documentation on it here:

https://shinmera.github.io/font-discovery/

tesseract-capi

Mon, 12 Oct 2020 23:24:53 +0300

It has nothing in common with the magic artefact, but anyway is able to do pretty cool things!

This library is a wrapper around Tesseract OCR and is suitable for an image to text transformations.

To test this system, you need to install the Tesseract library first. On OSX just do:

brew install tesseract

The CL library is not in the Quicklisp but can be installed from the https://ultralisp.org Follow the instruction from the repository:

https://github.com/egao1980/tesseract-capi

To make it work, you'll need a trained data for Tesseract engine. Happily, the Homebrew package includes some files for the English language.

To find out a path, do:

[poftheday] echo $(brew --prefix tesseract)/share/tessdata/
/usr/local/opt/tesseract/share/tessdata/

Now we can test this library on this sample from the repository:

Here is how to translate this image into the text:

POFTHEDAY> (tesseract-capi::tessversion)
"4.1.1"

POFTHEDAY> (let ((tesseract-capi:*tessdata-directory*
                   "/usr/local/opt/tesseract/share/tessdata/"))
             (tesseract-capi:image-to-text
              (asdf:system-relative-pathname
               :tesseract-capi "tests/data/fox.png")))
"This is a lot of 12 point text to test the
ocr code and see if it works on all types
of fle format

The quick brown dog jumped over the
lazy fox. The quick brown dog jumped
over the lazy fox. The quick brown dog
jumped over the lazy fox. The quick
brown dog jumped over the lazy fox
"

Now let's try it on this XKCD comic:

POFTHEDAY> (let ((tesseract-capi:*tessdata-directory*
                   "/usr/local/opt/tesseract/share/tessdata/"))
             (tesseract-capi:image-to-text
              "docs/media/0203/xkcd-312.png"))
"A GODS LAMENT
SOME SAID THE WORLD SHOULD BE IN PERL;
SOME. SAID IN LISP
NOW, HAVING GIVEN BOTH A WHIRL,
I HELD WITH THOSE WHO FAVORED PERL.
BUT I FEAR WE PASSED TO MEN
A DISAPPOINTING FOUNDING MYTH,
AND SHOULD WE WRITE IT ALL AGAIN,
TO ENO IT WITH
A CLOSE -PAREN.
SY i
\\ ad an ran
cS co V
a, em oo
2 ee eee
@ P= : } 0.\" :
a OO
"

As you can see, it outputs some junk at the end, but main text is recognized almost perfectly!

I think, this is a pretty cool library! For example, you can use it plus Montezuma to build a personal search engine for your image and scans collection. It's a matter of one night.

cl-cont

Sat, 10 Oct 2020 23:52:05 +0300

This is a pretty old system which implements Delimited Continuations for Common Lisp. Initially, it was part of the Weblocks web-framework.

Sadly, but cl-cont has no documentation. I found only one example on this page.

It always was hard to wrap my mind around continuations. Probably that is why I decided to remove their support from the core of the Weblocks when I did the refactoring.

Now it is time to dive into continuations and probably to return them to Weblocks as an additional library.

Let's see what continuation is and how they can be used in practice!

The first thing to note is that each piece of code which uses this magic should be wrapped into with-call/cc. The second thing to remember is that let/cc form allows you to capture the moment and to save the execution point somewhere.

The code below prints three lines. It prints "Begin", then captures the execution point, prints "Before returning" and returns the captured point:

POFTHEDAY> (cont:with-call/cc
             (format t "Begin~%")
             (cont:let/cc k
               (format t "Before returning k~%")
               k)
             (format t "End~%")
             :final-result)
Begin
Before returning k
#<FUNCTION (LAMBDA (&OPTIONAL #:G1455 &REST #:G1456)) {22A10A0B}>

What has happened to our third print "End"? It didn't have a chance to be executed yet. But we can continue execution, by calling the function we've received as the result on the previous code snippet:

POFTHEDAY> (funcall *)
End
:FINAL-RESULT

POFTHEDAY> (funcall **)
End
:FINAL-RESULT

That is why it is called "continuation"! Yeah! As you can see, we can call this captured function any amount of times.

Now, let's try to create a function which will interrupt its execution and return a continuation.

Our first attempt might be like this:

POFTHEDAY> (defun foo ()
             (cont:with-call/cc
               (format t "Begin foo~%")
               (cont:let/cc k
                 (format t "Before returning k from foo~%")
                 k)
               (format t "End foo~%")
               :final-result))

POFTHEDAY> (cont:with-call/cc
             (format t "Before foo~%")
             (foo)
             (format t "After foo~%"))
Before foo
Begin foo
Before returning k from foo
After foo ;; Ups! I've expected it will not output this
NIL       ;; and return a continuation function instead of NIL!

As you can see, only half of our function was executed and then control flow continued, printed "After foo" and finished without giving us any continuation to play with :(

To make this code work as expected, we need to move with-call/cc form and make it wrap the function definition:

POFTHEDAY> (cont:with-call/cc
             (defun foo-wrapped ()
               (format t "Begin foo~%")
               (cont:let/cc k
                 (format t "Before returning k from foo~%")
                 k)
               (format t "End foo~%")
               :final-result))


POFTHEDAY> (cont:with-call/cc
             (format t "Before foo~%")
             (foo-wrapped)
             (format t "After foo~%"))
Before foo
Begin foo
Before returning k from foo
#<CLOSURE (LAMBDA (&OPTIONAL #:G1561 &REST #:G1562)) {10067F637B}>

This version works exactly as I've expected. It halts execution inside the foo's call and returns this continuation.

Now we can call continuation to continue computation of the foo function and the rest of our top-level form:

POFTHEDAY> (funcall *)
End foo
After foo
NIL

The latter case works because cont:with-call/cc is smart enough and if it wraps the function foo-wrapped into a special funcallable object:

;; This function is usual:
POFTHEDAY> (fdefinition 'foo)
#<FUNCTION FOO>

;; But this one is not.
;; It supports nested continuations:
POFTHEDAY> (fdefinition 'foo-wrapped)
#<CL-CONT::FUNCALLABLE/CC {10063435FB}>

Now let's adapt some examples from this Wikipedia article about continuations. The first example shows how to save continuation into the global variable and what happens when you use the same function to create the second continuation:

POFTHEDAY> (defvar *the-continuation*)

POFTHEDAY> (cont:defun/cc test ()
             (let ((i 0))
               ;; let/cc binds to k symbol a variable representing
               ;; this point in the program as the argument to
               ;; that function.
               ;;
               ;; In this case, we assigns that
               ;; continuation to the variable *the-continuation*
               ;; and then return the incremented value of 'i'.
               ;;
               (cont:let/cc k
                 (setf *the-continuation* k)
                 (incf i))

               ;; The next time *the-continuation* is called,
               ;; we start here:
               (incf i)))

POFTHEDAY> (test)
1

POFTHEDAY> (funcall *the-continuation*)
2

POFTHEDAY> (funcall *the-continuation*)
3

;; Stores the current continuation (which will print 4 next) away
POFTHEDAY> (defparameter *another-continuation* *the-continuation*)

;; Resets *the-continuation*:
POFTHEDAY> (test)
1

POFTHEDAY> (funcall *the-continuation*)
2

;; Uses the previously stored continuation:
POFTHEDAY> (funcall *another-continuation*)
4

The second example is more interesting because it let us create a simple framework for running green threads.

First, we need to define such two primitives: fork and yield:

POFTHEDAY> (defparameter *queue* nil)

POFTHEDAY> (defun empty-queue? ()
             (null *queue*))

POFTHEDAY> (defun enqueue (func)
             (setf *queue*
                   (append *queue*
                           (list func))))

POFTHEDAY> (defun dequeue ()
             (pop *queue*))

;; This stops running the current thread by placing it into the queue
;; and starts running a (func).
POFTHEDAY> (cont:defun/cc fork (func &rest args)
             (cont:let/cc k
               (enqueue k)
               (apply func args)))

;; This stops running the current thread by placing it into the queue
;; and starts running the other thread from the queue if there is any:
POFTHEDAY> (cont:defun/cc yield ()
             (cont:let/cc k
               (enqueue k)
               (funcall (dequeue))))

How does fork function work?

;; This is the function we want to run in "parallel":
POFTHEDAY> (defun do-job ()
             (format t "Inside job~%"))

;; Initially, our queue is empty:
POFTHEDAY> *queue*
NIL

;; Now when we'll call the fork,
;; it will:
;;
;; - capture current continuation;
;; - put it into the queue;
;; - execute do-job function.
POFTHEDAY> (cont:with-call/cc
             (format t "Before fork~%")
             (fork #'do-job)
             (format t "After fork~%"))
Before fork
Inside job
NIL

;; Now queue has one function which is
;; the rest of our initial computation.
POFTHEDAY> *queue*
(#<FUNCTION (LAMBDA (&OPTIONAL #:G1655 &REST #:G1656)) {22A1719B}>)

;; When the rest of the computation gets called,
;; it prints "After fork" and exits:
POFTHEDAY> (funcall (dequeue))
After fork
NIL

Yield works similarly. It captures the current continuation, appends it to the queue, takes the next coroutine from the top of the queue and executes it.

To test how two coroutines will behave when running in parallel, let's create a function which will print its name in the loop. On each iteration a coroutine will call yield to give other coroutines a chance to get executed:

POFTHEDAY> (cont:defun/cc do-stuff-n-print (name)
             (loop for n from 1 upto 3
                   do (format t "~A ~A~%" name n)
                      (yield)
                      (sleep 1)))

;; We also need to add this primive to our framework
POFTHEDAY> (defun wait-for-threads ()
             (loop
               when (empty-queue?)
                 return nil
               do (funcall (dequeue))))

POFTHEDAY> (cont:with-call/cc
             (fork #'do-stuff-n-print "Foo")
             (fork #'do-stuff-n-print "Bar")
             (wait-for-threads))
Foo 1
Bar 2
Foo 3
Bar 1
Foo 2
Bar 3

The result we've got is the same as the result of the Wikipedia article. Messages from both coroutines are interleaving. That is great!

Now, cl-cont does not look so strange to me. It is time to reimplement continuation widgets for the Weblocks! :)

lmdb

Thu, 08 Oct 2020 23:55:01 +0300

This is a binding to the embedded database LMDB, mentioned in this tweet. LMDB is a fast key/value database which can be embedded into your app as a C library.

Documentation on LMDB says it is really fast. I found this performance benchmark which compares it to the BoltDB and Badger. According to it, LMDB is slightly faster than BoldDB, but both lose to Badger.

It would be interesting to make our own benchmarks, but to compare LMDB with LevelDB which also has a binding to Common Lisp. But that is a story for another day.

Here is a fixed and slightly modified example from the CL wrapper's README. It just writes a string by the key and reads it back:

POFTHEDAY> (let ((env (lmdb:make-environment #p"./the-database/")))
             (lmdb:with-environment (env)
               ;; Create a transaction
               (let ((txn (lmdb:make-transaction env)))
                 (lmdb:begin-transaction txn)
                 ;; Create a database access object
                 (let ((db (lmdb:make-database txn "db")))
                   (lmdb:with-database (db)
                     ;; Here is how we can write some data to the storage
                     (lmdb:put db "the key" "The string")
                     ;; and read it back:
                     (let ((vec (lmdb:get db "the key")))
                       (print vec)))))))

;; Pay attention, the data is returned as a vector and your
;; app have to interpret it:
#(84 104 101 32 115 116 114 105 110 103)

POFTHEDAY> (babel:octets-to-string *)
"The string"

What is interesting, I found this library was used in this Wiki software, written in Common Lisp: Antimer.

But LMDB backend was removed from Antimer at some moment and replaced with SQLite. Most probably because it needed the full power of SQL instead of simple key-value queries.

To finalize, this LMDB binding would be a good solution for small apps which makes simple queries and need a high-performance and low latency.

BTW, the LMDB's repository needs some love because there are some hanging pull requests and a few unanswered issues. Does somebody know how does Fernando Borretti feels himself? Maybe he needs some help?