You can support this project by donating at:

Or see the list of project sponsors.

cl-storedata-structuresdata-formats

Documentation	😀
Docstrings	😀
Tests	😀
Examples	😀
RepositoryActivity	🥺
CI	🥺

I use this library in a few of my projects. It is much like Python's pickle module, but a lot more extensible.

CL-Store is able to store and restore back almost any Lisp object. For example, lfarm, reviewed four days ago uses it to serialize and deserialize parameters when executing remote jobs on the server.

To demonstrate how it works, I'll create a vector of objects. Then we'll save this vector to the file and restore it back:

POFTHEDAY> (defclass user ()
             ((name :initarg :name
                    :reader user-name)))

POFTHEDAY> (defmethod print-object ((user user) stream)
             (print-unreadable-object (user stream :type t)
               (format stream "~A"
                       (user-name user))))

POFTHEDAY> (defparameter *users*
             (make-array 2
                         :initial-contents
                         (list (make-instance 'user :name "Bob")
                               (make-instance 'user :name "Alice"))))

POFTHEDAY> *users*
#(#<USER Bob> #<USER Alice>)

;; Now we are ready to store our data to the file
POFTHEDAY> (cl-store:store *users* #P"/tmp/users.bin")

;; and to restore it back:
POFTHEDAY> (cl-store:restore #P"/tmp/users.bin")
#(#<USER Bob> #<USER Alice>)

Here is the resulting file. Don't be deceived by the readable content. This is the binary format and it might contain nonreadable characters:

[art@poftheday] cat /tmp/users.bin
CLCL
#USER#  POFTHEDAY
#NAMEBobAlice

CL-Store can be extended. Originally it also provided a backend to serialize data into the XML. But now this backend is considered as deprecated.

Previously I didn't write backends for CL-Store, but today I found in its docs an example, how to write a backend, compatible with Python's pickle.

Documentation is 13 years old, let's see if we'll be able to reproduce it!

First, we need to define a new backend:

POFTHEDAY> (cl-store:defbackend pickle
             :stream-type 'character)
#<PICKLE {100369EBB3}>

;; This small call expands into this huge amount of code:

(eval-when (:load-toplevel :execute)
  (eval-when (:compile-toplevel :load-toplevel :execute)
    (defclass pickle (cl-store:backend) nil
              (:documentation
               "Autogenerated cl-store class for backend pickle."))
    (defmacro defstore-pickle
              ((cl-store::var type stream &optional cl-store::qualifier)
               &body cl-store::body)
      (cl-store::with-gensyms (cl-store::gbackend)
        `(defmethod cl-store:internal-store-object ,@(if cl-store::qualifier
                                                         (list
                                                          cl-store::qualifier)
                                                         nil)
                    ((,cl-store::gbackend ,'pickle) (,cl-store::var ,type)
                     ,stream)
           ,(format nil "Definition for storing an object of type ~A with ~
 backend ~A"
                    type 'pickle)
           (declare (ignorable ,cl-store::gbackend))
           ,@cl-store::body)))
    (defmacro defrestore-pickle
              ((type cl-store::place &optional cl-store::qualifier)
               &body cl-store::body)
      (cl-store::with-gensyms (cl-store::gbackend cl-store::gtype)
        `(defmethod cl-store::internal-restore-object ,@(if cl-store::qualifier
                                                            (list
                                                             cl-store::qualifier)
                                                            nil)
                    ((,cl-store::gbackend ,'pickle)
                     (,cl-store::gtype (eql ',type)) (,cl-store::place t))
           (declare (ignorable ,cl-store::gbackend ,cl-store::gtype))
           ,@cl-store::body))))
  (cl-store::register-backend 'pickle 'pickle nil 'character 'nil 'nil))

Now we can use defstore-pickle and defrestore-pickle macroses to define rules for the processing of different data types.

Here is where all real work is done:

POFTHEDAY> (defvar *pickle-mapping*
             '((#\S . string))
             "A mapping from Pickle's char codes
              to a Lisp data type")

POFTHEDAY> (defmethod cl-store:get-next-reader ((backend pickle) (stream stream))
             "This method is responsible for recognizing what
              type of object should be read next."
             (let ((type-code (read-char stream)))
               (or (cdr (assoc type-code *pickle-mapping*))
                   (values nil (format nil "Type ~A" type-code)))))

POFTHEDAY> (defrestore-pickle (noop stream)
             "We'll skip unknown objects")

POFTHEDAY> (defstore-pickle (obj string stream)
             "Here is how string should be written in Pickle's format."
             (format stream "S'~A'~%p0~%." obj))

POFTHEDAY> (defrestore-pickle (string stream)
             "And this is a code to read string back"
             (let ((val (read-line stream)))
               (read-line stream) ;; remove the PUSH op
               (read-line stream) ;; remove the END op
               (subseq val 1 (1- (length val)))))

It is time to test our functions. To make this old example work, I had to use old Python2.7, because Python3's pickle serializes into a little bit different format.

[art@poftheday] python2.7

>>> with open('/tmp/word.pickle', 'w') as f:
...     pickle.dump('Hello Lisp World!', f, protocol=0)
...
>>> ^D

[art@poftheday] cat /tmp/word.pickle
S'Hello Lisp World!'
p0
.

# Here is what I've got under Python3:

In [9]: with open('/tmp/word.pickle', 'bw') as f:
   ...:     pickle.dump('Hello Lisp World!', f, protocol=0)
   ...:

In [10]: !cat /tmp/word.pickle
VHello Lisp World!
p0
.
# This cl-store backend does not support V type code.
# Seems, it stands for a Unicode string.

Now we can read this file from Lisp and write it back:

POFTHEDAY> (cl-store:restore #P"/tmp/word.pickle"
                             'pickle)
"Hello Lisp World!"

POFTHEDAY> (cl-store:store "Howdy, Python!"
                           #P"/tmp/word.pickle"
                           'pickle)
"Howdy, Python!"

And finally, to ensure our backend works as expected, we'll read this response in Python2:

>>> import pickle
>>> with open('/tmp/word.pickle') as f:
...     pickle.load(f)
...
'Howdy, Python!'

That is it. If you need a time-proved serialization library, check out the CL-Store! To extend it, just read the docs.