RSS Feed

Lisp Project of the Day


You can support this project by donating at:

Donate using PatreonDonate using Liberapay

Or see the list of project sponsors.


Tests ๐Ÿฅบ
CI ๐Ÿฅบ

This library implements the async/await abstraction to make it easier to write parallel programs.

Now we'll turn "dexador" http library calls into async and will see if we can parallel 50 requests to the site which response in 5 seconds.

To create a function which can return a delayed result, a "promise", we have to use cl-async-await:defun-async:

POFTHEDAY> (cl-async-await:defun-async http-get (url &rest args)
             (apply #'dexador:get url args))

Now let's call this function. When called it returns a "promise" object not the real response from the site:

POFTHEDAY> (http-get "")

Now we can retrieve the real result, using cl-async-await:await function:

POFTHEDAY> (cl-async-await:await *)
  \"args\": {}, 
  \"data\": \"\", 
  \"files\": {}, 
  \"form\": {}, 
  \"headers\": {
    \"Accept\": \"*/*\", 
    \"Content-Length\": \"0\", 
    \"Host\": \"\", 
    \"User-Agent\": \"Dexador/0.9.14 (SBCL 2.0.8); Darwin; 19.5.0\", 
    \"X-Amzn-Trace-Id\": \"Root=1-5f9732d6-148ee9a305fab66c26a2dbfd\"
  \"origin\": \"\", 
  \"url\": \"\"
200 (8 bits, #xC8, #o310, #b11001000)
#<CL+SSL::SSL-STREAM for #<FD-STREAM for "socket, peer:" {10085B0BF3}>>

If we look a the promise object again, we'll see it has a state now:

  "args": {}, 
  "data": "", 
  "files": {}, 
  "form": {}, 
  "headers": {
    "Accept": "*/*", 
    "Content-Length": "0", 
    "Host": "", 
    "User-Agent": "Dexador/0.9.14 (SBCL 2.0.8); Darwin; 19.5.0", 
    "X-Amzn-Trace-Id": "Root=1-5f9732d6-148ee9a305fab66c26a2dbfd"
  "origin": "", 
  "url": ""

  200 #<HASH-TABLE :TEST EQUAL :COUNT 7 {1002987DE3}>
  #<SSL-STREAM for #<FD-STREAM for "socket, peer:" {10085B0BF3}>>) >

Ok, it is time to see if we can retrieve results from this site in parallel. To make it easier to test speed, I'll wrap all code into the separate function.

The function returns the total number of bytes in all 50 responses:

POFTHEDAY> (defun do-the-test ()
             (let ((promises
                     (loop repeat 50
                           collect (http-get ""
                                             :use-connection-pool nil
                                             :keep-alive nil))))
               ;; Now we have to fetch results from our promises.
               (loop for promise in promises
                     for response = (cl-async-await:await
                     summing (length response))))

POFTHEDAY> (time (do-the-test))
Evaluation took:
  6.509 seconds of real time
  2.496912 seconds of total run time (1.672766 user, 0.824146 system)
  38.36% CPU
  14,372,854,434 processor cycles
  1,519,664 bytes consed

As you can see, the function returns in 6.5 seconds instead of 250 seconds! This means cl-async-await works!

The only problem I found was this concurrency issue:

But probably it is only related to Dexador.

Brought to you by 40Ants under Creative Commons License