Dynamic Closures?

I have been experimenting with continuation closures (aka callbacks) in an Actors-based environment. Actors are objects that can only run on one thread at a time, and it doesn’t matter which thread. But they preserve single-thread semantics even in the face of SMP.

Blocking activities are the bane of threading and Actors, but you can’t always know when a function call will block, nor if it is actually blocking or just taking a long time to complete. Regardless, it is possible for an Actor to construct a callback closure and pass along as a reply-to argument. When activated, it sends a special message back to the Actor to have itself run its own continuation - to preserve single thread semantics. But a simple lexical closure is missing some things...

By using an expansion of the =BIND protocol from P. Graham, you can define extensions of closures that I call “Dynamic Closures”, so that when the closure is executed, it restores most of the dynamic environment that was in place when the closure was created. This is needed for HANDLER-BIND, CATCH, and many others. It does not restore active dynamic bindings unless you also use extended forms =LET and =LET*. And UNWIND-PROTECT remains problematic. (We need a DYNAMIC-WIND, which has its own problems. You can’t regenerate an irreversible past…)

But when I constructed =LET and =LET* it became immediately apparent that Delimited Continuations (if they exist) will have problems there too. It seems that, in general, the only safe Continuations would have to capture the entire stack in order to capture all the dynamic bindings also in effect at the time of creation. I am not overtly using Delimited Continuations, although you could do that with something like CL-CONT. I personally think that =BIND conveys a more Lisp centric display of what is happening, as constrasted with the hidden syntactic and semantic effects of ASYNC/AWAIT style.

So I did a search for “Dynamic Closures” in Lisp to see if anyone else had gone down this rabbit hole. I found a page by Paul Khuong talking about “Common Cold” as applied to SBCL and making serializable closures for web programming. But no libraries come up in searches.

Anyone else gone there?

- David McClain

_______________________________________________
Lisp Hug - the mailing list for LispWorks users
lisp-hug@lispworks.com
http://www.lispworks.com/support/lisp-hug.html

Re: Dynamic Closures?

Somewhat unrelated: there's an article by Kent Pitman that describes the
issues that happen between UNWIND-PROTECT and continuations, and you
might want to read it if you haven't yet -
http://www.nhplace.com/kent/PFAQ/unwind-protect-vs-continuations-original.html

~phoe

On 25.10.2020 17:56, dbm@refined-audiometrics.com wrote:
> I have been experimenting with continuation closures (aka callbacks) in an Actors-based environment. Actors are objects that can only run on one thread at a time, and it doesn’t matter which thread. But they preserve single-thread semantics even in the face of SMP. 
>
> Blocking activities are the bane of threading and Actors, but you can’t always know when a function call will block, nor if it is actually blocking or just taking a long time to complete. Regardless, it is possible for an Actor to construct a callback closure and pass along as a reply-to argument. When activated, it sends a special message back to the Actor to have itself run its own continuation - to preserve single thread semantics. But a simple lexical closure is missing some things...
>
> By using an expansion of the =BIND protocol from P. Graham, you can define extensions of closures that I call “Dynamic Closures”, so that when the closure is executed, it restores most of the dynamic environment that was in place when the closure was created. This is needed for HANDLER-BIND, CATCH, and many others. It does not restore active dynamic bindings unless you also use extended forms =LET and =LET*. And UNWIND-PROTECT remains problematic. (We need a DYNAMIC-WIND, which has its own problems. You can’t regenerate an irreversible past…)
>
> But when I constructed =LET and =LET* it became immediately apparent that Delimited Continuations (if they exist) will have problems there too. It seems that, in general, the only safe Continuations would have to capture the entire stack in order to capture all the dynamic bindings also in effect at the time of creation. I am not overtly using Delimited Continuations, although you could do that with something like CL-CONT. I personally think that =BIND conveys a more Lisp centric display of what is happening, as constrasted with the hidden syntactic and semantic effects of ASYNC/AWAIT style.
>
> So I did a search for “Dynamic Closures” in Lisp to see if anyone else had gone down this rabbit hole. I found a page by Paul Khuong talking about “Common Cold” as applied to SBCL and making serializable closures for web programming. But no libraries come up in searches.
>
> Anyone else gone there?
>
> - David McClain
>
>
> _______________________________________________
> Lisp Hug - the mailing list for LispWorks users
> lisp-hug@lispworks.com
> http://www.lispworks.com/support/lisp-hug.html

_______________________________________________
Lisp Hug - the mailing list for LispWorks users
lisp-hug@lispworks.com
http://www.lispworks.com/support/lisp-hug.html

Re: Dynamic Closures?

Yes! … Exactly right!

(BTW… I just discovered and bought your new book, TCLCS. Excellent! It has already helped, and I’m not a newbie…)

> On Oct 25, 2020, at 10:05 AM, Michał phoe Herda <phoe@disroot.org> wrote:
> 
> Somewhat unrelated: there's an article by Kent Pitman that describes the
> issues that happen between UNWIND-PROTECT and continuations, and you
> might want to read it if you haven't yet -
> http://www.nhplace.com/kent/PFAQ/unwind-protect-vs-continuations-original.html
> 
> ~phoe
> 
> On 25.10.2020 17:56, dbm@refined-audiometrics.com wrote:
>> I have been experimenting with continuation closures (aka callbacks) in an Actors-based environment. Actors are objects that can only run on one thread at a time, and it doesn’t matter which thread. But they preserve single-thread semantics even in the face of SMP. 
>> 
>> Blocking activities are the bane of threading and Actors, but you can’t always know when a function call will block, nor if it is actually blocking or just taking a long time to complete. Regardless, it is possible for an Actor to construct a callback closure and pass along as a reply-to argument. When activated, it sends a special message back to the Actor to have itself run its own continuation - to preserve single thread semantics. But a simple lexical closure is missing some things...
>> 
>> By using an expansion of the =BIND protocol from P. Graham, you can define extensions of closures that I call “Dynamic Closures”, so that when the closure is executed, it restores most of the dynamic environment that was in place when the closure was created. This is needed for HANDLER-BIND, CATCH, and many others. It does not restore active dynamic bindings unless you also use extended forms =LET and =LET*. And UNWIND-PROTECT remains problematic. (We need a DYNAMIC-WIND, which has its own problems. You can’t regenerate an irreversible past…)
>> 
>> But when I constructed =LET and =LET* it became immediately apparent that Delimited Continuations (if they exist) will have problems there too. It seems that, in general, the only safe Continuations would have to capture the entire stack in order to capture all the dynamic bindings also in effect at the time of creation. I am not overtly using Delimited Continuations, although you could do that with something like CL-CONT. I personally think that =BIND conveys a more Lisp centric display of what is happening, as constrasted with the hidden syntactic and semantic effects of ASYNC/AWAIT style.
>> 
>> So I did a search for “Dynamic Closures” in Lisp to see if anyone else had gone down this rabbit hole. I found a page by Paul Khuong talking about “Common Cold” as applied to SBCL and making serializable closures for web programming. But no libraries come up in searches.
>> 
>> Anyone else gone there?
>> 
>> - David McClain
>> 
>> 
>> _______________________________________________
>> Lisp Hug - the mailing list for LispWorks users
>> lisp-hug@lispworks.com
>> http://www.lispworks.com/support/lisp-hug.html


_______________________________________________
Lisp Hug - the mailing list for LispWorks users
lisp-hug@lispworks.com
http://www.lispworks.com/support/lisp-hug.html

Re: Dynamic Closures?

There is support for dynamic closures in Contextl. See the github page at https://github.com/pcostanza/contextl

Support for dynamic closures is implemented in cx-dynamic-environment.lisp and dynamic variables are implemented in cx-dynamic-variables.lisp. (They have like special variables, but need an extra level of indirection to work with features that capture the dynamic environment.)

Sorry for the lack of documentation, but the test cases in https://github.com/pcostanza/contextl/blob/master/test/dynamic-wind.lisp and https://github.com/pcostanza/contextl/blob/master/test/dynenv.lisp should give you an idea what you can do with this.

If you are only interested in dynamic closures and don’t want to buy into all of ContextL, there is even a system definition just for that in dynamic-wind.asd

Pascal

On 25 Oct 2020, at 17:56, dbm@refined-audiometrics.com wrote:

I have been experimenting with continuation closures (aka callbacks) in an Actors-based environment. Actors are objects that can only run on one thread at a time, and it doesn’t matter which thread. But they preserve single-thread semantics even in the face of SMP.

Blocking activities are the bane of threading and Actors, but you can’t always know when a function call will block, nor if it is actually blocking or just taking a long time to complete. Regardless, it is possible for an Actor to construct a callback closure and pass along as a reply-to argument. When activated, it sends a special message back to the Actor to have itself run its own continuation - to preserve single thread semantics. But a simple lexical closure is missing some things...

By using an expansion of the =BIND protocol from P. Graham, you can define extensions of closures that I call “Dynamic Closures”, so that when the closure is executed, it restores most of the dynamic environment that was in place when the closure was created. This is needed for HANDLER-BIND, CATCH, and many others. It does not restore active dynamic bindings unless you also use extended forms =LET and =LET*. And UNWIND-PROTECT remains problematic. (We need a DYNAMIC-WIND, which has its own problems. You can’t regenerate an irreversible past…)

But when I constructed =LET and =LET* it became immediately apparent that Delimited Continuations (if they exist) will have problems there too. It seems that, in general, the only safe Continuations would have to capture the entire stack in order to capture all the dynamic bindings also in effect at the time of creation. I am not overtly using Delimited Continuations, although you could do that with something like CL-CONT. I personally think that =BIND conveys a more Lisp centric display of what is happening, as constrasted with the hidden syntactic and semantic effects of ASYNC/AWAIT style.

So I did a search for “Dynamic Closures” in Lisp to see if anyone else had gone down this rabbit hole. I found a page by Paul Khuong talking about “Common Cold” as applied to SBCL and making serializable closures for web programming. But no libraries come up in searches.

Anyone else gone there?

- David McClain

_______________________________________________
Lisp Hug - the mailing list for LispWorks users
lisp-hug@lispworks.com
http://www.lispworks.com/support/lisp-hug.html

--
Pascal Costanza

Re: Dynamic Closures?

Indeed! DYNAMIC-WIND is really quite powerful, and far more expressive than what I had been doing.

The real advantage over my own method (which was really almost the same as yours), is the internal PROCEED macro, which allows me to pinpoint the location where the continuation closure should take over from the original code. That flexibility is really amazingly powerful.

- DM

On Nov 13, 2020, at 6:12 AM, Pascal Costanza <pc@p-cos.net> wrote:

There is support for dynamic closures in Contextl. See the github page at https://github.com/pcostanza/contextl

Support for dynamic closures is implemented in cx-dynamic-environment.lisp and dynamic variables are implemented in cx-dynamic-variables.lisp. (They have like special variables, but need an extra level of indirection to work with features that capture the dynamic environment.)

Sorry for the lack of documentation, but the test cases in https://github.com/pcostanza/contextl/blob/master/test/dynamic-wind.lisp and https://github.com/pcostanza/contextl/blob/master/test/dynenv.lisp should give you an idea what you can do with this.

If you are only interested in dynamic closures and don’t want to buy into all of ContextL, there is even a system definition just for that in dynamic-wind.asd

Pascal

On 25 Oct 2020, at 17:56, dbm@refined-audiometrics.com wrote:

I have been experimenting with continuation closures (aka callbacks) in an Actors-based environment. Actors are objects that can only run on one thread at a time, and it doesn’t matter which thread. But they preserve single-thread semantics even in the face of SMP.

Blocking activities are the bane of threading and Actors, but you can’t always know when a function call will block, nor if it is actually blocking or just taking a long time to complete. Regardless, it is possible for an Actor to construct a callback closure and pass along as a reply-to argument. When activated, it sends a special message back to the Actor to have itself run its own continuation - to preserve single thread semantics. But a simple lexical closure is missing some things...

By using an expansion of the =BIND protocol from P. Graham, you can define extensions of closures that I call “Dynamic Closures”, so that when the closure is executed, it restores most of the dynamic environment that was in place when the closure was created. This is needed for HANDLER-BIND, CATCH, and many others. It does not restore active dynamic bindings unless you also use extended forms =LET and =LET*. And UNWIND-PROTECT remains problematic. (We need a DYNAMIC-WIND, which has its own problems. You can’t regenerate an irreversible past…)

But when I constructed =LET and =LET* it became immediately apparent that Delimited Continuations (if they exist) will have problems there too. It seems that, in general, the only safe Continuations would have to capture the entire stack in order to capture all the dynamic bindings also in effect at the time of creation. I am not overtly using Delimited Continuations, although you could do that with something like CL-CONT. I personally think that =BIND conveys a more Lisp centric display of what is happening, as constrasted with the hidden syntactic and semantic effects of ASYNC/AWAIT style.

So I did a search for “Dynamic Closures” in Lisp to see if anyone else had gone down this rabbit hole. I found a page by Paul Khuong talking about “Common Cold” as applied to SBCL and making serializable closures for web programming. But no libraries come up in searches.

Anyone else gone there?

- David McClain

_______________________________________________
Lisp Hug - the mailing list for LispWorks users
lisp-hug@lispworks.com
http://www.lispworks.com/support/lisp-hug.html

--
Pascal Costanza

Re: Dynamic Closures?

… I would state that DYNAMIC-WIND plus PROCEED encompasses most (all?) of what delimited continuations offer. And the syntax is quite natural.

Example - notice how the PROCEED occupies the location of a pending operand. On a restart, that will pick up the conitnuation closure and evaluate it, instead of the original code which follows the PROCEED. The DYNAMIC-WIND allows me to re-instantiate dynamic bindings that were important in the first pass.

;; Intercept restartable queries to send back a response

;; from the following message, reflecting any errors back to

;; the caller.

(let ((original-ask-message (whole-message)))

(um:dynamic-wind

(let ((*whole-message* original-ask-message)

(*in-ask* t))

(handler-case

(send reply-to

(with-captured-ans-or-exn

(um:proceed

(apply #'self-call sub-msg))))

(no-immediate-answer ())

))))

On Nov 15, 2020, at 1:07 AM, dbm@refined-audiometrics.com wrote:

Indeed! DYNAMIC-WIND is really quite powerful, and far more expressive than what I had been doing.

The real advantage over my own method (which was really almost the same as yours), is the internal PROCEED macro, which allows me to pinpoint the location where the continuation closure should take over from the original code. That flexibility is really amazingly powerful.

- DM

On Nov 13, 2020, at 6:12 AM, Pascal Costanza <pc@p-cos.net> wrote:

There is support for dynamic closures in Contextl. See the github page at https://github.com/pcostanza/contextl

Support for dynamic closures is implemented in cx-dynamic-environment.lisp and dynamic variables are implemented in cx-dynamic-variables.lisp. (They have like special variables, but need an extra level of indirection to work with features that capture the dynamic environment.)

Sorry for the lack of documentation, but the test cases in https://github.com/pcostanza/contextl/blob/master/test/dynamic-wind.lisp and https://github.com/pcostanza/contextl/blob/master/test/dynenv.lisp should give you an idea what you can do with this.

If you are only interested in dynamic closures and don’t want to buy into all of ContextL, there is even a system definition just for that in dynamic-wind.asd

Pascal

On 25 Oct 2020, at 17:56, dbm@refined-audiometrics.com wrote:

I have been experimenting with continuation closures (aka callbacks) in an Actors-based environment. Actors are objects that can only run on one thread at a time, and it doesn’t matter which thread. But they preserve single-thread semantics even in the face of SMP.

Blocking activities are the bane of threading and Actors, but you can’t always know when a function call will block, nor if it is actually blocking or just taking a long time to complete. Regardless, it is possible for an Actor to construct a callback closure and pass along as a reply-to argument. When activated, it sends a special message back to the Actor to have itself run its own continuation - to preserve single thread semantics. But a simple lexical closure is missing some things...

By using an expansion of the =BIND protocol from P. Graham, you can define extensions of closures that I call “Dynamic Closures”, so that when the closure is executed, it restores most of the dynamic environment that was in place when the closure was created. This is needed for HANDLER-BIND, CATCH, and many others. It does not restore active dynamic bindings unless you also use extended forms =LET and =LET*. And UNWIND-PROTECT remains problematic. (We need a DYNAMIC-WIND, which has its own problems. You can’t regenerate an irreversible past…)

But when I constructed =LET and =LET* it became immediately apparent that Delimited Continuations (if they exist) will have problems there too. It seems that, in general, the only safe Continuations would have to capture the entire stack in order to capture all the dynamic bindings also in effect at the time of creation. I am not overtly using Delimited Continuations, although you could do that with something like CL-CONT. I personally think that =BIND conveys a more Lisp centric display of what is happening, as constrasted with the hidden syntactic and semantic effects of ASYNC/AWAIT style.

So I did a search for “Dynamic Closures” in Lisp to see if anyone else had gone down this rabbit hole. I found a page by Paul Khuong talking about “Common Cold” as applied to SBCL and making serializable closures for web programming. But no libraries come up in searches.

Anyone else gone there?

- David McClain

_______________________________________________
Lisp Hug - the mailing list for LispWorks users
lisp-hug@lispworks.com
http://www.lispworks.com/support/lisp-hug.html

--
Pascal Costanza

Re: Dynamic Closures?

Hi Pascal,

I am sure you have it somewhere, but could you repeat here a simple explanation of what DYNAMIC-WIND and PROCEED do?

Thanks

Marco

On Sun, Nov 15, 2020 at 9:24 AM David McClain <dbm@refined-audiometrics.com> wrote:

… I would state that DYNAMIC-WIND plus PROCEED encompasses most (all?) of what delimited continuations offer. And the syntax is quite natural.

Example - notice how the PROCEED occupies the location of a pending operand. On a restart, that will pick up the conitnuation closure and evaluate it, instead of the original code which follows the PROCEED. The DYNAMIC-WIND allows me to re-instantiate dynamic bindings that were important in the first pass.

;; Intercept restartable queries to send back a response
;; from the following message, reflecting any errors back to
;; the caller.
(let ((original-ask-message (whole-message)))
(um:dynamic-wind
(let ((*whole-message* original-ask-message)
(*in-ask* t))
(handler-case
(send reply-to
(with-captured-ans-or-exn
(um:proceed
(apply #'self-call sub-msg))))

(no-immediate-answer ())
))))

On Nov 15, 2020, at 1:07 AM, dbm@refined-audiometrics.com wrote:

Indeed! DYNAMIC-WIND is really quite powerful, and far more expressive than what I had been doing.

The real advantage over my own method (which was really almost the same as yours), is the internal PROCEED macro, which allows me to pinpoint the location where the continuation closure should take over from the original code. That flexibility is really amazingly powerful.

- DM

On Nov 13, 2020, at 6:12 AM, Pascal Costanza <pc@p-cos.net> wrote:

There is support for dynamic closures in Contextl. See the github page at https://github.com/pcostanza/contextl

Support for dynamic closures is implemented in cx-dynamic-environment.lisp and dynamic variables are implemented in cx-dynamic-variables.lisp. (They have like special variables, but need an extra level of indirection to work with features that capture the dynamic environment.)

Sorry for the lack of documentation, but the test cases in https://github.com/pcostanza/contextl/blob/master/test/dynamic-wind.lisp and https://github.com/pcostanza/contextl/blob/master/test/dynenv.lisp should give you an idea what you can do with this.

If you are only interested in dynamic closures and don’t want to buy into all of ContextL, there is even a system definition just for that in dynamic-wind.asd

Pascal

On 25 Oct 2020, at 17:56, dbm@refined-audiometrics.com wrote:

I have been experimenting with continuation closures (aka callbacks) in an Actors-based environment. Actors are objects that can only run on one thread at a time, and it doesn’t matter which thread. But they preserve single-thread semantics even in the face of SMP.

Blocking activities are the bane of threading and Actors, but you can’t always know when a function call will block, nor if it is actually blocking or just taking a long time to complete. Regardless, it is possible for an Actor to construct a callback closure and pass along as a reply-to argument. When activated, it sends a special message back to the Actor to have itself run its own continuation - to preserve single thread semantics. But a simple lexical closure is missing some things...

By using an expansion of the =BIND protocol from P. Graham, you can define extensions of closures that I call “Dynamic Closures”, so that when the closure is executed, it restores most of the dynamic environment that was in place when the closure was created. This is needed for HANDLER-BIND, CATCH, and many others. It does not restore active dynamic bindings unless you also use extended forms =LET and =LET*. And UNWIND-PROTECT remains problematic. (We need a DYNAMIC-WIND, which has its own problems. You can’t regenerate an irreversible past…)

But when I constructed =LET and =LET* it became immediately apparent that Delimited Continuations (if they exist) will have problems there too. It seems that, in general, the only safe Continuations would have to capture the entire stack in order to capture all the dynamic bindings also in effect at the time of creation. I am not overtly using Delimited Continuations, although you could do that with something like CL-CONT. I personally think that =BIND conveys a more Lisp centric display of what is happening, as constrasted with the hidden syntactic and semantic effects of ASYNC/AWAIT style.

So I did a search for “Dynamic Closures” in Lisp to see if anyone else had gone down this rabbit hole. I found a page by Paul Khuong talking about “Common Cold” as applied to SBCL and making serializable closures for web programming. But no libraries come up in searches.

Anyone else gone there?

- David McClain

_______________________________________________
Lisp Hug - the mailing list for LispWorks users
lisp-hug@lispworks.com
http://www.lispworks.com/support/lisp-hug.html

--
Pascal Costanza

Marco Antoniotti, Associate Professor tel. +39 - 02 64 48 79 01
DISCo, Università Milano Bicocca U14 2043 http://bimib.disco.unimib.it
Viale Sarca 336
I-20126 Milan (MI) ITALY

Re: Dynamic Closures?

Hi Marco,

Unfortunately, it has been a while since I wrote this code, so I would need to immerse myself again in my code for a bit to figure out myself what I was trying to achieve. Maybe David is in a better position to explain this right now. :)

Pascal

Sent from my iPad

On 15 Nov 2020, at 10:57, Marco Antoniotti <marco.antoniotti@unimib.it> wrote:

Hi Pascal,

I am sure you have it somewhere, but could you repeat here a simple explanation of what DYNAMIC-WIND and PROCEED do?

Thanks

Marco

On Sun, Nov 15, 2020 at 9:24 AM David McClain <dbm@refined-audiometrics.com> wrote:
… I would state that DYNAMIC-WIND plus PROCEED encompasses most (all?) of what delimited continuations offer. And the syntax is quite natural.

Example - notice how the PROCEED occupies the location of a pending operand. On a restart, that will pick up the conitnuation closure and evaluate it, instead of the original code which follows the PROCEED. The DYNAMIC-WIND allows me to re-instantiate dynamic bindings that were important in the first pass.

;; Intercept restartable queries to send back a response
;; from the following message, reflecting any errors back to
;; the caller.
(let ((original-ask-message (whole-message)))
(um:dynamic-wind
(let ((*whole-message* original-ask-message)
(*in-ask* t))
(handler-case
(send reply-to
(with-captured-ans-or-exn
(um:proceed
(apply #'self-call sub-msg))))

(no-immediate-answer ())
))))

On Nov 15, 2020, at 1:07 AM, dbm@refined-audiometrics.com wrote:

Indeed! DYNAMIC-WIND is really quite powerful, and far more expressive than what I had been doing.

The real advantage over my own method (which was really almost the same as yours), is the internal PROCEED macro, which allows me to pinpoint the location where the continuation closure should take over from the original code. That flexibility is really amazingly powerful.

- DM

On Nov 13, 2020, at 6:12 AM, Pascal Costanza <pc@p-cos.net> wrote:

There is support for dynamic closures in Contextl. See the github page at https://github.com/pcostanza/contextl

Support for dynamic closures is implemented in cx-dynamic-environment.lisp and dynamic variables are implemented in cx-dynamic-variables.lisp. (They have like special variables, but need an extra level of indirection to work with features that capture the dynamic environment.)

Sorry for the lack of documentation, but the test cases in https://github.com/pcostanza/contextl/blob/master/test/dynamic-wind.lisp and https://github.com/pcostanza/contextl/blob/master/test/dynenv.lisp should give you an idea what you can do with this.

If you are only interested in dynamic closures and don’t want to buy into all of ContextL, there is even a system definition just for that in dynamic-wind.asd

Pascal

On 25 Oct 2020, at 17:56, dbm@refined-audiometrics.com wrote:

I have been experimenting with continuation closures (aka callbacks) in an Actors-based environment. Actors are objects that can only run on one thread at a time, and it doesn’t matter which thread. But they preserve single-thread semantics even in the face of SMP.

Blocking activities are the bane of threading and Actors, but you can’t always know when a function call will block, nor if it is actually blocking or just taking a long time to complete. Regardless, it is possible for an Actor to construct a callback closure and pass along as a reply-to argument. When activated, it sends a special message back to the Actor to have itself run its own continuation - to preserve single thread semantics. But a simple lexical closure is missing some things...

By using an expansion of the =BIND protocol from P. Graham, you can define extensions of closures that I call “Dynamic Closures”, so that when the closure is executed, it restores most of the dynamic environment that was in place when the closure was created. This is needed for HANDLER-BIND, CATCH, and many others. It does not restore active dynamic bindings unless you also use extended forms =LET and =LET*. And UNWIND-PROTECT remains problematic. (We need a DYNAMIC-WIND, which has its own problems. You can’t regenerate an irreversible past…)

But when I constructed =LET and =LET* it became immediately apparent that Delimited Continuations (if they exist) will have problems there too. It seems that, in general, the only safe Continuations would have to capture the entire stack in order to capture all the dynamic bindings also in effect at the time of creation. I am not overtly using Delimited Continuations, although you could do that with something like CL-CONT. I personally think that =BIND conveys a more Lisp centric display of what is happening, as constrasted with the hidden syntactic and semantic effects of ASYNC/AWAIT style.

So I did a search for “Dynamic Closures” in Lisp to see if anyone else had gone down this rabbit hole. I found a page by Paul Khuong talking about “Common Cold” as applied to SBCL and making serializable closures for web programming. But no libraries come up in searches.

Anyone else gone there?

- David McClain

_______________________________________________
Lisp Hug - the mailing list for LispWorks users
lisp-hug@lispworks.com
http://www.lispworks.com/support/lisp-hug.html

--
Pascal Costanza

--
Marco Antoniotti, Associate Professor tel. +39 - 02 64 48 79 01
DISCo, Università Milano Bicocca U14 2043 http://bimib.disco..unimib.it
Viale Sarca 336
I-20126 Milan (MI) ITALY

Re: Dynamic Closures?

DYNAMIC-WIND - maintains a stack of closures. Each one consists of the an FLET on the body enclosed. But there is a MACROLET PROCEED inside the FLET. Wherever PROCEED is used, it becomes replaced by an IF statement. The FLET itself is parameterized by a thunk argument. Initially, the FLET is called with a null thunk. The IF of the PROCEED checks that thunk, and if null, it executes the code contained within the PROCEED macro. Otherwise it calls the thunk.

After some succession of DYNAMIC-WIND groups, you can call CAPTURE-DYNAMIC-ENVIRONMENT to produce a captured continuation state.

Later, you can invoke any function you like within the dynamic context at the time you captured it using CALL-WITH-DYNAMIC-ENVIROMENT, or using macro WITH-DYNAMIC-ENVIRONMENT. These reinstate the dynamic environment that you captured, before calling your function code. It calls the saved DYNAMIC-WIND flet functions in the same order as when they were created, ahead of your capturing of the dynamic context.

Pretty simple, really, but very clever and powerful. To my mind’s eye, I find the syntax considerably easier to grok than most implementations of delimited continuations, even though they accomplish essentially the same thing. I find DYNAMIC-WIND and PROCEED much more in line with Lisp syntax. Even more direct than Scheme’s CALL/CC.

- DM

> On Nov 15, 2020, at 3:39 AM, Pascal Costanza <pc@p-cos.net> wrote:
> 
> Hi Marco,
> 
> Unfortunately, it has been a while since I wrote this code, so I would need to immerse myself again in my code for a bit to figure out myself what I was trying to achieve. Maybe David is in a better position to explain this right now. :)
> 
> Pascal
> 
> Sent from my iPad
> 
>> On 15 Nov 2020, at 10:57, Marco Antoniotti <marco.antoniotti@unimib.it> wrote:
>> 
>> 
>> Hi Pascal,
>> 
>> I am sure you have it somewhere, but could you repeat here a simple explanation of what DYNAMIC-WIND and PROCEED do?
>> 
>> Thanks
>> 
>> Marco

_______________________________________________
Lisp Hug - the mailing list for LispWorks users
lisp-hug@lispworks.com
http://www.lispworks.com/support/lisp-hug.html

Re: Dynamic Closures?

Hmmm. Would you mind writing some sort of an article that describes and
exemplifies these? I don't think I have encountered any such piece of
writing before, and it would help me visualize what's going on under the
hood; I assume it could help other people with understanding this, too.

~phoe

On 15.11.2020 12:01, dbm@refined-audiometrics.com wrote:
> DYNAMIC-WIND - maintains a stack of closures. Each one consists of the an FLET on the body enclosed. But there is a MACROLET PROCEED inside the FLET. Wherever PROCEED is used, it becomes replaced by an IF statement. The FLET itself is parameterized by a thunk argument. Initially, the FLET is called with a null thunk. The IF of the PROCEED checks that thunk, and if null, it executes the code contained within the PROCEED macro. Otherwise it calls the thunk.
>
> After some succession of DYNAMIC-WIND groups, you can call CAPTURE-DYNAMIC-ENVIRONMENT to produce a captured continuation state.
>
> Later, you can invoke any function you like within the dynamic context at the time you captured it using CALL-WITH-DYNAMIC-ENVIROMENT, or using macro WITH-DYNAMIC-ENVIRONMENT. These reinstate the dynamic environment that you captured, before calling your function code. It calls the saved DYNAMIC-WIND flet functions in the same order as when they were created, ahead of your capturing of the dynamic context.
>
> Pretty simple, really, but very clever and powerful. To my mind’s eye, I find the syntax considerably easier to grok than most implementations of delimited continuations, even though they accomplish essentially the same thing. I find DYNAMIC-WIND and PROCEED much more in line with Lisp syntax. Even more direct than Scheme’s CALL/CC.
>
> - DM
>
>
>> On Nov 15, 2020, at 3:39 AM, Pascal Costanza <pc@p-cos.net> wrote:
>>
>> Hi Marco,
>>
>> Unfortunately, it has been a while since I wrote this code, so I would need to immerse myself again in my code for a bit to figure out myself what I was trying to achieve. Maybe David is in a better position to explain this right now. :)
>>
>> Pascal
>>
>> Sent from my iPad
>>
>>> On 15 Nov 2020, at 10:57, Marco Antoniotti <marco.antoniotti@unimib.it> wrote:
>>>
>>> 
>>> Hi Pascal,
>>>
>>> I am sure you have it somewhere, but could you repeat here a simple explanation of what DYNAMIC-WIND and PROCEED do?
>>>
>>> Thanks
>>>
>>> Marco
>
> _______________________________________________
> Lisp Hug - the mailing list for LispWorks users
> lisp-hug@lispworks.com
> http://www.lispworks.com/support/lisp-hug.html

_______________________________________________
Lisp Hug - the mailing list for LispWorks users
lisp-hug@lispworks.com
http://www.lispworks.com/support/lisp-hug.html

Re: Dynamic Closures?

You can see a live example in use in lines 339-369 of https://github.com/dbmcclain/Lisp-Actors/blob/main/Actors/actors.lisp

Then whenever I use an =BIND (a la Paul Graham) it constructs a closure of the body code before calling the code that produces the binding values. Think of =BIND as like MULTIPLE-VALUE-BIND or DESTRUCTURING-BIND. But in my system, I process that body code closure with a function =CONT that calls CAPTURE-DYNAMIC-ENVIRONMENT, then returns a closure that re-instates the dynamic context before calling the continuation closure.

;; -------------------------------------------

;; Turn a simple closure into a Continuation

(defun =cont (fn)

(let ((dyn-env (capture-dynamic-environment)))

(lambda (&rest args)

(with-dynamic-environment (dyn-env)

(apply fn args)))))

But if you like, I can write this up in a more connected format…

- DM

On Nov 15, 2020, at 4:06 AM, Michał phoe Herda <phoe@disroot.org> wrote:

Hmmm. Would you mind writing some sort of an article that describes and
exemplifies these? I don't think I have encountered any such piece of
writing before, and it would help me visualize what's going on under the
hood; I assume it could help other people with understanding this, too.

~phoe

On 15.11.2020 12:01, dbm@refined-audiometrics.com wrote:
DYNAMIC-WIND - maintains a stack of closures. Each one consists of the an FLET on the body enclosed. But there is a MACROLET PROCEED inside the FLET. Wherever PROCEED is used, it becomes replaced by an IF statement. The FLET itself is parameterized by a thunk argument. Initially, the FLET is called with a null thunk. The IF of the PROCEED checks that thunk, and if null, it executes the code contained within the PROCEED macro. Otherwise it calls the thunk.

After some succession of DYNAMIC-WIND groups, you can call CAPTURE-DYNAMIC-ENVIRONMENT to produce a captured continuation state.

Later, you can invoke any function you like within the dynamic context at the time you captured it using CALL-WITH-DYNAMIC-ENVIROMENT, or using macro WITH-DYNAMIC-ENVIRONMENT. These reinstate the dynamic environment that you captured, before calling your function code. It calls the saved DYNAMIC-WIND flet functions in the same order as when they were created, ahead of your capturing of the dynamic context.

Pretty simple, really, but very clever and powerful. To my mind’s eye, I find the syntax considerably easier to grok than most implementations of delimited continuations, even though they accomplish essentially the same thing. I find DYNAMIC-WIND and PROCEED much more in line with Lisp syntax. Even more direct than Scheme’s CALL/CC.

- DM

On Nov 15, 2020, at 3:39 AM, Pascal Costanza <pc@p-cos.net> wrote:

Hi Marco,

Unfortunately, it has been a while since I wrote this code, so I would need to immerse myself again in my code for a bit to figure out myself what I was trying to achieve. Maybe David is in a better position to explain this right now. :)

Pascal

Sent from my iPad

On 15 Nov 2020, at 10:57, Marco Antoniotti <marco.antoniotti@unimib.it> wrote:

Hi Pascal,

I am sure you have it somewhere, but could you repeat here a simple explanation of what DYNAMIC-WIND and PROCEED do?

Thanks

Marco

_______________________________________________
Lisp Hug - the mailing list for LispWorks users
lisp-hug@lispworks.com
http://www.lispworks.com/support/lisp-hug.html

Re: Dynamic Closures?

The code example I showed is used in Actors. Actors are guaranteed single-thread semantics. But if you call something that might block, you could choose to spawn that off into another worker (sub-actor) so that the main Actor isn’t tied up.

Problem is… when you have someone asking for the result of an Actor computation, there is no immediate answer available. So we set up a continuation callback, so that when the task is completed, that continuation will run once again in the actor. And if we were in the midst of an ASK by some client code, then we should send the result back to them. 

So the example code I gave, allows for an ASK that might not be able to complete right now. But it can be restarted later to finish the job. And using DYNAMIC-WIND makes this so easy to do.

- DM

> On Nov 15, 2020, at 4:06 AM, Michał phoe Herda <phoe@disroot.org> wrote:
> 
> Hmmm. Would you mind writing some sort of an article that describes and
> exemplifies these? I don't think I have encountered any such piece of
> writing before, and it would help me visualize what's going on under the
> hood; I assume it could help other people with understanding this, too.
> 
> ~phoe
> 
> On 15.11.2020 12:01, dbm@refined-audiometrics.com wrote:
>> DYNAMIC-WIND - maintains a stack of closures. Each one consists of the an FLET on the body enclosed. But there is a MACROLET PROCEED inside the FLET. Wherever PROCEED is used, it becomes replaced by an IF statement. The FLET itself is parameterized by a thunk argument. Initially, the FLET is called with a null thunk. The IF of the PROCEED checks that thunk, and if null, it executes the code contained within the PROCEED macro. Otherwise it calls the thunk.
>> 
>> After some succession of DYNAMIC-WIND groups, you can call CAPTURE-DYNAMIC-ENVIRONMENT to produce a captured continuation state.
>> 
>> Later, you can invoke any function you like within the dynamic context at the time you captured it using CALL-WITH-DYNAMIC-ENVIROMENT, or using macro WITH-DYNAMIC-ENVIRONMENT. These reinstate the dynamic environment that you captured, before calling your function code. It calls the saved DYNAMIC-WIND flet functions in the same order as when they were created, ahead of your capturing of the dynamic context.
>> 
>> Pretty simple, really, but very clever and powerful. To my mind’s eye, I find the syntax considerably easier to grok than most implementations of delimited continuations, even though they accomplish essentially the same thing. I find DYNAMIC-WIND and PROCEED much more in line with Lisp syntax. Even more direct than Scheme’s CALL/CC.
>> 
>> - DM
>> 
>> 
>>> On Nov 15, 2020, at 3:39 AM, Pascal Costanza <pc@p-cos.net> wrote:
>>> 
>>> Hi Marco,
>>> 
>>> Unfortunately, it has been a while since I wrote this code, so I would need to immerse myself again in my code for a bit to figure out myself what I was trying to achieve. Maybe David is in a better position to explain this right now. :)
>>> 
>>> Pascal
>>> 
>>> Sent from my iPad
>>> 
>>>> On 15 Nov 2020, at 10:57, Marco Antoniotti <marco.antoniotti@unimib.it> wrote:
>>>> 
>>>> 
>>>> Hi Pascal,
>>>> 
>>>> I am sure you have it somewhere, but could you repeat here a simple explanation of what DYNAMIC-WIND and PROCEED do?
>>>> 
>>>> Thanks
>>>> 
>>>> Marco
>> 
>> _______________________________________________
>> Lisp Hug - the mailing list for LispWorks users
>> lisp-hug@lispworks.com
>> http://www.lispworks.com/support/lisp-hug.html


_______________________________________________
Lisp Hug - the mailing list for LispWorks users
lisp-hug@lispworks.com
http://www.lispworks.com/support/lisp-hug.html