Lisp HUG Maillist Archive

Curious error...

Must be something I just don’t see…

Attempting to set a local variable from another thread seems to fail, but if the variable is global it works…

This version works properly:

(let (x)
  (defun tst ()
    (let ((sem (mp:make-semaphore :count 0)))
      (setf x nil)
      (mp:funcall-async (lambda ()
                          (unwind-protect
                              (setf x (bfly:!? "eval@Malachite.local"
                                               `(hash-scanner::grand-hash "~/projects/Lispworks/tools/FileSync/")))
                            (mp:semaphore-release sem))))
      (mp:semaphore-acquire sem :count 1) 
      (assert x)
      x)))

(tst) => #S(DIR-NODE :KEY "AB1D177EB7748A88990A8F97BFCA71FD3C83ED77D4014805598D561117F2767D2559744FBA20184E33FBBFAD27D2BF37EF77EA094A7974530AF1689B03E33B89" :NAME "~/projects/Lispworks/tools/FileSync/" :FTREE NIL :DTREE NIL)


And this version, with the x as a local, fails - every time:

(defun tst ()
  (let ((x nil)
        (sem (mp:make-semaphore :count 0)))
    (mp:funcall-async (lambda ()
                        (unwind-protect
                            (setf x (bfly:!? "eval@Malachite.local"
                                             `(hash-scanner::grand-hash "~/projects/Lispworks/tools/FileSync/")))
                          (mp:semaphore-release sem))))
    (mp:semaphore-acquire sem :count 1) 
    (assert x)
    x))

(tst) => 

Error: The assertion X inside HASH-SCANNER::TST failed.
  1 (continue) Retry assertion.
  2 (abort) Return to level 0.
  3 Return to top loop level 0.

Type :b for backtrace or :c <option number> to proceed.
Type :bug-form "<subject>" for a bug report template or :? for other options.

I don’t get it. The access to the semaphore appears to work properly in both cases, but access to the x variable fails when it is a frame local instead of a global.

Eh? What am I not seeing?

- DM



_______________________________________________
Lisp Hug - the mailing list for LispWorks users
lisp-hug@lispworks.com
http://www.lispworks.com/support/lisp-hug.html

Re: Curious error...

… never mind that I might be doing something complicated… This one fails too!

(defun tst ()
  (let (x
        (sem (mp:make-semaphore :count 0)))
    (mp:funcall-async (lambda ()
                        (unwind-protect
                            (setf x 15)
                          (mp:semaphore-release sem))))
    (mp:semaphore-acquire sem :count 1)
    (assert x)
    x))

 (tst) =>

Error: The assertion X inside HASH-SCANNER::TST failed.
  1 (continue) Retry assertion.
  2 (abort) Return to level 0.
  3 Return to top loop level 0.

Type :b for backtrace or :c <option number> to proceed.
Type :bug-form "<subject>" for a bug report template or :? for other options.

- DM


_______________________________________________
Lisp Hug - the mailing list for LispWorks users
lisp-hug@lispworks.com
http://www.lispworks.com/support/lisp-hug.html

Re: Curious error...

uh oh… I think I see what is going on here… Special Bindings !! Yikes! You really gotta watch out for unexpected side effects when global special bindings are around… It still seems like a bug that the local rebinding is ignored by another thread. Eh?

- DM

> On Oct 4, 2017, at 22:30, David McClain <dbm@refined-audiometrics.com> wrote:
> 
> … I tracked down that I had been using a defvar named X in earlier parts of the Lisp session. When I rename the local to something never before seen, like xzyyz instead of x, then the local binding version works as expected. That seems like a bug to me…
> 
> - DM
> 
> 
>> On Oct 4, 2017, at 22:22, David McClain <dbm@refined-audiometrics.com> wrote:
>> 
>> … never mind that I might be doing something complicated… This one fails too!
>> 
>> (defun tst ()
>> (let (x
>>       (sem (mp:make-semaphore :count 0)))
>>   (mp:funcall-async (lambda ()
>>                       (unwind-protect
>>                           (setf x 15)
>>                         (mp:semaphore-release sem))))
>>   (mp:semaphore-acquire sem :count 1)
>>   (assert x)
>>   x))
>> 
>> (tst) =>
>> 
>> Error: The assertion X inside HASH-SCANNER::TST failed.
>> 1 (continue) Retry assertion.
>> 2 (abort) Return to level 0.
>> 3 Return to top loop level 0.
>> 
>> Type :b for backtrace or :c <option number> to proceed.
>> Type :bug-form "<subject>" for a bug report template or :? for other options.
>> 
>> - DM
>> 
> 


_______________________________________________
Lisp Hug - the mailing list for LispWorks users
lisp-hug@lispworks.com
http://www.lispworks.com/support/lisp-hug.html

Re: Curious error...

… I tracked down that I had been using a defvar named X in earlier parts of the Lisp session. When I rename the local to something never before seen, like xzyyz instead of x, then the local binding version works as expected. That seems like a bug to me…

- DM


> On Oct 4, 2017, at 22:22, David McClain <dbm@refined-audiometrics.com> wrote:
> 
> … never mind that I might be doing something complicated… This one fails too!
> 
> (defun tst ()
>  (let (x
>        (sem (mp:make-semaphore :count 0)))
>    (mp:funcall-async (lambda ()
>                        (unwind-protect
>                            (setf x 15)
>                          (mp:semaphore-release sem))))
>    (mp:semaphore-acquire sem :count 1)
>    (assert x)
>    x))
> 
> (tst) =>
> 
> Error: The assertion X inside HASH-SCANNER::TST failed.
>  1 (continue) Retry assertion.
>  2 (abort) Return to level 0.
>  3 Return to top loop level 0.
> 
> Type :b for backtrace or :c <option number> to proceed.
> Type :bug-form "<subject>" for a bug report template or :? for other options.
> 
> - DM
> 


_______________________________________________
Lisp Hug - the mailing list for LispWorks users
lisp-hug@lispworks.com
http://www.lispworks.com/support/lisp-hug.html

Re: Curious error...




On 5 Oct 2017, at 07:33, David McClain <dbm@refined-audiometrics.com> wrote:

uh oh… I think I see what is going on here… Special Bindings !! Yikes! You really gotta watch out for unexpected side effects when global special bindings are around… It still seems like a bug that the local rebinding is ignored by another thread. Eh?

This is why all the global special variables must be named with stars! *X* not X.


-- 
__Pascal J. Bourguignon__

Re: Curious error...

Surely you jest… I always thought that was just an earmuff convention, not a requirement!

This simply cannot be correct….

(defun tst ()
  (let ((xxx 15))
    (mp:funcall-async #'(lambda ()
                          (print (list xxx))))
    ))

(tst) -> (15)

(defvar xxx 32)

(tst) -> 15

Now recompile the tst function and try again…

(tst) -> (32)  !!?

(unintern ‘xxx)

now recompile again

(tst) -> (15)

(defvar xxx 32)

(tst) -> 15

recompile…

(tst) -> (32)









On Oct 4, 2017, at 23:54, Pascal Bourguignon <pjb@informatimago.com> wrote:




On 5 Oct 2017, at 07:33, David McClain <dbm@refined-audiometrics.com> wrote:

uh oh… I think I see what is going on here… Special Bindings !! Yikes! You really gotta watch out for unexpected side effects when global special bindings are around… It still seems like a bug that the local rebinding is ignored by another thread. Eh?

This is why all the global special variables must be named with stars! *X* not X.


-- 
__Pascal J. Bourguignon__

Re: Curious error...

… so what happened to our closure? 

If you unintentionally use a local binding that happens to be a synonym of some already defined global symbol, then you are in for a world of hurt. And if you have correct running code and then later selectively recompile sections of your code after having made some additional unrelated declarations later on in your session then your code will stop working with bizarre and unexpected behaviors….

Surely, this cannot be correct… Why is the compiler refusing to use the closure lookup, and resorting to the nearest global declaration in lieu of all local rebindings?

- DM


On Oct 4, 2017, at 23:59, David McClain <dbm@refined-audiometrics.com> wrote:

Surely you jest… I always thought that was just an earmuff convention, not a requirement!

This simply cannot be correct….

(defun tst ()
  (let ((xxx 15))
    (mp:funcall-async #'(lambda ()
                          (print (list xxx))))
    ))

(tst) -> (15)

(defvar xxx 32)

(tst) -> 15

Now recompile the tst function and try again…

(tst) -> (32)  !!?

(unintern ‘xxx)

now recompile again

(tst) -> (15)

(defvar xxx 32)

(tst) -> 15

recompile…

(tst) -> (32)









On Oct 4, 2017, at 23:54, Pascal Bourguignon <pjb@informatimago.com> wrote:




On 5 Oct 2017, at 07:33, David McClain <dbm@refined-audiometrics.com> wrote:

uh oh… I think I see what is going on here… Special Bindings !! Yikes! You really gotta watch out for unexpected side effects when global special bindings are around… It still seems like a bug that the local rebinding is ignored by another thread. Eh?

This is why all the global special variables must be named with stars! *X* not X.


-- 
__Pascal J. Bourguignon__


Re: Curious error...

I am simply stunned by this discovery… I have been using Common Lisp for nearly 30 years, and I never realized this peculiar behavior. The Lisp Hyperspec has many examples of local rebindings of globally declared bindings, encouraging this by example.

Pascal is correct, at least as far as LW goes. Is this really the specified behavior of Common Lisp? Have I been spoiled by having used Scheme, OCaml, SML? If you follow Pascal’s advise then you will minimize problems along the way. But I am astonished that this is the accepted behavior for Common Lisp.

- DM


On Oct 5, 2017, at 00:04, David McClain <dbm@refined-audiometrics.com> wrote:

… so what happened to our closure? 

If you unintentionally use a local binding that happens to be a synonym of some already defined global symbol, then you are in for a world of hurt. And if you have correct running code and then later selectively recompile sections of your code after having made some additional unrelated declarations later on in your session then your code will stop working with bizarre and unexpected behaviors….

Surely, this cannot be correct… Why is the compiler refusing to use the closure lookup, and resorting to the nearest global declaration in lieu of all local rebindings?

- DM


On Oct 4, 2017, at 23:59, David McClain <dbm@refined-audiometrics.com> wrote:

Surely you jest… I always thought that was just an earmuff convention, not a requirement!

This simply cannot be correct….

(defun tst ()
  (let ((xxx 15))
    (mp:funcall-async #'(lambda ()
                          (print (list xxx))))
    ))

(tst) -> (15)

(defvar xxx 32)

(tst) -> 15

Now recompile the tst function and try again…

(tst) -> (32)  !!?

(unintern ‘xxx)

now recompile again

(tst) -> (15)

(defvar xxx 32)

(tst) -> 15

recompile…

(tst) -> (32)









On Oct 4, 2017, at 23:54, Pascal Bourguignon <pjb@informatimago.com> wrote:




On 5 Oct 2017, at 07:33, David McClain <dbm@refined-audiometrics.com> wrote:

uh oh… I think I see what is going on here… Special Bindings !! Yikes! You really gotta watch out for unexpected side effects when global special bindings are around… It still seems like a bug that the local rebinding is ignored by another thread. Eh?

This is why all the global special variables must be named with stars! *X* not X.


-- 
__Pascal J. Bourguignon__



Re: Curious error...


On 5 Oct 2017, at 08:59, David McClain <dbm@refined-audiometrics.com> wrote:

Surely you jest… I always thought that was just an earmuff convention, not a requirement!

This simply cannot be correct….

(defun tst ()
  (let ((xxx 15))
    (mp:funcall-async #'(lambda ()
                          (print (list xxx))))
    ))

(tst) -> (15)

(defvar xxx 32)

(tst) -> 15

Now recompile the tst function and try again…

(tst) -> (32)  !!?

(unintern ‘xxx)

now recompile again

(tst) -> (15)

(defvar xxx 32)

(tst) -> 15

recompile…

(tst) -> (32)


This is why all the global special variables must be named with stars! *X* not X.


(defun tst ()
  (let ((xxx 15))
    (bt:join-thread (bt:make-thread (lambda ()
                                      (declare (special *xxx*))
                                      (print (list xxx (when (boundp '*xxx*) *xxx*))))))))

(tst) ; --> (15 nil)

(defvar *xxx* 32)

(tst) ; --> (15 32)

;; Now recompile the tst function and try again…
(compile 'tst)

(tst) ; --> (15 32)

(unintern '*xxx*)

;; now recompile again
(compile 'tst)

(tst) ; --> (15 32)  ; recompiling doesn't re-read the source.

(defun tst ()
  (let ((xxx 15))
    (bt:join-thread (bt:make-thread (lambda ()
                                      (declare (special *xxx*))
                                      (print (list xxx (when (boundp '*xxx*) *xxx*))))))))
(tst) ; --> (15 nil)

(defvar *xxx* 32)

(tst) ; --> (15 32)

;; recompile…
(compile 'tst)

(tst) ; --> (15 32)




On Oct 4, 2017, at 23:54, Pascal Bourguignon <pjb@informatimago.com> wrote:




On 5 Oct 2017, at 07:33, David McClain <dbm@refined-audiometrics.com> wrote:

uh oh… I think I see what is going on here… Special Bindings !! Yikes! You really gotta watch out for unexpected side effects when global special bindings are around… It still seems like a bug that the local rebinding is ignored by another thread. Eh?

This is why all the global special variables must be named with stars! *X* not X.


-- 
__Pascal J. Bourguignon__


Re: Curious error...

There must have been some early historical confusion about just how lexical binding should work. Scheme came on stage and Common Lisp tried to emulate its binding behavior. But I see that under the hood, the early understanding of lexical binding is not the same as we commonly assume today. Even lowly C use the modern understanding of lexical binding - although it cannot understand special bindings by default.

Having written numerous compilers for myself, always with the modern understanding of local lexical binding taking precedence over anything in the global environment, I have led myself to believe that all lexical binding languages work the same way. Apparently not.

It is very interesting, to me, that we have managed to do so much with Common Lisp, and yet its bones are so primitive. I had always taken the earmuff convention as just a convention. And I’m sure I’m not alone. But Pascal really does point out an important aspect of Common Lisp. Beware! It is required.

- DM


_______________________________________________
Lisp Hug - the mailing list for LispWorks users
lisp-hug@lispworks.com
http://www.lispworks.com/support/lisp-hug.html

Re: Curious error...

.. as an aside… when I recompiled the tst function, I did it in the editor, forcing it to reread the source code. 

I’m still absorbing your comments about recompiling not rereading the source when you use (compile ‘tst).

On Oct 5, 2017, at 00:22, Pascal Bourguignon <pjb@informatimago.com> wrote:


On 5 Oct 2017, at 08:59, David McClain <dbm@refined-audiometrics.com> wrote:

Surely you jest… I always thought that was just an earmuff convention, not a requirement!

This simply cannot be correct….

(defun tst ()
  (let ((xxx 15))
    (mp:funcall-async #'(lambda ()
                          (print (list xxx))))
    ))

(tst) -> (15)

(defvar xxx 32)

(tst) -> 15

Now recompile the tst function and try again…

(tst) -> (32)  !!?

(unintern ‘xxx)

now recompile again

(tst) -> (15)

(defvar xxx 32)

(tst) -> 15

recompile…

(tst) -> (32)


This is why all the global special variables must be named with stars! *X* not X.


(defun tst ()
  (let ((xxx 15))
    (bt:join-thread (bt:make-thread (lambda ()
                                      (declare (special *xxx*))
                                      (print (list xxx (when (boundp '*xxx*) *xxx*))))))))

(tst) ; --> (15 nil)

(defvar *xxx* 32)

(tst) ; --> (15 32)

;; Now recompile the tst function and try again…
(compile 'tst)

(tst) ; --> (15 32)

(unintern '*xxx*)

;; now recompile again
(compile 'tst)

(tst) ; --> (15 32)  ; recompiling doesn't re-read the source.

(defun tst ()
  (let ((xxx 15))
    (bt:join-thread (bt:make-thread (lambda ()
                                      (declare (special *xxx*))
                                      (print (list xxx (when (boundp '*xxx*) *xxx*))))))))
(tst) ; --> (15 nil)

(defvar *xxx* 32)

(tst) ; --> (15 32)

;; recompile…
(compile 'tst)

(tst) ; --> (15 32)




On Oct 4, 2017, at 23:54, Pascal Bourguignon <pjb@informatimago.com> wrote:




On 5 Oct 2017, at 07:33, David McClain <dbm@refined-audiometrics.com> wrote:

uh oh… I think I see what is going on here… Special Bindings !! Yikes! You really gotta watch out for unexpected side effects when global special bindings are around… It still seems like a bug that the local rebinding is ignored by another thread. Eh?

This is why all the global special variables must be named with stars! *X* not X.


-- 
__Pascal J. Bourguignon__

Re: Curious error...

I always try to avoid using any global vars, when possible. Some people write Lisp like Fortran, but not me.

However, when diddling around in the listener, I frequently do a (setf x (some-computation-result …)) just to hang on to some long winded calculation for inspection and further diddling. This creates an (unintentional) special binding.

I guess I don’t know any way to create global non-special bindings, except to enclose the entire body of code inside some let-bindings. And there is no (declare not-special) available. Hence the leading of thinking to conclude that all local bindings always override any global bindings. But simply not so. A Common Lisp closure is not the same beast as an OCaml closure.

- DM


_______________________________________________
Lisp Hug - the mailing list for LispWorks users
lisp-hug@lispworks.com
http://www.lispworks.com/support/lisp-hug.html

Re: Curious error...

Hmmm… not sure we are talking about the same thing. Chapter 2.9 talks about COMPILE and COMPILE-FILE, not about bindings.

The specific problem I am addressing is the failure in multithreaded code to respect the local bindings furnished by a closure. Local bindings do always seem to override global bindings, even when they have the same name, as long as you stay in the resident thread. But constructing a closure with local bindings is not respected by foreign threads executing the code.

So, sans multithreading, Common Lisp would behave the same way as Scheme, SML, OCaml, and any number of other lexically bound languages. The problems arise when asking another thread to perform your closure.

- DM





On Oct 5, 2017, at 00:42, Dimitri Georganas <dg@biodys.com> wrote:

Lisp newbie here, reading On Lisp. Chapter 2.9 seems to explain this behaviour. 


On Thu, Oct 5, 2017 at 8:59 AM, David McClain <dbm@refined-audiometrics.com> wrote:
Surely you jest… I always thought that was just an earmuff convention, not a requirement!

This simply cannot be correct….

(defun tst ()
  (let ((xxx 15))
    (mp:funcall-async #'(lambda ()
                          (print (list xxx))))
    ))

(tst) -> (15)

(defvar xxx 32)

(tst) -> 15

Now recompile the tst function and try again…

(tst) -> (32)  !!?

(unintern ‘xxx)

now recompile again

(tst) -> (15)

(defvar xxx 32)

(tst) -> 15

recompile…

(tst) -> (32)









On Oct 4, 2017, at 23:54, Pascal Bourguignon <pjb@informatimago.com> wrote:




On 5 Oct 2017, at 07:33, David McClain <dbm@refined-audiometrics.com> wrote:

uh oh… I think I see what is going on here… Special Bindings !! Yikes! You really gotta watch out for unexpected side effects when global special bindings are around… It still seems like a bug that the local rebinding is ignored by another thread. Eh?

This is why all the global special variables must be named with stars! *X* not X.


-- 
__Pascal J. Bourguignon__



Re: Curious error...

The problem seems to entwined with the nature of special bindings. I had always thought that compiled code really compiled in lexical closure accesses when nearest lexical bindings are local. If you really want special binding behavior you either have to access a global special binding, or else use a (declare (special xx)) in the closure.

But there must be a runtime variation to allow for the possibility of special binding behavior. And that runtime switches back to some default setup when another thread is spawned on your closure. Apparently the runtime switching happens just ahead of actually accessing a closure local binding.

Given the need for (declare (special ..)) I am surprised that the compiler doesn’t just short circuit the non-special binding accesses and references.

- DM


_______________________________________________
Lisp Hug - the mailing list for LispWorks users
lisp-hug@lispworks.com
http://www.lispworks.com/support/lisp-hug.html

Re: Curious error...

* David McClain Wrote on Thu, 5 Oct 2017 00:30:22 -0700:

> There must have been some early historical confusion about just how
> lexical binding should work. Scheme came on stage and Common Lisp
> tried to emulate its binding behavior. But I see that under the hood,
> the early understanding of lexical binding is not the same as we
> commonly assume today. Even lowly C use the modern understanding of
> lexical binding - although it cannot understand special bindings by
> default.
>
> Having written numerous compilers for myself, always with the modern
> understanding of local lexical binding taking precedence over anything
> in the global environment, I have led myself to believe that all
> lexical binding languages work the same way. Apparently not.
>
> It is very interesting, to me, that we have managed to do so much with
> Common Lisp, and yet its bones are so primitive. I had always taken
> the earmuff convention as just a convention. And I’m sure I’m not
> alone. But Pascal really does point out an important aspect of Common
> Lisp. Beware! It is required.

Note Lispworks supports the earmuff convention, even warning you when a
earmuffed variable is bound lexically

* (compile nil (lambda () (let ((*foo* 10)) (expt *foo* 2))))
;;;*** Warning in 12: *FOO* bound lexically.


Lispworks also does not autodeclare variables as special when you use
setq, say in the REPL, (I believe you will disagree with my position
that it should).  Had you used (setq xxx) instead of (defvar xxx) in the
repl, then xxx would not have been globally special in lispworks. (My
position is that it should).

When you explicitly use a defvar or defparameter, the consequences are
well defined in the spec, and the game is over, it can never become a
lexical again. CL behaviour has been documented and consistent and
useful.   The examples in the spec however are buggy.

[I believed this trap you fell into is one of the first things that one
 learns when encountering CL, and after falling once, then one learns to
 use the behaviour to ones advantage.  I also notice an early
 ideological grounding in Scheme tends to close the mind to exploiting
 CL's behaviour in one's code.]

_______________________________________________
Lisp Hug - the mailing list for LispWorks users
lisp-hug@lispworks.com
http://www.lispworks.com/support/lisp-hug.html

Re: Curious error...

I guess I must now operate in a new aspect domain, cross-cutting my former view of Lisp. I think I know what you mean about Scheme biasing the use of the language, and I have intentionally avoided dynamic bindings except in a few rare instances. But that comes more from my early exposure to writing Pascal and C compilers, than any exposure to Scheme. (I actually began in earnest in Forth. So I have traveled far…)

Now I must think about how I could use this more creatively than before.

- DM

_______________________________________________
Lisp Hug - the mailing list for LispWorks users
lisp-hug@lispworks.com
http://www.lispworks.com/support/lisp-hug.html

Re: Curious error...




On 5 Oct 2017, at 09:33, David McClain <dbm@refined-audiometrics.com> wrote:

.. as an aside… when I recompiled the tst function, I did it in the editor, forcing it to reread the source code. 

I’m still absorbing your comments about recompiling not rereading the source when you use (compile ‘tst).


The meaning is that what is declared SPECIAL is the symbol naming the variable.
DEFVAR and DEFPARAMETER mark the name of the variable SPECIAL globally.
You can also do the same with a DECLAIM or PROCLAIM form.
There is no way to remove the global declaration, that is the SPECIAL mark put on symbol.

On the other hand, the local declaration SPECIAL is processed by the compiler (or interpreter), and doesn’t mark the symbol itself.

So indeed COMPILE only re-process the source of the function. It doesn’t read it again, so the symbols are directly referenced, and the only transition possible here is from non-SPECIAL to SPECIAL.
UNINTERN has no effect there (it only makes it more difficult for external code to access to the symbols in the source form).

On the other hand COMPILE-FILE will read a text file and intern all the symbols. So if you uninterned a symbol, you will get a newly interned symbol, which is different, and virgin, without a SPECIAL mark if you don’t declare it thus in the source.



See also:
https://groups.google.com/forum/#!msg/comp.lang.lisp/4VyopdWcFI4/1sDQU-3H8VgJ

-- 
__Pascal J. Bourguignon__

Re: Curious error...


> On 5 Oct 2017, at 09:41, David McClain <dbm@refined-audiometrics.com> wrote:
> 
> I always try to avoid using any global vars, when possible. Some people write Lisp like Fortran, but not me.
> 
> However, when diddling around in the listener, I frequently do a (setf x (some-computation-result …)) just to hang on to some long winded calculation for inspection and further diddling. This creates an (unintentional) special binding.
> 
> I guess I don’t know any way to create global non-special bindings, except to enclose the entire body of code inside some let-bindings. And there is no (declare not-special) available. Hence the leading of thinking to conclude that all local bindings always override any global bindings. But simply not so. A Common Lisp closure is not the same beast as an OCaml closure.



symbol-macros are global lexical, so you can use them to implement global lexical variables. There are libraries with such a defglobal or deflex macro.

-- 
__Pascal J. Bourguignon__


_______________________________________________
Lisp Hug - the mailing list for LispWorks users
lisp-hug@lispworks.com
http://www.lispworks.com/support/lisp-hug.html

Re: Curious error...



On 5 Oct 2017, at 09:49, David McClain <dbm@refined-audiometrics.com> wrote:

Hmmm… not sure we are talking about the same thing. Chapter 2.9 talks about COMPILE and COMPILE-FILE, not about bindings.

The specific problem I am addressing is the failure in multithreaded code to respect the local bindings furnished by a closure. Local bindings do always seem to override global bindings, even when they have the same name, as long as you stay in the resident thread. But constructing a closure with local bindings is not respected by foreign threads executing the code.

So, sans multithreading, Common Lisp would behave the same way as Scheme, SML, OCaml, and any number of other lexically bound languages. The problems arise when asking another thread to perform your closure.


The problem is unrelated to threads. It’s a problem of closure: you cannot enclose dynamic bindings, only lexical bindings can be enclosed in a closure.

(defun test-local ()
  (let ((x 42))
    (declare (special x))
    (let ((f  (lambda () (when (boundp 'x) x))))
      (values f (funcall f)))))


(defvar *x*)
(defun test-global ()
  (let ((*x* 33))
    (let ((f  (lambda () (when (boundp '*x*) *x*))))
      (values f (funcall f)))))

(multiple-value-bind (f v) (test-local)
  (values (funcall f) v))
--> nil
    42

(multiple-value-bind (f v) (test-global)
  (values (funcall f) v))
--> nil
    33

-- 
__Pascal J. Bourguignon__

Re: Curious error...

Yes, after altering my vantage point, I can see that it isn’t a problem of closures. I can see that the compiler is searching the global environment for special bindings before committing to closure reference.

By way of inspection of symbols, there seems no way to discern specialness. No flags mark these symbols as special, at least as reported in an inspector. So here is my helper:

(defmacro test-special (sym)
  (let ((gf (gensym)))
    `(let ((,gf (let ((,sym :lexical))
                  (lambda ()
                    ,sym))))
       (let ((,sym :dynamic))
         (funcall ,gf)))))

(test-special x) => :lexical

(defvar x 15)

(test-special x) => :dynamic

This reminds me a bit of the variable capture problem in Scheme. Thankfully, with an earmuff convention it isn’t nearly as big a problem as Scheme has with variable capture.

- DM


_______________________________________________
Lisp Hug - the mailing list for LispWorks users
lisp-hug@lispworks.com
http://www.lispworks.com/support/lisp-hug.html

Re: Curious error...

In Lispworks, there's #'system:declared-special-p, which may be what you want:

CL-USER 13 > (system:declared-special-p '*read-eval*)
T


On 5 Oct 2017, at 12:22 , David McClain <dbm@refined-audiometrics.com> wrote:

By way of inspection of symbols, there seems no way to discern specialness. No flags mark these symbols as special, at least as reported in an inspector. So here is my helper:

Re: Curious error...

Hi,
There is also LW's

(hcl:variable-information '*read-eval*)
:SPECIAL
NIL
NIL

Br
/Alexey

On Thu, Oct 5, 2017 at 12:43 PM, Raymond Wiker <rwiker@gmail.com> wrote:
In Lispworks, there's #'system:declared-special-p, which may be what you want:

CL-USER 13 > (system:declared-special-p '*read-eval*)
T


On 5 Oct 2017, at 12:22 , David McClain <dbm@refined-audiometrics.com> wrote:

By way of inspection of symbols, there seems no way to discern specialness. No flags mark these symbols as special, at least as reported in an inspector. So here is my helper:


Re: Curious error...

Thanks for those additional hints and tools. 

I guess the most alarming thing to me is that I have relied too long on thinking that I was performing intensional programming, where I thought that syntax dictates semantics. That simply isn’t true unless you abide strictly by the earmuff convention. To do otherwise invites the possibility of nondeterminism due to inadvertent name collisions at the REPL listener.

In a single thread, my ignorance was masked by making sometimes dynamic bindings locally, when I didn’t realize I was doing so. The code worked the way I expected within the lexical frame. However, any other functions that might have relied on the value of that global binding might have received a surprise, depending on when they looked.

The multithreaded environment finally opened my eyes, where each thread has its own dynamic binding environment. I gather that is because these dynamic bindings are held in stack frames, and threads each have their own stack.

The situation almost makes me want to define DLET and LLET macros so that I can return to intensional programming.

- DM

On Oct 5, 2017, at 03:51, Alexey Veretennikov <txm.fourier@gmail.com> wrote:

Hi,
There is also LW's

(hcl:variable-information '*read-eval*)
:SPECIAL
NIL
NIL

Br
/Alexey

On Thu, Oct 5, 2017 at 12:43 PM, Raymond Wiker <rwiker@gmail.com> wrote:
In Lispworks, there's #'system:declared-special-p, which may be what you want:

CL-USER 13 > (system:declared-special-p '*read-eval*)
T


On 5 Oct 2017, at 12:22 , David McClain <dbm@refined-audiometrics.com> wrote:

By way of inspection of symbols, there seems no way to discern specialness. No flags mark these symbols as special, at least as reported in an inspector. So here is my helper:



Re: Curious error...

As a happy conclusion to this episode, I checked my cache of 30 years of work in Lisp, and found that I have been pretty good at adhering to the earmuff convention.

But just as a safety precaution against ad-hoc diddling in the REPL, I invented a safety valve macro:

(defun ?specials (syms)
  ;; give us a list of symbols
  ;; return a list of those which are
  ;; declared special as viewed from current package
  (loop for sym in syms
        when (sys:declared-special-p sym)
        collect sym))

(defmacro error-if-special (&rest syms)
  ;; place (user:error-if-special ...syms...) in your code
  ;; Compiler will issue an error if any of them are SPECIAL bindings
  ;; Most useful with multithreaded code to be sure we don't accidentally
  ;; refer to a special binding instead of a lexical binding.
  (let ((specs (?specials syms)))
    (when specs
      (error "Symbols are SPECIAL: ~A" specs))))


The ?SPECIALS is useful from the keyboard if you want to check anticipated use of symbols. But the ERROR-IF-SPECIAL macro is good to plant permanently in the source of multithreaded exploits. A typical use scenario is the following example:

…
  (let* ((sem    (mp:make-semaphore :count 0))
...
    (multiple-value-bind (short-list rest-list)
        (nsplit-list nshort fns)
      (user:error-if-special fn sem)
      (map nil (lambda (fn)
                 (mp:funcall-async (lambda ()
                                     (unwind-protect
                                         (funcall fn)
                                       (mp:semaphore-release sem)))
                                   ))
           rest-list)
…

Here, the lambda being sent to funcall-async internally refers to values belonging to bindings named fn and sem. Both of these were intended to be seen as lexical bindings in this body of code. But if some inadvertent keyboard diddling causes one or both to be specially bound (you can’t ever know what will happen in the future), and if this body of code is subsequently recompiled from its source, the macro error-if-special will abort the recompilation with an error. 

My guess, about the asymmetry in the DECLARE clauses - i.e., declare special, without a corresponding declare lexical, arose from history, and too much old code would break unless we just automatically compiled references to extant special bindings in precedence over lexical bindings. I, personally, don’t much like dealing with accidents waiting to happen, but what can you do?

- DM

On Oct 5, 2017, at 04:19, David McClain <dbm@refined-audiometrics.com> wrote:

Thanks for those additional hints and tools. 

I guess the most alarming thing to me is that I have relied too long on thinking that I was performing intensional programming, where I thought that syntax dictates semantics. That simply isn’t true unless you abide strictly by the earmuff convention. To do otherwise invites the possibility of nondeterminism due to inadvertent name collisions at the REPL listener.

In a single thread, my ignorance was masked by making sometimes dynamic bindings locally, when I didn’t realize I was doing so. The code worked the way I expected within the lexical frame. However, any other functions that might have relied on the value of that global binding might have received a surprise, depending on when they looked.

The multithreaded environment finally opened my eyes, where each thread has its own dynamic binding environment. I gather that is because these dynamic bindings are held in stack frames, and threads each have their own stack.

The situation almost makes me want to define DLET and LLET macros so that I can return to intensional programming.

- DM

On Oct 5, 2017, at 03:51, Alexey Veretennikov <txm.fourier@gmail.com> wrote:

Hi,
There is also LW's

(hcl:variable-information '*read-eval*)
:SPECIAL
NIL
NIL

Br
/Alexey

On Thu, Oct 5, 2017 at 12:43 PM, Raymond Wiker <rwiker@gmail.com> wrote:
In Lispworks, there's #'system:declared-special-p, which may be what you want:

CL-USER 13 > (system:declared-special-p '*read-eval*)
T


On 5 Oct 2017, at 12:22 , David McClain <dbm@refined-audiometrics.com> wrote:

By way of inspection of symbols, there seems no way to discern specialness. No flags mark these symbols as special, at least as reported in an inspector. So here is my helper:




Updated at: 2020-12-10 08:30 UTC