1. Compiled code always works properly, seeing my intended lexical bindings as purely lexical. So I can put together the following kind of macro (bending some rules…)
(defmacro make-actor (name args state &body body &environment env)
(let* ((a!self (anaphor 'self))
(inner `(let (,a!self) ;; <— Build the macro expansion here, called inner...
(setf ,a!self (make-instance 'Actor
:name ,(if (consp name)
name
`',name)
:lambda-list ',args
:behav (behav ,args ,state ,@body)))
(add-actor ,a!self)
,a!self)))
(if (some (um:curry #'slot-value env) ;; <— peek where I probably shouldn't
'(compiler::compilation-env
compiler::fenv
compiler::venv))
inner ;; <— for assumed compile mode
;; else - we must be in eval mode...
`(funcall (compile nil (lambda ()
,inner))) )
))
And so far this always works, whether in eval mode in the Listener or Editor, or when compiling under ASDF.
2. The code that I’m producing with this macro actually has two portions. There is an outer frame that constructs an instance of Actor class and then installs it into the running system, after first backpatching a self-referential lexical binding. Then the inner frame is a portion of code actually stored inside that Actor instance.
You can see that outer frame in the example just shown. I construct a lexical binding named SELF, create that Actor instance, add to the running system, and return that instance. The (behav …) macro constructs the inner lexical closure that gets stored inside the Actor instance. And yes it *is* a lexical closure because SELF may be free in the body of the code, attempting to reference that outer binding shown above.
And yes, I understand that you can’t compile closures, only functions that construct closures. See what I do in that last line during eval mode to get things fully compiled anyway.
3. When I don’t fully compile using this treachery, and leave the code in interpreted form, the outer portion that constructs the Actor instance actually runs properly in the REPL thread. What doesn’t run properly is that inner eval mode code that gets embedded into the Actor instance. And the Actor always runs in some other Executive thread, not the Listener or Editor threads.
If there are no Executive threads when the expanded macro evaluates the (ADD-ACTOR …) clause, then Executive threads will be spawned to fill a thread pool. Under that situation, if the ADD-ACTOR was performed in eval mode in the Listener or Editor, then the resulting spawned Executive threads will fail to recognize inner references to SELF as lexical, and instead complain that my references are to an unbound symbol SELF. But that only happens if the free references are relatively deep inside the embedded code, such as within a contained LAMBDA form or inside a LABELS code body. Outer level references to SELF succeed.
4. Things always work properly if code is fully compiled. Things also work properly *IF* the very first thing performed in a newly spawned Executive thread happen to be eval-mode expressions from the Listener pane. But not if the very first thing performed in the Executive comes from compiled code.
——
Now, I don’t see any shared reference problems here, and it really appears to me that some residual setting from the compiling thread doesn’t get carried over to the Executive thread, when that Executive doesn’t get spawned during a REPL eval.
That’s why I thought the new threads were capturing some compiler environment for themselves. If they spawn during a REPL eval, that compiler environment info seems to get captured into the new threads, and they perform just fine. But if the newly spawned Executive threads come instead from a previously compiled expression evaluation, then they don’t have sufficient information to see that my future eval mode lexical bindings are lexical, not to unbound globals.
But, there’s always the possibility that I’m not seeing something that I caused for myself…
Cheers,
- DM
I'm not convinced that this is anything to do with threads, because they don't
have a lexical environment themselves.
Are you sure it isn't some variant of the "shared binding" problem:
(mapcar 'funcall (loop for x below 4 collect (lambda () x)))
which returns (4 4 4 4) rather than (0 1 2 3)?
--
Martin Simmons
LispWorks Ltd
http://www.lispworks.com/On Sat, 21 Oct 2017 10:51:20 -0700, David McClain said:
On second reflection, that doesn’t totally solve the problem.
The problem of incorrect interpretation — thinking that lexical references are instead to unbound globals — happens whenever the thread pool is recreated as a result of a simple non-binding form making the pool creation request at the REPL, or from automatically triggered compiled code.
But if the first thing to request the pool is an interpreted form which deep internal lexical bindings and macros, then the threads of a new pool formed during interpretation of that form are able to correctly evaluate the form. So this issue really has nothing to do with the Actions list.
It appears that macros used inside of interpreted forms are not examined until runtime. And when they rewrite themselves to executable form, the interpreter evaluates arguments of resulting function calls before calling the function. To do that, the interpreter needs to know where the value is for each symbol mentioned in the argument list.
When the pool is created while interpreting such a form, the environment is captured by all threads, so this environment must be a global. And that environment is rich enough to know the lexical bindings being referenced during arg evaluation. And so any of those threads can successfully interpret the user’s request.
But if the first thing you do which creates threads, does not need any such compilation environment, the thread-captured environment will be too poor to support proper interpretation of additional REPL forms.
So it appears that we somehow need to always spawn threads while a rich enough global compiler environment is available to be captured by each thread. That seems to do nothing to impair performance of compiled code, but it allows any thread to properly interpret REPL forms typed in by the user.
The problem I ran into is that the REPL in the Listener thread actually ran to completion properly. But what it produced were objects that were thrown at a pool of threads, and those objects further contained more interpreted code. By the time the threads caught the code to interpret for themselves, they either had a sufficiently rich environment captured from the Listener, or they didn’t. It was a toss up as to when your code would / would not run properly in interpreter mode.
The other solution, is to never use interpreted code. Always fully compile everything sent to a thread. That always seems to work properly.
This is not a criticism, but rather an attempt to understand the boundaries of the Lisp system under multithreading. I think I’m beginning to grasp some of it…
- DM
_______________________________________________
Lisp Hug - the mailing list for LispWorks users
lisp-hug@lispworks.com
http://www.lispworks.com/support/lisp-hug.html
_______________________________________________
Lisp Hug - the mailing list for LispWorks users
lisp-hug@lispworks.com
http://www.lispworks.com/support/lisp-hug.html