Re: Multiprocessing
On 16/02/2008, at 10:09, Denis Pousseur wrote:
>
> Hello,
>
> I try to learn more about multiprocessing mechanisms, and I would
> like to
> know if I well understand some basics. Thanks by advance for your
> great
> assistance :)
>
> 1)
> If a loop iterate at some time interval and send some tasks to other
> processes : Is it correct to say that, if the order of the tasks in
> the
> different processes will be done in parallel, on the other hand all
> the
> successive tasks (t1, t2, t3...) sent to a same process will be done
> sequentially, and that t2 never will be executed before the
> execution of t1
> is finished ?
>
> 2)
> If a task is always in the mailbox queue of the process, does it
> means that
> it is non executed (and so that it can be aborted) ? Does the
> process do :
>
> pop the queue
> execute the task
>
> OR
>
> execute the task (always in the queue)
> pop the queue
>
> ?
>
> For instance does this code do what I mean (replace a obsolete job
> by a new
> one of the same kind) ?
>
> (without-interrupts
> (when (eq (mp:mailbox-peek (mp:process-mailbox process1)) prev-task)
> (progn (mp:mailbox-read (mp:process-mailbox process1))
> (mp:process-send process1 new-task))))
>
> Or, maybe more efficient :
>
> (without-interrupts
> (when (eq (mp:mailbox-peek (mp:process-mailbox p1)) prev-task)
> (setf (cdr prev-task) (cdr new-task))) ;when the car is the same
>
>
>
> Thanks
>
> Denis
Hi Denis --
Your question is not easy to answer -- there are lots of issues here
and it's hard to figure where to start. My general advice would be
that you do some reading about the principles of doing multiprocessing
-- it is a notoriously difficult area and if you do not get it quite
right you are like to experience subtle errors can be very hard indeed
to track down. Believe me, such subtle errors will steal your time
and hurt your productivity so if you really want to work with multiple
processes, understanding multiprocessing principles will be a
worthwhile investment!
As far as Lispworks is concerned, it offers two synchronization
mechanisms to support your multiprocessing efforts: locks and
mailboxes, and additionally there's mp:without-preemption and
mp:without-interrupts. Both locks and mailboxes have the potential to
do all the synchronization needed, and they can be used in combination
too. The exact behaviour of the mp:without-... operators is not very
well documented (IMHO), so I try not to rely on them -- locks and
mailboxes are well-known from the literature, will do everything you
need, so I believe they're a safe bet. It is possible that
mp:without-... are more efficient under some circumstances - I don't
know anything about that.
The code you present above seems to 'sit on two chairs at the same
time': You seem to be using mp:without-interrupts to gain exclusive
access, and then you PEEK in the mailbox. Generally, I would not
think that is a good way to use a mailbox -- it has built-in exclusive
access, i.e. any Lisp process can safely use it without any other
synchronization being necessary.
To give you an idea of how you might use a mailbox consider the
following definitions:
(defvar *job-count* 0)
(defun each-job-has-own-mailbox (fn arglist)
(let ((mbox (make-mailbox)))
(flet ((retrieve-result ()
(apply #'values (mailbox-read mbox))))
(process-run-function (format nil "JOB-~d" (incf *job-count*))
nil
(lambda (fn1 arglist1)
(mailbox-send mbox (multiple-value-list
(apply fn1 arglist1))))
fn arglist)
#'retrieve-result)))
Here we're assigning a mailbox to each 'job' and offer an interface
where the user provides a function and an arglist, and gets in return
a closure to call to retrieve the result of the 'job'. The code above
offers solutions as to how to know WHEN to call the closure -- it will
be block until the result becomes available.
We might use this code like this
MP-EXAMPLE 4 > (flet ((job-fn (a b)
(- a b)))
(let ((joblist (loop for job in '((1 3) (10 10) (20
10))
collect (list job (each-job-has-
own-mailbox #'job-fn job)))))
(loop for jdesc in joblist
do (format t "~&JOB ~a is doing ~a." (second
jdesc) (first jdesc)))
;; collect results
(loop for jdesc in joblist
collect (list (first jdesc) (funcall (second
jdesc))))))
JOB #<Closure (EACH-JOB-HAS-OWN-MAILBOX . RETRIEVE-RESULT) 200BF4B2>
is doing (1 3).
JOB #<Closure (EACH-JOB-HAS-OWN-MAILBOX . RETRIEVE-RESULT) 200BF15A>
is doing (10 10).
JOB #<Closure (EACH-JOB-HAS-OWN-MAILBOX . RETRIEVE-RESULT) 200C0DBA>
is doing (20 10).
(((1 3) -2) ((10 10) 0) ((20 10) 10))
I am just print the 'jobs' to show the exist independently of the
results. Using this code we use a closure to abstract the interface
to the spawn job. The underlying mechanism used to synchronize is the
mailbox, but we could actually have used an MP:LOCK instead, with the
same result. (In fact the mailbox may been implemented using a lock,
but that's not important at all.)
A different way to accomplish the same thing, still using mailbox, is
to have just a single mailbox to which all results are sent. This
might look something like this
(defun arbiter-has-mailbox (fn joblist)
(let* ((mbox (make-mailbox))
(count 0))
(loop for job in joblist
do (process-run-function (format nil "JOB-~d" (incf *job-
count*))
nil
(lambda (mbox1 fn1 arglist1)
(mailbox-send mbox1 (apply fn1
arglist1)))
mbox fn job)
do (incf count))
(loop repeat count
collect (mailbox-read mbox))))
which might be used like this
MP-EXAMPLE 5 > (arbiter-has-mailbox (lambda (a b) (- a b)) '((1 3) (10
10) (20 10)))
(-2 10 0)
Not very elegant, perhaps, but effective. One could probably come up
with something more flexible. Clearly there's the issue of know how
many times to call MP:MAILBOX-READ on the mailbox, but we'll just
disregard that here.
Please note that this code is just to get you started thinking and
give you an idea of what using mailboxes might look like - it's not
for production.
HTH.
best regards,
-Klaus.
PS Longtime readers of lisphug may know that I personally believe that
Erlang's message-passing infrastructure addresses these issues in a
particularly elegant and powerful way, to the extent that I have
implemented large parts of it in cl-muproc. -K.
Re: Multiprocessing
> Hi Denis --
>
> Your question is not easy to answer -- there are lots of issues here
> and it's hard to figure where to start. My general advice would be
> that you do some reading about the principles of doing multiprocessing
> -- it is a notoriously difficult area and if you do not get it quite
> right you are like to experience subtle errors can be very hard indeed
> to track down. Believe me, such subtle errors will steal your time
> and hurt your productivity so if you really want to work with multiple
> processes, understanding multiprocessing principles will be a
> worthwhile investment!
>
> As far as Lispworks is concerned, it offers two synchronization
> mechanisms to support your multiprocessing efforts: locks and
> mailboxes, and additionally there's mp:without-preemption and
> mp:without-interrupts. Both locks and mailboxes have the potential to
> do all the synchronization needed, and they can be used in combination
> too. The exact behaviour of the mp:without-... operators is not very
> well documented (IMHO), so I try not to rely on them -- locks and
> mailboxes are well-known from the literature, will do everything you
> need, so I believe they're a safe bet. It is possible that
> mp:without-... are more efficient under some circumstances - I don't
> know anything about that.
>
> The code you present above seems to 'sit on two chairs at the same
> time': You seem to be using mp:without-interrupts to gain exclusive
> access, and then you PEEK in the mailbox. Generally, I would not
> think that is a good way to use a mailbox -- it has built-in exclusive
> access, i.e. any Lisp process can safely use it without any other
> synchronization being necessary.
>
> To give you an idea of how you might use a mailbox consider the
> following definitions:
>
> (defvar *job-count* 0)
>
> (defun each-job-has-own-mailbox (fn arglist)
> (let ((mbox (make-mailbox)))
> (flet ((retrieve-result ()
> (apply #'values (mailbox-read mbox))))
> (process-run-function (format nil "JOB-~d" (incf *job-count*))
> nil
> (lambda (fn1 arglist1)
> (mailbox-send mbox (multiple-value-list
> (apply fn1 arglist1))))
> fn arglist)
> #'retrieve-result)))
>
> Here we're assigning a mailbox to each 'job' and offer an interface
> where the user provides a function and an arglist, and gets in return
> a closure to call to retrieve the result of the 'job'. The code above
> offers solutions as to how to know WHEN to call the closure -- it will
> be block until the result becomes available.
>
> We might use this code like this
>
> MP-EXAMPLE 4 > (flet ((job-fn (a b)
> (- a b)))
> (let ((joblist (loop for job in '((1 3) (10 10) (20
> 10))
> collect (list job (each-job-has-
> own-mailbox #'job-fn job)))))
> (loop for jdesc in joblist
> do (format t "~&JOB ~a is doing ~a." (second
> jdesc) (first jdesc)))
> ;; collect results
> (loop for jdesc in joblist
> collect (list (first jdesc) (funcall (second
> jdesc))))))
> JOB #<Closure (EACH-JOB-HAS-OWN-MAILBOX . RETRIEVE-RESULT) 200BF4B2>
> is doing (1 3).
> JOB #<Closure (EACH-JOB-HAS-OWN-MAILBOX . RETRIEVE-RESULT) 200BF15A>
> is doing (10 10).
> JOB #<Closure (EACH-JOB-HAS-OWN-MAILBOX . RETRIEVE-RESULT) 200C0DBA>
> is doing (20 10).
> (((1 3) -2) ((10 10) 0) ((20 10) 10))
>
> I am just print the 'jobs' to show the exist independently of the
> results. Using this code we use a closure to abstract the interface
> to the spawn job. The underlying mechanism used to synchronize is the
> mailbox, but we could actually have used an MP:LOCK instead, with the
> same result. (In fact the mailbox may been implemented using a lock,
> but that's not important at all.)
>
> A different way to accomplish the same thing, still using mailbox, is
> to have just a single mailbox to which all results are sent. This
> might look something like this
>
> (defun arbiter-has-mailbox (fn joblist)
> (let* ((mbox (make-mailbox))
> (count 0))
> (loop for job in joblist
> do (process-run-function (format nil "JOB-~d" (incf *job-
> count*))
> nil
> (lambda (mbox1 fn1 arglist1)
> (mailbox-send mbox1 (apply fn1
> arglist1)))
> mbox fn job)
> do (incf count))
> (loop repeat count
> collect (mailbox-read mbox))))
>
> which might be used like this
>
> MP-EXAMPLE 5 > (arbiter-has-mailbox (lambda (a b) (- a b)) '((1 3) (10
> 10) (20 10)))
> (-2 10 0)
>
> Not very elegant, perhaps, but effective. One could probably come up
> with something more flexible. Clearly there's the issue of know how
> many times to call MP:MAILBOX-READ on the mailbox, but we'll just
> disregard that here.
>
> Please note that this code is just to get you started thinking and
> give you an idea of what using mailboxes might look like - it's not
> for production.
>
> HTH.
>
> best regards,
>
> -Klaus.
>
> PS Longtime readers of lisphug may know that I personally believe that
> Erlang's message-passing infrastructure addresses these issues in a
> particularly elegant and powerful way, to the extent that I have
> implemented large parts of it in cl-muproc. -K.
Hi Klaus,
Thanks a lot for your answer : I will take my time to examine and test it.
My first impression is that you are using mailbox in these examples to deal
with messages (result of tasks) when I use it (maybe incorrectly) to
implement a pile of tasks. My questions was thought for this particular
context, so it's better if I precise a little bit.
Here is how I define my processes at launch level with one mailbox per
process :
(defun make-background-process (process-name &optional (priority 0))
(let* ((wait-string (format nil "Waiting for tasks" process-name))
(return-string (format nil "Return from ~d" process-name))
(process (mp:process-run-function
process-name (list :priority priority)
#'(lambda ()
(with-simple-restart (abort return-string)
(loop do
(with-simple-restart (abort return-string)
(let ((self mp:*current-process*))
(mp:process-wait wait-string
#'mp:mailbox-peek (mp:process-mailbox self))
(let ((task (mp:mailbox-read
(mp:process-mailbox self))))
(apply (car task) (cdr task)))))))))))
(setf (mp:process-mailbox process) (mp:make-mailbox))
process))
And then to send task to processes I use this :
(defun apply-in-background-process (process priority function &rest args)
(push function args)
(mp:process-send process args :change-priority priority))
Best Regards
Denis
-------------------------------------------------------
Denis Pousseur
70 rue de Wansijn
1180 Bruxelles, Belgique
Tel : 32 (0)2 219 31 09
Mail : denis.pousseur@gmail.com
-------------------------------------------------------
Re: Multiprocessing
On 19/02/2008, at 14:12, Denis Pousseur wrote:
> Hi Klaus,
>
> Thanks a lot for your answer : I will take my time to examine and
> test it.
> My first impression is that you are using mailbox in these examples
> to deal
> with messages (result of tasks) when I use it (maybe incorrectly) to
> implement a pile of tasks. My questions was thought for this
> particular
> context, so it's better if I precise a little bit.
>
> Here is how I define my processes at launch level with one mailbox per
> process :
>
> (defun make-background-process (process-name &optional (priority 0))
> (let* ((wait-string (format nil "Waiting for tasks" process-name))
> (return-string (format nil "Return from ~d" process-name))
> (process (mp:process-run-function
> process-name (list :priority priority)
> #'(lambda ()
> (with-simple-restart (abort return-string)
> (loop do
> (with-simple-restart (abort return-
> string)
> (let ((self mp:*current-process*))
> (mp:process-wait wait-string
> #'mp:mailbox-peek (mp:process-mailbox self))
> (let ((task (mp:mailbox-read
> (mp:process-mailbox self))))
> (apply (car task) (cdr
> task)))))))))))
> (setf (mp:process-mailbox process) (mp:make-mailbox))
> process))
>
> And then to send task to processes I use this :
>
> (defun apply-in-background-process (process priority function &rest
> args)
> (push function args)
> (mp:process-send process args :change-priority priority))
>
>
> Best Regards
>
> Denis
>
> -------------------------------------------------------
> Denis Pousseur
> 70 rue de Wansijn
> 1180 Bruxelles, Belgique
>
> Tel : 32 (0)2 219 31 09
> Mail : denis.pousseur@gmail.com
> -------------------------------------------------------
>
>
Hi Denis --
Your code seems to work, and having seen your code, I think I
understand your questions a little better.
Basically, when you write multiprocessing code, you should assume that
all the processes run concurrently and that your code is responsible
for making them behave correctly, considered all together. In
principle, code executing in one thread may be stopped at any time and
another thread be resumed, except when you change that, by explicitly
adding synchronization operations to your code. So the answer to your
question
> 2)
> If a task is always in the mailbox queue of the process, does it
> means that
> it is non executed (and so that it can be aborted) ? Does the
> process do :
>
> pop the queue
> execute the task
>
> OR
>
> execute the task (always in the queue)
> pop the queue
>
> ?
is "It depends!" ... on how you synchronize your threads, using locks,
mailboxs and critical sections (that's what mp:without-... give you).
There is no magic, just synchronization code to make sure everything
happens just right.
In current versions of Lispworks, only one Lisp thread/process
actually execute at a time, even if you have a multi-core og multi-
processor machine.
A few comments to the code just posted (your code, for reference)
(defun make-background-process (process-name &optional (priority 0))
(let* ((wait-string (format nil "Waiting for tasks" process-name))
(return-string (format nil "Return from ~d" process-name))
(process (mp:process-run-function
process-name (list :priority priority)
#'(lambda ()
(with-simple-restart (abort return-string)
(loop do
(with-simple-restart (abort return-
string)
(let ((self mp:*current-process*))
(mp:process-wait wait-string
#'mp:mailbox-
peek (mp:process-mailbox self))
(let ((task (mp:mailbox-read
(mp:process-
mailbox self))))
(apply (car task) (cdr
task)))))))))))
(setf (mp:process-mailbox process) (mp:make-mailbox))
process))
(defun apply-in-background-process (process priority function
&rest args)
(push function args)
(mp:process-send process args :change-priority priority))
Although using PROCESS-WAIT permits you to set the thread status, I
find using both PROCESS-WAIT in combination with MAILBOX-PEEK _and_
MAILBOX-READ to be overly complicated - I'd certainly prefer to just
do a straight MAILBOX-READ. Also, I'd probably prefer to close over
mailbox (instead using PROCESS-MAILBOX) and return a closure to post
things to it, as in
(defun make-background-process (process-name &optional (priority 0))
(let ((mailbox (mp:make-mailbox)))
(flet ((worker (mbox name)
(with-simple-restart (abort (format nil "Abort job
known as ~a." name))
(loop with proceed = (format nil "Allow ~a to
proceed to next job." name)
for task = (mailbox-read mbox)
do (with-simple-restart (proceed proceed)
(apply (car task) (cdr task))))))
(submit-job (function &rest args)
(mp:mailbox-sed mailbox (push function args)
'ok)))
(mp:process-run-function process-name (list :priority
priority)
#'worker mailbox process-name)
#'submit-job)))
Using FLET instead of LAMBDA makes the code easier to read and also
helps because the printer will actually tell you the lexical context
of the closure (effectively telling you what kind it is). The above
code yields this
MP-EXAMPLE 1 > (defparameter *mylist* nil)
*MYLIST*
MP-EXAMPLE 2 > (let ((submit (make-background-process "WORKER-1")))
(funcall submit (lambda (x) (push x *mylist*)) 42))
OK
MP-EXAMPLE 3 > *mylist*
(42)
MP-EXAMPLE 4 >
Much of this is a matter of taste, of course. The above code does not
set the process status while waiting for a task, so that's a weakness
compared to yours.
HTH.
best regards,
-Klaus.
Re: Multiprocessing
> Hi Denis --
>
> Your code seems to work, and having seen your code, I think I
> understand your questions a little better.
>
> Basically, when you write multiprocessing code, you should assume that
> all the processes run concurrently and that your code is responsible
> for making them behave correctly, considered all together. In
> principle, code executing in one thread may be stopped at any time and
> another thread be resumed, except when you change that, by explicitly
> adding synchronization operations to your code. So the answer to your
> question
>
>> 2)
>> If a task is always in the mailbox queue of the process, does it
>> means that
>> it is non executed (and so that it can be aborted) ? Does the
>> process do :
>>
>> pop the queue
>> execute the task
>>
>> OR
>>
>> execute the task (always in the queue)
>> pop the queue
>>
>> ?
>
> is "It depends!" ... on how you synchronize your threads, using locks,
> mailboxs and critical sections (that's what mp:without-... give you).
> There is no magic, just synchronization code to make sure everything
> happens just right.
>
> In current versions of Lispworks, only one Lisp thread/process
> actually execute at a time, even if you have a multi-core og multi-
> processor machine.
>
> A few comments to the code just posted (your code, for reference)
>
> (defun make-background-process (process-name &optional (priority 0))
> (let* ((wait-string (format nil "Waiting for tasks" process-name))
> (return-string (format nil "Return from ~d" process-name))
> (process (mp:process-run-function
> process-name (list :priority priority)
> #'(lambda ()
> (with-simple-restart (abort return-string)
> (loop do
> (with-simple-restart (abort return-
> string)
> (let ((self mp:*current-process*))
> (mp:process-wait wait-string
> #'mp:mailbox-
> peek (mp:process-mailbox self))
> (let ((task (mp:mailbox-read
> (mp:process-
> mailbox self))))
> (apply (car task) (cdr
> task)))))))))))
> (setf (mp:process-mailbox process) (mp:make-mailbox))
> process))
>
> (defun apply-in-background-process (process priority function
> &rest args)
> (push function args)
> (mp:process-send process args :change-priority priority))
>
> Although using PROCESS-WAIT permits you to set the thread status, I
> find using both PROCESS-WAIT in combination with MAILBOX-PEEK _and_
> MAILBOX-READ to be overly complicated - I'd certainly prefer to just
> do a straight MAILBOX-READ. Also, I'd probably prefer to close over
> mailbox (instead using PROCESS-MAILBOX) and return a closure to post
> things to it, as in
>
> (defun make-background-process (process-name &optional (priority 0))
> (let ((mailbox (mp:make-mailbox)))
> (flet ((worker (mbox name)
> (with-simple-restart (abort (format nil "Abort job
> known as ~a." name))
> (loop with proceed = (format nil "Allow ~a to
> proceed to next job." name)
> for task = (mailbox-read mbox)
> do (with-simple-restart (proceed proceed)
> (apply (car task) (cdr task))))))
> (submit-job (function &rest args)
> (mp:mailbox-sed mailbox (push function args)
> 'ok)))
> (mp:process-run-function process-name (list :priority
> priority)
> #'worker mailbox process-name)
> #'submit-job)))
Hi Klaus,
About the process status, I am a little bit afraid : does not this loop,
with any wait-process call, completely overflow the system ?
For the other modifications you propose, I agree : I made this mix between
my codes and your proposals and, effectively, the codes are clearer (and
avoid the call to mp:*current-process* witch was not very convincing).
Thanks for your help !
(defun make-background-process (process-name &optional (priority 0))
(let ((mailbox (mp:make-mailbox)) process)
(flet ((worker (mbox name)
(with-simple-restart (abort (format nil "Abort job known as
~a." name))
(loop do (with-simple-restart (abort (format nil "Allow ~a to
proceed to next job." name))
(mp:process-wait (format nil "Waiting for tasks"
name) #'mp:mailbox-peek mbox)
(let ((task (mp:mailbox-read mbox)))
(apply (car task) (cdr task))))))))
(setf process (mp:process-run-function process-name (list :priority
priority) #'worker mailbox process-name)
(mp:process-mailbox process) mailbox)
process)))
Best Regards
Denis
>
> Using FLET instead of LAMBDA makes the code easier to read and also
> helps because the printer will actually tell you the lexical context
> of the closure (effectively telling you what kind it is). The above
> code yields this
>
> MP-EXAMPLE 1 > (defparameter *mylist* nil)
> *MYLIST*
>
> MP-EXAMPLE 2 > (let ((submit (make-background-process "WORKER-1")))
> (funcall submit (lambda (x) (push x *mylist*)) 42))
> OK
>
> MP-EXAMPLE 3 > *mylist*
> (42)
>
> MP-EXAMPLE 4 >
>
> Much of this is a matter of taste, of course. The above code does not
> set the process status while waiting for a task, so that's a weakness
> compared to yours.
>
> HTH.
>
> best regards,
>
> -Klaus.
-------------------------------------------------------
Denis Pousseur
70 rue de Wansijn
1180 Bruxelles, Belgique
Tel : 32 (0)2 219 31 09
Mail : denis.pousseur@gmail.com
-------------------------------------------------------
Re: Multiprocessing
> About the process status, I am a little bit afraid : does not this
> loop,
> with any wait-process call, completely overflow the system ?
Ah! - we're getting somewhere (this explains, at least to me, your use
of MP:PROCESS-WAIT): No, MP:MAILBOX-READ blocks until a message can be
read. Check the reference manual for MAILBOX-READ - http://www.lispworks.com/documentation/lw50/LWRM/html/lwref-382.htm#pgfId-938979
, second paragraph in Description. (Side effect: I learned that you
can specify a wait-reason in calls to MAILBOX-READ - forgot about that.)
best regards,
-Klaus.
Re: Multiprocessing
Le 20/02/08 10:27, « [NOM] » <[ADRESSE]> a écrit :
>> About the process status, I am a little bit afraid : does not this
>> loop,
>> with any wait-process call, completely overflow the system ?
>
> Ah! - we're getting somewhere (this explains, at least to me, your use
> of MP:PROCESS-WAIT): No, MP:MAILBOX-READ blocks until a message can be
> read. Check the reference manual for MAILBOX-READ -
> http://www.lispworks.com/documentation/lw50/LWRM/html/lwref-382.htm#pgfId-9389
> 79
> , second paragraph in Description. (Side effect: I learned that you
> can specify a wait-reason in calls to MAILBOX-READ - forgot about that.)
>
> best regards,
>
> -Klaus.
Ah well ! I understand my mistake. So this is better (removing the useless
process-wait and mailbox-peek calls) :
(defun make-background-process (process-name &optional (priority 0))
(let ((mailbox (mp:make-mailbox)) process)
(flet ((worker (mbox name)
(with-simple-restart (abort (format nil "Abort job known as
~a." name))
(loop for task = (mp:mailbox-read mbox)
do (with-simple-restart (abort (format nil "Allow ~a to
proceed to next job." name))
(apply (car task) (cdr task)))))))
(setf process (mp:process-run-function process-name (list :priority
priority) #'worker mailbox process-name)
(mp:process-mailbox process) mailbox)
process)))
Thanks !
Denis
-------------------------------------------------------
Denis Pousseur
70 rue de Wansijn
1180 Bruxelles, Belgique
Tel : 32 (0)2 219 31 09
Mail : denis.pousseur@gmail.com
-------------------------------------------------------