Portable Allegroserve and LispWorks 6

Has anyone else here tried to compile Portable Allegroserve in LispWorks 6?  I have been using it with great success in 5.1, but I find that I can no longer get it to compile.  The compilation errors occur with acl-compat..mp:process-revoke-run-reason and acl-compat.mp:process-add-run-reason.  Here is the report that I get from the compiler:
 
**++++ Error in ACL-COMPAT.MP:PROCESS-REVOKE-RUN-REASON: 
  WITHOUT-PREEMPTION is used
 **++++ Error in ACL-COMPAT.MP::INVOKE-WITH-TIMEOUT: 
  WITHOUT-INTERRUPTS is used
 
At this point, I'm not quite sure how to fix these problems.  I can find their reference in the source code file acl-mp.lisp:
 
(defun process-revoke-run-reason (process object)
  (mp:without-preemption
   (setf (mp:process-run-reasons process)
         (remove object (mp:process-run-reasons process))))
  (when (and (eq process mp:*current-process*)
             (not mp:*inhibit-scheduling-flag*))
    (mp:process-allow-scheduling)))
 
(defun invoke-with-timeout (timeout bodyfn timeoutfn)
  (block timeout
    (let* ((process mp:*current-process*)
           (unsheduled? nil)
           (timer (mp:make-timer
                   #'(lambda ()
                       (mp:process-interrupt process
                                             #'(lambda ()
                                                 (unless unsheduled?
                                                   (return-from timeout
                                                     (funcall timeoutfn)))))))))
      (mp:schedule-timer-relative timer timeout)
      (unwind-protect (funcall bodyfn)
        (without-interrupts
         (mp:unschedule-timer timer)
         (setf unsheduled? t))))))

Does anyone have a suggestion for a quick fix?  Thanks in advance.

Re: Portable Allegroserve and LispWorks 6

On Jan 19, 2010, at 7:41 PM, Richard Wojcik wrote:

> Has anyone else here tried to compile Portable Allegroserve in LispWorks 6?  I have been using it with great success in 5.1, but I find that I can no longer get it to compile.  The compilation errors occur with acl-compat.mp:process-revoke-run-reason and acl-compat.mp:process-add-run-reason.  Here is the report that I get from the compiler:
>  
> **++++ Error in ACL-COMPAT.MP:PROCESS-REVOKE-RUN-REASON: 
>   WITHOUT-PREEMPTION is used
>  **++++ Error in ACL-COMPAT.MP::INVOKE-WITH-TIMEOUT: 
>   WITHOUT-INTERRUPTS is used

<http://www.lispworks.com/documentation/lw60/LW/html/lw-238.htm#marker-893476>

Read this page. The short version is that under the new MP APIs something like mp:without-preemption or mp:without-interrupts has ambiguous semantics under the new APIs, so you need to rewrite this to do something else. IOW, LW 6 can't know whether the author of PAS meant to do something atomic by using without-interrupts, or meant to ensure that an operation completed without being aborted. So you must  either:

1. create a mp:lock and use mp:with-lock everywhere the code used to use mp:without-preemption or mp:without interrupts, or use the new atomic operations; this is for operations that need to be atomic with respect to a certain resource, for example reading or writing a composite object ensuring that no other thread modifies this composite object during the read, or tries to read its inconsistent state during a write.

2. block interrupts; this is for operations that need to complete in their entirety without being interrupted by the scheduler (i.e., not abort before the operation completes).

As a completely ham fisted first approximation you could define two new macros mp:without-preemption and mp:without-interrupts, and a single global mp:lock, and have both of these macros do the same thing, namely, inside the body of an mp:with-interrupts-blocked, grab the global lock and then execute the body. Sure to be sub-optimal, but it may get you part way to a working PAS. 

(in-package :mp)

(eval-when (:compile-toplevel :load-toplevel :execute)
  (defparameter *global-without-interrupts-hack-lock* (mp:make-lock)))

(defmacro without-interrupts (&rest body)
  `(mp:with-interrupts-blocked
    (mp:with-lock (*global-without-interrupts-hack-lock*)
      ,@body)))

(defmacro without-preemption (&rest body)
  `(mp:with-interrupts-blocked
    (mp:with-lock (*global-without-interrupts-hack-lock*)
      ,@body)))

FWIW this is not your only problem - once you do the above, PAS compiles, but running the test, you get complaint wrt the use of :arrest-reasons, which is deprecated in LW 6.0:
<http://www.lispworks.com/documentation/lw60/LW/html/lw-1012.htm#marker-888515>

warmest regards,

Ralph

Raffael Cavallaro
raffaelcavallaro@me.com

Re: Portable Allegroserve and LispWorks 6

Dear All,

I am just getting our stuff caught up on LW 6 and we also have a big
dependency on portableallegroserve.

It doesn't look like any of the code discussed in this thread has made
it into the CVS on sourceforge? If the code is available I am
certainly available to do some testing and hopefully we can get the
project current again.

It has probably diverged quite a bit from the Franz version as well,
since Franz are starting to do SMP and they will have to rewrite the
without-interrupts etc. as well.

Regards,

 -dave


On Thu, Feb 11, 2010 at 12:24 PM, Martin Simmons <martin@lispworks.com> wrote:
>
>>>>>> On Thu, 11 Feb 2010 16:51:49 +0100, Jochen Schmidt said:
>>
>> Am 11.02.2010 um 16:29 schrieb Martin Simmons:
>> >
>> > Thanks for the list.
>> >
>> > Many of those things are existing bugs though (e.g. unwind-protects that were
>> > never really safe w.r.t. with-timeout), so we might not address them at the
>> > moment.  Fixing them also requires non-portable changes in the main code or
>> > extensions to the acl-compat package for all of the supported platforms, but
>> > it isn't clear which is the best approach.
>>
>> I think the WITH-TIMEOUTS are not really a problem anymore if read-timeouts are used.
>> There are #+allegro conditionals sprinkled over the code were those timeouts
>> are used. LW read-timeouts should work out for those.
>
> Maybe, but with-http-response uses it, so it is currently wrapped around many
> response functions as well as the socket code.
>
>
>
>>                                                       Not all of the
>> acl-compat code is actually used by paserve.
>
> True, but unfortunately it has a life of its own too
> (http://www.cliki.net/ACL-COMPAT) so there might be other users.
>
>
>>                                              I strongly suggest implementing
>> the request dispatching and worker thread handling using LW native tools and
>> not like it is done now. I used a mailbox per worker. Requests just get
>> stuffed into the mailbox of the next free worker.
>
> Agreed.
>
>
>>                                                   The unwind-protects I
>> meant were not related to with-timeout - some of them maybe even weren't
>> there before ;-).
>>
>> I don't think that building it on ACL-COMPAT is the best approach here. When
>> I did the original port the MP facilities were a lot closer than they are
>> today. I came to the conclusion, that one needs much less code and the
>> result will work better when choosing to implement at least the dispatching
>> stuff LW native.
>
> OK.
>
>
>> Martin:
>> I think the only remaining thing in proxy.cl was some uses of
>> WITHOUT-SCHEDULING which you could just patch using a big lock for now. Did
>> you try my code?
>
> Yes, thanks for that.  It looks like it is similar to what we would like to
> see.
>
> --
> Martin Simmons
> LispWorks Ltd
> http://www.lispworks.com/
>
>



-- 
Dave Cooper, Genworks Support
dave@genworks.com, dave.genworks.com(skype)
USA: 248-327-3253(o), 1-248-330-2979(mobile)
UK: 0191 645 1699

Re: Portable Allegroserve and LispWorks 6

Hi Dave,

Sorry for not following up on this until now.

I have some fixes for Portable Allegroserve CVS HEAD to make it work with
LispWorks 6 that I can send you for testing if you like.

They have been tested to some extent, but AFAIK there hasn't been any stress
testing.  If you have some good tests to run then that would be useful.

What format do you want (e.g. diff -u, complete files)?

-- 
Martin Simmons
LispWorks Ltd
http://www.lispworks.com/



>>>>> On Thu, 9 Sep 2010 23:57:16 -0400, Dave Cooper said:
> 
> Dear All,
> 
> I am just getting our stuff caught up on LW 6 and we also have a big
> dependency on portableallegroserve.
> 
> It doesn't look like any of the code discussed in this thread has made
> it into the CVS on sourceforge? If the code is available I am
> certainly available to do some testing and hopefully we can get the
> project current again.
> 
> It has probably diverged quite a bit from the Franz version as well,
> since Franz are starting to do SMP and they will have to rewrite the
> without-interrupts etc. as well.
> 
> Regards,
> 
>  -dave
> 
> 
> On Thu, Feb 11, 2010 at 12:24 PM, Martin Simmons <martin@lispworks.com> wrote:
> >
> >>>>>> On Thu, 11 Feb 2010 16:51:49 +0100, Jochen Schmidt said:
> >>
> >> Am 11.02.2010 um 16:29 schrieb Martin Simmons:
> >> >
> >> > Thanks for the list.
> >> >
> >> > Many of those things are existing bugs though (e.g. unwind-protects that were
> >> > never really safe w.r.t. with-timeout), so we might not address them at the
> >> > moment.  Fixing them also requires non-portable changes in the main code or
> >> > extensions to the acl-compat package for all of the supported platforms, but
> >> > it isn't clear which is the best approach.
> >>
> >> I think the WITH-TIMEOUTS are not really a problem anymore if read-timeouts are used.
> >> There are #+allegro conditionals sprinkled over the code were those timeouts
> >> are used. LW read-timeouts should work out for those.
> >
> > Maybe, but with-http-response uses it, so it is currently wrapped around many
> > response functions as well as the socket code.
> >
> >
> >
> >>                                                       Not all of the
> >> acl-compat code is actually used by paserve.
> >
> > True, but unfortunately it has a life of its own too
> > (http://www.cliki.net/ACL-COMPAT) so there might be other users.
> >
> >
> >>                                              I strongly suggest implementing
> >> the request dispatching and worker thread handling using LW native tools and
> >> not like it is done now. I used a mailbox per worker. Requests just get
> >> stuffed into the mailbox of the next free worker.
> >
> > Agreed.
> >
> >
> >>                                                   The unwind-protects I
> >> meant were not related to with-timeout - some of them maybe even weren't
> >> there before ;-).
> >>
> >> I don't think that building it on ACL-COMPAT is the best approach here. When
> >> I did the original port the MP facilities were a lot closer than they are
> >> today. I came to the conclusion, that one needs much less code and the
> >> result will work better when choosing to implement at least the dispatching
> >> stuff LW native.
> >
> > OK.
> >
> >
> >> Martin:
> >> I think the only remaining thing in proxy.cl was some uses of
> >> WITHOUT-SCHEDULING which you could just patch using a big lock for now. Did
> >> you try my code?
> >
> > Yes, thanks for that.  It looks like it is similar to what we would like to
> > see.
> >
> > --
> > Martin Simmons
> > LispWorks Ltd
> > http://www.lispworks.com/
> >
> >
> 
> 
> 
> -- 
> Dave Cooper, Genworks Support
> dave@genworks.com, dave.genworks.com(skype)
> USA: 248-327-3253(o), 1-248-330-2979(mobile)
> UK: 0191 645 1699
>

Re: Portable Allegroserve and LispWorks 6

Hi Martin,

I am happy to start testing any time but I would need the full sources
to start. I will contact you and Kevin R if making any mods, until
this gets back "officially" into sourceforge or similar.

Have a good weekend!

 -dave


On Fri, Sep 24, 2010 at 8:33 AM, Martin Simmons <martin@lispworks.com> wrote:
> Hi Dave,
>
> Sorry for not following up on this until now.
>
> I have some fixes for Portable Allegroserve CVS HEAD to make it work with
> LispWorks 6 that I can send you for testing if you like.
>
> They have been tested to some extent, but AFAIK there hasn't been any stress
> testing.  If you have some good tests to run then that would be useful.
>
> What format do you want (e.g. diff -u, complete files)?
>
> --
> Martin Simmons
> LispWorks Ltd
> http://www.lispworks.com/
>
>
>
>>>>>> On Thu, 9 Sep 2010 23:57:16 -0400, Dave Cooper said:
>>
>> Dear All,
>>
>> I am just getting our stuff caught up on LW 6 and we also have a big
>> dependency on portableallegroserve.
>>
>> It doesn't look like any of the code discussed in this thread has made
>> it into the CVS on sourceforge? If the code is available I am
>> certainly available to do some testing and hopefully we can get the
>> project current again.
>>
>> It has probably diverged quite a bit from the Franz version as well,
>> since Franz are starting to do SMP and they will have to rewrite the
>> without-interrupts etc. as well.
>>
>> Regards,
>>
>>  -dave
>>
>>
>> On Thu, Feb 11, 2010 at 12:24 PM, Martin Simmons <martin@lispworks.com> wrote:
>> >
>> >>>>>> On Thu, 11 Feb 2010 16:51:49 +0100, Jochen Schmidt said:
>> >>
>> >> Am 11.02.2010 um 16:29 schrieb Martin Simmons:
>> >> >
>> >> > Thanks for the list.
>> >> >
>> >> > Many of those things are existing bugs though (e.g. unwind-protects that were
>> >> > never really safe w.r.t. with-timeout), so we might not address them at the
>> >> > moment.  Fixing them also requires non-portable changes in the main code or
>> >> > extensions to the acl-compat package for all of the supported platforms, but
>> >> > it isn't clear which is the best approach.
>> >>
>> >> I think the WITH-TIMEOUTS are not really a problem anymore if read-timeouts are used.
>> >> There are #+allegro conditionals sprinkled over the code were those timeouts
>> >> are used. LW read-timeouts should work out for those.
>> >
>> > Maybe, but with-http-response uses it, so it is currently wrapped around many
>> > response functions as well as the socket code.
>> >
>> >
>> >
>> >>                                                       Not all of the
>> >> acl-compat code is actually used by paserve.
>> >
>> > True, but unfortunately it has a life of its own too
>> > (http://www.cliki.net/ACL-COMPAT) so there might be other users.
>> >
>> >
>> >>                                              I strongly suggest implementing
>> >> the request dispatching and worker thread handling using LW native tools and
>> >> not like it is done now. I used a mailbox per worker. Requests just get
>> >> stuffed into the mailbox of the next free worker.
>> >
>> > Agreed.
>> >
>> >
>> >>                                                   The unwind-protects I
>> >> meant were not related to with-timeout - some of them maybe even weren't
>> >> there before ;-).
>> >>
>> >> I don't think that building it on ACL-COMPAT is the best approach here. When
>> >> I did the original port the MP facilities were a lot closer than they are
>> >> today. I came to the conclusion, that one needs much less code and the
>> >> result will work better when choosing to implement at least the dispatching
>> >> stuff LW native.
>> >
>> > OK.
>> >
>> >
>> >> Martin:
>> >> I think the only remaining thing in proxy.cl was some uses of
>> >> WITHOUT-SCHEDULING which you could just patch using a big lock for now. Did
>> >> you try my code?
>> >
>> > Yes, thanks for that.  It looks like it is similar to what we would like to
>> > see.
>> >
>> > --
>> > Martin Simmons
>> > LispWorks Ltd
>> > http://www.lispworks.com/
>> >
>> >
>>
>>
>>
>> --
>> Dave Cooper, Genworks Support
>> dave@genworks.com, dave.genworks.com(skype)
>> USA: 248-327-3253(o), 1-248-330-2979(mobile)
>> UK: 0191 645 1699
>>
>



-- 
Dave Cooper, Genworks Support
dave@genworks.com, dave.genworks.com(skype)
USA: 248-327-3253(o), 1-248-330-2979(mobile)
UK: 0191 645 1699