using multiple processors

Is there a means of using multiple processors for loading a lisp program (32-bit LW professional edition)?

On a Macintosh with 4 processors, the activity monitor indicates ~100% use of a single processor.

Since the load takes 3 days, I was hoping that using more than 1 processor might reduce this time.

Thanks,

Sheldon

Re: using multiple processors

Sheldon Ball <sheldon.ball@gmail.com> writes:

> Is there a means of using multiple processors for loading a lisp
> program (32-bit LW professional edition)?  On a Macintosh with 4
> processors, the activity monitor indicates ~100% use of a single
> processor.  Since the load takes 3 days, I was hoping that using more
> than 1 processor might reduce this time.

Theorically, it would be possible to create threads and load different
files at the same time.

In practice, AFAIK, MK:DEFSYSTEM and ASDF don't parallelize loading.  
I don't know about XCVB either, but perhaps it could be more easily be
made to load files in parallel, from its design.

Furthermore, most implementations have locks to prevent problems when
defining things in parallel (which should not be too often a problem,
assuming you don't load the same file twice at the same time, but
sometimes those locks are quite coarse (eg. sbcl).

I don't know if lispworks lock is coarse or not.  If it is not, then you
could write a load script to load several files at once.  The question
then is for you to know the dependencies between your files, and whether
you have groups of files that can be loaded at the same time (not
depending one on the other).

-- 
__Pascal Bourguignon__                     http://www.informatimago.com/
A bad day in () is better than a good day in {}.

_______________________________________________
Lisp Hug - the mailing list for LispWorks users
lisp-hug@lispworks.com
http://www.lispworks.com/support/lisp-hug.html

Re: using multiple processors

On 10 Sep 2012, at 22:36, Sheldon Ball wrote:

> Is there a means of using multiple processors for loading a lisp program (32-bit LW professional edition)?
> On a Macintosh with 4 processors, the activity monitor indicates ~100% use of a single processor.
> Since the load takes 3 days, I was hoping that using more than 1 processor might reduce this time.

If it takes 3 days then there must be some pretty extensive load-time computation going on (it can't just be a bunch of straightforward of definitions if it takes that long).  So I think the thing to do would be to understand where that is happening, and what its dependencies look like, and then to modify the thing to push that out into multiple processes, which can then run in parallel, using some kind of queue which will run up to n threads at once to keep the machine busy.

_______________________________________________
Lisp Hug - the mailing list for LispWorks users
lisp-hug@lispworks.com
http://www.lispworks.com/support/lisp-hug.html

Re: using multiple processors

Thanks to everyone for the replies.

The algorithm finds hyperlinks for a web application through a string matching function against a set of about 80,000 terms with a variable number of string permutations. Loading the same program with the hyperlinks preset takes only about 2 hours (rather than 3 days).

The program itself builds upon what it already knows as it gets larger, thus it seems, that criteria for parallel processing probably cannot be met.

Since I run the load over a weekend, a 2 day load time would be acceptable.

It appears my best options are:

1) get a faster computer

2) get a 64 bit compiler.

Thanks again,

Sheldon

On Tue, Sep 11, 2012 at 6:24 AM, Dan Corkill <dancorkill@comcast.net> wrote:

On Sep 11, 2012, at 2:25 AM, Tim Bradshaw <tfb@cley.com> wrote:

>

> On 10 Sep 2012, at 22:36, Sheldon Ball wrote:
>
>> Is there a means of using multiple processors for loading a lisp program (32-bit LW professional edition)?
>> On a Macintosh with 4 processors, the activity monitor indicates ~100% use of a single processor.
>> Since the load takes 3 days, I was hoping that using more than 1 processor might reduce this time.
>
> If it takes 3 days then there must be some pretty extensive load-time computation going on (it can't just be a bunch of straightforward of definitions if it takes that long). So I think the thing to do would be to understand where that is happening, and what its dependencies look like, and then to modify the thing to push that out into multiple processes, which can then run in parallel, using some kind of queue which will run up to n threads at once to keep the machine busy.

Extensive allocation & GCing could also be going on, so that possibility should be checked as well (and threads won't help that issue). Extensive incremental growth of hash-tables (rather than creating them initially with their eventual size or at least a large growth increment) can be particularly costly.

_______________________________________________
Lisp Hug - the mailing list for LispWorks users
lisp-hug@lispworks.com
http://www.lispworks.com/support/lisp-hug.html