Saving and loading big objects (or: limits for FASL files)

Hi,

I'm dumping some pretty big objects to FASL files by writing some
MAKE-LOAD-FORM methods and calling HCL:DUMP-FORMS-TO-FILE.

This works pretty well, but when the objects get very big (and the
resulting FASL files are somewhere between 5 and 10 MB), I get the
following error when I try to LOAD the FASL file:

  Error: Array size 2216391 is not an integer in the legal range 0 to 2096895.
    1 (continue) Try loading d:\frank\lisp\fflip\simulation-11200.fsl again.
    2 Give up loading d:\frank\lisp\fflip\simulation-11200.fsl.
    3 Try loading another file instead of d:\frank\lisp\fflip\simulation-11200.fsl.
    4 (abort) Return to level 0.
    5 Return to top loop level 0.

I need to save and load objects that are about 5 times bigger than the size
at which this error occurs, so I'm looking for solutions around this problem.

Some options I'm thinking about are:

 - Fine-tuning my MAKE-LOAD-FORM methods.
   I suppose I could optimize the forms that are generated by my MAKE-LOAD-FORM
   methods. At the moment, I mostly just call MAKE-LOAD-FORM-SAVING-SLOTS, which
   generates safe but rather long load-forms.
   With some work, I expect I may be able to gain a factor 2, but I doubt that
   I can get the factor 5 that I need.

 - Don't use HCL:DUMP-FORMS-TO-FILE, but just use WRITE to write the load-forms
   to a file. I suppose this will mean that loading the file will take a lot
   longer. And I'm not sure that this will actually solve the limitation problem.

 - Define my own file format and write my own reader/loader.
   That's always possible as a last resort, but I hope that won't be necessary.

All suggestions appreciated.

Thanks in advance,

Arthur Lemmens

Re: Saving and loading big objects (or: limits for FASL files)

Unable to parse email body. Email id is 1880

Re: Saving and loading big objects (or: limits for FASL files)

On Thu, 12 Feb 2004 12:43:32 GMT, Nick Levine <ndl@ravenbrook.com> wrote:

> In my case, becuase the structures I was dumping had very few
> pointers, it was very easy to write by own reader/loader - less than
> a day I think. It now runs an order of magnitude faster, and the
> files thus generated are not only much smaller but also will work
> for future versions of LW plus other platforms, which may turn out
> to be handy.

Do you mean a reader/loader that reads/loads a binary format specific
to your structures? Would you mind to sketch in a few sentences how it
worked and how you managed to make it much faster than
FASL-reading/loading?

Thanks in advance,
Edi.

Re: Saving and loading big objects (or: limits for FASL files)

Unable to parse email body. Email id is 1884