Creating large arrays

Hello,

I am trying to create a moderately large (4GB) array on LispWorks 64-bit on a barebone 16Gb Windows 10 PC, only LispWorks running, beside Windows services (HobbyistDV license):


(progn (make-array (* 4 1024 1024 1024)
             :allocation :long-lived
              :element-type '(unsigned-byte 8)
              :fill-pointer t)
  (values))


(Note: thanks to lispm on reddit for tip on using (values) to avoid huge output at REPL)


Even when compiling the above expression, I get this error message:

> Bad total size 4294967296 for array, should be in the inclusive
> range 0 to 536870910.


On same PC, the following works fine on ccl:

(progn
  (make-array (* 4 1024 1024 1024)
              :element-type '(unsigned-byte 8)
              :fill-pointer t) (values))


The only thing I could notice is that 536870910 is close to (expt 2 29) -> 536870912

Shouldn't The heap grow automatically by requesting more memory from the operating system?

In any case, based on this page,
http://www.lispworks.com/documentation/lw71/LW/html/lw-212.htm#37447

I tried using --relocate-image and --dynamic-space-size with same issue.


I was unable to find any other information that could help.

What am I doing wrong?

How can one use large arrays in LispWorks 64-bit, potentially more than 10 GB?



Thanks for any hint or direction,

Priyadarshan

_______________________________________________
Lisp Hug - the mailing list for LispWorks users
lisp-hug@lispworks.com
http://www.lispworks.com/support/lisp-hug.html

Re: Creating large arrays

See http://www.lispworks.com/documentation/lw71/LW/html/lw-233.htm#pgfId-891709.

It seems that 64-bit Lispworks does not support arrays of more than 2^29-1 elements, which seems a bit low for a 64-bit implementation.

A possible workaround is to use multiple arrays; another might be to use the FLI to allocate a large chunk of memory outside the Lisp heap. Neither of these seem like a great solution, though.

On Thu, Jul 18, 2019 at 1:01 PM Lisp Student <lisp@gmx.com> wrote:

Hello,

I am trying to create a moderately large (4GB) array on LispWorks 64-bit on a barebone 16Gb Windows 10 PC, only LispWorks running, beside Windows services (HobbyistDV license):

(progn (make-array (* 4 1024 1024 1024)
:allocation :long-lived
:element-type '(unsigned-byte 8)
:fill-pointer t)
(values))

(Note: thanks to lispm on reddit for tip on using (values) to avoid huge output at REPL)

Even when compiling the above expression, I get this error message:

> Bad total size 4294967296 for array, should be in the inclusive
> range 0 to 536870910.

On same PC, the following works fine on ccl:

(progn
(make-array (* 4 1024 1024 1024)
:element-type '(unsigned-byte 8)
:fill-pointer t) (values))

The only thing I could notice is that 536870910 is close to (expt 2 29) -> 536870912

Shouldn't The heap grow automatically by requesting more memory from the operating system?

In any case, based on this page,
http://www.lispworks.com/documentation/lw71/LW/html/lw-212.htm#37447

I tried using --relocate-image and --dynamic-space-size with same issue.

I was unable to find any other information that could help.

What am I doing wrong?

How can one use large arrays in LispWorks 64-bit, potentially more than 10 GB?

Thanks for any hint or direction,

Priyadarshan

_______________________________________________
Lisp Hug - the mailing list for LispWorks users
lisp-hug@lispworks.com
http://www.lispworks.com/support/lisp-hug.html

Re: Creating large arrays

Are there other fast-access data structures with no such limitations, on
LispWorks?

Thank you again,

Priyadarshan

I have created something called ’N-Dimensional Sparse Spaces’ as a solution to the large array allocation problem.

This initial implementation is one to replace a set of random points within an N-Dimensional Sparse Grid addressed using integers, hence the use of #’= in #’make-space.

It is more generallized using other equals functions such as #’eql (so you can use symbols, and hence make an addressable symbol space), #’equals and #equalp letting you use S-Expressions as your addresses.

I have a more general implementation somewhere in my code, but the above image is the code I initially wrote that can be springboarded off of to create your own implementation.

One trick I have thought of is to, after you have completed your work with the space, is to compile it to a more suituable data structure for read-only access.

The sparce space is for editing and loading of the data you want to work with. A complied space is for reading data from the space, giving up generality.

Feel free to write me to discuss this data structure further if you have any questions or comments on its usefulness and generality and applicability to your problem.

Burton Samograd

BusFactor1 Inc.

Re: Creating large arrays

Unable to parse email body. Email id is 14963