Re: Memory allocation fragmentation?

Unable to parse email body. Email id is 7237

Re: Memory allocation fragmentation?

Hi Osei,

have you tried doing the memory management yourself w/regard to the arrays?
Where you would pre-allocate arrays, in a stack, that were plenty large
enough to contain the data to be read in.  Then, as you finished with a
given array, you'd return it to a stack for re-use.  I used this scheme to
cut down on garbage generation and collection and improved performance by a
couple orders of magnitude.

Raymond Laning



                                                                           
                                                                           
             "Osei Poku"                                                   
             <osei.poku@gmail.                                             
             com>                                                       To 
             Sent by:                  lisp-hug@lispworks.com              
             owner-lisp-hug@li                                          cc 
             spworks.com                                                   
                                                                   Subject 
                                       Memory allocation fragmentation?    
             11/30/2007 01:42                                              
             PM                                                            
                                                                           
                                                                           
             Please respond to                                             
                "Osei Poku"                                                
             <osei.poku@gmail.                                             
                   com>                                                    
                                                                           
                                                                           
                                                                           





Hello,

I noticed an issue with memory allocation of large sections of memory.
 I will outline the steps below.

Note: I have a 64-bit machine with 32 gigs of RAM running a 64-bit LW
on Linux so (all memory - kernel memory) is potentially available.

1. Create a sequence of large arrays using make-array.  The reason for
using a sequence of arrays instead of one big array is that the size
of the big array would exceed the max size of an array.  So a sequence
of large arrays it is...  This works fine.  Think of this as
(defparameter *list-of-arrays* (list (make-array <big number>)
(make-array <big number>) ... )) for some arbitrary number of arrays.

2. This allocation takes up about 25 gigs or RAM simply because the
data is that big.  So the length of the list depends on <big number>
and total size of the array that needs to allocated.  Specifically, I
am READ-SEQUENCEing an array of integers directly from a file on disk.
 I allocate the array then I call READ-SEQUENCE to populate the array.

3. Now I do (setq *list-of-arrays* nil) and then make sure that the
object is not referenced anymore (ie typing 1 a bunch of times at the
REPL). I redo the above allocation and I get an error after a few but
less than half of the arrays have been allocated.  This error is of
type HEAP-MEMORY-ERROR and says that it cannot allocate memory of that
size (I don't remember the exact error type or message since this was
some time ago when I was under pressure to finish the job and I did
not have the foresight to save the debug message).

4. I reduce <big number> and thus the length of the list so that the
size of the individual arrays are reduced.  Now the array allocation
for the same total amount of memory works.

5. I conclude that fragmentation of the memory is occurring making is
impossible to allocate the same large contiguous chunks of memory as
before.  The large allocations only work in a fresh lisp image.

6. I try to find a garbage collecting solution to defragment the
memory to no avail.

Any suggestions?

Osei