Can anyone suggest some tools/techniques for tracking down a memory leak? We've identified that a leak is happening, repeating an operation that shouldn't grow the heap after the first invocation is steadily growing memory size. There is a small amount of expected leakage - it interns new symbols each time for some aspect of processing - but I don't think that accounts for the growth we're seeing. Thanks Guy
Unable to parse email body. Email id is 10771
On Nov 26, 2010, at 18:44 , Guy Footring wrote: > > Can anyone suggest some tools/techniques for tracking down a memory leak? We've identified that a leak is happening, repeating an operation that shouldn't grow the heap after the first invocation is steadily growing memory size. There is a small amount of expected leakage - it interns new symbols each time for some aspect of processing - but I don't think that accounts for the growth we're seeing. I was in a similar situation earlier this year, where I saw that the memory usage was increasing over time. It turned out that this was caused by a bug in Lispworks, where console output was being accumulated in the image. The solution for this was to periodically set win32::*console-outputs* to nil. I wrote some code to investigate the memory usage, including some functions for scanning through all objects in memory and counting the number of allocated objects of each type. This was trivial, given that I could use hcl:sweep-all-objects, but I'd be happy to share even this trivial code if you want something to start from.
On Sun, Nov 28, 2010 at 6:06 PM, Guy Footring <Guy@footring.demon.co.uk> wrote:
> In message <44884C84-072D-4708-87C9-902AC04667D1@gmail.com>, Raymond Wiker
> <rwiker@gmail.com> writes
>>
>> I was in a similar situation earlier this year, where I saw that the
>> memory usage was increasing over time. It turned out that this was caused by
>> a bug in Lispworks, where console output was being accumulated in the image.
>> The solution for this was to periodically set win32::*console-outputs* to
>> nil.
>>
>> I wrote some code to investigate the memory usage, including some
>> functions for scanning through all objects in memory and counting the number
>> of allocated objects of each type. This was trivial, given that I could use
>> hcl:sweep-all-objects, but I'd be happy to share even this trivial code if
>> you want something to start from.
>
> Hi Raymond.
> I've been playing around with sweep-all-objects, but I'd be interested to
> see what you have to see how it compares with my quick first-cut approach of
> doing a gc, then sticking everything found into hashtable and then seeing
> what's new after running the operation. I'm seeing all sorts of stuff that
> I wasn't expecting to see, making me wonder if objects will get found
> correctly in a hash table if GC has moved them...
Interference from other threads, maybe?
3 simple functions; the first lists the number of internal and
external symbols in each package, sorted by total symbol count per
package. The second lists the number of allocated objects per
class/type, while the third dumps all simple-strings to a file (which
I did because I noticed that the allocation of simple-strings appeared
to be growing more than I expected it to).
(defun package-sizes (&optional (stream t))
(dolist (package (sort (list-all-packages)
#'>
:key (lambda (p)
(+ (slot-value p
'system::internal-symbols-count)
(slot-value p
'system::external-symbols-count)))))
(with-slots (system::name system::internal-symbols-count
system::external-symbols-count)
package
(format stream "~&~40a: internal: ~5,' d, external: ~5,' d~%"
system::name
system::internal-symbols-count
system::external-symbols-count))))
(defun type-counts (&optional (stream t))
(let ((typemap (make-hash-table :test #'eq)))
(system:with-other-threads-disabled
(hcl:sweep-all-objects (lambda (o)
(incf (gethash (class-name (class-of
o)) typemap 0)))))
(let ((types-and-counts nil))
(maphash (lambda (k v)
(push (cons k v) types-and-counts))
typemap)
(dolist (type-and-count (sort types-and-counts #'> :key #'cdr))
(format stream "~&~80a: ~10d~%"
(concatenate 'string
(package-name (symbol-package (car
type-and-count)))
":"
(symbol-name (car type-and-count)))
(cdr type-and-count))))))
(defun dump-simple-strings (&optional (stream t))
(flet ((emit-char (c)
(if (and (typep c 'base-char) (graphic-char-p c) (not (char= c #\\)))
(format stream "~c" c)
(format stream "\\~4,'0X" (char-code c)))))
(system:with-other-threads-disabled
(hcl:sweep-all-objects
(lambda (o)
(when (eq (class-name (class-of o)) 'common-lisp:simple-string)
(let ((string-length (length o)))
(format stream "~&Length: ~7d; Value = " (length o))
(if (< (length o) 128)
(loop for c across o
do (emit-char c))
(progn
(loop for c across o
for i below 60
do (emit-char c))
(format stream " ... ")
(loop for i from (- string-length 60) below string-length
do (emit-char (char o i)))))
(terpri stream))))))))