Bad object in ... and an occasional crash


I'm running my delivered app on Linux and occasionally it spits out
this message:

  <**> Bad object in general 00000001 ptr 37ea4a4c previous 00000010
  <**> Bad object in general 00000001 ptr 37ea4a4c previous 00000010
  <**> Bad pointer found - tagged as general 206ec59c 37ea4a30 37ea4a30 00007308
  <**> Bad object in general 00000001 ptr 37ea4a4c previous 00000010

  <**> Illegal tag in mark-object 206e6576 ptr 00000010 previous 00000010
  <**>Bad pointer in entry table object 36073558 slot-pointer 36076cf8 pointer 206e6576
  Stack dump
  36073548  3c0a3e79 6d74682f 72613e6c 00000010
  36073558  04020000 03560417 0004ef50 00000001
  36073568  00575f00 74683c0a 0a3e6c6d 65683c0a
  36073578  0a3e6461 74656d3c 74682061 652d7074
  36073588  76697571 6f43223d 6e65746e 79542d74

The app often keeps going, but sometimes in crashes at this point.

Some background on the code: The program has 64 threads running and
takes up about 600 MB of ram.  Mostly it does disk and network io, but
it does cons a fair amount as it calculates what to write to disk.
Sometimes it takes up a lot less ram (300 MB or so).  When it does
take less RAM it seems to be be "happier" and print out the messages
much less often.

Has anyone else seen this? Any ideas on how I could fix it? I'm guess
it's in GC, but I really don't know for sure.

Regards,
Chris Dean

P.S.  Thanks to everyone on this list for all their help.  This has
      been a great community to be a part of and I really appreciate
      all of your help!

Re: Bad object in ... and an occasional crash

[Sorry, I cut off this part of the email]

On a crash, I get a message like this:

  ** << SERIOUS PROBLEMS DURING GARBAGE COLLECTION    >>  **

  SIG = 0000000b  PC = 2005f77f
  Simple backtrace:
  0805cbe4 200c6d88 #<cfo 200c6b1a>

  0805cc10 200c623f #<cfo 200c622a>

Regards,
Chris Dean

Re: Bad object in ... and an occasional crash

Well, we've found that Lispworks can exhibit this kind of behavior under either low-physical-memory conditions or very large image sizes. You say your app has reached 600mb; before restructuring things, our app at times exceeded 700mb on a 1GB machine and occassionally I'd see behavior similar to yours. I also think I saw a post here recently that mentioned image sizes in the ranges you and I have experienced approaches the location at which shared libraries are mapped, but I don't recall if this was Windows or some other platform.

We changed certain algorithms in our system to keep image sizes generally below 200mb, but that might not be an option for you.

- david

On 5/12/06, Chris Dean <ctdean@sokitomi.com> wrote:

[Sorry, I cut off this part of the email]

On a crash, I get a message like this:

** << SERIOUS PROBLEMS DURING GARBAGE COLLECTION >> **

SIG = 0000000b PC = 2005f77f
Simple backtrace:
0805cbe4 200c6d88 #<cfo 200c6b1a>

0805cc10 200c623f #<cfo 200c622a>

Regards,
Chris Dean

--
And now these three remain: faith, hope, and love.
But the greatest of these is love.
-- 1 Corinthians 13:13

For wisdom is more precious than rubies,
and nothing you desire can compare with her.
-- Proverbs 8:11

Re: Bad object in ... and an occasional crash


"David Young" <youngde811@gmail.com> writes:
> Well, we've found that Lispworks can exhibit this kind of behavior under
> either low-physical-memory conditions or very large image sizes. You say
> your app has reached 600mb; before restructuring things, our app at times
> exceeded 700mb on a 1GB machine and occassionally I'd see behavior similar
> to yours. 

Great info, thanks.  

I have plenty of memory (4 GB on those machines), but I'll try
partitioning into 2 copies each running 32 threads and see if that
helps.

Regards,
Chris Dean

Re: Bad object in ... and an occasional crash


> I have plenty of memory (4 GB on those machines), but I'll try
> partitioning into 2 copies each running 32 threads and see if that
> helps.

We partitioned down to 24 threads on each copy (about 300mb for each
image) and still have the same problem.  Thanks for the advice,
though.

Regards,
Chris Dean

Re: Bad object in ... and an occasional crash

Chris Dean <ctdean@sokitomi.com> writes:

> The app often keeps going, but sometimes in crashes at this point.
>
> Has anyone else seen this? Any ideas on how I could fix it? I'm guess
> it's in GC, but I really don't know for sure.

We have a system with very similar parameters to what you describe.
We have problems identical to yours.  We have tried to debug them
for over a year.  In a large system, with lots of FLI code, it's
hard to ever point the finger at any one thing, but I think there
is a problem with the GC.  I'm sad to say I don't have a lot of hope
for a fix at this point, besides the "voodoo programming let's rearrange
the world and hope it works" type "fix".

I'd be happy to compare notes with you privately about what our
system loads etc to help track down commonalities, if you wish.

                 -Alain Picard

Re: Bad object in ... and an occasional crash

Alain Picard <Alain.Picard@memetrics.com> writes:
> We have a system with very similar parameters to what you describe.
> We have problems identical to yours.  We have tried to debug them
> for over a year.  In a large system, with lots of FLI code

That's too bad - I hoped I was unique enough that it would turn out to
be an some obvious error on part.  For what it's worth, I have no FFI
code in the app that is crashing.

Thanks for the information.

Regards,
Chris Dean

Re: Bad object in ... and an occasional crash

Have you involved Lispworks support? Our experience with them has been very good.

david

On 5/15/06, Chris Dean <ctdean@sokitomi.com> wrote:

Alain Picard <Alain.Picard@memetrics.com> writes:
> We have a system with very similar parameters to what you describe..
> We have problems identical to yours.  We have tried to debug them
> for over a year.  In a large system, with lots of FLI code

That's too bad - I hoped I was unique enough that it would turn out to
be an some obvious error on part.  For what it's worth, I have no FFI
code in the app that is crashing.

Thanks for the information.

Regards,
Chris Dean

Re: Bad object in ... and an occasional crash

Oh, and something else I just considered. Are you delivering (via DELIVER) your app? If so, at what level? If it's anything other than 0, try delivering at level 0 and see what happens.

david

On 5/15/06, Chris Dean <ctdean@sokitomi.com> wrote:

Alain Picard <Alain.Picard@memetrics.com> writes:
> We have a system with very similar parameters to what you describe..
> We have problems identical to yours.  We have tried to debug them
> for over a year.  In a large system, with lots of FLI code

That's too bad - I hoped I was unique enough that it would turn out to
be an some obvious error on part.  For what it's worth, I have no FFI
code in the app that is crashing.

Thanks for the information.

Regards,
Chris Dean

Re: Bad object in ... and an occasional crash


"David Young" <youngde811@gmail.com> writes:
> Oh, and something else I just considered. Are you delivering (via DELIVER)
> your app? If so, at what level? If it's anything other than 0, try
> delivering at level 0 and see what happens.

I am delivering at level 0.  But thanks for the suggestion!

Regards,
Chris Dean