mmap & arrays

Does anyone have any experience with (or examples of) using mmap (or an mmap-related function) to map files into vectors.  My aim being the usual one of being able to talk nicely (ie not with a mass of seeking) to huge files which I do not want to have to copy into arrays).

I don't have much (any recent) experience with the FLI, but it looks to me like the best I can probably do is to create a foreign array which I can then talk to with foreign-aref, rather than something that will work with aref.  Is it reasonable to expect that foreign-aref is as quick (or in the ballpack of) as aref (assuming good type declarations)?

I'm happy for LW-specific answers.  I'm actually only working on OSX, but it would be nice if they were Unixoid-portable (and even nicer if they were portable to Unixoids and Windows).

Sorry if this is a question which has an obvious answer: I've not put much research into it.  Annoyingly I have some memory of doing this a long time ago but it just might not have been with LW.

Thanks

--tim

_______________________________________________
Lisp Hug - the mailing list for LispWorks users
lisp-hug@lispworks.com
http://www.lispworks.com/support/lisp-hug.html

Re: mmap & arrays

On 22 Feb 2018, at 15:07, Madhu <enometh@meer.net> wrote:

If you're using gnus with gmane, you can type `j 10540'
to retrieve a post by David Mcclain on Tue, 14 Sep 2010,
which announced an MMapper library for lispworks,
and `A T' to get the whole thread,
and `j 10555' to get the discussion on a parallel thread "Re: Mapper"

Thanks: I should be but am not, but I found this anyway.

[I don't think I used it myself: I couldn't find andy code which used
it. I do have some code from that time that used the fli to define
mmap and dealt directly with the array. which really didn't help when
dealing with lisp objects in lisp memory (if the mmaped file involves
lisp objects..)

No, they're files of bytes (or sometimes encoded floats, but the encoding is idiosyncratic as there is packing so I need to deal with that anyway), so I'm mostly OK there, as far as I have thought about it.

Allegro simple-streams had a mapped-file-simple-stream class, which
added an :mmaped t keyword argument to OPEN, and the voila file stream
would be backed by a mmapped file, but you could use lisp operations
on it. This is nice, if you're happy to seek with POSITION but the
concerns about efficiency you mentioned remained I never got around
to measuring them]

Yes, I vaguely remember that. I particularly want not to have to use stream semantics though (not that there's anything wrong with them).

--tim

Re: mmap & arrays

Unable to parse email body. Email id is 14740