Announce: MMapper Library Available

Hello all,

We decided to release our MMapper library to the public -- a library that extends the FLI in Lispworks to deal with memory mapped files. The library is in current use in our production code, handles OS X and Windows, both 32-bit and 64-bit versions of Lispworks. Our BTree code in Okeanos, among others, uses this library for its fast access. Of course, there are areas that could use some polish for extended uses.

We welcome your comments and improvements. The license is quite liberal -- have a look...

Download from:

http://refined-audiometrics.com/downloads/MMapper.zip

The library comes with an ASDF loader. Sections of the code that aren't germane to public use have been conditionally elided with the #+:COM.SD symbol. (i.e., you should not have :COM.SD in your *FEATURES* list, else you will run into missing components).

Documentation for the library is currently in the source files, and the TEST.LISP file shows examples of use.

There are two layers to the library -- the lowest layer implements FLI extensions to MMPTR objects which represent virtual address pointers relative to the start of a file. This lower layer is quite useful in its own right. You can perform FETCH and STORE of scalar items, vectors, arrays, ASCII strings, and struct slots.

Atop that lower layer, we build smart classes for persistent objects -- persistent in the sense that they refer to C structs that can be stored in the file. The example shown in TEST.LISP has plenty of demonstrable instances of the use of this higher level protocol. In every case, a C struct is defined first, for the information as it is stored on the file. Then a class is defined, derived from mmf:persistent-metalevel-class that refers to that C struct. Slots in the persistent objects can be normal slots, persistent slots, and computed slots. Slots can be cached for speedup of frequently examined pieces of file data.

I think it would be possible, but highly tedious, to port this code to some other Lisp, like Allegro or SBCL. I make no apologies for making this code LW-centric. The library code should be self-contained, needing no additional macros from other sources. Along with our code is a shamelessly hacked copy of the UUID library from Boian Tzonev. (We make heavy use of UUID's in our databases.)

Enjoy!

Dr. David McClain

Chief Technical Officer

Refined Audiometrics Laboratory

4391 N. Camino Ferreo

Tucson, AZ 85750

email: dbm@refined-audiometrics.com

phone: 1.520.390.3995

web: http://refined-audiometrics.com

Re: Announce: MMapper Library Available

Correction... UUID's are not needed in the code. So the UUID library is elided from MMapper.

Dr. David McClain

Chief Technical Officer

Refined Audiometrics Laboratory

4391 N. Camino Ferreo

Tucson, AZ 85750

email: dbm@refined-audiometrics.com

phone: 1.520.390.3995

web: http://refined-audiometrics.com

On Sep 14, 2010, at 12:58, David McClain wrote:

Hello all,

We decided to release our MMapper library to the public -- a library that extends the FLI in Lispworks to deal with memory mapped files. The library is in current use in our production code, handles OS X and Windows, both 32-bit and 64-bit versions of Lispworks. Our BTree code in Okeanos, among others, uses this library for its fast access. Of course, there are areas that could use some polish for extended uses.

We welcome your comments and improvements. The license is quite liberal -- have a look...

Download from:

http://refined-audiometrics.com/downloads/MMapper.zip

The library comes with an ASDF loader. Sections of the code that aren't germane to public use have been conditionally elided with the #+:COM.SD symbol. (i.e., you should not have :COM.SD in your *FEATURES* list, else you will run into missing components).

Documentation for the library is currently in the source files, and the TEST.LISP file shows examples of use.

There are two layers to the library -- the lowest layer implements FLI extensions to MMPTR objects which represent virtual address pointers relative to the start of a file. This lower layer is quite useful in its own right. You can perform FETCH and STORE of scalar items, vectors, arrays, ASCII strings, and struct slots.

Atop that lower layer, we build smart classes for persistent objects -- persistent in the sense that they refer to C structs that can be stored in the file. The example shown in TEST.LISP has plenty of demonstrable instances of the use of this higher level protocol. In every case, a C struct is defined first, for the information as it is stored on the file. Then a class is defined, derived from mmf:persistent-metalevel-class that refers to that C struct. Slots in the persistent objects can be normal slots, persistent slots, and computed slots. Slots can be cached for speedup of frequently examined pieces of file data.

I think it would be possible, but highly tedious, to port this code to some other Lisp, like Allegro or SBCL. I make no apologies for making this code LW-centric. The library code should be self-contained, needing no additional macros from other sources. Along with our code is a shamelessly hacked copy of the UUID library from Boian Tzonev. (We make heavy use of UUID's in our databases.)

Enjoy!

Dr. David McClain
Chief Technical Officer
Refined Audiometrics Laboratory
4391 N. Camino Ferreo
Tucson, AZ 85750

email: dbm@refined-audiometrics.com
phone: 1.520.390.3995
web: http://refined-audiometrics.com

Re: Announce: MMapper Library Available

My preference would be for the fine folks at LispWorks to set up an account on github to manage LispWorks contributed code. I just looked at the contributed code directory on the lispworks FTP server and there is no activity on it.

Short of LispWorks setting up a repository, I'd be willing to create an organization through my account on github to manage LispWorks specific projects. I'd like to include LispWorks in the name of the organization, but probably need the their blessing to do it. As a last resort, I could just create a public github repository with some generic name to get the ball rolling.

~ Tom

----------------------------------------------------------------
Thomas M. Hermann
Odonata Research LLC
http://www.odonata-research.com/
http://www.linkedin.com/in/thomasmhermann

On Thu, Sep 16, 2010 at 5:56 AM, Alexandre Paes <alex.paes@streetdogstudio.com> wrote:

I completely understand David's reservations about creating an online repository (git, darcs, whatever...) after all it does sound like it means a lot of trouble. But if i'm allowed the suggestion, maybe David won't really mind if another user (not offering myself here) takes up on this and creates the project while obviously respecting and giving the due credit to David (or even David's company)?

This could be the start of a new Common Lisp library, who knows? Just my 2 cents here.

Cheers,

Alexandre Paes

Re: Announce: MMapper Library Available

Unable to parse email body. Email id is 10556

Re: Announce: MMapper Library Available

I have no preference between github and common-lisp.net. The main criteria for me is that the various LispWorks contributions that are circulating get organized and tracked.

Thank you,

~ Tom

----------------------------------------------------------------
Thomas M. Hermann
Odonata Research LLC
http://www..odonata-research.com/
http://www.linkedin.com/in/thomasmhermann

On Fri, Sep 17, 2010 at 12:22 PM, Dave Fox <davef@lispworks.com> wrote:

Thanks for the suggestions.

Is there a reason for contributors not to use common-lisp.net, which
also has git ?

We are thinking that this would be better in cases where a
LispWorks-specific contribution later gets ported to another Common
Lisp implementation.

--
Dave Fox
LispWorks Ltd
St John's Innovation Centre, Cowley Road
Cambridge CB4 0WS
England
http://www.lispworks.com/

LispWorks Ltd is registered in England, company number 5114963. The
company's registered address is c/o Streets Whitmarsh Sterland LLP,
62 Hills Road, Cambridge CB2 1LA (but customers and suppliers should
write to our office address above). LispWorks Ltd is registered for
VAT with EC VAT ID: GB 833329531

> My preference would be for the fine folks at LispWorks to set up an account
> on github to manage LispWorks contributed code. I just looked at the
> contributed code directory on the lispworks FTP server and there is no
> activity on it.
>
> Short of LispWorks setting up a repository, I'd be willing to create an
> organization through my account on github to manage LispWorks specific
> projects. I'd like to include LispWorks in the name of the organization, but
> probably need the their blessing to do it. As a last resort, I could just
> create a public github repository with some generic name to get the ball
> rolling.
>
> ~ Tom
> ----------------------------------------------------------------
> Thomas M. Hermann
> Odonata Research LLC
> http://www.odonata-research.com/
> http://www.linkedin.com/in/thomasmhermann
>
>
> On Thu, Sep 16, 2010 at 5:56 AM, Alexandre Paes <
> alex.paes@streetdogstudio.com> wrote:
>
> >
> > I completely understand David's reservations about creating an online
> > repository (git, darcs, whatever...) after all it does sound like it means a
> > lot of trouble. But if i'm allowed the suggestion, maybe David won't really
> > mind if another user (not offering myself here) takes up on this and creates
> > the project while obviously respecting and giving the due credit to David
> > (or even David's company)?
> >
> > This could be the start of a new Common Lisp library, who knows? Just my 2
> > cents here.
> >
> >
> > Cheers,
> >
> > Alexandre Paes