Re: MMapper

Maybe this goes without saying, but a prime reason for our invention of the MMapper library follows from measurements which show that direct memory mapped file I/O runs about 100x faster than conventional stream based file I/O. Modern OS's implement paged memory very efficiently, obviating our implementation of a parallel layer in Lisp.

The other reason is that, by means of the implemented persistent object class, we can directly access portions of a file as though they were file resident C structs. Byte endian issues are handled automatically in MMapper to allow files written on one host to be read / written compatibly on an entirely different host. (e.g., PPC vs Intel)

Dr. David McClain

Chief Technical Officer

Refined Audiometrics Laboratory

4391 N. Camino Ferreo

Tucson, AZ 85750

email: dbm@refined-audiometrics.com

phone: 1.520.390.3995

web: http://refined-audiometrics.com

Re: MMapper

On 17 Sep 2010, at 19:00, David McClain wrote:

> Maybe this goes without saying, but a prime reason for our invention of the MMapper library follows from measurements which show that direct memory mapped file I/O runs about 100x faster than conventional stream based file I/O. Modern OS's implement paged memory very efficiently, obviating our implementation of a parallel layer in Lisp.

It would be interesting (though off-topic, probably) to know where the stream overhead comes from.  My (somewhat informed, but for a small selection of platforms) is that you don't need to memory-map files to get very good I/O performance, for programs which do their I/O carefully.  However these are C programs taking considerable advantage of various I/O options various unices let you specify.  Of course, for at least some of these platforms, the system memory-maps everything.

Re: MMapper

Martin Cracauer gave a talk about this at the second ECLM in 2006.
When they started with this, they used CMUCL, and that was quite some
time ago.  If I remember correctly, only the most basic plumbing is
done in C/C++, though, while the rest is Lisp.

Edi.


On Sat, Sep 18, 2010 at 7:56 PM, David McClain
<dbm@refined-audiometrics.com> wrote:
> Hi Chun,
> I find your comments interesting -- er, those of ITA using C++ to do the
> dirty work... I would be interested to hear just how much improvement is
> made by going to C++ for this effort. For myself, I find that I'm using ever
> less C / C++ in my work, resorting only when absolutely necessary. And with
> the capabilities already present in the LW FLI, I find that isn't very often
> at all. Perhaps ITA did the C++ route because they were using SBCL?
> Dr. David McClain
> Chief Technical Officer
> Refined Audiometrics Laboratory
> 4391 N. Camino Ferreo
> Tucson, AZ  85750
> email: dbm@refined-audiometrics.com
> phone: 1.520.390.3995
> web: http://refined-audiometrics.com
>
>
> On Sep 17, 2010, at 18:53, Chun Tian (binghe) wrote:
>
> Hi, David
> Actually your MMapper made me remember something related on ITA Software's
> web site [1]. In the section "C++ at ITA Software", they said:
> """
> While most of ITA's algorithm-intensive code is implemented in high-level
> languages like Lisp, we use C/C++ to provide system support to our
> Lisp-based applications where the Common Lisp runtime support is inadequate.
> For instance, we represent large static datasets as densely packed arrays
> stored in files. We use C++ to make the mmap() calls and fetch pointers into
> these mappings for Lisp to access.
> When virtual memory address space has been a limitation, we have employed
> tricks to fetch memory-mapped data by interacting with cooperating processes
> running on the same machine. We use C/C++ to manage this communication and
> the associated setup and error handling.
> """
> So, I think mapping has at least one important features which streaming I/O
> don't have: random access of large static datasets. And for 32-bit
> LispWorks, this feature seems allowing us access very large of such file
> which exceed virtual memory address space, isn't it? If this is true, I
> consider your MMapper very useful, and do this work in a Lisp way.
> --binghe
> [1] http://www.itasoftware.com/careers/work-at-ita/ita-engineering.html
> 在 2010-9-18，05:16， David McClain 写道：
>
> With regard to caching (persistent objects) and mmaping, the following
> page has some related facts.
>
> FWIW, when I first implemented the cacheing in MMapper (and elsewhere at
> even higher levels in Okeanos), I was able to discern approximately 10-20%
> speedup. Not a whole lot, but on long data analysis runs, it mounts up. I
> think the real payoff will be highly access-pattern dependent.
> Dr. David McClain
> Chief Technical Officer
> Refined Audiometrics Laboratory
> 4391 N. Camino Ferreo
> Tucson, AZ  85750
> email: dbm@refined-audiometrics.com
> phone: 1.520.390.3995
> web: http://refined-audiometrics.com
>
>
> On Sep 17, 2010, at 12:59, Ala'a Mohammad wrote:
>
>
>
>