Re: How to create an array which size is larger then array-total-size-limit
I use a json parser jsown: https://github.com/madnificent/jsown
@Raymond, It only accept string as input so a little extra work should
be done to support memory mapped access.
@Devon, The idea to redefine acessor is good but reading them in
several steps is a little more convenient for me to parse it(as input
file is an array of json):
(with-open-file (in path)
(loop with eof-p = nil
with step = 20000
with result = nil
until eof-p
do (let* ((json (with-output-to-string (stream)
(write-char #\[ stream)
(loop for j from 0 below step
for line = (read-line in nil nil)
do (cond (line (write-sequence
line stream)
(write-char #\, stream))
(t (setf eof-p t)
(loop-finish)))))))
(setf (aref json (1- (length json))) #\])
(setf result (append result (jsown:parse json))))
finally (return result)))
Thanks all for your interesting ideas :)
With Best Regards,
jingtao.
On Wed, Apr 6, 2016 at 3:29 PM, Devon Sean McCullough
<LispWorks@jovi.net> wrote:
> Or redefine char/aref/whatever accessor this parser uses
> to read memory mapped octets instead of string characters.
>
> Peace
> --Devon
>
> P.S. If this parser can read from streams, just open the file.
>
> On Wed, April 6, 2016 02:13, Raymond Wiker wrote:
>> If this parser can read from streams it should be possible to use a
>> combination of make-string-input-stream and make-concatenated-stream... it
>> may also be possible to have a stream (gray stream?) that uses a
>> memory-mapped file or some other mechanism.
>>
>>
>>
>> On Wed, Apr 6, 2016 at 8:23 AM, Jens Teich <info@jensteich.de> wrote:
>>
>>
>>> What does this parser do?
>>>
>>>
>>> Am 06.04.16 um 08:04 schrieb Jingtao Xu:
>>>
>>>
>>>
>>>> Hi Raymond,
>>>>
>>>>
>>>> Thanks for your advice but the memory-mapped solution can't be used
>>>> directly because the parser only accept simple string as input.
>>>>
>>>> Maybe lispworks should support a very large value for
>>>> array-total-size-limit.
>>>>
>>>> With Best Regards,
>>>> jingtao.
>>>>
>>>>
>>>> On Wed, Apr 6, 2016 at 11:16 AM, Raymond Wiker <rwiker@gmail.com>
>>>> wrote:
>>>>
>>>>
>>>>> Maybe an abstraction over memory-mapped files might be useful? E.g,
>>>>> http://permalink.gmane.org/gmane.lisp.lispworks.general/10540.
>>>>>
>>>>>
>>>>> On 06 Apr 2016, at 05:01 , Jingtao Xu <jingtaozf@gmail.com> wrote:
>>>>>
>>>>>
>>>>>
>>>>> That's what I did now, but it's inconvenient.
>>>>>
>>>>>
>>>>> I find that sbcl has a very large value for array-total-size-limit
>>>>> but lispworks has a very small one,that's inconvenient for large
>>>>> data processing.
>>>>>
>>>>>
>>>>> With best regards,
>>>>> jingtao.
>>>>>
>>>>> On Wed, Apr 6, 2016 at 10:31 AM, Robert Smith <quad@symbo1ics.com>
>>>>> wrote:
>>>>>
>>>>>
>>>>> How about making multiple arrays in a list?
>>>>>
>>>>>
>>>>> Robert
>>>>>
>>>>>
>>>>> On Tue, Apr 5, 2016 at 6:54 PM, Jingtao Xu <jingtaozf@gmail.com>
>>>>> wrote:
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Hello all,
>>>>>
>>>>>
>>>>>
>>>>> Common lisp has a constant variable array-total-size-limit which
>>>>> limit the maximum size of an array, but in some situation I want to
>>>>> read a very large file(for example its size is 1G) to one string
>>>>> then parse it.
>>>>>
>>>>> So my question is that Is there one way to create a very large
>>>>> string which exceed the size specified by array-total-size-limit?
>>>>>
>>>>> Thanks.
>>>>>
>>>>>
>>>>> With Best Regards,
>>>>> jingtao.
>>>>>
>>>>> _______________________________________________
>>>>> Lisp Hug - the mailing list for LispWorks users
>>>>> lisp-hug@lispworks.com http://www.lispworks.com/support/lisp-hug.html
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Lisp Hug - the mailing list for LispWorks users
>>>>> lisp-hug@lispworks.com http://www.lispworks.com/support/lisp-hug.html
>>>>>
>>>>>
>>>>>
>>>>>
>>>> _______________________________________________
>>>> Lisp Hug - the mailing list for LispWorks users
>>>> lisp-hug@lispworks.com http://www.lispworks.com/support/lisp-hug.html
>>>>
>>>>
>>>>
>>>>
>>>
>>
>
_______________________________________________
Lisp Hug - the mailing list for LispWorks users
lisp-hug@lispworks.com
http://www.lispworks.com/support/lisp-hug.html