Large Uploads Through mod_lisp Sometimes Failing

Hi Marc,
Hi All,

I have been using mod_lisp for several years and I am very happy with  
it.
Thanks Marc, for writing, publishing and maintaining it - I known  
from experience that doing this takes a lot of effort!

My KPAX Common Lisp Web Application Framework works with three front- 
ends:
either mod_lisp, portableaserve and my own s-http-server.

This week we were adding some functionality to one of our web  
applications that required uploading files using a multipart form  
encoded data POST request. We did this before in several situations  
and also this time things worked as expected.

However, there seems to be a problem when uploading large (multi- 
megabyte files): this fails almost consistently.

The culprit seems to be that read-sequence, trying to read the body  
string from the mod_lisp connection in one go, fails to read  
everything at once (returning less than the buffer length). Normally,  
one would then loop and try again with an offset (:start), although I  
never had to do this in the past (so I assume that the read-sequence  
implementation handles some if not most of the buffering/networking  
issues). Now, even after retrying multiple times, no more data comes  
from the mod_lisp connection socket.

I think it has to be something in mod_lisp/apache, because the same  
example test code works consistently with large files when using my  
own, very simple, s-http-server. In this case, read-sequence is used  
in exactly the same way, reading the whole posted body in one go,  
without retrying. The only difference is that the socket connection  
being read from comes directly from the browser instead of from  
mod_lisp/apache.

In the C code I see that mod_lisp is using ap_send_fb_length to send  
the body - but that doesn't help me.

Has anybody else ever encountered similar problems ?

I am using Apache/1.3.33 (Darwin) mod_lisp/2.43 with LispWorks Pro  
4.4.6 on Mac OS X 10.4.3

Thx,

Sven

--
Sven Van Caekenberghe - http://homepage.mac.com/svc
Beta Nine - software engineering - http://www.beta9.be

"Lisp isn't a language, it's a building material." - Alan Kay

Re: Large Uploads Through mod_lisp Sometimes Failing

Hi Sven!

On Sun, 11 Dec 2005 11:47:51 +0100, Sven Van Caekenberghe <sven@beta9.be> wrote:

> However, there seems to be a problem when uploading large (multi-
> megabyte files): this fails almost consistently.

I cannot confirm that there is a general problem with mod_lisp.  I've
just tried again and I can upload pretty large files (one example was
460 MB) without any problems using the test code from TBNL.  (Setup:
Debian testing, Apache 1.3.33, mod_lisp 2.43, LWL 4.4.6, TBNL 0.8.8.)

> The culprit seems to be that read-sequence, trying to read the body
> string from the mod_lisp connection in one go, fails to read
> everything at once (returning less than the buffer length).

Hmm, this behaviour (returning part of the buffer without blocking) is
advertised as an optional feature on AllegroCL but I wouldn't expect
LW to do that if the stream isn't closed.

Maybe you're seeing a Mac bug (in OS X, Apache, or LW)?

Cheers,
Edi.

Re: Large Uploads Through mod_lisp Sometimes Failing

On Dec 11, 2005, at 2:47 AM, Sven Van Caekenberghe wrote:
>
> However, there seems to be a problem when uploading large (multi- 
> megabyte files): this fails almost consistently.
>
> The culprit seems to be that read-sequence, trying to read the body  
> string from the mod_lisp connection in one go, fails to read  
> everything at once (returning less than the buffer length).  
> Normally, one would then loop and try again with an offset  
> (:start), although I never had to do this in the past (so I assume  
> that the read-sequence implementation handles some if not most of  
> the buffering/networking issues). Now, even after retrying multiple  
> times, no more data comes from the mod_lisp connection socket.

Unfortunately I don't have much more to offer but I have run into  
similar problems in Apache2/SVN after upgrading a machine from 10.3  
to 10.4. Checkouts worked fine but checkins (uploads) of anything  
substantial failed. I eventually found this bug filed against APR.

http://issues.apache.org/bugzilla/show_bug.cgi?id=34332

I also noticed that Apple's recent instructions on getting subversion  
up an running instructed one to manually apply the same basic patch  
and avoid the the poll() system call.

http://developer.apple.com/tools/subversionxcode.html

Another tool I use; ganglia, fell victim to the busted poll() system  
call as well.

http://bugzilla.ganglia.info/cgi-bin/bugzilla/show_bug.cgi?id=78
http://lists.apple.com/archives/darwin-dev/2005/May/msg00220.html

Where I'm going with all this is that socket behavior does appear to  
have changed enough under 10.4 that more than one piece of code  
broken under certain conditions. It possible that Apache 1.33 is also  
running into problems.

Hope that helps.

-greg

_______________________________________________________
Greg Wuller                             greg@wuller.com
_______________________________________________________

Re: Large Uploads Through mod_lisp Sometimes Failing

On Sun, 11 Dec 2005 19:27:45 +0100, Sven Van Caekenberghe <sven@beta9.be> wrote:

> I haven't yet looked at how you actually try to download the POST
> body, but I assume you are doing with read-sequence directly in a
> buffer sized by the Content-Length ?

No, I think that won't work well with really large files.  The stream
is read more or less one octet at a time and at the same time parse by
Janis Dzerins' RFC 2388 code.

> I still think/feel that it is mod_lisp or apache's fault, not
> LispWork's.

It could also be the operating system.

> Maybe I should test apache2+mod_lisp2 

I would try it if I were you - one more data point.

Re: Large Uploads Through mod_lisp Sometimes Failing

On Sun, 11 Dec 2005 18:41:14 -0800, Greg Wuller <greg@falcon.csc.calpoly.edu> wrote:

> For some reason Safari didn't (initially) like TNBL upload test. Any
> time I tried to upload a file it would sit there for a long time
> seemingly doing nothing. Firefox worked much more reliably but two
> of the early upload attempts did result in a "Document Contains No
> Data" error dialog when redisplaying the new upload.html page.

Would you mind testing this with TBNL in "stand-alone" mode (without
Apache/mod_lisp) to make sure this is not a bug in TBNL?  I don't have
a Mac to test with.

Thanks,
Edi.

Re: Large Uploads Through mod_lisp Sometimes Failing

On Tue, 13 Dec 2005 10:06:52 +0100, Sven Van Caekenberghe <sven@beta9.be> wrote:

> I looked at the TBNL source code, and there too, read-sequence is
> used for the whole large body at once,

As I already said when I answered your question this is /not/ the
case.

Re: Large Uploads Through mod_lisp Sometimes Failing

On Tue, 13 Dec 2005 10:45:25 +0100, Sven Van Caekenberghe <sven@beta9.be> wrote:

> Then I must have misread your code, the (most) relevant use of read-
> sequence in your code sems to be:
>
> In file request.lisp:
>
> (defun get-post-data (&optional (request *request*))
>    "Reads the posted data from the stream and stores the raw contents
> in the corresponding slot of the REQUEST object."
>    (let* ((headers-in (headers-in request))
>           (content-length (string-assoc "content-length" headers-in)))
>      (when content-length
>        (let ((content (make-string (parse-integer content-length
>                                                   :junk-allowed t))))
>          (read-sequence content (string-assoc "content-stream"
>          headers-in))
>          (setf (slot-value request 'raw-post-data) content)))))
>
> That is reading everything at once, no ?

That's how "application/x-www-form-urlencoded" data is read.  File
uploads are sent with a "multipart/form-data" content type.  Also from
request.lisp:

  (cond ((string-equal "application/x-www-form-urlencoded" content-type)
         (and (string-assoc "content-length" headers-in)
              (form-url-encoded-list-to-alist
               (cl-ppcre:split "&"
                               (get-post-data request)))))
        ((starts-with-p content-type "multipart/form-data;"
                        :test #'char-equal)
         (handler-case
           (parse-rfc2388-form-data (string-assoc "content-stream"
                                                  headers-in)
                                    content-type)
           (error (msg)
             (log-message :error
                          "While parsing multipart/form-data parameters: ~A"
                          msg)
             nil))))

Look at the code for PARSE-RFC2388-FORM-DATA.

Re: Large Uploads Through mod_lisp Sometimes Failing

On Tue, 13 Dec 2005 13:23:20 +0100, Sven Van Caekenberghe <sven@beta9.be> wrote:

> OK, now I see why I didn't find it, read-seqence isn't used at all,
> the data is read using read-char!
>
> Function: read-until-next-boundary
> File: rfc2388.lisp
>
> So I guess this means an implicit reliance on buffering inside the
> streams.

Yes.

> On the other end, the data is (optionally) pushed to a real file
> instead of being held in memory.
>
> Unless I am misreading the code again ;-)

No, that's correct.

FWIW, you can easily test with READ-SEQUENCE if your files aren't too
large by replacing this piece of code in request.lisp

                       (parse-rfc2388-form-data (string-assoc "content-stream"
                                                              headers-in)
                                                content-type)

with this one:

                       (let ((data (get-post-data request)))
                         (parse-rfc2388-form-data (make-string-input-stream data)
                                                  content-type))

On Linux this works fine for files up to about 4.5 MB - if they're
much larger you'll hit the maximum string size limitation in LW (and
you'll see a TBNL error message in Apache's log file).

Does this work on your Mac?

Cheers,
Edi.