Lisp HUG Maillist Archive

Curious case of slowdown with file output

Hi,

We have encountered a curious case of slowdown with file output: When performing an (open pathname :direction :output :if-exists :supersede), if the file indeed already exists, output is dramatically slower than if the file didn’t exist before. We see a degradation of a factor of 3, which can hurt a lot for very large files (several gigabytes). By doing a (when-let (path (probe-file pathname)) (assert (delete-file path))) before the call to ‘open, the slowdown can be avoided.

Does anybody know a different / better keyword to pass to :if-exists to avoid this issue? :rename-and-delete sounds clunky, but maybe that’s better?

This is in LispWorks 64bit on Linux.

Thanks,
Pascal

--
Pascal Costanza, ExaScience Life Lab, Intel, Belgium
pascal.costanza@intel.com



Intel Corporation NV/SA
Kings Square, Veldkant 31
2550 Kontich
RPM (Bruxelles) 0415.497.718. 
Citibank, Brussels, account 570/1031255/09

This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.


_______________________________________________
Lisp Hug - the mailing list for LispWorks users
lisp-hug@lispworks.com
http://www.lispworks.com/support/lisp-hug.html

Re: Curious case of slowdown with file output

Have you checked the I/O load on the machine?  Using :if-exists :supersede
causes the file to be opened with O_TRUNC, so this might be interacting badly
with the auto_da_alloc mount option if you are using ext4 (see
https://www.kernel.org/doc/Documentation/filesystems/ext4.txt).

:rename-and-delete will not be better because we treat it as a synonym for
:supersede.

-- 
Martin Simmons
LispWorks Ltd
http://www.lispworks.com/


>>>>> On Fri, 23 May 2014 09:06:42 +0000, Costanza, Pascal said:
> 
> Hi,
> 
> We have encountered a curious case of slowdown with file output: When performing an (open pathname :direction :output :if-exists :supersede), if the file indeed already exists, output is dramatically slower than if the file didn’t exist before. We see a degradation of a factor of 3, which can hurt a lot for very large files (several gigabytes). By doing a (when-let (path (probe-file pathname)) (assert (delete-file path))) before the call to ‘open, the slowdown can be avoided.
> 
> Does anybody know a different / better keyword to pass to :if-exists to avoid this issue? :rename-and-delete sounds clunky, but maybe that’s better?
> 
> This is in LispWorks 64bit on Linux.
> 
> Thanks,
> Pascal
> 
> --
> Pascal Costanza, ExaScience Life Lab, Intel, Belgium
> pascal.costanza@intel.com
> 
> 
> 
> Intel Corporation NV/SA
> Kings Square, Veldkant 31
> 2550 Kontich
> RPM (Bruxelles) 0415.497.718. 
> Citibank, Brussels, account 570/1031255/09
> 
> This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.
> 
> 
> _______________________________________________
> Lisp Hug - the mailing list for LispWorks users
> lisp-hug@lispworks.com
> http://www.lispworks.com/support/lisp-hug.html
> 

_______________________________________________
Lisp Hug - the mailing list for LispWorks users
lisp-hug@lispworks.com
http://www.lispworks.com/support/lisp-hug.html
Updated at: 2020-12-10 08:34 UTC