Aha, I think this is helpful. So, if I'm correct, the cffi and groveling that's used requires the compiling procedures which are gone after delivering, so I think I need to:
- Somehow add all the .64xfasl files to my projects directory/cache (as well as the actual *.dylib libraries used when generating it)
- lispworks -build my-delivery.lisp, making sure things are factored such that it doesn't load cl-freetype2 at all during this stage
- The delivered application can load cl-freetype2 during it's startup methods.
MacOS doesn't support unloading unfortunately. There is also a problem if you
close a module that has explicit references (such as the :library argument to
cffi:defcfun).
The recommended approach is to not load the library before creating the
deliverable (and then load it on startup as you mentioend below).
--
Martin Simmons
LispWorks Ltd
>>>>> On Fri, 04 Dec 2020 20:00:22 -0800, Steven Githens said:
>
> Hi Colin,
>
> Thanks for this meticulously crafted set of references. I'm on MacOS Catalina with Lispworks 7.1.2, so everything should be 64-bit since it doesn't support 32-bit apps. I actually went ahead and hacked up cl-freetype2 quick to use embedded-modules since that would be great, but pretty much get the same errors.
>
> I did try out the deliver option ":in-memory-delivery t" and of course it works fine, so it's definitely some particular issue with loading the lisp image back up.
>
> Also randomly, I just did some small testing in the REPL with unloading the dylib, and got results that surprised me, in that I was still able to use the library after unloading it... I don't think it's being aggressively reloaded, because when I unload it again I get `nil` which would be expected...
>
>
> ~Steve
>
> On Fri, Dec 4, 2020, at 6:14 PM, Colin J.E. Lupton wrote:
> > I came upon a similar issue, and was able to find the solution in the LispWorks documentation.
> >
> > You can work your way through the following list of solutions from top to bottom:
> >
> > - LW 29.8, external libraries used with 64-bit LispWorks must be 64-bit libraries, with 32-bit LispWorks must be 32-bit libraries
> > - DV 4.4.5, be certain the calls to load the foreign libraries with CFFI, register-module (or the embedded-module interface mentioned below) are made within function bodies and not top-level. This ensures the dynamic library is loaded in full at runtime, when the wrapper function is called, and not during compilation or delivery.
> > - DV 10.6, there are some useful notes here on how the FLI converts between foreign and Lisp objects, and parts of the delivery process that might clobber your FLI; mostly missing templates, occasionally memory errors.
> > - FLI 5.5, how to correctly load and access both C and C++ dynamic libraries; there’s some extra work with C++. Haven’t used FREETYPE2 myself, so not immediately clear on its implementation.
> > - FLI 5.6, covers embedding dynamically-loaded foreign libraries inside the Lisp image, instead of relying separate libraries loaded at runtime. The embedded-module interface handles a lot of extra complexity, if it can be used with CFFI. Maybe try this on a new branch of your source control some other time.
> >
> > That should cover all of the possible cases for the errors you’re getting. If not, see LW 11..4 for a review of memory management in 64-bit LispWorks, or LW 11.3 for 32-bit.
> >
> > If you’re interested in why you’re getting those specific memory error patterns, LW 11.6 has a few notes on reducing image size and garbage collection of foreign-objects that explains most of it. For example, clean-down is called automatically before delivery, IIRC.
> >
> >
> > --
> >
> > Colin J.E. Lupton
> > *CEO, Founder, Principal Quantum Metamachinist*
> > *Black Brane Systems, Inc.*
> >
> > phone: +1 (833) 227-2630 x700
> > fax: +1 (833) 227-2630
> >
> >
> > *Sent:* Friday, December 4, 2020 7:48:39 PM
> > *Subject:* Re: Building deliverables using CFFI libraries
> >
> > Hi again,
> >
> > Thanks for the responses, it's been helpful to look through all the example code. Also, thanks Arnold for pointing out that CFFI actually does use fli under the covers, it was helpful to clone the CFFI repo and look through how that actually works.
> >
> > I think I'm starting to get close... in my Delivery script I'm using:
> >
> > (cffi:close-foreign-library 'FREETYPE2::FREETYPE2)
> >
> > and I can see that it's returning T.
> >
> > and in application start defun I'm using:
> >
> > (pushnew
> > (make-pathname :directory
> > (append (butlast (pathname-directory (lw:lisp-image-name)))
> > '("Resources" "libs")))
> > cffi:*foreign-library-directories* :test #'equal)
> > (cffi:use-foreign-library freetype2::freetype2)
> >
> > Which appears to be reloading it, although I reliably get a handful of different loading errors. (some of them below) The path in these errors IS the correct path to the library file though...
> >
> > -Steve
> >
> >
>
> > Error: Segmentation violation(11) [code 0] at 1006BF128
>
> > Foreign code offset #x38 from symbol "FT_Stream_New"
>
>
> > rax 1EC00 ; rbx 108164810 ; rcx 600000008600 ; rdx 70000101FB58
>
> > rsp 70000101FAE0 ; rbp 70000101FB10 ; rdi 1080F5570 ; rsi 50
>
> > r8 1 ; r9 40A03B6D3B ; r10 4040070C6B ; r11 1006C00A0
>
> > r12 70000101FBB0 ; r13 600000008680 ; r14 70000101FB58 ; r15 1080F5570
>
> > 1 (abort) Quit process.
>
> >
> > or
> >
> >
>
> > Error: FreeType error: INVALID-LIBRARY-HANDLE
>
> > 1 (abort) Quit process.
>
> >
> >
> >
> > On Fri, Dec 4, 2020, at 2:11 PM, Arnold Noronha wrote:
> >>
> >> On Fri, Dec 04, 2020 at 01:02:50PM -0800, Steven Githens wrote:
> >> > Hi all,
> >> >
> >> > This is a convoluted question, but to start, I'm wondering if folks have successfully used the lisp works delivery mechanism incorporating libraries that use the CFFI module (not the builtin ffi module that is part of lispworks).
> >>
> >> Since CFFI uses FLI under the hood you should still be able to use FLI
> >> functions.
> >>
> >> I used fli:disconnect-module just before delivery. (I am using my own
> >> library though, so not 100% sure it's easy to do this from an external
> >> library loaded with CFFI). Then use fli:get-embedded-module in your
> >> app, and fli:install-embedded-module in your init function in the
> >> delivered app.
> >>
> >> --Arnold
> >>
> >
>
_______________________________________________
Lisp Hug - the mailing list for LispWorks users