Re: How to load a custom SQLite library using CFFI on macOS
Excellent! Thank you very much, Carlos!
—me
> On May 28, 2020, at 5:09 PM, Carlos Ungil <carlos.ungil@gmail.com> wrote:
>
> Hello,
>
> the following works for me (High Sierra):
>
> ====
>
> (ql:quickload :cffi)
>
> (push "/usr/local/Cellar/sqlite/3.31.1/lib/" cffi:*foreign-library-directories*)
>
> (ql:quickload :sqlite)
>
> ;; you can verify that the right library has been loaded
> (fli:print-foreign-modules)
>
> ;; cl-sqlite function definitions are equivalent to this
> (cffi:defcfun (bad-sqlite3-libversion "sqlite3_libversion") :string)
>
> (bad-sqlite3-libversion)
>
> ;; but you need to specify the library you want to use
> (cffi:defcfun (sqlite3-libversion "sqlite3_libversion" :library sqlite-ffi::sqlite3-lib) :string)
>
> (sqlite3-libversion)
>
> ====
>
> The problem is that cl-sqlite defines a library and loads it, but won’t really refer to it and the symbols that are already available will be used (inspecting #'|%cffi-foreign-function/BAD-SQLITE3-LIBVERSION| we see that it points to sqlite3_libversion in module NIL).
>
> You’ll need to patch the defcfun calls in sqlite-ffi.lisp to include the :library sqlite3-lib as above.
>
> The form (cffi:defcfun ("sqlite3_libversion" :library sqlite-ffi::sqlite3-lib) :string) works as well and I think (cffi:defcfun (sqlite3-libversion :library sqlite-ffi::sqlite3-lib) :string) should work but apparently it doesn’t.
>
> Cheers,
>
> Carlos
>
> PS: Using FLI, the code you sent uses the symbols it finds somewhere (you don’t need to load any library for it to work) but you can make it look it the right place:
>
> ====
>
> (fli:define-foreign-function
> (sqlite3-libversion "sqlite3_libversion" :source)
> ()
> :result-type (:pointer :char)
> :language :ansi-c
> :module "/usr/local/Cellar/sqlite/3.31.1/lib/libsqlite3.dylib")
>
> (fli:convert-from-foreign-string (sqlite3-libversion))
>
> ====
>
>
>
>
>> On 28 May 2020, at 17:39, mikel evins (as mevins at me dot com) <lisp-hug@lispworks.com> wrote:
>>
>> Try this:
>>
>> 1. Find out what version of SQLite is preinstalled on your system. For example, do
>>
>> $ which sqlite3
>>
>> If that will run the preinstalled SQLite, then it should tell you
>>
>> /usr/bin/sqlite3
>>
>> 2. Run sqlite:
>>
>> $ sqlite3
>>
>> 3. Find out the version of the installed SQLite:
>>
>> sqlite> select sqlite_version();
>>
>> On my Catalina dev box, it's 3.28.0. On my High Sierra test system it's 3.19.3.
>>
>> 4. Obtain a different version of SQLite. For example, fetch and build the amalgamation, or download a prebuilt binary. Make sure it's suitable for your OS version and its word size is the same as your Lispworks (presumably 64-bit on any plausibly recent macOS). Put it somewhere and take note of the full pathname to it. Also, make sure it's a dylib.
>>
>> 5. In Lispworks, load CFFI. For example,
>>
>> (ql:quickload :cffi)
>>
>> 6. Push the directory containing your custom SQLite dylib onto CFFI:*FOREIGN-LIBRARY-DIRECTORIES*.
>>
>> 7. Try loading the library. The simplest way is with quicklisp:
>>
>> (ql:quickload :sqlite)
>>
>> …but you could also define the sqlite library by hand using CFFI:DEFINE-FOREIGN-LIBRARY and load it using CFFI:USE-FOREIGN-LIBRARY or CFFI:LOAD-FOREIGN-LIBRARY. I;ve tried all of these (among other things).
>>
>> 8. Ask the loaded SQLite for its version.
>>
>> One way is to make an FLI call to the C API, something like this:
>>
>> (fli:define-foreign-function
>> (sqlite3-libversion "sqlite3_libversion" :source)
>> ()
>> :result-type (:pointer :char)
>> :language :ansi-c)
>>
>> then:
>>
>> (fli:convert-from-foreign-string (delectus::sqlite3-libversion))
>>
>> Another way is to use the same SELECT statement I showed above, but by passing it to SQLite with the cl-sqlite API:
>>
>> (defun sqlite-library-version (path)
>> (with-open-database (db path)
>> (execute-to-list db "SELECT sqlite_version()")))
>>
>> On Catalina, I always get version 3.28.0, and on High Sierra I always get 3.19.3, even if I loaded my hand-built copy of 3.28.0. I've tried the easy way (pushing my directory onto CFFI:*FOREIGN-LIBRARY-DIRECTORIES*. I've tried calling load-foreign-library directly with the absolute path of my dylib. No matter what I try, on High Sierra the library reports version 3.19.3.
>>
>> I could dispense with CFFI and just use the Lispworks FLI directly, but then I have to build my own API for SQLite instead of using the one in cl-sqlite. That's not a terrible lot of work, but for the sake of progress on my project it's just easier to use a different SQL query to get the results I want (and the different query performs well enough, so no sweat).
>>
>> So I'm not blocked, but I am still curious why I can't seem to make cl-sqlite and CFFI work the way I expect.
>>
>> Maybe I should try it with bare FLI, just to see what happens.
>>
>>> On May 28, 2020, at 9:23 AM, Frank Goenninger | Goenninger B&T <frank.goenninger@goenninger.net> wrote:
>>>
>>> … show me the code ;-) Seriously, let’s look at your code and try it here on my Macs.
>>>
>>> // Frank
>>>
>>>> Am 27.05.2020 um 22:39 schrieb mikel evins (as mevins at me dot com) <lisp-hug@lispworks.com>:
>>>>
>>>> My question's in the subject line.
>>>>
>>>> I've tried the obvious: pushing the directory that contains the dylib onto CFFI's foreign library directories list.
>>>>
>>>> I've also tried less obvious things, including modifying the cl-sqlite code to defer loading the library until I do it explicitly (with an absolute pathname), and even explicitly loading my local SQLite build by hand in the Listener.
>>>>
>>>> On macOS Catalina, the system -provided SQLite 3 (v 3.28.0) has the features I need; on my High Sierra test system, the system-provided SQLite (v 3.19.3) lacks some of them. No matter what I've tried with CFFI, and whether I've tested in the IDE or in my delivered app, Lispworks on High Sierra always loads the system-provided SQLite 3.19.3 (as confirmed by calling "SELECT sqlite_version()", and also by using FLI to call the "sqlite3_libversion" API).
>>>>
>>>> I noticed that I can define and call the "sqlite3_libversion" API function from a fresh Lispworks, without loading any SQLite library. Does Lispworks have a reference to SQLite3 built into it? If so, is there something I can do to enable it to use the library I supply instead?
>>>>
>>>> All advice and corrections gratefully received.
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Lisp Hug - the mailing list for LispWorks users
>>>> lisp-hug@lispworks.com
>>>> http://www.lispworks.com/support/lisp-hug.html
>>>
>>
>>
>> _______________________________________________
>> Lisp Hug - the mailing list for LispWorks users
>> lisp-hug@lispworks.com
>> http://www.lispworks.com/support/lisp-hug.html
>
_______________________________________________
Lisp Hug - the mailing list for LispWorks users
lisp-hug@lispworks.com
http://www.lispworks.com/support/lisp-hug.html