Re: Question for the experts...

I vaguely remember having a similar problem some time ago, I think the way  
it worked properly on W7 was something along these lines (note the  
keyword):

(fli:disconnect-module  
"C:/Users/Yuri/Misc/Code/C++/AudioStream/Debug/AudioStream.dll" :remove t)
(fli:register-module  
"C:/Users/Yuri/Misc/Code/C++/AudioStream/Debug/AudioStream.dll")

However, if the problem lies within the OS cache that might not help.  
Also, as far as I can remember, in order to overwrite a library it had to  
be disconnected first, unfortunately I do not have things set up in order  
to verify it now.



On Sat, 01 Aug 2015 20:44:10 +0100, David McClain  
<dbm@refined-audiometrics.com> wrote:

>
> During the creation of a C dynamic library for running time critical  
> routines, called from Lisp. I found that whenever I made modifications  
> to the library through the C compiler, I could not coax Lisp to drop  
> what it thinks it knows about the library, and reconnect to the new  
> version.
>
> This work was on Mac OS X so, being Unix-like, there was no problem  
> (unlike a Windows environment) overwriting the active library image from  
> the compiler while it was in use. The active previous image was already  
> in memory and hooked to the running Lisp session, and while its inode  
> may have been orphaned by the overwrite, that’s okay.
>
> I always had the following code for connecting to the library:
>
> (fli:disconnect-module :hsiirlib)
> (fli:register-module :hsiirlib
>                      :real-name
>                      (translate-logical-pathname
>                       #+:MAC   "PROJECTS:DYLIB;libHsIIR.dylib"
>                       #+:WIN32 "PROJECTS:DYLIB;libHsIIR.dll"))
>
> What I tried doing whenever a fresh image of the library had been  
> produced, was to re-execute that DISCONNECT-MODULE and REGISTER-MODULE.  
> Looking at FLI::*CURRENT-MODULES-LIST* with the inspector, or with  
> FLI:PRINT-FOREIGN-MODULES, always showed that the result of this  
> re-execution cleared out the cached entry points established from prior  
> executions of library routines from FLI code.
>
> But regardless of this cleared cache state, it seems that the system  
> dynloader always opted for what it already had seen before, presumably  
> still in memory cache somewhere. The library in question had only one  
> user — the Lisp code. It was a special purpose library, not used by  
> anything else in the running system.
>
> The only way I could see an indication of Lisp running new library code  
> was to fully quit the Lisp session and begin anew.
>
> Has anyone else figured out how to keep an interactive session alive and  
> also invoking fresh images of dynamic libraries?
>
> - DM
>
> _______________________________________________
> Lisp Hug - the mailing list for LispWorks users
> lisp-hug@lispworks.com
> http://www.lispworks.com/support/lisp-hug.html
>


-- 
Regards,

Yuri

http://www.disclosure.ie

_______________________________________________
Lisp Hug - the mailing list for LispWorks users
lisp-hug@lispworks.com
http://www.lispworks.com/support/lisp-hug.html

Re: Question for the experts...

>>> But regardless of this cleared cache state, it seems that the system dynloader always opted for what it already had seen before, presumably still in memory cache somewhere. The library in question had only one user — the Lisp code. It was a special purpose library, not used by anything else in the running system.
>>>
>>> The only way I could see an indication of Lisp running new library code was to fully quit the Lisp session and begin anew.
>>>
>>> Has anyone else figured out how to keep an interactive session alive and also invoking fresh images of dynamic libraries?
>>>
I don't know about Windows DLLs, but with Linux, you need to dlclose
and/or to dlopen with RTLD_DEEPBIND (a non-standard extension) so that
the new versions of the function are not shadowed by the old ones.

Alternatively, if you're generating the code, you could output the
SHA256 hash of your source code to the C function names, ensuring that
the linker does no shadowing whatsoever. Can be done with a macro
#define _(x) x##_12345

—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org
Politicians are like diapers: they must be changed often.
And for the same reasons. [Also, adults don't need either of them. — Faré]

_______________________________________________
Lisp Hug - the mailing list for LispWorks users
lisp-hug@lispworks.com
http://www.lispworks.com/support/lisp-hug.html

Re: Question for the experts...

<opinion>

There is no way it should be all that hard to do.  That is that hard is due to bugs or at least quite nasty code "smells".    Perhaps something can be learned from scripting languages that do this sort of thing quite a bit?

</opinion>

Re: Question for the experts...

David McClain <dbm@refined-audiometrics.com> writes:

> I have to agree with Samantha on this. I’m still puzzled by the
> responses I got from Duane Rettig about this, and even back then, I
> had a hard time understanding why the system dynloader should be so
> primitive? Perhaps it is a matter of Unix lore and tradition?

Or more exactly, limitations on some unix implementation of dynamic
libraries.  If you mostly use GNU/Linux and MacOSX, you might not
realize how primitive some other unices were.

-- 
__Pascal Bourguignon__                 http://www.informatimago.com/
“The factory of the future will have only two employees, a man and a
dog. The man will be there to feed the dog. The dog will be there to
keep the man from touching the equipment.” -- Carl Bass CEO Autodesk

_______________________________________________
Lisp Hug - the mailing list for LispWorks users
lisp-hug@lispworks.com
http://www.lispworks.com/support/lisp-hug.html

Re: Question for the experts...

Unable to parse email body. Email id is 13508

Re: Question for the experts...

Unable to parse email body. Email id is 13519

Re: Question for the experts...

David McClain <dbm@refined-audiometrics.com> writes:

> Nope… once again, tests confirm that the simple way of upgrading external libs does *NOT* work on LWM-7/64 with Mac OS X 10.10.4
>
> So is this a bug? LW??

I can't test LW for now, but here are some links and further information
that should help you determine what's happening.


https://developer.apple.com/library/mac/documentation/DeveloperTools/Conceptual/DynamicLibraries/000-Introduction/Introduction.html

Notice in particular those famous environment variables:

https://developer.apple.com/library/mac/documentation/DeveloperTools/Conceptual/DynamicLibraries/100-Articles/LoggingDynamicLoaderEvents.html#//apple_ref/doc/uid/TP40002077-SW1

(famous, because one of such environment variables allows you to get
root access ;-))

Anyways, this should help you see what happens on MacOSX when you work
with dynamic libraries.  Define all those variables in the terminal, and
launch LW from that terminal; this should log those message either in
the terminal or in Console.app.


https://developer.apple.com/library/mac/documentation/DeveloperTools/Conceptual/DynamicLibraries/100-Articles/UsingDynamicLibraries.html#//apple_ref/doc/uid/TP40002182-SW13

Notice:

https://developer.apple.com/library/mac/documentation/Darwin/Reference/ManPages/man3/dlclose.3.html#//apple_ref/c/func/dlclose

     dlclose() releases a reference to the dynamic library or bundle
     referenced by handle.  If the reference count drops to 0, the
     bundle is removed from the address space, and handle is rendered
     invalid.  Just before removing a dynamic library or bundle in this
     way, any termination routines in it are called.  handle is the
     value returned by a previous call to dlopen.

Does the reference count of your library go down to 0?
(probably)

     Prior to Mac OS X 10.5, only bundles could be unloaded.  Starting
     in Mac OS X 10.5, dynamic libraries may also be unloaded.  There
     are a couple of cases in which a dynamic library will never be
     unloaded: 1) the main executable links against it, 2) An API that
     does not supoort unloading (e.g. NSAddImage()) was used to load it
     or some other dynnamic library that depends on it, 3) the dynamic
     library is in dyld's shared cache.

Is your library in the dyld's shared cache?
(I don't know how it would get there; probably not.)

https://developer.apple.com/library/prerelease/mac/documentation/Darwin/Reference/ManPages/man1/update_dyld_shared_cache.1.html


I had old information, it seems that since 10.5, libraries can also be
unloaded.  So that should be possible, unless you are using an older
MacOSX.


-- 
__Pascal Bourguignon__                 http://www.informatimago.com/
“The factory of the future will have only two employees, a man and a
dog. The man will be there to feed the dog. The dog will be there to
keep the man from touching the equipment.” -- Carl Bass CEO Autodesk

_______________________________________________
Lisp Hug - the mailing list for LispWorks users
lisp-hug@lispworks.com
http://www.lispworks.com/support/lisp-hug.html

Re: Question for the experts...

David McClain <dbm@refined-audiometrics.com> writes:

> Hi Camille,
>
> I just tried a direct experiment here. I wrote a direct FLI interface
> to DLOPEN and DLCLOSE for myself:
>
>
> (fli:define-foreign-function (_dlclose "dlclose" :source)
>     ((handle :long))
>   :result-type :long
>   :language :ansi-c)
>
> (fli:define-foreign-function (_dlopen "dlopen" :source)
>     ((path (:reference-pass (:ef-mb-string :null-terminated-p t)))
>      (flags :long))
>   :result-type :long
>   :language :ansi-c)
>
> Then I looked at the list of modules that Lisp had opened, via the
> Mac Activity Monitor. Then I looked for a junk library that I wrote
> that had not yet been loaded. In my case, I had one laying around
> called /usr/local/lib64/libLispFDStuff.dylib.
>
> So here is the interaction:
>
> CL-USER 5 > (setf x (_dlopen "/usr/local/lib64/
> libLispFDFileStuff.dylib" 0))
> 105553117454144
>
> Now verify that the Activity Monitor shows that library as currently
> opened in the Lisp image…. Yes, it does. So try to close it and see
> what happens…
>
> CL-USER 6 > (_dlclose x)
> 0
>
> Now see what the Activity Monitor shows…. Hmmm…. Interesting… It’s
> gone!!

Ok. Now try actually using some symbol, call _dlsym.

-- 
__Pascal Bourguignon__                 http://www.informatimago.com/
“The factory of the future will have only two employees, a man and a
dog. The man will be there to feed the dog. The dog will be there to
keep the man from touching the equipment.” -- Carl Bass CEO Autodesk

_______________________________________________
Lisp Hug - the mailing list for LispWorks users
lisp-hug@lispworks.com
http://www.lispworks.com/support/lisp-hug.html

Re: Question for the experts...

> On Aug 4, 2015, at 8:07 PM, David McClain <dbm@refined-audiometrics.com> wrote:
> 
> I just looked at the man page for DLOPEN… I wonder if this flag got set along the way??
> 
> RTLD_NODELETE   The specified image is tagged so that will never be removed from the address space,
>                      even after all clients have released it via 
> dlclose
> ()

Note that there are various things that can cause a dynamic library never to be unloaded on OSX. Some are listed in the man page for dlclose:

https://developer.apple.com/library/prerelease/mac/documentation/Darwin/Reference/ManPages/man3/dlclose.3.html



_______________________________________________
Lisp Hug - the mailing list for LispWorks users
lisp-hug@lispworks.com
http://www.lispworks.com/support/lisp-hug.html

Re: Question for the experts...

David McClain <dbm@refined-audiometrics.com> writes:

> I just looked at the man page for DLOPEN… I wonder if this flag got
> set along the way??
>
>
> RTLD_NODELETE   The specified image is tagged so that will never be removed from the address space,
>                      even after all clients have released it via dlclose()

This most certainly is an option you don't want to set for your use
case.

But notice that if you have defined FFI functions using the result of
dlsym, (and therefore storing it in a closure or some other place), then
you will have lisp objects ("foreign function" objects), that will refer
to the address of the dynamically loaded symbol.  If you dlclose it, and
it gets removed from the address space, then this address will be
"dangling", and calling this foreign function will crash.

So you must keep track of any result of dlsym you've stored in the lisp
image, and disable it before calling dlclose.

For your test, it won't matter, because you won't store the result of
dlsym or try to call it after dlclose.

The point here, is whether the FFI of lispworks takes care of this.  If
not, that would be a reason to use the RTLD_NODELETE option.

But it seems you can set the :dlopen-flags from the register-module
call. You just need to pass a fixnum with the right flag value.

http://www.lispworks.com/documentation/lw70/FLI/html/fli-111.htm#marker-1609192

    The keyword values of dlopen-flags correspond to combinations of
    RTLD_* constants (see /usr/include/dlfcn.h). The value t means the
    same as :local-lazy. The value nil means the same as t except on
    Darwin. On Darwin the value nil means do not use dlopen, and use the
    older interfaces instead.

-- 
__Pascal Bourguignon__                 http://www.informatimago.com/
“The factory of the future will have only two employees, a man and a
dog. The man will be there to feed the dog. The dog will be there to
keep the man from touching the equipment.” -- Carl Bass CEO Autodesk

_______________________________________________
Lisp Hug - the mailing list for LispWorks users
lisp-hug@lispworks.com
http://www.lispworks.com/support/lisp-hug.html

Re: Question for the experts...

David McClain <dbm@refined-audiometrics.com> writes:

> Hi Pascal,
>
> I don’t think I want to use that flag… I was suggesting that perhaps
> Lipspworks might be using that flag, since I cannot unload the library
> that I originally loaded. I don’t know what to think about this.

Yes, I mean the same.

dlopen-flags

    Controls use of dlopen on Unix-based systems. One of t, nil,
    :local-now, :global-now, :global-lazy, :local-lazy, or a fixnum. The
    default value is nil on Darwin, and t on other platforms.

The point here is that most probably the option you currently use
implies including RTDL_NODELETE, or, if you leave the default value on
darwin it is nil and means to not use dlopen but some other older API.

Reading some more of the doc on the same page:

lifetime

    A keyword specifying the lifetime of the connection. One of
    :indefinite or :session. The default value is :indefinite.

I wouldn't be surprised if having a lifetime :indefinite might also have
an influence on the dlopen-flag and add automatically RTDL_NODELETE.  So
I would rather use explicitely :lifetime :session.  

You may test the behavior, but if you do not have the source of the
implementation, how can you be sure that the behavior won't change in
some other circumstance and nullify your test?



>> On Aug 4, 2015, at 7:08 PM, Pascal J. Bourguignon <pjb@informatimago.com> wrote:
>> 
>> But it seems you can set the :dlopen-flags from the register-module
>> call. You just need to pass a fixnum with the right flag value.
>> 
>> http://www.lispworks.com/documentation/lw70/FLI/html/fli-111.htm#marker-1609192
>> 
>>    The keyword values of dlopen-flags correspond to combinations of
>>    RTLD_* constants (see /usr/include/dlfcn.h). The value t means the
>>    same as :local-lazy. The value nil means the same as t except on
>>    Darwin. On Darwin the value nil means do not use dlopen, and use the
>>    older interfaces instead.

-- 
__Pascal Bourguignon__                 http://www.informatimago.com/
“The factory of the future will have only two employees, a man and a
dog. The man will be there to feed the dog. The dog will be there to
keep the man from touching the equipment.” -- Carl Bass CEO Autodesk

_______________________________________________
Lisp Hug - the mailing list for LispWorks users
lisp-hug@lispworks.com
http://www.lispworks.com/support/lisp-hug.html

Re: Question for the experts...

David McClain <dbm@refined-audiometrics.com> writes:

> Here’s a stab at a Q&D implementation of dynlib management for LWM on
> Mac OS X 10.10.4 and LWM 7.0.0/64. YMMV
> […]
> This really does give the answer 42!  Feel free to improve and pass
> along your improvements to me and others.

Sounds good.

-- 
__Pascal Bourguignon__                 http://www.informatimago.com/
“The factory of the future will have only two employees, a man and a
dog. The man will be there to feed the dog. The dog will be there to
keep the man from touching the equipment.” -- Carl Bass CEO Autodesk

_______________________________________________
Lisp Hug - the mailing list for LispWorks users
lisp-hug@lispworks.com
http://www.lispworks.com/support/lisp-hug.html

Re: Question for the experts...

>>>>> On Wed, 05 Aug 2015 04:34:40 +0200, Pascal J Bourguignon said:
> 
> David McClain <dbm@refined-audiometrics.com> writes:
> 
> > Hi Pascal,
> >
> > I don’t think I want to use that flag… I was suggesting that perhaps
> > Lipspworks might be using that flag, since I cannot unload the library
> > that I originally loaded. I don’t know what to think about this.
> 
> Yes, I mean the same.
> 
> dlopen-flags
> 
>     Controls use of dlopen on Unix-based systems. One of t, nil,
>     :local-now, :global-now, :global-lazy, :local-lazy, or a fixnum. The
>     default value is nil on Darwin, and t on other platforms.

Yes, that is the key to it.

There were problems using dlopen on the Mac for some libraries, so we continue
to use the older Mac-specific APIs that don't support unloading.  If you want
unloading, then you need :dlopen-flags t (or one of the other non-nil values).


> Reading some more of the doc on the same page:
> 
> lifetime
> 
>     A keyword specifying the lifetime of the connection. One of
>     :indefinite or :session. The default value is :indefinite.
> 
> I wouldn't be surprised if having a lifetime :indefinite might also have
> an influence on the dlopen-flag and add automatically RTDL_NODELETE.  So
> I would rather use explicitely :lifetime :session.  

Using lifetime :indefinite doesn't affect dlopen-flags.

The lifetime keyword controls what happens if you call register-module
followed by save-image (or deliver).  With :indefinite, the module remains
registered when you start the saved image, but with :session it is removed
(useful if you want to reload it from a different path for example).  In
general though we recommend loading module in an initialization function
rather than before saving the image because it allows better error reporting
if loading fails.

-- 
Martin Simmons
LispWorks Ltd
http://www.lispworks.com/

_______________________________________________
Lisp Hug - the mailing list for LispWorks users
lisp-hug@lispworks.com
http://www.lispworks.com/support/lisp-hug.html