Lisp HUG Maillist Archive

ASDF weirdness

For some reason I have lost track of I decided I should make things so that various of my packages can use ASDF.  I'm using 64-bit LW 7.1.1 on OSX.  I think I'm using the bundled ASDF, but I might have updated it.  ASDF reports itself as being 3.3.0.

To configure the source registry system I have this form:

(asdf:initialize-source-registry
 '(:source-registry
   :inherit-configuration
   (:tree
    (:root "Local" "tfb" "packages" "asdf"))))

I know this is not how you are meant to do that, but it means I can have the configuration in a single file which I control with git rather than some obscure config file which I have no idea where it is, and I can put my packages in "/Local/tfb/packages/asdf/...".

And this works fine:

> (asdf:locate-system "org.tfeb.hax")
t
nil
#P"/Local/tfb/packages/asdf/org.tfeb.hax/org.tfeb.hax.asd"
nil
nil

All is shiny and good.

Except ... not.  This is great in the LW GUI, but I also have console images which are made by

<lispworks-binary> -build make-console-image.lisp

Where make-console-image.lisp is:

(in-package :cl-user)

;;; This is not needed, but harmless
(load-all-patches)

(save-image "lw"
            :clean-down t
            :console t
            :environment nil)
(quit)

And in these images nothing works:

CL-USER 1 > (asdf:locate-system "org.tfeb.hax")
nil
nil
nil
nil
nil

CL-USER 2 > asdf:*source-registry-parameter*
(:source-registry
 :inherit-configuration
 (:tree (:root "Local" "tfb" "packages" "asdf")))

As you can see the console image has read the init file.  I've also tried explicitly clearing the registry and redefining it.  No good.

Does anyone understand ASDF well enough to explain what's going on?  Every time I try and understand it it just fills me with depression and I end up using LW's defsystem and/or writing another makefile because it's easier, but I also realise that I kind of need to have a defsystem I can use on platforms other than LW if I want to make any of my code available to people.

Just knowing what function I should trace or hook I should set to tell ASDF to show me where it's looking would be good.

Thanks

--tim

_______________________________________________
Lisp Hug - the mailing list for LispWorks users
lisp-hug@lispworks.com
http://www.lispworks.com/support/lisp-hug.html

Re: ASDF weirdness

I'm not in a position to test this myself, but my best guess about what is wrong is that you need to invoke asdf:initialize-source-registry again, after the image restarts. It sounds like you might have tried this.

Minor note: the proper API function is asdf:find-system, which has a simpler return form than locate-system.

Also, when I look at the source registry form you posted, I note that there's an unquoted top in there, but I don't know if that's just a cut-and-paste error.

You can also ask questions on the asdf-devel mailing list (see the project page for ASDF on common-lisp.net)

Finally, have a look at the FAQ in the ASDF manual on the common-lisp.net page: ISTR that I put something in there about how to debug configurations.

Ping me if that is not enough to put you on the right track.

On 9 Jun 2020, at 12:43, Tim Bradshaw wrote:

For some reason I have lost track of I decided I should make things so that various of my packages can use ASDF. I'm using 64-bit LW 7.1.1 on OSX. I think I'm using the bundled ASDF, but I might have updated it. ASDF reports itself as being 3.3.0.

To configure the source registry system I have this form:

(asdf:initialize-source-registry
'(:source-registry
:inherit-configuration
(:tree
(:root "Local" "tfb" "packages" "asdf"top ))))

I know this is not how you are meant to do that, but it means I can have the configuration in a single file which I control with git rather than some obscure config file which I have no idea where it is, and I can put my packages in "/Local/tfb/packages/asdf/...".

And this works fine:

(asdf:locate-system "org.tfeb.hax")

t
nil
#P"/Local/tfb/packages/asdf/org.tfeb.hax/org.tfeb.hax.asd"
nil
nil

All is shiny and good.

Except ... not. This is great in the LW GUI, but I also have console images which are made by

<lispworks-binary> -build make-console-image.lisp

Where make-console-image.lisp is:

(in-package :cl-user)

;;; This is not needed, but harmless
(load-all-patches)

(save-image "lw"
:clean-down t
:console t
:environment nil)
(quit)

And in these images nothing works:

CL-USER 1 > (asdf:locate-system "org.tfeb.hax")
nil
nil
nil
nil
nil

CL-USER 2 > asdf:*source-registry-parameter*
(:source-registry
:inherit-configuration
(:tree (:root "Local" "tfb" "packages" "asdf")))

As you can see the console image has read the init file. I've also tried explicitly clearing the registry and redefining it. No good.

Does anyone understand ASDF well enough to explain what's going on? Every time I try and understand it it just fills me with depression and I end up using LW's defsystem and/or writing another makefile because it's easier, but I also realise that I kind of need to have a defsystem I can use on platforms other than LW if I want to make any of my code available to people.

Just knowing what function I should trace or hook I should set to tell ASDF to show me where it's looking would be good.

Thanks

--tim

_______________________________________________
Lisp Hug - the mailing list for LispWorks users
lisp-hug@lispworks.com
http://www.lispworks.com/support/lisp-hug.html

Robert P. Goldman
Research Fellow
Smart Information Flow Technologies (d/b/a SIFT, LLC)

319 N. First Ave., Suite 400
Minneapolis, MN 55401

Voice: (612) 326-3934
Email: rpgoldman@SIFT.net

Re: ASDF weirdness

On 9 Jun 2020, at 19:26, Robert Goldman <rpgoldman@sift.net> wrote:

I'm not in a position to test this myself, but my best guess about what is wrong is that you need to invoke asdf:initialize-source-registry again, after the image restarts. It sounds like you might have tried this.

Minor note: the proper API function is asdf:find-system, which has a simpler return form than locate-system.



Yes, I did try resetting it after the image starts properly.  Yes, sorry, I got confused: I'd tried both forms but pasted the wrong one.

Also, when I look at the source registry form you posted, I note that there's an unquoted top in there, but I don't know if that's just a cut-and-paste error.



It was, the actual form is right.

You can also ask questions on the asdf-devel mailing list (see the project page for ASDF on common-lisp.net)

Finally, have a look at the FAQ in the ASDF manual on the common-lisp.net page: ISTR that I put something in there about how to debug configurations.



Thanks I will have a look at it.

Ping me if that is not enough to put you on the right track.

On 9 Jun 2020, at 12:43, Tim Bradshaw wrote:

For some reason I have lost track of I decided I should make things so that various of my packages can use ASDF. I'm using 64-bit LW 7.1.1 on OSX. I think I'm using the bundled ASDF, but I might have updated it. ASDF reports itself as being 3.3.0.

To configure the source registry system I have this form:

(asdf:initialize-source-registry
'(:source-registry
:inherit-configuration
(:tree
(:root "Local" "tfb" "packages" "asdf"top ))))

I know this is not how you are meant to do that, but it means I can have the configuration in a single file which I control with git rather than some obscure config file which I have no idea where it is, and I can put my packages in "/Local/tfb/packages/asdf/...".

And this works fine:

(asdf:locate-system "org.tfeb.hax")

t
nil
#P"/Local/tfb/packages/asdf/org.tfeb.hax/org.tfeb.hax.asd"
nil
nil

All is shiny and good.

Except ... not. This is great in the LW GUI, but I also have console images which are made by

<lispworks-binary> -build make-console-image.lisp

Where make-console-image.lisp is:

(in-package :cl-user)

;;; This is not needed, but harmless
(load-all-patches)

(save-image "lw"
:clean-down t
:console t
:environment nil)
(quit)

And in these images nothing works:

CL-USER 1 > (asdf:locate-system "org.tfeb.hax")
nil
nil
nil
nil
nil

CL-USER 2 > asdf:*source-registry-parameter*
(:source-registry
:inherit-configuration
(:tree (:root "Local" "tfb" "packages" "asdf")))

As you can see the console image has read the init file. I've also tried explicitly clearing the registry and redefining it. No good.

Does anyone understand ASDF well enough to explain what's going on? Every time I try and understand it it just fills me with depression and I end up using LW's defsystem and/or writing another makefile because it's easier, but I also realise that I kind of need to have a defsystem I can use on platforms other than LW if I want to make any of my code available to people.

Just knowing what function I should trace or hook I should set to tell ASDF to show me where it's looking would be good.

Thanks

--tim

_______________________________________________
Lisp Hug - the mailing list for LispWorks users
lisp-hug@lispworks.com
http://www.lispworks.com/support/lisp-hug.html

Robert P. Goldman
Research Fellow
Smart Information Flow Technologies (d/b/a SIFT, LLC)

319 N. First Ave., Suite 400
Minneapolis, MN 55401

Voice: (612) 326-3934
Email: rpgoldman@SIFT.net


Re: ASDF weirdness

Unable to parse email body. Email id is 15279

Re: ASDF weirdness

On 10 Jun 2020, at 11:34, Martin Simmons <martin@lispworks.com> wrote:
> 
> Note that the pathspec is relative, so it wil depend on the current working
> directory and cl:*default-pathname-defaults*.  I don't know if this is a bug
> or a feature, but it is probably not what you want.
> 
> It might work better if you use (:tree "/Local/tfb/packages/asdf").

Thanks, this is it.  The LW GUI starts up in / (which is kind of annoying: I think I'd rather it started up in the user's home directory, but there's clearly not really any good choice) while a console image has a good idea of where it is, and this caused the search to be silently wrong.

And this as because I was confused about what :root is meant to do: I'd naively assumed it meant, well, the root directory.  But it doesn't:

  :ROOT
    ;; magic, for output-translations source only: paths that are relative
    ;; to the root of the source host and device

It means the root directory but only in some cases, and not in this case when it means, I think, nothing.

I've since also discovered that find-system doesn't actually search for systems as best I can see: that search happens when the source registry is defined.  So if you define the source registry at image start time, and there's a big tree to search, the image is slow to start: on my machine

  (initialize-source-registry
   '(:source-registry
     :inherit-configuration
     (:tree "/Local")))

makes the image take a great chunk of a minute to start (well, the first time: once the filesystem cache is warm it's better).

Well, OK, I'm going to revive my antique hack which knew how to search for things based on their names and modify it so it will look for asd files, and make sure that that tree is not where ASDF searches by default so my images start quickly.

However, thanks: this has solved it.

--tim





_______________________________________________
Lisp Hug - the mailing list for LispWorks users
lisp-hug@lispworks.com
http://www.lispworks.com/support/lisp-hug.html

Re: ASDF weirdness

On 10 Jun 2020, at 6:09, Tim Bradshaw wrote:

I've since also discovered that find-system doesn't actually search for systems as best I can see: that search happens when the source registry is defined. So if you define the source registry at image start time, and there's a big tree to search, the image is slow to start: on my machine

(initialize-source-registry
'(:source-registry
:inherit-configuration
(:tree "/Local")))

makes the image take a great chunk of a minute to start (well, the first time: once the filesystem cache is warm it's better).

Yes, that was done by Faré a long time ago. I believe that this was done to handle very large lisp systems like the ones that ITA was building, but you are right that it doesn't make a great tradeoff for people with more modest systems.

If you like looking for systems on an as-needed basis, you can use asdf:*central-registry*, the old school way of specifying where to look for systems, but note that it does not have the equivalent of :tree. See below

Well, OK, I'm going to revive my antique hack which knew how to search for things based on their names and modify it so it will look for asd files, and make sure that that tree is not where ASDF searches by default so my images start quickly.

I have an equivalent to this hack, called asd-finder. LMK off list if you would like a copy of it.

Also note that if you would like things to start more quickly, you can provide a smaller region for the :tree search to cover, possibly by having two different regions (one for common libraries and one for the actual system you are loading), and searching two smaller trees instead of one big one.

Best,
R

Re: ASDF weirdness

On 10 Jun 2020, at 17:08, Robert Goldman <rpgoldman@sift.net> wrote:

I have an equivalent to this hack, called asd-finder. LMK off list if you would like a copy of it.


I might do, but I'll have a go at reviving my old code first -- I'll mail you privately if so, thank you for the offer.

Also note that if you would like things to start more quickly, you can provide a smaller region for the :tree search to cover, possibly by having two different regions (one for common libraries and one for the actual system you are loading), and searching two smaller trees instead of one big one



Yes, in fact the thing that was irritating (and I think still is) is that if I have some external process which takes a system and then dumps it into a place where it should be found, then ASDF doesn't look for it without kicking it.

I realise though that I'm basically whining about a system which I'm getting for free and which I'm not contributing any resource to.  I hope that I didn't seem rude: I was not meaning to be, and I'm very grateful for all the people who've made ASDF exist!

--tim

Re: ASDF weirdness

On 11 Jun 2020, at 10:13, Tim Bradshaw wrote:

Also note that if you would like things to start more quickly, you can provide a smaller region for the `:tree` search to cover, possibly by having two different regions (one for common libraries and one for the actual system you are loading), and searching two smaller trees instead of one big one

Yes, in fact the thing that was irritating (and I think still is) is that if I have some external process which takes a system and then dumps it into a place where it should be found, then ASDF doesn't look for it without kicking it.

Yes, ASDF does now lock you into eagerly finding systems, rather than lazily finding them, unless you use asdf:*central-registry*, but if you do that, you can find things lazily.

The docs encourage you to use the new shiny thing, but *central-registry* is simple and has worked for me forever.

Updated at: 2020-12-10 08:28 UTC