Lisp HUG Maillist Archive

On the road to replacing C++

Hello All,

 

Before I get into the questions I would like to ask, I want to mention something about my area of interest. I am primarily a system programmer. What I mean by “system programmer” in this case is that I develop things that run on a server (bulk processing, routing, monitoring, etc). I do this work primarily for financial institutions. I also do a fair amount of work in networking and system monitoring, management and virtualization. I have been doing a lot of looking around/reading/testing and this doesn’t appear to be an area that very many Lispers are working in. One of the main issues I have come up against is performance. I know that performance is an area that many developers are unnecessarily preoccupied with, but that is not the case here.  I am trying to use Lisp as a higher level (hopefully) lower maintenance replacement for several things that are currently written in C++. My personal intention for the use of Lisp is to provide the higher level glue logic for these programs. I envision a layer of C code providing OS access and things like image processing etc.. and using Lisp to provide the higher level logic that glues it all together. My questions will inevitably be in this vain, so if this list is not the appropriate place for such questions please let me know.

 

Why lisp-hug? Well.. my reasoning is that I’m a licensed LispWorks customer and I very much want the opinion of folks who are using Lisp (LispWorks) in a commercial/professional setting. If I am wrong about this, please let me know.

 

I am finding that a lot of the Lisp libraries/packages available on the web are not exactly written with the system programmer mindset. Some of them are beautiful and very educational, but they lack a focus on performance, scalability, and maintainability. I am not one to reinvent the wheel unless there is no other option, but it’s beginning to look like I’m going to have to start pretty close to the OS on a lot of the things I need. I’m wondering if y’all would agree with my findings or if you could possibly point me to some Lisp packages that are more in the vain I described.

 

I am currently working on some benchmarks that will hopefully help me figure out the depth and scope of the C layer that will be necessary to accomplish the goal of replacing C++. I’m wondering if y’all could share some of your experiences with me that involve using the existing Lisp i/o facilities available (ie; disk, network) and the smp/threading capabilities of LispWorks and other. I do plan to do benchmarks, but I think it would be great to hear from y’all, due to my being so new to Lisp. Would it be considered acceptable to post code fragments to this list to get a second opinion on code that I think should run faster, but doesn’t?

 

Interfacing Lisp with C is something that I’m going to be working on a lot. I’m curious about the overhead involved in calling C functions from Lisp and vice-versa. How does the Lispworks FLI compare to CFFI and others? Which one would y’all use and why? The particulars of Lisp/C and C/Lisp behavior in multi-threaded applications is also an area that I going to need to understand pretty well. The LispWorks docs provide some good information on this, but I trying to distill it down to some simple rules for access between Lisp/C and C/Lisp where threads are involved. Are there any docs/papers around that address this in good detail?

 

Thanks,

Gerry

 

    

RE: On the road to replacing C++

Hi Paul,

 

I think perhaps my post was unclear. I don’t have a performance issue at all with C++. I have been looking for a C++ replacement for a number of reasons, but performance was not one of them. For example, I don’t like the direction C++ is headed at all. The performance issues I’m having are with the various Lisp libraries I have tried. I’m trying to figure out what it will take to make Lisp a viable C++ alternative for the work I do.  

 

Thanks,

Gerry

 

 

From: Paul Billington [mailto:paulfwb@mac.com]
Sent: Sunday, February 17, 2013 3:40 AM
To: Gerry Weaver
Cc: lisp-hug@lispworks.com
Subject: Re: On the road to replacing C++

 

Having read your post, I am unclear why you wish to replace C++. From my experience as a programmer of performant systems C++ (a superset of C) can be very performant if care is taken, e.g. Avoiding inappropriate run-time bindings and over zealous use of object creation and destruction etc.

 

What aspect of C++ are you thinking of when referring to poor performance?

 

Best regards,

Paul

Paul F. Billington

+41 79439 2281

paulfwb@acm.org

 

"The past is a foreign country: they do things differently there." (L. P. Hartley)


On 16 Feb 2013, at 23:54, Gerry Weaver <gerryw@compvia.com> wrote:

Hello All,

 

Before I get into the questions I would like to ask, I want to mention something about my area of interest. I am primarily a system programmer. What I mean by “system programmer” in this case is that I develop things that run on a server (bulk processing, routing, monitoring, etc). I do this work primarily for financial institutions. I also do a fair amount of work in networking and system monitoring, management and virtualization. I have been doing a lot of looking around/reading/testing and this doesn’t appear to be an area that very many Lispers are working in. One of the main issues I have come up against is performance. I know that performance is an area that many developers are unnecessarily preoccupied with, but that is not the case here.  I am trying to use Lisp as a higher level (hopefully) lower maintenance replacement for several things that are currently written in C++. My personal intention for the use of Lisp is to provide the higher level glue logic for these programs. I envision a layer of C code providing OS access and things like image processing etc. and using Lisp to provide the higher level logic that glues it all together. My questions will inevitably be in this vain, so if this list is not the appropriate place for such questions please let me know.

 

Why lisp-hug? Well.. my reasoning is that I’m a licensed LispWorks customer and I very much want the opinion of folks who are using Lisp (LispWorks) in a commercial/professional setting. If I am wrong about this, please let me know.

 

I am finding that a lot of the Lisp libraries/packages available on the web are not exactly written with the system programmer mindset. Some of them are beautiful and very educational, but they lack a focus on performance, scalability, and maintainability. I am not one to reinvent the wheel unless there is no other option, but it’s beginning to look like I’m going to have to start pretty close to the OS on a lot of the things I need. I’m wondering if y’all would agree with my findings or if you could possibly point me to some Lisp packages that are more in the vain I described.

 

I am currently working on some benchmarks that will hopefully help me figure out the depth and scope of the C layer that will be necessary to accomplish the goal of replacing C++. I’m wondering if y’all could share some of your experiences with me that involve using the existing Lisp i/o facilities available (ie; disk, network) and the smp/threading capabilities of LispWorks and other. I do plan to do benchmarks, but I think it would be great to hear from y’all, due to my being so new to Lisp. Would it be considered acceptable to post code fragments to this list to get a second opinion on code that I think should run faster, but doesn’t?

 

Interfacing Lisp with C is something that I’m going to be working on a lot. I’m curious about the overhead involved in calling C functions from Lisp and vice-versa. How does the Lispworks FLI compare to CFFI and others? Which one would y’all use and why? The particulars of Lisp/C and C/Lisp behavior in multi-threaded applications is also an area that I going to need to understand pretty well. The LispWorks docs provide some good information on this, but I trying to distill it down to some simple rules for access between Lisp/C and C/Lisp where threads are involved. Are there any docs/papers around that address this in good detail?

 

Thanks,

Gerry

 

    

RE: On the road to replacing C++

Hi Pascal,

 

Yes. You hit the nail right on the head. The new C++ (BOOST) is a good example of why I’m trying to make the move to Lisp.  

 

Thanks,

Gerry

 

 

From: Pascal Costanza [mailto:pc@p-cos.net]
Sent: Sunday, February 17, 2013 7:06 AM
To: Gerry Weaver
Cc: Paul Billington; lisp-hug@lispworks.com
Subject: Re: On the road to replacing C++

 

Hi Gerry,

 

In my experience, with Common Lisp, and specifically with LispWorks, you can easily get in the range of a factor of 1.5 or so of the speed of good C++ code. So in principle, nothing speaks against using Common Lisp as a replacement for C++ for systems programming.

 

However, Common Lisp is a language designed with productivity as a primary goal, and only then with performance as a secondary goal. This means that you can only achieve performance if you are willing to put in extra work to get all the type declarations and optimization declarations right. I don't believe that the majority of existing Common Lisp libraries is written with such attention to low-level performance details in mind. So I would expect that you have to put in more work into tuning libraries yourself, or simply be more careful during library selection.

 

C++ libraries can also suffer from this problem, because as you said yourself, C++ also has a number of features that can lead to excess overheads if used without care. So I'm not sure that C++ is strictly better in this regard. You can probably also easily choose the wrong library, and only later discover that there are unexpected, unnecessary performance overheads. (If someone uses BOOST, I'm immediately also wary of potential problems… ;)

 

Pascal

 

On 17 Feb 2013, at 11:02, Gerry Weaver <gerryw@compvia.com> wrote:



Hi Paul,

 

I think perhaps my post was unclear. I don’t have a performance issue at all with C++. I have been looking for a C++ replacement for a number of reasons, but performance was not one of them. For example, I don’t like the direction C++ is headed at all. The performance issues I’m having are with the various Lisp libraries I have tried. I’m trying to figure out what it will take to make Lisp a viable C++ alternative for the work I do.  

 

Thanks,

Gerry

 

 

From: Paul Billington [mailto:paulfwb@mac.com] 
Sent: Sunday, February 17, 2013 3:40 AM
To: Gerry Weaver
Cc: lisp-hug@lispworks..com
Subject: Re: On the road to replacing C++

 

Having read your post, I am unclear why you wish to replace C++. From my experience as a programmer of performant systems C++ (a superset of C) can be very performant if care is taken, e.g. Avoiding inappropriate run-time bindings and over zealous use of object creation and destruction etc.

 

What aspect of C++ are you thinking of when referring to poor performance?

 

Best regards,

Paul

Paul F. Billington

+41 79439 2281

paulfwb@acm.org

 

"The past is a foreign country: they do things differently there." (L. P. Hartley)


On 16 Feb 2013, at 23:54, Gerry Weaver <gerryw@compvia.com> wrote:

Hello All,

 

Before I get into the questions I would like to ask, I want to mention something about my area of interest. I am primarily a system programmer. What I mean by “system programmer” in this case is that I develop things that run on a server (bulk processing, routing, monitoring, etc). I do this work primarily for financial institutions. I also do a fair amount of work in networking and system monitoring, management and virtualization. I have been doing a lot of looking around/reading/testing and this doesn’t appear to be an area that very many Lispers are working in. One of the main issues I have come up against is performance. I know that performance is an area that many developers are unnecessarily preoccupied with, but that is not the case here.  I am trying to use Lisp as a higher level (hopefully) lower maintenance replacement for several things that are currently written in C++. My personal intention for the use of Lisp is to provide the higher level glue logic for these programs. I envision a layer of C code providing OS access and things like image processing etc. and using Lisp to provide the higher level logic that glues it all together. My questions will inevitably be in this vain, so if this list is not the appropriate place for such questions please let me know.

 

Why lisp-hug? Well.. my reasoning is that I’m a licensed LispWorks customer and I very much want the opinion of folks who are using Lisp (LispWorks) in a commercial/professional setting. If I am wrong about this, please let me know.

 

I am finding that a lot of the Lisp libraries/packages available on the web are not exactly written with the system programmer mindset. Some of them are beautiful and very educational, but they lack a focus on performance, scalability, and maintainability. I am not one to reinvent the wheel unless there is no other option, but it’s beginning to look like I’m going to have to start pretty close to the OS on a lot of the things I need. I’m wondering if y’all would agree with my findings or if you could possibly point me to some Lisp packages that are more in the vain I described.

 

I am currently working on some benchmarks that will hopefully help me figure out the depth and scope of the C layer that will be necessary to accomplish the goal of replacing C++. I’m wondering if y’all could share some of your experiences with me that involve using the existing Lisp i/o facilities available (ie; disk, network) and the smp/threading capabilities of LispWorks and other. I do plan to do benchmarks, but I think it would be great to hear from y’all, due to my being so new to Lisp. Would it be considered acceptable to post code fragments to this list to get a second opinion on code that I think should run faster, but doesn’t?

 

Interfacing Lisp with C is something that I’m going to be working on a lot. I’m curious about the overhead involved in calling C functions from Lisp and vice-versa. How does the Lispworks FLI compare to CFFI and others? Which one would y’all use and why? The particulars of Lisp/C and C/Lisp behavior in multi-threaded applications is also an area that I going to need to understand pretty well. The LispWorks docs provide some good information on this, but I trying to distill it down to some simple rules for access between Lisp/C and C/Lisp where threads are involved. Are there any docs/papers around that address this in good detail?

 

Thanks,

Gerry

 

    

 

--

Pascal Costanza

The views expressed in this email are my own, and not those of my employer.

 

 

 

RE: On the road to replacing C++

Hi Bradford,

 

I’m sorry. I guess the way I worded my post is confusing everyone .When I speak of C in the context of replacing C++, I mean that I’m trying to figure out what C (FLI) code will be necessary to bring Lisp performance into the range of C++ for my requirements. For example, would I use a particular Lisp socket library, the library provided by LispWorks, or would I write my own FLI interface to one that I’ve written in C? The performance issues I refer to are only with regard to the various Lisp libraries I’ve tried. I hope that helps clarify things a bit. It was pretty late when I wrote the post and I think my brain was a little fried ;-)

 

Thanks,

Gerry

 

From: Bradford Miller [mailto:bradfordmiller@me.com]
Sent: Sunday, February 17, 2013 7:15 AM
To: Gerry Weaver
Subject: Re: On the road to replacing C++

 

Gerry,

 

If I understand your question, you want to keep your algorithmic code in C++ or whatever, and use Lisp as the coordination or 'scripting' code that schedules how these lower level algorithms process some unit of work, correct?

 

In that case, I'm not sure why performance is an issue - the amount of time spent by your system is presumably mostly in the underlying algorithms, and only a small fraction in the control code.

 

So I'm assuming I'm not really understanding your problem…

 

On Feb 16, 2013, at 5:54 PM, Gerry Weaver <gerryw@compvia.com> wrote:



Hello All,

 

Before I get into the questions I would like to ask, I want to mention something about my area of interest. I am primarily a system programmer. What I mean by “system programmer” in this case is that I develop things that run on a server (bulk processing, routing, monitoring, etc). I do this work primarily for financial institutions. I also do a fair amount of work in networking and system monitoring, management and virtualization. I have been doing a lot of looking around/reading/testing and this doesn’t appear to be an area that very many Lispers are working in. One of the main issues I have come up against is performance. I know that performance is an area that many developers are unnecessarily preoccupied with, but that is not the case here.  I am trying to use Lisp as a higher level (hopefully) lower maintenance replacement for several things that are currently written in C++. My personal intention for the use of Lisp is to provide the higher level glue logic for these programs. I envision a layer of C code providing OS access and things like image processing etc. and using Lisp to provide the higher level logic that glues it all together. My questions will inevitably be in this vain, so if this list is not the appropriate place for such questions please let me know.

 

Why lisp-hug? Well.. my reasoning is that I’m a licensed LispWorks customer and I very much want the opinion of folks who are using Lisp (LispWorks) in a commercial/professional setting. If I am wrong about this, please let me know.

 

I am finding that a lot of the Lisp libraries/packages available on the web are not exactly written with the system programmer mindset. Some of them are beautiful and very educational, but they lack a focus on performance, scalability, and maintainability. I am not one to reinvent the wheel unless there is no other option, but it’s beginning to look like I’m going to have to start pretty close to the OS on a lot of the things I need. I’m wondering if y’all would agree with my findings or if you could possibly point me to some Lisp packages that are more in the vain I described.

 

I am currently working on some benchmarks that will hopefully help me figure out the depth and scope of the C layer that will be necessary to accomplish the goal of replacing C++. I’m wondering if y’all could share some of your experiences with me that involve using the existing Lisp i/o facilities available (ie; disk, network) and the smp/threading capabilities of LispWorks and other. I do plan to do benchmarks, but I think it would be great to hear from y’all, due to my being so new to Lisp. Would it be considered acceptable to post code fragments to this list to get a second opinion on code that I think should run faster, but doesn’t?

 

Interfacing Lisp with C is something that I’m going to be working on a lot. I’m curious about the overhead involved in calling C functions from Lisp and vice-versa. How does the Lispworks FLI compare to CFFI and others? Which one would y’all use and why? The particulars of Lisp/C and C/Lisp behavior in multi-threaded applications is also an area that I going to need to understand pretty well. The LispWorks docs provide some good information on this, but I trying to distill it down to some simple rules for access between Lisp/C and C/Lisp where threads are involved. Are there any docs/papers around that address this in good detail?

 

Thanks,

Gerry

 

    

 

RE: On the road to replacing C++

Hi David,

 

That’s very encouraging. I would be quite happy if I could achieve even half of that gain in productivity. Would you recommend using the LispWorks FLI or do you use something else? Also, do you specify types in the Lisp code or do use that as a way to optimize after profiling?

 

Thanks,

Gerry

 

 

From: David McClain [mailto:david@acudora.com]
Sent: Sunday, February 17, 2013 12:42 PM
To: Gerry Weaver
Cc: Paul Billington; lisp-hug@lispworks.com
Subject: Re: On the road to replacing C++

 

Ditto the comments from Pascal... I coded an elaborate LIFO cached direct file mapping system entirely in Lisp and achieved a 100x speedup in file access for my database system. So the fault is not with Lisp per se. Using Lisp made it possible to write that massive systems programming chore in a few days and have it working properly. 

 

Anecdotally, I find that I can write and debug Lisp about 10x faster than I can write C/C++, and one line of Lisp corresponds to about 10-100 lines of C/C++. So I see a productivity increase of around 100x-1000x in just sticking with Lisp as much as possible, and using C/C++ only for a thin veneer of glue routines.

 

Regardless of what language you use, you will probably run out of gas at around 100 KLOC. So any major project will self-destruct unless you use a productivity enhancer like Lisp.  See Greenspun's 10th Rule...

 

http://en.wikipedia.org/wiki/Greenspun's_tenth_rule

 

- DM

 

On Feb 17, 2013, at 03:02 AM, Gerry Weaver <gerryw@compvia.com> wrote:



Hi Paul,

 

I think perhaps my post was unclear. I don’t have a performance issue at all with C++. I have been looking for a C++ replacement for a number of reasons, but performance was not one of them. For example, I don’t like the direction C++ is headed at all. The performance issues I’m having are with the various Lisp libraries I have tried. I’m trying to figure out what it will take to make Lisp a viable C++ alternative for the work I do.  

 

Thanks,

Gerry

 

 

From: Paul Billington [mailto:paulfwb@mac.com] 
Sent: Sunday, February 17, 2013 3:40 AM
To: Gerry Weaver
Cc: lisp-hug@lispworks..com
Subject: Re: On the road to replacing C++

 

Having read your post, I am unclear why you wish to replace C++. From my experience as a programmer of performant systems C++ (a superset of C) can be very performant if care is taken, e.g. Avoiding inappropriate run-time bindings and over zealous use of object creation and destruction etc.

 

What aspect of C++ are you thinking of when referring to poor performance?

 

Best regards,

Paul

Paul F. Billington

+41 79439 2281

paulfwb@acm.org

 

"The past is a foreign country: they do things differently there." (L. P. Hartley)


On 16 Feb 2013, at 23:54, Gerry Weaver <gerryw@compvia.com> wrote:

Hello All,

 

Before I get into the questions I would like to ask, I want to mention something about my area of interest. I am primarily a system programmer. What I mean by “system programmer” in this case is that I develop things that run on a server (bulk processing, routing, monitoring, etc). I do this work primarily for financial institutions. I also do a fair amount of work in networking and system monitoring, management and virtualization. I have been doing a lot of looking around/reading/testing and this doesn’t appear to be an area that very many Lispers are working in. One of the main issues I have come up against is performance. I know that performance is an area that many developers are unnecessarily preoccupied with, but that is not the case here.  I am trying to use Lisp as a higher level (hopefully) lower maintenance replacement for several things that are currently written in C++. My personal intention for the use of Lisp is to provide the higher level glue logic for these programs. I envision a layer of C code providing OS access and things like image processing etc. and using Lisp to provide the higher level logic that glues it all together. My questions will inevitably be in this vain, so if this list is not the appropriate place for such questions please let me know.

 

Why lisp-hug? Well.. my reasoning is that I’m a licensed LispWorks customer and I very much want the opinion of folks who are using Lisp (LispWorks) in a commercial/professional setting. If I am wrong about this, please let me know.

 

I am finding that a lot of the Lisp libraries/packages available on the web are not exactly written with the system programmer mindset. Some of them are beautiful and very educational, but they lack a focus on performance, scalability, and maintainability. I am not one to reinvent the wheel unless there is no other option, but it’s beginning to look like I’m going to have to start pretty close to the OS on a lot of the things I need. I’m wondering if y’all would agree with my findings or if you could possibly point me to some Lisp packages that are more in the vain I described.

 

I am currently working on some benchmarks that will hopefully help me figure out the depth and scope of the C layer that will be necessary to accomplish the goal of replacing C++. I’m wondering if y’all could share some of your experiences with me that involve using the existing Lisp i/o facilities available (ie; disk, network) and the smp/threading capabilities of LispWorks and other. I do plan to do benchmarks, but I think it would be great to hear from y’all, due to my being so new to Lisp. Would it be considered acceptable to post code fragments to this list to get a second opinion on code that I think should run faster, but doesn’t?

 

Interfacing Lisp with C is something that I’m going to be working on a lot. I’m curious about the overhead involved in calling C functions from Lisp and vice-versa. How does the Lispworks FLI compare to CFFI and others? Which one would y’all use and why? The particulars of Lisp/C and C/Lisp behavior in multi-threaded applications is also an area that I going to need to understand pretty well. The LispWorks docs provide some good information on this, but I trying to distill it down to some simple rules for access between Lisp/C and C/Lisp where threads are involved. Are there any docs/papers around that address this in good detail?

 

Thanks,

Gerry

 

    

 

Dr. David McClain
CTO & Co-Founder
Acudora (TM)
e-mail: david@acudora.com
Tel: (+1) 520-529-2437

 

RE: On the road to replacing C++

Hi Mikel,

This is an excellent point. I've never been able to get into Python. I can get things done, but I don't enjoy the huge boost in productivity that so many folks do. Conversely, I had an incredible experience with Prolog of all things. I don't think I have ever learned a new language so quickly and effortlessly. Do y'all remember when Borland Turbo Prolog came out?  Anyway, I have found that when learning any new language, I need to commit a certain amount of it to memory. The need to constantly look things up slows things down a lot.  


Thanks,
Gerry


-----Original Message-----
From: mikel evins [mailto:mevins@clozure.com] 
Sent: Sunday, February 17, 2013 1:49 PM
To: David McClain
Cc: mikel evins; Gerry Weaver; Paul Billington; lisp-hug@lispworks.com
Subject: Re: On the road to replacing C++


On Feb 17, 2013, at 12:41 PM, David McClain <david@acudora.com> wrote:

> Ditto the comments from Pascal... I coded an elaborate LIFO cached direct file mapping system entirely in Lisp and achieved a 100x speedup in file access for my database system. So the fault is not with Lisp per se. Using Lisp made it possible to write that massive systems programming chore in a few days and have it working properly. 
> 
> Anecdotally, I find that I can write and debug Lisp about 10x faster than I can write C/C++, and one line of Lisp corresponds to about 10-100 lines of C/C++. So I see a productivity increase of around 100x-1000x in just sticking with Lisp as much as possible, and using C/C++ only for a thin veneer of glue routines.
> 
> Regardless of what language you use, you will probably run out of gas at around 100 KLOC. So any major project will self-destruct unless you use a productivity enhancer like Lisp.  See Greenspun's 10th Rule...

I buy your numbers, based on personal experience, but I don't think Lisp works the same way for everyone. I've known smart, productive programmers for whom Lisp was not a productivity enhancer. 

I think that Lisp is natural for some people, and not for others. If you have the sort of mind that benefits from Lisp, then soon after exposure to it you will find it natural and comfortable; if you don't, then that won't happen.

If you have the kind of mind that finds it natural and comfortable, then you will see a large increase in productivity. If you don't, you won't.



_______________________________________________
Lisp Hug - the mailing list for LispWorks users
lisp-hug@lispworks.com
http://www.lispworks.com/support/lisp-hug.html


RE: On the road to replacing C++

Hi Wayne,

 

It’s good to hear that you like the LispWorks rule engine. KnowledgeWorks was one of my main considerations when deciding which version of LispWorks to purchase. I am trying very hard to keep myself from looking at it until I get a little further along with the language though ;-) I haven’t looked yet, but I’m going to be trying to find a good genetic algorithm library at some point as well.

 

Thanks,

Gerry

 

 

From: Wayne Peterson [mailto:wcpeterson@me.com]
Sent: Sunday, February 17, 2013 3:53 PM
To: Gerry Weaver
Subject: Re: On the road to replacing C++

 

Yes, I learned PROLOG before I learned Lisp. I still have a couple of old prolog books and like the prolog that comes with KnowledgeWorks. 

 

In Lisp you get a much richer OO language than C++. You get multiple inheritance, method overrides, mixins and MOP. You also get incremental compilation. I love LispWorks and I am impressed how fast the rule engine is. If I write C++, I generate the C++ code in Lisp. I see Lisp (with PROLOG) and programming with two hands and C++ with one hand tied behind my back. LispWorks also has a profiler which allows you to optimize the bottlenecks (where performance really matters). 

 

Good luck Gerry,

 

Wayne

 

On Feb 17, 2013, at 1:41 PM, Gerry Weaver wrote:




Hi Mikel,

This is an excellent point. I've never been able to get into Python. I can get things done, but I don't enjoy the huge boost in productivity that so many folks do. Conversely, I had an incredible experience with Prolog of all things. I don't think I have ever learned a new language so quickly and effortlessly. Do y'all remember when Borland Turbo Prolog came out?  Anyway, I have found that when learning any new language, I need to commit a certain amount of it to memory. The need to constantly look things up slows things down a lot.  


Thanks,
Gerry


-----Original Message-----
From: mikel evins [mailto:mevins@clozure.com]
Sent: Sunday, February 17, 2013 1:49 PM
To: David McClain
Cc: mikel evins; Gerry Weaver; Paul Billington; lisp-hug@lispworks.com
Subject: Re: On the road to replacing C++


On Feb 17, 2013, at 12:41 PM, David McClain <david@acudora.com> wrote:


Ditto the comments from Pascal... I coded an elaborate LIFO cached direct file mapping system entirely in Lisp and achieved a 100x speedup in file access for my database system. So the fault is not with Lisp per se. Using Lisp made it possible to write that massive systems programming chore in a few days and have it working properly.

 

Anecdotally, I find that I can write and debug Lisp about 10x faster than I can write C/C++, and one line of Lisp corresponds to about 10-100 lines of C/C++. So I see a productivity increase of around 100x-1000x in just sticking with Lisp as much as possible, and using C/C++ only for a thin veneer of glue routines.

 

Regardless of what language you use, you will probably run out of gas at around 100 KLOC. So any major project will self-destruct unless you use a productivity enhancer like Lisp.  See Greenspun's 10th Rule...


I buy your numbers, based on personal experience, but I don't think Lisp works the same way for everyone. I've known smart, productive programmers for whom Lisp was not a productivity enhancer.

I think that Lisp is natural for some people, and not for others. If you have the sort of mind that benefits from Lisp, then soon after exposure to it you will find it natural and comfortable; if you don't, then that won't happen.

If you have the kind of mind that finds it natural and comfortable, then you will see a large increase in productivity. If you don't, you won't.



_______________________________________________
Lisp Hug - the mailing list for LispWorks users
lisp-hug@lispworks.com
http://www.lispworks.com/support/lisp-hug.html

 

RE: On the road to replacing C++

Hi David,

I hear what you're saying. I've never minded doing libraries in C (which I typically use for that). I'm generally accessing the OS/hardware for some purpose, so it makes sense there. Using C for very specific tasks like that is pretty comfortable. My frustration comes when I'm trying to do the application/server/logic layer on top of them. I muttered to myself "there has got to be a better way" many times when doing such work. Up until a few years ago, I worked for large corporations and the politics of introducing something like Lisp would make it quite impossible. I would just trudge forward while promising myself that a better day would come. I actually left that world a few years ago to do my own thing. A few months ago I woke up and realized that the moment that I had been dreaming ab
 out to solve my problem was not just close, it was here and now. I immediately put everything on the slow track in order to find a better language. I adopted a philosophy of "assume nothing!
 ". I didn't want my search to be tainted by anything that I had read or heard previously. I looked at, and worked with, every language I could find regardless of reputation or popularity. I many cases I was surprised by what I found.  I think anyone who is making a career out of software development owes it to themselves to go through that exercise. It is also never too late. I've been programming for over 25 years and my only regret is not having done it sooner. Furthermore, believe me when I tell you that you can't trust everything you read and hear about programming languages. This is particularly true when coming from the so called experts (which IMO, due to the depth of our chosen field, simply can't exist). Selecting a programming language is not unlike shopping for a good chair. No
  other programmer/person can tell you which one is the right one for you. I ultimately settled on Lisp as you already know.

I have built up a rather large collection of C code and libraries over the course of my career, so the use of FLI/CFFI has an added benefit for me. It would allow me to leverage a lot of that code in Lisp. I have played a little bit with FLI, but I definitely intend to spend some time with CFFI. The portability aspect of it would be nice. I guess it will boil down to what the tradeoffs are relative to FLI in the end.

Thanks,
Gerry


-----Original Message-----
From: owner-lisp-hug@lispworks.com [mailto:owner-lisp-hug@lispworks.com] On Behalf Of David McClain
Sent: Sunday, February 17, 2013 8:58 PM
To: lisp-hug@lispworks.com Lispworks
Subject: Re: On the road to replacing C++


.... also, very important to remember the 90/10 rule... 90% of the time is spent in 10% of the code. So profiling might help you locate that 10%, and there I might take the trouble to specify the types of operands in the Lisp code. But that doesn't happen very often for me.

You might gain more by just choosing a different / better data structure, and certainly you gain much much more by using a better algorithm when they exist.

But your mileage may vary. I don't do the kind of work you describe for yourself. 

The productivity gains in Lisp come mostly from the already rich collection of data structures and primitive algorithms that exist in the base Lisp system, then and especially from the fact that you live inside of an incremental and interactive programming environment. A code exploratorium where you can try out any new ideas that you like, without the long disconnected edit / compile / debug cycle offered by most other languages. But this same benefit also existed in Forth.

- DM

Dr. David McClain
CTO & Co-Founder
Acudora (TM)
e-mail: david@acudora.com
Tel: (+1) 520-529-2437




_______________________________________________
Lisp Hug - the mailing list for LispWorks users lisp-hug@lispworks.com http://www.lispworks.com/support/lisp-hug.html


_______________________________________________
Lisp Hug - the mailing list for LispWorks users
lisp-hug@lispworks.com
http://www.lispworks.com/support/lisp-hug.html


RE: On the road to replacing C++

Hi David,

Yes. I think that the portability is must have feature. I'm still trying to decide whether or not I'm going to worry about portability between Lisp implementations. I would guess that I will probably end up just making that call on a per project basis.

 I am quite happy thus far with my choice of LispWorks. I like the idea that they are a commercial company that has a financial incentive to do what they do. Although, I also very much appreciate the good folks in the open source community for the incredible amount of work they do.   

There is only one thing that I wish Lisp had that it doesn't so far. I really like that Erlang actor/message passing thing. After looking at Erlang, that really stuck with me. I would sacrifice Corba in a heartbeat for a Lispy version of that ;-)

Thanks,
Gerry
 

-----Original Message-----
From: David McClain [mailto:david@acudora.com] 
Sent: Sunday, February 17, 2013 11:29 PM
To: lisp-hug@lispworks.com Lispworks
Cc: Gerry Weaver
Subject: Re: On the road to replacing C++

.... of course, that bit about the relative lack of code-rot in Lisp is due in no small part to the efforts of the folks at Lispworks, who are the first line of defense against underlying OS changes. These guys / gals are real heroes in my book, and make my life so much easier.

- DM

On Feb 17, 2013, at 22:11 PM, David McClain <david@acudora.com> wrote:

> 
> Hi Gerry,
> 
> I think you will find working in Lisp for the higher levels very rewarding. I certainly do. As long as your C interfaces are pretty straightforward you will probably find it just as easy in CFFI as with the LW FLI. But the LW FLI offers many more exotic marshaling capabilities that might spoil you... most notably in the passing of ASCII, multibyte, or Unicode strings, and also for more elaborate data structures being passed by value.
> 
> I try not to spend too much time in the C layer, just because of the severe cost in efficiency for me, compared to working in Lisp. And I'm certainly facile in C / C++, having been the creator of their compilers in the past.
> 
> When I first learned SML / OCaml, I was struck by what a royal pain in the a** it was to get it to compile cleanly, but then once a clean compile was obtained, it ran perfectly well for me, with practically zero debugging effort. That was on a problem that we had worked more than a year on in another Fortran-like language (not Python, but nearly as bad...).
> 
> However, a later effort in OCaml showed me that this was an accident -- intermittent positive reinforcement, just like what affects habitual gamblers... The OCaml world is not all it has been cracked up to be. And so I relapsed into Lisp with increased fervor. Now I hardly go a day without pounding out some pretty significant code in Lisp.
> 
> When I began about 40 years ago, the fellow who invented the Forth language (Chuck Moore) was credited with stating that the average significant program took about a year to produce, and he wanted to do 10 per year. I push that further, saying I want to do hundreds per year. And that can only happen with the environment and accumulation of useful tools in Lisp.
> 
> Another area that I have spoken on in the past is the relative freedom from "code-rot" in Lisp. I'm still using modules that I wrote nearly 20 years ago, and have hardly ever needed any updating despite changes to underlying operating systems and glue languages (C++ has evolved horribly over the past 20 years). And my code is immediately cross-platform between Windows and Mac.
> 
> Enjoy!!
> 
> - DM
> 
> 
> On Feb 17, 2013, at 21:18 PM, Gerry Weaver <gerryw@compvia.com> wrote:
> 
>> 
>> Hi David,
>> 
>> I hear what you're saying. I've never minded doing libraries in C (which I typically use for that). I'm generally accessing the OS/hardware for some purpose, so it makes sense there. Using C for very specific tasks like that is pretty comfortable. My frustration comes when I'm trying to do the application/server/logic layer on top of them. I muttered to myself "there has got to be a better way" many times when doing such work. Up until a few years ago, I worked for large corporations and the politics of introducing something like Lisp would make it quite impossible. I would just trudge forward while promising myself that a better day would come. I actually left that world a few years ago to do my own thing. A few months ago I woke up and realized that the moment that I had been dreaming
  about to solve my problem was not just close, it was here and now. I immediately put everything on the slow track in order to find a better language. I adopted a philosophy of "assume noth!
 i!
> ng!
>> ". I didn't want my search to be tainted by anything that I had read or heard previously. I looked at, and worked with, every language I could find regardless of reputation or popularity. I many cases I was surprised by what I found.  I think anyone who is making a career out of software development owes it to themselves to go through that exercise. It is also never too late. I've been programming for over 25 years and my only regret is not having done it sooner. Furthermore, believe me when I tell you that you can't trust everything you read and hear about programming languages. This is particularly true when coming from the so called experts (which IMO, due to the depth of our chosen field, simply can't exist). Selecting a programming language is not unlike shopping for a good chair. 
 No other programmer/person can tell you which one is the right one for you. I ultimately settled on Lisp as you already know.
>> 
>> I have built up a rather large collection of C code and libraries over the course of my career, so the use of FLI/CFFI has an added benefit for me. It would allow me to leverage a lot of that code in Lisp. I have played a little bit with FLI, but I definitely intend to spend some time with CFFI. The portability aspect of it would be nice. I guess it will boil down to what the tradeoffs are relative to FLI in the end.
>> 
>> Thanks,
>> Gerry
>> 
>> 
>> -----Original Message-----
>> From: owner-lisp-hug@lispworks.com 
>> [mailto:owner-lisp-hug@lispworks.com] On Behalf Of David McClain
>> Sent: Sunday, February 17, 2013 8:58 PM
>> To: lisp-hug@lispworks.com Lispworks
>> Subject: Re: On the road to replacing C++
>> 
>> 
>> ... also, very important to remember the 90/10 rule... 90% of the time is spent in 10% of the code. So profiling might help you locate that 10%, and there I might take the trouble to specify the types of operands in the Lisp code. But that doesn't happen very often for me.
>> 
>> You might gain more by just choosing a different / better data structure, and certainly you gain much much more by using a better algorithm when they exist.
>> 
>> But your mileage may vary. I don't do the kind of work you describe for yourself. 
>> 
>> The productivity gains in Lisp come mostly from the already rich collection of data structures and primitive algorithms that exist in the base Lisp system, then and especially from the fact that you live inside of an incremental and interactive programming environment. A code exploratorium where you can try out any new ideas that you like, without the long disconnected edit / compile / debug cycle offered by most other languages. But this same benefit also existed in Forth.
>> 
>> - DM
>> 
>> Dr. David McClain
>> CTO & Co-Founder
>> Acudora (TM)
>> e-mail: david@acudora.com
>> Tel: (+1) 520-529-2437
>> 
>> 
>> 
>> 
>> _______________________________________________
>> Lisp Hug - the mailing list for LispWorks users 
>> lisp-hug@lispworks.com http://www.lispworks.com/support/lisp-hug.html
>> 
>> 
>> _______________________________________________
>> Lisp Hug - the mailing list for LispWorks users 
>> lisp-hug@lispworks.com http://www.lispworks.com/support/lisp-hug.html
>> 
>> 
> 
> Dr. David McClain
> CTO & Co-Founder
> Acudora (TM)
> e-mail: david@acudora.com
> Tel: (+1) 520-529-2437
> 
> 
> 
> 
> _______________________________________________
> Lisp Hug - the mailing list for LispWorks users lisp-hug@lispworks.com 
> http://www.lispworks.com/support/lisp-hug.html
> 
> 

Dr. David McClain
CTO & Co-Founder
Acudora (TM)
e-mail: david@acudora.com
Tel: (+1) 520-529-2437




_______________________________________________
Lisp Hug - the mailing list for LispWorks users
lisp-hug@lispworks.com
http://www.lispworks.com/support/lisp-hug.html


RE: On the road to replacing C++

Hi Pascal,

Oh this is cool! This is very cool! Mailboxes... I remembered seeing them in the API reference. I just didn't realize what they were. Thanks for the pointer!

I also just realized that FLI supports multi-threaded callbacks! Any more of this and I may just have to jump out of my chair a do my happy dance ;-) Alas, I'm beginning to think that LispWorks is going to earn my money on multiple platforms.  

Thanks,
Gerry


-----Original Message-----
From: Pascal Costanza [mailto:pc@p-cos.net] 
Sent: Monday, February 18, 2013 1:15 AM
To: Gerry Weaver
Cc: David McClain; lisp-hug@lispworks.com
Subject: Re: On the road to replacing C++

Hi,

If you are looking for message passing as an implicit synchronization mechanism for shared-memory parallelism, then look no further than LispWorks's mailboxes. They should give you everything that you like about the message passing paradigm, without forcing you into an actor-based programming style (which you can, but do not necessarily have to use in LispWorks as well). In my experience, LispWorks's mailboxes give you all the synchronization you need in 95% of the time.

For distributed programming, I don't have an overview of what the options are. I believe there are some alternatives, available as part of Quicklisp for example, but don't have a good idea what they provide. (I wish there was some well-maintained Lisp interface to MPI. There is CL-MPI, but it looks incomplete to me, unfortunately...)

Pascal

On 18 Feb 2013, at 06:55, Gerry Weaver <gerryw@compvia.com> wrote:

> 
> Hi David,
> 
> Yes. I think that the portability is must have feature. I'm still trying to decide whether or not I'm going to worry about portability between Lisp implementations. I would guess that I will probably end up just making that call on a per project basis.
> 
> I am quite happy thus far with my choice of LispWorks. I like the idea that they are a commercial company that has a financial incentive to do what they do. Although, I also very much appreciate the good folks in the open source community for the incredible amount of work they do.   
> 
> There is only one thing that I wish Lisp had that it doesn't so far. I 
> really like that Erlang actor/message passing thing. After looking at 
> Erlang, that really stuck with me. I would sacrifice Corba in a 
> heartbeat for a Lispy version of that ;-)
> 
> Thanks,
> Gerry
> 
> 
> -----Original Message-----
> From: David McClain [mailto:david@acudora.com]
> Sent: Sunday, February 17, 2013 11:29 PM
> To: lisp-hug@lispworks.com Lispworks
> Cc: Gerry Weaver
> Subject: Re: On the road to replacing C++
> 
> ... of course, that bit about the relative lack of code-rot in Lisp is due in no small part to the efforts of the folks at Lispworks, who are the first line of defense against underlying OS changes. These guys / gals are real heroes in my book, and make my life so much easier.
> 
> - DM
> 
> On Feb 17, 2013, at 22:11 PM, David McClain <david@acudora.com> wrote:
> 
>> 
>> Hi Gerry,
>> 
>> I think you will find working in Lisp for the higher levels very rewarding. I certainly do. As long as your C interfaces are pretty straightforward you will probably find it just as easy in CFFI as with the LW FLI. But the LW FLI offers many more exotic marshaling capabilities that might spoil you... most notably in the passing of ASCII, multibyte, or Unicode strings, and also for more elaborate data structures being passed by value.
>> 
>> I try not to spend too much time in the C layer, just because of the severe cost in efficiency for me, compared to working in Lisp. And I'm certainly facile in C / C++, having been the creator of their compilers in the past.
>> 
>> When I first learned SML / OCaml, I was struck by what a royal pain in the a** it was to get it to compile cleanly, but then once a clean compile was obtained, it ran perfectly well for me, with practically zero debugging effort. That was on a problem that we had worked more than a year on in another Fortran-like language (not Python, but nearly as bad...).
>> 
>> However, a later effort in OCaml showed me that this was an accident -- intermittent positive reinforcement, just like what affects habitual gamblers... The OCaml world is not all it has been cracked up to be. And so I relapsed into Lisp with increased fervor. Now I hardly go a day without pounding out some pretty significant code in Lisp.
>> 
>> When I began about 40 years ago, the fellow who invented the Forth language (Chuck Moore) was credited with stating that the average significant program took about a year to produce, and he wanted to do 10 per year. I push that further, saying I want to do hundreds per year. And that can only happen with the environment and accumulation of useful tools in Lisp.
>> 
>> Another area that I have spoken on in the past is the relative freedom from "code-rot" in Lisp. I'm still using modules that I wrote nearly 20 years ago, and have hardly ever needed any updating despite changes to underlying operating systems and glue languages (C++ has evolved horribly over the past 20 years). And my code is immediately cross-platform between Windows and Mac.
>> 
>> Enjoy!!
>> 
>> - DM
>> 
>> 
>> On Feb 17, 2013, at 21:18 PM, Gerry Weaver <gerryw@compvia.com> wrote:
>> 
>>> 
>>> Hi David,
>>> 
>>> I hear what you're saying. I've never minded doing libraries in C (which I typically use for that). I'm generally accessing the OS/hardware for some purpose, so it makes sense there. Using C for very specific tasks like that is pretty comfortable. My frustration comes when I'm trying to do the application/server/logic layer on top of them. I muttered to myself "there has got to be a better way" many times when doing such work. Up until a few years ago, I worked for large corporations and the politics of introducing something like Lisp would make it quite impossible. I would just trudge forward while promising myself that a better day would come. I actually left that world a few years ago to do my own thing. A few months ago I woke up and realized that the moment that I had been dreamin
 g about to solve my problem was not just close, it was here and now. I immediately put everything on the slow track in order to find a better language. I adopted a philosophy of "assume not!
 h!
> i!
>> ng!
>>> ". I didn't want my search to be tainted by anything that I had read or heard previously. I looked at, and worked with, every language I could find regardless of reputation or popularity. I many cases I was surprised by what I found.  I think anyone who is making a career out of software development owes it to themselves to go through that exercise. It is also never too late. I've been programming for over 25 years and my only regret is not having done it sooner. Furthermore, believe me when I tell you that you can't trust everything you read and hear about programming languages. This is particularly true when coming from the so called experts (which IMO, due to the depth of our chosen field, simply can't exist). Selecting a programming language is not unlike shopping for a good chair.
  No other programmer/person can tell you which one is the right one for you. I ultimately settled on Lisp as you already know.
>>> 
>>> I have built up a rather large collection of C code and libraries over the course of my career, so the use of FLI/CFFI has an added benefit for me. It would allow me to leverage a lot of that code in Lisp. I have played a little bit with FLI, but I definitely intend to spend some time with CFFI. The portability aspect of it would be nice. I guess it will boil down to what the tradeoffs are relative to FLI in the end.
>>> 
>>> Thanks,
>>> Gerry
>>> 
>>> 
>>> -----Original Message-----
>>> From: owner-lisp-hug@lispworks.com
>>> [mailto:owner-lisp-hug@lispworks.com] On Behalf Of David McClain
>>> Sent: Sunday, February 17, 2013 8:58 PM
>>> To: lisp-hug@lispworks.com Lispworks
>>> Subject: Re: On the road to replacing C++
>>> 
>>> 
>>> ... also, very important to remember the 90/10 rule... 90% of the time is spent in 10% of the code. So profiling might help you locate that 10%, and there I might take the trouble to specify the types of operands in the Lisp code. But that doesn't happen very often for me.
>>> 
>>> You might gain more by just choosing a different / better data structure, and certainly you gain much much more by using a better algorithm when they exist.
>>> 
>>> But your mileage may vary. I don't do the kind of work you describe for yourself. 
>>> 
>>> The productivity gains in Lisp come mostly from the already rich collection of data structures and primitive algorithms that exist in the base Lisp system, then and especially from the fact that you live inside of an incremental and interactive programming environment. A code exploratorium where you can try out any new ideas that you like, without the long disconnected edit / compile / debug cycle offered by most other languages. But this same benefit also existed in Forth.
>>> 
>>> - DM
>>> 
>>> Dr. David McClain
>>> CTO & Co-Founder
>>> Acudora (TM)
>>> e-mail: david@acudora.com
>>> Tel: (+1) 520-529-2437
>>> 
>>> 
>>> 
>>> 
>>> _______________________________________________
>>> Lisp Hug - the mailing list for LispWorks users 
>>> lisp-hug@lispworks.com 
>>> http://www.lispworks.com/support/lisp-hug.html
>>> 
>>> 
>>> _______________________________________________
>>> Lisp Hug - the mailing list for LispWorks users 
>>> lisp-hug@lispworks.com 
>>> http://www.lispworks.com/support/lisp-hug.html
>>> 
>>> 
>> 
>> Dr. David McClain
>> CTO & Co-Founder
>> Acudora (TM)
>> e-mail: david@acudora.com
>> Tel: (+1) 520-529-2437
>> 
>> 
>> 
>> 
>> _______________________________________________
>> Lisp Hug - the mailing list for LispWorks users 
>> lisp-hug@lispworks.com http://www.lispworks.com/support/lisp-hug.html
>> 
>> 
> 
> Dr. David McClain
> CTO & Co-Founder
> Acudora (TM)
> e-mail: david@acudora.com
> Tel: (+1) 520-529-2437
> 
> 
> 
> 
> _______________________________________________
> Lisp Hug - the mailing list for LispWorks users lisp-hug@lispworks.com 
> http://www.lispworks.com/support/lisp-hug.html
> 

--
Pascal Costanza




_______________________________________________
Lisp Hug - the mailing list for LispWorks users
lisp-hug@lispworks.com
http://www.lispworks.com/support/lisp-hug.html


RE: On the road to replacing C++

Hi Pascal,

Sorry. I meant to mention that there is a Lisp binding to 0MQ. I believe it also supports LispWorks. Perhaps one could use that in conjunction with the mailboxes. It's just an idea. I don't know if it would actually work well.

Thanks,
Gerry



-----Original Message-----
From: Pascal Costanza [mailto:pc@p-cos.net] 
Sent: Monday, February 18, 2013 1:15 AM
To: Gerry Weaver
Cc: David McClain; lisp-hug@lispworks.com
Subject: Re: On the road to replacing C++

Hi,

If you are looking for message passing as an implicit synchronization mechanism for shared-memory parallelism, then look no further than LispWorks's mailboxes. They should give you everything that you like about the message passing paradigm, without forcing you into an actor-based programming style (which you can, but do not necessarily have to use in LispWorks as well). In my experience, LispWorks's mailboxes give you all the synchronization you need in 95% of the time.

For distributed programming, I don't have an overview of what the options are. I believe there are some alternatives, available as part of Quicklisp for example, but don't have a good idea what they provide. (I wish there was some well-maintained Lisp interface to MPI. There is CL-MPI, but it looks incomplete to me, unfortunately...)

Pascal

On 18 Feb 2013, at 06:55, Gerry Weaver <gerryw@compvia.com> wrote:

> 
> Hi David,
> 
> Yes. I think that the portability is must have feature. I'm still trying to decide whether or not I'm going to worry about portability between Lisp implementations. I would guess that I will probably end up just making that call on a per project basis.
> 
> I am quite happy thus far with my choice of LispWorks. I like the idea that they are a commercial company that has a financial incentive to do what they do. Although, I also very much appreciate the good folks in the open source community for the incredible amount of work they do.   
> 
> There is only one thing that I wish Lisp had that it doesn't so far. I 
> really like that Erlang actor/message passing thing. After looking at 
> Erlang, that really stuck with me. I would sacrifice Corba in a 
> heartbeat for a Lispy version of that ;-)
> 
> Thanks,
> Gerry
> 
> 
> -----Original Message-----
> From: David McClain [mailto:david@acudora.com]
> Sent: Sunday, February 17, 2013 11:29 PM
> To: lisp-hug@lispworks.com Lispworks
> Cc: Gerry Weaver
> Subject: Re: On the road to replacing C++
> 
> ... of course, that bit about the relative lack of code-rot in Lisp is due in no small part to the efforts of the folks at Lispworks, who are the first line of defense against underlying OS changes. These guys / gals are real heroes in my book, and make my life so much easier.
> 
> - DM
> 
> On Feb 17, 2013, at 22:11 PM, David McClain <david@acudora.com> wrote:
> 
>> 
>> Hi Gerry,
>> 
>> I think you will find working in Lisp for the higher levels very rewarding. I certainly do. As long as your C interfaces are pretty straightforward you will probably find it just as easy in CFFI as with the LW FLI. But the LW FLI offers many more exotic marshaling capabilities that might spoil you... most notably in the passing of ASCII, multibyte, or Unicode strings, and also for more elaborate data structures being passed by value.
>> 
>> I try not to spend too much time in the C layer, just because of the severe cost in efficiency for me, compared to working in Lisp. And I'm certainly facile in C / C++, having been the creator of their compilers in the past.
>> 
>> When I first learned SML / OCaml, I was struck by what a royal pain in the a** it was to get it to compile cleanly, but then once a clean compile was obtained, it ran perfectly well for me, with practically zero debugging effort. That was on a problem that we had worked more than a year on in another Fortran-like language (not Python, but nearly as bad...).
>> 
>> However, a later effort in OCaml showed me that this was an accident -- intermittent positive reinforcement, just like what affects habitual gamblers... The OCaml world is not all it has been cracked up to be. And so I relapsed into Lisp with increased fervor. Now I hardly go a day without pounding out some pretty significant code in Lisp.
>> 
>> When I began about 40 years ago, the fellow who invented the Forth language (Chuck Moore) was credited with stating that the average significant program took about a year to produce, and he wanted to do 10 per year. I push that further, saying I want to do hundreds per year. And that can only happen with the environment and accumulation of useful tools in Lisp.
>> 
>> Another area that I have spoken on in the past is the relative freedom from "code-rot" in Lisp. I'm still using modules that I wrote nearly 20 years ago, and have hardly ever needed any updating despite changes to underlying operating systems and glue languages (C++ has evolved horribly over the past 20 years). And my code is immediately cross-platform between Windows and Mac.
>> 
>> Enjoy!!
>> 
>> - DM
>> 
>> 
>> On Feb 17, 2013, at 21:18 PM, Gerry Weaver <gerryw@compvia.com> wrote:
>> 
>>> 
>>> Hi David,
>>> 
>>> I hear what you're saying. I've never minded doing libraries in C (which I typically use for that). I'm generally accessing the OS/hardware for some purpose, so it makes sense there. Using C for very specific tasks like that is pretty comfortable. My frustration comes when I'm trying to do the application/server/logic layer on top of them. I muttered to myself "there has got to be a better way" many times when doing such work. Up until a few years ago, I worked for large corporations and the politics of introducing something like Lisp would make it quite impossible. I would just trudge forward while promising myself that a better day would come. I actually left that world a few years ago to do my own thing. A few months ago I woke up and realized that the moment that I had been dreamin
 g about to solve my problem was not just close, it was here and now. I immediately put everything on the slow track in order to find a better language. I adopted a philosophy of "assume not!
 h!
> i!
>> ng!
>>> ". I didn't want my search to be tainted by anything that I had read or heard previously. I looked at, and worked with, every language I could find regardless of reputation or popularity. I many cases I was surprised by what I found.  I think anyone who is making a career out of software development owes it to themselves to go through that exercise. It is also never too late. I've been programming for over 25 years and my only regret is not having done it sooner. Furthermore, believe me when I tell you that you can't trust everything you read and hear about programming languages. This is particularly true when coming from the so called experts (which IMO, due to the depth of our chosen field, simply can't exist). Selecting a programming language is not unlike shopping for a good chair.
  No other programmer/person can tell you which one is the right one for you. I ultimately settled on Lisp as you already know.
>>> 
>>> I have built up a rather large collection of C code and libraries over the course of my career, so the use of FLI/CFFI has an added benefit for me. It would allow me to leverage a lot of that code in Lisp. I have played a little bit with FLI, but I definitely intend to spend some time with CFFI. The portability aspect of it would be nice. I guess it will boil down to what the tradeoffs are relative to FLI in the end.
>>> 
>>> Thanks,
>>> Gerry
>>> 
>>> 
>>> -----Original Message-----
>>> From: owner-lisp-hug@lispworks.com
>>> [mailto:owner-lisp-hug@lispworks.com] On Behalf Of David McClain
>>> Sent: Sunday, February 17, 2013 8:58 PM
>>> To: lisp-hug@lispworks.com Lispworks
>>> Subject: Re: On the road to replacing C++
>>> 
>>> 
>>> ... also, very important to remember the 90/10 rule... 90% of the time is spent in 10% of the code. So profiling might help you locate that 10%, and there I might take the trouble to specify the types of operands in the Lisp code. But that doesn't happen very often for me.
>>> 
>>> You might gain more by just choosing a different / better data structure, and certainly you gain much much more by using a better algorithm when they exist.
>>> 
>>> But your mileage may vary. I don't do the kind of work you describe for yourself. 
>>> 
>>> The productivity gains in Lisp come mostly from the already rich collection of data structures and primitive algorithms that exist in the base Lisp system, then and especially from the fact that you live inside of an incremental and interactive programming environment. A code exploratorium where you can try out any new ideas that you like, without the long disconnected edit / compile / debug cycle offered by most other languages. But this same benefit also existed in Forth.
>>> 
>>> - DM
>>> 
>>> Dr. David McClain
>>> CTO & Co-Founder
>>> Acudora (TM)
>>> e-mail: david@acudora.com
>>> Tel: (+1) 520-529-2437
>>> 
>>> 
>>> 
>>> 
>>> _______________________________________________
>>> Lisp Hug - the mailing list for LispWorks users 
>>> lisp-hug@lispworks.com 
>>> http://www.lispworks.com/support/lisp-hug.html
>>> 
>>> 
>>> _______________________________________________
>>> Lisp Hug - the mailing list for LispWorks users 
>>> lisp-hug@lispworks.com 
>>> http://www.lispworks.com/support/lisp-hug.html
>>> 
>>> 
>> 
>> Dr. David McClain
>> CTO & Co-Founder
>> Acudora (TM)
>> e-mail: david@acudora.com
>> Tel: (+1) 520-529-2437
>> 
>> 
>> 
>> 
>> _______________________________________________
>> Lisp Hug - the mailing list for LispWorks users 
>> lisp-hug@lispworks.com http://www.lispworks.com/support/lisp-hug.html
>> 
>> 
> 
> Dr. David McClain
> CTO & Co-Founder
> Acudora (TM)
> e-mail: david@acudora.com
> Tel: (+1) 520-529-2437
> 
> 
> 
> 
> _______________________________________________
> Lisp Hug - the mailing list for LispWorks users lisp-hug@lispworks.com 
> http://www.lispworks.com/support/lisp-hug.html
> 

--
Pascal Costanza




_______________________________________________
Lisp Hug - the mailing list for LispWorks users
lisp-hug@lispworks.com
http://www.lispworks.com/support/lisp-hug.html


Re: On the road to replacing C++

Gerry Weaver <gerryw@compvia.com> writes:

> I have built up a rather large collection of C code and libraries over
> the course of my career, […]

I had also a few personal C libraries.   

When I switched to CL, I rewrote them in lisp.  But only 1/10 of them
were still needed in CL (the rest being covered by the CL standard
library), and of those that remained, the resulting lisp LoC was 1/10 to
1/30 the C LoC.

I would advise you to try and start rewriting your libraries in lisp.
You might be quite edified.


-- 
__Pascal Bourguignon__                     http://www.informatimago.com/
A bad day in () is better than a good day in {}.

_______________________________________________
Lisp Hug - the mailing list for LispWorks users
lisp-hug@lispworks.com
http://www.lispworks.com/support/lisp-hug.html


Re: On the road to replacing C++

You know I am completely fascinated by the different reason people like Lisp. I would not have the slightest idea what LEX and YACC is. Its not my background  But it was with Lisp that I first learned object-oriented programming. It was FLAVORS which came with the Symbolics computer. The entire operating system (later called Genera) was written in FLAVORS. Later they combined with LOOPS by Zerox to form CLOS. When Booch and Rumbaugh came along with Object-Oriented Design and Analysis I was quite disappointed because they left out so many of the key features of OO as I knew them. The first thing I liked about Lisp is  (/ 4 6) gives you 3/2 not 1.5. I guess having a math background I prefer leaving fractions as fractions. Then of course it was the list structures that allowed one to nest list inside of list. That was great for working with networks of nodes and segments. But LispWorks goes way beyond Lisp in general. With Prolog, one can even factor algebraic expression or solve differential equations. In fact PROLOG is still used today by the department of Homeland Security because it is easy to extend queries into rules. Then they add in forward chaining rules with KnowledgeWorks and you have a great mixed chaining of logic. So I can't think of a better language for exploring AI. It seems the breadth of features is what has the appeal.

The only reason I think it hasn't had a broader appeal is that it was faster on Lisp-based computers like Symbolics, and did not run nearly as fast on Sun or on mini-computers. But I think with todays computers that is really not as much of a big deal. All I can say as someone who has worked with Ilog JRules, the dominant rule engine for about 15 years, Knowledgeworks just blows it away. And at a far, far cheaper price. Actually I can think of another reason. IT in general seems to follow the lemming theory. It follows the herd, even if that means going over the cliff.

Love this discussion, as too few people really program in Lisp.
BTW: I am willing to bet I having even used most of the features of Lisp. It's a language I will never master. And I think that is a good thing.

Wayne


On Feb 18, 2013, at 12:12 PM, Paul Tarvydas wrote:


Allow me to paraphrase David's, et al's, comments in as terse and controversial manner as possible:

You don't yet know how much you're going to like CL.

CL has LEX and YACC built into it.

CL doesn't force premature typing.[1]

CL doesn't force premature choice of paradigm.

Every new software project - by definition[2] - involves design and research.

To have free reign over a new design, one needs all of the above. CL is a language that supports all of the above ; per Greenspun's 10th, all other languages don't.

In contrast, premature (aka strong / static) typing forces one to make (bad / incomplete) decisions too early in the design cycle, which come back to bite later.  Same with premature choice of paradigms, or worse, a single choice of paradigm.

Erlang: see, also, FBP (flow based programming)[3][4].

You might find Jack Harper's ECLM talk http://blip.tv/eclm/episode-5741240 to be reassuring.

As to David's point about design / algorithmic-changes for enhanced efficiency, Doug Hoyte[5] argues that web servers should not use threads[6] and built Antiweb in CL http://hoytech.com/antiweb/manual.awp/design.html .

The only drawback, IMO, of CL is that it doesn't have a formalized method for parsing diagrammatic syntax[7].

pt


[1] I used to be a typle thumper (e.g. I dropped C for Concurrent Euclid, Eiffel, etc).
[2] If a software project does not require design and research, the code for it already exists and can be merely copied.
[3] http://www.jpaulmorrison.com/fbp/ and https://groups.google.com/forum/?hl=en&fromgroups#!forum/flow-based-programming
[4] visualframeworksinc.com
[5] author of "Let Over Lambda"
[6] well, exactly 1 thread.
[7] Some stupid pet tricks and 800 lines of code can give you a diagram compiler, though, e.g. https://github.com/guitarvydas/vsh , now we just need some visual reader macros...

_______________________________________________
Lisp Hug - the mailing list for LispWorks users
lisp-hug@lispworks.com
http://www.lispworks.com/support/lisp-hug.html


RE: On the road to replacing C++

Hi Paul,

 

I too am enjoying this discussion immensely. It is so cool to hear about what Lisp means to different people. I guess I never really said what it was about Lisp that made me choose it over the other languages. When I looked at Lisp code for the first time, I was struck almost immediately by two things. How easy it would be to parse and how the code structure reminded me of XML. I even did a little playing around by translating an XML file to Lisp. Beyond this, it is definitely the fact that Lisp supports multiple paradigms. I have learned that certain programming paradigms lend themselves better than others to solving certain kinds of problems. I feel like Lisp is kind of the Swiss Army Knife of programming languages. If you add the fact that you get the benefit of a high level language at potentially C/C++ performance, than what’s not to like?   Then there is LispWorks. What can I say? I mean talk about batteries included! I would imagine I will still be finding and learning new things every day for a very long time. It makes me kind of sad when I think about the fact that Lisp never really got to shine the way it should have. I guess that was largely because of its association with the “AI winter”. I have read that some folks find Lisp difficult to learn. Thankfully, I don’t seem to be one of them. I am finding the learning experience to be fairly quick paced at this point. Then again, maybe I’m just weird. I actually kind of liked the whole () thing from the beginning. I’m afraid I’m well past the point of no return ;-)

 

I’d like to take a moment to thank everyone for taking the time to answer my questions. Your time and patience for a newbie is very much appreciated. I don’t remember the last time I had such an enjoyable and enlightening conversation.  Once I get to the point of being able to help others like myself, I will definitely hang around and try to give something back.

 

BTW: I have been having a little trouble sleeping at night. I keep finding things that just make my head spin with possibilities. I don’t seem to be able to turn it off! Does it ever stop?

 

Thanks,

Gerry

 

 

From: owner-lisp-hug@lispworks.com [mailto:owner-lisp-hug@lispworks.com] On Behalf Of Wayne Peterson
Sent: Monday, February 18, 2013 2:58 PM
To: Paul Tarvydas
Cc: lisp-hug@lispworks.com
Subject: Re: On the road to replacing C++

 

You know I am completely fascinated by the different reason people like Lisp. I would not have the slightest idea what LEX and YACC is. Its not my background  But it was with Lisp that I first learned object-oriented programming. It was FLAVORS which came with the Symbolics computer. The entire operating system (later called Genera) was written in FLAVORS. Later they combined with LOOPS by Zerox to form CLOS. When Booch and Rumbaugh came along with Object-Oriented Design and Analysis I was quite disappointed because they left out so many of the key features of OO as I knew them. The first thing I liked about Lisp is  (/ 4 6) gives you 3/2 not 1.5. I guess having a math background I prefer leaving fractions as fractions. Then of course it was the list structures that allowed one to nest list inside of list. That was great for working with networks of nodes and segments. But LispWorks goes way beyond Lisp in general. With Prolog, one can even factor algebraic expression or solve differential equations. In fact PROLOG is still used today by the department of Homeland Security because it is easy to extend queries into rules. Then they add in forward chaining rules with KnowledgeWorks and you have a great mixed chaining of logic. So I can't think of a better language for exploring AI. It seems the breadth of features is what has the appeal.

 

The only reason I think it hasn't had a broader appeal is that it was faster on Lisp-based computers like Symbolics, and did not run nearly as fast on Sun or on mini-computers. But I think with todays computers that is really not as much of a big deal. All I can say as someone who has worked with Ilog JRules, the dominant rule engine for about 15 years, Knowledgeworks just blows it away. And at a far, far cheaper price. Actually I can think of another reason.. IT in general seems to follow the lemming theory. It follows the herd, even if that means going over the cliff.

 

Love this discussion, as too few people really program in Lisp.

BTW: I am willing to bet I having even used most of the features of Lisp. It's a language I will never master. And I think that is a good thing.

 

Wayne

 

 

On Feb 18, 2013, at 12:12 PM, Paul Tarvydas wrote:




Allow me to paraphrase David's, et al's, comments in as terse and controversial manner as possible:

You don't yet know how much you're going to like CL.

CL has LEX and YACC built into it.

CL doesn't force premature typing.[1]

CL doesn't force premature choice of paradigm.

Every new software project - by definition[2] - involves design and research.

To have free reign over a new design, one needs all of the above. CL is a language that supports all of the above ; per Greenspun's 10th, all other languages don't.

In contrast, premature (aka strong / static) typing forces one to make (bad / incomplete) decisions too early in the design cycle, which come back to bite later.  Same with premature choice of paradigms, or worse, a single choice of paradigm.

Erlang: see, also, FBP (flow based programming)[3][4].

You might find Jack Harper's ECLM talk http://blip.tv/eclm/episode-5741240 to be reassuring.

As to David's point about design / algorithmic-changes for enhanced efficiency, Doug Hoyte[5] argues that web servers should not use threads[6] and built Antiweb in CL http://hoytech.com/antiweb/manual.awp/design.html .

The only drawback, IMO, of CL is that it doesn't have a formalized method for parsing diagrammatic syntax[7].

pt


[1] I used to be a typle thumper (e.g. I dropped C for Concurrent Euclid, Eiffel, etc).
[2] If a software project does not require design and research, the code for it already exists and can be merely copied.
[3] http://www.jpaulmorrison.com/fbp/ and https://groups.google.com/forum/?hl=en&fromgroups#!forum/flow-based-programming
[4] visualframeworksinc.com
[5] author of "Let Over Lambda"
[6] well, exactly 1 thread.
[7] Some stupid pet tricks and 800 lines of code can give you a diagram compiler, though, e.g. https://github.com/guitarvydas/vsh , now we just need some visual reader macros...

_______________________________________________
Lisp Hug - the mailing list for LispWorks users
lisp-hug@lispworks.com
http://www.lispworks.com/support/lisp-hug.html

 

RE: On the road to replacing C++

Hi Wayne, Paul,

 

Sorry. I got things mixed up.

 

Thanks,

Gerry

 

From: owner-lisp-hug@lispworks.com [mailto:owner-lisp-hug@lispworks.com] On Behalf Of Gerry Weaver
Sent: Monday, February 18, 2013 4:22 PM
To: Wayne Peterson
Cc: lisp-hug@lispworks.com
Subject: RE: On the road to replacing C++

 

Hi Paul,

 

I too am enjoying this discussion immensely. It is so cool to hear about what Lisp means to different people. I guess I never really said what it was about Lisp that made me choose it over the other languages. When I looked at Lisp code for the first time, I was struck almost immediately by two things. How easy it would be to parse and how the code structure reminded me of XML. I even did a little playing around by translating an XML file to Lisp. Beyond this, it is definitely the fact that Lisp supports multiple paradigms. I have learned that certain programming paradigms lend themselves better than others to solving certain kinds of problems. I feel like Lisp is kind of the Swiss Army Knife of programming languages. If you add the fact that you get the benefit of a high level language at potentially C/C++ performance, than what’s not to like?   Then there is LispWorks. What can I say? I mean talk about batteries included! I would imagine I will still be finding and learning new things every day for a very long time. It makes me kind of sad when I think about the fact that Lisp never really got to shine the way it should have. I guess that was largely because of its association with the “AI winter”. I have read that some folks find Lisp difficult to learn. Thankfully, I don’t seem to be one of them. I am finding the learning experience to be fairly quick paced at this point. Then again, maybe I’m just weird. I actually kind of liked the whole () thing from the beginning. I’m afraid I’m well past the point of no return ;-)

 

I’d like to take a moment to thank everyone for taking the time to answer my questions. Your time and patience for a newbie is very much appreciated. I don’t remember the last time I had such an enjoyable and enlightening conversation.  Once I get to the point of being able to help others like myself, I will definitely hang around and try to give something back.

 

BTW: I have been having a little trouble sleeping at night. I keep finding things that just make my head spin with possibilities. I don’t seem to be able to turn it off! Does it ever stop?

 

Thanks,

Gerry

 

 

From: owner-lisp-hug@lispworks.com [mailto:owner-lisp-hug@lispworks.com] On Behalf Of Wayne Peterson
Sent: Monday, February 18, 2013 2:58 PM
To: Paul Tarvydas
Cc: lisp-hug@lispworks.com
Subject: Re: On the road to replacing C++

 

You know I am completely fascinated by the different reason people like Lisp. I would not have the slightest idea what LEX and YACC is. Its not my background  But it was with Lisp that I first learned object-oriented programming. It was FLAVORS which came with the Symbolics computer. The entire operating system (later called Genera) was written in FLAVORS. Later they combined with LOOPS by Zerox to form CLOS. When Booch and Rumbaugh came along with Object-Oriented Design and Analysis I was quite disappointed because they left out so many of the key features of OO as I knew them. The first thing I liked about Lisp is  (/ 4 6) gives you 3/2 not 1.5. I guess having a math background I prefer leaving fractions as fractions. Then of course it was the list structures that allowed one to nest list inside of list. That was great for working with networks of nodes and segments. But LispWorks goes way beyond Lisp in general. With Prolog, one can even factor algebraic expression or solve differential equations. In fact PROLOG is still used today by the department of Homeland Security because it is easy to extend queries into rules. Then they add in forward chaining rules with KnowledgeWorks and you have a great mixed chaining of logic. So I can't think of a better language for exploring AI. It seems the breadth of features is what has the appeal.

 

The only reason I think it hasn't had a broader appeal is that it was faster on Lisp-based computers like Symbolics, and did not run nearly as fast on Sun or on mini-computers. But I think with todays computers that is really not as much of a big deal. All I can say as someone who has worked with Ilog JRules, the dominant rule engine for about 15 years, Knowledgeworks just blows it away. And at a far, far cheaper price. Actually I can think of another reason.. IT in general seems to follow the lemming theory. It follows the herd, even if that means going over the cliff.

 

Love this discussion, as too few people really program in Lisp.

BTW: I am willing to bet I having even used most of the features of Lisp. It's a language I will never master. And I think that is a good thing.

 

Wayne

 

 

On Feb 18, 2013, at 12:12 PM, Paul Tarvydas wrote:

 


Allow me to paraphrase David's, et al's, comments in as terse and controversial manner as possible:

You don't yet know how much you're going to like CL.

CL has LEX and YACC built into it.

CL doesn't force premature typing.[1]

CL doesn't force premature choice of paradigm.

Every new software project - by definition[2] - involves design and research.

To have free reign over a new design, one needs all of the above. CL is a language that supports all of the above ; per Greenspun's 10th, all other languages don't.

In contrast, premature (aka strong / static) typing forces one to make (bad / incomplete) decisions too early in the design cycle, which come back to bite later.  Same with premature choice of paradigms, or worse, a single choice of paradigm.

Erlang: see, also, FBP (flow based programming)[3][4].

You might find Jack Harper's ECLM talk http://blip.tv/eclm/episode-5741240 to be reassuring.

As to David's point about design / algorithmic-changes for enhanced efficiency, Doug Hoyte[5] argues that web servers should not use threads[6] and built Antiweb in CL http://hoytech.com/antiweb/manual.awp/design.html .

The only drawback, IMO, of CL is that it doesn't have a formalized method for parsing diagrammatic syntax[7].

pt


[1] I used to be a typle thumper (e.g. I dropped C for Concurrent Euclid, Eiffel, etc).
[2] If a software project does not require design and research, the code for it already exists and can be merely copied.
[3] http://www.jpaulmorrison.com/fbp/ and https://groups.google.com/forum/?hl=en&fromgroups#!forum/flow-based-programming
[4] visualframeworksinc.com
[5] author of "Let Over Lambda"
[6] well, exactly 1 thread.
[7] Some stupid pet tricks and 800 lines of code can give you a diagram compiler, though, e.g. https://github.com/guitarvydas/vsh , now we just need some visual reader macros...

_______________________________________________
Lisp Hug - the mailing list for LispWorks users
lisp-hug@lispworks.com
http://www.lispworks.com/support/lisp-hug.html

 

RE: On the road to replacing C++

Hi David,

 

This reminds of something that I was thinking about a while ago. It would be interesting to have the capability to use Lisp as a scripting extension to itself. I guess some kind of sandboxed interpreter. Perhaps there could be some mechanism for registering the functions that a script can call. Lisp could then be used for configuration files and application user extensions. I have several C++ applications that make use of something like this. They use a library called “AngelScript”. Has anyone ever looked at doing something like this for Lisp?

 

Thanks,

Gerry

 

 

From: owner-lisp-hug@lispworks.com [mailto:owner-lisp-hug@lispworks.com] On Behalf Of David McClain
Sent: Monday, February 18, 2013 3:51 PM
To: lisp-hug@lispworks.com Lispworks
Subject: Re: On the road to replacing C++

 

 

On Feb 18, 2013, at 13:58 PM, Wayne Peterson <wcpeterson@me.com> wrote:

Love this discussion, as too few people really program in Lisp.

BTW: I am willing to bet I having even used most of the features of Lisp. It's a language I will never master. And I think that is a good thing.

 

Wayne

 

I had to confront an old wive's tale the other night, about *NEVER USE EVAL!!!!*  Well, there really are some places that have legitimate uses, and I had to dig to find them.

 

But meanwhile, I had an expression like '(* 15 (1+ (* 32 (1+ (* 43 .... ))) that I wanted both to take apart to some level, and to evaluate the tail of the remaining portion after the take-apart.. How to do this without using EVAL??

 

So I came up with this: 

 

(eval e)  ==>  (funcall (compile nil `(lambda () ,e))))

 

Eh? Anyone else have a better idea? Has to evaluate in the current dynamic environment and the null lexical environment, just like EVAL.

 

(Sometimes Lisp presents some real brain-twisters...)

 

- DM

 

 

RE: On the road to replacing C++

Hi Tim,

 

Thanks for the info. This sounds like something that I’m not going to be ready to tackle for a while. I guess a more ham fisted approach would be to do an FLI binding to one of the lighter weight embeddable Lisp interpeters. I may have to give that some thought.

 

Thanks,

Gerry

 

 

From: Tim Bradshaw [mailto:tfb@cley.com]
Sent: Monday, February 18, 2013 6:51 PM
To: Gerry Weaver
Cc: David McClain; lisp-hug@lispworks.com
Subject: Re: On the road to replacing C++

 

On 18 Feb 2013, at 22:42, Gerry Weaver wrote:



This reminds of something that I was thinking about a while ago. It would be interesting to have the capability to use Lisp as a scripting extension to itself. I guess some kind of sandboxed interpreter. Perhaps there could be some mechanism for registering the functions that a script can call.

 

I did some stuff around this a fairly long time ago.  The trick was essentially to define a "safe language" which is some subset of CL (or possibly a variant lisp built on CL, it does not matter).  That "safe language" is defined as consisting of a bunch of symbols which you then arrange in a package. You then read the form you want to evaluate in a package which has USEd the safe lisp package (so basically a user package) and which has no internal symbols initially.  You use very safe and controlled reader settings (and you are completely at the mercy of the implementation's reader here). At the end of this you have an object which you walk, checking it for toxins, where a toxin includes at least any symbol which is not directly present in either the user package or the safe package (you can hack keywords into the safe language, but this is fairly fiddly to get right)  If it passes then you evaluate the resulting thing.

 

This is missing various things: in particular you can't tell from the form if the thing really is safe, unless the language is very restrictive indeed.  And you also have the problem that the form can name symbols which you *don't* allow but which have already leaked into packages you don't want it to see (for instance the keyword package).  So although this can't be used to communicate between evaluations by an attacker it can cause leaks.

 

I had an implementation of this, but it was not really heavy duty, and I am not sure if the approach is viable at all any more.

 

To answer another message: ARP spoofing is only an interesting attack if you do not trust your layer 2 broadcast domain (ie an attacker can physically attach a machine to it).  If you don't, you probably have already given up any serious pretence of security: there is a lot to be said for locking the doors to your datacentre.

Re: On the road to replacing C++

David McClain <david@acudora.com> writes:

> On Feb 18, 2013, at 13:58 PM, Wayne Peterson <wcpeterson@me.com>
> wrote:
>
>     Love this discussion, as too few people really program in Lisp.
>     BTW: I am willing to bet I having even used most of the features
>     of Lisp. It's a language I will never master. And I think that is
>     a good thing.
>    
>     Wayne
>
>
> I had to confront an old wive's tale the other night, about *NEVER
> USE EVAL!!!!*  Well, there really are some places that have
> legitimate uses, and I had to dig to find them.
>
> But meanwhile, I had an expression like '(* 15 (1+ (* 32 (1+ (* 43
> .... ))) that I wanted both to take apart to some level, and to
> evaluate the tail of the remaining portion after the take-apart. How
> to do this without using EVAL??
>
> So I came up with this: 
>
> (eval e)  ==>  (funcall (compile nil `(lambda () ,e))))
>
> Eh? 

Right.  It's no better.  For all we know, the implementation could
have:

   (defun eval (e) (funcall (compile nil `(lambda () ,e))))


> Anyone else have a better idea?  Has to evaluate in the current
> dynamic environment and the null lexical environment, just like EVAL.

Then just use EVAL.  This is the reason why EVAL is frowned upon, but if
that's what you need then you need EVAL.


A form without free variables can be EVAL'ed freely.

  (eval '(* 15 (1+ (* 32 (1+ (* 43 2))))))


A form whose free variables are all special variables can be EVAL'ed
freely in a dynamic environment where those free variables are special
and bound as needed.

 (defvar *v1*)
 (let ((*v1* 1)
       (v2   2))
   (declare (special v2))
   (eval '(list (incf *v1*) (incf v2))))

Now if your form contains free variables that are not special, then you
just cannot use EVAL meaningfully on it.  You could use EVAL and get
implementation dependent effects (such has having "undefined" variable
errors, or such as declaraing them automatically special, or not).  Or
you could use funcall compile lambda, binding those free variables in a
let wrapping the form (therefore reverting to the first case above):

  (let ((v1 1)
        (v2 2))
    (let ((form '(list v1 (incf v2))))
      (funcall (compile nil `(lambda () 
                               (let ((v1 ',v1) (v2 ',v2))
                                 ,form))))))
   --> (1 3)


-- 
__Pascal Bourguignon__                     http://www.informatimago.com/
A bad day in () is better than a good day in {}.

_______________________________________________
Lisp Hug - the mailing list for LispWorks users
lisp-hug@lispworks.com
http://www.lispworks.com/support/lisp-hug.html


Re: On the road to replacing C++

Never use EVAL is silly, IMO.

The question really is, do you trust the source? For example, the LW IDE uses EVAL (well, LOAD, probably) on your ~/.lispworks file, because you're the user and what would it matter if you put a "format c:" in your ~/.lispworks file?

What is important is remembering that EVAL will do just that - evaluate. Similarly, it's always important to remember that READ is capable of calling EVAL if *READ-EVAL* is non-nil.

Sometimes I really wish that Lisp had something similar to Erlang's file:consult/1 (http://www.erlang.org/doc/man/file.html#consult-1) function. You can get pretty close, though:

(defun consult (pathname)
  (let ((*read-eval* nil))
    (with-open-file (s pathname)
      (loop :for term := (read s nil nil) :while term :collect term))))

It's just not quite the same. But, most of the time it fits the bill perfectly.

As for having some kind of scripting language in Lisp that's somehow a subset of Lisp or something else, I used to think this would be cool, and then I come back to asking myself why? What would be the point?

If I trust the source (a user having bought a product or myself), then give them the full access. Why not? After all, then you'll have the totally awesome capability to bug fix a running system for them, send patch files, etc. In my experience, if you trust the source and don't provide full access, you'll end up sticking in a backdoor for yourself, and then once that cat's out of the bag, you'll be kicking yourself for putting tons of effort into creating a sandbox that's easily bypassed (which will happen). 

And, if you don't trust the source of the "script," then you have a whole host of other problems to think about besides whether or not to use EVAL. :-)

My 2 cents.

Jeff M.


On Wed, Feb 20, 2013 at 1:27 PM, David McClain <david@acudora.com> wrote:

>> Anyone else have a better idea?  Has to evaluate in the current
>> dynamic environment and the null lexical environment, just like EVAL.
>
> Then just use EVAL.  This is the reason why EVAL is frowned upon, but if
> that's what you need then you need EVAL.

Yes, every now and again, I revisit old prescriptions...  like "never use EVAL", "never use global vars", "never use goto"... I'm old enough to have been through a number of revolutions, starting with the advent of "structured programming", "object oriented programming", and so on.

There was no "C" compiler when I started -- just PL/1 and Assembly, Cobol, Snobol, and RPG... We went through some great times, but sad to say, things often get pushed beyond the point of reasonableness. C++ now looks like a pig in lipstick.

Whenever I hear a rule repeated in my head, I need to ask why? what was the context of the time? What were the reasons for believing these edicts?

Just trying to clear out old cobwebs...

- DM


_______________________________________________
Lisp Hug - the mailing list for LispWorks users
lisp-hug@lispworks.com
http://www.lispworks.com/support/lisp-hug.html


RE: On the road to replacing C++

Hello All,

 

I know I’m new to Lisp, but I agree with Jeff in the general sense. I have done this in many applications. If you want to add a little additional protection, you may want to consider zipping up the source files with password protection and/or creating a little command line utility that will zip/sign them. You could then crack open the zip and/or verify the signature before evaluating them. Just a thought.

 

Thanks,

Gerry

 

 

From: owner-lisp-hug@lispworks.com [mailto:owner-lisp-hug@lispworks.com] On Behalf Of Jeff Massung
Sent: Wednesday, February 20, 2013 2:00 PM
To: lisp-hug@lispworks.com Lispworks
Subject: Re: On the road to replacing C++

 

Never use EVAL is silly, IMO.

 

The question really is, do you trust the source? For example, the LW IDE uses EVAL (well, LOAD, probably) on your ~/.lispworks file, because you're the user and what would it matter if you put a "format c:" in your ~/.lispworks file?

 

What is important is remembering that EVAL will do just that - evaluate. Similarly, it's always important to remember that READ is capable of calling EVAL if *READ-EVAL* is non-nil.

 

Sometimes I really wish that Lisp had something similar to Erlang's file:consult/1 (http://www.erlang.org/doc/man/file.html#consult-1) function. You can get pretty close, though:

 

(defun consult (pathname)

  (let ((*read-eval* nil))

    (with-open-file (s pathname)

      (loop :for term := (read s nil nil) :while term :collect term))))

 

It's just not quite the same. But, most of the time it fits the bill perfectly.

 

As for having some kind of scripting language in Lisp that's somehow a subset of Lisp or something else, I used to think this would be cool, and then I come back to asking myself why? What would be the point?

 

If I trust the source (a user having bought a product or myself), then give them the full access. Why not? After all, then you'll have the totally awesome capability to bug fix a running system for them, send patch files, etc. In my experience, if you trust the source and don't provide full access, you'll end up sticking in a backdoor for yourself, and then once that cat's out of the bag, you'll be kicking yourself for putting tons of effort into creating a sandbox that's easily bypassed (which will happen). 

 

And, if you don't trust the source of the "script," then you have a whole host of other problems to think about besides whether or not to use EVAL. :-)

 

My 2 cents.

 

Jeff M.

 

On Wed, Feb 20, 2013 at 1:27 PM, David McClain <david@acudora.com> wrote:


>> Anyone else have a better idea?  Has to evaluate in the current
>> dynamic environment and the null lexical environment, just like EVAL.
>
> Then just use EVAL.  This is the reason why EVAL is frowned upon, but if
> that's what you need then you need EVAL.

Yes, every now and again, I revisit old prescriptions...  like "never use EVAL", "never use global vars", "never use goto"... I'm old enough to have been through a number of revolutions, starting with the advent of "structured programming", "object oriented programming", and so on.

There was no "C" compiler when I started -- just PL/1 and Assembly, Cobol, Snobol, and RPG... We went through some great times, but sad to say, things often get pushed beyond the point of reasonableness. C++ now looks like a pig in lipstick.

Whenever I hear a rule repeated in my head, I need to ask why? what was the context of the time? What were the reasons for believing these edicts?

Just trying to clear out old cobwebs...

- DM



_______________________________________________
Lisp Hug - the mailing list for LispWorks users
lisp-hug@lispworks.com
http://www.lispworks.com/support/lisp-hug.html

 

RE: On the road to replacing C++

Hello All,

 

I have been thinking about how to provide remote REPL access to my delivered application. I am extremely interested in doing that. I’ve been toying with the idea of creating an “enable remote support” feature in the application that would connect to a server at my office over https and present me with a REPL. Once the support session was finished, the user could “disable remote support”, which would then close the connection and shutdown the “remote support” process in the image. This should provide a reasonably secure way to handle this with the added benefit of allowing the customer/user to initiate/control the session. I’m curious what y’all think about something like this.

 

Thanks,

Gerry

 

 

From: owner-lisp-hug@lispworks.com [mailto:owner-lisp-hug@lispworks.com] On Behalf Of Gerry Weaver
Sent: Wednesday, February 20, 2013 2:18 PM
To: Jeff Massung
Cc: lisp-hug@lispworks.com
Subject: RE: On the road to replacing C++

 

Hello All,

 

I know I’m new to Lisp, but I agree with Jeff in the general sense. I have done this in many applications. If you want to add a little additional protection, you may want to consider zipping up the source files with password protection and/or creating a little command line utility that will zip/sign them. You could then crack open the zip and/or verify the signature before evaluating them. Just a thought.

 

Thanks,

Gerry

 

 

From: owner-lisp-hug@lispworks.com [mailto:owner-lisp-hug@lispworks.com] On Behalf Of Jeff Massung
Sent: Wednesday, February 20, 2013 2:00 PM
To: lisp-hug@lispworks.com Lispworks
Subject: Re: On the road to replacing C++

 

Never use EVAL is silly, IMO.

 

The question really is, do you trust the source? For example, the LW IDE uses EVAL (well, LOAD, probably) on your ~/.lispworks file, because you're the user and what would it matter if you put a "format c:" in your ~/.lispworks file?

 

What is important is remembering that EVAL will do just that - evaluate. Similarly, it's always important to remember that READ is capable of calling EVAL if *READ-EVAL* is non-nil.

 

Sometimes I really wish that Lisp had something similar to Erlang's file:consult/1 (http://www.erlang.org/doc/man/file.html#consult-1) function. You can get pretty close, though:

 

(defun consult (pathname)

  (let ((*read-eval* nil))

    (with-open-file (s pathname)

      (loop :for term := (read s nil nil) :while term :collect term))))

 

It's just not quite the same. But, most of the time it fits the bill perfectly.

 

As for having some kind of scripting language in Lisp that's somehow a subset of Lisp or something else, I used to think this would be cool, and then I come back to asking myself why? What would be the point?

 

If I trust the source (a user having bought a product or myself), then give them the full access. Why not? After all, then you'll have the totally awesome capability to bug fix a running system for them, send patch files, etc. In my experience, if you trust the source and don't provide full access, you'll end up sticking in a backdoor for yourself, and then once that cat's out of the bag, you'll be kicking yourself for putting tons of effort into creating a sandbox that's easily bypassed (which will happen). 

 

And, if you don't trust the source of the "script," then you have a whole host of other problems to think about besides whether or not to use EVAL. :-)

 

My 2 cents.

 

Jeff M.

 

On Wed, Feb 20, 2013 at 1:27 PM, David McClain <david@acudora.com> wrote:


>> Anyone else have a better idea?  Has to evaluate in the current
>> dynamic environment and the null lexical environment, just like EVAL.
>
> Then just use EVAL.  This is the reason why EVAL is frowned upon, but if
> that's what you need then you need EVAL.

Yes, every now and again, I revisit old prescriptions...  like "never use EVAL", "never use global vars", "never use goto"... I'm old enough to have been through a number of revolutions, starting with the advent of "structured programming", "object oriented programming", and so on.

There was no "C" compiler when I started -- just PL/1 and Assembly, Cobol, Snobol, and RPG... We went through some great times, but sad to say, things often get pushed beyond the point of reasonableness. C++ now looks like a pig in lipstick.

Whenever I hear a rule repeated in my head, I need to ask why? what was the context of the time? What were the reasons for believing these edicts?

Just trying to clear out old cobwebs...

- DM



_______________________________________________
Lisp Hug - the mailing list for LispWorks users
lisp-hug@lispworks.com
http://www.lispworks.com/support/lisp-hug.html

 

RE: On the road to replacing C++

Hi there Gerry,

 

“I have been thinking about how to provide remote REPL access to my delivered application.”

 

Maybe have a peek at an example that ships with LispWorks, for some ideas along those lines:

 

   …\examples\capi\applications\remote-debugger.lisp

   …\examples\capi\applications\remote-debugger-client.lisp

 

Brian Connoy

 

RE: On the road to replacing C++

Hi Brian,

 

Thanks for the pointer. I am continually amazed by the depth and breadth of the LispWorks product.

 

Thanks,

Gerry

 

From: Brian Connoy [mailto:BConnoy@morrisonhershfield.com]
Sent: Wednesday, February 20, 2013 2:49 PM
To: Gerry Weaver
Cc: lisp-hug@lispworks.com
Subject: RE: On the road to replacing C++

 

Hi there Gerry,

 

“I have been thinking about how to provide remote REPL access to my delivered application.”

 

Maybe have a peek at an example that ships with LispWorks, for some ideas along those lines:

 

   …\examples\capi\applications\remote-debugger.lisp

   …\examples\capi\applications\remote-debugger-client.lisp

 

Brian Connoy

 

RE: On the road to replacing C++

Hi Tim,

 

You make some good points..  Let me try to explain better.

 

1.       The user calls for support

2.       I direct them to enable remote support

3.       A process is started in the application that connects to my server over https

4.       My server accepts the connection a opens a REPL window

5.       I do whatever is necessary (the application logs my activity)

6.       I instruct the user to disable remote support

7.       The connection is closed and the process is stopped in the application

 

Basically the support code/process only runs if the user enables remote support. If there is no activity for some period it would shut itself down. Most firewalls allow outgoing https, so that isn’t normally a problem. There would never be an incoming support connection to the application. The connection would always originate from the customer’s location. I actually have something like this already that allows me to enable a trace and capture the output remotely. I can have the user reproduce the problem and watch what the application is doing. This would just be a variation of the same theme for Lisp with the added feature of a possible hot fix. Most of my customers are banks and this has worked well there. I guess it would largely depend on the relationship with the customer and their security policy.

 

I hope this made more sense.

 

Thanks,

Gerry

 

 

 

From: Tim Bradshaw [mailto:tfb@cley.com]
Sent: Wednesday, February 20, 2013 7:20 PM
To: Gerry Weaver
Cc: lisp-hug@lispworks.com
Subject: Re: On the road to replacing C++

 

I don't know what your target is, but in the sorts of places where I work(ed), there are at least a couple of things which make this nontrivial:

 

(1) firewalls make it extremely unlikely that any machine on the internal network would ever be accessible from the outside, with the exceptions being systems that people would certainly not allow this sort of thing to happen on (public-facing web servers and so on).  There are ways around this which involve the internal machine making some (proxied) outgoing connection to the vendor's systems, but they are laborious.

(2) No-one not completely trusted (where I'm leaving "completely" undefined, because it is not well-defined as far as I know) would be allowed to type at an internal system anyway, or at least not unless they were physically supervised by someone who was.

 

Some vendors did provide remote support tools, but in general they were fairly obviously too fragile to use.  There needs to be some way by which is is not possible for a command to be executed without the trusted person saying so, and that is often hard to ensure: the tool I looked at mostly tried to ensure this by making it impossible for the vendor person to type return, but there are other ways of getting most Unix shells to execute things, unfortunately.  I think the only reliable way of doing this is to have some thing where the vendor person types some stuff and the trusted person then takes some action which sends it to the application, and that becomes little better than some IRC-alike + cut & paste.

 

(This isn't military BTW: I hope that they are a whole bunch fussier than we were, though I fear they may not be.)

 

--tim

 

 

On 20 Feb 2013, at 20:36, Gerry Weaver wrote:



Hello All,

 

I have been thinking about how to provide remote REPL access to my delivered application. I am extremely interested in doing that. I’ve been toying with the idea of creating an “enable remote support” feature in the application that would connect to a server at my office over https and present me with a REPL. Once the support session was finished, the user could “disable remote support”, which would then close the connection and shutdown the “remote support” process in the image. This should provide a reasonably secure way to handle this with the added benefit of allowing the customer/user to initiate/control the session. I’m curious what y’all think about something like this.

 

Thanks,

Gerry

 

 

From: owner-lisp-hug@lispworks.com [mailto:owner-lisp-hug@lispworks.com] On Behalf Of Gerry Weaver
Sent: Wednesday, February 20, 2013 2:18 PM
To: Jeff Massung
Cc: lisp-hug@lispworks.com
Subject: RE: On the road to replacing C++

 

Hello All,

 

I know I’m new to Lisp, but I agree with Jeff in the general sense. I have done this in many applications. If you want to add a little additional protection, you may want to consider zipping up the source files with password protection and/or creating a little command line utility that will zip/sign them. You could then crack open the zip and/or verify the signature before evaluating them. Just a thought.

 

Thanks,

Gerry

 

 

From: owner-lisp-hug@lispworks.com [mailto:owner-lisp-hug@lispworks.com] On Behalf Of Jeff Massung
Sent: Wednesday, February 20, 2013 2:00 PM
To: lisp-hug@lispworks.com Lispworks
Subject: Re: On the road to replacing C++

 

Never use EVAL is silly, IMO.

 

The question really is, do you trust the source? For example, the LW IDE uses EVAL (well, LOAD, probably) on your ~/.lispworks file, because you're the user and what would it matter if you put a "format c:" in your ~/.lispworks file?

 

What is important is remembering that EVAL will do just that - evaluate. Similarly, it's always important to remember that READ is capable of calling EVAL if *READ-EVAL* is non-nil.

 

Sometimes I really wish that Lisp had something similar to Erlang's file:consult/1 (http://www.erlang.org/doc/man/file.html#consult-1) function. You can get pretty close, though:

 

(defun consult (pathname)

  (let ((*read-eval* nil))

    (with-open-file (s pathname)

      (loop :for term := (read s nil nil) :while term :collect term))))

 

It's just not quite the same. But, most of the time it fits the bill perfectly.

 

As for having some kind of scripting language in Lisp that's somehow a subset of Lisp or something else, I used to think this would be cool, and then I come back to asking myself why? What would be the point?

 

If I trust the source (a user having bought a product or myself), then give them the full access. Why not? After all, then you'll have the totally awesome capability to bug fix a running system for them, send patch files, etc. In my experience, if you trust the source and don't provide full access, you'll end up sticking in a backdoor for yourself, and then once that cat's out of the bag, you'll be kicking yourself for putting tons of effort into creating a sandbox that's easily bypassed (which will happen). 

 

And, if you don't trust the source of the "script," then you have a whole host of other problems to think about besides whether or not to use EVAL. :-)

 

My 2 cents.

 

Jeff M.

 

On Wed, Feb 20, 2013 at 1:27 PM, David McClain <david@acudora.com> wrote:


>> Anyone else have a better idea?  Has to evaluate in the current
>> dynamic environment and the null lexical environment, just like EVAL.
>
> Then just use EVAL.  This is the reason why EVAL is frowned upon, but if
> that's what you need then you need EVAL.

Yes, every now and again, I revisit old prescriptions...  like "never use EVAL", "never use global vars", "never use goto"... I'm old enough to have been through a number of revolutions, starting with the advent of "structured programming", "object oriented programming", and so on.

There was no "C" compiler when I started -- just PL/1 and Assembly, Cobol, Snobol, and RPG... We went through some great times, but sad to say, things often get pushed beyond the point of reasonableness. C++ now looks like a pig in lipstick.

Whenever I hear a rule repeated in my head, I need to ask why? what was the context of the time? What were the reasons for believing these edicts?

Just trying to clear out old cobwebs...

- DM



_______________________________________________
Lisp Hug - the mailing list for LispWorks users
lisp-hug@lispworks.com
http://www.lispworks.com/support/lisp-hug.html

 

 

RE: On the road to replacing C++

Hi Jeff,

 

That is exactly what I’m talking about. I guess I must not be doing a very good job of explaining it ;-( The customer always initiates the connection from behind their firewall to a server running at my office. Maybe it’s my use of the word “process” to describe the customer functionality. I used that because, in the current C/C++ version, the application server runs the remote support session in a separate thread. There is actually server type code running on both sides of the connection, but the customer application instance always initiates the connection. It works similar to Bumgar, Webex, etc.

 

Thanks,

Gerry

 

 

From: Jeffrey Massung [mailto:massung@gmail.com]
Sent: Wednesday, February 20, 2013 10:16 PM
To: Gerry Weaver
Cc: Tim Bradshaw; lisp-hug@lispworks.com
Subject: Re: On the road to replacing C++

 

Gerry,

 

If the user is going to have to enable something, and security isn't a major concern, then why not just run a service on your machine that when the client "enables remote debugging" actually connects to your server and they are the client. You can then - from the server - send code which the client executes? That way you don't have to worry about firewalls, nat traversal, etc. and most of all the client doesn't have a back door into their system that someone else can get into.

 

Just a thought...

 

Jeff M.


Sent from my iPad


On Feb 20, 2013, at 9:23 PM, Gerry Weaver <gerryw@compvia.com> wrote:

Hi Tim,

 

You make some good points.  Let me try to explain better.

 

1.      The user calls for support

2.      I direct them to enable remote support

3.      A process is started in the application that connects to my server over https

4.      My server accepts the connection a opens a REPL window

5.      I do whatever is necessary (the application logs my activity)

6.      I instruct the user to disable remote support

7.      The connection is closed and the process is stopped in the application

 

Basically the support code/process only runs if the user enables remote support. If there is no activity for some period it would shut itself down. Most firewalls allow outgoing https, so that isn’t normally a problem. There would never be an incoming support connection to the application. The connection would always originate from the customer’s location. I actually have something like this already that allows me to enable a trace and capture the output remotely. I can have the user reproduce the problem and watch what the application is doing. This would just be a variation of the same theme for Lisp with the added feature of a possible hot fix. Most of my customers are banks and this has worked well there. I guess it would largely depend on the relationship with the customer and their security policy.

 

I hope this made more sense.

 

Thanks,

Gerry

 

 

 

From: Tim Bradshaw [mailto:tfb@cley.com]
Sent: Wednesday, February 20, 2013 7:20 PM
To: Gerry Weaver
Cc: lisp-hug@lispworks.com
Subject: Re: On the road to replacing C++

 

I don't know what your target is, but in the sorts of places where I work(ed), there are at least a couple of things which make this nontrivial:

 

(1) firewalls make it extremely unlikely that any machine on the internal network would ever be accessible from the outside, with the exceptions being systems that people would certainly not allow this sort of thing to happen on (public-facing web servers and so on).  There are ways around this which involve the internal machine making some (proxied) outgoing connection to the vendor's systems, but they are laborious.

(2) No-one not completely trusted (where I'm leaving "completely" undefined, because it is not well-defined as far as I know) would be allowed to type at an internal system anyway, or at least not unless they were physically supervised by someone who was.

 

Some vendors did provide remote support tools, but in general they were fairly obviously too fragile to use.  There needs to be some way by which is is not possible for a command to be executed without the trusted person saying so, and that is often hard to ensure: the tool I looked at mostly tried to ensure this by making it impossible for the vendor person to type return, but there are other ways of getting most Unix shells to execute things, unfortunately.  I think the only reliable way of doing this is to have some thing where the vendor person types some stuff and the trusted person then takes some action which sends it to the application, and that becomes little better than some IRC-alike + cut & paste.

 

(This isn't military BTW: I hope that they are a whole bunch fussier than we were, though I fear they may not be.)

 

--tim

 

 

On 20 Feb 2013, at 20:36, Gerry Weaver wrote:




Hello All,

 

I have been thinking about how to provide remote REPL access to my delivered application. I am extremely interested in doing that. I’ve been toying with the idea of creating an “enable remote support” feature in the application that would connect to a server at my office over https and present me with a REPL. Once the support session was finished, the user could “disable remote support”, which would then close the connection and shutdown the “remote support” process in the image. This should provide a reasonably secure way to handle this with the added benefit of allowing the customer/user to initiate/control the session. I’m curious what y’all think about something like this.

 

Thanks,

Gerry

 

 

From: owner-lisp-hug@lispworks.com [mailto:owner-lisp-hug@lispworks.com] On Behalf Of Gerry Weaver
Sent: Wednesday, February 20, 2013 2:18 PM
To: Jeff Massung
Cc: lisp-hug@lispworks.com
Subject: RE: On the road to replacing C++

 

Hello All,

 

I know I’m new to Lisp, but I agree with Jeff in the general sense. I have done this in many applications. If you want to add a little additional protection, you may want to consider zipping up the source files with password protection and/or creating a little command line utility that will zip/sign them. You could then crack open the zip and/or verify the signature before evaluating them. Just a thought.

 

Thanks,

Gerry

 

 

From: owner-lisp-hug@lispworks.com [mailto:owner-lisp-hug@lispworks.com] On Behalf Of Jeff Massung
Sent: Wednesday, February 20, 2013 2:00 PM
To: lisp-hug@lispworks.com Lispworks
Subject: Re: On the road to replacing C++

 

Never use EVAL is silly, IMO.

 

The question really is, do you trust the source? For example, the LW IDE uses EVAL (well, LOAD, probably) on your ~/.lispworks file, because you're the user and what would it matter if you put a "format c:" in your ~/.lispworks file?

 

What is important is remembering that EVAL will do just that - evaluate. Similarly, it's always important to remember that READ is capable of calling EVAL if *READ-EVAL* is non-nil.

 

Sometimes I really wish that Lisp had something similar to Erlang's file:consult/1 (http://www.erlang.org/doc/man/file.html#consult-1) function. You can get pretty close, though:

 

(defun consult (pathname)

  (let ((*read-eval* nil))

    (with-open-file (s pathname)

      (loop :for term := (read s nil nil) :while term :collect term))))

 

It's just not quite the same. But, most of the time it fits the bill perfectly.

 

As for having some kind of scripting language in Lisp that's somehow a subset of Lisp or something else, I used to think this would be cool, and then I come back to asking myself why? What would be the point?

 

If I trust the source (a user having bought a product or myself), then give them the full access. Why not? After all, then you'll have the totally awesome capability to bug fix a running system for them, send patch files, etc. In my experience, if you trust the source and don't provide full access, you'll end up sticking in a backdoor for yourself, and then once that cat's out of the bag, you'll be kicking yourself for putting tons of effort into creating a sandbox that's easily bypassed (which will happen). 

 

And, if you don't trust the source of the "script," then you have a whole host of other problems to think about besides whether or not to use EVAL. :-)

 

My 2 cents.

 

Jeff M.

 

On Wed, Feb 20, 2013 at 1:27 PM, David McClain <david@acudora.com> wrote:


>> Anyone else have a better idea?  Has to evaluate in the current
>> dynamic environment and the null lexical environment, just like EVAL.
>
> Then just use EVAL.  This is the reason why EVAL is frowned upon, but if
> that's what you need then you need EVAL.

Yes, every now and again, I revisit old prescriptions...  like "never use EVAL", "never use global vars", "never use goto"... I'm old enough to have been through a number of revolutions, starting with the advent of "structured programming", "object oriented programming", and so on.

There was no "C" compiler when I started -- just PL/1 and Assembly, Cobol, Snobol, and RPG... We went through some great times, but sad to say, things often get pushed beyond the point of reasonableness. C++ now looks like a pig in lipstick.

Whenever I hear a rule repeated in my head, I need to ask why? what was the context of the time? What were the reasons for believing these edicts?

Just trying to clear out old cobwebs...

- DM



_______________________________________________
Lisp Hug - the mailing list for LispWorks users
lisp-hug@lispworks.com
http://www.lispworks.com/support/lisp-hug.html

 

 

Re: On the road to replacing C++

Heh, after re-reading what you wrote (I had just skimmed it) you explained it well enough. Reading comprehension for the loss.

Well, let's just leave it at great minds think alike? ;-)

Jeff M.


On Wed, Feb 20, 2013 at 11:11 PM, Gerry Weaver <gerryw@compvia.com> wrote:

Hi Jeff,

 

That is exactly what I’m talking about. I guess I must not be doing a very good job of explaining it ;-( The customer always initiates the connection from behind their firewall to a server running at my office. Maybe it’s my use of the word “process” to describe the customer functionality. I used that because, in the current C/C++ version, the application server runs the remote support session in a separate thread. There is actually server type code running on both sides of the connection, but the customer application instance always initiates the connection. It works similar to Bumgar, Webex, etc.

 

Thanks,

Gerry

 

 

From: Jeffrey Massung [mailto:massung@gmail.com]
Sent: Wednesday, February 20, 2013 10:16 PM
To: Gerry Weaver
Cc: Tim Bradshaw; lisp-hug@lispworks.com


Subject: Re: On the road to replacing C++

 

Gerry,

 

If the user is going to have to enable something, and security isn't a major concern, then why not just run a service on your machine that when the client "enables remote debugging" actually connects to your server and they are the client. You can then - from the server - send code which the client executes? That way you don't have to worry about firewalls, nat traversal, etc. and most of all the client doesn't have a back door into their system that someone else can get into.

 

Just a thought...

 

Jeff M.


Sent from my iPad


On Feb 20, 2013, at 9:23 PM, Gerry Weaver <gerryw@compvia.com> wrote:

Hi Tim,

 

You make some good points..  Let me try to explain better.

 

1.      The user calls for support

2.      I direct them to enable remote support

3.      A process is started in the application that connects to my server over https

4.      My server accepts the connection a opens a REPL window

5.      I do whatever is necessary (the application logs my activity)

6.      I instruct the user to disable remote support

7.      The connection is closed and the process is stopped in the application

 

Basically the support code/process only runs if the user enables remote support. If there is no activity for some period it would shut itself down. Most firewalls allow outgoing https, so that isn’t normally a problem. There would never be an incoming support connection to the application. The connection would always originate from the customer’s location. I actually have something like this already that allows me to enable a trace and capture the output remotely. I can have the user reproduce the problem and watch what the application is doing. This would just be a variation of the same theme for Lisp with the added feature of a possible hot fix. Most of my customers are banks and this has worked well there. I guess it would largely depend on the relationship with the customer and their security policy.

 

I hope this made more sense.

 

Thanks,

Gerry

 

 

 

From: Tim Bradshaw [mailto:tfb@cley.com]
Sent: Wednesday, February 20, 2013 7:20 PM
To: Gerry Weaver
Cc: lisp-hug@lispworks.com
Subject: Re: On the road to replacing C++

 

I don't know what your target is, but in the sorts of places where I work(ed), there are at least a couple of things which make this nontrivial:

 

(1) firewalls make it extremely unlikely that any machine on the internal network would ever be accessible from the outside, with the exceptions being systems that people would certainly not allow this sort of thing to happen on (public-facing web servers and so on).  There are ways around this which involve the internal machine making some (proxied) outgoing connection to the vendor's systems, but they are laborious.

(2) No-one not completely trusted (where I'm leaving "completely" undefined, because it is not well-defined as far as I know) would be allowed to type at an internal system anyway, or at least not unless they were physically supervised by someone who was.

 

Some vendors did provide remote support tools, but in general they were fairly obviously too fragile to use.  There needs to be some way by which is is not possible for a command to be executed without the trusted person saying so, and that is often hard to ensure: the tool I looked at mostly tried to ensure this by making it impossible for the vendor person to type return, but there are other ways of getting most Unix shells to execute things, unfortunately.  I think the only reliable way of doing this is to have some thing where the vendor person types some stuff and the trusted person then takes some action which sends it to the application, and that becomes little better than some IRC-alike + cut & paste.

 

(This isn't military BTW: I hope that they are a whole bunch fussier than we were, though I fear they may not be.)

 

--tim

 

 

On 20 Feb 2013, at 20:36, Gerry Weaver wrote:




Hello All,

 

I have been thinking about how to provide remote REPL access to my delivered application. I am extremely interested in doing that. I’ve been toying with the idea of creating an “enable remote support” feature in the application that would connect to a server at my office over https and present me with a REPL. Once the support session was finished, the user could “disable remote support”, which would then close the connection and shutdown the “remote support” process in the image. This should provide a reasonably secure way to handle this with the added benefit of allowing the customer/user to initiate/control the session. I’m curious what y’all think about something like this.

 

Thanks,

Gerry

 

 

From: owner-lisp-hug@lispworks.com [mailto:owner-lisp-hug@lispworks.com] On Behalf Of Gerry Weaver
Sent: Wednesday, February 20, 2013 2:18 PM
To: Jeff Massung
Cc: lisp-hug@lispworks.com
Subject: RE: On the road to replacing C++

 

Hello All,

 

I know I’m new to Lisp, but I agree with Jeff in the general sense. I have done this in many applications. If you want to add a little additional protection, you may want to consider zipping up the source files with password protection and/or creating a little command line utility that will zip/sign them. You could then crack open the zip and/or verify the signature before evaluating them. Just a thought.

 

Thanks,

Gerry

 

 

From: owner-lisp-hug@lispworks.com [mailto:owner-lisp-hug@lispworks.com] On Behalf Of Jeff Massung
Sent: Wednesday, February 20, 2013 2:00 PM
To: lisp-hug@lispworks.com Lispworks
Subject: Re: On the road to replacing C++

 

Never use EVAL is silly, IMO.

 

The question really is, do you trust the source? For example, the LW IDE uses EVAL (well, LOAD, probably) on your ~/.lispworks file, because you're the user and what would it matter if you put a "format c:" in your ~/.lispworks file?

 

What is important is remembering that EVAL will do just that - evaluate. Similarly, it's always important to remember that READ is capable of calling EVAL if *READ-EVAL* is non-nil.

 

Sometimes I really wish that Lisp had something similar to Erlang's file:consult/1 (http://www.erlang.org/doc/man/file..html#consult-1) function. You can get pretty close, though:

 

(defun consult (pathname)

  (let ((*read-eval* nil))

    (with-open-file (s pathname)

      (loop :for term := (read s nil nil) :while term :collect term))))

 

It's just not quite the same. But, most of the time it fits the bill perfectly.

 

As for having some kind of scripting language in Lisp that's somehow a subset of Lisp or something else, I used to think this would be cool, and then I come back to asking myself why? What would be the point?

 

If I trust the source (a user having bought a product or myself), then give them the full access. Why not? After all, then you'll have the totally awesome capability to bug fix a running system for them, send patch files, etc. In my experience, if you trust the source and don't provide full access, you'll end up sticking in a backdoor for yourself, and then once that cat's out of the bag, you'll be kicking yourself for putting tons of effort into creating a sandbox that's easily bypassed (which will happen). 

 

And, if you don't trust the source of the "script," then you have a whole host of other problems to think about besides whether or not to use EVAL. :-)

 

My 2 cents.

 

Jeff M.

 

On Wed, Feb 20, 2013 at 1:27 PM, David McClain <david@acudora.com> wrote:


>> Anyone else have a better idea?  Has to evaluate in the current
>> dynamic environment and the null lexical environment, just like EVAL.
>
> Then just use EVAL.  This is the reason why EVAL is frowned upon, but if
> that's what you need then you need EVAL.

Yes, every now and again, I revisit old prescriptions...  like "never use EVAL", "never use global vars", "never use goto"... I'm old enough to have been through a number of revolutions, starting with the advent of "structured programming", "object oriented programming", and so on.

There was no "C" compiler when I started -- just PL/1 and Assembly, Cobol, Snobol, and RPG... We went through some great times, but sad to say, things often get pushed beyond the point of reasonableness. C++ now looks like a pig in lipstick.

Whenever I hear a rule repeated in my head, I need to ask why? what was the context of the time? What were the reasons for believing these edicts?

Just trying to clear out old cobwebs...

- DM



_______________________________________________
Lisp Hug - the mailing list for LispWorks users
lisp-hug@lispworks.com
http://www.lispworks.com/support/lisp-hug.html

 

 


RE: On the road to replacing C++

Hi Jeff,

 

No worries.  I was beginning to think y’all were just messing around with me ;-) Then again, sitting in a dimly lit room, pounding mountain dew, and cursing the infernal machine, might be having a negative impact on my communication skills.

 

BTW: I just received my spankin’ new Filco Ninja Tactile (brown switch) keyboard today. These keyboards just keep getting better. If you ever want to treat yourself, they’re not a bad way to go.

 

I’m actually a little curious about what platforms y’all develop on. I mean what is your main dev platform? It sounded like David’s is Mac. I use Ubuntu server 12.04 LTS 64bit with the latest Mate desktop installed. I haven’t decided whether I’m going to do client apps on Mac/Win with LispWorks yet. I need to spend some quality time with capi first. Qt is pretty quick and easy though.

 

UNWINDPROTECT, BLOCK, RETURN-FROM, CATCH, and THROW. Where have you been all my life J

 

Thanks,

Gerry

 

 

 

 

From: owner-lisp-hug@lispworks.com [mailto:owner-lisp-hug@lispworks.com] On Behalf Of Jeff Massung
Sent: Wednesday, February 20, 2013 11:33 PM
To: lisp-hug@lispworks.com
Subject: Re: On the road to replacing C++

 

Heh, after re-reading what you wrote (I had just skimmed it) you explained it well enough. Reading comprehension for the loss.

 

Well, let's just leave it at great minds think alike? ;-)

 

Jeff M.

 

On Wed, Feb 20, 2013 at 11:11 PM, Gerry Weaver <gerryw@compvia.com> wrote:

Hi Jeff,

 

That is exactly what I’m talking about. I guess I must not be doing a very good job of explaining it ;-( The customer always initiates the connection from behind their firewall to a server running at my office. Maybe it’s my use of the word “process” to describe the customer functionality. I used that because, in the current C/C++ version, the application server runs the remote support session in a separate thread. There is actually server type code running on both sides of the connection, but the customer application instance always initiates the connection. It works similar to Bumgar, Webex, etc.

 

Thanks,

Gerry

 

 

From: Jeffrey Massung [mailto:massung@gmail.com]
Sent: Wednesday, February 20, 2013 10:16 PM
To: Gerry Weaver
Cc: Tim Bradshaw; lisp-hug@lispworks.com


Subject: Re: On the road to replacing C++

 

Gerry,

 

If the user is going to have to enable something, and security isn't a major concern, then why not just run a service on your machine that when the client "enables remote debugging" actually connects to your server and they are the client. You can then - from the server - send code which the client executes? That way you don't have to worry about firewalls, nat traversal, etc. and most of all the client doesn't have a back door into their system that someone else can get into.

 

Just a thought...

 

Jeff M.


Sent from my iPad


On Feb 20, 2013, at 9:23 PM, Gerry Weaver <gerryw@compvia.com> wrote:

Hi Tim,

 

You make some good points.  Let me try to explain better.

 

1.      The user calls for support

2.      I direct them to enable remote support

3.      A process is started in the application that connects to my server over https

4.      My server accepts the connection a opens a REPL window

5.      I do whatever is necessary (the application logs my activity)

6.      I instruct the user to disable remote support

7.      The connection is closed and the process is stopped in the application

 

Basically the support code/process only runs if the user enables remote support. If there is no activity for some period it would shut itself down. Most firewalls allow outgoing https, so that isn’t normally a problem. There would never be an incoming support connection to the application. The connection would always originate from the customer’s location. I actually have something like this already that allows me to enable a trace and capture the output remotely. I can have the user reproduce the problem and watch what the application is doing. This would just be a variation of the same theme for Lisp with the added feature of a possible hot fix. Most of my customers are banks and this has worked well there. I guess it would largely depend on the relationship with the customer and their security policy.

 

I hope this made more sense.

 

Thanks,

Gerry

 

 

 

From: Tim Bradshaw [mailto:tfb@cley.com]
Sent: Wednesday, February 20, 2013 7:20 PM
To: Gerry Weaver
Cc: lisp-hug@lispworks.com
Subject: Re: On the road to replacing C++

 

I don't know what your target is, but in the sorts of places where I work(ed), there are at least a couple of things which make this nontrivial:

 

(1) firewalls make it extremely unlikely that any machine on the internal network would ever be accessible from the outside, with the exceptions being systems that people would certainly not allow this sort of thing to happen on (public-facing web servers and so on).  There are ways around this which involve the internal machine making some (proxied) outgoing connection to the vendor's systems, but they are laborious.

(2) No-one not completely trusted (where I'm leaving "completely" undefined, because it is not well-defined as far as I know) would be allowed to type at an internal system anyway, or at least not unless they were physically supervised by someone who was.

 

Some vendors did provide remote support tools, but in general they were fairly obviously too fragile to use.  There needs to be some way by which is is not possible for a command to be executed without the trusted person saying so, and that is often hard to ensure: the tool I looked at mostly tried to ensure this by making it impossible for the vendor person to type return, but there are other ways of getting most Unix shells to execute things, unfortunately.  I think the only reliable way of doing this is to have some thing where the vendor person types some stuff and the trusted person then takes some action which sends it to the application, and that becomes little better than some IRC-alike + cut & paste.

 

(This isn't military BTW: I hope that they are a whole bunch fussier than we were, though I fear they may not be.)

 

--tim

 

 

On 20 Feb 2013, at 20:36, Gerry Weaver wrote:



Hello All,

 

I have been thinking about how to provide remote REPL access to my delivered application. I am extremely interested in doing that. I’ve been toying with the idea of creating an “enable remote support” feature in the application that would connect to a server at my office over https and present me with a REPL. Once the support session was finished, the user could “disable remote support”, which would then close the connection and shutdown the “remote support” process in the image. This should provide a reasonably secure way to handle this with the added benefit of allowing the customer/user to initiate/control the session. I’m curious what y’all think about something like this.

 

Thanks,

Gerry

 

 

From: owner-lisp-hug@lispworks.com [mailto:owner-lisp-hug@lispworks.com] On Behalf Of Gerry Weaver
Sent: Wednesday, February 20, 2013 2:18 PM
To: Jeff Massung
Cc: lisp-hug@lispworks.com
Subject: RE: On the road to replacing C++

 

Hello All,

 

I know I’m new to Lisp, but I agree with Jeff in the general sense. I have done this in many applications. If you want to add a little additional protection, you may want to consider zipping up the source files with password protection and/or creating a little command line utility that will zip/sign them. You could then crack open the zip and/or verify the signature before evaluating them. Just a thought.

 

Thanks,

Gerry

 

 

From: owner-lisp-hug@lispworks.com [mailto:owner-lisp-hug@lispworks.com] On Behalf Of Jeff Massung
Sent: Wednesday, February 20, 2013 2:00 PM
To: lisp-hug@lispworks.com Lispworks
Subject: Re: On the road to replacing C++

 

Never use EVAL is silly, IMO.

 

The question really is, do you trust the source? For example, the LW IDE uses EVAL (well, LOAD, probably) on your ~/.lispworks file, because you're the user and what would it matter if you put a "format c:" in your ~/.lispworks file?

 

What is important is remembering that EVAL will do just that - evaluate. Similarly, it's always important to remember that READ is capable of calling EVAL if *READ-EVAL* is non-nil.

 

Sometimes I really wish that Lisp had something similar to Erlang's file:consult/1 (http://www.erlang.org/doc/man/file.html#consult-1) function. You can get pretty close, though:

 

(defun consult (pathname)

  (let ((*read-eval* nil))

    (with-open-file (s pathname)

      (loop :for term := (read s nil nil) :while term :collect term))))

 

It's just not quite the same. But, most of the time it fits the bill perfectly.

 

As for having some kind of scripting language in Lisp that's somehow a subset of Lisp or something else, I used to think this would be cool, and then I come back to asking myself why? What would be the point?

 

If I trust the source (a user having bought a product or myself), then give them the full access. Why not? After all, then you'll have the totally awesome capability to bug fix a running system for them, send patch files, etc. In my experience, if you trust the source and don't provide full access, you'll end up sticking in a backdoor for yourself, and then once that cat's out of the bag, you'll be kicking yourself for putting tons of effort into creating a sandbox that's easily bypassed (which will happen). 

 

And, if you don't trust the source of the "script," then you have a whole host of other problems to think about besides whether or not to use EVAL. :-)

 

My 2 cents.

 

Jeff M.

 

On Wed, Feb 20, 2013 at 1:27 PM, David McClain <david@acudora.com> wrote:


>> Anyone else have a better idea?  Has to evaluate in the current
>> dynamic environment and the null lexical environment, just like EVAL.
>
> Then just use EVAL.  This is the reason why EVAL is frowned upon, but if
> that's what you need then you need EVAL.

Yes, every now and again, I revisit old prescriptions...  like "never use EVAL", "never use global vars", "never use goto"... I'm old enough to have been through a number of revolutions, starting with the advent of "structured programming", "object oriented programming", and so on.

There was no "C" compiler when I started -- just PL/1 and Assembly, Cobol, Snobol, and RPG... We went through some great times, but sad to say, things often get pushed beyond the point of reasonableness. C++ now looks like a pig in lipstick.

Whenever I hear a rule repeated in my head, I need to ask why? what was the context of the time? What were the reasons for believing these edicts?

Just trying to clear out old cobwebs...

- DM



_______________________________________________
Lisp Hug - the mailing list for LispWorks users
lisp-hug@lispworks.com
http://www.lispworks.com/support/lisp-hug.html

 

 

 

RE: On the road to replacing C++

I run a (non-Lisp) eCommerce business with a significant custom manufacturing component. Many of our vendors need to perform remote support or maintenance on their equipment in the plants and we'd had a too-loose policy when we first launched.

We've since tightened that up significantly and use http://www.xceedium.com/ devices to control access. However, it does not provide the tight level of control "within machine" that Tim contemplates in #2, but does a good job on controlling and making auditable #1. Depending on the application, that might make sense. In our case, we need/intend to limit vendor access to the devices (computers or PLCs) that they "own", but since they own them entirely, we're much less concerned with limiting or auditing their actions on the machine. We also have many such instances all at the same site and so spreading the costs of access control across all of them makes it easy to swallow.

 It's all about the application and the organization's risk tolerance. Banks and military are a lot more worried than a hair salon or pet shop. :)

--Jim

PS: no connection to exceedium other than as a reasonably satisfied customer.

________________________________________
From: owner-lisp-hug@lispworks.com [owner-lisp-hug@lispworks.com] on behalf of Tim Bradshaw [tfb@cley.com]
Sent: Wednesday, February 20, 2013 8:19 PM
To: Gerry Weaver
Cc: lisp-hug@lispworks.com
Subject: Re: On the road to replacing C++

I don't know what your target is, but in the sorts of places where I work(ed), there are at least a couple of things which make this nontrivial:

(1) firewalls make it extremely unlikely that any machine on the internal network would ever be accessible from the outside, with the exceptions being systems that people would certainly not allow this sort of thing to happen on (public-facing web servers and so on).  There are ways around this which involve the internal machine making some (proxied) outgoing connection to the vendor's systems, but they are laborious.
(2) No-one not completely trusted (where I'm leaving "completely" undefined, because it is not well-defined as far as I know) would be allowed to type at an internal system anyway, or at least not unless they were physically supervised by someone who was.

Some vendors did provide remote support tools, but in general they were fairly obviously too fragile to use.  There needs to be some way by which is is not possible for a command to be executed without the trusted person saying so, and that is often hard to ensure: the tool I looked at mostly tried to ensure this by making it impossible for the vendor person to type return, but there are other ways of getting most Unix shells to execute things, unfortunately.  I think the only reliable way of doing this is to have some thing where the vendor person types some stuff and the trusted person then takes some action which sends it to the application, and that becomes little better than some IRC-alike + cut & paste.

(This isn't military BTW: I hope that they are a whole bunch fussier than we were, though I fear they may not be.)

--tim


On 20 Feb 2013, at 20:36, Gerry Weaver wrote:

Hello All,

I have been thinking about how to provide remote REPL access to my delivered application. I am extremely interested in doing that. I’ve been toying with the idea of creating an “enable remote support” feature in the application that would connect to a server at my office over https and present me with a REPL. Once the support session was finished, the user could “disable remote support”, which would then close the connection and shutdown the “remote support” process in the image. This should provide a reasonably secure way to handle this with the added benefit of allowing the customer/user to initiate/control the session. I’m curious what y’all think about something like this.

Thanks,
Gerry


From: owner-lisp-hug@lispworks.com<mailto:owner-lisp-hug@lispworks.com> [mailto:owner-lisp-hug@lispworks.com] On Behalf Of Gerry Weaver
Sent: Wednesday, February 20, 2013 2:18 PM
To: Jeff Massung
Cc: lisp-hug@lispworks.com<mailto:lisp-hug@lispworks.com>
Subject: RE: On the road to replacing C++

Hello All,

I know I’m new to Lisp, but I agree with Jeff in the general sense. I have done this in many applications. If you want to add a little additional protection, you may want to consider zipping up the source files with password protection and/or creating a little command line utility that will zip/sign them. You could then crack open the zip and/or verify the signature before evaluating them. Just a thought.

Thanks,
Gerry


From: owner-lisp-hug@lispworks.com<mailto:owner-lisp-hug@lispworks.com> [mailto:owner-lisp-hug@lispworks.com]<mailto:[mailto:owner-lisp-hug@lispworks.com]> On Behalf Of Jeff Massung
Sent: Wednesday, February 20, 2013 2:00 PM
To: lisp-hug@lispworks.com<mailto:lisp-hug@lispworks.com> Lispworks
Subject: Re: On the road to replacing C++

Never use EVAL is silly, IMO.

The question really is, do you trust the source? For example, the LW IDE uses EVAL (well, LOAD, probably) on your ~/.lispworks file, because you're the user and what would it matter if you put a "format c:" in your ~/.lispworks file?

What is important is remembering that EVAL will do just that - evaluate. Similarly, it's always important to remember that READ is capable of calling EVAL if *READ-EVAL* is non-nil.

Sometimes I really wish that Lisp had something similar to Erlang's file:consult/1 (http://www.erlang.org/doc/man/file.html#consult-1) function. You can get pretty close, though:

(defun consult (pathname)
  (let ((*read-eval* nil))
    (with-open-file (s pathname)
      (loop :for term := (read s nil nil) :while term :collect term))))

It's just not quite the same. But, most of the time it fits the bill perfectly.

As for having some kind of scripting language in Lisp that's somehow a subset of Lisp or something else, I used to think this would be cool, and then I come back to asking myself why? What would be the point?

If I trust the source (a user having bought a product or myself), then give them the full access. Why not? After all, then you'll have the totally awesome capability to bug fix a running system for them, send patch files, etc. In my experience, if you trust the source and don't provide full access, you'll end up sticking in a backdoor for yourself, and then once that cat's out of the bag, you'll be kicking yourself for putting tons of effort into creating a sandbox that's easily bypassed (which will happen).

And, if you don't trust the source of the "script," then you have a whole host of other problems to think about besides whether or not to use EVAL.. :-)

My 2 cents.

Jeff M.

On Wed, Feb 20, 2013 at 1:27 PM, David McClain <david@acudora.com<mailto:david@acudora.com>> wrote:

>> Anyone else have a better idea?  Has to evaluate in the current
>> dynamic environment and the null lexical environment, just like EVAL.
>
> Then just use EVAL.  This is the reason why EVAL is frowned upon, but if
> that's what you need then you need EVAL.
Yes, every now and again, I revisit old prescriptions...  like "never use EVAL", "never use global vars", "never use goto"... I'm old enough to have been through a number of revolutions, starting with the advent of "structured programming", "object oriented programming", and so on.

There was no "C" compiler when I started -- just PL/1 and Assembly, Cobol, Snobol, and RPG... We went through some great times, but sad to say, things often get pushed beyond the point of reasonableness. C++ now looks like a pig in lipstick.

Whenever I hear a rule repeated in my head, I need to ask why? what was the context of the time? What were the reasons for believing these edicts?

Just trying to clear out old cobwebs...

- DM


_______________________________________________
Lisp Hug - the mailing list for LispWorks users
lisp-hug@lispworks.com<mailto:lisp-hug@lispworks.com>
http://www.lispworks.com/support/lisp-hug.html



_______________________________________________
Lisp Hug - the mailing list for LispWorks users
lisp-hug@lispworks.com
http://www.lispworks.com/support/lisp-hug.html


Re: On the road to replacing C++

Hi Gerry,

I really enjoy the quality discussion we have on this forum, and wanted to contribute with a bit of my experience with Lisp.

I come from a Mac OS X / Cocoa background.  For a year I worked on the Windows port of an application I wrote for Mac OS X.  The Windows expert I was working with advised on using C++ with the Boost libraries, because at this moment it seemed the most obvious, popular and supported common denominator for the Windows and OS X platforms.

This project has been a failure on the technical side, but it has been very enlightening for me.  C++ was not the right choice for me, I won't discuss further about this language and the reasons of its popularity.


On 16 févr. 2013, at 23:54, Gerry Weaver <gerryw@compvia.com> wrote:

> I am finding that a lot of the Lisp libraries/packages available on the web are not exactly written with the system programmer mindset. Some of them are beautiful and very educational, but they lack a focus on performance, scalability, and maintainability. I am not one to reinvent the wheel unless there is no other option, but it’s beginning to look like I’m going to have to start pretty close to the OS on a lot of the things I need. I’m wondering if y’all would agree with my findings or if you could possibly point me to some Lisp packages that are more in the vain I described.

In my opinion, the problem with the library approach is that most of the time the author had a subtly different goal than yours when he designed his library.  Whenever possible, I try to minimize dependancies; of course there are some libraries I wouldn't even dare writing myself.

I have not been convinced Quicklisp would be useful for my needs at least in its current state.  My biggest problem with Quicklisp is that you can't be sure when a component will be updated.  This is important for reliability reasons, if a library is used in several projects, you want to keep the one that worked with your code until you decide to update your work.

The another reason is related to dependency minimization.  It's too easy with Quicklisp to load libraries that are on top of a multi-layered cake of dependencies.  Therefore I don't use it, or only when I need to sketch a quick project and I don't want to crawl through the dependancies.

As I said, ASDF is still needed when I need to use third-party code.  I have set-up ASDF so it knows where to find all my ASDF loadable libraries, it works like this:

- create the directory ~/.config/common-lisp/
- in this directory write the following in a file called source-registry.conf:
(:SOURCE-REGISTRY :INHERIT-CONFIGURATION (:TREE "~/lisp/asdf/"))

- where ~/lisp/asdf/ is the directory where I store ASDF loadable libraries.


When I find a library that I like and I want to use, I search for an online repository (Git for example) and clone it to the ASDF libraries directory.  This is a bit of work but I have less than a dozen libraries so once it's done you don't have to do it again.  If I can't find an available repository, I just create a local git repository and manage it myself, it's just a matter of recording the versions when I update the code.

The code I write uses defsystem.  I find it faster, less verbose and better integrated with the IDE.  If need to load ASDF dependancies, I create a file like in the following example:

    ;; dependancies.lisp
    (require "asdf")
    (asdf:load-system :CXML)
    (asdf:load-system :CL-PPCRE)


ASDF is great but if you write code for yourself on LispWorks, then you don't need to use ASDF or Quicklisp.
Try to reduce entropy at its minimum.


> I am currently working on some benchmarks that will hopefully help me figure out the depth and scope of the C layer that will be necessary to accomplish the goal of replacing C++. I’m wondering if y’all could share some of your experiences with me that involve using the existing Lisp i/o facilities available (ie; disk, network) and the smp/threading capabilities of LispWorks and other. I do plan to do benchmarks, but I think it would be great to hear from y’all, due to my being so new to Lisp. Would it be considered acceptable to post code fragments to this list to get a second opinion on code that I think should run faster, but doesn’t?

I am sure the fine members of this list will be happy to help you.


> Interfacing Lisp with C is something that I’m going to be working on a lot. I’m curious about the overhead involved in calling C functions from Lisp and vice-versa. How does the Lispworks FLI compare to CFFI and others? Which one would y’all use and why? The particulars of Lisp/C and C/Lisp behavior in multi-threaded applications is also an area that I going to need to understand pretty well. The LispWorks docs provide some good information on this, but I trying to distill it down to some simple rules for access between Lisp/C and C/Lisp where threads are involved. Are there any docs/papers around that address this in good detail?

I've had excellent results with foreign code interfacing.  If you don't need to write portable code, don't use CFFI, the LispWorks FLI has everything needed to interface with C, not more not less.

I have recently discovered FLI:DEFINE-POINTER-TYPE which is something I haven't found in CFFI.  This feature is not very well documented, basically is lets you extend the notion of pointer as a struct.  This way you can attach data related to a pointer instead of wrapping the pointer with a structure, you pointer is the structure.

Regarding the multi-threaded aspect of FLI, it is not much different from when you write a C application with multiple threads.  Foreign created threads are properly turned into Lisp processes and Lisp processes can execute foreign code with no problem.


Finally, please allow me to cite two items that may help you during your Lisp adventure:

One of the most important concept I learned from Common Lisp is the conditions system.  I warmly encourage you to learn how it works, and spend some time experimenting.  Read why and how Ken Pitman designed it [1], this article by the way finely explains the differences between handler-bind and handler-case, which are not obvious despite the similar name.

Another thing I wouldn't miss mentionning is the excellent MP library.  With mailboxes and process you are near to have everything you need to create multi-process, performant and reliable systems.


Best,
Cam

[1] http://www.nhplace.com/kent/Papers/Condition-Handling-2001.html


_______________________________________________
Lisp Hug - the mailing list for LispWorks users
lisp-hug@lispworks.com
http://www.lispworks.com/support/lisp-hug.html


Re: On the road to replacing C++

Camille Troillard <camille@osculator.net> writes:

> I have not been convinced Quicklisp would be useful for my needs at
> least in its current state.  My biggest problem with Quicklisp is that
> you can't be sure when a component will be updated.  This is important
> for reliability reasons, if a library is used in several projects, you
> want to keep the one that worked with your code until you decide to
> update your work.

Quicklisp does actually have provisions to lock in a set of consistent
versions, but unfortunately it's poorly documented. The only way to know
about it is from reading the source code, which is not an acceptable
form of documentation IMO.

Quicklisp will never update libraries on its own. It must always be
triggered by (ql:update-dist "quicklisp") or (ql:update-all-dists). 

What is more likely to happen is that different installations of
Quicklisp have differing sets of libraries; by default, Quicklisp
installation will grab the latest dist. So you might have computer A
with quicklisp dist "2012-08-11" from when you started a project in
August, but computer B has quicklisp dist version "2012-12-23" with
newer libraries, and computer C has an even more recent version
"2013-02-17".

However, you can synchronize which sets of libraries are in use across
multiple computers like so:

Find out which version you want to lock to, by running the following on
the "critical" system:

  (in-package ql-dist-user)
  (ql-dist::canonical-distinfo-url (dist "quicklisp"))

This will yield something like
"http://beta.quicklisp.org/dist/quicklisp/2012-08-11/distinfo.txt". Then,
on the systems which should use the same sets of libraries, use this:

  (in-package ql-dist-user)
  (install-dist "http://beta.quicklisp.org/dist/quicklisp/2012-08-11/distinfo.txt" :prompt nil :replace t)
  (inhibit-subscription (dist "quicklisp"))

That is one way to ensure that each system with Quicklisp is using the
same set of libraries, and that nothing is updated even if
(ql:update-dist "quicklisp") is called.

I would also like to add a system for exporting a set of
Quicklisp-supplied libraries to a directory structure so they can be
bundled up by some other non-Quicklisp process, e.g. for delivery in an
application, so Quicklisp is not needed at build- or run-time.

There are a number of other things to improve to make this process
smoother. I'm slowly working on them.

Hope this helps,
Zach

_______________________________________________
Lisp Hug - the mailing list for LispWorks users
lisp-hug@lispworks.com
http://www.lispworks.com/support/lisp-hug.html


Re: On the road to replacing C++

Another thing worth considering is having quicklisp crawl through the ASDF dependencies for version info, and supply appropriate versions of systems if specified.  I've not used it, but the ASDF manual suggests that you can define version info in the system definition. 

This would be helpful in supporting fine grain control over version control; suppose I have two projects, one I want locked to a particular set of libraries, another is a scratch project or something I want up-to-date.

Nick Patrick


On Thu, Feb 21, 2013 at 1:26 PM, Zach Beane <xach@xach.com> wrote:

Camille Troillard <camille@osculator.net> writes:

> I have not been convinced Quicklisp would be useful for my needs at
> least in its current state.  My biggest problem with Quicklisp is that
> you can't be sure when a component will be updated.  This is important
> for reliability reasons, if a library is used in several projects, you
> want to keep the one that worked with your code until you decide to
> update your work.

Quicklisp does actually have provisions to lock in a set of consistent
versions, but unfortunately it's poorly documented. The only way to know
about it is from reading the source code, which is not an acceptable
form of documentation IMO.

Quicklisp will never update libraries on its own. It must always be
triggered by (ql:update-dist "quicklisp") or (ql:update-all-dists).

What is more likely to happen is that different installations of
Quicklisp have differing sets of libraries; by default, Quicklisp
installation will grab the latest dist. So you might have computer A
with quicklisp dist "2012-08-11" from when you started a project in
August, but computer B has quicklisp dist version "2012-12-23" with
newer libraries, and computer C has an even more recent version
"2013-02-17".

However, you can synchronize which sets of libraries are in use across
multiple computers like so:

Find out which version you want to lock to, by running the following on
the "critical" system:

  (in-package ql-dist-user)
  (ql-dist::canonical-distinfo-url (dist "quicklisp"))

This will yield something like
"http://beta.quicklisp.org/dist/quicklisp/2012-08-11/distinfo.txt". Then,
on the systems which should use the same sets of libraries, use this:

  (in-package ql-dist-user)
  (install-dist "http://beta.quicklisp.org/dist/quicklisp/2012-08-11/distinfo.txt" :prompt nil :replace t)
  (inhibit-subscription (dist "quicklisp"))

That is one way to ensure that each system with Quicklisp is using the
same set of libraries, and that nothing is updated even if
(ql:update-dist "quicklisp") is called.

I would also like to add a system for exporting a set of
Quicklisp-supplied libraries to a directory structure so they can be
bundled up by some other non-Quicklisp process, e.g. for delivery in an
application, so Quicklisp is not needed at build- or run-time.

There are a number of other things to improve to make this process
smoother. I'm slowly working on them.

Hope this helps,
Zach

_______________________________________________
Lisp Hug - the mailing list for LispWorks users
lisp-hug@lispworks.com
http://www.lispworks.com/support/lisp-hug.html


Re: On the road to replacing C++

Nicholas Patrick <npatrick04@gmail.com> writes:

> Another thing worth considering is having quicklisp crawl through the ASDF
> dependencies for version info, and supply appropriate versions of systems
> if specified.  I've not used it, but the ASDF manual suggests that you can
> define version info in the system definition.
>
> This would be helpful in supporting fine grain control over version
> control; suppose I have two projects, one I want locked to a particular set
> of libraries, another is a scratch project or something I want up-to-date.

When I want that, I tend to have a quicklisp installation directly in
the "locked" project's directory.

Zach

_______________________________________________
Lisp Hug - the mailing list for LispWorks users
lisp-hug@lispworks.com
http://www.lispworks.com/support/lisp-hug.html


Updated at: 2020-12-10 08:35 UTC