test frameworks
Unable to parse email body. Email id is 11556
Unable to parse email body. Email id is 11556
I need a test framework for regression-testing and unit-testing some
libraries and applications written using LispWorks. Suggestions? I
have read this:
<http://aperiodic.net/phil/archives/Geekery/notes-on-lisp-testing-frameworks.html>
and looked through quite a few of the linked pages, as a result of
which I am favouring FiveAM, and thinking of building some
infrastructure for testing CAPI widgets.
Nick B
Hi, Just out of curiosity, what's wrong with just running some Lisp scripts calling assert? Pascal Sent from my iPad On 18 Apr 2012, at 22:41, Nick Barnes <Nick.Barnes@pobox.com> wrote: > > I need a test framework for regression-testing and unit-testing some > libraries and applications written using LispWorks. Suggestions? I > have read this: > > <http://aperiodic.net/phil/archives/Geekery/notes-on-lisp-testing-frameworks.html> > > and looked through quite a few of the linked pages, as a result of > which I am favouring FiveAM, and thinking of building some > infrastructure for testing CAPI widgets. > > Nick B >
Unable to parse email body. Email id is 11560
On 19 Apr 2012, at 10:16, Nick Barnes wrote: > At 2012-04-19 04:51:03+0000, Pascal Costanza writes: > >> Just out of curiosity, what's wrong with just running some Lisp scripts calling assert? > > The same things that are wrong with using wget/fetch/curl as a web > browser, or with programming in machine code. I don't think the analogy holds. In my experience, setting up test suites for languages like Java, Smalltalk, C++, etc., is pretty difficult and benefits strongly from testing frameworks. On the other hand, I never experienced the need for a test framework in Lisp, because writing the necessary test scripts that just call 'assert in the right places is pretty straightforward, and substantially easier to do than in those languages. So to rephrase my question, under what circumstances is testing without additional support even in Lisp difficult? Pascal -- Pascal Costanza
Unable to parse email body. Email id is 11563
On 19 Apr 2012, at 14:21, Nick Barnes wrote: > At 2012-04-19 12:00:39+0000, Pascal Costanza writes: >> >> On 19 Apr 2012, at 10:16, Nick Barnes wrote: >> >>> At 2012-04-19 04:51:03+0000, Pascal Costanza writes: >>> >>>> Just out of curiosity, what's wrong with just running some Lisp scripts calling assert? >>> >>> The same things that are wrong with using wget/fetch/curl as a web >>> browser, or with programming in machine code. >> >> >> I don't think the analogy holds. In my experience, setting up test suites for languages like Java, Smalltalk, C++, etc., is pretty difficult and benefits strongly from testing frameworks. On the other hand, I never experienced the need for a test framework in Lisp, because writing the necessary test scripts that just call 'assert in the right places is pretty straightforward, and substantially easier to do than in those languages. So to rephrase my question, under what circumstances is testing without additional support even in Lisp difficult? > > Well, in the current instance I want to be able to run many thousands > of regression tests in an automated fashion, and get a detailed > collated report of all unexpected behaviour [*] from the entire run, > distinguishing between detected defects [*] and test system errors > [*]. During this automated run, many different test fixtures and > combinations of fixtures will need to be created and torn down in a > reliable and consistent way. Some tests will have fixed data, others > will need to be randomized, in a manner which is reproducible from > information contained in the report. I will need to be able to > manually run a single test from the suite, or a hierarchical subset, > so both tests and subsets should be named, preferably with meaningful > names. I will need to be able to manually specify parameters to an > individual test or subset of tests. > > Any system like that, in any language, is going to include hundreds of > lines of test framework, not counting the actual test and fixture > code. Although I've built such frameworks in various languages over > the years, and although some languages make it easier than others (it > was easier in Python than in C, and it's probably easier in Lisp than > it was in Python), I'd rather not have to do it again. > > [*] Avoiding the term "test failure", which is ambiguous and also > communicates exactly the wrong message about the purpose of software > testing. A test which detects a defect is a success by any reasonable > definition. OK, thank you very much. This is a very clear and helpful answer. Best, Pascal -- Pascal Costanza
Pascal Costanza <pc@p-cos.net> writes:
> On 19 Apr 2012, at 10:16, Nick Barnes wrote:
>
>> At 2012-04-19 04:51:03+0000, Pascal Costanza writes:
>>
>>> Just out of curiosity, what's wrong with just running some Lisp scripts calling assert?
>>
>> The same things that are wrong with using wget/fetch/curl as a web
>> browser, or with programming in machine code.
>
>
> I don't think the analogy holds. In my experience, setting up test
> suites for languages like Java, Smalltalk, C++, etc., is pretty
> difficult and benefits strongly from testing frameworks. On the other
> hand, I never experienced the need for a test framework in Lisp,
> because writing the necessary test scripts that just call 'assert in
> the right places is pretty straightforward, and substantially easier
> to do than in those languages. So to rephrase my question, under what
> circumstances is testing without additional support even in Lisp
> difficult?
I second the question.
- one of my projects where testing was big, "unit" test frameworks
weren't useful because what we wanted to test was end-to-end and
round-trip of a SMTP based system. I wrote that in emacs lisp (it was
before I switched to CL and that was what motivated me to switch to
CL).
- on another of my project where I needed more than (defun test/feature
() (assert …)… :success), I wrote my own simple test framework in less
than 130 lines of lisp. I lost 10% of that time browsing the
documentation of the other lisp test frameworks, without being
convinced of their usefulness for my project.
Not to say that they are never useful, but the cases are rare IMO.
One thing on the other hand that may be more useful is to provide a test
"target" to our ASDF system. Not a lot of libraries have it.
--
__Pascal Bourguignon__ http://www.informatimago.com/
A bad day in () is better than a good day in {}.
[ snipped lots of stuff ]
Not to say that they are never useful, but the cases are rare IMO.
One thing on the other hand that may be more useful is to provide a test
"target" to our ASDF system. Not a lot of libraries have it.
Nick Barnes wrote on Wed, 18 Apr 2012 21:41:48 +0100 00:41: | I need a test framework for regression-testing and unit-testing some | libraries and applications written using LispWorks. Suggestions? I | have read this: | | <http://aperiodic.net/phil/archives/Geekery/notes-on-lisp-testing-frame | works.html> | | and looked through quite a few of the linked pages, as a result of | which I am favouring FiveAM, and thinking of building some | infrastructure for testing CAPI widgets. Looking at cliki.net and cl-user.net could also be valuable. I would recommend the tester module from the acl-compat library. For a small except from it I use on LispWorks, please see http://lisp.ystok.ru/projects.html#acl-compat -- Sincerely, Dmitriy Ivanov lisp.ystok.ru