Project: automated testing of libraries with ECL
Written by jjgarcia on 2008-08
Dear lispers,
this message is for not only for ECL users looking for a place to contribute, but also for developers of libraries that want them to be tested with our implementation of Common Lisp.
We have reached a point where the ANSI compatibility test suite is not enough. Undocumented or undefined behavior that changes between releases may cause some libraries to break in unexpected ways -- we have plenty of messages in the mailing list about them.
So my idea is to add new directories to ecl-test for automating the downloading and testing of other libraries on a daily, weekly or monthly basis, depending on how things move on.
As a guidance you may have a look at how ecl-test/ansi is implemented: Makefile checks for the existence of that directory and otherwise checks out with cvs a fresh new copy of the test suite. Tests are then loaded and run, and the output is logged into output/ecl/ansi.log. We could have similar directories for cffi, babel, flexi-streams, and other packages. It should not be that difficult.
However, given that this is to be run on at least five different platforms, a few restrictions apply:
1) Source code can only be retrieved using cvs, ftp or wget/curl 2) Running the tests should not take more than half an hour on a computer with few resources: 512Mb, P4 (this is a typical configuration for my virtual machines) 3) The test suite should be comprehensive enough. 4) If possible, libraries should not have other dependencies. 5) One should only test stable releases.
If neither of these points is satisfied, then I do not see the point on going any further. However, not all restrictions are equally important. Points 1 and 3 are crucial, and 3 most so: why running a test suite when it is not able to produce a meaningful answer? Point 4 is not so relevant. If it cannot be satisfied by all libraries then the design might become a bit more complex. Point 5 can be negotiated: libraries that wish to be tested before release could be configured to look for an almost stable branch of the code.
What I provide is a rather powerful computer (quad-core, 64 bits, 8GB), running kvm with at least the following virtual machines:
- Windows XP + Microsoft Visual Studio 2008
- Ubuntu Linux 64bits
- Fedora 32bits
- FreeBSD
- NetBSD
- OpenBSD
Interested parties should contact us at the mailing list.
Juanjo
