FReT is a software package for testing software packages written in Lisp. The intent is to include a variety of testing tools that will be of use to Lisp programmers. We start with a basic framework that provides unit tests. We then build on top of that framework, and provide features for automated testing, historical analysis, etc. More specifically, we would like to develop the following facilities in the longer term:
Defining test suites and test cases. FReT's structure is inspired by JUnit. There are parts of JUnit that I consider braindead, which I've tried to address. FReT defines a class for each test suite, test case and test script. While testing, we instantiate and test an instance of such a test class.
Writing and reading test cases. Any test case or test suite authored in FReT should be easy to read and write from some storage. The most likely storage location is going to be the file system, but it is easy to build an abstraction that will support more. So we should have it. The storage policy should be easy to hook into a system definition facility.
Managing test cases. At the very minimum, FReT should have a console UI capable of operating on existing test suites and test cases. The list of operations should include add, delete, rearrange, modify, inspect, etc.
Running tests. FReT should be able to run test cases both as part of the development process, and off-line. The execution of test cases should tie in nicely with a system definition facility.
Historical analysis. FReT should support the analysis of test runs over time, giving the programmer some insight into the development process itself.
FReT organizes its
test cases
into suites,
and suites may be organized in an
arbitrary inheritance hierarchy. A script
may also be defined that cuts across the inhertance hierarchy to give
us a customized test routine. Before any case is run, the setup fixtures for
the test case are
run. And after the test case has run, the teardown fixtures
are run. The setup
fixtures are intended for setting up the necessary context for running
the test case, and the teardown fixtures should clean up that context.
A test case succeeds if it executes without any error. We provide a set of ensure macros that provide a convenient means of signaling a test failure errors. An error in a fixture is a different type of error. An error here means the test case was unable to run, not just that it failed.
When executing a test object, we first execute its setup fixtures. We then execute the tests contained within the object, and finally execute the teardown fixtures. For a test script, that means executing the setup, followed by the suites and cases listed in the script, and finally the teardown. A test suite isn't executed, rather the individual cases in the suite are executed. The suite merely organizes them. When executing a test case, all the setup fixtures for the suite containing the case are executed, from the most general to the most specific, where general and specific are defined as they would during CLOS finalization. Then the test case is executed. Finally the teardown fixtures for the suite containing the case are executed, from the most specific to the most general. The execution of a suite thus involves gathering all the test cases in the suite, and testing each case within the context created by the suite fixtures.
Please see the test cases in ../test/fret-test-script.lisp to get a feel for how suites and cases are defined and arranged. Support code is provided in ../test-support/framework-tests-support.lisp. The cases here define a test suite that tests FReT's functionality. At present no further documentation is available.
If you are using ASDF to manage FReT, you can also test FReT through ASDF. Load FReT, and execute (fret:test-fret). The ASDF integration of FReT is better than this test suggests, for an explanation see the comments in ../src/asdf-interface.lisp.
Please see the source code. There is no explicit documentation available at present. Also see package.lisp for the exported symbols. This should help focus the search in the source code.