New features in darcs elephant-1.0 head
February 7th, 2009
- Alex fixed bugs in the postmodern backend and added new
- Ian promoted a new CLP backend which is an all-lisp solution based
on cl-prevalence and cl-containers that is currently at Alpha quality.
It requires the latest patches to cl-containers and a Mercurial fork
of cl-prevalence which can be retrieved using 'hg clone
- We are deprecating the CLSQL backend by moving it into contrib; it
will not be a part of the 1.0 release. You can still use it by
creating a symlink in the elephant root directory to the appropriate
ele-xx files in src/contrib/rread/db-clsql
Elephant 1.0 Alpha 2
January 15th, 2009
Bug fixes and minor feature enhancements
Elephant 1.0 Alpha 1
January 3rd, 2009
Significant development work has been taking place on the darcs
elephant-unstable repository over the past year. The last stable
release (0.9.1) is quite stale, lacking many bug fixes and features
that are part of the upcoming 1.0 release.
We've decided to release a stable snapshot of the development tree
and recommend that all users engaged in active development switch to
this version of Elephant. This includes changing your target darcs
repository from elephant-unstable to elephant-1.0.
While the Alpha 1 snapshot has not undergone rigorous testing, it
is in daily use in a number of live applications and our test suites
are currently green for all 0.9.1 and many 1.0 features on all data
stores on SBCL/Linux. We invite users of other platforms to help
expand the scope of testing of the 1.0 tree to ensure that any
platform specific bugs are resolved prior to a final release.
The following new features have been introduced since our last
release, 0.9.1, in November 2007.
- New slot types
- Hierarchical indexed slots
- Indexed slots can now index a base class slot as well as all subclasses inheriting that slot. Index inheritance can be inhibited by redefining the slot in a base class.
- Virtual derived index slots
- A better way to manage derived indexes (the old derived index model is deprecated). You can define a virtual slot that is used to create the index. A slot argument defines the function of slot values that populates the index. Hints are available that inhibit updates to only occur when certain slots are written.
- Set-valued slots
- A simple API to an unordered aggregate slot using the pset api for access
- Many-to-1 and many-to-many associations between slots in two different classes. They also uses a variation of the pset api.
- 'Checkout' slot caching
- A simple means for runtime switching between persistent and transient slot behavior with the ability to sync transient values back to the store. Can buy up to 100x in read/write throughput at the cost of automatic thread safety
- Dynamic schema evolution
- Global persistent instance map (all instances are reachable unless deleted)
- Each class has a DB schema that stores the class structure
- Multiple schema versions of a class may exist in the database concurrently
- Upgrade of DB instances to the latest schema can be done greed or lazily
- Dereferencing pointers to deleted instances returns nil; this is a lazy form of referential integrity.
- Berkeley DB Store
- BDB 4.7 support (now recommended)
- Support to globally enable MVCC from my-config.sexp;
:mvcc t to
:snapshot t in
- Read/write performance has been improved by up to 2x
- db_deadlock support deprecated in favor of built-in operators (no external process needed)
- Postmodern Store
- Data caching within large transactions
- Caching between transactions on a per-thread basis
- Classes instances can exist in more than one store
- Multi-store operations are safer (transaction interleaving, for example)
- Serializer performance enhancements and bug fixes
- Migration bug fixes as well and upgrade support from 0.9.1
- Too many bug fixes and small enhancements to enumerate...
- Garbage Collection (prototype)
The code base contains a working prototype of an online garbage
collector using a mark and sweep strategy. Testing and documentation
are not complete but you can experiment with it. Details are available
on the mailing list archive from Jan 5th, 2009. Complete functionality
will be available in the final release.
- Query Language and Interpreter (prototype)
We also have a working sketch of a simple query language for
elephant (query.lisp) that uses indices and a preliminary sketch of a
query planner and interpreter (query2.lisp) that does something akin
to joins. A modestly optimized form of this will be available as part
of the final 1.0 release. Comments on the simple constraint language
will be appreciated.
There is still much to do to iron down the final release:
- Associations still contain a few bugs, so we don't recommend depending on them yet.
- Derived index slots haven't been extensively used or tested
- Cached slots have not been extensively tested for corner cases, but do have some modest regression tests to validate their basic functionality and should be in good shape for single-threaded use
- Schema evolution There are also some minor problems that can crop up occasionally with schema evolution, but there are almost always workarounds so if you run into trouble e-mail elephant-devel
- Garbage collection has not been fully validated and should not be considered reliable
- Support a full constraint language specification in the query interface
- We need better test coverage (particularly new features)
- Documentation of new features
Ian Eslick has primary responsibility for the features in this
release. Alex Mizrahi, Henrik Hjelte and Robert Read have put in
significant work upgrading and improving the SQL backends. Special
thanks to Leslie Polzer for various contributions and picking up some
of the support slack.
Other noteable contributors to this release in the form of small patches,
tracking down hairy bugs, suggestions and documentation include:
Christoph Ludwig, Anton Kazennikov, Glen Tarcea, Frank Schorr, Daniel
Salama, Lukas Giessmann, Patrick May, Quan Hu, Red Daly, Richard
Szopa, Sean Ross, Tiarnan O'Corrain, and Elliott Slaughter. Our
apologies if we have left anyone out (email eslick at media mit edu if
Elephant 0.9.1 now available
November 16th, 2007
Elephant 0.9.1 is now officially released.
This release provides official support for using the postmodern
interface to Postgres as a back-end store. This back-end is about
three times faster than the generic SQL interface on top of Postgres.
Note that existing Postgres repositories are not compatible with a
Postmodern repository. However, Elephant supports multi-repository
migration. If you have an existing Postgres repository and want to
migrate to Postmodern, you should create a completely new Postmodern
repository and migrate all of your data into it. The migration can be
accomplished with a single command.
If you have been using the (unofficially released) postmodern backend,
you will have to migrate your data into a CL-SQL based postgres backend,
or into a BDB backend, and then migrate into your new postmodern
repository. This release is not compatible with previous (development)
versions of the postmodern functionality.
This release has been tested on a 64-bit Linux with SBCL 1.0.10, and
other platforms as well. If anybody downloads it and tests it, please
run the (large) set of automated tests and report your results to
elphant-devel at common-lisp.net. We will update the platforms page as we receive your updates.
Elephant 0.9 now available
June 7th, 2007
Elephant 0.9 is a significant improvement in stability, platform
support and feature set over Elephant 0.6.0, our last major release.
This release should be considered a stable release, and any major bugs
that are uncovered will be accommodated by a patch release.
Major Features of Elephant 0.9
- Expanded and improved platform support
- SBCL 1.0, Allegro 8.0/7.0, Lispworks 5.0 and OpenMCL 1.0
- Elephant has been verified on every major OS (Linux, Mac OS X or Windows) for which these lisps are available
- New support for 64-bit lisps and improved support for Unicode
- Elephant is now thread-safe under both BDB and SQL data stores
- A wide set of configurations have now been validated.
- New datastructures and interfaces
- Persistent sets - lightweight collections with a simple API
- Mapping operators - Efficient, cons free access to btrees, sets and class indices
- All new users manual
- A new, basic tutorial incorporating the major new features
- Expanded user guide for specific features
- A new rough guide to the internal APIs for developers
- Other benefits or features
- Simplified user configuration using a single file, my-config.sexp
- Proper upgrades from 0.6, via the ability to open 0.6 databases in 0.9
- The Berkeley DB backend is up-to-date with the latest library version and requires Berkeley DB 4.5
- A host of performance improvements, bug fixes and minor feature enhancements (see the development list or prior announcements for some of the details)
We are planning a string of interim releases between 0.9 and 1.0, but
we recommend that people using Elephant for active web sites and
important applications use 0.9 until 1.0 is available. We will
not guarantee clean upgrades between 0.9.x releases if the database
format changes and cannot guarantee the same level of stability
outside these major release milestones.
The features planned for the 1.0 release of Elephant are recorded as tickets
the Trac project
Visit the documentation page to read the new reference manual
and the download page to get the release.
Elephant 0.9 RC1 announced
April 30th, 2007
With the marked improvement in stability and functionality of the code
base, the maintainers have decided to abandon the 0.6.1 release label
and change the upcoming release to 0.9. This change puts us on a new
leading to a 1.0 release later in 2007.
New features and improvements in release 0.9 RC1
Additional features integrated since the Beta 0.6.1 announcement:
- Full platform support
- SBCL 1.0, Allegro 8.0/7.0, Lispworks 5.0 and OpenMCL 1.0
- All these lisps should work on every major OS (Linux, Mac OS X, Windows) they are released on
- A wide set of configurations has been tested.
- 64-bit support has been validated, but not recently and not on all platforms (testers welcome!)
- Mapping operators
- BTrees, BTree indicies, class index and slot/derived indices all have a map operation
- Enable mapping a function over values and ranges
- Map in both ascending and descending order
- Collect function results into a list
- Efficient stack allocation where appropriate
- Persistent sets
- A lighter-weight API for persistent collections
- Provides insert, remove, find, map and list operations
- With appropriate methods, makes one to many slots easy to construct,
see tutorial and user-guide for details.
- Complete update and rewrite of the users manual
- All new and up-to-date tutorial
- New section containing more detailed implementation information for advanced users
- New sections for prospective data store developers
- Detailed installation instructions (supercedes INSTALL file)
- A list of use case scenarios that should help developers understand Elephant better
- Much work remains to clean up the manual for 1.0, but the bulk of the rewrite is done
and we invite comment on what is missing or unclear.
- Feature tweaks
- Drop commands for persistent collections to reclaim storage for reuse.
- Expanded support for nested aggregates in serializer and
migrate (i.e. an array of persistent objects will be
properly migrated, before only btree collections of
persistent objects were properly copied).
- Better error handling during object creation, change class operations, etc.
- Better error checking during cross-store operations.
- Persistent instances passed to print-object print their oid by default to create a recognizable signature
- And much more...
- Performance enhancements
- Option to increase cache size for BDB backend
- Improve contention performance while mapping btrees
- Improved map performance
- And much more...
Beta 0.6.1 Release available via CVS
March 21st, 2007
- See the platform page for details
- OpenMCL will be verified as part of the beta cycle
- CMU should work, but there are no users so it is not officially supported
Download and Install
The beta release will only be available via CVS. Download the latest development HEAD.
cvs -z3 -d :pserver:anonymous:email@example.com:/project/elephant/cvsroot checkout elephant
This will put the elephant release into the directory 'elephant'. Read the INSTALL, UPGRADE and UPGRADE-BDB files for further instructions.
Purpose of the beta release
The tree is effectively a release candidate. We only need to verify OpenMCL, beef up the documentation and examples and make sure we have good test coverage. Otherwise this code base appears stable enough to release.
New features in 0.6.1
The beta release is a more stable version of the feature set in the alpha release. A list of the changes between alpha and beta can be found in the doc/oldfiles/TODO file.
Alpha 0.6.1 Release available via CVS
February 20th, 2007
- SBCL, Allegro, CMU, OpenMCL, Lispworks (build system is not automated for Windows)
- CMU, OpenMCL and Lispworks have not been fully tested and may require minor bug fixes
Download and Install
The alpha release will only be available via CVS
cvs -z3 -d :pserver:anonymous:firstname.lastname@example.org:/project/elephant/cvsroot checkout -r ELEPHANT-0-6-1-alpha elephant
This will put the elephant release into the directory: elephant. Read the INSTALL, UPGRADE and UPGRADE-BDB files for further instructions.
Purpose of the alpha release
The primary developers do not have ready access to all supported platforms and would like to ask the community to help validate the current implementation on other platforms. We also are using the alpha to improve up our test suite. For more details please see the TODO
file. If you have a favorite feature such as multi-threading, 64-bit, etc., please free to submit a test for the test suite.
New features in 0.6.1
- Simplified build and site configuration support:
- See config.sexp in root directory for site customization (no more editing code files)
- Linux and Mac systems should automatically build libraries when asdf is called
- 64-bit lisps are now supported
- 64-bit and 32-bit lisps can read and run off the same database files (on the same machine)
- Elephant should now be thread safe, including sharing a single store-controller across threads
- Read comments in src/elephant/transactions.lisp and BDB users should read src/db-bdb/bdb-transactions.lisp
- Improved support for mixing transactions and store-controllers
- Elephant 0.6.1 can open and directly manipulate 0.6.0 databases
- Existing 0.6.0 databases can be upgraded via the 'upgrade' function which takes
- Upgrading is required for 64-bit systems
- New serializer-independant metadata should enable future upgrades easier
- NOTE: There may be some problems upgrading SQL databases
- Berkeley DB backend
- BDB 4.5 required (see UPGRADE-BDB)
- *auto-commit* is no longer required. All data access methods auto-commit if there is no active transaction
- store-controller accepts :deadlock-detect keyword (t or nil) which will run db_deadlock as a background process to abort deadlocked threads
- optimize-storage is a new store-controller method currently supported by BDB backend. It compacts a whole database or only a specific BTree and returns free pages to the free list or to the file system.
- Minor features
- Re-organization to the internal structure in this release
- Renaming including removing defunct sleepycat naming scheme, backend packages, etc.
- The serializer was modularized to allow future releases to change serializer
- strategies or implement custom serializers and to be able to open legacy databases
- Remove various warnings in SBCL build, etc.
- Separated utilities into their own package and directory
- Serializer improvements
- 0.6.1 databases can be shared across lisps running on hardware of the same endianness
(i.e. all x86 platforms or PPC/Alpha, etc)
- Simplified unicode serialization support across all platforms
- Performance improvements in multi-threading situations
- Pushing :elephant-without-optimize onto *features* will disable optimization declarations and simplifying debugging
- Other fixes and features documented in the TODO file
See the downloads page
for more information.
Release 0.6.0 Available
May 27th, 2006
The latest version is 0.6.0, is offically released as of May 27, 2006. It has been tested on a variety of lisp implementations and with a variety of storage repositories. This release is mainly the work of Ian Eslick, and offers a significant refactoring as well as the very convenient feature of automatic slot indexing. Julian Stecklina submitted a patch for FreeBSD which is in the CVS repository now but not in the 0.6.0 release.