Common Lisp Librarian

Library dowloader and manager for Common Lisp.

Table of Contents


One problem I always had when hacking CL were libraries. And it was not the lack of them – contrary to popular belief, there is quite a lot of Lisp libraries. The real problem is how to comfortably find and manage them. For finding, CLiki and Common Lisp Directory work great. Managing, however, has to be done by hand. There was an attempt to solve this problem, ASDF-Install, but it relies on a very inconvenient (at least for me) assumption: that libraries are versioned .tar.gz files. In other languages it may be true, but Lisp is different (as usual). Many libraries are in a `permanently rolling mudball' state, where the current version is accessible only by various version control systems, and tarballs appear rarely and quickly become outdated.

There are other library collections and collectors:

What is more, all above are targeted towards fresh users and their goal is to provide an easy Lisp installation. There is no `library manager' which would be usable and convenient for the power user, and which would just sit on top of the ASDF managing libraries.

That is what CL-Librarian aims to be: a manager for Common Lisp libraries, focused on libraries updated dynamically by version control systems and targeted at power users. It handles downloading such libraries, and keeping them up-to-date. This may seem like a trivial task, but after maintaining an UnCommon Web installation for over a year I am quite sure it's not. Library management could use some automation.

CL-Librarian's primary platform is SBCL on Linux. It should work with other Lisp implementations (for the only non-portable part: executing external commands, it uses the EXTERNAL-PROGRAM library, which is portable). It should work on other Unix systems, including OS X. Porting it to Windows should not be very hard, but I am not a Windows expert, so most probably it won't work out of the box.

CL-Librarian is copyright (C) 2007-2008 Maciej Pasternacki <> and available on BSD license without advertisement clause. See file =COPYING= for details.

1.1 Note of warning

As of now, this code is under heavy development. Both implementation and interface may change. I try to document fixed interfaces, but I cannot promise much at the moment. When you upgrade Librarian, please read the commit description. You have been warned.

1.2 Download

CL-Librarian lives in a darcs repository. To dowload the source, type:

 $ darcs get

It depends on, and Those can be installed manually, or – on UNIX-like systems that use symlinks – the included script can be used to download dependencies (except ASDF, which is included in all CL implementations that I use) and .asd files into Librarian's main directory:

 cl-librarian$ sh ./

Librarian can be also downloaded with

1.3 Required binaries

CL-Librarian makes extensive use of system binaries. Required binaries are:

As far as I know, all of them are ported to most of commonly used systems.

1.4 Contact

CL-Librarian is written and maintained by Maciek Pasternacki <>. Feel free to contact me by e-mail or jabber (JID is the same as e-mail). After sending me an e-mail from an address previously unknown to me, a confirmation message will be sent back; only after the sender replies to this message I will see the original mail.

There is no mailing list at the moment.

2 Documentation

CL-Librarian is just an ASDF library. Once it gets its prerequisites, either with, ASDF-Install, or by loading the necessary libraries by hand, it doesn't need any further set up other than being loaded:

 (asdf:operate 'asdf:load-op :cl-librarian)

2.1 Library shelf model

CL-Librarian is designed around the concept of a library shelf. A shelf is a single unit consisting of several ASDF-defined libraries managed by the Librarian. The Librarian doesn't bother moving around single libraries; only whole library shelves, which are self-contained sets of libraries, are downloaded or updated at a time.

Each shelf consists of library repositories. Libraries are distinguished with names (symbols); it is assumed that name identifies a library uniquely, and that it is the same as library's main .asd file. However, single library can have multiple sources (e.g. stable tarball from release site or ASDF-Install, and development darcs repository). Shelf can contain only one source of every library.

Shelves and library repositories can be defined either in Lisp files with .shelf extension placed in directory specified by `*SHELVES-DIR*' variable (by default, shelves subdirectory of Librarian installation directory), or inline in normal Lisp code. First method is loosely based on ASDF's .asd files.

A shelf specification is a single Lisp file, with the .shelf extension, containing a DEFSHELF form. They resemble ASDF's .asd files with DEFSYSTEM forms, which is intentional. Shelves managed by Librarian are located in the shelves subdirectory. CL-Librarian is capable of downloading shelves from the network or copying them from other files. The shelf contents are a directory, located in shelves subdirectory of Librarian, whose name is the basename of .shelf file. It contains one subdirectory per each library it includes.

A shelf can include other shelves. By these I mean literal inclusions rather than inheritance – each shelf has its own copy of every library it directly or indirectly includes. This is to keep the shelf's internal dependencies consistent and to make it possible to upgrade one shelf without touching others.

A sample Librarian's description of a shelf, describing libraries used by the Librarian itself, looks like this:

(defshelf librarian-base ()
  ((external-program tarball-repo :source "asdf-install:external-program")
   (split-sequence tarball-repo :source "cclan:split-sequence.tar.gz"))

DEFSHELF form above defines a shelf, named LIBRARIAN-BASE, including no other shelves, and consisting of the EXTERNAL-PROGRAM and SPLIT-SEQUENCE packages, downloaded as .tar.gz file (`tarball') from CClan and ASDF-Install link at CLiki.

2.2 Finding and downloading shelves

Shelves are distinguished by names, which are symbols. As in ASDF systems, symbols are distinguished only by their name, regardless of their package. In order to find and load a shelve definitions we use the FIND-SHELF function:

(defun find-shelf (name &key force-reload)
  "Find shelf `NAME', returning its shelf object.

`NAME' can be a symbol or string.  If it's a symbol or string not
containing the colon and dot characters, it names the shelf that is
already managed.  If the shelf isn't already loaded, or `FORCE-RELOAD'
is true, an appropriate .shelf file from `*SHELVES-DIR*' directory is
loaded, and the shelf object is returned.

If `NAME' is any other string, it is treated as an URL or full pathname
to a .shelf file.  The file is downloaded or copied into
`*SHELVES-DIR*' and then loaded."

For downloading contents of shelved libraries, Librarian exports DOWNLOAD-SHELF function:

(defun download-shelf (shelf)
  "Download libraries included in `SHELF' into shelf's subdir of `*SHELVES-DIR*'.

`SHELF' may be a SHELF instance, or a symbol, or string.  If it's
a symbol or string, `FIND-SHELF' is invoked to get the SHELF

2.3 Using shelved libraries

CL-Librarian provides SEARCH-SYSTEM-IN-SHELVES function that hooks into ASDF:*SYSTEM-DEFINITION-SEARCH-FUNCTIONS* and is used to find single ASDF systems among used shelves. List of used shelves is stored in *USED-SHELVES* and can be examined and updated manually. Convenience functions (USE-SHELF SHELF) and (UNUSE-SHELF SHELF) are also provided.

2.4 TODO Keeping shelves up to date

2.5 TODO Customizing Librarian


2.6 TODO Defining shelves

Example shelves are in shelves/

2.7 TODO Extending Librarian

3 Hacking

3.1 DONE 2007-11-01 Thu Separate library and shelf definition

for easier using of the same library in different shelves

3.2 BUG Invent way of updating tarball libraries

3.3 BUG Detect include loops

3.4 TODO [#A] Do update shelf contents

3.5 DONE 2007-11-01 Thu Convenient using and unusing shelved libraries

2007-11-01 Thu Separate shelf and systems directories. USE-SHELF and UNUSE-SHELF convenience functions added.

3.6 BUG Check authenticity of downloaded libraries.

No authenticity checking is currently done on downloaded libraries. Care must be taken to make sure if a correct, trusted source is used and to check downloaded library's integrity.

3.7 TODO Checking shelf's completness

3.8 TODO Running shelf contents' ASDF:TEST-OP

3.9 DONE Explicitly set download dir

CLOSED: 2007-12-16 Sun 23:15

3.10 DONE Implicit automagic using shelf object or name

CLOSED: 2008-03-05 Wed 11:25
(WITH-SHELF (name) …) -> (let ((name (find-shelf name))) …)

3.11 TODO Support for other VC systems

3.11.1 TODO [#A] git

3.11.2 TODO Mercurial

3.11.3 TODO Bazaar-NG

3.12 TODO Updating included subshelves only

3.13 Skel

3.13.1 TODO Support for dumping Lisp image

Author: Maciej Pasternacki <>

Date: 2008/10/18 23:12:10