Next: , Previous: The object model of ASDF, Up: Top

8 Controlling where ASDF searches for systems

8.1 Configurations

Configurations specify paths where to find system files.

  1. The search registry may use some hardcoded wrapping registry specification. This allows some implementations (notably SBCL) to specify where to find some special implementation-provided systems that need to precisely match the version of the implementation itself.
  2. An application may explicitly initialize the source-registry configuration using the configuration API (see Configuration API, below) in which case this takes precedence. It may itself compute this configuration from the command-line, from a script, from its own configuration file, etc.
  3. The source registry will be configured from the environment variable CL_SOURCE_REGISTRY if it exists.
  4. The source registry will be configured from user configuration file $XDG_CONFIG_DIRS/common-lisp/source-registry.conf (which defaults to ~/.config/common-lisp/source-registry.conf) if it exists.
  5. The source registry will be configured from user configuration directory $XDG_CONFIG_DIRS/common-lisp/source-registry.conf.d/ (which defaults to ~/.config/common-lisp/source-registry.conf.d/) if it exists.
  6. The source registry will be configured from system configuration file /etc/common-lisp/source-registry.conf if it exists/
  7. The source registry will be configured from system configuration directory /etc/common-lisp/source-registry.conf.d/ if it exists.
  8. The source registry will be configured from a default configuration. This configuration may allow for implementation-specific systems to be found, for systems to be found the current directory (at the time that the configuration is initialized) as well as :directory entries for $XDG_DATA_DIRS/common-lisp/systems/ and :tree entries for $XDG_DATA_DIRS/common-lisp/source/. For instance, SBCL will include directories for its contribs when it can find them; it will look for them where SBCL was installed, or at the location specified by the SBCL_HOME environment variable.

Each of these configurations is specified as an s-expression in a trivial domain-specific language (defined below). Additionally, a more shell-friendly syntax is available for the environment variable (defined yet below).

Each of these configurations is only used if the previous configuration explicitly or implicitly specifies that it includes its inherited configuration.

Additionally, some implementation-specific directories may be automatically prepended to whatever directories are specified in configuration files, no matter if the last one inherits or not.

8.2 Truenames and other dangers

One great innovation of the original ASDF was its ability to leverage CL:TRUENAME to locate where your source code was and where to build it, allowing for symlink farms as a simple but effective configuration mechanism that is easy to control programmatically. ASDF 3 still supports this configuration style, and it is enabled by default; however we recommend you instead use our source-registry configuration mechanism described below, because it is easier to setup in a portable way across users and implementations.

Addtionally, some people dislike truename, either because it is very slow on their system, or because they are using content-addressed storage where the truename of a file is related to a digest of its individual contents, and not to other files in the same intended project. For these people, ASDF 3 allows to eschew the TRUENAME mechanism, by setting the variable asdf:*resolve-symlinks* to nil.

PS: Yes, if you haven't read Vernor Vinge's short but great classic “True Names... and Other Dangers” then you're in for a treat.

8.3 XDG base directory

Note that we purport to respect the XDG base directory specification as to where configuration files are located, where data files are located, where output file caches are located. Mentions of XDG variables refer to that document.

This specification allows the user to specify some environment variables to customize how applications behave to his preferences.

On Windows platforms, when not using Cygwin, instead of the XDG base directory specification, we try to use folder configuration from the registry regarding Common AppData and similar directories. Since support for querying the Windows registry is not possible to do in reasonable amounts of portable Common Lisp code, ASDF 3 relies on the environment variables that Windows usually exports.

8.4 Backward Compatibility

For backward compatibility as well as to provide a practical backdoor for hackers, ASDF will first search for .asd files in the directories specified in asdf:*central-registry* before it searches in the source registry above.

See Configuring ASDF to find your systems — old style.

By default, asdf:*central-registry* will be empty.

This old mechanism will therefore not affect you if you don't use it, but will take precedence over the new mechanism if you do use it.

8.5 Configuration DSL

Here is the grammar of the s-expression (SEXP) DSL for source-registry configuration:

     ;; A configuration is a single SEXP starting with keyword :source-registry
     ;; followed by a list of directives.
     CONFIGURATION := (:source-registry DIRECTIVE ...)
     ;; A directive is one of the following:
         ;; Your configuration expression MUST contain
         ;; exactly one of either of these:
         :inherit-configuration | ; splices inherited configuration (often specified last)
         :ignore-inherited-configuration | ; drop inherited configuration (specified anywhere)
         ;; forward compatibility directive (since ASDF 2.011.4), useful when
         ;; you want to use new configuration features but have to bootstrap a
         ;; the newer required ASDF from an older release that doesn't sport said features:
         :ignore-invalid-entries | ; drops subsequent invalid entries instead of erroring out
         ;; add a single directory to be scanned (no recursion)
         ;; add a directory hierarchy, recursing but excluding specified patterns
         ;; override the defaults for exclusion patterns
         (:exclude EXCLUSION-PATTERN ...) |
         ;; augment the defaults for exclusion patterns
         (:also-exclude EXCLUSION-PATTERN ...) |
         ;; Note that the scope of a an exclude pattern specification is
         ;; the rest of the current configuration expression or file.
         ;; splice the parsed contents of another config file
         ;; This directive specifies that some default must be spliced.
     DIRECTORY-PATHNAME-DESIGNATOR := PATHNAME-DESIGNATOR ;; interpreted as a directory name
         NIL | ;; Special: skip this entry.
     EXCLUSION-PATTERN := a string without wildcards, that will be matched exactly
     	against the name of a any subdirectory in the directory component
             of a path. e.g. "_darcs" will match #p"/foo/bar/_darcs/src/bar.asd"

Pathnames are designated using another DSL, shared with the output-translations configuration DSL below. The DSL is resolved by the function asdf::resolve-location, to be documented and exported at some point in the future.

         STRING | ;; namestring (better be absolute or bust, directory assumed where applicable).
                  ;; In output-translations, directory is assumed and **/*.*.* added if it's last.
                  ;; On MCL, a MacOSX-style POSIX namestring (for MacOS9 style, use #p"...");
                  ;; Note that none of the above applies to strings used in *central-registry*,
                  ;; which doesn't use this DSL: they are processed as normal namestrings.
                  ;; however, you can compute what you put in the *central-registry*
                  ;; based on the results of say (asdf::resolve-location "/Users/fare/cl/cl-foo/")
         PATHNAME | ;; pathname (better be an absolute path, or bust)
                    ;; In output-translations, unless followed by relative components,
                    ;; it better have appropriate wildcards, as in **/*.*.*
         :HOME | ;; designates the user-homedir-pathname ~/
         :USER-CACHE | ;; designates the default location for the user cache
         :HERE | ;; designates the location of the configuration file
                 ;; (or *default-pathname-defaults*, if invoked interactively)
         :ROOT ;; magic, for output-translations source only: paths that are relative
               ;; to the root of the source host and device
         ;; Not valid anymore: :SYSTEM-CACHE (was a security hazard)
         STRING | ;; relative directory pathname as interpreted by parse-unix-namestring.
                  ;; In output translations, if last component, **/*.*.* is added
         PATHNAME | ;; pathname; unless last component, directory is assumed.
         :IMPLEMENTATION | ;; directory based on implementation, e.g. sbcl-1.0.45-linux-x64
         :IMPLEMENTATION-TYPE | ;; a directory based on lisp-implementation-type only, e.g. sbcl
         :DEFAULT-DIRECTORY | ;; a relativized version of the default directory
         :*/ | ;; any direct subdirectory (since ASDF 2.011.4)
         :**/ | ;; any recursively inferior subdirectory (since ASDF 2.011.4)
         :*.*.* | ;; any file (since ASDF 2.011.4)
         ;; Not supported (anymore): :UID and :USERNAME

For instance, as a simple case, my ~/.config/common-lisp/source-registry.conf, which is the default place ASDF looks for this configuration, once contained:

       (:tree (:home "cl")) ;; will expand to e.g. "/home/joeluser/cl/"

8.6 Configuration Directories

Configuration directories consist in files each containing a list of directives without any enclosing (:source-registry ...) form. The files will be sorted by namestring as if by string< and the lists of directives of these files with be concatenated in order. An implicit :inherit-configuration will be included at the end of the list.

This allows for packaging software that has file granularity (e.g. Debian's dpkg or some future version of clbuild) to easily include configuration information about distributed software.

The convention is that, for sorting purposes, the names of files in such a directory begin with two digits that determine the order in which these entries will be read. Also, the type of these files is conventionally "conf" and as a limitation to some implementations (e.g. GNU clisp), the type cannot be nil.

Directories may be included by specifying a directory pathname or namestring in an :include directive, e.g.:

     	(:include "/foo/bar/")

Hence, to achieve the same effect as my example ~/.config/common-lisp/source-registry.conf above, I could simply create a file ~/.config/common-lisp/source-registry.conf.d/33-home-fare-cl.conf alone in its directory with the following contents:

     (:tree "/home/fare/cl/")

8.6.1 The :here directive

The :here directive is an absolute pathname designator that refers to the directory containing the configuration file currently being processed.

The :here directive is intended to simplify the delivery of complex CL systems, and for easy configuration of projects shared through revision control systems, in accordance with our design principle that each participant should be able to provide all and only the information available to him or her.

Consider a person X who has set up the source code repository for a complex project with a master directory dir/. Ordinarily, one might simply have the user add a directive that would look something like this:

        (:tree  "path/to/dir")

But what if X knows that there are very large subtrees under dir that are filled with, e.g., Java source code, image files for icons, etc.? All of the asdf system definitions are contained in the subdirectories dir/src/lisp/ and dir/extlib/lisp/, and these are the only directories that should be searched.

In this case, X can put into dir/ a file asdf.conf that contains the following:

        (:tree (:here "src/lisp/"))
        (:tree (:here "extlib/lisp"))
        (:directory (:here "outlier/")))

Then when someone else (call her Y) checks out a copy of this repository, she need only add

     (:include "/path/to/my/checkout/directory/asdf.conf")

to one of her previously-existing asdf source location configuration files, or invoke initialize-source-registry with a configuration form containing that s-expression. ASDF will find the .conf file that X has provided, and then set up source locations within the working directory according to X's (relative) instructions.

8.7 Shell-friendly syntax for configuration

When considering environment variable CL_SOURCE_REGISTRY ASDF will skip to next configuration if it's an empty string. It will READ the string as a SEXP in the DSL if it begins with a paren ( and it will be interpreted much like TEXINPUTS list of paths, where

* paths are separated by a : (colon) on Unix platforms (including cygwin), by a ; (semicolon) on other platforms (mainly, Windows).

* each entry is a directory to add to the search path.

* if the entry ends with a double slash // then it instead indicates a tree in the subdirectories of which to recurse.

* if the entry is the empty string (which may only appear once), then it indicates that the inherited configuration should be spliced there.

8.8 Search Algorithm

In case that isn't clear, the semantics of the configuration is that when searching for a system of a given name, directives are processed in order.

When looking in a directory, if the system is found, the search succeeds, otherwise it continues.

When looking in a tree, if one system is found, the search succeeds. If multiple systems are found, the consequences are unspecified: the search may succeed with any of the found systems, or an error may be raised. ASDF currently returns the first system found, XCVB currently raised an error. If none is found, the search continues.

Exclude statements specify patterns of subdirectories the systems from which to ignore. Typically you don't want to use copies of files kept by such version control systems as Darcs. Exclude statements are not propagated to further included or inherited configuration files or expressions; instead the defaults are reset around every configuration statement to the default defaults from asdf::*default-source-registry-exclusions*.

Include statements cause the search to recurse with the path specifications from the file specified.

An inherit-configuration statement cause the search to recurse with the path specifications from the next configuration (see Configurations above).

8.9 Caching Results

The implementation is allowed to either eagerly compute the information from the configurations and file system, or to lazily re-compute it every time, or to cache any part of it as it goes. To explicitly flush any information cached by the system, use the API below.

8.10 Configuration API

The specified functions are exported from your build system's package. Thus for ASDF the corresponding functions are in package ASDF, and for XCVB the corresponding functions are in package XCVB.

— Function: initialize-source-registry &optional PARAMETER

will read the configuration and initialize all internal variables. You may extend or override configuration from the environment and configuration files with the given PARAMETER, which can be nil (no configuration override), or a SEXP (in the SEXP DSL), a string (as in the string DSL), a pathname (of a file or directory with configuration), or a symbol (fbound to function that when called returns one of the above).

— Function: clear-source-registry

undoes any source registry configuration and clears any cache for the search algorithm. You might want to call this function (or better, clear-configuration) before you dump an image that would be resumed with a different configuration, and return an empty configuration. Note that this does not include clearing information about systems defined in the current image, only about where to look for systems not yet defined.

— Function: ensure-source-registry &optional PARAMETER

checks whether a source registry has been initialized. If not, initialize it with the given PARAMETER.

Every time you use ASDF's find-system, or anything that uses it (such as operate, load-system, etc.), ensure-source-registry is called with parameter nil, which the first time around causes your configuration to be read. If you change a configuration file, you need to explicitly initialize-source-registry again, or maybe simply to clear-source-registry (or clear-configuration) which will cause the initialization to happen next time around.

8.11 Introspection

8.11.1 *source-registry-parameter* variable

We have made available the variable *source-registry-parameter* that can be used by code that wishes to introspect about the (past) configuration of ASDF's source registry. This variable should never be set! It will be set as a side-effect of calling initialize-source-registry; user code should treat it as read-only.

8.11.2 Information about system dependencies

ASDF makes available three functions to read system interdependencies. These are intended to aid programmers who wish to perform dependency analyses.

— Function: system-defsystem-depends-on system
— Function: system-depends-on system
— Function: system-weakly-depends-on system

Returns a list of names of systems that are weakly depended on by system. Weakly depended on systems are optionally loaded only if ASDF can find them; failure to find such systems does not cause an error in loading.

Note that the return value for system-weakly-depends-on is simpler than the return values of the other two system dependency introspection functions.

8.12 Status

This mechanism is vastly successful, and we have declared that asdf:*central-registry* is not recommended anymore, though we will continue to support it. All hooks into implementation-specific search mechanisms have been integrated in the wrapping-source-registry that everyone uses implicitly.

8.13 Rejected ideas

Alternatives I considered and rejected included:

  1. Keep asdf:*central-registry* as the master with its current semantics, and somehow the configuration parser expands the new configuration language into a expanded series of directories of subdirectories to lookup, pre-recursing through specified hierarchies. This is kludgy, and leaves little space of future cleanups and extensions.
  2. Keep asdf:*central-registry* remains the master but extend its semantics in completely new ways, so that new kinds of entries may be implemented as a recursive search, etc. This seems somewhat backwards.
  3. Completely remove asdf:*central-registry* and break backwards compatibility. Hopefully this will happen in a few years after everyone migrate to a better ASDF and/or to XCVB, but it would be very bad to do it now.
  4. Replace asdf:*central-registry* by a symbol-macro with appropriate magic when you dereference it or setf it. Only the new variable with new semantics is handled by the new search procedure. Complex and still introduces subtle semantic issues.

I've been suggested the below features, but have rejected them, for the sake of keeping ASDF no more complex than strictly necessary.

8.14 TODO

8.15 Credits for the source-registry

Thanks a lot to Stelian Ionescu for the initial idea.

Thanks to Rommel Martinez for the initial implementation attempt.

All bad design ideas and implementation bugs are mine, not theirs. But so are good design ideas and elegant implementation tricks.

— Francois-Rene Rideau, Mon, 22 Feb 2010 00:07:33 -0500