Next: , Previous: Controlling where ASDF searches for systems, Up: Top


9 Controlling where ASDF saves compiled files

Each Common Lisp implementation has its own format for compiled files (fasls for short, short for “fast loading”). If you use multiple implementations (or multiple versions of the same implementation), you'll soon find your source directories littered with various fasls, dfsls, cfsls and so on. Worse yet, some implementations use the same file extension while changing formats from version to version (or platform to platform) which means that you'll have to recompile binaries as you switch from one implementation to the next.

Since ASDF 2, ASDF includes the asdf-output-translations facility to mitigate the problem.

9.1 Configurations

Configurations specify mappings from input locations to output locations. Once again we rely on the XDG base directory specification for configuration. See XDG base directory.

  1. Some hardcoded wrapping output translations configuration may be used. This allows special output translations (or usually, invariant directories) to be specified corresponding to the similar special entries in the source registry.
  2. An application may explicitly initialize the output-translations configuration using the Configuration API in which case this takes precedence. (see Configuration API.) It may itself compute this configuration from the command-line, from a script, from its own configuration file, etc.
  3. The source registry will be configured from the environment variable ASDF_OUTPUT_TRANSLATIONS if it exists.
  4. The source registry will be configured from user configuration file $XDG_CONFIG_DIRS/common-lisp/asdf-output-translations.conf (which defaults to ~/.config/common-lisp/asdf-output-translations.conf) if it exists.
  5. The source registry will be configured from user configuration directory $XDG_CONFIG_DIRS/common-lisp/asdf-output-translations.conf.d/ (which defaults to ~/.config/common-lisp/asdf-output-translations.conf.d/) if it exists.
  6. The source registry will be configured from system configuration file /etc/common-lisp/asdf-output-translations.conf if it exists.
  7. The source registry will be configured from system configuration directory /etc/common-lisp/asdf-output-translations.conf.d/ if it exists.

Each of these configurations is specified as a SEXP in a trival domain-specific language (defined below). Additionally, a more shell-friendly syntax is available for the environment variable (defined yet below).

Each of these configurations is only used if the previous configuration explicitly or implicitly specifies that it includes its inherited configuration.

Note that by default, a per-user cache is used for output files. This allows the seamless use of shared installations of software between several users, and takes files out of the way of the developers when they browse source code, at the expense of taking a small toll when developers have to clean up output files and find they need to get familiar with output-translations first.

9.2 Backward Compatibility

We purposefully do NOT provide backward compatibility with earlier versions of ASDF-Binary-Locations (8 Sept 2009), common-lisp-controller (7.0) or cl-launch (2.35), each of which had similar general capabilities. The previous APIs of these programs were not designed for configuration by the end-user in an easy way with configuration files. Recent versions of same packages use the new asdf-output-translations API as defined below: common-lisp-controller (7.2) and cl-launch (3.000). ASDF-Binary-Locations is fully superseded and not to be used anymore.

This incompatibility shouldn't inconvenience many people. Indeed, few people use and customize these packages; these few people are experts who can trivially adapt to the new configuration. Most people are not experts, could not properly configure these features (except inasmuch as the default configuration of common-lisp-controller and/or cl-launch might have been doing the right thing for some users), and yet will experience software that “just works”, as configured by the system distributor, or by default.

Nevertheless, if you are a fan of ASDF-Binary-Locations, we provide a limited emulation mode:

— Function: enable-asdf-binary-locations-compatibility &key centralize-lisp-binaries default-toplevel-directory include-per-user-information map-all-source-files source-to-target-mappings

This function will initialize the new asdf-output-translations facility in a way that emulates the behavior of the old ASDF-Binary-Locations facility. Where you would previously set global variables *centralize-lisp-binaries*, *default-toplevel-directory*, *include-per-user-information*, *map-all-source-files* or *source-to-target-mappings* you will now have to pass the same values as keyword arguments to this function. Note however that as an extension the :source-to-target-mappings keyword argument will accept any valid pathname designator for asdf-output-translations instead of just strings and pathnames.

If you insist, you can also keep using the old ASDF-Binary-Locations (the one available as an extension to load of top of ASDF, not the one built into a few old versions of ASDF), but first you must disable asdf-output-translations with (asdf:disable-output-translations), or you might experience “interesting” issues.

Also, note that output translation is enabled by default. To disable it, use (asdf:disable-output-translations).

9.3 Configuration DSL

Here is the grammar of the SEXP DSL for asdf-output-translations configuration:

;; A configuration is single SEXP starting with keyword :source-registry
;; followed by a list of directives.
CONFIGURATION := (:output-translations DIRECTIVE ...)

;; A directive is one of the following:
DIRECTIVE :=
    ;; INHERITANCE DIRECTIVE:
    ;; Your configuration expression MUST contain
    ;; exactly one of either of these:
    :inherit-configuration | ; splices inherited configuration (often specified last)
    :ignore-inherited-configuration | ; drop inherited configuration (specified anywhere)

    ;; forward compatibility directive (since ASDF 2.011.4), useful when
    ;; you want to use new configuration features but have to bootstrap a
    ;; the newer required ASDF from an older release that doesn't sport said features:
    :ignore-invalid-entries | ; drops subsequent invalid entries instead of erroring out

    ;; include a configuration file or directory
    (:include PATHNAME-DESIGNATOR) |

    ;; enable global cache in ~/.common-lisp/cache/sbcl-1.0.45-linux-amd64/ or something.
    :enable-user-cache |
    ;; Disable global cache. Map / to /
    :disable-cache |

    ;; add a single directory to be scanned (no recursion)
    (DIRECTORY-DESIGNATOR DIRECTORY-DESIGNATOR)

    ;; use a function to return the translation of a directory designator
    (DIRECTORY-DESIGNATOR (:function TRANSLATION-FUNCTION))

DIRECTORY-DESIGNATOR :=
    NIL | ;; As source: skip this entry. As destination: same as source
    T | ;; as source matches anything, as destination leaves pathname unmapped.
    ABSOLUTE-COMPONENT-DESIGNATOR ;; same as in the source-registry language

TRANSLATION-FUNCTION :=
    SYMBOL | ;; symbol of a function that takes two arguments,
             ;; the pathname to be translated and the matching DIRECTORY-DESIGNATOR
    LAMBDA   ;; A form which evalutates to a function taking two arguments consisting of
             ;; the pathname to be translated and the matching DIRECTORY-DESIGNATOR

Relative components better be either relative or subdirectories of the path before them, or bust.

The last component, if not a pathname, is notionally completed by /**/*.*. You can specify more fine-grained patterns by using a pathname object as the last component e.g. #p"some/path/**/foo*/bar-*.fasl"

You may use #+features to customize the configuration file.

The second designator of a mapping may be nil, indicating that files are not mapped to anything but themselves (same as if the second designator was the same as the first).

When the first designator is t, the mapping always matches. When the first designator starts with :root, the mapping matches any host and device. In either of these cases, if the second designator isn't t and doesn't start with :root, then strings indicating the host and pathname are somehow copied in the beginning of the directory component of the source pathname before it is translated.

When the second designator is t, the mapping is the identity. When the second designator starts with :root, the mapping preserves the host and device of the original pathname. Notably, this allows you to map files to a subdirectory of the whichever directory the file is in. Though the syntax is not quite as easy to use as we'd like, you can have an (source destination) mapping entry such as follows in your configuration file, or you may use enable-asdf-binary-locations-compatibility with :centralize-lisp-binaries nil which will do the same thing internally for you:

  #.(let ((wild-subdir (make-pathname :directory '(:relative :wild-inferiors)))
          (wild-file (make-pathname :name :wild :version :wild :type :wild)))
     `((:root ,wild-subdir ,wild-file) ;; Or using the implicit wildcard, just :root
       (:root ,wild-subdir :implementation ,wild-file)))
Starting with ASDF 2.011.4, you can use the simpler: `(:root (:root :**/ :implementation :*.*.*))

:include statements cause the search to recurse with the path specifications from the file specified.

If the translate-pathname mechanism cannot achieve a desired translation, the user may provide a function which provides the required algorithim. Such a translation function is specified by supplying a list as the second directory-designator the first element of which is the keyword :function, and the second element of which is either a symbol which designates a function or a lambda expression. The function designated by the second argument must take two arguments, the first being the pathname of the source file, the second being the wildcard that was matched. The result of the function invocation should be the translated pathname.

An :inherit-configuration statement cause the search to recurse with the path specifications from the next configuration. See Configurations, above.

9.4 Configuration Directories

Configuration directories consist in files each contains a list of directives without any enclosing (:output-translations ...) form. The files will be sorted by namestring as if by string< and the lists of directives of these files with be concatenated in order. An implicit :inherit-configuration will be included at the end of the list.

This allows for packaging software that has file granularity (e.g. Debian's dpkg or some future version of clbuild) to easily include configuration information about software being distributed.

The convention is that, for sorting purposes, the names of files in such a directory begin with two digits that determine the order in which these entries will be read. Also, the type of these files is conventionally "conf" and as a limitation of some implementations, the type cannot be nil.

Directories may be included by specifying a directory pathname or namestring in an :include directive, e.g.:

	(:include "/foo/bar/")

9.5 Shell-friendly syntax for configuration

When considering environment variable ASDF_OUTPUT_TRANSLATIONS ASDF will skip to next configuration if it's an empty string. It will READ the string as an SEXP in the DSL if it begins with a paren ( and it will be interpreted as a list of directories. Directories should come by pairs, indicating a mapping directive. Entries are separated by a : (colon) on Unix platforms (including cygwin), by a ; (semicolon) on other platforms (mainly, Windows).

The magic empty entry, if it comes in what would otherwise be the first entry in a pair, indicates the splicing of inherited configuration. If it comes as the second entry in a pair, it indicates that the directory specified first is to be left untranslated (which has the same effect as if the directory had been repeated).

9.6 Semantics of Output Translations

From the specified configuration, a list of mappings is extracted in a straightforward way: mappings are collected in order, recursing through included or inherited configuration as specified. To this list is prepended some implementation-specific mappings, and is appended a global default.

The list is then compiled to a mapping table as follows: for each entry, in order, resolve the first designated directory into an actual directory pathname for source locations. If no mapping was specified yet for that location, resolve the second designated directory to an output location directory add a mapping to the table mapping the source location to the output location, and add another mapping from the output location to itself (unless a mapping already exists for the output location).

Based on the table, a mapping function is defined, mapping source pathnames to output pathnames: given a source pathname, locate the longest matching prefix in the source column of the mapping table. Replace that prefix by the corresponding output column in the same row of the table, and return the result. If no match is found, return the source pathname. (A global default mapping the filesystem root to itself may ensure that there will always be a match, with same fall-through semantics).

9.7 Caching Results

The implementation is allowed to either eagerly compute the information from the configurations and file system, or to lazily re-compute it every time, or to cache any part of it as it goes. To explicitly flush any information cached by the system, use the API below.

9.8 Output location API

The specified functions are exported from package ASDF.

— Function: initialize-output-translations &optional PARAMETER

will read the configuration and initialize all internal variables. You may extend or override configuration from the environment and configuration files with the given PARAMETER, which can be nil (no configuration override), or a SEXP (in the SEXP DSL), a string (as in the string DSL), a pathname (of a file or directory with configuration), or a symbol (fbound to function that when called returns one of the above).

— Function: disable-output-translations

will initialize output translations in a way that maps every pathname to itself, effectively disabling the output translation facility.

— Function: clear-output-translations

undoes any output translation configuration and clears any cache for the mapping algorithm. You might want to call this function (or better, clear-configuration) before you dump an image that would be resumed with a different configuration, and return an empty configuration. Note that this does not include clearing information about systems defined in the current image, only about where to look for systems not yet defined.

— Function: ensure-output-translations &optional PARAMETER

checks whether output translations have been initialized. If not, initialize them with the given PARAMETER. This function will be called before any attempt to operate on a system.

— Function: apply-output-translations PATHNAME

Applies the configured output location translations to PATHNAME (calls ensure-output-translations for the translations).

Every time you use ASDF's output-files, or anything that uses it (that may compile, such as operate, perform, etc.), ensure-output-translations is called with parameter nil, which the first time around causes your configuration to be read. If you change a configuration file, you need to explicitly initialize-output-translations again, or maybe clear-output-translations (or clear-configuration), which will cause the initialization to happen next time around.

9.9 Credits for output translations

Thanks a lot to Bjorn Lindberg and Gary King for ASDF-Binary-Locations, and to Peter van Eynde for Common Lisp Controller.

All bad design ideas and implementation bugs are to mine, not theirs. But so are good design ideas and elegant implementation tricks.

— Francois-Rene Rideau fare@tunes.org