Functions COMPILED-FILE-P and ABI-VERSION

Author

Sam Steingold

Related

ANSI Common Lisp standard function compile-file.

Abstract

A facility to determine whether a file is a valid compiled file for the specific implementation and to identify the current compiled file format.

Rationale

Build tools, like defsystem or asdf, have to determine whether a file needs to be recompiled.

Obviously, when the compiled file is older than the source file, recompilation is in order.

Alas, there are other situations when this might be necessary, e.g., when the implementation changes the compiled file format or when two implementations use the same name for their compiled files (.fasl is used by both SBCL and ACL).

Traditionally, system definition facilities have taken the route of creating a separate directory for each combination of implementation type, version, operating system, and architecture. This is wasteful because the the compiled file format does not necessarily change between versions and does not even have to depend on OS and architecture.

The proposed functions will simplify the build directory tree structure and reduce the number of binary distribution bundles.

Current Practice

Implementation-dependent.

Cost of adoption

For COMPILED-FILE-P, probably tiny: an implementation must be able to check for compiled file validity, so all it takes is to export the necessary functionality, e.g.:

#+clisp
(defun compiled-file-p (file-name)
  (with-open-file (in file-name :direction :input :if-does-not-exist nil)
    (and in (char= #\( (peek-char nil in))
         (let ((form (ignore-errors (read in nil nil))))
           (and (consp form)
                (eq (car form) 'SYSTEM::VERSION)
                (null (nth-value 1 (ignore-errors (eval form)))))))))

For ABI-VERSION, it probably depends on the implementation; for some it might be trivial:

#+clisp
(defun abi-version (&-optional (object nil supplied-p))
  (if supplied-p
      (handler-case (progn (system::version (list object)) t)
        (error (e) nil))
      (car (system::version))))

and for others it might not.

Cost of non-adoption

Users will suffer random errors when trying to load invalid binary files.

Specification

Function COMPILED-FILE-P

Function

(compiled-file-p file-name) ==> valid-p

Returns

true
if the file appears to be a valid compiled file (i.e., exists, is readable, and the contents appears to be valid for this implementation),
false
otherwise.

Implementations are required to inspect the contents (e.g., checking just the pathname type is not sufficient). Although the completeness of the inspection is not required, this function should be able to detect, e.g., file format changes between versions.

Exceptional situations

Examples

(compiled-file-p "foo.lisp") ==> NIL
(compiled-file-p (compile-file "foo.lisp")) ==> T

Function ABI-VERSION

Function

(abi-version &optional object) ==> object

identifies the ABI (Application Binary Interface) of the presently used Common Lisp implementation.

When called without arguments, returns an implementation-defined object which uniquely identifies the compiled file format produced by the implementation.

The return value must satisfy three conditions:

  1. (princ-to-string value) must be a valid logical pathname component
  2. (equalp value (let ((*package* (find-package "KEYWORD"))) (read-from-string (princ-to-string value)))) must be true
  3. If (equalp a b) is true, then (abi-version a) and (abi-version b) should either both be true, or both be false

which guarantee that it can be used to name directories where the compiled files are stored and the ABI version can be recovered from a directory name.

When called with an argument, returns a generalized boolean:

true
if the argument identifies an ABI version which can be consumed by this implementation version,
false
otherwise.

Exceptional situations

Examples

(abi-version (abi-version))
==> T
;; in clisp 2.49:
(abi-version)
==> 20080430
(abi-version 20080430)
==> T
(abi-version "foo")
==> NIL
;; in a hypothetical lisp implementation MyCL v7 with a native compiler:
(abi-version)
==> :x86_64-7 ; or, say, 64007.
(abi-version :x86_64-7)
==> T
(abi-version :x86_64-6)
==> T ; if files compiled by v6 are acceptable for v7
(abi-version :x86_64-8)
==> NIL ; since forward compatibility cannot be assumed
(abi-version :sparc-7)
==> NIL ; since native compilation is usually architecture-dependent
(abi-version "bar")
==> NIL
(compile-file "foo.lisp" :output-file
  (make-pathname :directory
    (list (lisp-implementation-type) (princ-to-string (abi-version)))))
==> #P"CLISP/20080430/foo.fas" ; or
==> #P"MyCL/x86_64-7/foo.fasl"

Reference Implementation

See above.

History

The compiled-file-p spec was accepted as CLRFI-2 (in 2004).

Notes

The trivial implementation:

(defun compiled-file-p (file-name)
  (not (nth-value 1 (ignore-errors (load file-name)))))

is wrong because,

  1. load may fail even though the file is valid: even when foo.lisp contains calls to error,
    (compiled-file-p (compile-file "foo.lisp"))
    should still return T.
  2. this is not side-effect-free, i.e., this may define new functions and macros (or, worse yet, redefine some existing functions and macros or execute some malicious code).

If we could require (abi-version file) to return either the abi-version of the implementation which produced the compiled file, or nil if the file is not a compiled file for this implementation, then we could define

(defun compiled-file-p (file) (equalp (abi-version) (abi-version file)))

however, it is not obvious that all implementation can actually do this without unwelcome invasive changes.

Copying and License

This work may be distributed and/or modified under the conditions of the LaTeX Project Public License (LPPL), either version 1.3 or (at your option) any later version.

This work has the LPPL maintenance status maintained.

The Current Maintainer of this work is Sam Steingold.