4.1 Compiler Introduction
This chapter contains information about the compiler that every
CMUCL user should be familiar with. Chapter 5 goes into greater
depth, describing ways to use more advanced features.
The CMUCL compiler (also known as Python, not to be confused with
the programming language of the same name) has many features that
are seldom or never supported by conventional Common Lisp
compilers:
- Source level debugging of compiled code (see
chapter 3.)
- Type error compiler warnings for type errors
detectable at compile time.
- Compiler error messages that provide a good
indication of where the error appeared in the source.
- Full run-time checking of all potential type
errors, with optimization of type checks to minimize the cost.
- Scheme-like features such as proper tail
recursion and extensive source-level optimization.
- Advanced tuning and optimization features
such as comprehensive efficiency notes, flow analysis, and untagged
number representations (see chapter 5.)
4.2 Calling the Compiler
Functions may be compiled using
compile, compile-file,
or compile-from-stream.
[Function]
compile name &optional definition
This function compiles the function whose name is name. If name is
nil, the compiled function object is
returned. If definition is supplied, it
should be a lambda expression that is to be compiled and then
placed in the function cell of name. As
per the proposed X3J13 cleanup
“compile-argument-problems”, definition may also be an interpreted function.
The return values are as per the proposed X3J13 cleanup
“compiler-diagnostics”. The first value is the function
name or function object. The second value is nil if no compiler diagnostics were issued, and
t otherwise. The third value is nil if no compiler diagnostics other than style
warnings were issued. A non-nil value
indicates that there were “serious” compiler
diagnostics issued, or that other conditions of type error or warning (but not style-warning) were signaled
during compilation.
[Function]
compile-file input-pathname &key
:output-file :error-file :trace-file
:error-output :verbose
:print :progress
:load :block-compile
:entry-points
:byte-compile
The CMUCL compile-file is extended through
the addition of several new keywords and an additional
interpretation of input-pathname:
- input-pathname
- If this argument is a list of input files,
rather than a single input pathname, then all the source files are
compiled into a single object file. In this case, the name of the
first file is used to determine the default output file names. This
is especially useful in combination with block-compile.
- :output-file
- This argument specifies the name of the output
file. t gives the default name, nil suppresses the output file.
- :error-file
- A listing of all the error output is directed
to this file. If there are no errors, then no error file is
produced (and any existing error file is deleted.) t gives "name.err" (the default), and nil
suppresses the output file.
- :error-output
- If t (the default), then
error output is sent to *error-output*. If a
stream, then output is sent to that stream instead. If nil, then error output is suppressed. Note that this
error output is in addition to (but the same as) the output placed
in the error-file.
- :verbose
- If t (the default), then
the compiler prints to error output at the start and end of
compilation of each file. See *compile-verbose*.
- :print
- If t (the default), then
the compiler prints to error output when each function is compiled.
See *compile-print*.
- :progress
- If t (default nil), then the compiler prints to error output progress
information about the phases of compilation of each function. This
is a CMUCL extension that is useful mainly in large block
compilations. See *compile-progress*.
- :trace-file
- If t, several of the
intermediate representations (including annotated assembly code)
are dumped out to this file. t gives
"name.trace". Trace
output is off by default. See section 5.12.5.
- :load
- If t, load the resulting
output file.
- :block-compile
- Controls the compile-time resolution of
function calls. By default, only self-recursive calls are resolved,
unless an ext:block-start declaration appears
in the source file. See section 5.7.3.
- :entry-points
- If non-null, then this is a list of the names
of all functions in the file that should have global definitions
installed (because they are referenced in other files.) See
section 5.7.3.
- :byte-compile
- If t, compiling to a
compact interpreted byte code is enabled. Possible values are
t, nil, and :maybe (the default.) See *byte-compile-default* and See
section 5.9.
The return values are as per the proposed X3J13 cleanup
“compiler-diagnostics”. The first value from compile-file is the truename of the output file, or
nil if the file could not be created. The
interpretation of the second and third values is described above
for compile.
[Variable]
*compile-verbose*
[Variable]
*compile-print*
[Variable]
*compile-progress*
These variables determine the default values for the :verbose, :print and :progress arguments to compile-file.
[Function]
extensions:compile-from-stream input-stream &key
:error-stream
:trace-stream
:block-compile :entry-points
:byte-compile
This function is similar to compile-file, but
it takes all its arguments as streams. It reads Common Lisp code
from input-stream until end of file is
reached, compiling into the current environment. This function
returns the same two values as the last two values of compile. No output files are produced.
4.3 Compilation Units
CMUCL supports the with-compilation-unit macro added to the language by
the X3J13 “with-compilation-unit” compiler cleanup
issue. This provides a mechanism for eliminating spurious undefined
warnings when there are forward references across files, and also
provides a standard way to access compiler extensions.
[Macro]
with-compilation-unit ({key value}*) {form}*
This macro evaluates the forms in an
environment that causes warnings for undefined variables, functions
and types to be delayed until all the forms have been evaluated.
Each keyword value is an evaluated form.
These keyword options are recognized:
- :override
- If uses of with-compilation-unit are dynamically nested, the
outermost use will take precedence, suppressing printing of
undefined warnings by inner uses. However, when the override option is true this shadowing is inhibited; an
inner use will print summary warnings for the compilations within
the inner scope.
- :optimize
- This is a CMUCL extension that specifies of the
“global” compilation policy for the dynamic extent of
the body. The argument should evaluate to an optimize declare form, like:
(optimize (speed 3) (safety 0))
See section 4.7.1
- :optimize-interface
- Similar to :optimize, but
specifies the compilation policy for function interfaces (argument
count and type checking) for the dynamic extent of the body. See
section 4.7.2.
- :context-declarations
- This is a CMUCL extension that pattern-matches
on function names, automatically splicing in any appropriate
declarations at the head of the function definition. See
section 5.7.5.
4.3.1 Undefined Warnings
Warnings about undefined variables,
functions and types are delayed until the end of the current
compilation unit. The compiler entry functions (compile, etc.) implicitly use with-compilation-unit, so undefined warnings will be
printed at the end of the compilation unless there is an enclosing
with-compilation-unit. In order the gain the
benefit of this mechanism, you should wrap a single with-compilation-unit around the calls to compile-file, i.e.:
(with-compilation-unit ()
(compile-file "file1")
(compile-file "file2")
...)
Unlike for functions and types, undefined warnings for variables
are not suppressed when a definition (e.g. defvar) appears after the reference (but in the same
compilation unit.) This is because doing special declarations out
of order just doesn't work—although early references will be
compiled as special, bindings will be done lexically.
Undefined warnings are printed with full source context (see
section 4.4), which tremendously
simplifies the problem of finding undefined references that
resulted from macroexpansion. After printing detailed information
about the undefined uses of each name, with-compilation-unit also prints summary listings of
the names of all the undefined functions, types and variables.
[Variable]
*undefined-warning-limit*
This variable controls the number of undefined warnings for each
distinct name that are printed with full source context when the
compilation unit ends. If there are more undefined references than
this, then they are condensed into a single warning:
Warning: count more uses of undefined function name.
When the value is 0, then the undefined
warnings are not broken down by name at all: only the summary
listing of undefined names is printed.
4.4 Interpreting Error Messages
One of Python's unique
features is the level of source location information it provides in
error messages. The error messages contain a lot of detail in a
terse format, to they may be confusing at first. Error messages
will be illustrated using this example program:
(defmacro zoq (x)
`(roq (ploq (+ ,x 3))))
(defun foo (y)
(declare (symbol y))
(zoq y))
The main problem with this program is that it is trying to add
3 to a symbol. Note also that the functions
roq and ploq aren't
defined anywhere.
4.4.1 The Parts of the Error Message
The compiler will produce this warning:
File: /usr/me/stuff.lisp
In: DEFUN FOO
(ZOQ Y)
–> ROQ PLOQ +
==>
Y
Warning: Result is a SYMBOL, not a NUMBER.
In this example we see each of the six possible parts of a compiler
error message:
- File:
/usr/me/stuff.lisp
- This is the file that
the compiler read the relevant code from. The file name is
displayed because it may not be immediately obvious when there is
an error during compilation of a large system, especially when
with-compilation-unit is used to delay
undefined warnings.
- In: DEFUN FOO
- This is the definition or top-level form responsible for the
error. It is obtained by taking the first two elements of the
enclosing form whose first element is a symbol beginning with
“DEF”. If there is no enclosing
defmumble, then the outermost form is
used. If there are multiple defmumbles,
then they are all printed from the out in, separated by =>'s. In this example, the problem was in the
defun for foo.
- (ZOQ Y)
- This is the original source form
responsible for the error. Original source means that the form
directly appeared in the original input to the compiler, i.e. in
the lambda passed to compile or the top-level
form read from the source file. In this example, the expansion of
the zoq macro was responsible for the
error.
- –> ROQ PLOQ
+
- This is the processing path that the
compiler used to produce the errorful code. The processing path is
a representation of the evaluated forms enclosing the actual source
that the compiler encountered when processing the original source.
The path is the first element of each form, or the form itself if
the form is not a list. These forms result from the expansion of
macros or source-to-source transformation done by the compiler. In
this example, the enclosing evaluated forms are the calls to
roq, ploq and
+. These calls resulted from the expansion of
the zoq macro.
- ==> Y
- This is the actual source responsible
for the error. If the actual source appears in the explanation,
then we print the next enclosing evaluated form, instead of
printing the actual source twice. (This is the form that would
otherwise have been the last form of the processing path.) In this
example, the problem is with the evaluation of the reference to the
variable y.
- Warning: Result is a SYMBOL,
not a NUMBER.
- This is the explanation the problem. In this example, the
problem is that y evaluates to a symbol, but is in a context where a number is required
(the argument to +).
Note that each part of the error message is distinctively marked:
- File: and In: mark the file and definition, respectively.
- The original source is an indented form with
no prefix.
- Each line of the processing path is prefixed
with –>.
- The actual source form is indented like the
original source, but is marked by a preceding ==> line. This is like the “macroexpands
to” notation used in Common Lisp: The Language.
- The explanation is prefixed with the error
severity (see section 4.4.4),
either Error:, Warning:, or Note:.
Each part of the error message is more specific than the preceding
one. If consecutive error messages are for nearby locations, then
the front part of the error messages would be the same. In this
case, the compiler omits as much of the second message as in common
with the first. For example:
File: /usr/me/stuff.lisp
In: DEFUN FOO
(ZOQ Y)
–> ROQ
==>
(PLOQ (+ Y 3))
Warning: Undefined function: PLOQ
==>
(ROQ (PLOQ (+ Y 3)))
Warning: Undefined function: ROQ
In this example, the file, definition and original source are
identical for the two messages, so the compiler omits them in the
second message. If consecutive messages are entirely identical,
then the compiler prints only the first message, followed by:
[Last message occurs repeats times]
where repeats is the number of times the
message was given.
If the source was not from a file, then no file line is printed. If
the actual source is the same as the original source, then the
processing path and actual source will be omitted. If no forms
intervene between the original source and the actual source, then
the processing path will also be omitted.
4.4.2 The Original and Actual Source
The
original source displayed will almost always be a list. If
the actual source for an error message is a symbol, the original
source will be the immediately enclosing evaluated list form. So
even if the offending symbol does appear in the original source,
the compiler will print the enclosing list and then print the
symbol as the actual source (as though the symbol were introduced
by a macro.)
When the actual source is displayed (and is not a symbol),
it will always be code that resulted from the expansion of a macro
or a source-to-source compiler optimization. This is code that did
not appear in the original source program; it was introduced by the
compiler.
Keep in mind that when the compiler displays a source form in an
error message, it always displays the most specific (innermost)
responsible form. For example, compiling this function:
(defun bar (x)
(let (a)
(declare (fixnum a))
(setq a (foo x))
a))
gives this error message:
In: DEFUN BAR
(LET (A) (DECLARE (FIXNUM A)) (SETQ A (FOO X)) A)
Warning: The binding of A is not a FIXNUM:
NIL
This error message is not saying “there's a problem somewhere
in this let”—it is saying that
there is a problem with the let itself. In
this example, the problem is that a's
nil initial value is not a fixnum.
4.4.3 The Processing Path
The processing path is mainly useful for
debugging macros, so if you don't write macros, you can ignore the
processing path. Consider this example:
(defun foo (n)
(dotimes (i n *undefined*)))
Compiling results in this error message:
In: DEFUN FOO
(DOTIMES (I N *UNDEFINED*))
–> DO BLOCK LET TAGBODY RETURN-FROM
==>
(PROGN *UNDEFINED*)
Warning: Undefined variable: *UNDEFINED*
Note that do appears in the processing path.
This is because dotimes expands into:
(do ((i 0 (1+ i)) (#:g1 n))
((>= i #:g1) *undefined*)
(declare (type unsigned-byte i)))
The rest of the processing path results from the expansion of
do:
(block nil
(let ((i 0) (#:g1 n))
(declare (type unsigned-byte i))
(tagbody (go #:g3)
#:g2 (psetq i (1+ i))
#:g3 (unless (>= i #:g1) (go #:g2))
(return-from nil (progn *undefined*)))))
In this example, the compiler descended into the block, let, tagbody and return-from to reach
the progn printed as the actual source. This
is a place where the “actual source appears in
explanation” rule was applied. The innermost actual source
form was the symbol *undefined* itself, but
that also appeared in the explanation, so the compiler backed out
one level.
4.4.4 Error Severity
There are three
levels of compiler error severity:
- Error
- This severity is used when the compiler
encounters a problem serious enough to prevent normal processing of
a form. Instead of compiling the form, the compiler compiles a call
to error. Errors are used mainly for
signaling syntax errors. If an error happens during macroexpansion,
the compiler will handle it. The compiler also handles and attempts
to proceed from read errors.
- Warning
- Warnings are used when the compiler can prove
that something bad will happen if a portion of the program is
executed, but the compiler can proceed by compiling code that
signals an error at runtime if the problem has not been fixed:
- Violation of type declarations, or
- Function calls that have the wrong number of
arguments or malformed keyword argument lists, or
- Referencing a variable declared ignore, or unrecognized declaration specifiers.
In the language of the Common Lisp standard, these are situations
where the compiler can determine that a situation with undefined
consequences or that would cause an error to be signaled would
result at runtime.
- Note
- Notes are used when there is something that
seems a bit odd, but that might reasonably appear in correct
programs.
Note that the compiler does not fully conform to the proposed X3J13
“compiler-diagnostics” cleanup. Errors, warnings and
notes mostly correspond to errors, warnings and style-warnings, but
many things that the cleanup considers to be style-warnings are
printed as warnings rather than notes. Also, warnings,
style-warnings and most errors aren't really signaled using the
condition system.
4.4.5 Errors During Macroexpansion
The compiler handles errors that happen
during macroexpansion, turning them into compiler errors. If you
want to debug the error (to debug a macro), you can set *break-on-signals* to error. For
example, this definition:
(defun foo (e l)
(do ((current l (cdr current))
((atom current) nil))
(when (eq (car current) e) (return current))))
gives this error:
In: DEFUN FOO
(DO ((CURRENT L #) (# NIL)) (WHEN (EQ # E) (RETURN CURRENT)) )
Error: (during macroexpansion)
Error in function LISP::DO-DO-BODY.
DO step variable is not a symbol: (ATOM CURRENT)
4.4.6 Read Errors
The compiler also handles errors while
reading the source. For example:
Error: Read error at 2:
"(,/\foo)"
Error in function LISP::COMMA-MACRO.
Comma not inside a backquote.
The “at 2” refers to the
character position in the source file at which the error was
signaled, which is generally immediately after the erroneous text.
The next line, “(,/\foo)”, is the
line in the source that contains the error file position. The
“/\ ” indicates the error
position within that line (in this example, immediately after the
offending comma.)
When in Hemlock (or any other EMACS-like editor), you can go to a
character position with:
M-< C-u position C-f
Note that if the source is from a Hemlock buffer, then the position
is relative to the start of the compiled region or defun, not the file or buffer start.
After printing a read error message, the compiler attempts to
recover from the error by backing up to the start of the enclosing
top-level form and reading again with *read-suppress* true. If the compiler can recover from
the error, then it substitutes a call to cerror for the unreadable form and proceeds to compile
the rest of the file normally.
If there is a read error when the file position is at the end of
the file (i.e., an unexpected EOF error), then the error message
looks like this:
Error: Read error in form starting at 14:
"(defun test ()"
Error in function LISP::FLUSH-WHITESPACE.
EOF while reading #<Stream for file "/usr/me/test.lisp">
In this case, “starting at 14”
indicates the character position at which the compiler started
reading, i.e. the position before the start of the form that was
missing the closing delimiter. The line "(defun
test ()" is first line after the starting position that the
compiler thinks might contain the unmatched open delimiter.
4.4.7 Error Message Parameterization
There is some
control over the verbosity of error messages. See also *undefined-warning-limit*,
*efficiency-note-limit* and *efficiency-note-cost-threshold*.
[Variable]
*enclosing-source-cutoff*
This variable specifies the number of enclosing actual source forms
that are printed in full, rather than in the abbreviated processing
path format. Increasing the value from its default of 1 allows you to see more of the guts of the
macroexpanded source, which is useful when debugging
macros.
[Variable]
*error-print-length*
[Variable]
*error-print-level*
These variables are the print level and print length used in
printing error messages. The default values are 5 and 3. If null, the global
values of *print-level* and *print-length* are used.
[Macro]
extensions:def-source-context name lambda-list
{form}*
This macro defines how to extract an abbreviated source context
from the named form when it appears in
the compiler input. lambda-list is a
defmacro style lambda-list used to parse the
arguments. The body should return a list
of subforms that can be printed on about one line. There are
predefined methods for defstruct, defmethod, etc. If no method is defined, then the first
two subforms are returned. Note that this facility implicitly
determines the string name associated with anonymous
functions.
4.5 Types in Python
A big difference between Python and all
other Common Lisp compilers is the approach to type checking and
amount of knowledge about types:
- Python treats type declarations much
differently that other Lisp compilers do. Python doesn't blindly
believe type declarations; it considers them assertions about the
program that should be checked.
- Python also has a tremendously greater
knowledge of the Common Lisp type system than other compilers.
Support is incomplete only for the not,
and and satisfies
types.
See also sections 5.2 and 5.3.
4.5.1 Compile Time Type Errors
If the
compiler can prove at compile time that some portion of the program
cannot be executed without a type error, then it will give a
warning at compile time. It is possible that the offending code
would never actually be executed at run-time due to some higher
level consistency constraint unknown to the compiler, so a type
warning doesn't always indicate an incorrect program. For example,
consider this code fragment:
(defun raz (foo)
(let ((x (case foo
(:this 13)
(:that 9)
(:the-other 42))))
(declare (fixnum x))
(foo x)))
Compilation produces this warning:
In: DEFUN RAZ
(CASE FOO (:THIS 13) (:THAT 9) (:THE-OTHER 42))
–> LET COND IF COND IF COND IF
==>
(COND)
Warning: This is not a FIXNUM:
NIL
In this case, the warning is telling you that if foo isn't any of :this,
:that or :the-other,
then x will be initialized to nil, which the fixnum declaration
makes illegal. The warning will go away if ecase is used instead of case, or
if :the-other is changed to t.
This sort of spurious type warning happens moderately often in the
expansion of complex macros and in inline functions. In such cases,
there may be dead code that is impossible to correctly execute. The
compiler can't always prove this code is dead (could never be
executed), so it compiles the erroneous code (which will always
signal an error if it is executed) and gives a warning.
[Function]
extensions:required-argument
This function can be used as the default value for keyword
arguments that must always be supplied. Since it is known by the
compiler to never return, it will avoid any compile-time type
warnings that would result from a default value inconsistent with
the declared type. When this function is called, it signals an
error indicating that a required keyword argument was not supplied.
This function is also useful for defstruct
slot defaults corresponding to required arguments. See
section 5.2.5.
Although this function is a CMUCL extension, it is relatively
harmless to use it in otherwise portable code, since you can easily
define it yourself:
(defun required-argument ()
(error "A required keyword argument was not supplied."))
Type warnings are inhibited when the extensions:inhibit-warnings optimization quality is
3 (see section 4.7.) This can be used in a local
declaration to inhibit type warnings in a code fragment that has
spurious warnings.
4.5.2 Precise Type Checking
With the
default compilation policy, all type assertions1 are precisely checked. Precise
checking means that the check is done as though typep had been called with the exact type specifier
that appeared in the declaration. Python uses policy to determine whether to trust type
assertions (see section 4.7).
Type assertions from declarations are indistinguishable from the
type assertions on arguments to built-in functions. In Python,
adding type declarations makes code safer.
If a variable is declared to be (integer 3
17), then its value must always always be an integer between
3 and 17. If multiple
type declarations apply to a single variable, then all the
declarations must be correct; it is as though all the types were
intersected producing a single and type
specifier.
Argument type declarations are automatically enforced. If you
declare the type of a function argument, a type check will be done
when that function is called. In a function call, the called
function does the argument type checking, which means that a more
restrictive type assertion in the calling function (e.g., from
the) may be lost.
The types of structure slots are also checked. The value of a
structure slot must always be of the type indicated in any
:type slot option.2 Because of precise type checking,
the arguments to slot accessors are checked to be the correct type
of structure.
In traditional Common Lisp compilers, not all type assertions are
checked, and type checks are not precise. Traditional compilers
blindly trust explicit type declarations, but may check the
argument type assertions for built-in functions. Type checking is
not precise, since the argument type checks will be for the most
general type legal for that argument. In many systems, type
declarations suppress what little type checking is being done, so
adding type declarations makes code unsafe. This is a problem since
it discourages writing type declarations during initial coding. In
addition to being more error prone, adding type declarations during
tuning also loses all the benefits of debugging with checked type
assertions.
To gain maximum benefit from Python's type checking, you should
always declare the types of function arguments and structure slots
as precisely as possible. This often involves the use of or, member and other list-style
type specifiers. Paradoxically, even though adding type
declarations introduces type checks, it usually reduces the overall
amount of type checking. This is especially true for structure slot
type declarations.
Python uses the safety optimization quality
(rather than presence or absence of declarations) to choose one of
three levels of run-time type error checking: see
section 4.7.1. See
section 5.2 for more
information about types in Python.
4.5.3 Weakened Type Checking
When the
value for the speed optimization quality is
greater than safety, and safety is not 0, then type
checking is weakened to reduce the speed and space penalty. In
structure-intensive code this can double the speed, yet still catch
most type errors. Weakened type checks provide a level of safety
similar to that of “safe” code in other Common Lisp
compilers.
A type check is weakened by changing the check to be for some
convenient supertype of the asserted type. For example, (integer 3 17) is changed to fixnum, (simple-vector 17) to
simple-vector, and structure types are
changed to structure. A complex check like:
(or node hunk (member :foo :bar :baz))
will be omitted entirely (i.e., the check is weakened to *.) If a precise check can be done for no extra cost,
then no weakening is done.
Although weakened type checking is similar to type checking done by
other compilers, it is sometimes safer and sometimes less safe.
Weakened checks are done in the same places is precise checks, so
all the preceding discussion about where checking is done still
applies. Weakened checking is sometimes somewhat unsafe because
although the check is weakened, the precise type is still input
into type inference. In some contexts this will result in type
inferences not justified by the weakened check, and hence deletion
of some type checks that would be done by conventional
compilers.
For example, if this code was compiled with weakened checks:
(defstruct foo
(a nil :type simple-string))
(defstruct bar
(a nil :type single-float))
(defun myfun (x)
(declare (type bar x))
(* (bar-a x) 3.0))
and myfun was passed a foo, then no type error would be signaled, and we would
try to multiply a simple-vector as though it
were a float (with unpredictable results.) This is because the
check for bar was weakened to structure, yet when compiling the call to bar-a, the compiler thinks it knows it has a bar.
Note that normally even weakened type checks report the precise
type in error messages. For example, if myfun's bar check is weakened to
structure, and the argument is nil, then the error will be:
Type-error in MYFUN:
NIL is not of type BAR
However, there is some speed and space cost for signaling a precise
error, so the weakened type is reported if the speed optimization quality is 3
or debug quality is less than 1:
Type-error in MYFUN:
NIL is not of type STRUCTURE
See section 4.7.1 for
further discussion of the optimize
declaration.
4.6 Getting Existing Programs to Run
Since Python does much more comprehensive type
checking than other Lisp compilers, Python will detect type errors
in many programs that have been debugged using other compilers.
These errors are mostly incorrect declarations, although
compile-time type errors can find actual bugs if parts of the
program have never been tested.
Some incorrect declarations can only be detected by run-time type
checking. It is very important to initially compile programs with
full type checks and then test this version. After the checking
version has been tested, then you can consider weakening or
eliminating type checks. This applies even to previously
debugged programs. Python does much more type inference than
other Common Lisp compilers, so believing an incorrect declaration
does much more damage.
The most common problem is with variables whose initial value
doesn't match the type declaration. Incorrect initial values will
always be flagged by a compile-time type error, and they are simple
to fix once located. Consider this code fragment:
(prog (foo)
(declare (fixnum foo))
(setq foo ...)
...)
Here the variable foo is given an initial
value of nil, but is declared to be a
fixnum. Even if it is never read, the initial
value of a variable must match the declared type. There are two
ways to fix this problem. Change the declaration:
(prog (foo)
(declare (type (or fixnum null) foo))
(setq foo ...)
...)
or change the initial value:
(prog ((foo 0))
(declare (fixnum foo))
(setq foo ...)
...)
It is generally preferable to change to a legal initial value
rather than to weaken the declaration, but sometimes it is simpler
to weaken the declaration than to try to make an initial value of
the appropriate type.
Another declaration problem occasionally encountered is incorrect
declarations on defmacro arguments. This
probably usually happens when a function is converted into a macro.
Consider this macro:
(defmacro my-1+ (x)
(declare (fixnum x))
`(the fixnum (1+ ,x)))
Although legal and well-defined Common Lisp, this meaning of this
definition is almost certainly not what the writer intended. For
example, this call is illegal:
(my-1+ (+ 4 5))
The call is illegal because the argument to the macro is (+ 4 5), which is a list, not a
fixnum. Because of macro semantics, it is
hardly ever useful to declare the types of macro arguments. If you
really want to assert something about the type of the result of
evaluating a macro argument, then put a the
in the expansion:
(defmacro my-1+ (x)
`(the fixnum (1+ (the fixnum ,x))))
In this case, it would be stylistically preferable to change this
macro back to a function and declare it inline. Macros have no
efficiency advantage over inline functions when using Python. See
section 5.8.
Some more subtle problems are caused by incorrect declarations that
can't be detected at compile time. Consider this code:
(do ((pos 0 (position #\a string :start (1+ pos))))
((null pos))
(declare (fixnum pos))
...)
Although pos is almost always a fixnum, it is nil at the end of
the loop. If this example is compiled with full type checks (the
default), then running it will signal a type error at the end of
the loop. If compiled without type checks, the program will go into
an infinite loop (or perhaps position will
complain because (1+ nil) isn't a sensible
start.) Why? Because if you compile without type checks, the
compiler just quietly believes the type declaration. Since
pos is always a fixnum,
it is never nil, so (null
pos) is never true, and the loop exit test is optimized away.
Such errors are sometimes flagged by unreachable code notes (see
section 5.4.5), but it is still
important to initially compile any system with full type checks,
even if the system works fine when compiled using other
compilers.
In this case, the fix is to weaken the type declaration to
(or fixnum null).3 Note that there is usually little
performance penalty for weakening a declaration in this way. Any
numeric operations in the body can still assume the variable is a
fixnum, since nil is
not a legal numeric argument. Another possible fix would be to say:
(do ((pos 0 (position #\a string :start (1+ pos))))
((null pos))
(let ((pos pos))
(declare (fixnum pos))
...))
This would be preferable in some circumstances, since it would
allow a non-standard representation to be used for the local
pos variable in the loop body (see section
5.11.3.)
In summary, remember that all values that a variable
ever has must be of the declared type, and that you should
test using safe code initially.
4.7 Compiler Policy
The policy is what
tells the compiler how to compile a
program. This is logically (and often textually) distinct from the
program itself. Broad control of policy is provided by the
optimize declaration; other declarations and
variables control more specific aspects of compilation.
4.7.1 The Optimize Declaration
The
optimize declaration recognizes six different
qualities. The qualities are conceptually
independent aspects of program performance. In reality, increasing
one quality tends to have adverse effects on other qualities. The
compiler compares the relative values of qualities when it needs to
make a trade-off; i.e., if speed is greater
than safety, then improve speed at the cost
of safety.
The default for all qualities (except debug)
is 1. Whenever qualities are equal, ties are
broken according to a broad idea of what a good default environment
is supposed to be. Generally this downplays speed, compile-speed and
space in favor of safety and debug. Novice and
casual users should stick to the default policy. Advanced users
often want to improve speed and memory usage at the cost of safety
and debuggability.
If the value for a quality is 0 or 3, then it may have a special interpretation. A value
of 0 means “totally unimportant”,
and a 3 means “ultimately
important.” These extreme optimization values enable
“heroic” compilation strategies that are not always
desirable and sometimes self-defeating. Specifying more than one
quality as 3 is not desirable, since it
doesn't tell the compiler which quality is most important.
These are the optimization qualities:
- speed
- How fast the program
should is run. speed 3 enables some
optimizations that hurt debuggability.
- compilation-speed
- How fast the compiler
should run. Note that increasing this above safety weakens type checking.
- space
- How much space the
compiled code should take up. Inline expansion is mostly inhibited
when space is greater than speed. A value of 0 enables
promiscuous inline expansion. Wide use of a 0
value is not recommended, as it may waste so much space that run
time is slowed. See section 5.8 for a discussion of
inline expansion.
- debug
- How debuggable the
program should be. The quality is treated differently from the
other qualities: each value indicates a particular level of
debugger information; it is not compared with the other qualities.
See section 3.6
for more details.
- safety
- How much error
checking should be done. If speed, space or compilation-speed is
more important than safety, then type
checking is weakened (see section 4.5.3). If safety
if 0, then no run time error checking is
done. In addition to suppressing type checks, 0 also suppresses argument count checking,
unbound-symbol checking and array bounds checks.
- extensions:inhibit-warnings
- This is a CMUCL
extension that determines how little (or how much) diagnostic
output should be printed during compilation. This quality is
compared to other qualities to determine whether to print style
notes and warnings concerning those qualities. If speed is greater than inhibit-warnings, then notes about how to improve speed
will be printed, etc. The default value is 1,
so raising the value for any standard quality above its default
enables notes for that quality. If inhibit-warnings is 3, then all
notes and most non-serious warnings are inhibited. This is useful
with declare to suppress warnings about
unavoidable problems.
4.7.2 The Optimize-Interface
Declaration
The extensions:optimize-interface declaration is identical
in syntax to the optimize declaration, but it
specifies the policy used during compilation of code the compiler
automatically generates to check the number and type of arguments
supplied to a function. It is useful to specify this policy
separately, since even thoroughly debugged functions are vulnerable
to being passed the wrong arguments. The optimize-interface declaration can specify that
arguments should be checked even when the general optimize policy is unsafe.
Note that this argument checking is the checking of user-supplied
arguments to any functions defined within the scope of the
declaration, not the checking of arguments to
Common Lisp primitives that appear in those definitions.
The idea behind this declaration is that it allows the definition
of functions that appear fully safe to other callers, but that do
no internal error checking. Of course, it is possible that
arguments may be invalid in ways other than having incorrect type.
Functions compiled unsafely must still protect themselves against
things like user-supplied array indices that are out of bounds and
improper lists. See also the :context-declarations option to with-compilation-unit.
4.8 Open Coding and Inline Expansion
Since Common Lisp forbids the redefinition of
standard functions4, the compiler can have special knowledge of
these standard functions embedded in it. This special knowledge is
used in various ways (open coding, inline expansion, source
transformation), but the implications to the user are basically the
same:
- Attempts to redefine standard functions may
be frustrated, since the function may never be called. Although it
is technically illegal to redefine standard functions, users
sometimes want to implicitly redefine these functions when they are
debugging using the trace macro.
Special-casing of standard functions can be inhibited using the
notinline declaration.
- The compiler can have multiple alternate
implementations of standard functions that implement different
trade-offs of speed, space and safety. This selection is based on
the compiler policy, see section 4.7.
When a function call is open coded, inline code whose
effect is equivalent to the function call is substituted for that
function call. When a function call is closed coded, it is
usually left as is, although it might be turned into a call to a
different function with different arguments. As an example, if
nthcdr were to be open coded, then
(nthcdr 4 foobar)
might turn into
(cdr (cdr (cdr (cdr foobar))))
or even
(do ((i 0 (1+ i))
(list foobar (cdr foobar)))
((= i 4) list))
If nth is closed coded, then
(nth x l)
might stay the same, or turn into something like:
(car (nthcdr x l))
In general, open coding sacrifices space for speed, but some
functions (such as car) are so simple that
they are always open-coded. Even when not open-coded, a call to a
standard function may be transformed into a different function call
(as in the last example) or compiled as static call.
Static function call uses a more efficient calling convention that
forbids redefinition.
- 1
- There are a few circumstances where a type
declaration is discarded rather than being used as type assertion.
This doesn't affect safety much, since such discarded declarations
are also not believed to be true by the compiler.
- 2
- The initial value need not be of this type as
long as the corresponding argument to the constructor is always
supplied, but this will cause a compile-time type warning unless
required-argument is used.
- 3
- Actually, this declaration is totally
unnecessary in Python, since it already knows position returns a non-negative fixnum or nil.
- 4
- See the proposed X3J13
“lisp-symbol-redefinition” cleanup.