The cmucl debugger is unique in its level of support for source-level debugging of compiled code. Although some other debuggers allow access of variables by name, this seems to be the first Common Lisp debugger that:
These features allow the debugging of compiled code to be made almost indistinguishable from interpreted code debugging.
The debugger is an interactive command loop that allows a user to examine the function call stack. The debugger is invoked when:
Note: there are two debugger interfaces in cmucl: the TTY debugger (described below) and the Motif debugger. Since the difference is only in the user interface, much of this chapter also applies to the Motif version. See section 2.9.1 for a very brief discussion of the graphical interface.
When you enter the TTY debugger, it looks something like this:
Error in function CAR. Wrong type argument, 3, should have been of type LIST. Restarts: 0: Return to Top-Level. Debug (type H for help) (CAR 3) 0]
The first group of lines describe what the error was that put us in the debugger. In this case car was called on 3. After Restarts: is a list of all the ways that we can restart execution after this error. In this case, the only option is to return to top-level. After printing its banner, the debugger prints the current frame and the debugger prompt.
The debugger is an interactive read-eval-print loop much like the normal top-level, but some symbols are interpreted as debugger commands instead of being evaluated. A debugger command starts with the symbol name of the command, possibly followed by some arguments on the same line. Some commands prompt for additional input. Debugger commands can be abbreviated by any unambiguous prefix: help can be typed as h, he, etc. For convenience, some commands have ambiguous one-letter abbreviations: f for frame.
The package is not significant in debugger commands; any symbol with the name of a debugger command will work. If you want to show the value of a variable that happens also to be the name of a debugger command, you can use the list-locals command or the debug:var function, or you can wrap the variable in a progn to hide it from the command loop.
The debugger prompt is “frame]”, where frame is the number of the current frame. Frames are numbered starting from zero at the top (most recent call), increasing down to the bottom. The current frame is the frame that commands refer to. The current frame also provides the lexical environment for evaluation of non-command forms.
The debugger evaluates forms in the lexical environment of the functions being debugged. The debugger can only access variables. You can’t go or return-from into a function, and you can’t call local functions. Special variable references are evaluated with their current value (the innermost binding around the debugger invocation)—you don’t get the value that the special had in the current frame. See section 3.4 for more information on debugger variable access.
A stack frame is the run-time representation of a call to a function; the frame stores the state that a function needs to remember what it is doing. Frames have:
These commands move to a new stack frame and print the name of the function and the values of its arguments in the style of a Lisp function call:
A frame is printed to look like a function call, but with the actual argument values in the argument positions. So the frame for this call in the source:
(myfun (+ 3 4) ’a)
would look like this:
(MYFUN 7 A)
All keyword and optional arguments are displayed with their actual values; if the corresponding argument was not supplied, the value will be the default. So this call:
(subseq "foo" 1)
would look like this:
(SUBSEQ "foo" 1 3)
And this call:
(string-upcase "test case")
would look like this:
(STRING-UPCASE "test case" :START 0 :END NIL)
The arguments to a function call are displayed by accessing the argument variables. Although those variables are initialized to the actual argument values, they can be set inside the function; in this case the new value will be displayed.
&rest arguments are handled somewhat differently. The value of the rest argument variable is displayed as the spread-out arguments to the call, so:
(format t "~A is a ~A." "This" ’test)
would look like this:
(FORMAT T "~A is a ~A." "This" ’TEST)
Rest arguments cause an exception to the normal display of keyword arguments in functions that have both &rest and &key arguments. In this case, the keyword argument variables are not displayed at all; the rest arg is displayed instead. So for these functions, only the keywords actually supplied will be shown, and the values displayed will be the argument values, not values of the (possibly modified) variables.
If the variable for an argument is never referenced by the function, it will be deleted. The variable value is then unavailable, so the debugger prints #<unused-arg> instead of the value. Similarly, if for any of a number of reasons (described in more detail in section 3.4) the value of the variable is unavailable or not known to be available, then #<unavailable-arg> will be printed instead of the argument value.
Printing of argument values is controlled by *debug-print-level* and *debug-print-length*.
If a function is defined by defun, labels, or flet, then the debugger will print the actual function name after the open parenthesis, like:
(STRING-UPCASE "test case" :START 0 :END NIL) ((SETF AREF) #\a "for" 1)
Otherwise, the function name is a string, and will be printed in quotes:
("DEFUN MYFUN" BAR)
("DEFMACRO DO" (DO ((I 0 (1+ I))) ((= I 13))) NIL)
("SETQ *GC-NOTIFY-BEFORE*")
This string name is derived from the defmumble form that encloses or expanded into the lambda, or the outermost enclosing form if there is no defmumble.
Sometimes the evaluator introduces new functions that are used to implement a user function, but are not directly specified in the source. The main place this is done is for checking argument type and syntax. Usually these functions do their thing and then go away, and thus are not seen on the stack in the debugger. But when you get some sort of error during lambda-list processing, you end up in the debugger on one of these funny frames.
These funny frames are flagged by printing “[keyword]” after the parentheses. For example, this call:
(car ’a ’b)
will look like this:
(CAR 2 A) [:EXTERNAL]
And this call:
(string-upcase "test case" :end)
would look like this:
("DEFUN STRING-UPCASE" "test case" 335544424 1) [:OPTIONAL]
As you can see, these frames have only a vague resemblance to the original call. Fortunately, the error message displayed when you enter the debugger will usually tell you what problem is (in these cases, too many arguments and odd keyword arguments.) Also, if you go down the stack to the frame for the calling function, you can display the original source (see section 3.5.)
With recursive or block compiled functions (see section 5.7), an :EXTERNAL frame may appear before the frame representing the first call to the recursive function or entry to the compiled block. This is a consequence of the way the compiler does block compilation: there is nothing odd with your program. You will also see :CLEANUP frames during the execution of unwind-protect cleanup code. Note that inline expansion and open-coding affect what frames are present in the debugger, see sections 3.6 and 4.8.
Both the compiler and the interpreter are “properly tail recursive.” If a function call is in a tail-recursive position, the stack frame will be deallocated at the time of the call, rather than after the call returns. Consider this backtrace:
(BAR ...) (FOO ...)
Because of tail recursion, it is not necessarily the case that FOO directly called BAR. It may be that FOO called some other function FOO2 which then called BAR tail-recursively, as in this example:
(defun foo () ... (foo2 ...) ...) (defun foo2 (...) ... (bar ...)) (defun bar (...) ...)
Usually the elimination of tail-recursive frames makes debugging more pleasant, since theses frames are mostly uninformative. If there is any doubt about how one function called another, it can usually be eliminated by finding the source location in the calling frame (section 3.5.)
The elimination of tail-recursive frames can be prevented by disabling tail-recursion optimization, which happens when the debug optimization quality is greater than 2 (see section 3.6.)
For a more thorough discussion of tail recursion, see section 5.5.
The debugger operates using special debugging information attached to the compiled code. This debug information tells the debugger what it needs to know about the locations in the code where the debugger can be invoked. If the debugger somehow encounters a location not described in the debug information, then it is said to be unknown. If the code location for a frame is unknown, then some variables may be inaccessible, and the source location cannot be precisely displayed.
There are three reasons why a code location could be unknown:
In the last two cases, the values of argument variables are accessible, but may be incorrect. See section 3.4.1 for more details on when variable values are accessible.
It is possible for an interrupt to happen when a function call or return is in progress. The debugger may then flame out with some obscure error or insist that the bottom of the stack has been reached, when the real problem is that the current stack frame can’t be located. If this happens, return from the interrupt and try again.
When running interpreted code, all locations should be known. However, an interrupt might catch some subfunction of the interpreter at an unknown location. In this case, you should be able to go up the stack a frame or two and reach an interpreted frame which can be debugged.
There are three ways to access the current frame’s local variables in the debugger. The simplest is to type the variable’s name into the debugger’s read-eval-print loop. The debugger will evaluate the variable reference as though it had appeared inside that frame.
The debugger doesn’t really understand lexical scoping; it has just one namespace for all the variables in a function. If a symbol is the name of multiple variables in the same function, then the reference appears ambiguous, even though lexical scoping specifies which value is visible at any given source location. If the scopes of the two variables are not nested, then the debugger can resolve the ambiguity by observing that only one variable is accessible.
When there are ambiguous variables, the evaluator assigns each one a small integer identifier. The debug:var function and the list-locals command use this identifier to distinguish between ambiguous variables:
This function returns the value of the variable in the current frame with the specified name. If supplied, identifier determines which value to return when there are ambiguous variables.
When name is a symbol, it is interpreted as the symbol name of the variable, i.e. the package is significant. If name is an uninterned symbol (gensym), then return the value of the uninterned variable with the same name. If name is a string, debug:var interprets it as the prefix of a variable name, and must unambiguously complete to the name of a valid variable.
This function is useful mainly for accessing the value of uninterned or ambiguous variables, since most variables can be evaluated directly.
The value of a variable may be unavailable to the debugger in portions of the program where Common Lisp says that the variable is defined. If a variable value is not available, the debugger will not let you read or write that variable. With one exception, the debugger will never display an incorrect value for a variable. Rather than displaying incorrect values, the debugger tells you the value is unavailable.
The one exception is this: if you interrupt (e.g., with ^C) or if there is an unexpected hardware error such as “bus error” (which should only happen in unsafe code), then the values displayed for arguments to the interrupted frame might be incorrect.1 This exception applies only to the interrupted frame: any frame farther down the stack will be fine.
The value of a variable may be unavailable for these reasons:
Since it is especially useful to be able to get the arguments to a function, argument variables are treated specially when the speed optimization quality is less than 3 and the debug quality is at least 1. With this compilation policy, the values of argument variables are almost always available everywhere in the function, even at unknown locations. For non-argument variables, debug must be at least 2 for values to be available, and even then, values are only available at known locations.
When the debugger command loop establishes variable bindings for available variables, these variable bindings have lexical scope and dynamic extent.2 You can close over them, but such closures can’t be used as upward funargs.
You can also set local variables using setq, but if the variable was closed over in the original source and never set, then setting the variable in the debugger may not change the value in all the functions the variable is defined in. Another risk of setting variables is that you may assign a value of a type that the compiler proved the variable could never take on. This may result in bad things happening.
One of cmucl’s unique capabilities is source level debugging of compiled code. These commands display the source location for the current frame:
The source form for a location in the code is the innermost list present in the original source that encloses the form responsible for generating that code. If the actual source form is not a list, then some enclosing list will be printed. For example, if the source form was a reference to the variable *some-random-special*, then the innermost enclosing evaluated form will be printed. Here are some possible enclosing forms:
(let ((a *some-random-special*)) ...) (+ *some-random-special* ...)
If the code at a location was generated from the expansion of a macro or a source-level compiler optimization, then the form in the original source that expanded into that code will be printed. Suppose the file /usr/me/mystuff.lisp looked like this:
(defmacro mymac () ’(myfun)) (defun foo () (mymac) ...)
If foo has called myfun, and is waiting for it to return, then the source command would print:
; File: /usr/me/mystuff.lisp (MYMAC)
Note that the macro use was printed, not the actual function call form, (myfun).
If enclosing source is printed by giving an argument to source or vsource, then the actual source form is marked by wrapping it in a list whose first element is #:***HERE***. In the previous example, source 1 would print:
; File: /usr/me/mystuff.lisp (DEFUN FOO () (#:***HERE*** (MYMAC)) ...)
If the code was defined from Common Lisp by compile or eval, then the source can always be reliably located. If the code was defined from a fasl file created by compile-file, then the debugger gets the source forms it prints by reading them from the original source file. This is a potential problem, since the source file might have moved or changed since the time it was compiled.
The source file is opened using the truename of the source file pathname originally given to the compiler. This is an absolute pathname with all logical names and symbolic links expanded. If the file can’t be located using this name, then the debugger gives up and signals an error.
If the source file can be found, but has been modified since the time it was compiled, the debugger prints this warning:
; File has been modified since compilation: ; filename ; Using form offset instead of character position.
where filename is the name of the source file. It then proceeds using a robust but not foolproof heuristic for locating the source. This heuristic works if:
If the heuristic doesn’t work, the displayed source will be wrong, but will probably be near the actual source. If the “shape” of the top-level form in the source file is too different from the original form, then an error will be signaled. When the heuristic is used, the the source location commands are noticeably slowed.
Source location printing can also be confused if (after the source was compiled) a read-macro you used in the code was redefined to expand into something different, or if a read-macro ever returns the same eq list twice. If you don’t define read macros and don’t use ## in perverted ways, you don’t need to worry about this.
Source location information is only available when the debug optimization quality is at least 2. If source location information is unavailable, the source commands will give an error message.
If source location information is available, but the source location is unknown because of an interrupt or unexpected hardware error (see section 3.3.6), then the command will print:
Unknown location: using block start.
and then proceed to print the source location for the start of the basic block enclosing the code location. It’s a bit complicated to explain exactly what a basic block is, but here are some properties of the block start location:
In other words, the true location lies between the printed location and the next conditional (but watch out because the compiler may have changed the program on you.)
The compilation policy specified by optimize declarations affects the behavior seen in the debugger. The debug quality directly affects the debugger by controlling the amount of debugger information dumped. Other optimization qualities have indirect but observable effects due to changes in the way compilation is done.
Unlike the other optimization qualities (which are compared in relative value to evaluate tradeoffs), the debug optimization quality is directly translated to a level of debug information. This absolute interpretation allows the user to count on a particular amount of debug information being available even when the values of the other qualities are changed during compilation. These are the levels of debug information that correspond to the values of the debug quality:
As you can see, if the speed quality is 3, debugger performance is degraded. This effect comes from the elimination of argument variable special-casing (see section 3.4.1.) Some degree of speed/debuggability tradeoff is unavoidable, but the effect is not too drastic when debug is at least 2.
In addition to inline and notinline declarations, the relative values of the speed and space qualities also change whether functions are inline expanded (see section 5.8.) If a function is inline expanded, then there will be no frame to represent the call, and the arguments will be treated like any other local variable. Functions may also be “semi-inline”, in which case there is a frame to represent the call, but the call is to an optimized local version of the function, not to the original function.
These commands get you out of the debugger.
Most of these commands print information about the current frame or function, but a few show general information.
cmucl supports setting of breakpoints inside compiled functions and stepping of compiled code. Breakpoints can only be set at at known locations (see section 3.3.6), so these commands are largely useless unless the debug optimize quality is at least 2 (see section 3.6). These commands manipulate breakpoints:
If consecutive locations have the same source, then a numeric range like 3-5: will be printed. For example, a default function call has a known location both immediately before and after the call, which would result in two code locations with the same source. The listed function becomes the new default function for breakpoint setting (via the breakpoint) command.3: (1- N)
Consider this definition of the factorial function:
(defun ! (n)
(if (zerop n)
1
(* n (! (1- n)))))
This debugger session demonstrates the use of breakpoints:
common-lisp-user> (break) ; Invoke debugger Break Restarts: 0: [CONTINUE] Return from BREAK. 1: [ABORT ] Return to Top-Level. Debug (type H for help) (INTERACTIVE-EVAL (BREAK)) 0] ll #’! 0: #’(LAMBDA (N) (BLOCK ! (IF # 1 #))) 1: (ZEROP N) 2: (* N (! (1- N))) 3: (1- N) 4: (! (1- N)) 5: (* N (! (1- N))) 6: #’(LAMBDA (N) (BLOCK ! (IF # 1 #))) 0] br 2 (* N (! (1- N))) 1: 2 in ! Added. 0] q common-lisp-user> (! 10) ; Call the function *Breakpoint hit* Restarts: 0: [CONTINUE] Return from BREAK. 1: [ABORT ] Return to Top-Level. Debug (type H for help) (! 10) ; We are now in first call (arg 10) before the multiply Source: (* N (! (1- N))) 3] st *Step* (! 10) ; We have finished evaluation of (1- n) Source: (1- N) 3] st *Breakpoint hit* Restarts: 0: [CONTINUE] Return from BREAK. 1: [ABORT ] Return to Top-Level. Debug (type H for help) (! 9) ; We hit the breakpoint in the recursive call Source: (* N (! (1- N))) 3]
The tracer causes selected functions to print their arguments and their results whenever they are called. Options allow conditional printing of the trace information and conditional breakpoints on function entry or exit.
trace is a debugging tool that prints information when specified functions are called. In its simplest form:
(trace name-1 name-2 ...)trace causes a printout on *trace-output* each time that one of the named functions is entered or returns (the names are not evaluated.) Trace output is indented according to the number of pending traced calls, and this trace depth is printed at the beginning of each line of output. Printing verbosity of arguments and return values is controlled by *debug-print-level* and *debug-print-length*.
Local functions defined by flet and labels can be traced using the syntax (flet f f1 f2 ...) or (labels f f1 f2 ...) where f is the flet or labels function we want to trace and f1, f2, are the functions containing the local function f. Invidiual methods can also be traced using the syntax (method name qualifiers specializers). See 2.23.7 for more information.
If no names or options are are given, trace returns the list of all currently traced functions, *traced-function-list*.
Trace options can cause the normal printout to be suppressed, or cause extra information to be printed. Each option is a pair of an option keyword and a value form. Options may be interspersed with function names. Options only affect tracing of the function whose name they appear immediately after. Global options are specified before the first name, and affect all functions traced by a given use of trace. If an already traced function is traced again, any new options replace the old options. The following options are defined:
- :condition form, :condition-after form, :condition-all form
- If :condition is specified, then trace does nothing unless form evaluates to true at the time of the call. :condition-after is similar, but suppresses the initial printout, and is tested when the function returns. :condition-all tries both before and after.
- :wherein names
- If specified, names is a function name or list of names. trace does nothing unless a call to one of those functions encloses the call to this function (i.e. it would appear in a backtrace.) Anonymous functions have string names like "DEFUN FOO". Individual methods can also be traced. See section 2.23.7.
- :wherein-only names
- If specified, this is just like :wherein, but trace produces output only if the immediate caller of the traced function is one of the functions listed in names.
- :break form, :break-after form, :break-all form
- If specified, and form evaluates to true, then the debugger is invoked at the start of the function, at the end of the function, or both, according to the respective option.
- :print form, :print-after form, :print-all form
- In addition to the usual printout, the result of evaluating form is printed at the start of the function, at the end of the function, or both, according to the respective option. Multiple print options cause multiple values to be printed.
- :function function-form
- This is a not really an option, but rather another way of specifying what function to trace. The function-form is evaluated immediately, and the resulting function is traced.
- :encapsulate {:default | t | nil}
- In cmucl, tracing can be done either by temporarily redefining the function name (encapsulation), or using breakpoints. When breakpoints are used, the function object itself is destructively modified to cause the tracing action. The advantage of using breakpoints is that tracing works even when the function is anonymously called via funcall.
When :encapsulate is true, tracing is done via encapsulation. :default is the default, and means to use encapsulation for interpreted functions and funcallable instances, breakpoints otherwise. When encapsulation is used, forms are not evaluated in the function’s lexical environment, but debug:arg can still be used.
Note that if you trace using :encapsulate, you will only get a trace or breakpoint at the outermost call to the traced function, not on recursive calls.
In the case of functions where the known return convention is used to optimize, encapsulation may be necessary in order to make tracing work at all. The symptom of this occurring is an error stating
Error in function foo: :FUNCTION-END breakpoints are currently unsupported for the known return convention.in such cases we recommend using (trace foo :encapsulate t)
:condition, :break and :print forms are evaluated in the lexical environment of the called function; debug:var and debug:arg can be used. The -after and -all forms are evaluated in the null environment.
This macro turns off tracing for the specified functions, and removes their names from *traced-function-list*. If no function-names are given, then all currently traced functions are untraced.
A list of function names maintained and used by trace, untrace, and untrace-all. This list should contain the names of all functions currently being traced.
The maximum number of spaces which should be used to indent trace printout. This variable is initially set to 40.
A list of package names. Functions from these packages are traced using encapsulation instead of function-end breakpoints. This list should at least include those packages containing functions used directly or indirectly in the implementation of trace.
The encapsulation functions provide a mechanism for intercepting the arguments and results of a function. encapsulate changes the function definition of a symbol, and saves it so that it can be restored later. The new definition normally calls the original definition. The Common Lisp fdefinition function always returns the original definition, stripping off any encapsulation.
The original definition of the symbol can be restored at any time by the unencapsulate function. encapsulate and unencapsulate allow a symbol to be multiply encapsulated in such a way that different encapsulations can be completely transparent to each other.
Each encapsulation has a type which may be an arbitrary lisp object. If a symbol has several encapsulations of different types, then any one of them can be removed without affecting more recent ones. A symbol may have more than one encapsulation of the same type, but only the most recent one can be undone.
Saves the current definition of symbol, and replaces it with a function which returns the result of evaluating the form, body. Type is an arbitrary lisp object which is the type of encapsulation.
When the new function is called, the following variables are bound for the evaluation of body:
- extensions:argument-list
- A list of the arguments to the function.
- extensions:basic-definition
- The unencapsulated definition of the function.
The unencapsulated definition may be called with the original arguments by including the form
(apply extensions:basic-definition extensions:argument-list)encapsulate always returns symbol.
Undoes symbol’s most recent encapsulation of type type. Type is compared with eq. Encapsulations of other types are left in place.
Returns t if symbol has an encapsulation of type type. Returns nil otherwise. type is compared with eq.
Here is an example of tracing with some of the possible options. For simplicity, this is the function:
(defun fact (n)
(declare (double-float n) (optimize speed))
(if (zerop n)
1d0
(* n (fact (1- n)))))
(compile ’fact)
This example shows how to use the :condition option:
(trace fact :condition (= 4d0 (debug:arg 0)))
(fact 10d0) ->
0: (FACT 4.0d0)
0: FACT returned 24.0d0
3628800.0d0
As we can see, we produced output when the condition was satisfied.
Here’s another example:
(untrace)
(trace fact :break (= 4d0 (debug:arg 0)))
(fact 10d0) ->
0: (FACT 5.0d0)
1: (FACT 4.0d0)
Breaking before traced call to FACT:
[Condition of type SIMPLE-CONDITION]
Restarts:
0: [CONTINUE] Return from BREAK.
1: [ABORT ] Return to Top-Level.
Debug (type H for help)
In this example, we see that normal tracing occurs until we the argument reaches 4d0, at which point, we break into the debugger.
These are the special variables that control the debugger action.
*print-level* and *print-length* are bound to these values during the execution of some debug commands. When evaluating arbitrary expressions in the debugger, the normal values of *print-level* and *print-length* are in effect. These variables are initially set to 3 and 5, respectively.