Instead of referring to the parts of our form by (second x) and such, we would like to give names to the various pieces of the form, and somehow have the (second x) automatically generated. This is done by a macro called defmacro. It is easiest to explain what defmacro does by showing an example. Here is how you would write the for macro using defmacro:

The (var lower upper . body) is a pattern to match against the body of the form (to be more precise, to match against the cdr of the argument to the macro's expander function). If defmacro tries to match the two lists (var lower upper . body) and (a 1 100 (print a) (print (* a a))) var is bound to the symbol a, lower to the fixnum 1, upper to the fixnum 100, and body to the list ((print a) (print (* a a))). var, lower, upper, and body are then used by the body of the macro definition.

Note that the pattern need not be a list the way a lambda-list must. In the above example, the pattern was a dotted list, since the symbol body was supposed to match the cddddr of the macro form. If we wanted a new iteration form, like for except that our example would look like (for a (1 100) (print a) (print (* a a))) (just because we thought that was a nicer syntax), then we could do it merely by modifying the pattern of the defmacro above; the new pattern would be (var (lower upper) . body).

Here is how we would write our other examples using defmacro:

All of these were very simple macros and have very simple patterns, but these examples show that we can replace the (cadr x) with a readable mnemonic name such as the-list or symbol, which makes the program clearer, and enables documentation facilities such as the arglist function to describe the syntax of the special form defined by the macro.

The pattern in a defmacro is more like the lambda list of a normal function than revealed above. It is allowed to contain certain &-keywords. Subpatterns of the lambda list pattern can also use &-keywords, a usage not allowed in functions.

&optional is followed by variable, (variable), (variable default), or (variable default present-p), exactly the same as in a function. Note that default is still a form to be evaluated, even though variable is not being bound to the value of a form. variable does not have to be a symbol; it can be a pattern. In this case the first form is disallowed because it is syntactically ambigous. The pattern must at least be enclosed in a singleton list. If variable is a pattern, default can be evaluated more than once. Example: (defmacro foo (&optional ((x &optional y) '(a))) ...) Here the first argument of foo is optional, and should be a list of one or two elements which become x and y. If foo is given no arguments, the list (a) is decomposed to get x and y, so that x's value is a and y's value is nil.

Using &rest is the same as using a dotted list as the pattern, except that it may be easier to read and leaves a place to put &aux.

When &key is used in a defmacro pattern, the keywords are decoded at macro expansion time. Therefore, they must be constants. Example:

&aux is the same in a macro as in a function, and has nothing to do with pattern matching.

defmacro implements a few additional keywords not allowed in functions.

&body is identical to &rest except that it informs the editor and the grinder that the remaining subforms constitute a ``body'' rather than ordinary arguments and should be indented accordingly. Example: (defmacro with-open-file ((streamvar filename &rest options) &body body) ...)

&whole causes the variable that follows it to be bound to the entire macro call, just as the form-arg variable in macro would be. &whole exists to make defmacro able to do anything that macro can be used for, for the sake of Common Lisp, in which defmacro is the primitive and macro does not exist. &whole is also useful in macrolet.

&environment causes the variable that follows it to be bound to the local macros environment of the macro call being expanded. This is useful if the code for expanding this macro needs to invoke macroexpand on subforms of the macro call. Then, to achieve correct interaction with macrolet, this local macros environment should be passed to macroexpand as its second argument.

&list-of pattern requires that the corresponding position of the form being translated must contain a list (or nil). It matches pattern against each element of that list. Each variable in pattern is bound to a list of the corresponding values in each element of the list matched by the &list-of. This may be clarified by an example. Suppose we want to be able to say things like:

We could define a send-commands macro as follows:

Note that this example uses &body together with &list-of, so you don't see the list itself; the list is just the rest of the macro-form.

You can combine &optional and &list-of. Consider the following example:

In this example we aren't using &body or anything like it, so you do see the list itself; that is why you see parentheses around the (bar 3).

Now we deal with the other problem: the long strings of calls to cons and list. This problem is relieved by introducing some new characters that are special to the Lisp reader. Just as the single-quote character makes it easier to type things of the form (quote x), so backquote and comma make it easier to type forms that create new list structure. They allow you to create a list from a template including constant and variable parts.

The backquote facility is used by giving a backquote character ( ` ), followed by a list or vector. If the comma character does not appear within the text for the list or vector, the backquote acts just like a single quote: it creates a form which, when evaluated, produces the list or vector specified. For example, '(a b c) => (a b c) `(a b c) => (a b c) `#(a b) => #(a b) So in the simple cases, backquote is just like the regular single-quote macro. The way to get it to do interesting things is to include a comma somewhere inside of the form following the backquote. The comma is followed by a form, and that form gets evaluated even though it is inside the backquote. For example, (setq b 1) `(a b c) => (a b c) `(a ,b c) => (a 1 c) `(abc ,(+ b 4) ,(- b 1) (def ,b)) => (abc 5 0 (def 1)) `#(a ,b) => #(a 1) In other words, backquote quotes everything except expressions preceded by a comma; those get evaluated.

The list or vector following a backquote can be thought of as a template for some new data structure. The parts of it that are preceded by commas are forms that fill in slots in the template; everything else is just constant structure that appears as written in the result. This is usually what you want in the body of a macro. Some of the form generated by the macro is constant, the same thing on every invocation of the macro. Other parts are different every time the macro is called, often being functions of the form that the macro appeared in (the arguments of the macro). The latter parts are the ones for which you would use the comma. Several examples of this sort of use follow.

When the reader sees the `(a ,b c) it is actually generating a form such as (list 'a b 'c). The actual form generated may use list, cons, append, or whatever might be a good idea; you should never have to concern yourself with what it actually turns into. All you need to care about is what it evaluates to. Actually, it doesn't use the regular functions cons, list, and so forth, but uses special ones instead so that the grinder can recognize a form which was created with the backquote syntax, and print it using backquote so that it looks like what you typed in. You should never write any program that depends on this, anyway, because backquote makes no guarantees about how it does what it does. In particular, in some circumstances it may decide to create constant forms, which will cause sharing of list structure at run time, or it may decide to create forms that will create new list structure at run time. For example, if the reader sees `(r . ,nil), it may produce the same thing as (cons 'r nil), or '(r . nil). Be careful that your program does not depend on which of these it does.

This is generally found to be pretty confusing by most people; the best way to explain further seems to be with examples. Here is how we would write our three simple macros using both the defmacro and backquote facilities.

To demonstrate finally how easy it is to define macros with these two facilities, here is the final form of the for macro. (defmacro for (var lower upper . body) `(do ((,var ,lower (1+ ,var))) ((> ,var ,upper)) . ,body)) Look at how much simpler that is than the original definition. Also, look how closely it resembles the code it is producing. The functionality of the for really stands right out when written this way.

If a comma inside a backquote form is followed by an at-sign character (`@'), it has a special meaning. The `,@' should be followed by a form whose value is a list; then each of the elements of the list is put into the list being created by the backquote. In other words, instead of generating a call to the cons function, backquote generates a call to append. For example, if a is bound to (x y z), then `(1 ,a 2) would evaluate to (1 (x y z) 2), but `(1 ,@a 2) would evaluate to (1 x y z 2).

Here is an example of a macro definition that uses the `,@' construction. One way to define do-forever would be for it to expand (do-forever form1 form2 form3) into (tagbody a form1 form2 form3 (go a))

You could define the macro by (defmacro do-forever (&body body) `(tagbody a ,@body (go a))) (This definition has the disadvantage of interfering with use of the go tag a to go from the body of the do-forever to a tag defined outside of it. A more robust implementation would construct a new tag each time, using gensym.)

A similar construct is `,.' (comma, dot). This means the same thing as `,@' except that the list which is the value of the following form may be modified destructively; backquote uses nconc rather than append. This should, of course, be used with caution.

Backquote does not make any guarantees about what parts of the structure it shares and what parts it copies. You should not do destructive operations such as nconc on the results of backquote forms such as `(,a b c d) since backquote might choose to implement this as (cons a '(b c d)) and nconc would smash the constant. On the other hand, it would be safe to nconc the result of `(a b ,c ,d) since any possible expansion of this would make a new list. One possible expansion is (list 'a 'b c d)

Backquote of course guarantees not to do any destructive operations (rplaca, rplacd, nconc) on the components of the structure it builds, unless the `,.' syntax is used.

Advanced macro writers sometimes write macro-defining macros: forms which expand into forms which, when evaluated, define macros. In such macros it is often useful to use nested backquote constructs. For example, here is a very simple version of defstruct (see ) which does not allow any options and only the simplest slot descriptors. Its invocation looks like: (defstruct (name) item1 item2 ...) We would like this form to expand into (progn (defmacro item1 (x) `(aref ,x 0)) (defmacro item2 (x) `(aref ,x 1)) (defmacro item3 (x) `(aref ,x 2)) (defmacro item4 (x) `(aref ,x 3)) ...)

Here is the macro to perform the expansion: (defmacro defstruct ((name) . items) (do ((item-list items (cdr item-list)) (ans nil) (i 0 (1+ i))) ((null item-list) `(progn . ,(nreverse ans))) (push `(defmacro ,(car item-list) (x) `(aref ,x ,',i)) ans)))

The interesting part of this definition is the body of the (inner) defmacro form: `(aref ,x ,',i) Instead of using this backquote construction, we could have written (list 'aref x ,i) That is, the ,', acts like a comma that matches the outer backquote, while the comma preceding the x matches with the inner backquote. Thus, the symbol i is evaluated when the defstruct form is expanded, whereas the symbol x is evaluated when the accessor macros are expanded.

Backquote can be useful in situations other than the writing of macros. Whenever there is a piece of list structure to be consed up, most of which is constant, the use of backquote can make the program considerably clearer.

One of the most common errors in writing macros is best illustrated by example. Suppose we wanted to write dolist (see ) as a macro that expanded into a do (see ). The first step, as always, is to figure out what the expansion should look like. Let's pick a representative example form, and figure out what its expansion should be. Here is a typical dolist form.

We want to create a do form that does the thing that the above dolist form says to do. That is the basic goal of the macro: it must expand into code that does the same thing that the original code says to do, but it should be in terms of existing Lisp constructs. The do form might look like this:

Now we could start writing the macro that would generate this code, and in general convert any dolist into a do, in an analogous way. However, there is a problem with the above scheme for expanding the dolist. The above example's expansion works fine. But what if the input form had been the following:

This is just like the form we saw above, except that the programmer happened to decide to name the looping variable list rather than element. The corresponding expansion would be:

This doesn't work at all! In fact, this is not even a valid program, since it contains a do that uses the same variable in two different iteration clauses.

Here's another example that causes trouble:

If you work out the expansion of this form, you will see that there are two variables named list, and that the programmer meant to refer to the outer one but the generated code for the push actually uses the inner one.

The problem here is an accidental name conflict. This can happen in any macro that has to create a new variable. If that variable ever appears in a context in which user code might access it, then you have to worry that it might conflict with some other name that the user is using for his own program.

One way to avoid this problem is to choose a name that is very unlikely to be picked by the user, simply by choosing an unusual name, in a package which only you will write code in. This will probably work, but it is inelegant since there is no guarantee that the user won't just happen to choose the same name. The way to avoid the name conflict reliably is to use an uninterned symbol as the variable in the generated code. The function gensym (see ) is useful for creating such symbols.

Here is the expansion of the original form, using an uninterned symbol created by gensym.

This is the right kind of thing to expand into. (This is how the expression would print; this text would not read in properly because a new uninterned symbol would be created by each use of #:.) Now that we understand how the expansion works, we are ready to actually write the macro. Here it is:

Many system macros do not use gensym for the internal variables in their expansions. Instead they use symbols whose print names begin and end with a dot. This provides meaningful names for these variables when looking at the generated code and when looking at the state of a computation in the error-handler. These symbols are in the si package; as a result, a name conflict is possible only in code which uses variables in the si package. This would not normally happen in user code, which resides in other packages.

A related problem occurs when you write a macro that expands into a prog or do (or anything equivalent) behind the user's back (unlike dolist, which is documented to be like do). Consider the error-restart special form (see ). Suppose we wanted to implement it as a macro that expands into a do-forever, which becomes a prog. Then the following (contrived) Lisp program would not behave correctly:

The problem is that the return would return from the error-restart instead of the prog.

There are two possible ways to avoid this. The best is to make the expanded code use only explicit block's with obscure or gensymmed block names, and never a prog or do.

The other is to give any prog or do the name t. t as a prog name is special; it causes the prog to generate only a block named t, omitting the usual block named nil which is normally generated as well. Because only blocks named nil affect return, the problem is avoided.

When error-restart's expansion is supposed to return from the prog named t, it uses return-from t.

Macros like dolist specifically should expand into an ordinary do, because the user expects to be able to exit them with return.

Sometimes a macro wants to do several different things when its expansion is evaluated. Another way to say this is that sometimes a macro wants to expand into several things, all of which should happen sequentially at run time (not macro-expand time). For example, suppose you wanted to implement defconst (see ) as a macro. defconst must do two things, declare the variable to be special and set the variable to its initial value. (Here we implement a simplified defconst that does only these two things, and doesn't have any options.) What should a defconst form expand into? Well, what we would like is for an appearance of (defconst a (+ 4 b)) in a file to be the same thing as the appearance of the following two forms: (proclaim '(special a)) (setq a (+ 4 b)) However, because of the way that macros work, they only expand into one form, not two. So we need to have a defconst form expand into one form that is just like having two forms in the file.

There is such a form. It looks like this: (progn (proclaim '(special a)) (setq a (+ 4 b))) In interpreted Lisp, it is easy to see what happens here. This is a progn special form, and so all its subforms are evaluated, in turn. The proclaim form and the setq form are evaluated. The compiler recognizes progn specially and treats each argument of the progn form as if it had been encountered at top level. Here is the macro definition:

Here is another example of a form that wants to expand into several things. We implement a special form called define-command, which is intended to be used in order to define commands in some interactive user subsystem. For each command, there are two things provided by the define-command form: a function that executes the command, and a character that should invoke the function in this subsystem. Suppose that in this subsystem, commands are always functions of no arguments, and characters are used to index a vector called dispatch-table to find the function to use. A typical call to define-command would look like:

The define-command expands into three forms. The first one sets up the specified character to invoke this command. The second one puts the command name onto the list of all command names. The third one is the defun that actually defines the function itself. Note that the setf and push happen at load-time (when the file is loaded); the function, of course, also gets defined at load time. (See the description of eval-when () for more discussion of the differences between compile time, load time, and eval time.)

This technique makes Lisp a powerful language in which to implement your own language. When you write a large system in Lisp, frequently you can make things much more convenient and clear by using macros to extend Lisp into a customized language for your application. In the above example, we have created a little language extension: a new special form that defines commands for our system. It lets the writer of the system attach the code for a command character to the character itself. Macro expansion allows the function definitions and the command dispatch table to be made from the same source code.

There is a particular kind of macro that is very useful for many applications. This is a macro that you place ``around'' some Lisp code, in order to make the evaluation of that code happen in a modified context. For a very simple example, we could define a macro called with-output-in-base, that executes the forms within its body with any output of numbers that is done defaulting to a specified base. (defmacro with-output-in-base ((base-form) &body body) `(let ((*print-base* ,base-form)) . ,body)) A typical use of this macro might look like: (with-output-in-base (*default-base*) (print x) (print y)) which would expand into (let ((*print-base* *default-base*)) (print x) (print y))

This example is too trivial to be very useful; it is intended to demonstrate some stylistic issues. There are standard Zetalisp constructs that are similar to this macro; see with-open-file () and with-input-from-string (), for example. The really interesting thing, of course, is that you can define your own such constructs for your applications. One very powerful application of this technique was used in a system that manipulates and solves the Rubik's cube puzzle. The system heavily uses a construct called with-front-and-top, whose meaning is ``evaluate this code in a context in which this specified face of the cube is considered the front face, and this other specified face is considered the top face''.

The first thing to keep in mind when you write this sort of macro is that you can make your macro much clearer to people who might read your program if you conform to a set of loose standards of syntactic style. By convention, the names of such constructs start with ``with-''. This seems to be a clear way of expressing the concept that we are setting up a context; the meaning of the construct is ``do this stuff with the following things true''. Another convention is that any ``parameters'' to the construct should appear in a list that is the first subform of the construct, and that the rest of the elements should make up a body of forms that are evaluated sequentially with the last one returned. All of the examples cited above work this way. In our with-output-in-base example, there was one parameter (the base), which appears as the first (and only) element of a list that is the first subform of the construct. The extra level of parentheses in the printed representation serves to separate the ``parameter'' forms from the ``body'' forms so that it is textually apparent which is which; it also provides a convenient way to provide default parameters (a good example is the with-input-from-string construct (), which takes two required and two optional parameters). Another convention/technique is to use the &body keyword in the defmacro to tell the editor how to indent the elements of the body (see ).

The other thing to keep in mind is that control can leave the construct either by the last form's returning, or by a non-local exit (go, return or throw). You should write the definition in such a way that everything is cleaned up appropriately no matter how control exits. In our with-output-in-base example, there is no problem, because non-local exits undo lambda-bindings. However, in even slightly more complicated cases, an unwind-protect form (see ) is needed: the macro must expand into an unwind-protect that surrounds the body, with ``cleanup'' forms that undo the context-setting-up that the macro did. For example, using-resource (see ) expands (using-resource (window menu-resource) body...) into (let ((window nil)) (unwind-protect (progn (setq window (allocate-resource 'menu-resource)) body...) (and window (deallocate-resource 'menu-resource window)))) This way the allocated resource item is deallocated whenever control leaves the using-resource special form.

In any macro, you should always pay attention to the problem of multiple or out-of-order evaluation of user subforms. Here is an example of a macro with such a problem. This macro defines a special form with two subforms. The first is a reference, and the second is a form. The special form is defined to create a cons whose car and cdr are both the value of the second subform, and then to set the reference to be that cons. Here is a possible definition: (defmacro test (reference form) `(setf ,reference (cons ,form ,form))) Simple cases work all right: (test foo 3) ==> (setf foo (cons 3 3)) But a more complex example, in which the subform has side effects, can produce surprising results: (test foo (setq x (1+ x))) ==> (setf foo (cons (setq x (1+ x)) (setq x (1+ x)))) The resulting code evaluates the setq form twice, and so x is increased by two instead of by one. A better definition of test that avoids this problem is: (defmacro test (reference form) (let ((value (gensym))) `(let ((,value ,form)) (setf ,reference (cons ,value ,value))))) With this definition, the expansion works as follows: (test foo (setq x (1+ x))) ==> (let ((#:g0005 (setq x (1+ x)))) (setf foo (cons #:g0005 #:g0005))) Once again, the expansion would print this way, but this text would not read in as a valid expression due to the inevitable problems of #:.

In general, when you define a new construct which contains one or more argument forms, you must be careful that the expansion evaluates the argument forms the proper number of times and in the proper order. There's nothing fundamentally wrong with multiple or out-of-order evalation if that is really what you want and if it is what you document your special form to do. But if this happens unexpectedly, it can make invocations fail to work as they appear they should.

once-only is a macro that can be used to avoid multiple evaluation. It is most easily explained by example. You would write test using once-only as follows: (defmacro test (reference form) (once-only (form) `(setf ,reference (cons ,form ,form)))) This defines test in such a way that the form is only evaluated once, and references to form inside the macro body refer to that value. once-only automatically introduces a lambda-binding of a generated symbol to hold the value of the form. Actually, it is more clever than that; it avoids introducing the lambda-binding for forms whose evaluation is trivial and may be repeated without harm or cost, such as numbers, symbols, and quoted structure. This is just an optimization that helps produce more efficient code.

The once-only macro makes it easier to follow the principle, but it does not completely or automatically solve the problems of multiple and out-of-order evaluation. It is just a tool that can solve some of the problems some of the time; it is not a panacea.

The following description attempts to explain what once-only does, but it is a lot easier to use once-only by imitating the example above than by trying to understand once-only's rather tricky definition.

A useful technique for building language extensions is to define programming constructs that employ two special forms, one of which is used inside the body of the other. Here is a simple example. There are two special forms. The outer one is called with-collection, and the inner one is called collect. collect takes one subform, which it evaluates; with-collection just has a body, whose forms it evaluates sequentially. with-collection returns a list of all of the values that were given to collect during the evaluation of the with-collection's body. For example, (with-collection (dotimes (i 5) (collect i))) => (1 2 3 4 5) Remembering the first piece of advice we gave about macros, the next thing to do is to figure out what the expansion looks like. Here is how the above example could expand:

Now, how do we write the definition of the macros? Well, with-collection is pretty easy:

The hard part is writing collect. Let's try it: (defmacro collect (argument) `(push ,argument ,var))

Note that something unusual is going on here: collect is using the variable var freely. It is depending on the binding that takes place in the body of with-collection in order to get access to the value of var. Unfortunately, that binding took place when with-collection got expanded; with-collection's expander function bound var, and the binding of var was unmade when the expander function was done. By the time the collect form gets expanded, the binding is long gone. The macro definitions above do not work. Somehow the expander function of with-collection has to communicate with the expander function of collect to pass over the generated symbol.

The only way for with-collection to convey information to the expander function of collect is for it to expand into something that passes that information.

One way to write these macros is using macrolet:

Here with-collection expands into code which defines collect specially to know about which variable to collect into. ,', causes var's value to be substituted when the outer backquote, the one around the macrolet, is executed. argument, however, is substituted in when the inner backquote is executed, which happens when collect is expanded.

This technique has the interesting consequence that collect is defined only within the body of a with-collection. It would simply not be recognized elsewhere; or it could have another definition, for some other purpose, globally. This has both advantages and disadvantages.

Another technique is to communicate through local declarations. The code generated by with-collection can contain a local-declare. The expansion of collect can examine the declararion with getdecl to decide what to do. Here is the code:

Another way, used before getdecl existed, was with compiler-let (see ). compiler-let is identical to let as far as the interpreter is concerned, so the macro continues to work in the interpreter with this change. When the compiler encounters a compiler-let, however, it actually performs the bindings that the compiler-let specifies and proceeds to compile the body of the compiler-let with all of those bindings in effect. In other words, it acts as the interpreter would.

Here's the right way to write these macros in this fashion:

The technique of defining functions to be used during macro expansion deserves explicit mention here. It may not occur to you, but a macro expander function is a Lisp program like any other Lisp program, and it can benefit in all the usual ways by being broken down into a collection of functions that do various parts of its work. Usually macro expander functions are pretty simple Lisp programs that take things apart and put them together slightly differently, but some macros are quite complex and do a lot of work. Several features of Zetalisp, including flavors, loop, and defstruct, are implemented using very complex macros, which, like any complex well-written Lisp program, are broken down into modular functions. You should keep this in mind if you ever invent an advanced language extension or ever find yourself writing a five-page expander function.

A particular thing to note is that any functions used by macro-expander functions must be available at compile-time. You can make a function available at compile time by surrounding its defining form with an (eval-when (compile load eval) ...); see for more details. Doing this means that at compile time the definition of the function is interpreted, not compiled, and hence runs more slowly.

Another approach is to separate macro definitions and the functions they call during expansion into a separate file, often called a ``defs'' (definitions) file. This file defines all the macros, and also all functions that the macros call. It can be separately compiled and loaded up before compiling the main part of the program, which uses the macros. The system facility (see ) helps keep these various files straight, compiling and loading things in the right order.