Star Sapphire Common LISP Home

Download Star Saphire
Index

3. SCOPE AND EXTENT

This chapter contains the following sections:

3.1 Introduction to Lexical Scoping

3.2 Introduction to Dynamic Scoping

3.3 Advanced Scoping and Extent

3.4 Scoping and Extent Rules

This chapter discusses the semantics or meaning of LISP programs. The key to understanding LISP semantics is understanding scope and extent.

Scoping refers to a restricted area in space where references to some variable can occur. Extent refers to a duration in time when references to a variable can occur.

The following two sections discuss lexically and dynamically scoped variables as an introduction to the subject; the final two sections in this chapter give a more formal treatment of this issue.

3.1 Introduction to Lexical Scoping

Lexical scoping is basically a way of defining variables which take on values only while execution is occuring in a given context. This context can be visualized by checking the parenthesis which most closely surround the form that the variable is defined in.

Lexical scoping is the default mechanism for establishing values for variables in Common LISP.

Here is a simple example.

Consider the following function:

(defun foo (x)

(setq x 15)

(print x))

The defun form defines a function called foo with one argument named x. The setq special form is used to assign a value to either a lexically or dynamically scoped variable.

Assume the environment has just been booted from scratch: in particular, no other uses of the symbol x have occured yet.

In this case x is a lexically scoped variable. In brief, referring to x has no meaning outside of the function foo.

The effect of running foo with any value is the same: 15 is printed and returned as the value of the form.

LISP: (foo 3)

15 15

LISP: (foo 23432)

15 15

LISP: (foo "hello, world!")

15 15

At this point if you tried to obtain the value of x for instance by just typing x at the LISP prompt, you will get an error; x has no value, is unbound.

The explanation is as follows. The variable x is a symbol, certainly. If you had previously assigned a value to x by typing (setq x "abc") at the LISP prompt, then typing x at the LISP prompt would return the string "abc".

However, by writing (x) for the parameter list of the function foo, we are essentially telling the system that x is to be used as an alias for 'the first and only argument to this function in the body of the function foo.

Implementationally, when a given function is called, a stack frame is created. In Star Sapphire, the LISP stack is different from the system stack; it is a stack of virtual addresses, which is nevertheless kept in physical memory for the sake of efficiency. When the function foo is called it expects to get one argument. This Note that foo does not actually do anything with the input value; the very first thing that it does is clobber whatever value is in that slot on the stack with the number 15.

Note that we haven't talked about x at all in the last two paragraphs. This is because x at this point is long gone, having been converted into a lexical variable object by the incremental compiler. The setq call now looks like this:

(setq #<LEXVAR 1:0> 15)

This means, roughly, "stick the number 15 in the 0th slot in this functions' stack frame". Again, x has nothing to do with this at all.

NOTE: To print out the compiled form of a function, set the global variable *print-funobj* to t:

(setq *print-funobj* t)

and then ask for the description of foo's functional obect:

(describe #'foo)

or (using the Star Sapphire specific #? describe shorthand);

#?#'foo

 

 

This is why if you type x after running foo, it comes up as having no value: all foo does is to store the number 15 on the stack in a particular location. This stack location can be referenced symbolically as x in that function, but this is just a matter of convenience for the programmer.

When you type x in any context for the first time, an entry in the user package is made for it. This entry, the symbol x has a value cell. In Star Sapphire, the symbol is stored in the virtual heap; this is completely distinct from the LISP stack, which is stored in real memory. Assigning 15 to the x which has been translated into a lexical variable, a LISP stack reference, has no effect on the value cell of the symbol x on the virtual heap.

If this is beginning to sound a little like Monty Python's "dead parrot" sketch, we apologise. However, it is important to fully understand what is going on when we say that a variable is lexically scoped before moving on the dynamic scoping.

3.2 Introduction to Dynamic Scoping

In many simple cases, dynamically and lexically scoped variables appear to have exactly the same effect. A variable takes on a value in a restricted context.

The difference is that a lexical variable can be said to exist only in the given lexical context it is written in. The dynamic variable persists outside the given lexical context. It may or may not have a value outside of this lexical context; before being 'bound' to (assigned) a value in the given context any previous values are saved; after exiting the context the value is restored. The value is saved on the stack, so dynamically scoped variable bindings (assignments) can nest.

Here is an example of a dynamically scoped variable.

(defun foo(*x*)

(declare special *x*)

(setq *x* 15)

(print *x*))

The (declare special *x*) clause tells the compiler that *x* is to be treated as a dynamically scoped variable. Dynamically scoped variables are often called 'special'. For clarity we have written x *x*; it is traditional to surround global variables or special symbols with asterisks to distinguish them from lexically scoped variables.

This has exactly the same effect as the previous example.

(foo 3)

15 15

(foo "hello, world!")

15 15

Implementationally, when a dynamically scoped variable is declared, instances of that variable name are not converted by the incremental compiler. If you print out the compiled form of the function foo now you will find that *x* is still embedded in the function code, instead of being converted to a #<LEXVAR> object.

So the setq actually alters the value cell of *x* by storing 15 there. When print gets *x*, it prints the value stored on the value cell of *x*.

So why is *x* still unbound after running foo?

The special declaration doesn't just suppress the compilation of the name *x*. When the function foo is entered, the interpreter knows that it has to store the current value of the symbol *x* on the stack. On exit from foo, what ever was stored in the value cell of *x* prior to running foo is restored. Furthermore, if *x* has no value previous to running foo, it will be made unbound (in Star Sapphire LISP, the value vnil, which indicates an unbound symbol, is saved and restored in this case).

So what?, you may ask. The only difference seems to be that instead of storing the current binding on the LISP stack in physical memory it is stashed in the virtual heap (which, as you might guess, is a lot less efficient in terms of access). The previous binding is what is stored on the LISP stack in this case.

This is a valid concern. LISP dynamically scoped variables are sort of a backwards compatability feature for Common LISP. This was the default scoping mechanism in some previous dialects of LISP. The dynamically scoped version of foo takes a major performance hit, however; the use of dynamically scoped variables is not advised unless there is a good reason to do so.

Historical note:

There are two techniques for performing dynamic scoping, deep and shallow binding. This was a hot issue in the epoch when dynamically scoped LISPs roamed the Earth; the use of lexical scoping has deflated the importance of this debate. If anyone asks, Star Sapphire uses 'shallow' binding. (Moreover, if this is the first question somebody asks you about Star Sapphire LISP you can take this as a valid carbon-14 dating of their knowledge of LISP to the Ford or Carter presidency).

If you are interested in this kind of topic, check out either the first section of Performance and Evaluation of LISP Systems by Richard P. Gabriel (MIT Press, 1986), or Anatomy of LISP by John Allen (McGraw-Hill 1978). Both of these are of use primarily to LISP implementor. The former book is also of use if you benchmark LISP systems for a living (nice work if you can get it); the latter book is definitely B.C. (Before Common LISP).

So what is the difference between lexical and dynamic scoping? The distinction is made clear when we call another function which also alters the global symbol (without declaring it special).

Consider two functions:

(defun foo (*x*)

(declare special *x*)

(bar)

(print *x*))

(defun bar() (setq *x* 15))

The effect of the new foo is the same as before.

On entry to foo the current global value of *x* will be stored on the LISP stack.

By calling the bar function, the current value of *x* will be set to 15.

The variable *x* will have the value 15 when print is called; print will output 15. Whatever value *x* had before foo was called will be restored on exit; if *x* was unbound before foo was called, it will be unbound after foo returns.

Throughout this example, *x* refers to the global symbol *x*. The only thing stored on the stack is the value of *x* in the surrounding enviroment of the call to foo.

Now, if you mechanically translate this to a lexically scoped version of the above two routines, it will present fundamentally different behavior.

Let's take out the special declaration, and, for clarity, change *x* to x:

(defun foo(x)

(bar)

(print x))

(defun bar() (setq x 15))

Note that bar has no arguments. The symbol x in the body of bar refers to the global symbol x. The symbol x in the body of foo refers to the first argument on foo's stack frame.

Hence if you call this version of foo, it will actually print the argument to foo and return it:

(foo "hello, world!")

"hello, world!" "hello, world!"

However, now if you type x at the LISP prompt, the value of x will be printed as 15. The function bar has no influence on the lexical variable named x in the body of foo. However, it has changed the value stashed in the value cell of the symbol x on the heap to 15.

Since x is not declared special in bar, the previous value of the symbol x will not be saved or restored.

3.3 Advanced Scoping and Extent

The above two sections have only touched on two (albeit very common) concrete examples of scoping and extent in Common LISP. This and the next section will present a more comprehensive formal definition of the semantics of Common LISP.

The following terms are useful when discussing scoping and extent.

In Common LISP, a entity which can be referenced is said to be 'established' by the execution of some language construct.

By entity, we mean any Common LISP data object, as well as variable bindings (either lexical or special), catchers, and go targets.

A binding is a particular instance of a parameter; for instance (foo "hello, world!") and (foo 3) bind, in the first case "hello, world!", and in the second case 3 to the parameter x of the function foo.

An entity can have a name. It is crucial to distinguish between the entity and the name of the entity: as can be seen above, the variable binding for x in the lexical version of the foo defun is completely disassociated from the symbol x by the fact of being compiled into a lexvar object.

The following varieties of scope and extent are given distinct terms:

3.3.1. Lexical scope.

Lexical scope means that references to the established entity can occur only within the establishing construct. Normally, the part of the construct in which references to the established entity can occur is called the body of the construct.

3.3.2. Indefinite Scope.

Indefinite scope means that, as in the case of predefined global constants such as pi or the functional value of the symbol car, references can occur anywhere in any Common LISP program (although, of course in practice these values must be established at some point during the LISP boot sequence).

3.3.3 Dynamic Extent

Dynamic extent means that references may occur at any time between the establishment of the entity and the explicit disestablishment of the entity. The entity is said to be disestablished when the execution of the establishing construct terminates in one way or another.

Entities with dynamic extent can stack if their establishing constructs' execution nest.

As described above, a special variables' binding has dynamic extent.

3.3.4 Indefinite extent.

In this case, the entity continues to exist as long as the possibility of reference exists. When the object has no possibility of reference, it becomes fair game for the garbage collector.

Most Common LISP data objects have indefinite extent.

By definition, the bindings of lexically scoped parameters of a function have indefinite extent. Under normal conditions, the effect of successive calls to a function overwrites previous values of the function parameters. An exception to this can occur when the function is saved in a 'closure'. (NOTE: this discussion will only make sense if you understand closures, see function).

For instance,

(defun append-prefix (prefix)

#'(lambda (suffix) (concatenate 'string prefix "-" suffix))))

(setq append-anti (append-prefix "anti"))

(funcall append-anti "disestablishmentarianism")

=> "anti-disestablishmentarianism"

When a functional object is saved in the variable append-anti, the binding for the lexical variable 'prefix' "anti" (which normally would evaporate on the next call to the function append-prefix) effectively gets saved.

Then when append-anti gets used (via funcall), the prefix "anti" gets added to the string "disestablishmentarianism".

In Star Sapphire Common LISP, lexical bindings get stored in the virtual heap, only when a closure is detected through the use of the function special form. Note that the language definition implies that all lexical bindings get saved in this fashion, which of course would impose an heavy burden on performance. Other Common LISP implementations fudge on this point in one way or another, so Star Sapphire is not necessarily abnormal or non-conformant in this respect.

3.3.5 Dynamic Scope

The term dynamic scope is, stricly speaking, inaccurate.

Dynamic scope actually means an entity has indefinite scope and dynamic extent. For this reason, the term special variable is used; a special variable can be referred to anywhere as long as its binding is in effect.

3.3.6 Shadowing

As with other languages with nested constructs, an instance of a name in a more deeply nested construct can override a matching name in a less deeply nested construct, referring to two completely different items.

A simple example:

(defun foo (x) ; x defined as foo's parameter

(let ((x 15)) ; x defined in let

(print x)) ; x defined in let

(print x)) ; x defined as foo's parameter

The let special form sets up a local set of lexical variables, optionally allowing the programmer to initialize their value. In the form above, a local lexical variable named x, shadowing foo's parameter x gets initialized to 15.

If given the argument "hello, world!", foo will print 15, then "hello, world!", and return "hello, world!". This is because the first print statement encountered in the body of foo is nested in the let form; here x refers to the let variable, which has been assigned the value 15. The second print statement is outside the let body and hence refers to x, foo's parameter. The return value from the print statement is the value of x the argument bound to x when foo is called is returned.

In Common LISP, shadowing also applies to dynamic extents, yet another unique aspect of the language. In essence, reference by name to an entity with dynamic extent will always refer to the most recently established (and not yet disestablished) instance of that entity. This is of importance, for instance, when two or more catchers are active. Only the most recently established catcher will be intercepted by a throw statement (see catch and throw).

This should not be taken to imply that dynamically scoped variables shadow: They do not. A reference to a special variable always refers to the same symbol even though it may be declared special in some more deeply nested construct; only the bindings nest.

Of course, using shadowing extensively can complicate your code and make it more difficult to read, maintain and debug. If you use this effect on purpose, it is good practice to clearly comment on the fact.

3.4 Scoping and Extent Rules

This section defines the scoping and extent rules used with various Common LISP constructs and entities. To fully understand some of these rules, you will also need to refer to the entry for the given construct.