2.1.4 Generators

In all of the above clauses, the driver variable is updated on each iteration. Sometimes it is desirable to have greater control over updating. For instance, consider the problem of associating numbers, in increasing order and with no gaps, with the non-nil elements of a list. One obvious first pass at writing this is:

  (iter (for el in list)
        (for i upfrom 1)
        (if el (collect (cons el i))))

But on the list (a b nil c) this produces ((a . 1) (b . 2) (c . 4)) instead of the desired ((a . 1) (b . 2) (c . 3)). The problem is that i is incremented each time through the loop, even when el is nil.

The problem could be solved elegantly if we could step i only when we wished to. This can be accomplished for any iterate driver by writing generate (or its synonym generating) instead of for. Doing so produces a generator—a driver whose values are yielded explicitly. To obtain the next value of a generator variable v, write (next v). The value of a next form is the next value of v, as determined by its associated driver clause. next also has the side-effect of updating v to that value. If there is no next value, next will terminate the loop, just as with a normal driver.

Using generators, we can now write our example like this:

  (iter (for el in list)
        (generate i upfrom 1)
        (if el (collect (cons el (next i)))))

Now i is updated only when (next i) is executed, and this occurs only when el is non-nil.

To better understand the relationship between ordinary drivers and generators, observe that we can rewrite an ordinary driver using its generator form immediately followed by next, as this example shows:

  (iter (generating i from 1 to 10)
        (next i)
        ...)

Provided that the loop body contains no (next i) forms, this will behave just as if we had written (for i from 1 to 10).

We can still refer to a driver variable v without using next; in this case, its value is that given to it by the last evaluation of (next v). Before (next v) has been called the first time, the value of v is undefined.

This semantics is more flexible than one in which v begins the loop bound to its first value and calls of next supply subsequent values, because it means the loop will not terminate too soon if the generator's sequence is empty. For instance, consider the following code, which tags non-nil elements of a list using a list of tags, and also counts the null elements. (We assume there are at least as many tags as non-nil elements.)

  (let* ((counter 0)
         (tagged-list (iter (for el in list)
                            (generating tag in tag-list)
                            (if (null el)
                                (incf counter)
                                (collect (cons el (next tag)))))))
    ...)

It may be that there are just as many tags as non-null elements of list. If all the elements of list are null, we still want the counting to proceed, even though tag-list is nil. If tag had to be assigned its first value before the loop begins, we would have had to terminate the loop before the first iteration, since when tag-list is nil, tag has no first value. With the existing semantics, however, (next tag) will never execute, so the iteration will cover all the elements of list.

When the “variable” of a driver clause is actually a destructuring template containing several variables, all the variables are eligible for use with next. As before, (next v) evaluates to v's next value; but the effect is to update all of the template's variables. For instance, the following code will return the list (a 2 c).

  (iter (generating (key . item) in '((a . 1) (b . 2) (c . 3)))
        (collect (next key))
        (collect (next item)))

Only driver clauses with variables can be made into generators. This includes all clauses mentioned so far except for repeat. It does not include for... previous, for... =, for... initially... then or for... first... then (see below).