CLPython Manual

Last update: 2011.02.12

» Introduction
» Manual

1. Installing CLPython

1.1   Supported Common Lisp implementations
1.2   Library dependencies
1.3   Loading CLPython
1.4   Running the test suite

2. Running Python code

2.1   A note on completeness
2.2   Evaluating a Python expression
2.3   Executing a Python module
2.4   Python runtime environments
2.5   Compiling versus interpreting
2.6   Python interpreter
2.7   Mixed-syntax Lisp/Python mode TODO
2.8   Handling of errors

3. Accessing Lisp from Python

3.1   Lisp functions
3.2   Lisp classes TODO
3.3   Lisp packages TODO

4. Accessing Python from Lisp

4.1   Python values and functions
4.2   Python class instantiating

5. Debugging Python programs

5.1   Tracing function calls TODO
5.2   Profiling functions TODO
5.3   Breakpoints and stepping TODO
5.4   Code coverage reports TODO

6. Python development using the Slime IDE

6.1   Installation TODO
6.2   Inspecting Python objects TODO
6.3   Redefining functions and classes TODO
6.4   Breakpoints and stepping TODO

7. Analyzing Python source code

7.1   Module dependency graph TODO
7.2   Abstract syntax tree analysis with Prolog TODO

8. Extending Python

8.1   Custom import handlers TODO
8.2   Extending Python syntax TODO

9. Compiler implementation details

9.1   Python object representation
9.2   Compilation by macroexpansion

10. Completeness and compatibility

10.1   Compatibility target
10.2   Python-the-language
10.3   Built-in functions TODO
10.4   Built-in classes and methods TODO
10.5   Built-in modules TODO
10.6   Compatibility with CPython C extensions

11. Contributing

11.1   CLPython source on Github
11.2   Announcement mailing list
11.3   Developement mailing list
11.4   Bug tracker
11.5   Task: Implementing builtin modules
11.6   Task: Porting CLPython
11.7   Task: Supporting C Extensions


1.  Installing CLPython

1.1  Supported Common Lisp implementations

CLPython is developed with portability in mind. The latest status of the supported Common Lisp implementations is as follows:
  • Allegro CL  (2011.02.11: test success on Allegro CL 8.2 on linux-x86, both ANSI and Modern mode)
  • Clozure CL  (2011.02.11: test success on Clozure CL 1.5-r13651 on linux-x86)
  • CMU Common Lisp  (2011.02.11: test success on CMUCL-20b-pre2 on linux-x86
  • ECL  (2011.02.11: test success on ECL from current git, on linux-x86)
  • LispWorks  (2011.02.11: test success on LispWorks 6.0 on linux-x86)
  • SBCL  (2011.02.11: test success on sbcl-1.0.45 on linux-x86

Your help is appreciated in making CLPython work on even more implementations, see Task: Porting CLPython.

[back to top]

1.2  Library dependencies

CLPython depends on the following libraries:

  • ASDF which already comes with all the above implementations: used to load CLPython;
  • Closer to MOP by Pascal Costanza: used to access MOP funtionality in a uniform way.

The libraries above are all dependencies to run CLPython in Allegro CL. But in other Lisp implementations the following libraries are also required:

[back to top]

1.3  Loading CLPython

CLPython is loaded with ASDF:
(asdf:operate 'asdf:load-op :clpython)
If this leads to an error that the "clpython component can not be found", it might be possible to continue by loading the system definition file manually:
(load "clpython.asd")
(asdf:operate 'asdf:load-op :clpython)

Upon loading of CLPython a "quick start guide" is printed. It shows a few ways in which Python source code can be evaluated:
CLPython quick start guide:
  Run a string of Python code:           (clpython:run "for i in range(4): print i")
  Run a Python file:                     (clpython:run #p"~/example/")
  Start the Python "interpreter" (REPL): (
  To start mixed Python/Lisp input mode: (clpython:enter-mixed-lisp-python-syntax)
  Run the test suite:                    (asdf:operate 'asdf:test-op :clpython)
[back to top]

1.4  Running the test suite

The test suite can be run to verify that CLPython runs fine in your Lisp implementation. It generates a lot of output followed by the number of test successes and failures. There can be errors, but they should all be marked "known failures".
(asdf:operate 'asdf:test-op :clpython)
... a lot of output ...
End CLPython test
Errors detected in this test: 4 (all known failures)
Successes this test: 1305
[back to top]

2.  Running Python code

2.1  A note on completeness

While many built-in Python classes, functions, and modules are implemented in CLPython, there is also a lot of functionality not available yet. Running valid Python code in CLPython will often fail due to missing functionality. Please see Completeness and compatibility for the details. Of course it would be appreciated if you would contribute missing functionality.

[back to top]

2.2  Evaluating a Python expression

The simplest way to execute a string of Python code is by calling generic function run on it:
(clpython:run "2 + 3")

As for the return value of run: this depends on whether the source code is a statement or an expression. Statements are used for definitions and control flow, like: if, def, class, try/except, import. Statements have expressions inside them. Expressions are forms that evaluate to a value, like: a+b, [a,b], f(1,2,3).

If the source supplied to run is an expression then its value is returned, like 5 in the example above. In case of statements the return value is an undefined implementation detail. Below is an example of running a statement:

(run "for i in range(4): print i")

(Please see Mixed-syntax Lisp/Python mode on how to run Python code directly in the Lisp listener, as alternative to using run.)

[back to top]

2.3  Executing a Python module

Given a file with this contents:

def f():
print 'hello world'

It can be executed using run passing the filename as absolute or relative pathname:

(run #p"")
hello world

To load a "package", which is basically a directory containing an file, load that file:

(run #p"/tmp/zut/")

Relative filenames will be looked up in the directories in *clpython-module-search-paths* and the Python value sys.path.

Filenames should always be passed as pathnames, not strings, because strings are treated directly as source code (see § 2.2) leading to errors like:

(run "")
NameError: Variable `foo' is unbound.

[back to top]

2.4  Python runtime environments

By default every time a Python expression is executed with run a fresh, empty execution environment is used. Referring to a variable bound in a previous call won't work, for example:

(run "a = 3")
(run "print a")
Error: NameError: Variable `a' is unbound.

This can be made to work if the calls share the execution environment, called habitat. If *habitat* is bound, it is used as execution environment, otherwise a new habitat is created and used:

(setf *habitat* (make-habitat))
#<habitat stdin=nil stdout=nil stderr=nil #loaded-modules=0 cmd-line-args=nil search-paths=#(".") @ #x2292eb52>
(run "a = 3")  ;; stores the variable binding in *habitat*
(run "print a")  ;; looks up the variable in *habitat*

Moreover you can use different habitats concurrently, and explicitly pass the one to use to run:

(defvar *other-habitat* (make-habitat))
(run "a = 4" :habitat *other-habitat*)
(run "print a" :habitat *other-habitat*)

And back to the orignal environment:

(run "print a" :habitat *habitat*)
(run "print a")
[back to top]

2.5  Compiling versus interpreting

Function run takes the keyword argument compile which determines whether the source should be compiled before being run. This options is only relevant if the Lisp implementation has both an interpreter and compiler, which is the case in e.g. LispWorks and Allegro CL. In compiler-only implementations like SBCL this option has no effect.

This option is in particular relevant when running benchmarks:

(run #p"/usr/lib/python2.6/test/" <b>:compile nil</b> :args '("" "3000"))
Pystone(1.1) time for 3000 passes = 26.709991
This machine benchmarks at 112.31752 pystones/second
(run #p"/usr/lib/python2.6/test/" <b>:compile t</b> :args '("" "3000"))
Pystone(1.1) time for 3000 passes = 0.11001587
This machine benchmarks at 27268.793 pystones/second
[back to top]

2.6  Python interpreter

A common way to evaluate Python expressions is by typing them into an "interpreter" or "read-eval-print loop" (REPL). CLPython comes with a REPL that acts like the regular Python interpreter:

Welcome to CLPython, an implementation of Python in Common Lisp.
Running on: International Allegro CL Free Express Edition 8.2 [Windows] (May 16, 2010 12:01)
REPL shortcuts: `:q' = quit, `:h' = help.
Using 1 default module search paths set in *clpython-module-search-paths*
>>> 1 + 23 24 >>> print 'hello Lisp world!' hello Lisp world! >>>

For more information, please see the output of typing :h in the repl, Handling of errors, and the documentation for function repl.

This interpreter is useful for evaluating Python code, but not so practical when it comes to mixing Python and Lisp code. Use the more advanced mixed-mode interpreter for that instead .

[back to top]

2.7  Mixed-syntax Lisp/Python mode

CLPython is able to turn a regular Lisp listener (REPL) into a "mixed-mode" listener that supports both Lisp and Python source as input:

clpython(213): (clpython:enter-mixed-lisp-python-syntax)
; The mixed Lisp/Python syntax mode is now enabled;
; Lispy *readtable* is now set.

clpython(214): print 123 * 2
clpython(215): range(100)[98:2:-2]
#(98 96 94 92 90 88 86 84 82 80 ...)
clpython(216): (+ 1 2)

It supports multi-line Python statements as long as the next lines are properly indented:

clpython(70): for i in range(4):
  print i,
  print i*2
0 0 1 2 2 4 3 6

See the documentation for macro enter-mixed-lisp-python-syntax for all "mixed-mode" options, and also see Handling of errors.

This mode can also be enabled in Lisp source files:


You may find this mixed-mode handy for Python development using the Slime IDE.

[back to top]

2.8  Handling of errors

In case an unhandled error happens during evaluation of Python code, the behaviour of CLPython is different from standard Python. You may find yourself suddenly in the Lisp debugger without a clear idea of what is happening or how to continue.

In standard Python, when an unhandled error occurs the error message is printed and the stack is unwound:

Regular Python
>>> 123 + 4/0
Traceback (most recent call last):
File "<stdin>", line 1, in
ZeroDivisionError: integer division or modulo by zero

CLPython implements Python exceptions as Common Lisp errors (see Python object representation). The Common Lisp way to deal with errors is:

  1. The error is reported to the user;
  2. Options are presented on how to proceed: so-called "restarts";
  3. The user chooses an option by "invoking a restart" and execution will continue.

The same error in the CLPython REPL:

>>> 123 + 4/0
ZeroDivisionError: Attempt to divide 4 by zero.

Restart actions (select using :continue):
0: Use another divisor instead of zero
1: Retry evaluating this top-level form in module `__main__'
2: Skip this top-level form in module `__main__'
3: Retry the expression: "123 + 4/0" (:re).
4: Return to Python top level (:pt).
5: Return to Top Level (an "abort" restart).
6: Abort entirely from this (lisp) process.
[1] cl-user(30):

The execution is now paused, not aborted. The most interesting restarts in the list are:

  • 0: Use another divisor instead of zero

    This allows you to specific another divisor value to be used in place of 0. The calculation will continue with the provided value:

    [1] cl-user(30): :continue 0
    New divisor value: 2

    Here the calculation proceeded as if it were: 123 + 4/2 = 125.

  • 3: Retry the expression: "123 + 4/0" (:re).

    For this calculation a retry does not make sense, as 4/0 will always fail. But in other situations a retry might make sense. For example if an attempt is made to read a non-existent file, the file could now be created and then the file operation can be retried.

    [1] cl-user(30): :continue 3
    ZeroDivisionError: Attempt to divide 4 by zero.

    Restart actions (select using :continue):
    0: Use another divisor instead of zero
    1: Retry evaluating this top-level form in module `__main__'
    2: Skip this top-level form in module `__main__'
    3: Retry the expression: "123 + 4/0" (:re).
    4: Return to Python top level (:pt).
    5: Return to Top Level (an "abort" restart).
    6: Abort entirely from this (lisp) process.
    [1] cl-user(31):
  • 4: Return to Python top-level

    This will abort the evaluation of the current input, then unwind the stack down to the top-level of the Python interpreter. This option implements what the standard Python interpreter does:

    [1] cl-user(30): :continue 4
  • 5: Return to Top Level

    This will unwind the stack even further than the previous option does, down to the Lisp listener. The Python interpreter will be aborted along the way:

    [1] cl-user(30): :continue 5

The exact way to invoke a restart is implementation-dependent:

  • Allegro CL:

    The examples above show the interaction in Allegro CL.
    Instead of :continue number also the abbreviation :cont number can be used.
    Two handy restarts in the CLPython REPL have their own shortcut:
    :re will invoke: Retry the expression
    :pt will invoke: Return to Python top level

  • CMUCL:

    A restart is invoked by typing its number, e.g. 3

  • Clozure CL:

    A restart is invoked by typing ("c" followed by space) and its number, e.g. c 3

  • ECL:

    A restart is invoked by typing r and its number, e.g. r3

  • LispWorks:

    A restart is invoked by typing ("c" followed by space) and its number, e.g. c 3

  • SBCL:

    A restart is invoked by typing its number, e.g. 3

[back to top]

3.  Accessing Lisp from Python

3.1  Lisp functions

In the Mixed-syntax Lisp/Python mode you can access Lisp functions in Python code directly:

clpython(26): vectorp( [1,2,3] )
clpython(27): vectorp( (1,2,3) )

If the Lisp name contains dashes, which makes them invalid Python identifiers, replace the dashes by underscores in the Python code:

clpython(28): make_hash_table
#<Function make-hash-table>
clpython(29): make_hash_table()
#<eql hash-table with 0 entries @ #x218e1a82>

An example use of calling print-object on a Python object and the Python standard output:

clpython(30): print_object
#<standard-generic-function print-object>
clpython(31): import sys
#<The clpython.module.sys package>
clpython(32): sys.stdout
#<py-hash/py-==->lisp-val hash-table with 0 entries @ #x21678432>
clpython(33): print_object( {}, sys.stdout)
#<py-hash/py-==->lisp-val hash-table with 0 entries @ #x219305a2>
[back to top]

3.2  Lisp classes


[back to top]

3.3  Lisp packages


[back to top]

4.  Accessing Python from Lisp

4.1  Python values and functions

In the Mixed-syntax Lisp/Python mode you can access Python values (like global variables, functions, and modules) from Lisp directly:

clpython(70): def py(n):
  print "python got: %s" % n
#<python-function py (interpreted) (__main__/py) @ #x218e6b52>
clpython(71): (py (+ 1 2))
python got: 3

For Python expressions that are not a simple identifier there is the ~ (tilde) reader macro. Think of the tilde as a snake! It works for attribute lookup:

clpython(72): import math
#<The clpython.module.math package>
clpython(73): (list cl:pi ~math.pi)
(3.141592653589793d0 3.141592653589793d0)

And it works for object subscription:

clpython(74): x = [1,2,3]
#(1 2 3)
clpython(75): (list ~x[1])
[back to top]

4.2  Python class instantiating

Section § 4.1 showed how to access Python values including class objects. Python classes are instantiated using py-call:

clpython(76): class C:
#<class C @ #x21eca942>
clpython(77): (setq x (py-call ~C))
#<C @ #x21f1d9e2>
clpython(78): x.a = 3
clpython(79): print x.a
[back to top]

5.  Debugging Python programs

5.1  Tracing function calls


[back to top]

5.2  Profiling functions


[back to top]

5.3  Breakpoints and stepping


[back to top]

5.4  Code coverage reports


[back to top]

6.  Python development using the Slime IDE

6.1  Installation


[back to top]

6.2  Inspecting Python objects


[back to top]

6.3  Redefining functions and classes


[back to top]

6.4  Breakpoints and stepping


[back to top]

7.  Analyzing Python source code

7.1  Module dependency graph


[back to top]

7.2  Abstract syntax tree analysis with Prolog


[back to top]

8.  Extending Python

8.1  Custom import handlers


[back to top]

8.2  Extending Python syntax


[back to top]

9.  Compiler implementation details

9.1  Python object representation

Python objects are represented by an equivalent Lisp value where possible, and as CLOS instances otherwise:

Python data typeRepresentation in CLPython
ClassCLOS class
Instance of user-defined classCLOS instance
Generator functionClosure
(Unicode) StringUnicode string
ListAdjustable vector
TupleConsed list
Long, Integer, BooleanInteger

[back to top]

9.2  Compilation by macroexpansion

CLPython first translates Python code into an abstract syntax tree (AST), and then translates the AST into Lisp code. A lot of work is carried out by macros. Example:

if 4 > 3:
  print 'y'
  print 'n')

This is parse into the following abstract syntax tree:

clpython(1): (defvar *source* "
if 4 > 3:
  print 'y'
  print 'n'")
clpython(2): (defvar *ast* (parse *source*))
clpython(3): (with-ast-user-pprinter () (prin1 *ast*) (values))
([module-stmt] ([suite-stmt] (([if-stmt] ((([comparison-expr] [>] ([literal-expr] :number 4) ([literal-expr] :number 3)) ([suite-stmt] (([print-stmt] nil (([literal-expr] :string "y")) nil))))) ([suite-stmt] (([print-stmt] nil (([literal-expr] :string "n")) nil)))))))
Here [if-stmt] is a macro that expands a Python if statement into a Lisp cond form:
(with-ast-user-pprinter ()
  (prin1 (macroexpand-1
        ((([comparison-expr] [>] ([literal-expr] :number 4) ([literal-expr] :number 3))
          ([suite-stmt] (([print-stmt] nil (([literal-expr] :string "y")) nil)))))
        ([suite-stmt] (([print-stmt] nil (([literal-expr] :string "n")) nil))))))
(cond ((clpython::py-val->lisp-bool ([comparison-expr] [>] ([literal-expr] :number 4) ([literal-expr] :number 3))) ([suite-stmt] (([print-stmt] nil (([literal-expr] :string "y")) nil)))) (t ([suite-stmt] (([print-stmt] nil (([literal-expr] :string "n")) nil)))))

The expansion of [if-stmt] thus contains a call to function py-val->lisp-bool. Expansion of the [comparison-expr] form includes a call to py->, and the expansion of [print-stmt] calls py-print.

These functions implement the Python semantics for truth, comparison and printing, which can be hairy at times. They are part of the CLPython runtime. The generated Lisp code is thus not self-contained and independent, but requires CLPython to be loaded every time the code is executed.

[back to top]

10.  Completeness and compatibility

10.1  Compatibility target

When it comes to compatibility, the moving target is the most recent 2.x release of CPython, which is currently Python 2.7.1.

[back to top]

10.2  Python-the-language

Many commonly used language features are implemented, including:

  • modules
  • classes and metaclasses
  • functions, and function decorators
  • generators, list comprehensions, and generator expressions
  • functions locals() and globals()
  • exceptions
  • and of course lambda


  • the bytes type
  • and perhaps other recently introduced types
[back to top]

10.3  Built-in functions

The following built-in functions are implemented:




[back to top]

10.4  Built-in classes and methods


[back to top]

10.5  Built-in modules


[back to top]

10.6  Compatibility with CPython C extensions

There are many extension libraries written in C for the standard Python. One example is NumPy. These modules can not currently be used in CLPython, as CLPython does not implement the Python C API. Interesting is that IronPython, the .Net implementation of Python, has started to support C extensions with a component called IronClad, with apparently most of NumPy already working (implementation details). That they got it working is a nice surprise. But I don't know yet if their approach would work for CLPython.

Other extensions are based on the ctypes module. It seems doable to port this module using CFFI.

[back to top]

11.  Contributing

11.1  CLPython source on Github

The source code of CLPython is hosted in a repository at GitHub. There you can easily clone the project, start hacking, share patches, and report issues.

[back to top]

11.2  Announcement mailing list

To be informed about new releases, please subscribe to clpython-announce.

[back to top]

11.3  Developement mailing list

Please send questions and feature requests to clpython-devel.

[back to top]

11.4  Bug tracker

Please report issues in the issue tracker at GibHub.

[back to top]

11.5  Task: Implementing builtin modules

Probably the first reason some Python code does not run on CLPython, is that some built-in module is partly or completely missing. Extending suppport for built-in features is a task that can always use help.

[back to top]

11.6  Task: Porting CLPython

The intention is to support all Common Lisp implementations over time. Many implementations are already supported. Implementations that are not yet supported are e.g. ABCL and CLISP.

The first requirement on new platforms is that the libraries Closer to MOP and CL-Yacc can be run on them. Porting CLPython can then best started by just loading the system, resolving the roadblocks as they appear. The parts in the code prefixed by reader conditionals (like #+allegro) are obvious cases that need to be extended to support other Lisp implementations. Once CLPython can be loaded successfully, running the test suite without unexpected errors is the next goal.

Please mention porting attempts and progress on clpython-devel.

[back to top]

11.7  Task: Supporting C Extensions

Your help is welcome in investigating whether CLPython can support C extension modules, see Compatibility with CPython C extensions.

[back to top]