1.  Installing CLPython

1.1  Supported Common Lisp implementations

• Allegro CL  (2011.02.11: test success on Allegro CL 8.2 on linux-x86, both ANSI and Modern mode)
• Clozure CL  (2011.02.11: test success on Clozure CL 1.5-r13651 on linux-x86)
• CMU Common Lisp  (2011.02.11: test success on CMUCL-20b-pre2 on linux-x86
• ECL  (2011.02.11: test success on ECL from current git, on linux-x86)
• LispWorks  (2011.02.11: test success on LispWorks 6.0 on linux-x86)
• SBCL  (2011.02.11: test success on sbcl-1.0.45 on linux-x86

Your help is appreciated in making CLPython work on even more implementations, see Task: Porting CLPython.

1.2  Library dependencies

CLPython depends on the following libraries:

• ASDF which already comes with all the above implementations: used to load CLPython;
• Closer to MOP by Pascal Costanza: used to access MOP funtionality in a uniform way.

The libraries above are all dependencies to run CLPython in Allegro CL. But in other Lisp implementations the following libraries are also required:

• CL-Yacc by Juliusz Chroboczek: used to parse Python source code;
• ptester is necessary for running the test suite.

1.3  Loading CLPython

(asdf:operate 'asdf:load-op :clpython)

(load "clpython.asd")
(asdf:operate 'asdf:load-op :clpython)

CLPython quick start guide:
  Run a string of Python code:           (clpython:run "for i in range(4): print i")
  Run a Python file:                     (clpython:run #p"~/example/foo.py")
  Start the Python "interpreter" (REPL): (clpython.app.repl:repl)
  To start mixed Python/Lisp input mode: (clpython:enter-mixed-lisp-python-syntax)
  Run the test suite:                    (asdf:operate 'asdf:test-op :clpython)

1.4  Running the test suite

(asdf:operate 'asdf:test-op :clpython)
... a lot of output ...
End CLPython test
Errors detected in this test: 4 (all known failures)
Successes this test: 1305

2.  Running Python code

2.1  A note on completeness

While many built-in Python classes, functions, and modules are implemented in CLPython, there is also a lot of functionality not available yet. Running valid Python code in CLPython will often fail due to missing functionality. Please see Completeness and compatibility for the details. Of course it would be appreciated if you would contribute missing functionality.

2.2  Evaluating a Python expression

(clpython:run "2 + 3")
5

As for the return value of run: this depends on whether the source code is a statement or an expression. Statements are used for definitions and control flow, like: if, def, class, try/except, import. Statements have expressions inside them. Expressions are forms that evaluate to a value, like: a+b, [a,b], f(1,2,3).

If the source supplied to run is an expression then its value is returned, like 5 in the example above. In case of statements the return value is an undefined implementation detail. Below is an example of running a statement:

(run "for i in range(4): print i")
0
1
2
3
nil

(Please see Mixed-syntax Lisp/Python mode on how to run Python code directly in the Lisp listener, as alternative to using run.)

2.3  Executing a Python module

Given a file foo.py with this contents:

def f():
  print 'hello world'
f()

It can be executed using run passing the filename as absolute or relative pathname:

(run #p"foo.py")
hello world
None

To load a "package", which is basically a directory containing an __init__.py file, load that file:

(run #p"/tmp/zut/__init__.py")

Relative filenames will be looked up in the directories in *clpython-module-search-paths* and the Python value sys.path.

Filenames should always be passed as pathnames, not strings, because strings are treated directly as source code (see § 2.2) leading to errors like:

(run "foo.py")
NameError: Variable `foo' is unbound.

2.4  Python runtime environments

By default every time a Python expression is executed with run a fresh, empty execution environment is used. Referring to a variable bound in a previous call won't work, for example:

(run "a = 3")

(run "print a")
Error: NameError: Variable `a' is unbound.

This can be made to work if the calls share the execution environment, called habitat. If *habitat* is bound, it is used as execution environment, otherwise a new habitat is created and used:

(setf *habitat* (make-habitat))
#<habitat stdin=nil stdout=nil stderr=nil #loaded-modules=0 cmd-line-args=nil search-paths=#(".") @ #x2292eb52>

(run "a = 3")  ;; stores the variable binding in *habitat*

(run "print a")  ;; looks up the variable in *habitat*
3

Moreover you can use different habitats concurrently, and explicitly pass the one to use to run:

(defvar *other-habitat* (make-habitat))
*other-habitat*

(run "a = 4" :habitat *other-habitat*)

(run "print a" :habitat *other-habitat*)
4

And back to the orignal environment:

(run "print a" :habitat *habitat*)
3

(run "print a")
3

2.5  Compiling versus interpreting

Function run takes the keyword argument compile which determines whether the source should be compiled before being run. This options is only relevant if the Lisp implementation has both an interpreter and compiler, which is the case in e.g. LispWorks and Allegro CL. In compiler-only implementations like SBCL this option has no effect.

This option is in particular relevant when running benchmarks:

(run #p"/usr/lib/python2.6/test/pystone.py" <b>:compile nil</b> :args '("pystone.py" "3000"))
Pystone(1.1) time for 3000 passes = 26.709991
This machine benchmarks at 112.31752 pystones/second
None

(run #p"/usr/lib/python2.6/test/pystone.py" <b>:compile t</b> :args '("pystone.py" "3000"))
Pystone(1.1) time for 3000 passes = 0.11001587
This machine benchmarks at 27268.793 pystones/second
None

2.6  Python interpreter

A common way to evaluate Python expressions is by typing them into an "interpreter" or "read-eval-print loop" (REPL). CLPython comes with a REPL that acts like the regular Python interpreter:

(clpython:repl)
Welcome to CLPython, an implementation of Python in Common Lisp.
Running on: International Allegro CL Free Express Edition 8.2 [Windows] (May 16, 2010 12:01)
REPL shortcuts: `:q' = quit, `:h' = help.
Using 1 default module search paths set in *clpython-module-search-paths*
>>> 1 + 23
24
>>> print 'hello Lisp world!'
hello Lisp world!
>>> 

For more information, please see the output of typing :h in the repl, Handling of errors, and the documentation for function repl.

This interpreter is useful for evaluating Python code, but not so practical when it comes to mixing Python and Lisp code. Use the more advanced mixed-mode interpreter for that instead .

2.7  Mixed-syntax Lisp/Python mode

CLPython is able to turn a regular Lisp listener (REPL) into a "mixed-mode" listener that supports both Lisp and Python source as input:

clpython(213): (clpython:enter-mixed-lisp-python-syntax)
; The mixed Lisp/Python syntax mode is now enabled;
; Lispy *readtable* is now set.
clpython(214): print 123 * 2
246
clpython(215): range(100)[98:2:-2]
#(98 96 94 92 90 88 86 84 82 80 ...)
clpython(216): (+ 1 2)
3

It supports multi-line Python statements as long as the next lines are properly indented:

clpython(70): for i in range(4):
  print i,
  print i*2
0 0
1 2
2 4
3 6

See the documentation for macro enter-mixed-lisp-python-syntax for all "mixed-mode" options, and also see Handling of errors.

This mode can also be enabled in Lisp source files:

TODO

You may find this mixed-mode handy for Python development using the Slime IDE.

2.8  Handling of errors

In case an unhandled error happens during evaluation of Python code, the behaviour of CLPython is different from standard Python. You may find yourself suddenly in the Lisp debugger without a clear idea of what is happening or how to continue.

In standard Python, when an unhandled error occurs the error message is printed and the stack is unwound:

Regular Python
>>> 123 + 4/0
Traceback (most recent call last): 
  File "<stdin>", line 1, in 
ZeroDivisionError: integer division or modulo by zero
>>> 

CLPython implements Python exceptions as Common Lisp errors (see Python object representation). The Common Lisp way to deal with errors is:

1. The error is reported to the user;
2. Options are presented on how to proceed: so-called "restarts";
3. The user chooses an option by "invoking a restart" and execution will continue.

The same error in the CLPython REPL:

CLPython
>>> 123 + 4/0
ZeroDivisionError: Attempt to divide 4 by zero.

Restart actions (select using :continue): 
0: Use another divisor instead of zero
1: Retry evaluating this top-level form in module `__main__'
2: Skip this top-level form in module `__main__'
3: Retry the expression: "123 + 4/0" (:re).
4: Return to Python top level (:pt).
5: Return to Top Level (an "abort" restart).
6: Abort entirely from this (lisp) process.
[1] cl-user(30): 

The execution is now paused, not aborted. The most interesting restarts in the list are:

• 0: Use another divisor instead of zero

  This allows you to specific another divisor value to be used in place of 0. The calculation will continue with the provided value:

  [1] cl-user(30): :continue 0
  New divisor value: 2
  125

  Here the calculation proceeded as if it were: 123 + 4/2 = 125.

• 3: Retry the expression: "123 + 4/0" (:re).

  For this calculation a retry does not make sense, as 4/0 will always fail. But in other situations a retry might make sense. For example if an attempt is made to read a non-existent file, the file could now be created and then the file operation can be retried.

  [1] cl-user(30): :continue 3
  ZeroDivisionError: Attempt to divide 4 by zero.
  
  Restart actions (select using :continue): 
  0: Use another divisor instead of zero
  1: Retry evaluating this top-level form in module `__main__'
  2: Skip this top-level form in module `__main__'
  3: Retry the expression: "123 + 4/0" (:re).
  4: Return to Python top level (:pt).
  5: Return to Top Level (an "abort" restart).
  6: Abort entirely from this (lisp) process.
  [1] cl-user(31): 

• 4: Return to Python top-level

  This will abort the evaluation of the current input, then unwind the stack down to the top-level of the Python interpreter. This option implements what the standard Python interpreter does:

  [1] cl-user(30): :continue 4
  >>> 

• 5: Return to Top Level

  This will unwind the stack even further than the previous option does, down to the Lisp listener. The Python interpreter will be aborted along the way:

  [1] cl-user(30): :continue 5
  cl-user(31):

The exact way to invoke a restart is implementation-dependent:

• Allegro CL:

  The examples above show the interaction in Allegro CL.
  Instead of :continue number also the abbreviation :cont number can be used.
  Two handy restarts in the CLPython REPL have their own shortcut:
  :re will invoke: Retry the expression
:pt will invoke: Return to Python top level

• CMUCL:

  A restart is invoked by typing its number, e.g. 3

• Clozure CL:

  A restart is invoked by typing c  ("c" followed by space) and its number, e.g. c 3

• ECL:

  A restart is invoked by typing r and its number, e.g. r3

• LispWorks:

  A restart is invoked by typing c  ("c" followed by space) and its number, e.g. c 3

• SBCL:

  A restart is invoked by typing its number, e.g. 3

3.  Accessing Lisp from Python

3.1  Lisp functions

In the Mixed-syntax Lisp/Python mode you can access Lisp functions in Python code directly:

clpython(26): vectorp( [1,2,3] )
t

clpython(27): vectorp( (1,2,3) )
nil

If the Lisp name contains dashes, which makes them invalid Python identifiers, replace the dashes by underscores in the Python code:

clpython(28): make_hash_table
#<Function make-hash-table>

clpython(29): make_hash_table()
#<eql hash-table with 0 entries @ #x218e1a82>

An example use of calling print-object on a Python object and the Python standard output:

clpython(30): print_object
#<standard-generic-function print-object>

clpython(31): import sys
#<The clpython.module.sys package>

clpython(32): sys.stdout
#<py-hash/py-==->lisp-val hash-table with 0 entries @ #x21678432>

clpython(33): print_object( {}, sys.stdout)
#<py-hash/py-==->lisp-val hash-table with 0 entries @ #x219305a2>

3.2  Lisp classes

TODO

3.3  Lisp packages

TODO

4.  Accessing Python from Lisp

4.1  Python values and functions

In the Mixed-syntax Lisp/Python mode you can access Python values (like global variables, functions, and modules) from Lisp directly:

clpython(70): def py(n):
  print "python got: %s" % n
#<python-function py (interpreted) (__main__/py) @ #x218e6b52>

clpython(71): (py (+ 1 2))
python got: 3

For Python expressions that are not a simple identifier there is the ~ (tilde) reader macro. Think of the tilde as a snake! It works for attribute lookup:

clpython(72): import math
#<The clpython.module.math package>

clpython(73): (list cl:pi ~math.pi)
(3.141592653589793d0 3.141592653589793d0)

And it works for object subscription:

clpython(74): x = [1,2,3]
#(1 2 3)

clpython(75): (list ~x[1])
(2)

4.2  Python class instantiating

Section § 4.1 showed how to access Python values including class objects. Python classes are instantiated using py-call:

clpython(76): class C:
  pass
#<class C @ #x21eca942>

clpython(77): (setq x (py-call ~C))
#<C @ #x21f1d9e2>

clpython(78): x.a = 3

clpython(79): print x.a
3

5.  Debugging Python programs

5.1  Tracing function calls

TODO

5.2  Profiling functions

TODO

5.3  Breakpoints and stepping

TODO

5.4  Code coverage reports

TODO

6.  Python development using the Slime IDE

6.1  Installation

TODO

6.2  Inspecting Python objects

TODO

6.3  Redefining functions and classes

TODO

6.4  Breakpoints and stepping

TODO

7.  Analyzing Python source code

7.1  Module dependency graph

TODO

7.2  Abstract syntax tree analysis with Prolog

TODO

8.  Extending Python

8.1  Custom import handlers

TODO

8.2  Extending Python syntax

TODO

9.  Compiler implementation details

9.1  Python object representation

Python objects are represented by an equivalent Lisp value where possible, and as CLOS instances otherwise:

9.2  Compilation by macroexpansion

CLPython first translates Python code into an abstract syntax tree (AST), and then translates the AST into Lisp code. A lot of work is carried out by macros. Example:

if 4 > 3:
  print 'y'
else:
  print 'n')

This is parse into the following abstract syntax tree:

clpython(1): (defvar *source* "
if 4 > 3:
  print 'y'
else:
  print 'n'")
*source*

clpython(2): (defvar *ast* (parse *source*))
*ast*

clpython(3): (with-ast-user-pprinter () (prin1 *ast*) (values))
([module-stmt]
  ([suite-stmt]
    (([if-stmt]
       ((([comparison-expr] [>] ([literal-expr] :number 4) ([literal-expr] :number 3))
         ([suite-stmt] (([print-stmt] nil (([literal-expr] :string "y")) nil)))))
       ([suite-stmt] (([print-stmt] nil (([literal-expr] :string "n")) nil)))))))

(with-ast-user-pprinter ()
  (prin1 (macroexpand-1
     '([if-stmt]
        ((([comparison-expr] [>] ([literal-expr] :number 4) ([literal-expr] :number 3))
          ([suite-stmt] (([print-stmt] nil (([literal-expr] :string "y")) nil)))))
        ([suite-stmt] (([print-stmt] nil (([literal-expr] :string "n")) nil))))))
  (values))
(cond ((clpython::py-val->lisp-bool
           ([comparison-expr] [>] ([literal-expr] :number 4) ([literal-expr] :number 3)))
       ([suite-stmt] (([print-stmt] nil (([literal-expr] :string "y")) nil))))
      (t
       ([suite-stmt] (([print-stmt] nil (([literal-expr] :string "n")) nil)))))

The expansion of [if-stmt] thus contains a call to function py-val->lisp-bool. Expansion of the [comparison-expr] form includes a call to py->, and the expansion of [print-stmt] calls py-print.

These functions implement the Python semantics for truth, comparison and printing, which can be hairy at times. They are part of the CLPython runtime. The generated Lisp code is thus not self-contained and independent, but requires CLPython to be loaded every time the code is executed.

10.  Completeness and compatibility

10.1  Compatibility target

When it comes to compatibility, the moving target is the most recent 2.x release of CPython, which is currently Python 2.7.1.

10.2  Python-the-language

Many commonly used language features are implemented, including:

• modules
• classes and metaclasses
• functions, and function decorators
• generators, list comprehensions, and generator expressions
• functions locals() and globals()
• exceptions
• and of course lambda

Missing:

• the bytes type
• and perhaps other recently introduced types

10.3  Built-in functions

The following built-in functions are implemented:

TODO

Missing:

TODO

10.4  Built-in classes and methods

TODO

10.5  Built-in modules

TODO

10.6  Compatibility with CPython C extensions

There are many extension libraries written in C for the standard Python. One example is NumPy. These modules can not currently be used in CLPython, as CLPython does not implement the Python C API. Interesting is that IronPython, the .Net implementation of Python, has started to support C extensions with a component called IronClad, with apparently most of NumPy already working (implementation details). That they got it working is a nice surprise. But I don't know yet if their approach would work for CLPython.

Other extensions are based on the ctypes module. It seems doable to port this module using CFFI.

11.  Contributing

11.1  CLPython source on Github

The source code of CLPython is hosted in a repository at GitHub. There you can easily clone the project, start hacking, share patches, and report issues.

11.2  Announcement mailing list

To be informed about new releases, please subscribe to clpython-announce.

11.3  Developement mailing list

Please send questions and feature requests to clpython-devel.

11.4  Bug tracker

Please report issues in the issue tracker at GibHub.

11.5  Task: Implementing builtin modules

Probably the first reason some Python code does not run on CLPython, is that some built-in module is partly or completely missing. Extending suppport for built-in features is a task that can always use help.

11.6  Task: Porting CLPython

The intention is to support all Common Lisp implementations over time. Many implementations are already supported. Implementations that are not yet supported are e.g. ABCL and CLISP.

The first requirement on new platforms is that the libraries Closer to MOP and CL-Yacc can be run on them. Porting CLPython can then best started by just loading the system, resolving the roadblocks as they appear. The parts in the code prefixed by reader conditionals (like #+allegro) are obvious cases that need to be extended to support other Lisp implementations. Once CLPython can be loaded successfully, running the test suite without unexpected errors is the next goal.

Please mention porting attempts and progress on clpython-devel.

11.7  Task: Supporting C Extensions

Your help is welcome in investigating whether CLPython can support C extension modules, see Compatibility with CPython C extensions.