;;;(C) By Jeremy Smith February 2006 ;;;level9@decompiler.org ;;;BSD license ;;;Gotchas: ;;;Don't use too much Python or LispEval from Python, or you'll end up with spaghetti code requiring 2 languages to unscramble. ;;;Don't run an infinite Python process (such as a webserver) before saving your work, as you might not be able to Ctrl+C out. You could find another way for the process to terminate. For instance, I found that when you run web.py, it goes into a server loop. Ctrl+C doesn't work, or so it seems, until you hard-reload a webpage served up by web.py, and then it breaks out of Python with a stack trace. ;;;Don't let anyone who has access to Python run malicious code (they could call pol.LispEval even if you secure Python), so observe all the usual security requirements. You could also disable pol.LispEval by specifying 'T' (secure) to init-python. eg (init-python "program name" T) ;;;I will eventually be moving to keyword arguments for the more complex functions (such as ":secure T" to init-python). ;;;I haven't put OS or distribution specific checks in (this is an Alpha release), so set *python-library-path* to the right value below, and make sure CFFI is in the right place ;;;Bugfixes and new features ;;; 2006-02-03 ;;; Fixed the message "NIL is not a Lisp string or pointer" by replacing the NIL's in the module table with (cffi:null-pointer). It was expecting a pointer, not a NIL. This bug should be fixed on Linux now, too, because that's where it was reported as occurring. ;;; I got pythonlisp up and running in Lispworks on Windows. I noticed none of Python's printed text was making it to the Lispworks console, probably because stdout is not the same as in Clisp's console window (which I'd been testing it on). Now dumpstd outputs straight to the REPL window using print. ;;; redirect-stdout is assumed to be set by default to avoid confusing first-time users who can't see anything being output by Python. ;;; Removed that 'import' macro thing - it stopped pythonlisp.lisp loading into Lispworks 4.4 properly. ;;; Now tested with Clisp (2.37/2.38) ACL 7.0 and Lispworks 4.4 on Windows with CFFI 060120 ;;;2006-02-04 ;;; Added the REPL toy, which lets you interact directly with Python, and even define indented blocks a la the official Python REPL ;;;2006-02-05 ;;; Added get-mbox-messages which demonstrates making custom callback tables, and avoiding the use of intermediate variables to pass data between the two languages. Because Python and Lisp both have dynamically-allocated string classes, it is quite efficient to pass enormous strings between them without the usual security holes of using C ;;; Made module arrays static allocations to avoid Python crashing when py is run with a module definition, then called again without it, but calling that module's functions. ;;; Tidied up the pythonlisp function (and the entire code file) ;;; Added 'secure' option to init-python ;;;2006-02-19 ;;; New release! Lets you define Python functions with defpyfun (excuse the pun). I changed all the current module function definitions to use defpyfun, so look at those for examples of how to easily make callbacks. ;;;Known bugs/problems ;;; 2006-02-05 ;;; There's a way for a callback function (defined as METHOD_VARARGS, or 4) to say that the number of arguments passed to it was wrong, and then Python throws an error. I'll look into this soon, as it's not essential to a working system right now, but it is desirable. ;;; There should be a way to just print stdout to the Lisp REPL, but not add it to a huge string. I'm working on it. ;;; *** Start of simple documentation *** ;;; "Why" is a complex question, with a simple answer: I like Lisp's code and Python's extremely simple-to-use and easy-to-install library collection. I've never got a webserver to run in Clisp, but I got web.py (Aaron Swartz's server framework) running in minutes. ;;; Requirements: ;;; Lisp (!) ;;; Python must be installed. Not the 'devel' version, but just the general consumer version. All you need is the python/lib libraries and the Python library binary (on Windows, it's "python22.dll" or "python24.dll" etc, and on Linux it's (I think) libpython2.3.so.1.0 etc). If you want to re-distribute, you can in some cases include the Python DLL even in commercial distributions, perhaps the specific python/lib .py libraries used. It's easiest just to ask people to download Python first, but that's about 6mb. ;;; For those who don't want to read the code (I know I wouldn't!), here's some tips on using it. ;;; First, this code is only tested in Clisp, Allegro and Lispworks on Windows, and Clisp on Linux (but should work with the other two in Linux as well). ;;; To run a bit of Python code and print its results to the Lisp console, type "(py::py [code])" where [code] is a string with the Python code you want to execute. It will also return a string with the text printed out. eg, (py::py "print 'abc'") will print "abc" to the Lisp REPL and return "abc" to the caller. ;;; Use the ' for strings in Python code so you don't have to escape the quote character " ;;; To escape Lisp calls to Python, use \" ;;; When calling back from Python to Lisp, you may need to escape strings (if calling in a string from Lisp) with \\\" (which is \" in escaped Python code). ;;; To run code and not print anything (the most efficient method), "(py::pythonlisp [code] T)" ;;; To run code and define a module of callbacks which Python can run by module name and function name, add it as a 3rd parameter to pythonlisp or define it with add-python-module, and then you need to copy and paste the callbacks and rewrite as appropriate. ;;; get-mbox-messages is a good example of how to pass a module definition to add-python-module, as is the definition in init-python ;;; mbox-add-field is an example of a callback with 2 string args ;;; SetString is 1 arg ;;; GetString is no args but a return value. I haven't done any work on having 'l' args (long integer), so use 'atol' with 's' args. ;;; There are some toys: ;;; To interact with Python, "(py::py-repl)". Be very careful in this mode, in the same way you would be in the Lisp REPL. Don't paste code into the window, don't do anything like play around with file access unless you know what you're doing, and treat it as you would any console. ;;; To get a webpage in a string, "(py::get-web-page [url])" ;;; To get all the messages in a Unix-style mbox mail file, "(py::get-mbox-messages [filename])" - this does not open any files for write-access (only read-binary), so should be okay in all situations, but the file handles may remain open (untested). Works with Netscape mail files, probably Mozilla and Thunderbird too. It is not OS-dependent, so the same code will work in Windows, Linux, etc. ;;; *** End of simple documentation *** ;;; *** Start of pol.lisp code *** ;;;Call a function in this file with (py::[function]) or (pythononlisp::[function]) (defpackage :pythononlisp (:nicknames :py)) (in-package :pythononlisp) ;I really should read more Lisp documentation, in particular, how to load packages just once, and only execute code once per session (defvar *loaded* (progn ;;;;Really completist (load "asdf.lisp") ;;;;;;;Load CFFI first (load "cffi-060120/cffi.asd") ;;;;(load "cffi.asd") (asdf:operate 'asdf:load-op :cffi) T)) ;;;Uncomment one of the following and rename the other as appropriate: ;Windows: ;(defparameter *python-library-path* "python24.dll") ;Linux (could we use wildcards on libpython.*so.*?): (defvar *python-library-path* "libpython2.3.so.1.0") ;Helper macro, takes same arguments as format but returns a string. I'm not sure how efficient it is. (defmacro str-format (&rest body) `(with-output-to-string (out) (format out ,@body))) ;Helper function, turns a numeric string (integer or floating-point) into a number. Used temporarily with 's' args until I get that sorted out. (defun atof (decimal-string) (with-input-from-string (s (concatenate 'string decimal-string)) (read s))) ;;; *** Python wrapper definitions *** ;Note: 'long and 'string seem to be locked in the py:: namespace - I don't know how to fix this yet. (defun generate-parsetuple-format(input) (let ((format-arg "")) (dolist (var input) ;py::string and py::long actually (when (eq (second var) 'string) (setf format-arg (str-format "~as" format-arg))) (when (eq (second var) 'long) (setf format-arg (str-format "~al" format-arg)))) format-arg)) ;[Done]Needs to check if there's any input - if not, don't include the parsetuple call ;Special thanks to Pascal Bourguignon for basically writing this macro! I did add a few things so it's not exactly the same. ;The formatting is absolutely atrocious, but I don't know how to indent loop. (defun gen-pycallback (function-name return-type arguments &optional in-body) "Generates the macro code to take a function name, return type, and list of args, then creates the ParseTuple call that (at runtime) parses the args from the Python args binding, and makes Lisp bindings (it does this later). It creates the BuildValue call which returns the return value, in Lisp, or returns a NIL return value (an empty string). Finally, it wraps it all within with-foreign-pointer calls which bind the local Lisp variables with the given name, to the values created within ParseTuple. It's complex!" (loop :with body = `(progn ,(if arguments `(cffi:foreign-funcall "PyArg_ParseTuple" :pointer args :string ,(generate-parsetuple-format arguments) ,@(loop :for ( ) :in arguments :nconc `(:pointer ,))) (list)) (if ,in-body ,in-body ,(if return-type `(Py_BuildValue ,(generate-parsetuple-format (list(list 'dummy return-type))) (,function-name ,@(loop :for ( ) :in arguments :collect `(pygetarg , (quote ,))))) (progn `(progn (,function-name ,@(loop :for ( ) :in arguments :collect `(pygetarg , (quote ,)))) (Py_BuildValue "" ""))) ))) :for ( ) :in (reverse arguments) :do (setf body `(cffi:with-foreign-pointer (, 255) ,body)) :finally (return body))) ;This generates a 'pyfun' macro, which really just takes the arguments value and shoves the first (the variable names) into the lambda binding list of a standard defun. See defpyfun, below. (defun gen-depyfun(name return-type args body) `(defun ,name ,(let ((arglist)) (dolist (arg args) (push (first arg) arglist)) (nreverse arglist)) (progn ,@body))) ;Helper function for python arguments (auto-called by gen-pycallback) (defun pygetarg(variable type) (case type (long (cffi:mem-ref variable :long)) (string (cffi:foreign-string-to-lisp(cffi:mem-ref variable :string))))) ;Define a callback entry, which is the cffi:def-callback stuff (defmacro defpycallbackentry(function-name return-type input) (gen-pycallback function-name return-type input)) ;Define a callback function, which is the defun stuff, but with stock Lisp arguments (defmacro defpycallbackfunction(function-name return-type input body) (gen-depyfun function-name return-type input body)) ;Define both a callback entry, and a callback function, and translate the args (eg, '((key string) (value string))') into Python-calling code. Definitely a great help. ;A pycallback function has to have a defun with a name matching the name argument and an arg list the same size as the arg list specified. A callback entry has to have variable type definitions, and a reference in memory. (defmacro defpyfun(name return-type args &rest body) `(progn (cffi:defcallback ,name :pointer ((self :pointer) (args :pointer)) (defpycallbackentry ,name ,return-type ,args)) (defpycallbackfunction ,name ,return-type ,args ,body))) ;This is the Python function that initialises a module. The first 2 arguments are all that's required - the last 3 can be NIL. Py_InitModule is a C macro which calls this with NILs. ;methods is an array of PyMethodDefs (cffi:defcfun ("Py_InitModule4" Py_InitModule4) :void (name :string) (methods :pointer) (doc :string) (self :pointer) (apiver :int)) ;This sets the executable filename (I think) (cffi:defcfun ("Py_SetProgramName" Py_SetProgramName) :void (name :string)) ;Name says it all. It initialises the Python working state in memory (cffi:defcfun ("Py_Initialize" Py_Initialize) :void) ;This is basically a kind of 'eval string' for Python, and is where all the work takes plae (cffi:defcfun ("PyRun_SimpleString" PyRun_SimpleString) :void (code :string)) ;This is, amongst other things, to build return values. It *should* take multiple arguments, a la sprintf, in the same manner as PyArg_ParseTuple, where the number and types of arguments is expressed in eg, Py_BuildValue("ss",var1,var2). More work on this is needed. (cffi:defcfun ("Py_BuildValue" Py_BuildValue) :pointer (types :string) (value :string)) ;PyArg_ParseTuple takes n+2 arguments: The argument tuple (passed in to every callback), the types expressed as a c-string with one character per type, and then, for each character, a variable to hold the value (hence the pointer dereferencing with mem-ref to get the value). These values do not need to be freed up, because Python does all that automatically ;This wrapper does not include 'n+2' arguments, just 3, but this wrapper is used in LispEval and SetString because they just use one argument. See below in mbox-add-field for the use of this function with more than 3 arguments. (cffi:defcfun ("PyArg_ParseTuple" PyArg_ParseTuple) :pointer (args :pointer) (types :string) (value :pointer)) ;;; *** End of Python wrapper definitions *** 1;;; *** Start of pol default module functions *** ;This holds data between Lisp and Python. Not thread-safe? Perhaps a better way of passing values is to run LispEval from within Python with the return value, to whatever closure (defparameter *callbackstr* "") ;Holds any text printed out during the session via dumpstd, returned by pythonlisp (defparameter *session-stdout* NIL) (defpyfun LispEval string((code string)) (str-format "~A" (eval(read-from-string code)))) (defpyfun GetString string NIL *callbackstr*) (defpyfun SetString NIL ((text string)) (setf *callbackstr* text)) (defpyfun dumpstd NIL ((text string)) (when *session-stdout* (setf *session-stdout* (str-format "~A~A" *session-stdout* (pygetarg text 'string)))) (format t text)) (defvar *python-initialised* NIL) ;Do all the dirty work before running any code. The argument 'program-name' should specify the path to the Lisp executable, but I'm not sure where in Python the value is used (defun init-python(&optional (program-name "Lisp") (secure NIL)) (cffi:load-foreign-library *python-library-path*) (Py_SetProgramName program-name) (Py_Initialize) (if secure (add-python-module (list "pol" (list "SetString" 1 'SetString) (list "GetString" 4 'GetString) (list "dumpstd" 1 'dumpstd))) (add-python-module (list "pol" (list "SetString" 1 'SetString) (list "GetString" 4 'GetString) (list "LispEval" 1 'LispEval) (list "dumpstd" 1 'dumpstd)))) (setf *python-initialised* T) (pythonlisp "print \"If you can see this, Python is loaded and working\"") ) ;;; *** Start of module-creation functions *** ;This is to define an entry in the Python module array. (cffi:defcstruct PyMethodDef (ml_name :string) (ml_meth :pointer) (ml_flags :int) (ml_doc :pointer) ) ;Create exactly one callback array entry from the data given - rather tied to add-python-module due to its assumption of 'methodptr', needs rewriting (defmacro defpycallstruct(count name flags callback &optional NULL) `(progn (setf (cffi:foreign-slot-value (cffi:mem-aref methodptr 'PyMethodDef ,count) 'PyMethodDef 'ml_name) ,name) (setf (cffi:foreign-slot-value (cffi:mem-aref methodptr 'PyMethodDef ,count) 'PyMethodDef 'ml_flags),flags) (setf (cffi:foreign-slot-value (cffi:mem-aref methodptr 'PyMethodDef ,count) 'PyMethodDef 'ml_meth) (if ,NULL (cffi:null-pointer)(cffi:get-callback ,callback))) (setf (cffi:foreign-slot-value (cffi:mem-aref methodptr 'PyMethodDef ,count) 'PyMethodDef 'ml_doc) (cffi:null-pointer)))) ;This function takes a name for the module, then a list of function names, and generates an array to the length of the methods list+1, creating 4 lines for each entry in the list with an increasing array offset, and adding a NULL entry at the end (hence +1). If there is no NULL entry, Python keeps reading past the end of the module array to goodness knows where and that is not good. ;;;The module is permanent, until Lisp is exited, and the memory used is *not* freed up. The reason for this is because Python can continue to call a module even after the memory has been freed up in the caller - which would cause an unwanted crash. (defun add-python-module(methods) ;length+1 for the NULL item (let ((methodptr (cffi:foreign-alloc 'PyMethodDef :count (+(length (rest methods))1)))) (dotimes (i (length(rest methods))) (defpycallstruct i (first(elt(rest methods)i)) (second(elt(rest methods)i)) (third(elt(rest methods)i)))) (defpycallstruct (length(rest methods))(cffi:null-pointer) 0 (cffi:null-pointer) T) ;;1011 is Python 2.2 which I use - don't panic if it gives a warning about the wrong version (Py_InitModule4 (first methods) methodptr (cffi:null-pointer) (cffi:null-pointer) 1011) (pyrun_simplestring (str-format "import ~a" (first methods))))) ;;; *** End of module-creation functions *** ;;; *** Start of PyRun_SimpleString wrapper *** ;Saves on all that typing! (defmacro py(code) `(pythonlisp ,code)) ;Call a length of Python code, as a string. The second argument is the module you want to create for this bit of Python code (or not) but will stay in memory for any additional sessions. The third argument (default switched on) states that you want to see any Python output in the Lisp REPL and/or returned from pythonlisp as a string. It adds overhead, so turn it off for extensive Python calls where you don't need to see what's going on. (defun pythonlisp(code &optional (redirect-stdout 'CONSOLE) methods) ;;If the user has not initialised Python first, there's no point in doing nothing. (unless *python-initialised* (init-python)) (when redirect-stdout (when (eq redirect-stdout 'CONSOLE) (setf *session-stdout* NIL)) (when (eq redirect-stdout 'STRING) (setf *session-stdout* ""))) ;;Adds them permanently. I think the 'methods' parameter to pythonlisp could be removed completely, and I'd rather ask people to specifically define a module before calling it. (when methods (add-python-module methods)) ;;I don't know if this has to be imported every time, or just once per session. Non-Clisp Lisps don't catch any Python stdout without this. I think it can be improved by removing the 'class Sout' and the imports. ;;At least the 'sys.std*' lines have to be called every time ;;This inline code has to be aligned far-left, so Python can get the right indents (let ((header "class Sout: def write(self, stdstring): pol.dumpstd(stdstring) import pol import sys sys.stdout = Sout() sys.stderr = Sout() #sys.sydin = None ")) (let ((code-to-execute (if methods (str-format "import ~a~%~a" (first methods) code) code))) ;;Redirecting stdout is necessary if you want to get Python's printed output, but creates additional overhead (if redirect-stdout (pyrun_simplestring (str-format "~a~a~%" header code-to-execute)) (pyrun_simplestring code-to-execute))) *session-stdout*)) ;;; *** End of PyRun_SimpleString wrapper *** ;;; *** Start of some simple example 'toys' *** ;Python REPL emulator (just for fun) ;Use ~% for linebreaks (terrible use of str-format for this) ;Kind of 'intelligent' like the Python REPL (prompt=">>>"): after a line ending in ':', it prints new lines starting with the prompt "...". You indent each line appropriately until you have finished the block, and then you insert an empty line to execute the code, all in one go. Seems to work! Even the Python REPL doesn't evaluate until the indented block is at an end (checks for a blank line). ;Doesn't seem to evaluate expressions on their own (eg, just running the Python code "20" will print nothing, but "print 20" will), you have to use print, perhaps this could be added to the execution string (defun py-repl() (format t "Welcome to the Python-on-lisp REPL emulator - enter on a blank line to quit~%>>> ") (loop for line = (read-line) while (not(equal line "")) do ;;Doesn't work if the line ends in a ": " or similiar (if (eq (elt (reverse line) 0) #\:) (let ((indented-block line)) (format t "... ") ;;;Can't be bothered to see if the line is validly indented or not! (loop for line = (read-line) while (not(equal line "")) do (setf indented-block (str-format "~a~%~a" indented-block line)) (format t "... ")) (py (str-format "~a~%" indented-block))) (py (str-format line))) (format t ">>> "))) ;Returns the web page text (though it is more efficient to use SetString) or a list of errors if something went wrong (defun get-web-page(url) (pythonlisp (str-format "from httplib import HTTP import urllib import string f = urllib.urlopen(\"~a\") print f.read() #More efficient than print f.read() #pol.SetString(f.read() " url)'STRING)) ;MBOX utility (defparameter *mbox-message* NIL) (defparameter *mbox-messages* NIL) (defparameter *mbox-ids* (make-hash-table :test #'equal)) ; Gets an mbox file into a list of (sort of alists) lists. Demonstrates callbacks and creating a custom callback module, and the ease of Python to do certain things. ; Just a note of interest, but mbox_add_field could take a numeric value to specify which map the value is to go into, the value being incremented at the start of get-mbox-messages and this could be threadsafe, in that you could call get-mbox-messages from several threads. It wouldn't be totally threadsafe, but it's a start. ; Even better might just be to create a lambda expression with its own message list and field hashmap, and pass the closure as a new CFFI callback. That might be possible using gensym. (defun get-mbox-messages(filename &optional fresh) (setf *mbox-message* (list)) (when fresh (setf *mbox-messages* (list))) (pythonlisp (str-format "import mailbox,rfc822,sys,os,string,re mb = mailbox.UnixMailbox(file(\"~a\",'rb')) msg = mb.next() while msg is not None: for i in msg.keys(): polmbox.mbox_add_field(i,msg.getheader(i)) polmbox.mbox_add_field('Contents',msg.fp.read()) polmbox.mbox_add_email() msg = mb.next()" filename) ;Here is the argument to pythonlisp: a custom callback table, with a module name (which Python sees), a table of: ;;strings (the name Python will see) ;;numbers (function return type, which I should define an ENUM for, use '1' for now) and ;;callbacks (as a symbol to the CFFI callback lookup function) ;No stdout, as we're using solely callbacks for data NIL ;Create this module, polmbox (list "polmbox" (list "mbox_add_header" 1 'mbox-add-header) (list "mbox_add_field" 1 'mbox-add-field) (list "mbox_add_email" 4 'mbox-add-email))) ;After execution, return this variable which will now contain all the mbox messages *mbox-messages*) ;These 2 functions are totally experimental, and for my use only ;(defun write-mbox-message(message stream) ; ;This is required to separate messages in an MBOX file - but Python does not handle it ; (format stream "From - Sat Apr 01 20:38:32 1994~%") ; (format stream "From: ~A~%" (second(assoc "from" message :test #'equal))) ; (dolist (header message) ; (when (not (equal (first header) "Contents")) ; (format stream "~a: ~a~%" (first header) (second header)))) ; (format stream "~A~%" (second(assoc "Contents" message :test #'equal)))) ;(defun write-mbox-messages(messages filename) ; (with-open-file (file filename :direction :output :if-exists :supersede :if-does-not-exist :create) ; (dolist (message messages) ; (write-mbox-message message file)))) (defpyfun mbox-add-field NIL ((key string) (value string)) ;Detects messages with duplicate IDs ; (when (equal key "message-id") ; (when (gethash value *mbox-ids*) ; (break)) ; (setf (gethash value *mbox-ids*) T)) (push (list key value)*mbox-message*)) (defpyfun mbox-add-header NIL ((value string)) (push value *mbox-message*)) ;Here, the return type and args are both NIL, this works fine. (defpyfun mbox-add-email NIL NIL (push *mbox-message* *mbox-messages*) (setf *mbox-message* (list))) ;An example of what you can do with this data (defun find-mbox-author(messages author) (let ((retval)) (dolist (msg messages) (when (equal author (second(assoc "from" msg :test #'equal))) (push msg retval))) retval)) ;Warning: Don't call any defun function the same name as a defpyfun. That's because it's the same name, so the latter will overwrite the former. Just treat a pyfun like any other. (defun testmynum() (pythonlisp "import test test.testnum(50) " NIL (list "test" (list "testnum" 1 'testnum)))) (defpyfun testnum NIL ((number long)) (print number)) ;;; *** End of some simple example 'toys' *** ;;; *** End of pol.lisp code ***