Copyright © 2005 James Bielman <jamesjb at jamesjb.com>
Copyright © 2005, 2006 Luís Oliveira
<loliveira at common-lisp.net>
Copyright © 2006 Stephen Compall <s11 at member.fsf.org>
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
The software is provided “as is”, without warranty of any kind, express or implied, including but not limited to the warranties of merchantability, fitness for a particular purpose and noninfringement. In no event shall the authors or copyright holders be liable for any claim, damages or other liability, whether in an action of contract, tort or otherwise, arising from, out of or in connection with the software or the use or other dealings in the software.
--- Dictionary ---
Foreign Types
Pointers
Strings
Variables
Functions
Libraries
Callbacks
Finalizers
CFFI is the Common Foreign Function Interface for ANSI Common Lisp systems. By foreign function we mean a function written in another programming language and having different data and calling conventions than Common Lisp, namely, C. CFFI allows you to call foreign functions and access foreign variables, all without leaving the Lisp image.
We consider this manual ever a work in progress. If you have difficulty with anything CFFI-specific presented in the manual, please contact the developers with details.
See What makes Lisp different, for an argument in favor of FFI in general.
CFFI's primary role in any image is to mediate between Lisp developers and the widely varying FFIs present in the various Lisp implementations it supports. With CFFI, you can define foreign function interfaces while still maintaining portability between implementations. It is not the first Common Lisp package with this objective; however, it is meant to be a more malleable framework than similar packages.
CFFI supports various free and commercial Lisp implementations: Allegro CL, Corman CL, clisp, CMUCL, ECL, LispWorks, OpenMCL, SBCL and the Scieneer CL.
There are also plans to support Digitool MCL, and GCL.
Tested platforms: linux/x86, linux/ppc, win32/x86, darwin/ppc.
Version 7.0 is supported. The 8.0 beta is also known to work. Earlier versions are untested and unsupported but patches to support them are welcome.
:long-long type.
Tested platforms: win32/x86.
Versions prior to 2.51 are untested and unsupported. Also, you will
need to avoid Corman's buggy COMPILE-FILE and fasl
loader. Please follow these instructions by Edi Weitz to setup ASDF for Corman CL in a way that
works around these issues.
foreign-funcall.
Tested platforms: linux/x86, linux/ppc, win32/x86, darwin/ppc.
Version is 2.34 or newer is required on win32/x86. For other platforms version 2.35 or newer is required.
Tested platforms: linux/x86, darwin/ppc.
Versions prior to 19B are untested. For darwin/ppc, the 2006-02 (19C) snapshot or later is recommended.
Tested platforms: needs testing...
As of November 2005, the CVS version of ECL is required. It is reported to pass all tests.
:long-long type.
:dffi is not present in *features*),
cffi:load-foreign-library does not work and you must use ECL's
own ffi:load-foreign-library with a constant string argument.
Tested platforms: linux/x86, win32/x86, darwin/ppc.
Versions prior to 4.4 are untested.
:long-long type.
Tested platforms: darwin/ppc, linux/ppc.
OpenMCL 1.0 or newer is recommended.
Tested platforms: linux/x86, linux/ppc, darwin/ppc.
Version 0.9.6 or newer is recommended.
Tested platforms: linux/x86, linux/amd64.
Version 1.2.10 or newer is recommended. Passes all tests. The x86 and AMD64 ports feature long-double support.
Users of many popular languages bearing semantic similarity to Lisp, such as Perl and Python, are accustomed to having access to popular C libraries, such as GTK, by way of “bindings”. In Lisp, we do something similar, but take a fundamentally different approach. This tutorial first explains this difference, then explains how you can use CFFI, a powerful system for calling out to C and C++ and access C data from many Common Lisp implementations.
The concept can be generalized to other languages; at the time of writing, only CFFI's C support is fairly complete, but C++ support is being worked on. Therefore, we will interchangeably refer to foreign functions and foreign data, and “C functions” and “C data”. At no time will the word “foreign” carry its usual, non-programming meaning.
This tutorial expects you to have a working understanding of both Common Lisp and C, including the Common Lisp macro system.
The following sums up how bindings to foreign libraries are usually implemented in other languages, then in Common Lisp:
PyObject
in Python, and whatever C object is called for, often using C
functions provided by the implementation. It also translates between
the calling conventions of the language and C.
We believe the advantages of the Common Lisp approach far outweigh any disadvantages. Incremental development with a listener can be as productive for C binding development as it is with other Lisp development. Keeping it “in the [Lisp] family”, as it were, makes it much easier for you and other Lisp programmers to load and use the bindings. Common Lisp implementations such as CMUCL, freed from having to provide a C interface to their own objects, are thus freed to be implemented in another language (as CMUCL is) while still allowing programmers to call foreign functions.
Perhaps the greatest advantage is that using an FFI doesn't obligate you to become a professional binding developer. Writers of bindings for other languages usually end up maintaining or failing to maintain complete bindings to the foreign library. Using an FFI, however, means if you only need one or two functions, you can write bindings for only those functions, and be assured that you can just as easily add to the bindings if need be.
The removal of the C compiler, or C interpretation of any kind,
creates the main disadvantage: some of C's “abstractions” are not
available, violating information encapsulation. For example,
structs that must be passed on the stack, or used as return
values, without corresponding functional abstractions to create and
manage the structs, must be declared explicitly in Lisp. This
is fine for structs whose contents are “public”, but is not so
pleasant when a struct is supposed to be “opaque” by convention,
even though it is not so defined.1
Without an abstraction to create the struct, Lisp needs to be able to lay out the struct in memory, so must know its internal details.
In these cases, you can create a minimal C library to provide the
missing abstractions, without destroying all the advantages of the
Common Lisp approach discussed above. In the case of structs,
you can write simple, pure C functions that tell you how many bytes a
struct requires or allocate new structs, read and write fields of the
struct, or whatever operations are supposed to be
public.2
Another disadvantage appears when you would rather use the foreign language than Lisp. However, someone who prefers C to Lisp is not a likely candidate for developing a Lisp interface to a C library.
The widely available libcurl is a library for downloading files
over protocols like HTTP. We will use libcurl with
CFFI to download a web page.
Please note that there are many other ways to download files from the
web, not least the cl-curl project to provide bindings to
libcurl via a similar FFI.3
libcurl-tutorial(3)
is a tutorial for libcurl programming in C. We will follow
that to develop a binding to download a file. We will also use
curl.h, easy.h, and the man pages for the
libcurl function, all available in the `curl-dev' package
or equivalent for your system, or in the cURL source code
package. If you have the development package, the headers should be
installed in /usr/include/curl/, and the man pages
may be accessed through your favorite man facility.
First of all, we will create a package to work in. You can save these
forms in a file, or just send them to the listener as they are. If
creating bindings for an ASDF package of yours, you will
want to add :cffi to the :depends-on list in your
.asd file. Otherwise, just use the asdf:oos function to
load CFFI.
(asdf:oos 'asdf:load-op :cffi) ;;; Nothing special about the "CFFI-USER" package. We're just ;;; using it as a substitute for your own CL package. (defpackage :cffi-user (:use :common-lisp :cffi)) (in-package :cffi-user) (define-foreign-library libcurl (:unix (:or "libcurl.so.3" "libcurl.so")) (t (:default "libcurl"))) (use-foreign-library libcurl)
Using define-foreign-library and use-foreign-library, we
have loaded libcurl into Lisp, much as the linker does when you
start a C program, or common-lisp:load does with a Lisp source
file or FASL file. We special-cased for unix machines
to always load a particular version, the one this tutorial was tested
with; for those who don't care, the define-foreign-library
clause (t (:default "libcurl")) should be satisfactory, and
will adapt to various operating systems.
libcurlAfter the introductory matter, the tutorial goes on to present the first function you should use.
CURLcode curl_global_init(long flags);
Let's pick this apart into appropriate Lisp code:
;;; A CURLcode is the universal error code. curl/curl.h says ;;; no return code will ever be removed, and new ones will be ;;; added to the end. (defctype curl-code :int) ;;; Initialize libcurl with FLAGS. (defcfun "curl_global_init" curl-code (flags :long))
Implementor's note: CFFI currently assumes the UNIX viewpoint that there is one C symbol namespace, containing all symbols in all loaded objects. This is not so on Windows and Darwin. The interface may be changed to deal with this.
Note the parallels with the original C declaration. We've defined
curl-code as a wrapping type for :int; right now, it
only marks it as special, but later we will do something more
interesting with it. The point is that we don't have to do it yet.
Looking at curl.h, CURL_GLOBAL_NOTHING, a possible value
for flags above, is defined as `0'. So we can now call
the function:
cffi-user> (curl-global-init 0)
=> 0
Looking at curl.h again, 0 means CURLE_OK, so it
looks like the call succeeded. Note that CFFI converted the
function name to a Lisp-friendly name. You can specify your own name
if you want; use ("curl_global_init" your-name-here) as
the name argument to defcfun.
The tutorial goes on to have us allocate a handle. For good measure, we should also include the deallocator. Let's look at these functions:
CURL *curl_easy_init( ); void curl_easy_cleanup(CURL *handle);
Advanced users may want to define special pointer types; we will explore this possibility later. For now, just treat every pointer as the same:
(defcfun "curl_easy_init" :pointer) (defcfun "curl_easy_cleanup" :void (easy-handle :pointer))
Now we can continue with the tutorial:
cffi-user> (defparameter *easy-handle* (curl-easy-init)) => *EASY-HANDLE* cffi-user> *easy-handle* => #<FOREIGN-ADDRESS #x09844EE0>
Note the print representation of a pointer. It changes depending on what Lisp you are using, but that doesn't make any difference to CFFI.
The libcurl tutorial says we'll want to set many options before
performing any download actions. This is done through
curl_easy_setopt:
CURLcode curl_easy_setopt(CURL *curl, CURLoption option, ...);
We've introduced a new twist: variable arguments. There is no obvious
translation to the defcfun form, particularly as there are four
possible argument types. Because of the way C works, we could define
four wrappers around curl_easy_setopt, one for each type; in
this case, however, we'll use the general-purpose macro
foreign-funcall to call this function.
To make things easier on ourselves, we'll create an enumeration of the
kinds of options we want to set. The enum CURLoption isn't the
most straightforward, but reading the CINIT C macro definition
should be enlightening.
(defmacro define-curl-options (name type-offsets &rest enum-args) "As with CFFI:DEFCENUM, except each of ENUM-ARGS is as follows: (NAME TYPE NUMBER) Where the arguments are as they are with the CINIT macro defined in curl.h, except NAME is a keyword. TYPE-OFFSETS is a plist of TYPEs to their integer offsets, as defined by the CURLOPTTYPE_LONG et al constants in curl.h." (flet ((enumerated-value (type offset) (+ (getf type-offsets type) offset))) `(progn (defcenum ,name ,@(loop for (name type number) in enum-args collect (list name (enumerated-value type number)))) ',name))) ;for REPL users' sanity (define-curl-options curl-option (long 0 objectpoint 10000 functionpoint 20000 off-t 30000) (:noprogress long 43) (:nosignal long 99) (:errorbuffer objectpoint 10) (:url objectpoint 2))
With some well-placed Emacs query-replace-regexps, you could
probably similarly define the entire CURLoption enumeration. I
have selected to transcribe a few that we will use in this tutorial.
If you're having trouble following the macrology, just macroexpand the
curl-option definition, or see the following macroexpansion,
conveniently downcased and reformatted:
(progn
(defcenum curl-option
(:noprogress 43)
(:nosignal 99)
(:errorbuffer 10010)
(:url 10002))
'curl-option)
That seems more than reasonable. You may notice that we only use the type to compute the real enumeration offset; we will also need the type information later.
First, however, let's make sure a simple call to the foreign function works:
cffi-user> (foreign-funcall "curl_easy_setopt"
:pointer *easy-handle*
curl-option :nosignal :long 1 curl-code)
=> 0
foreign-funcall, despite its surface simplicity, can be used to
call any C function. Its first argument is a string, naming the
function to be called. Next, for each argument, we pass the name of
the C type, which is the same as in defcfun, followed by a Lisp
object representing the data to be passed as the argument. The final
argument is the return type, for which we use the curl-code
type defined earlier.
defcfun just puts a convenient façade on
foreign-funcall.4 Our earlier call to
curl-global-init could have been written as follows:
cffi-user> (foreign-funcall "curl_global_init" :long 0
curl-code)
=> 0
Before we continue, we will take a look at what CFFI can and can't do, and why this is so.
In What makes Lisp different, we mentioned
that writing an FFI sometimes requires depending on
information not provided as part of the interface. The easy option
CURLOPT_WRITEDATA, which we will not provide as part of the
Lisp interface, illustrates this issue.
Strictly speaking, the curl-option enumeration is not
necessary; we could have used :int 99 instead of
curl-option :nosignal in our call to curl_easy_setopt
above. We defined it anyway, in part to hide the fact that we are
breaking the abstraction that the C enum provides. If the
cURL developers decide to change those numbers later, we
must change the Lisp enumeration, because enumeration values are not
provided in the compiled C library, libcurl.so.3.
CFFI works because the most useful things in C libraries —
non-static functions and non-static variables — are included
accessibly in libcurl.so.3. A C compiler that violated this
would be considered a worthless compiler.
The other thing define-curl-options does is give the “type”
of the third argument passed to curl_easy_setopt. Using this
information, we can tell that the :nosignal option should
accept a long integer argument. We can implicitly assume t
== 1 and nil == 0, as it is in C, which takes care
of the fact that CURLOPT_NOSIGNAL is really asking for a
boolean.
The “type” of CURLOPT_WRITEDATA is objectpoint.
However, it is really looking for a FILE*.
CURLOPT_ERRORBUFFER is looking for a char*, so there is
no obvious CFFI type but :pointer.
The first thing to note is that nowhere in the C interface includes
this information; it can only be found in the manual. We could
disjoin these clearly different types ourselves, by splitting
objectpoint into filepoint and charpoint, but we
are still breaking the abstraction, because we have to augment the
entire enumeration form with this additional
information.5
The second is that the CURLOPT_WRITEDATA argument is completely
incompatible with the desired Lisp data, a
stream.6 It is probably acceptable if we are controlling every file
we might want to use as this argument, in which case we can just call
the foreign function fopen. Regardless, though, we can't write
to arbitrary streams, which is exactly what we want to do for this
application.
Finally, note that the curl_easy_setopt interface itself is a
hack, intended to work around some of the drawbacks of C. The
definition of Curl_setopt, while long, is far less cluttered
than the equivalent disjoint-function set would be; in addition,
setting a new option in an old libcurl can generate a run-time
error rather than breaking the compile. Lisp can just as concisely
generate functions as compare values, and the “undefined function”
error is just as useful as any explicit error we could define here
might be.
We could use foreign-funcall directly every time we wanted to
call curl_easy_setopt. However, we can encapsulate some of the
necessary information with the following.
;;; We will use this typedef later in a more creative way. For ;;; now, just consider it a marker that this isn't just any ;;; pointer. (defctype easy-handle :pointer) (defmacro curl-easy-setopt (easy-handle enumerated-name value-type new-value) "Call `curl_easy_setopt' on EASY-HANDLE, using ENUMERATED-NAME as the OPTION. VALUE-TYPE is the CFFI foreign type of the third argument, and NEW-VALUE is the Lisp data to be translated to the third argument. VALUE-TYPE is not evaluated." `(foreign-funcall "curl_easy_setopt" easy-handle ,easy-handle curl-option ,enumerated-name ,value-type ,new-value curl-code))
Now we define a function for each kind of argument that encodes the
correct value-type in the above. This can be done reasonably
in the define-curl-options macroexpansion; after all, that is
where the different options are listed!
We could make cl:defun forms in the expansion that simply call
curl-easy-setopt; however, it is probably easier and clearer to
use defcfun. define-curl-options was becoming unwieldy,
so I defined some helpers in this new definition.
(defun curry-curl-option-setter (function-name option-keyword) "Wrap the function named by FUNCTION-NAME with a version that curries the second argument as OPTION-KEYWORD. This function is intended for use in DEFINE-CURL-OPTION-SETTER." (setf (symbol-function function-name) (let ((c-function (symbol-function function-name))) (lambda (easy-handle new-value) (funcall c-function easy-handle option-keyword new-value))))) (defmacro define-curl-option-setter (name option-type option-value foreign-type) "Define (with DEFCFUN) a function NAME that calls curl_easy_setopt. OPTION-TYPE and OPTION-VALUE are the CFFI foreign type and value to be passed as the second argument to easy_setopt, and FOREIGN-TYPE is the CFFI foreign type to be used for the resultant function's third argument. This macro is intended for use in DEFINE-CURL-OPTIONS." `(progn (defcfun ("curl_easy_setopt" ,name) curl-code (easy-handle easy-handle) (option ,option-type) (new-value ,foreign-type)) (curry-curl-option-setter ',name ',option-value))) (defmacro define-curl-options (type-name type-offsets &rest enum-args) "As with CFFI:DEFCENUM, except each of ENUM-ARGS is as follows: (NAME TYPE NUMBER) Where the arguments are as they are with the CINIT macro defined in curl.h, except NAME is a keyword. TYPE-OFFSETS is a plist of TYPEs to their integer offsets, as defined by the CURLOPTTYPE_LONG et al constants in curl.h. Also, define functions for each option named set-`TYPE-NAME'-`OPTION-NAME', where OPTION-NAME is the NAME from the above destructuring." (flet ((enumerated-value (type offset) (+ (getf type-offsets type) offset)) ;;map PROCEDURE, destructuring each of ENUM-ARGS (map-enum-args (procedure) (mapcar (lambda (arg) (apply procedure arg)) enum-args)) ;;build a name like SET-CURL-OPTION-NOSIGNAL (make-setter-name (option-name) (intern (concatenate 'string "SET-" (symbol-name type-name) "-" (symbol-name option-name))))) `(progn (defcenum ,type-name ,@(map-enum-args (lambda (name type number) (list name (enumerated-value type number))))) ,@(map-enum-args (lambda (name type number) (declare (ignore number)) `(define-curl-option-setter ,(make-setter-name name) ,type-name ,name ,(ecase type (long :long) (objectpoint :pointer) (functionpoint :pointer) (off-t :long))))) ',type-name)))
Macroexpanding our define-curl-options form once more, we
see something different:
(progn
(defcenum curl-option
(:noprogress 43)
(:nosignal 99)
(:errorbuffer 10010)
(:url 10002))
(define-curl-option-setter set-curl-option-noprogress
curl-option :noprogress :long)
(define-curl-option-setter set-curl-option-nosignal
curl-option :nosignal :long)
(define-curl-option-setter set-curl-option-errorbuffer
curl-option :errorbuffer :pointer)
(define-curl-option-setter set-curl-option-url
curl-option :url :pointer)
'curl-option)
Macroexpanding one of the new define-curl-option-setter
forms yields the following:
(progn
(defcfun ("curl_easy_setopt" set-curl-option-nosignal) curl-code
(easy-handle easy-handle)
(option curl-option)
(new-value :long))
(curry-curl-option-setter 'set-curl-option-nosignal ':nosignal))
Finally, let's try this out:
cffi-user> (set-curl-option-nosignal *easy-handle* 1)
=> 0
Looks like it works just as well. This interface is now reasonably
high-level to wash out some of the ugliness of the thinnest possible
curl_easy_setopt FFI, without obscuring the remaining
C bookkeeping details we will explore.
According to the documentation for curl_easy_setopt, the type
of the third argument when option is CURLOPT_ERRORBUFFER
is char*. Above, we've defined
set-curl-option-errorbuffer to accept a :pointer as the
new option value. However, there is a CFFI type :string,
which translates Lisp strings to C strings when passed as arguments to
foreign function calls. Why not, then, use :string as the
CFFI type of the third argument? There are two reasons, both
related to the necessity of breaking abstraction described in
Breaking the abstraction.
The first reason also applies to CURLOPT_URL, which we will use
to illustrate the point. Assuming we have changed the type of the
third argument underlying set-curl-option-url to
:string, look at these two equivalent forms.
(set-curl-option-url *easy-handle* "http://www.cliki.net/CFFI") == (with-foreign-string (url "http://www.cliki.net/CFFI") (foreign-funcall "curl_easy_setopt" easy-handle *easy-handle* curl-option :url :pointer url curl-code))
The latter, in fact, is mostly equivalent to what a foreign function
call's macroexpansion actually does. As you can see, the Lisp string
"http://www.cliki.net/CFFI" is copied into a char array and
null-terminated; the pointer to beginning of this array, now a C
string, is passed as a CFFI :pointer to the foreign
function.
Unfortunately, the C abstraction has failed us, and we must break it.
While :string works well for many char* arguments, it
does not for cases like this. As the curl_easy_setopt
documentation explains, “The string must remain present until curl no
longer needs it, as it doesn't copy the string.” The C string
created by with-foreign-string, however, only has dynamic
extent: it is “deallocated” when the body (above containing the
foreign-funcall form) exits.
If we are supposed to keep the C string around, but it goes away, what
happens when some libcurl function tries to access the
URL string? We have reentered the dreaded world of C
“undefined behavior”. In some Lisps, it will probably get a chunk
of the Lisp/C stack. You may segfault. You may get some random piece
of other data from the heap. Maybe, in a world where “dynamic
extent” is defined to be “infinite extent”, everything will turn
out fine. Regardless, results are likely to be almost universally
unpleasant.7
Returning to the current set-curl-option-url interface, here is
what we must do:
(let (easy-handle)
(unwind-protect
(with-foreign-string (url "http://www.cliki.net/CFFI")
(setf easy-handle (curl-easy-init))
(set-curl-option-url easy-handle url)
#|do more with the easy-handle, like actually get the URL|#)
(when easy-handle
(curl-easy-cleanup easy-handle))))
That is fine for the single string defined here, but for every string
option we want to pass, we have to surround the body of
with-foreign-string with another with-foreign-string
wrapper, or else do some extremely error-prone pointer manipulation
and size calculation in advance. We could alleviate some of the pain
with a recursively expanding macro, but this would not remove the need
to modify the block every time we want to add an option, anathema as
it is to a modular interface.
Before modifying the code to account for this case, consider the other
reason we can't simply use :string as the foreign type. In C,
a char * is a char *, not necessarily a string. The
option CURLOPT_ERRORBUFFER accepts a char *, but does
not expect anything about the data there. However, it does expect
that some libcurl function we call later can write a C string
of up to 255 characters there. We, the callers of the function, are
expected to read the C string at a later time, exactly the opposite of
what :string implies.
With the semantics for an input string in mind — namely, that the
string should be kept around until we curl_easy_cleanup the
easy handle — we are ready to extend the Lisp interface:
(defvar *easy-handle-cstrings* (make-hash-table) "Hashtable of easy handles to lists of C strings that may be safely freed after the handle is freed.") (defun make-easy-handle () "Answer a new CURL easy interface handle, to which the lifetime of C strings may be tied. See `add-curl-handle-cstring'." (let ((easy-handle (curl-easy-init))) (setf (gethash easy-handle *easy-handle-cstrings*) '()) easy-handle)) (defun free-easy-handle (handle) "Free CURL easy interface HANDLE and any C strings created to be its options." (curl-easy-cleanup handle) (mapc #'foreign-string-free (gethash handle *easy-handle-cstrings*)) (remhash handle *easy-handle-cstrings*)) (defun add-curl-handle-cstring (handle cstring) "Add CSTRING to be freed when HANDLE is, answering CSTRING." (car (push cstring (gethash handle *easy-handle-cstrings*))))
Here we have redefined the interface to create and free handles, to
associate a list of allocated C strings with each handle while it
exists. The strategy of using different function names to wrap around
simple foreign functions is more common than the solution implemented
earlier with curry-curl-option-setter, which was to modify the
function name's function slot.8
Incidentally, the next step is to redefine
curry-curl-option-setter to allocate C strings for the
appropriate length of time, given a Lisp string as the
new-value argument:
(defun curry-curl-option-setter (function-name option-keyword)
"Wrap the function named by FUNCTION-NAME with a version that
curries the second argument as OPTION-KEYWORD.
This function is intended for use in DEFINE-CURL-OPTION-SETTER."
(setf (symbol-function function-name)
(let ((c-function (symbol-function function-name)))
(lambda (easy-handle new-value)
(funcall c-function easy-handle option-keyword
(if (stringp new-value)
(add-curl-handle-cstring
easy-handle
(foreign-string-alloc new-value))
new-value))))))
A quick analysis of the code shows that you need only reevaluate the
curl-option enumeration definition to take advantage of these
new semantics. Now, for good measure, let's reallocate the handle
with the new functions we just defined, and set its URL:
cffi-user> (curl-easy-cleanup *easy-handle*) => NIL cffi-user> (setf *easy-handle* (make-easy-handle)) => #<FOREIGN-ADDRESS #x09844EE0> cffi-user> (set-curl-option-nosignal *easy-handle* 1) => 0 cffi-user> (set-curl-option-url *easy-handle* "http://www.cliki.net/CFFI") => 0
For fun, let's inspect the Lisp value of the C string that was created
to hold "http://www.cliki.net/CFFI". By virtue of the implementation of
add-curl-handle-cstring, it should be accessible through the
hash table defined:
cffi-user> (foreign-string-to-lisp
(car (gethash *easy-handle* *easy-handle-cstrings*)))
=> "http://www.cliki.net/CFFI"
Looks like that worked, and libcurl now knows what
URL we want to retrieve.
Finally, we turn back to the :errorbuffer option mentioned at
the beginning of this section. Whereas the abstraction added to
support string inputs works fine for cases like CURLOPT_URL, it
hides the detail of keeping the C string; for :errorbuffer,
however, we need that C string.
In a moment, we'll define something slightly cleaner, but for now,
remember that you can always hack around anything. We're modifying
handle creation, so make sure you free the old handle before
redefining free-easy-handle.
(defvar *easy-handle-errorbuffers* (make-hash-table) "Hashtable of easy handles to C strings serving as error writeback buffers.") ;;; An extra byte is very little to pay for peace of mind. (defparameter *curl-error-size* 257 "Minimum char[] size used by cURL to report errors.") (defun make-easy-handle () "Answer a new CURL easy interface handle, to which the lifetime of C strings may be tied. See `add-curl-handle-cstring'." (let ((easy-handle (curl-easy-init))) (setf (gethash easy-handle *easy-handle-cstrings*) '()) (setf (gethash easy-handle *easy-handle-errorbuffers*) (foreign-alloc :char :count *curl-error-size* :initial-element 0)) easy-handle)) (defun free-easy-handle (handle) "Free CURL easy interface HANDLE and any C strings created to be its options." (curl-easy-cleanup handle) (foreign-free (gethash handle *easy-handle-errorbuffers*)) (remhash handle *easy-handle-errorbuffers*) (mapc #'foreign-string-free (gethash handle *easy-handle-cstrings*)) (remhash handle *easy-handle-cstrings*)) (defun get-easy-handle-error (handle) "Answer a string containing HANDLE's current error message." (foreign-string-to-lisp (gethash handle *easy-handle-errorbuffers*)))
Be sure to once again set the options we've set thus far. You may wish to define yet another wrapper function to do this.
If you have been reading
curl_easy_setopt(3), you should have noticed that some options
accept a function pointer. In particular, we need one function
pointer to set as CURLOPT_WRITEFUNCTION, to be called by
libcurl rather than the reverse, in order to receive data as it
is downloaded.
A binding writer without the aid of FFI usually approaches this problem by writing a C function that accepts C data, converts to the language's internal objects, and calls the callback provided by the user, again in a reverse of usual practices.
The CFFI approach to callbacks precisely mirrors its differences
with the non-FFI approach on the “calling C from Lisp”
side, which we have dealt with exclusively up to now. That is, you
define a callback function in Lisp using defcallback, and
CFFI effectively creates a C function to be passed as a function
pointer.
Implementor's note: This is much trickier than calling C functions from Lisp, as it literally involves somehow generating a new C function that is as good as any created by the compiler. Therefore, not all Lisps support them. See Implementation Support, for information about CFFI support issues in this and other areas. You may want to consider changing to a Lisp that supports callbacks in order to continue with this tutorial.
Defining a callback is very similar to defining a callout; the main
difference is that we must provide some Lisp forms to be evaluated as
part of the callback. Here is the signature for the function the
:writefunction option takes:
size_t function(void *ptr, size_t size, size_t nmemb, void *stream);
Implementor's note: size_t is almost always an unsigned int. You can get this and many other types using feature tests for your system by using cffi-grovel.
The above signature trivially translates into a CFFI
defcallback form, as follows.
;;; Alias in case size_t changes. (defctype size :unsigned-int) ;;; To be set as the CURLOPT_WRITEFUNCTION of every easy handle. (defcallback easy-write size ((ptr :pointer) (size size) (nmemb size) (stream :pointer)) (let ((data-size (* size nmemb))) (handler-case ;; We use the dynamically-bound *easy-write-procedure* to ;; call a closure with useful lexical context. (progn (funcall (symbol-value '*easy-write-procedure*) (foreign-string-to-lisp ptr data-size nil)) data-size) ;indicates success ;; The WRITEFUNCTION should return something other than the ;; #bytes available to signal an error. (error () (if (zerop data-size) 1 0)))))
First, note the correlation of the first few forms, used to declare the C function's signature, with the signature in C syntax. We provide a Lisp name for the function, its return type, and a name and type for each argument.
In the body, we call the dynamically-bound
*easy-write-procedure* with a “finished” translation, of
pulling together the raw data and size into a Lisp string, rather than
deal with the data directly. As part of calling
curl_easy_perform later, we'll bind that variable to a closure
with more useful lexical bindings than the top-level
defcallback form.
Finally, we make a halfhearted effort to prevent non-local exits from
unwinding the C stack, covering the most likely case with an
error handler, which is usually triggered
unexpectedly.9 The reason is that most C code is written to
understand its own idiosyncratic error condition, implemented above in
the case of curl_easy_perform, and more “undefined behavior”
can result if we just wipe C stack frames without allowing them to
execute whatever cleanup actions as they like.
Using the CURLoption enumeration in curl.h once more, we
can describe the new option by modifying and reevaluating
define-curl-options.
(define-curl-options curl-option
(long 0 objectpoint 10000 functionpoint 20000 off-t 30000)
(:noprogress long 43)
(:nosignal long 99)
(:errorbuffer objectpoint 10)
(:url objectpoint 2)
(:writefunction functionpoint 11)) ;new item here
Finally, we can use the defined callback and the new
set-curl-option-writefunction to finish configuring the easy
handle, using the callback macro to retrieve a CFFI
:pointer, which works like a function pointer in C code.
cffi-user> (set-curl-option-writefunction
*easy-handle* (callback easy-write))
=> 0
With all options finally set and a medium-level interface developed, we can finish the definition and retrieve http://www.cliki.net/CFFI, as is done in the tutorial.
(defcfun "curl_easy_perform" curl-code
(handle easy-handle))
cffi-user> (with-output-to-string (contents)
(let ((*easy-write-procedure*
(lambda (string)
(write-string string contents))))
(declare (special *easy-write-procedure*))
(curl-easy-perform *easy-handle*)))
=> "<!DOCTYPE HTML PUBLIC \"-//W3C//DTD HTML 4.01//EN\"
....
Now fear, comprehensively</P>
"
Of course, that itself is slightly unwieldy, so you may want to define a function around it that simply retrieves a URL. I will leave synthesis of all the relevant REPL forms presented thus far into a single function as an exercise for the reader.
The remaining sections of this tutorial explore some advanced features of CFFI; the definition of new types will receive special attention. Some of these features are essential for particular foreign function calls; some are very helpful when trying to develop a Lispy interface to C.
We've occasionally used the defctype macro in previous sections
as a kind of documentation, much what you'd use typedef for in
C. We also tried one special kind of type definition, the
defcenum type. See defcstruct, for a definition macro that
may come in handy if you need to use C structs as data.
However, all of these are mostly sugar for the powerful underlying
foreign type interface called type translators. You can easily
define new translators for any simple named foreign type. Since we've
defined the new type curl-code to use as the return type for
various libcurl functions, we can use that to directly convert
cURL errors to Lisp errors.
The CURLcode enumeration seems to follow the typical error code
convention of `0' meaning all is well, and each non-zero integer
indicating a different kind of error. We can apply that trivially to
differentiate between normal exits and error exits.
(define-condition curl-code-error (error) (($code :initarg :curl-code :reader curl-error-code)) (:report (lambda (c stream) (format stream "libcurl function returned error ~A" (curl-error-code c)))) (:documentation "Signalled when a libcurl function answers a code other than CURLE_OK.")) (defmethod translate-from-foreign (value (name (eql 'curl-code))) "Raise a CURL-CODE-ERROR if VALUE, a curl-code, is non-zero." (if (zerop value) :curle-ok (error 'curl-code-error :curl-code value)))
The heart of this translator is new method
translate-from-foreign. By specializing the name
parameter on (eql 'type-name), we immediately modify the
behavior of every function that returns a curl-code to pass the
result through this new method.
To see the translator in action, try invoking a function that returns
a curl-code.
cffi-user> (set-curl-option-nosignal *easy-handle* 1)
=> :CURLE-OK
As the result was `0', the new method returned :curle-ok,
just as specified.10 I will leave disjoining the separate
CURLcodes into condition types and improving the :report
function as an exercise for you.
The creation of *easy-handle-cstrings* and
*easy-handle-errorbuffers* as properties of easy-handles
is a kluge. What we really want is a Lisp structure that stores these
properties along with the C pointer. Unfortunately,
easy-handle is currently just a fancy name for the foreign type
:pointer; the actual pointer object varies from Common Lisp
implementation to implementation, needing only to satisfy
pointerp and be returned from make-pointer and friends.
One solution that would allow us to define a new Lisp structure to
represent easy-handles would be to write a wrapper around every
function that currently takes an easy-handle; the wrapper would
extract the pointer and pass it to the foreign function. However, we
can use type translators to more elegantly integrate this
“translation” into the foreign function calling framework, using
translate-to-foreign.
(defclass easy-handle () ((pointer :initform (curl-easy-init) :documentation "Foreign pointer from curl_easy_init") (error-buffer :initform (foreign-alloc :char :count *curl-error-size* :initial-element 0) :documentation "C string describing last error") (c-strings :initform '() :documentation "C strings set as options")) (:documentation "I am a parameterization you may pass to curl-easy-perform to perform a cURL network protocol request.")) (defmethod initialize-instance :after ((self easy-handle) &key) (set-curl-option-errorbuffer self (slot-value self 'error-buffer))) (defun add-curl-handle-cstring (handle cstring) "Add CSTRING to be freed when HANDLE is, answering CSTRING." (car (push cstring (slot-value handle 'c-strings)))) (defun get-easy-handle-error (handle) "Answer a string containing HANDLE's current error message." (foreign-string-to-lisp (slot-value handle 'error-buffer))) (defun free-easy-handle (handle) "Free CURL easy interface HANDLE and any C strings created to be its options." (with-slots (pointer error-buffer c-strings) handle (curl-easy-cleanup pointer) (foreign-free error-buffer) (mapc #'foreign-string-free c-strings))) (defmethod translate-to-foreign (handle (name (eql 'easy-handle))) "Extract the pointer from an easy-HANDLE." (slot-value handle 'pointer))
While we changed some of the Lisp functions defined earlier to use CLOS slots rather than hash tables, the foreign functions work just as well as they did before.
The greatest strength, and the greatest limitation, of the type translator comes from its generalized interface. As stated previously, we could define all foreign function calls in terms of the primitive foreign types provided by CFFI. The type translator interface allows us to cleanly specify the relationship between Lisp and C data, independent of where it appears in a function call. This independence comes at a price; for example, it cannot be used to modify translation semantics based on other arguments to a function call. In these cases, you should rely on other features of Lisp, rather than the powerful, yet domain-specific, type translator interface.
CFFI provides a rich and powerful foundation for communicating with foreign libraries; as we have seen, it is up to you to make that experience a pleasantly Lispy one. This tutorial does not cover all the features of CFFI; please see the rest of the manual for details. In particular, if something seems obviously missing, it is likely that either code or a good reason for lack of code is already present.
Implementor's note: There are some other things in CFFI that might deserve tutorial sections, such as define-foreign-type, free-translated-object, or structs. Let us know which ones you care about.
CFFI's interface is designed for human programmers, being aimed at aesthetic as well as technical sophistication. However, there are a few programs aimed at translating C and C++ header files, or approximations thereof, into CFFI forms constituting a foreign interface to the symbols in those files.
These wrapper generators are known to support output of CFFI forms.
First, this manual does not describe use of these other programs; they have documentation of their own. If you have problems using a generated interface, please look at the output CFFI forms and verify that they are a correct CFFI interface to the library in question; if they are correct, contact CFFI developers with details, keeping in mind that they communicate in terms of those forms rather than any particular wrapper generator. Otherwise, contact the maintainers of the wrapper generator you are using, provided you can reasonably expect more accuracy from the generator.
When is more accuracy an unreasonable expectation? As described in the tutorial (see Breaking the abstraction), the information in C declarations is insufficient to completely describe every interface. In fact, it is quite common to run into an interface that cannot be handled automatically, and generators should be excused from generating a complete interface in these cases.
As further described in the tutorial, the thinnest Lisp interface to a C function is not always the most pleasant one. In many cases, you will want to manually write a Lispier interface to the C functions that interest you.
Wrapper generators should be treated as time-savers, not complete automation of the full foreign interface writing job. Reports of the amount of work done by generators vary from 30% to 90%. The incremental development style enabled by CFFI generally reduces this proportion below that for languages like Python.
Foreign types describe how data is translated back and forth between C and Lisp. CFFI provides various built-in types and allows the user to define new types.
Dictionary
These types correspond to the native C integer types according to the ABI of the Lisp implementation's host system.
For convenience, the above types are provided as shortcuts for
unsigned-char, unsigned-short, unsigned-int,
unsigned-long, long-long and unsigned-long-long,
respectively.
:long-long and :unsigned-long-long are not supported on
all implementations. When those types are not available, the
symbol cffi-features:no-long-long is pushed into
*features*.
Foreign integer types of specific sizes, corresponding to the C types
defined in stdint.h.
On all systems, the :float and :double types represent a
C float and double, respectively. On most but not all
systems, :float and :double represent a Lisp
single-float and double-float, respectively. It is not
so useful to consider the relationship between Lisp types and C types
as isomorphic, as simply to recognize the relationship, and relative
precision, among each respective category.
This type is only supported on SCL.
A foreign pointer to an object of any type, corresponding to
void *.
No type at all. Only valid as the return type of a function.
CFFI also provides a few useful types that aren't built-in C types.
The :string type performs automatic conversion between Lisp and
C strings. Note that, in the case of functions the converted C string
will have dynamic extent (i.e. it will be automatically freed after
the foreign function returns).
In addition to Lisp strings, this type will also convert
Lisp arrays of element type (unsigned-byte 8) and will pass
foreign pointers unmodified.
A method for free-translated-object is specialized for this type. So, for example, foreign strings allocated by this type and passed to a foreign function will be freed after the function returns.
CFFI> (foreign-funcall "getenv" :string "SHELL" :string) => "/bin/bash" CFFI> (with-foreign-string (str "abcdef") (foreign-funcall "strlen" :string str :int)) => 6 CFFI> (let ((str (make-array 4 :element-type '(unsigned-byte 8) :initial-element 65))) (foreign-funcall "strlen" :string str :int)) => 4
Like :string but returns a list with two values when convert
from C to Lisp: a Lisp string and the C string's foreign pointer.
CFFI> (foreign-funcall "getenv" :string "SHELL" :string+ptr) => ("/bin/bash" #.(SB-SYS:INT-SAP #XBFFFFC6F))
The :boolean type converts between a Lisp boolean and a C
boolean. It canonicalizes to base-type which is :int by
default.
(convert-to-foreign nil :boolean) => 0 (convert-to-foreign t :boolean) => 1 (convert-from-foreign 0 :boolean) => nil (convert-from-foreign 1 :boolean) => t
The :wrapper type stores two symbols passed to the to-c
and from-c arguments. When a value is being translated to or
from C, this type funcalls the respective symbol.
:wrapper types will be typedefs for base-type and will
inherit its translators, if any.
Here's an example of how the :boolean type could be defined in
terms of :wrapper.
(defun bool-c-to-lisp (value) (not (zerop value))) (defun bool-lisp-to-c (value) (if value 1 0)) (defctype my-bool (:wrapper :int :from-c bool-c-to-lisp :to-c bool-lisp-to-c)) (convert-to-foreign nil 'my-bool) => 0 (convert-from-foreign 1 'my-bool) => t
Typedefs are similar to typedefs in C, except they are more
like “type wrappers” than aliases, for reasons that will become
clear in the next section.
Defining a typedef is as simple as giving defctype a new name
and the name of the type to be wrapped. Here is how a simpler version
of the built-in :boolean type could be defined:
;;; Define MY-BOOLEAN as an alias for the built-in type :INT. (defctype my-boolean :int)
With this type definition, one can declare arguments to foreign
functions as having the type my-boolean, and they will be
passed as integers. No conversion is taking place—if nil is
passed as a my-boolean, a type error will be signalled.
Type translators are used to automatically convert Lisp values to or
from foreign values. For example, using type translators, one can
define a boolean type which converts a Lisp generalized boolean
(nil vs. non-nil) to a C boolean (zero vs.
non-zero).
We created the my-boolean type in the previous section. To
tell CFFI how to automatically convert Lisp values to
my-boolean values, specialize the generic function
translate-to-foreign using an eql-specializer with the
name of your type, in this case the symbol my-boolean:
;;; Define a method that converts Lisp booleans to C booleans. (defmethod translate-to-foreign (value (type (eql 'my-boolean))) (if value 1 0))
Now, when an object is passed as a my-boolean to a foreign
function, this method will be invoked to convert the Lisp value to an
integer. To perform the inverse operation, which is needed for
functions that return a