Next: , Previous: , Up: Top   [Contents][Index]

4 Extensions


Next: , Up: Extensions   [Contents][Index]

4.1 System building


Next: , Up: System building   [Contents][Index]

4.1.1 Compiling with ECL

In this section we will introduce topics on compiling Lisp programs. ECL is especially powerful on combining lisp programs with C programs. You can embed ECL as a lisp engine in C programs, or call C functions via Foreign Function Interface. We explain file types generated by some compilation approaches. A GNU/Linux system and gcc as a development environment are assumed.

You can generate the following files with ECL:

  1. Portable FASL file (.fasc)
  2. Native FASL file (.fas, .fasb)
  3. Object file (.o)
  4. Static library
  5. Shared library
  6. Executable file

Relations among them are depicted below:

figures/file-types

Figure 4.1: Build file types


Next: , Up: Compiling with ECL   [Contents][Index]

4.1.1.1 Portable FASL

ECL provides two compilers (bytecodes compiler, and C/C++ compiler). Portable FASL files are built from source lisp files by the bytecodes compiler. Generally FASC files are portable across architectures and operating systems providing a convenient way of shipping portable modules. Portable FASL files may be concatenated, what leads to bundles. FASC files are faster to compile, but generally slower to run.

;; install bytecodes compiler
(ext:install-bytecodes-compiler)

;; compile hello.lisp file to hello.fasc
(compile-file "hello1.lisp")
(compile-file "hello2.lisp")

;; reinitialize C/C++ compiler back
(ext:install-c-compiler)

;; FASC file may be loaded dynamically from lisp program
(load "hello1.fasc")

;; ... concatenated into a bundle with other FASC
(with-open-file (output "hello.fasc"
                        :direction :output
                        :if-exists :supersede)
  (ext:run-program
   "cat" '("hello1.fasc" "hello2.fasc")  :output output))

;; ... and loaded dynamically from lisp program
(load "hello.fasc")

Next: , Previous: , Up: Compiling with ECL   [Contents][Index]

4.1.1.2 Native FASL

If you want to make a library which is loaded dynamically from a lisp program, you should choose the fasl file format. Under the hood native fasls are just a shared library files.

This means you can load fasl files with dlopen and initialize it by calling a init function from C programs, but this is not an intended usage. The recommended usage is to load fasl files by calling the load lisp function. To work with Native FASL files ECL has to be compiled with --enable-shared configure option (enabled by default).

Creating a fasl file from one lisp file is very easy.

(compile-file "hello.lisp")

To create a fasl file from more lisp files, firstly you have to compile each lisp file into an object file, and then combine them with c:build-fasl.

;; generates hello.o
(compile-file "hello.lisp" :system-p t)
;; generates goodbye.o
(compile-file "goodbye.lisp" :system-p t)

;; generates hello-goodbye.fas
(c:build-fasl "hello-goodbye"
              :lisp-files '("hello.o" "goodbye.o"))

;; fasls may be built from mix of objects and libraries (both shared and
;; static)
(c:build-fasl "mixed-bundle"
              :lisp-files '("hello1.o" "hello2.a" "hello3.so"))

Next: , Previous: , Up: Compiling with ECL   [Contents][Index]

4.1.1.3 Object file

Object files work as an intermediate file format. If you want to compile more than two lisp files, you might better to compile with a :system-p t option, which generates object files (instead of a fasl).

On linux systems, ECL invokes gcc -c to generate object files.

An object file consists of some functions in C:

Consider the example below.

(defun say-hello ()
  (print "Hello, world"))

During compilation, this simple lisp program is translated into the C program, and then compiled into the object file. The C program contains two functions:

In order to use these object files from your C program, you have to call initialization functions before using lisp functions (such as say-hello). However the name of an init function is seemed to be randomized and not user-friendly. This is because object files are not intended to be used directly.

ECL provides other user-friendly ways to generate compiled lisp programs (as static/shared libraries or executables), and in each approach, object files act as intermediate files.


Next: , Previous: , Up: Compiling with ECL   [Contents][Index]

4.1.1.4 Static library

ECL can compile lisp programs to static libraries, which can be linked with C programs. A static library is created by c:build-static-library with some compiled object files.

;; generates hello.o
(compile-file "hello.lsp" :system-p t)
;; generates goodbye.o
(compile-file "goodbye.lsp" :system-p t)

;; generates libhello-goodbye.a
(c:build-static-library "hello-goodbye"
                        :lisp-files '("hello.o" "goodbye.o")
                        :init-name "init_hello_goodbye")

When you use a static/shared library, you have to call its init function. The name of this function is specified by the :init-name option. In this example, it is then init_hello_goodbye. The usage of this function is shown below:

#include <ecl/ecl.h>
extern void init_hello_goodbye(cl_object cblock);

int
main(int argc, char **argv)
{
    /* setup the lisp runtime */
    cl_boot(argc, argv);

    /* call the init function via read_VV */
    read_VV(OBJNULL, init_hello_goodbye);

    /* ... */

    /* shutdown the lisp runtime */
    cl_shutdown();

    return 0;
}

Because the program itself does not know the type of the init function, a prototype declaration is inserted. After booting up the lisp environment, it invokes init_hello_goodbye via read_VV. init_hello_goodbye takes an argument, and read_VV supplies an appropriate one. Now that the initialization is finished, we can use functions and other stuffs defined in the library.


Next: , Previous: , Up: Compiling with ECL   [Contents][Index]

4.1.1.5 Shared library

Almost the same as with a static library. The user has to use c:build-shared-library:

;; generates hello.o
(compile-file "hello.lsp" :system-p t)
;; generates goodbye.o
(compile-file "goodbye.lsp" :system-p t)

;; generates libhello-goodbye.so
(c:build-shared-library "hello-goodbye"
                        :lisp-files '("hello.o" "goodbye.o")
                        :init-name "init_hello_goodbye")

Next: , Previous: , Up: Compiling with ECL   [Contents][Index]

4.1.1.6 Executable

ECL supports the generation of executable files. To create a standalone executable from a lisp program, compile all lisp files to object files. After that, calling c:build-program creates the executable:

;; generates hello.o
(compile-file "hello.lsp" :system-p t)
;; generates goodbye.o
(compile-file "goodbye.lsp" :system-p t)

;; generates hello-goodbye
(c:build-program "hello-goodbye"
                 :lisp-files '("hello.o" "goodbye.o"))

Like with native FASL, the program may be built also from libraries.


Previous: , Up: Compiling with ECL   [Contents][Index]

4.1.1.7 Summary

In this post, some file types that can be compiled with ECL were introduced. Each file type has an adequate purpose:

ECL provides a high-level interface c:build-* for each native format. In case of Portable FASL the bytecodes compiler is needed.


Previous: , Up: System building   [Contents][Index]

4.1.2 Compiling with ASDF

First, let’s disregard the simple situation in which we write Lisp without depending on any other Lisp libraries. A more practical example is to build a library that depends on other asdf systems. ECL provides a useful extension for asdf called asdf:make-build, which offers an abstraction for building libraries directly from system definitions.

To download dependencies you may use Quicklisp to load your system (with dependencies defined). Make sure you can successfully load and run your library in the ECL REPL (or *slime-repl*). Don’t worry about other libraries loaded in your image – ECL will only build and pack libraries your project depends on (that is, all dependencies you put in your .asd file, and their dependencies - nothing more, despite the fact that other libraries may be loaded).


Next: , Up: Compiling with ASDF   [Contents][Index]

4.1.2.1 Example code to build

We use a simple project that depends on alexandria to demonstrate the interface. The example consists of example-with-dep.asd, package.lisp and example.lisp (included in the examples/asdf_with_dependence/ directory in the ECL source tree). Before any kind of build you need to push the full path of this directory to asdf:*central-registry* (or link it in a location already recognized by ASDF).


Next: , Previous: , Up: Compiling with ASDF   [Contents][Index]

4.1.2.2 Build it as an single executable

Use this in the REPL to make an executable:

(asdf:make-build :example-with-dep
                 :type :program
                 :move-here #P"./"
                 :epilogue-code '(progn (example:test-function 5)
                                        (si:exit)))

Here the :epilogue-code is executed after loading our library; we can use arbitrary Lisp forms here. You can also put this code in your Lisp files and directly build them without this :epilogue-code option to achieve the same result. Running the program in a console will display the following and exit:

Factorial of 5 is: 120

Next: , Previous: , Up: Compiling with ASDF   [Contents][Index]

4.1.2.3 Build it as shared library and use in C

Use this in the REPL to make a shared library:

(asdf:make-build :example-with-dep
                 :type :shared-library
                 :move-here #P"./"
                 :monolithic t
                 :init-name "init_example")

Here :monolithic t means that ECL will compile the library and all its dependencies into a single library named example-with-dep--all-systems.so. The :move-here parameter is self-explanatory. :init-name sets the name of the initialization function. Each library linked from C/C++ code must be initialized, and this is a mechanism to specify the initialization function’s name.

To use it, we write a simple C program:

/* test.c */
#include <ecl/ecl.h>

int main (int argc, char **argv) {
  extern void init_dll_example(cl_object);
  
  cl_boot(argc, argv);
  ecl_init_module(NULL, init_dll_example);

  /* do things with the Lisp library */
  cl_eval(c_string_to_object("(example:test-function 5)"));

  cl_shutdown();
  return 0;
}

Compile the file using a standard C compiler (note we’re linking to libecl.so with -lecl, which provides the lisp runtime1):

gcc test.c example-with-dep--all-systems.so -o test -lecl

If ECL is installed in a non-standard location you may need to provide flags for the compiler and the linker. You may read them with:

ecl-config --cflags
ecl-config --libs

Since our shared object is not in the standard location, you need to provide LD_LIBRARY_PATH pointing to the current directory to run the application:

LD_LIBRARY_PATH=`pwd` ./test

This will show:

Factorial of 5 is: 120

You can also build all dependent libraries separately as a few .so files and link them together. For example, if you are building a library called complex-example, that depends on alexandria and cl-fad, you can do the following (in the REPL):

(asdf:make-build :complex-example
                 :type :shared-library
                 :move-here #P"./"
                 :init-name "init_example")

(asdf:make-build :alexandria
                 :type :shared-library
                 :move-here #P"./"
                 :init-name "init_alexandria")

(asdf:make-build :cl-fad
                 :type :shared-library
                 :move-here #P"./"
                 :init-name "init_fad")

(asdf:make-build :bordeaux-threads
                 :type :shared-library
                 :move-here #P"./"
                 :init-name "init_bt")

Note that we haven’t specified :monolithic t, so we need to build bordeaux-threads as well because cl-fad depends on it. The building sequence doesn’t matter and the resultant ~.so~ files can also be used in your future programs if these libraries are not modified.

We need to initialize all these modules using ecl_init_module in the correct order. (bordeaux-threads must be initialized before cl-fad; cl-fad and alexandria must be initialized before complex-ecample.)

Here is a code snippet (not a full program):

extern void init_fad(cl_object);
extern void init_alexandria(cl_object);
extern void init_bt(cl_object);
extern void init_example(cl_object);

/* call these *after* cl_boot(argc, argv); 
   if B depends on A, you should first init A then B. */
ecl_init_module(NULL, init_bt);
ecl_init_module(NULL, init_fad);
ecl_init_module(NULL, init_alexandria);
ecl_init_module(NULL, init_example);

Previous: , Up: Compiling with ASDF   [Contents][Index]

4.1.2.4 Build it as static library and use in C

To build a static library, use:

(asdf:make-build :example-with-dep
                 :type :static-library
                 :move-here #P"./"
                 :monolithic t
                 :init-name "init_example")

This will generate example-with-dep--all-systems.a in the current directory which we need to initialize with the init_example function. Compile it using:

gcc test.c example-with-dep--all-systems.a -o test-static -lecl

Then run it:

./test-static

This will show:

Factorial of 5 is: 120

Note we don’t need to pass the current path in ~LD_LIBRARY_PATH~ here, since our Lisp library is statically bundled with the executable. The result is the same as the shared library example above. You can also build all dependent libraries separately as static libraries.


Next: , Previous: , Up: Extensions   [Contents][Index]

4.2 Operating System Interface


Next: , Up: Operating System Interface   [Contents][Index]

4.2.1 Command line arguments

Variable: string ext:*help-message*

Command line help message. Initial value is ECL help message. This variable contains the help message which is output when ECL is invoked with the --help.

Variable: list-of-pathname-designators ext:*lisp-init-file-list*

ECL initialization files. Initial value is '("~/.ecl" "~/.eclrc"). This variable contains the names of initialization files that are loaded by ECL or embedding programs. The loading of initialization files happens automatically in ECL unless invoked with the option --norc. Whether initialization files are loaded or not is controlled by the command line options rules, as described in ext:process-command-args.

Variable: list-of-lists ext:+default-command-arg-rules+

ECL command line options. This constant contains a list of rules for parsing the command line arguments. This list is made of all the options which ECL accepts by default. It can be passed as first argument to ext:process-command-args, and you can use it as a starting point to extend ECL.

Function: ext:command-args

Original list of command line arguments. This function returns the list of command line arguments passed to either ECL or the program it was embedded in. The output is a list of strings and it corresponds to the argv vector in a C program. Typically, the first argument is the name of the program as it was invoked. You should not count on ths filename to be resolved.

Function: ext:process-command-args &key args rules
argument: args

A list of strings. Defaults to the output of ext:command-args.

argument: rules

A list of lists. Defaults to the value of ext:+default-command-arg-rules+.

This function processes the command line arguments passed to either ECL or the program that embeds it. It uses the list of rules rules, which has the following syntax:

(option-name nargs template [:stop | :noloadrc | :loadrc]*)

opt: option-name

A string with the option prefix as typed by the user. For instance --help, -?, --compile, etc.

opt: nargs

A non-negative integer denoting the number of arguments taken by this option.

opt: template

A lisp form, not evaluated, where numbers from 0 to nargs will be replaced by the corresponding option argument.

opt: :STOP

If present, parsing of arguments stops after this option is found and processed. The list of remaining arguments is passed to the rule. ECL’s top-level uses this option with the -- command line option to set ext:*unprocessed-ecl-command-args* to the list of remaining arguments.

opt: :NOLOADRC
opt: :LOADRC

Determine whether the lisp initalization file (ext:*lisp-init-file-list*) will be loaded before processing all forms.

ext:process-command-args works as follows. First of all, it parses all the command line arguments, except for the first one, which is assumed to contain the program name. Each of these arguments is matched against the rules, sequentially, until one of the patterns succeeeds.

A special name *DEFAULT*, matches any unknown command line option. If there is no *DEFAULT* rule and no match is found, an error is signalled. For each rule that succeeds, the function constructs a lisp statement using the template.

After all arguments have been processed, ext:process-command-args, and there were no occurences of :noloadrc, one of the files listed in ext:*lisp-init-file-list* will be loaded. Finally, the list of lisp statements will be evaluated.

The following piece of code implements the ls command using lisp. Instructions for building this program are found under ecl/examples/cmdline/ls.lsp.

(setq ext:*help-message* "
ls [--help | -?] filename*
     Lists the file that match the given patterns.
")

(defun print-directory (pathnames)
  (format t "~{~A~%~}"
          (mapcar #'(lambda (x) (enough-namestring x (si::getcwd)))
                  (mapcan #'directory (or pathnames '("*.*" "*/"))))))

(defconstant +ls-rules+
  '(("--help" 0 (progn (princ ext:*help-message* *standard-output*) (ext:quit 0)))
    ("-?" 0 (progn (princ ext:*help-message* *standard-output*) (ext:quit 0)))
    ("*DEFAULT*" 1 (print-directory 1) :stop)))

(let ((ext:*lisp-init-file-list* NIL)) ; No initialization files
  (handler-case (ext:process-command-args :rules +ls-rules+)
                (error (c)
                       (princ ext:*help-message* *error-output*)
                       (ext:quit 1))))
(ext:quit 0)

Next: , Previous: , Up: Operating System Interface   [Contents][Index]

4.2.2 External processes

ECL provides several facilities for invoking and communicating with ext:external-process. If one just wishes to execute some program, without caring for its output, then probably ext:system is the best function. In all other cases it is preferable to use ext:run-program, which opens pipes to communicate with the program and manipulate it while it runs on the background.

External process is a structure created with ext:run-program (returned as third value). It is programmer responsibility, to call ext:external-process-wait on finished processes, however during garbage collection object will be finalized.

Function: ext:external-process-pid process

Returns process PID or nil if already finished.

Function: ext:external-process-status process

Updates process status. ext:external-process-status calls ext:external-process-wait if proces has not finished yet (non-blocking call). Returns two values:

status - member of (:abort :error :exited :signalled :stopped :resumed :running)

code - if process exited it is a returned value, if terminated it is a signal code. Otherwise NIL.

Function: ext:external-process-wait proces wait

If the second argument is non-NIL, function blocks until external process is finished. Otherwise status is updated. Returns two values (see ext:external-process-status).

Function: ext:terminate-process process &optional force

Terminates external process.

Function: ext:external-process-input process
Function: ext:external-process-output process
Function: ext:external-process-error-stream process

Process stream accessors (read-only).

Function: ext:run-program command argv &key input output error wait environ if-input-does-not-exist if-output-exists if-error-exists external-format #+windows escape-arguments

run-program creates a new process specified by the command argument. argv are the standard arguments that can be passed to a program. For no arguments, use nil (which means that just the name of the program is passed as arg 0).

run-program will return three values - two-way stream for communication, return code or nil (if process is called asynchronously), and ext:external-process object holding process state.

It is programmer responsibility to call ext:external-process-wait on finished process, however ECL associates finalizer with the object calling it when the object is garbage collected. If process didn’t finish but is not referenced, finalizer will be invoked once more during next garbage collection.

The &key arguments have the following meanings:

argument: input

Either t, nil, a pathname, a string, a stream or :stream. If t the standard input for the current process is inherited. If nil, /dev/null is used. If a pathname (or a string), the file so specified is used. If a stream, all the input is read from that stream and sent to the subprocess. If :stream, the external-process-input slot is filled in with a stream that sends its output to the process. Defaults to :stream.

argument: if-input-does-not-exist

can be one of: :error to generate an error :create to create an empty file nil (the default) to return nil from run-program

argument: output

Either t, nil, a pathname, a string, a stream, or :stream. If t, the standard output for the current process is inherited. If nil, /dev/null is used. If a pathname (or as string), the file so specified is used. If a stream, all the output from the process is written to this stream. If :stream, the external-process-output slot is filled in with a stream that can be read to get the output. Defaults to :stream.

argument: if-output-exists
argument: error

Same as :output, except that :error can also be specified as :output in which case all error output is routed to the same place as normal output. Defaults to :output.

argument: if-error-exists

Same as :if-output-exists.

argument: wait

If non-NIL (default), wait until the created process finishes. If nil, continue running Lisp until the program finishes.

argument: environ

A list of STRINGs describing the new Unix environment (as in "man environ"). The default is to copy the environment of the current process. To extend existing environment (instead of replacing it), use :environ (append *my-env* (ext:environ)).

If non-NIL environ argument is supplied, then first argument to ext:run-program command must be full path to the file.

argument: external-format

The external-format to use for :input, :output, and :error STREAMs.

Windows specific options:

argument: escape-arguments

Controls escaping of the arguments passed to CreateProcess.


Previous: , Up: Operating System Interface   [Contents][Index]

4.2.3 Operating System Interface Reference

Function: ext:system command

Run shell command ignoring its output. Uses fork.

Function: ext:make-pipe

Creates a pipe and wraps it in a two way stream.

Function: ext:quit &optional exit-code kill-all-threads

This function abruptly stops the execution of the program in which ECL is embedded. Depending on the platform, several other functions will be invoked to free resources, close loaded modules, etc.

The exit code is the code seen by the parent process that invoked this program. Normally a code other than zero denotes an error.

If kill-all-threads is non-NIL, tries to gently kill and join with running threads.

Function: ext:environ
Function: ext:getenv variable
Function: ext:setenv variable value

Environment accessors.

Function: ext:getpid
Function: ext:getuid
Function: ext:getcwd
Function: ext:chdir
Function: ext:file-kind
Function: ext:copy-file
Function: ext:chmod

Common operating system functions.


Next: , Previous: , Up: Extensions   [Contents][Index]

4.3 Foreign Function Interface


Next: , Up: Foreign Function Interface   [Contents][Index]

4.3.1 What is a FFI?

A Foreign Function Interface, or FFI for short, is a means for a programming language to interface with libraries written in other programming languages, the foreign code. You will see this concept most often being used in interpreted environments, such as Python, Ruby or Lisp, where one wants to reuse the big number of libraries written in C and C++ for dealing with graphical interfaces, networking, filesystems, etc.

A FFI is made of at least three components:

Foreign objects management

This is the data that the foreign code will use. A FFI needs to provide means to build and manipulate foreign data, with automatic conversions to and from lisp data types whenever possible, and it also has to deal with issues like garbage collection and finalization.

Foreign code loader

To actually use a foreign routine, the code must reside in memory. The process of loading this code and finding out the addresses of the routines we want to use is normally done by an independent component.

Foreign function invocation

This is the part of the FFI that deals with actually calling the foreign routines we want to use. For that one typically has to tell the FFI what are the arguments that these routines expect, what are the calling conventions and where are these routines to be found.

On top of these components sits a higher level interface written entirely in lisp, with which you will actually declare and use foreign variables, functions and libraries. In the following sections we describe both the details of the low-level components (Section 3.2, Section 3.3), and of the higher level interface (Section 3.4). It is highly recommended that you read all sections.


Next: , Previous: , Up: Foreign Function Interface   [Contents][Index]

4.3.2 Two kinds of FFI

ECL allows for two different appraoches when building a FFI. Both approaches have a different implementation philosophy and affect the places where you can use the FFI and how.

Static FFI (SFFI)

For every foreign function and variable you might need to use, a wrapper is automatically written in C with the help of ffi:c-inline. These wrappers are compiled using an ordinary C compiler and linked against both the foreign libraries you want to use and against the ECL library. The result is a FASL file that can be loaded from ECL and where the wrappers appear as ordinary lisp functions and variables that the user may directly invoked.

Dynamic FFI (DFFI)

First of all, the foreign libraries are loaded in memory using the facilities of the operating system. Similar routines are used to find out and register the memory location of all the functions and variables we want to use. Finally, when actually accessing these functions, a little piece of assembly code does the job of translating the lisp data into foreign objects, storing the arguments in the stack and in CPU registers, calling the function and converting back the output of the function to lisp.

ECL for this purpose utilizes libffi, a portable foreign-function interface library.

figures/ffi

Figure 4.2: FFI components

As you see, the first approach uses rather portable technices based on a programming language (C, C++) which is strongly supported by the operating system. The conversion of data is performed by a calling routines in the ECL library and we need not care about the precise details (organizing the stack, CPU registers, etc) when calling a function: the compiler does this for us.

On the other hand, the dynamic approach allows us to choose the libraries we load at any time, look for the functions and invoke them even from the toplevel, but it relies on unportable techniques and requires the developers to know very well both the assembly code of the machine the code runs on and the calling conventions of that particular operating system. For these reasons ECL doesn’t maintain it’s own implementation of the DFFI but rather relies on the third party library.

ECL currently supports the static method on all platforms, and the dynamical one a wide range of the most popular ones, shown in the section Supported Platforms at https://sourceware.org/libffi/.

You can test if your copy of ECL was built with DFFI by inspecting whether the symbol :DFFI is present in the list from variable *FEATURES*.


Next: , Previous: , Up: Foreign Function Interface   [Contents][Index]

4.3.3 Foreign objects

While the foreign function invocation protocols differ strongly between platforms and implementations, foreign objects are pretty easy to handle portably. For ECL, a foreign object is just a bunch of bytes stored in memory. The lisp object for a foreign object encapsulates several bits of information:

A foreign object may contain many different kinds of data: integers, floating point numbers, C structures, unions, etc. The actual type of the object is stored in a list or a symbol which is understood by the higher level interface (Section 3.4).

The most important component of the object is the memory region where data is stored. By default ECL assumes that the user will perform automatic management of this memory, deleting the object when it is no longer needed. The first reason is that this block may have been allocated by a foreign routine using malloc(), or mmap(), or statically, by referring to a C constant. The second reason is that foreign functions may store references to this memory which ECL is not aware of and, in order to keep these references valid, ECL should not attempt to automatically destroy the object.

In many cases, however, it is desirable to automatically destroy foreign objects once they have been used. The higher level interfaces UFFI and CFFI provide tools for doing this. For instance, in the following example adapted from the UFFI documentation, the string NAME is automatically deallocated

(def-function "gethostname" 
  ((name (* :unsigned-char))
   (len :int))
  :returning :int)

(if (zerop (c-gethostname (ffi:char-array-to-pointer name) 256))
    (format t "Hostname: ~S" (ffi:convert-from-foreign-string name))
    (error "gethostname() failed."))

Next: , Previous: , Up: Foreign Function Interface   [Contents][Index]

4.3.4 Higher level interfaces

Up to now we have only discussed vague ideas about how a FFI works, but you are probably more interested on how to actually code all these things in lisp. You have here three possibilities:

In the following two subsections we will discuss two practical examples of using the native UFFI and the CFFI library.

UFFI example

The example below shows how to use UFFI in an application. There are several important ingredients:

#|
Build and load this module with (compile-file "uffi.lsp" :load t)
|#
;;
;; This toplevel statement notifies the compiler that we will
;; need this shared library at runtime. We do not need this
;; statement in windows.
;;
#-(or ming32 windows)
(ffi:load-foreign-library #+darwin "/usr/lib/libm.dylib"
			  #-darwin "/usr/lib/libm.so")
;;
;; With this other statement, we import the C function sin(),
;; which operates on IEEE doubles.
;;
(ffi:def-function ("sin" c-sin) ((arg :double))
                  :returning :double)
;;
;; We now use this function and compare with the lisp version.
;;
(format t "~%Lisp sin:~t~d~%C sin:~t~d~%Difference:~t~d"
	(sin 1.0d0) (c-sin 1.0d0) (- (sin 1.0d0) (c-sin 1.0d0)))

CFFI example

The CFFI library is an independent project and it is not shipped with ECL. If you wish to use it you can go to their homepage, download the code and build it using ASDF.

CFFI differs slightly from UFFI in that functions may be used even without being declared beforehand.

#|
Build and load this module with (compile-file "cffi.lsp" :load t)
|#
;;
;; This toplevel statement notifies the compiler that we will
;; need this shared library at runtime. We do not need this
;; statement in windows.
;;
#-(or ming32 windows)
(cffi:load-foreign-library #+darwin "/usr/lib/libm.dylib"
			   #-darwin "/usr/lib/libm.so")
;;
;; With this other statement, we import the C function sin(),
;; which operates on IEEE doubles.
;;
(cffi:defcfun ("sin" c-sin) :double '(:double))
;;
;; We now use this function and compare with the lisp version.
;;
(format t "~%Lisp sin:~t~d~%C sin:~t~d~%Difference:~t~d"
	(sin 1.0d0) (c-sin 1.0d0) (- (sin 1.0d0) (c-sin 1.0d0)))
;;
;; The following also works: no declaration!
;;
(let ((c-cos (cffi:foreign-funcall "cos" :double 1.0d0 :double)))
   (format t "~%Lisp cos:~t~d~%C cos:~t~d~%Difference:~t~d"
	(cos 1.0d0) c-cos (- (cos 1.0d0) c-cos)))

SFFI example (low level inlining)

To compare with the previous pieces of code, we show how the previous programs would be written using ffi:clines and ffi:c-inline.

#|
Build and load this module with (compile-file "ecl.lsp" :load t)
|#
;;
;; With this other statement, we import the C function sin(), which
;; operates on IEEE doubles. Notice that we include the C header to
;; get the full declaration.
;;
(defun c-sin (x)
  (ffi:clines "#include <math.h>")
  (ffi:c-inline (x) (:double) :double "sin(#0)" :one-liner t))
;;
;; We now use this function and compare with the lisp version.
;;
(format t "~%Lisp sin:~t~d~%C sin:~t~d~%Difference:~t~d"
	(sin 1.0d0) (c-sin 1.0d0) (- (sin 1.0d0) (c-sin 1.0d0)))

Next: , Previous: , Up: Foreign Function Interface   [Contents][Index]

4.3.5 SFFI Reference

Reference

Special Form: ffi:clines c/c++-code*

Insert C declarations and definitions

c/c++-code

One or more strings with C definitions. Not evaluated.

returns

No value.

Description

This special form inserts C code from strings passed in the arguments directly in the file that results from compiling lisp sources. Contrary to ffi:c-inline, this function may have no executable statements, accepts no input value and returns no value.

The main use of FFI:CLINES is to declare or define C variables and functions that are going to be used later in other FFI statements. All statements from arguments are grouped at the beginning of the produced header file.

FFI:CLINES is a special form that can only be used in lisp compiled files as a toplevel form. Other uses will lead to an error being signaled, either at the compilation time or when loading the file.

Examples

In this example the FFI:CLINES statement is required to get access to the C function cos:

(ffi:clines "#include <math.h>")
(defun cos (x)
  (ffi:c-inline (x) (:double) :double "cos(#0)" :on-liner t))
Special Form: ffi:c-inline (lisp-values) (arg-c-types) return-type c/c++-code &key (side-effects t) (one-liner nil)

Inline C code in a lisp form

lisp-values

One or more lisp expressions. Evaluated.

arg-c-types

One or more valid FFI types. Evaluated.

return-type

Valid FFI type or (VALUES ffi-type*).

c/c++-code

String containing valid C code plus some valid escape forms.

one-liner

Boolean indicating, if the expression is a valid R-value. Defaults to NIL.

side-effects

Boolean indicating, if the expression causes side effects. Defaults to T.

returns

One or more lisp values.

Description

This is a special form which can be only used in compiled code and whose purpose is to execute some C code getting and returning values from and to the lisp environment.

The first argument lisp-values is a list of lisp forms. These forms are going to be evaluated and their lisp values will be transformed to the corresponding C types denoted by the elements in the list arg-c-types.

The input values are used to create a valid C expression using the template in C/C++-code. This is a string of arbitrary size which mixes C expressions with two kind of escape forms.

The first kind of escape form are made of a hash and a letter or a number, as in: #0, #1, ..., until #z. These codes are replaced by the corresponding input values. The second kind of escape form has the format @(return [n]), it can be used as lvalue in a C expression and it is used to set the n-th output value of the ffi:c-inline form.

When the parameter one-liner is true, then the C template must be a simple C statement that outputs a value. In this case the use of @(return) is not allowed. When the parameter one-liner is false, then the C template may be a more complicated block form, with braces, conditionals, loops and spanning multiple lines. In this case the output of the form can only be set using @(return).

Parameter side-effects set to true will indicate, that the functions causes no side-effects. This information is used by the compiler to optimize the resulting code. If side-effects is set to true, but the function may cause the side effects, then results are undefined.

Note that the conversion between lisp arguments and FFI types is automatic. Note also that ffi:c-inline cannot be used in interpreted or bytecompiled code! Such usage will signal an error.

Examples

The following example implements the transcendental function SIN using the C equivalent:

(ffi:c-lines "#include <math.h>")
(defun mysin (x)
  (ffi:c-inline (x) (:double) :double
    "sin(#0)"
    :one-liner t
    :side-effects nil))

This function can also be implemented using the @(return) form as follows:

(defun mysin (x)
  (ffi:c-inline (x) (:double) :double
    "@(return)=sin(#0);"
    :side-effects nil))

The following example is slightly more complicated as it involves loops and two output values:

(defun sample (x)
  (ffi:c-inline (x (+ x 2)) (:int :int) (values :int :int) "{
    int n1 = #0, n2 = #1, out1 = 0, out2 = 1;
    while (n1 <= n2) {
      out1 += n1;
      out2 *= n1;
      n1++;
    }
    @(return 0)= out1;
    @(return 1)= out2;
    }"
   :side-effects nil))
Special Form: ffi:c-progn args &body body

Interleave C statements with the Lisp code

args

Lisp arguments. Evaluated.

returns

No value.

Description

This form is used for it’s side effects. It allows for interleaving C statements with the Lisp code. The argument types doesn’t have to be declared – in such case the objects type in the C world will be cl_object.

Examples

(lambda (i)
  (let* ((limit i)
         (iterator 0)
         (custom-var (cons 1 2)))
    (declare (:int limit iterator))
    (ffi:c-progn (limit iterator custom-var)
                 "cl_object cv = #2;"
                 "ecl_print(cv, ECL_T);"
                 "for (#1 = 0; #1 < #0; #1++) {"
                 (format t "~&Iterator: ~A, I: ~A~%" iterator i)
                 "}")))
Special Form: ffi:defcallback name ret-type arg-desc &body body
name

Name of the lisp function.

ret-type

Declaration of the return type which function returns.

arg-desc

List of pairs (arg-name arg-type).

body

Function body.

returns

Pointer to the defined callback.

Description

Defines Lisp function and generates a callback for the C world, which may be passed to these functions. Note, that this special operator has also a dynamic variant (with the same name and interface).

Macro: ffi:defcbody name arg-types result-type c-expression

Define C function under the lisp name

name

Defined function name.

arg-types

Argument types of the defined Lisp function.

result-type

Result type of the C function (may be (values ...).

returns

Defined function name.

Description

The compiler defines a Lisp function named by NAME whose body consists of the C code of the string C-EXPRESSION. In the C-EXPRESSION one can reference the arguments of the function as #0, #1, etc.

The interpreter ignores this form.

Macro: ffi:defentry name arg-types c-name &key no-interrupts
name

Lisp name for the function.

arg-types

Argument types of the C function (one of the symbols OBJECT, INT, CHAR, CHAR*, FLOAT, DOUBLE).

c-name

If C-NAME is a list, then C function result type is declared as (CAR C-NAME) and its name is (STRING (CDR C-NAME)).

If it’s an atom, then the result type is OBJECT, and function name is (STRING C-NAME).

returns

Lisp function NAME.

Description

The compiler defines a Lisp function named by NAME whose body consists of a calling sequence to the C language function named by FUNCTION-NAME.

The interpreter ignores this form. ARG-TYPES are argument types of the C function and RESULT-TYPE is its return type. Symbols OBJECT, INT, CHAR, CHAR*, FLOAT, DOUBLE are allowed for these types.

Macro: ffi:defla name args &body body

Provide Lisp alternative for interpreted code.

Description

Used to DEFine Lisp Alternative. For the interpreter, DEFLA is equivalent to DEFUN, but the compiler ignores this form.


Previous: , Up: Foreign Function Interface   [Contents][Index]

4.3.6 UFFI Reference


Next: , Up: UFFI Reference   [Contents][Index]

4.3.6.1 Primitive Types

Primitive types have a single value, these include characters, numbers, and pointers. They are all symbols in the keyword package.

:char
:unsigned-char

Signed/unsigned 8-bits. Dereferenced pointer returns a character.

:byte
:unsigned-byte

Signed/unsigned 8-bits. Dereferenced pointer returns an integer.

:short
:unsigned-short
:int
:unsigned-int
:long
:unsigned-long

Standard integer types (16-bit, 32-bit and 32/64-bit).

:int16_t
:uint16_t
:int32_t
:uint32_t
:int64_t
:uint64_t

Integer types with guaranteed bitness.

:float
:double

Floating point numerals (32-bit and 64-bit).

:cstring

A NULL terminated string used for passing and returning characters strings with a C function.

:void

The absence of a value. Used to indicate that a function does not return a value.

:pointer-void

Points to a generic object.

*

Used to declare a pointer to an object.

Reference

Macro: ffi:def-constant name value &key (export nil)

Binds a symbol to a constant.

name

A symbol that will be bound to the value.

value

An evaluated form that is bound the the name.

export

When T, the name is exported from the current package. Defaults to NIL.

returns

Constant name.

Description

This is a thin wrapper around defconstant. It evaluates at compile-time and optionally exports the symbol from the package.

Examples

(ffi:def-constant pi2 (* 2 pi))
(ffi:def-constant exported-pi2 (* 2 pi) :export t)

Side Effects

Creats a new special variable.

Macro: ffi:def-foreign-type name definition

Defines a new foreign type

name

A symbol naming the new foreign type.

value

A form that is not evaluated that defines the new foreign type.

returns

Foreign type designator (value).

Description

Defines a new foreign type

Examples

(def-foreign-type my-generic-pointer :pointer-void)
(def-foreign-type a-double-float :double-float)
(def-foreign-type char-ptr (* :char))

Side effects

Defines a new foreign type.

Function: ffi:null-char-p char

Tests a character for NULL value

char

A character or integer.

returns

A boolean flag indicating if char is a NULL value.

Description

A predicate testing if a character or integer is NULL. This abstracts the difference in implementations where some return a character and some return a integer whence dereferencing a C character pointer.

Examples

(ffi:def-array-pointer ca :unsigned-char)
  (let ((fs (ffi:convert-to-foreign-string "ab")))
    (values (ffi:null-char-p (ffi:deref-array fs 'ca 0))
            (ffi:null-char-p (ffi:deref-array fs 'ca 2))))
;; => NIL T

Next: , Previous: , Up: UFFI Reference   [Contents][Index]

4.3.6.2 Aggregate Types

Overview

Aggregate types are comprised of one or more primitive types.

Reference

Macro: ffi:def-enum name fields &key separator-key

Defines a C enumeration

name

A symbol that names the enumeration.

fields

A list of field defintions. Each definition can be a symbol or a list of two elements. Symbols get assigned a value of the current counter which starts at 0 and increments by 1 for each subsequent symbol. It the field definition is a list, the first position is the symbol and the second position is the value to assign the the symbol. The current counter gets set to 1+ this value.

returns

A string that governs the creation of constants. The default is "#".

Description

Declares a C enumeration. It generates constants with integer values for the elements of the enumeration. The symbols for the these constant values are created by the concatenation of the enumeration name, separator-string, and field symbol. Also creates a foreign type with the name name of type :int.

Examples

(ffi:def-enum abc (:a :b :c)) 
;; Creates constants abc#a (1), abc#b (2), abc#c (3) and defines
;; the foreign type "abc" to be :int

(ffi:def-enum efoo (:e1 (:e2 10) :e3) :separator-string "-")
;; Creates constants efoo-e1 (1), efoo-e2 (10), efoo-e3 (11) and defines
;; the foreign type efoo to be :int

Side effects

Creates a :int foreign type, defines constants.

Macro: ffi:def-struct name &rest fields

Defines a C structure

name

A symbol that names the structure.

fields

A variable number of field defintions. Each definition is a list consisting of a symbol naming the field followed by its foreign type.

Description

Declares a structure. A special type is available as a slot in the field. It is a pointer that points to an instance of the parent structure. It’s type is :pointer-self.

Examples

(ffi:def-struct foo (a :unsigned-int) 
  (b    (* :char)) 
  (c    (:array :int 10)) 
  (next :pointer-self))

Side effects

Creates a foreign type.

Function: ffi:get-slot-value obj type field

Retrieves a value from a slot of a structure

obj

A pointer to the foreign structure.

type

A name of the foreign structure.

field

A name of the desired field in foreign structure.

returns

The value of the field in the structure obj.

Description

Accesses a slot value from a structure. This is generalized and can be used with SETF-able.

Examples

(get-slot-value foo-ptr 'foo-structure 'field-name)
(setf (get-slot-value foo-ptr 'foo-structure 'field-name) 10)
Function: ffi:get-slot-pointer obj type field

Retrieves a pointer from a slot of a structure

obj

A pointer to the foreign structure.

type

A name of the foreign structure.

field

A name of the desired field in foreign structure.

returns

The value of the pointer field in the structure obj.

Description

This is similar to get-slot-value. It is used when the value of a slot is a pointer type.

Examples

(get-slot-pointer foo-ptr 'foo-structure 'my-char-ptr)
Macro: ffi:def-array-pointer name type

Defines a pointer to an array of type

name

A name of the new foreign type.

type

The foreign type of the array elements.

Description

Defines a type that is a pointer to an array of type.

Examples

(def-array-pointer byte-array-pointer :unsigned-char)

Side effects

Defines a new foreign type.

Function: ffi:deref-array array type position

Deference an array

array

A foreign array.

type

The foreign type of the array.

position

An integer specifying the position to retrieve from the array.

returns

The value stored in the position of the array.

Description

Dereferences (retrieves) the value of the foreign array element. SETF-able.

Examples

(ffi:def-array-pointer ca :char) (let ((fs (ffi:convert-to-foreign-string "ab"))) (values (ffi:null-char-p (ffi:deref-array fs ’ca 0)) (ffi:null-char-p (ffi:deref-array fs ’ca 2)))) ;; => NIL T

Macro: ffi:def-union name &rest fields

Defines a foreign union type

name

A name of the new union type.

fields

A list of fields of the union in form (field-name fields-type).

Description

Defines a foreign union type.

Examples

(ffi:def-union test-union
  (a-char :char)
  (an-int :int))

(let ((u (ffi:allocate-foreign-object 'test-union)))
  (setf (ffi:get-slot-value u 'test-union 'an-int) (+ 65 (* 66 256)))
  (prog1
     (ffi:ensure-char-character (ffi:get-slot-value u 'test-union 'a-char))
   (ffi:free-foreign-object u)))
;; => #\A

Side effects

Defines a new foreign type.


Next: , Previous: , Up: UFFI Reference   [Contents][Index]

4.3.6.3 Foreign Objects

Overview

Objects are entities that can allocated, referred to by pointers, and can be freed.

Reference

Function: ffi:allocate-foreign-object type &optional size

Allocates an instance of a foreign object

type

The type of foreign object to allocate. This parameter is evaluated.

size

An optional size parameter that is evaluated. If specified, allocates and returns an array of type that is size members long. This parameter is evaluated.

returns

A pointer to the foreign object.

Description

Allocates an instance of a foreign object. It returns a pointer to the object.

Examples

(ffi:def-struct ab (a :int) (b :double))
;; => (:STRUCT (A :INT) (B :DOUBLE))
(ffi:allocate-foreign-object 'ab)
;; => #<foreign AB>
Function: ffi:free-foreign-object ptr

Frees memory that was allocated for a foreign object

ptr

A pointer to the allocated foreign object to free.

Description

Frees memory that was allocated for a foreign object.

Macro: ffi:with-foreign-object (var type) &body body

Wraps the allocation, binding and destruction of a foreign object around a body of code

var

Variable name to bind.

type

Type of foreign object to allocate. This parameter is evaluated.

body

Code to be evaluated.

returns

The result of evaluating the body.

Description

This function wraps the allocation, binding, and destruction of a foreign object around the body of code.

Examples

(defun gethostname2 ()
  "Returns the hostname"
  (ffi:with-foreign-object (name '(:array :unsigned-char 256))
    (if (zerop (c-gethostname (ffi:char-array-to-pointer name) 256))
        (ffi:convert-from-foreign-string name)
        (error "gethostname() failed."))))
Macro: ffi:size-of-foreign-type ftype

Returns the number of data bytes used by a foreign object type

ftype

A foreign type specifier. This parameter is evaluated.

returns

Number of data bytes used by a foreign object ftype.

Description

Returns the number of data bytes used by a foreign object type. This does not include any Lisp storage overhead.

Examples

(ffi:size-of-foreign-type :unsigned-byte)
;; => 1
(ffi:size-of-foreign-type 'my-100-byte-vector-type)
;; => 100
Function: ffi:pointer-address ptr

Returns the address of a pointer

ptr

A pointer to a foreign object.

returns

An integer representing the pointer’s address.

Description

Returns the address as an integer of a pointer.

Function: ffi:deref-pointer ptr ftype

Deferences a pointer

ptr

Pointer to a foreign object.

ftype

Foreign type of the object being pointed to.

returns

The value of the object where the pointer points.

Description

Returns the object to which a pointer points. SETF-able.

Notes

Casting of the pointer may be performed with WITH-CAST-POINTER together with the DEREF-POINTER/DEREF-ARRAY.

Examples

(let ((intp (ffi:allocate-foreign-object :int)))
  (setf (ffi:deref-pointer intp :int) 10)
  (prog1
      (ffi:deref-pointer intp :int)
    (ffi:free-foreign-object intp)))
;; => 10
Function: ffi:ensure-char-character object

Ensures that a dereferenced :char pointer is a character

object

Either a character or a integer specifying a character code.

returns

A character.

Description

Ensures that an objects obtained by dereferencing :char and :unsigned-char pointers are a lisp character.

Examples

(let ((fs (ffi:convert-to-foreign-string "a")))
  (prog1 
      (ffi:ensure-char-character (ffi:deref-pointer fs :char))
    (ffi:free-foreign-object fs)))
;; => #\a

Exceptional Situations

Depending upon the implementation and what UFFI expects, this macro may signal an error if the object is not a character or integer.

Function: ffi:ensure-char-integer object

Ensures that a dereferenced :char pointer is an integer

object

Either a character or a integer specifying a character code.

returns

An integer.

Description

Ensures that an objects obtained by dereferencing :char and :unsigned-char pointers is a lisp integer.

Examples

(let ((fs (ffi:convert-to-foreign-string "a")))
  (prog1 
      (ffi:ensure-char-integer (ffi:deref-pointer fs :char))
    (ffi:free-foreign-object fs)))
;; => 96

Exceptional Situations

Depending upon the implementation and what UFFI expects, this macro may signal an error if the object is not a character or integer.

Function: ffi:make-null-pointer ftype

Create a NULL pointer of a specified type

ftype

A type of object to which the pointer refers.

returns

The NULL pointer of type ftype.

Function: ffi:null-pointer-p ptr

Tests a pointer for NULL value

ptr

A foreign object pointer.

returns

The boolean flag.

FFI: +null-cstring-pointer+

A NULL cstring pointer. This can be used for testing if a cstring returned by a function is NULL.

Macro: ffi:with-cast-pointer (var ptr ftype) &body body

Wraps a body of code with a pointer cast to a new type

var

Symbol which will be bound to the casted object.

ptr

Pointer to a foreign object.

ftype

A foreign type of the object being pointed to.

returns

The value of the object where the pointer points.

Description

Executes BODY with PTR cast to be a pointer to type FTYPE. VAR is will be bound to this value during the execution of BODY.

Examples

(ffi:with-foreign-object (size :int)
  ;; FOO is a foreign function returning a :POINTER-VOID
  (let ((memory (foo size)))
    (when (mumble)
      ;; at this point we know for some reason that MEMORY points
      ;; to an array of unsigned bytes
      (ffi:with-cast-pointer (memory :unsigned-byte)
        (dotimes (i (deref-pointer size :int))
          (do-something-with
              (ffi:deref-array memory '(:array :unsigned-byte) i)))))))
Macro: ffi:def-foreign-var name type module

Defines a symbol macro to access a variable in foreign code

name

A string or list specificying the symbol macro’s name. If it is a string, that names the foreign variable. A Lisp name is created by translating #\_ to #\- and by converting to upper-case in case-insensitive Lisp implementations.

If it is a list, the first item is a string specifying the foreign variable name and the second it is a symbol stating the Lisp name.

type

A foreign type of the foreign variable.

module

A string specifying the module (or library) the foreign variable resides in.

Description

Defines a symbol macro which can be used to access (get and set) the value of a variable in foreign code.

Examples

C code defining foreign structure, standalone integer and the accessor:

int baz = 3;

typedef struct {
  int x;
  double y;
} foo_struct;

foo_struct the_struct = { 42, 3.2 };

int foo () {
  return baz;
}

Lisp code defining C structure, function and a variable:

(ffi:def-struct foo-struct
  (x :int)
  (y :double))

(ffi:def-function ("foo" foo) ()
  :returning :int
  :module "foo")

(ffi:def-foreign-var ("baz" *baz*) :int "foo")
(ffi:def-foreign-var ("the_struct" *the-struct*) foo-struct "foo")

*baz*           ;; => 3
(incf *baz*)    ;; => 4
(foo)           ;; => 4

Next: , Previous: , Up: UFFI Reference   [Contents][Index]

4.3.6.4 Foreign Strings

Overview

UFFI has functions to two types of C-compatible strings: cstring and foreign strings. cstrings are used only as parameters to and from functions. In some implementations a cstring is not a foreign type but rather the Lisp string itself. On other platforms a cstring is a newly allocated foreign vector for storing characters. The following is an example of using cstrings to both send and return a value.

(ffi:def-function ("getenv" c-getenv) 
    ((name :cstring))
  :returning :cstring)

(defun my-getenv (key)
  "Returns an environment variable, or NIL if it does not exist"
  (check-type key string)
  (ffi:with-cstring (key-native key)
    (ffi:convert-from-cstring (c-getenv key-native))))

In contrast, foreign strings are always a foreign vector of characters which have memory allocated. Thus, if you need to allocate memory to hold the return value of a string, you must use a foreign string and not a cstring. The following is an example of using a foreign string for a return value.

(ffi:def-function ("gethostname" c-gethostname)
    ((name (* :unsigned-char))
     (len :int))
  :returning :int)

(defun gethostname ()
  "Returns the hostname"
  (let* ((name (ffi:allocate-foreign-string 256))
         (result-code (c-gethostname name 256))
         (hostname (when (zerop result-code)
                     (ffi:convert-from-foreign-string name))))
    ;; UFFI does not yet provide a universal way to free
    ;; memory allocated by C's malloc. At this point, a program
    ;; needs to call C's free function to free such memory.
    (unless (zerop result-code)
      (error "gethostname() failed."))))

Foreign functions that return pointers to freshly allocated strings should in general not return cstrings, but foreign strings. (There is no portable way to release such cstrings from Lisp.) The following is an example of handling such a function.

(ffi:def-function ("readline" c-readline)
    ((prompt :cstring))
  :returning (* :char))

(defun readline (prompt)
  "Reads a string from console with line-editing."
  (ffi:with-cstring (c-prompt prompt)
    (let* ((c-str (c-readline c-prompt))
           (str (ffi:convert-from-foreign-string c-str)))
      (ffi:free-foreign-object c-str)
      str)))

Reference

Macro: ffi:convert-from-cstring object

Converts a cstring to a Lisp string

object

cstring

returns

Lisp string

Description

Converts a Lisp string to a cstring. This is most often used when processing the results of a foreign function that returns a cstring.

Macro: ffi:convert-to-cstring object

Converts a Lisp string to a cstring

object

Lisp string

returns

cstring

Description

Converts a Lisp string to a cstring. The cstring should be freed with free-cstring.

Side Effects

This function allocates memory.

Macro: ffi:convert-from-cstring cstring

Free memory used by cstring

cstring

cstring to be freed.

Description

Frees any memory possibly allocated by convert-to-cstring. On ECL, a cstring is just the Lisp string itself.

Macro: ffi:with-cstring (cstring string) &body body

Binds a newly created cstring

cstring

A symbol naming the cstring to be created.

string

A Lisp string that will be translated to a cstring.

body

The body of where the cstring will be bound.

returns

Result of evaluating the body.

Description

Binds a symbol to a cstring created from conversion of a string. Automatically frees the cstring.

Examples

(ffi:def-function ("getenv" c-getenv) 
    ((name :cstring))
  :returning :cstring)

(defun getenv (key)
  "Returns an environment variable, or NIL if it does not exist"
  (check-type key string)
  (ffi:with-cstring (key-cstring key)
    (ffi:convert-from-cstring (c-getenv key-cstring))))
Macro: ffi:with-cstrings bindings &body body

Binds a newly created cstrings

bindings

List of pairs (cstring string), where cstring is a name for a cstring translated from Lisp string string.

body

The body of where the bindings will be bound.

returns

Result of evaluating the body.

Description

Binds a symbols to a cstrings created from conversion of a strings. Automatically frees the cstrings. This macro works similar to LET*. Based on with-cstring.

Macro: ffi:convert-from-foreign-string foreign-string &key length (null-terminated-p t)

Converts a foreign string into a Lisp string

foreign-string

A foreign string.

length

The length of the foreign string to convert. The default is the length of the string until a NULL character is reached.

null-terminated-p

A boolean flag with a default value of T When true, the string is converted until the first NULL character is reached.

returns

A Lisp string.

Description

Returns a Lisp string from a foreign string. Can translated ASCII and binary strings.

Macro: ffi:convert-to-foreign-string

Converts a Lisp string to a foreign string

string

A Lisp string.

returns

A foreign string.

Description

Converts a Lisp string to a foreign string. Memory should be freed with free-foreign-object.

Macro: ffi:allocate-foreign-string size &key unsigned

Allocates space for a foreign string

size

The size of the space to be allocated in bytes.

unsigned

A boolean flag with a default value of T. When true, marks the pointer as an :unsigned-char.

returns

A foreign string which has undefined contents.

Description

Allocates space for a foreign string. Memory should be freed with free-foreign-object.

Macro: ffi:with-foreign-string (foreign-string string) &body body

Binds a newly allocated foreign-string

foreign-string

A symbol naming the foreign string to be created.

string

A Lisp string that will be translated to a foreign string.

body

The body of where the foreign-string will be bound.

returns

Result of evaluating the body.

Description

Binds a symbol to a foreign-string created from conversion of a string. Automatically deallocates the foreign-string.

Examples

Macro: ffi:with-foreign-strings bindings &body body

Binds a newly created foreign string

bindings

List of pairs (foreign-string string), where foreign-string is a name for a foreign string translated from Lisp string string.

body

The body of where the bindings will be bound.

returns

Result of evaluating the body.

Description

Binds a symbols to a foreign-strings created from conversion of a strings. Automatically frees the foreign-strings. This macro works similar to LET*. Based on with-foreign-string.


Previous: , Up: UFFI Reference   [Contents][Index]

4.3.6.5 Functions and Libraries

Reference

Macro: ffi:def-function name args &key module (returning :void) (call :cdecl)
name

A string or list specificying the function name. If it is a string, that names the foreign function. A Lisp name is created by translating #\_ to #\- and by converting to upper-case in case-insensitive Lisp implementations. If it is a list, the first item is a string specifying the foreign function name and the second it is a symbol stating the Lisp name.

args

A list of argument declarations. If NIL, indicates that the function does not take any arguments.

module

A string specifying which module (or library) that the foreign function resides.

call

Function calling convention. May be one of :default, :cdecl, :sysv, :stdcall, :win64 and unix64.

This argument is used only when we’re using the dynamic function interface. If ECL is built without the DFFI support, then it uses SFFI the call argument is ignored.

returning

A declaration specifying the result type of the foreign function. If :void indicates module does not return any value.

Description

Declares a foreign function.

Examples

(def-function "gethostname"
    ((name (* :unsigned-char))
     (len :int))
  :returning) :int)
Macro: ffi:load-foreign-library filename &key module supporting-libraries force-load system-library
filename

A string or pathname specifying the library location in the filesystem.

module

IGNORED A string designating the name of the module to apply to functions in this library.

supporting-libraries

IGNORED A list of strings naming the libraries required to link the foreign library.

force-load

IGNORED Forces the loading of the library if it has been previously loaded.

system-library

Denotes if the loaded library is a system library (accessible with the correct linker flags). If T, then SFFI is used and the linking is performed after compilation of the module. Otherwise (default) both SFFI and DFFI are used, but SFFI only during the compilation.

returns

A generalized boolean true if the library was able to be loaded successfully or if the library has been previously loaded, otherwise NIL.

Description

Loads a foreign library. Ensures that a library is only loaded once during a session.

Examples

(ffi:load-foreign-library #p"/usr/lib/libmagic.so.1")
;; => #<codeblock "/usr/lib/libmagic.so">

Side Effects

Loads the foreign code into the Lisp system.

Affected by

Ability to load the file.

Function: ffi:find-foreign-library names directories &key drive-letters types

Finds a foreign library file

names

A string or list of strings containing the base name of the library file.

directories

A string or list of strings containing the directory the library file.

drive-letters

A string or list of strings containing the drive letters for the library file.

types

A string or list of strings containing the file type of the library file. Default is NIL. If NIL, will use a default type based on the currently running implementation.

returns

A path containing the path to the first file found, or NIL if the library file was not found.

Description

Finds a foreign library by searching through a number of possible locations. Returns the path of the first found file.

Examples

(ffi:find-foreign-library '("libz" "libmagic")
                          '("/usr/local/lib/" "/usr/lib/")
                          :types '("so" "dll"))
;; => #P"/usr/lib/libz.so.1.2.8"

Next: , Previous: , Up: Extensions   [Contents][Index]

4.4 Native threads


Next: , Up: Native threads   [Contents][Index]

4.4.1 Tasks, threads or processes

On most platforms, ECL supports native multithreading. That means there can be several tasks executing lisp code on parallel and sharing memory, variables and files. The interface for multitasking in ECL, like those of most other implementations, is based on a set of functions and types that resemble the multiprocessing capabilities of old Lisp Machines.

This backward compatibility is why tasks or threads are called "processes". However, they should not be confused with operating system processes, which are made of programs running in separate contexts and without access to each other’s memory.

The implementation of threads in ECL is purely native and based on Posix Threads wherever avaiable. The use of native threads has advantanges. For instance, they allow for non-blocking file operations, so that while one task is reading a file, a different one is performing a computation.

As mentioned above, tasks share the same memory, as well as the set of open files and sockets. This manifests on two features. First of all, different tasks can operate on the same lisp objects, reading and writing their slots, or manipulating the same arrays. Second, while threads share global variables, constants and function definitions they can also have thread-local bindings to special variables that are not seen by other tasks.

The fact that different tasks have access to the same set of data allows both for flexibility and a greater risk. In order to control access to different resources, ECL provides the user with locks, as explained in the next section.


Next: , Previous: , Up: Native threads   [Contents][Index]

4.4.2 Processes (native threads)

Process is a primitive representing native thread.


Next: , Previous: , Up: Native threads   [Contents][Index]

4.4.3 Processes dictionary

Function: cl_object mp_all_processes ()
Function: mp:all-processes

Returns the list of processes associated to running tasks. The list is a fresh new one and can be destructively modified. However, it may happen that the output list is not up to date, because some of the tasks has expired before this copy is returned.

Function: cl_object mp_all_processes () ecl_attr_noreturn
Function: mp:exit_process

When called from a running task, this function immediately causes the task to finish. When invoked from the main thread, it is equivalent to invoking ext:quit with exit code 0.

Function: cl_object mp_interrupt_process (cl_object process, cl_object function)
Function: mp:interrupt_process process function

Interrupt a task. This function sends a signal to a running process. When the task is free to process that signal, it will stop whatever it is doing and execute the given function.

Example:

Kill a task that is doing nothing (See mp:process-kill).

(flet ((task-to-be-killed ()
         ;; Infinite loop
         (loop (sleep 1))))
  (let ((task (mp:process-run-function 'background #'task-to-be-killed)))
    (sleep 10)
    (mp:interrupt-process task 'mp:exit-process)))
Function: cl_object mp_make_process (cl_narg narg, ...)
Function: mp:make-process &key name initial-bindings

Create a new thread. This function creates a separate task with a name set to name, set of variable bindings initial-bindings and no function to run. See also mp:process-run-function. Returns newly created process.

Function: cl_object mp_make_process (cl_object process)
Function: mp:process-active-p process

Returns t when process is active, nil otherwise. Signals an error if process doesn’t designate a valid process.

Function: cl_object mp_process_enable (cl_object process)
Function: mp:process-enable process

The argument to this function should be a process created by mp:make-process, which has a function associated as per mp:process-preset but which is not yet running. After invoking this function a new thread will be created in which the associated function will be executed.

(defun process-run-function (process-name process-function &rest args)
  (let ((process (mp:make-process name)))
    (apply #'mp:process-preset process function args)
    (mp:process-enable process)))
Function: cl_object mp_process_yield ()
Function: mp:process-yield

Yield the processor to other threads.

Function: cl_object mp_process_join (cl_object process)
Function: mp:process-join process

Suspend current thread until process exits. Return the result values of the process function. If process is the current thread, signal an error with.

Function: cl_object mp_process_kill (cl_object process)
Function: mp:process-kill process

Try to stop a running task. Killing a process may fail if the task has disabled interrupts.

Example:

Kill a task that is doing nothing

(flet ((task-to-be-killed ()
         ;; Infinite loop
         (loop (sleep 1))))
  (let ((task (mp:process-run-function 'background #'task-to-be-killed)))
    (sleep 10)
    (mp:process-kill task)))
Function: cl_object mp_process_suspend (cl_object process)
Function: mp:process-suspend process

Suspend a running process. May be resumed with mp:process-resume.

Example:

(flet ((ticking-task ()
         ;; Infinite loop
         (loop
            (sleep 1)
            (print :tick))))
  (print "Running task (one tick per second)")
  (let ((task (mp:process-run-function 'background #'ticking-task)))
    (sleep 5)
    (print "Suspending task for 5 seconds")
    (mp:process-suspend task)
    (sleep 5)
    (print "Resuming task for 5 seconds")
    (mp:process-resume task)
    (sleep 5)
    (print "Killing task")
    (mp:process-kill task)))
Function: cl_object mp_process_resume (cl_object process)
Function: mp:process-resume process

Resumes a suspended process. See example in mp:process-suspend.

Function: cl_object mp_process_name (cl_object process)
Function: mp:process-name process

Returns the name of a process (if any).

Function: cl_object mp_process_preset (cl_narg narg, cl_object process, cl_object function, ...)
Function: mp:process-preset process function &rest function-args

Associates a function to call with the arguments function-args, with a stopped process. The function will be the entry point when the task is enabled in the future.

See mp:enable-process and mp:process-run-function.

Function: cl_object mp_process_run_function (cl_narg narg, cl_object name, cl_object function, ...)
Function: mp:process_run_function name function &rest funciton-args

Create a new process using mp:make-process, associate a function to it and start it using mp:process-preset.

Example:

(flet ((count-numbers (end-number)
         (dotimes (i end-number)
	   (format t "~%;;; Counting: ~i" i)
	   (terpri)
	   (sleep 1))))
  (mp:process-run-function 'counter #'count-numbers 10))
Function: cl_object mp_current_process ()
Function: mp:current-process

Returns the current process of a caller.

Function: cl_object mp_block_signals ()
Function: mp:block-signals

Blocks process for interrupts and returns the previous sigmask.

See mp:interrupt-process.

Function: cl_object mp_restore_signals (cl_object sigmask)
Function: mp:restor-signals sigmask

Enables the interrupts from sigmask.

See mp:interrupt-process.

Macro: mp:without-interrupts &body body

Executes body with all deferrable interrupts disabled. Deferrable interrupts arriving during execution of the body take effect after body has been executed.

Deferrable interrupts include most blockable POSIX signals, and mp:interrupt-thread. Does not interfere with garbage collection, and unlike in many traditional Lisps using userspace threads, in ECL mp:without-interrupts does not inhibit scheduling of other threads.

Binds allow-with-interrupts, with-local-interrupts and with-restored-interrupts as a local macros.

with-restored-interrupts executes the body with interrupts enabled if and only if the without-interrupts was in an environment in which interrupts were allowed.

allow-with-interrupts allows the with-interrupts to take effect during the dynamic scope of its body, unless there is an outer without-interrupts without a corresponding allow-with-interrupts.

with-local-interrupts executes its body with interrupts enabled provided that for there is an allow-with-interrupts for every without-interrupts surrounding the current one. with-local-interrupts is equivalent to:

(allow-with-interrupts (with-interrupts ...))

Care must be taken not to let either allow-with-interrupts or with-local-interrupts appear in a function that escapes from inside the without-interrupts in:

(without-interrupts
  ;; The body of the lambda would be executed with WITH-INTERRUPTS allowed
  ;; regardless of the interrupt policy in effect when it is called.
  (lambda () (allow-with-interrupts ...)))

(without-interrupts
  ;; The body of the lambda would be executed with interrupts enabled
  ;; regardless of the interrupt policy in effect when it is called.
  (lambda () (with-local-interrupts ...)))
Macro: mp:with-interrupts &body body

Executes body with deferrable interrupts conditionally enabled. If there are pending interrupts they take effect prior to executing body.

As interrupts are normally allowed with-interrupts only makes sense if there is an outer without-interrupts with a corresponding allow-with-interrupts: interrupts are not enabled if any outer without-interrupts is not accompanied by allow-with-interrupts.


Next: , Previous: , Up: Native threads   [Contents][Index]

4.4.4 Locks (mutexes)

Locks are used to synchronize access to the shared data. Lock may be owned only by a single thread at any given time. Recursive locks may be re-acquired by the same thread multiple times (and non-recursive locks can’t).


Next: , Previous: , Up: Native threads   [Contents][Index]

4.4.5 Locks dictionary

Function: ecl_make_lock (cl_object name, bool recursive)

C/C++ equivalent of mp:make-lock without key arguments.

See mp:make-lock.

Function: mp:make-lock &key name (recursive nil)

Creates a lock name. If recursive isn’t nil, then the created lock is recursive.

Function: cl_object mp_recursive_lock_p (cl_object lock)
Function: mp:recursive-lock-p lock

Predicate verifying if lock is recursive.

Function: cl_object mp_holding_lock_p (cl_object lock)
Function: mp:holding-lock-p lock

Predicate verifying if the current thread holds lock.

Function: cl_object mp_lock_name (cl_object lock)
Function: mp:lock_name lock

Returns lock name.

Function: cl_object mp_lock_owner (cl_object lock)
Function: mp:lock_owner lock

Returns process owning lock (or nil if it is free). For testing whether the current thread is holding a lock see mp:holding-lock-p.

Function: cl_object mp_lock_count (cl_object lock)
Function: mp:lock-count lock

Returns number of processes waiting for lock.

Function: cl_object mp_get_lock_wait (cl_object lock)

Grabs a lock (blocking if lock is already takene). Returns ECL_T.

Function: cl_object mp_get_lock_nowait

Grabs a lock if free (non-blocking). If lock is already taken returns ECL_NIL, otherwise ECL_T.

Function: mp:get-lock lock &optional (wait t)

Tries to acquire a lock. wait indicates whenever function should block or give up if lock is already taken. If wait is nil and lock can’t be acquired returns nil. Succesful operation returns t.

Function: cl_object mp_giveup_lock (cl_object lock)
Function: mp:giveup-lock

Releases lock.

Macro: mp:with-lock (lock-form) &body body

Acquire lock for the dynamic scope of body, which is executed with the lock held by current thread, and with-lock returns the values of body.


Next: , Previous: , Up: Native threads   [Contents][Index]

4.4.6 Readers-writer locks

Readers-writer (or shared-exclusive ) lock allows concurrent access for read-only operations and write operations require exclusive access. mp:rwlock is non-recursive.


Next: , Previous: , Up: Native threads   [Contents][Index]

4.4.7 Read-Write locks dictionary

Function: ecl_make_rwlock (cl_object name)

C/C++ equivalent of mp:make-rwlock without key arguments.

See mp:make-rwlock.

Function: mp:make_rwlock &key name

Creates a rwlock with name.

Function: cl_object mp_rwlock_name (cl_object lock)
Function: mp:rwlock_name lock

Returns lock name.

Function: cl_object mp_get_rwlock_read_wait (cl_object lock)

Acquires lock (blocks if lock is already taken with mp:get-rwlock-write. Lock may be acquired by multiple readers). Returns ECL_T.

Function: cl_object mp_get_rwlock_read_nowait

Tries to acquire lock. if lock is already taken with mp:get-rwlock-write returns ECL_NIL, otherwise ECL_T.

Function: mp:get-rwlock-read lock &optional (wait t)

Tries to acquire lock. wait indicates whenever function should block or give up if lock is already taken with mp:get-rwlock-write.

Function: cl_object mp_get_rwlock_write_wait (cl_object lock)

Acquires lock (blocks if lock is already taken). Returns ECL_T.

Function: cl_object mp_get_rwlock_write_nowait

Tries to acquire lock. If lock is already taken returns ECL_NIL, otherwise ECL_T.

Function: mp:get-rwlock-write lock &optional (wait t)

Tries to acquire lock. wait indicates whenever function should block or give up if lock is already taken.

Function: cl_object mp_giveup_rwlock_read (cl_object lock)
Function: cl_object mp_giveup_rwlock_write (cl_object lock)
Function: mp:giveup_rwlock_read lock
Function: mp:giveup_rwlock_write lock

Release lock.

Macro: mp:with-rwlock (lock op) &body body

Acquire rwlock for the dynamic scope of body for operation op, which is executed with the lock held by current thread, and with-rwlock returns the values of body.

Valid values of argument op are :read or :write (for reader and writer access accordingly).


Next: , Previous: , Up: Native threads   [Contents][Index]

4.4.8 Condition variables

Condition variables are used to wait for a particular condition becoming true (e.g new client connects to the server).


Next: , Previous: , Up: Native threads   [Contents][Index]

4.4.9 Condition variables dictionary

Function: cl_object mp_make_condition_variable ()
Function: mp:make-condition-variable

Create a condition variable.

Function: cl_object mp_condition_variable_wait (cl_object cv, cl_object lock)
Function: mp:condition-variable-wait cv lock

Release lock and suspend thread until condition mp:condition-variable-signal is called on cv. When thread resumes re-aquire lock.

Function: cl_object mp_condition_variable_timedwait (cl_object cv, cl_object lock, cl_object seconds)
Function: mp:condition-variable-timedwait cv lock seconds

mp:condition-variable-wait which timeouts after seconds seconds.

Function: cl_object mp_condition_variable_signal (cl_object cv)
Function: mp:condition-variable-signal cv

Signal cv (wakes up only one waiter). After signal, signaling thread keeps lock, waking thread goes on the queue waiting for the lock.

See mp:condition-variable-wait.

Function: cl_object mp_condition_variable_broadcast (cl_object cv)
Function: mp:condition-variable-broadcast cv

Signal cv (wakes up all waiters).

See mp:condition-variable-wait and mp:condition-variable-signal.


Next: , Previous: , Up: Native threads   [Contents][Index]

4.4.10 Semaphores

Semaphores are objects which allow an arbitrary resource count. Semaphores are used for shared access to resources where number of concurrent threads allowed to access it is limited.


Previous: , Up: Native threads   [Contents][Index]

4.4.11 Semaphores dictionary

Function: cl_object ecl_make_semaphore (cl_object name, cl_fixnum count)

C/C++ equivalent of mp:make-sempahore without key arguments.

See mp:make-sempahore.

Function: mp:make-semaphore &key name count

Creates a counting semaphore name with a resource count count.

Function: cl_object mp_semaphore_name (cl_object semaphore)
Function: mp:semaphore-name semaphore

Returns semaphore name.

Function: cl_object mp_semaphore_count (cl_object semaphore)
Function: mp:semaphore-count semaphore

Returns semaphore count of resources.

Function: cl_object mp_semaphore_wait_count (cl_object semaphore)
Function: mp:semaphore-wait-count semaphore

Returns number of threads waiting on semaphore.

Function: cl_object mp_wait_on_semaphore (cl_object semaphore)
Function: mp:wait-on-semaphore semaphore

Waits on semaphore until it can grab the resource (blocking). Returns resource count before semaphore was acquired.

Function: cl_object mp_try_get_semaphore (cl_object semaphore)
Function: mp:try_get_semaphore semaphore

Tries to get a semaphore (non-blocking). If there is no resource left returns NIL, otherwise returns resource count before semaphore was acquired.

Function: cl_object mp_signal_semaphore (cl_narg n, cl_object sem, ...);
Function: mp:signal-semaphore semaphore &optional (count 1)

Releases count units of a resource on semaphore.


Next: , Previous: , Up: Extensions   [Contents][Index]

4.5 Signals and Interrupts


Next: , Up: Signals and Interrupts   [Contents][Index]

4.5.1 Problems associated to signals

POSIX contemplates the notion of "signals", which are events that cause a process or a thread to be interrupted. Windows uses the term "exception", which includes also a more general kind of errors.

In both cases the consequence is that a thread or process may be interrupted at any time, either by causes which are intrinsic to them (synchronous signals), such as floating point exceptions, or extrinsic (asynchronous signals), such as the process being aborted by the user.

Of course, those interruptions are not always welcome. When the interrupt is delivered and a handler is invoked, the thread or even the whole program may be in an inconsistent state. For instance the thread may have acquired a lock, or it may be in the process of filling the fields of a structure. Furthermore, sometimes the signal that a process receives may not even be related to it, as in the case when a user presses Cltr-C and a SIGINT signal is delivered to an arbitrary thread, or when the process receives the Windows exception CTRL_CLOSE_EVENT denoting that the terminal window is being closed.

Understanding this, POSIX restricts severely what functions can be called from a signal handler, thereby limiting its usefulness. However, Common Lisp users expect to be able to handle floating point exceptions and to gracefully manage user interrupts, program exits, etc. In an attempt to solve this seemingly impossible problem, ECL has taken a pragmatic approach that works, it is rather safe, but involves some work on the ECL maintainers and also on users that want to embed ECL as a library.


Next: , Previous: , Up: Signals and Interrupts   [Contents][Index]

4.5.2 Kinds of signals


Next: , Up: Signals and Interrupts - Kinds of signals   [Contents][Index]

4.5.2.1 Synchronous signals

The name derives from POSIX and it denotes interrupts that occur due to the code that a particular thread executes. They are largely equivalent to C++ and Java exceptions, and in Windows they are called "unchecked exceptions."

Common Lisp programs may generate mostly three kinds of synchronous signals:

The first family of signals are generated by the floating point processing hardware in the computer, and they typically happen when code is compiled with low security settings, performing mathematical operations without checks.

The second family of signals may seem rare, but unfortunately they still happen quite often. One scenario is wrong code that handles memory directly via FFI. Another one is undetected stack overflows, which typically result in access to protected memory regions. Finally, a very common cause of these kind of exceptions is invoking a function that has been compiled with very low security settings with arguments that are not of the expected type – for instance, passing a float when a structure is expected.

The third family is related to the multiprocessing capabilities in Common Lisp systems and more precisely to the mp:interrupt-process function which is used to kill, interrupt and inspect arbitrary threads. In POSIX systems ECL informs a given thread about the need to interrupt its execution by sending a particular signal from the set which is available to the user.

Note that in neither of these cases we should let the signal pass unnoticed. Access violations and floating point exceptions may propagate through the program causing more harm than expected, and without process interrupts we will not be able to stop and cancel different threads. The only question that remains, though, is whether such signals can be handled by the thread in which they were generated and how.


Previous: , Up: Signals and Interrupts - Kinds of signals   [Contents][Index]

4.5.2.2 Asynchronous signals

In addition to the set of synchronous signals or "exceptions", we have a set of signals that denote "events", things that happen while the program is being executed, and "requests". Some typical examples are:

The important difference with synchronous signals is that we have no thread that causes the interrupt and thus there is no preferred way of handling them. Moreover, the operating system will typically dispatch these signals to an arbitrary thread, unless we set up mechanisms to prevent it. This can have nasty consequences if the incoming signal interrupt a system call, or leaves the interrupted thread in an inconsistent state.


Next: , Previous: , Up: Signals and Interrupts   [Contents][Index]

4.5.3 Signals and interrupts in ECL

The signal handling facilities in ECL are constrained by two needs. First of all, we can not ignore the synchronous signals mentioned in Signals and Interrupts - Synchronous signals. Second, all other signals should cause the least harm to the running threads. Third, when a signal is handled synchronously using a signal handler, the handler should do almost nothing unless we are completely sure that we are in an interruptible region, that is outside system calls, in code that ECL knows and controls.

The way in which this is solved is based on the existence of both synchronous and asynchronous signal handling code, as explained in the following two sections.


Next: , Up: Signals and Interrupts - Signals and interrupts in ECL   [Contents][Index]

4.5.3.1 Handling of asynchronous signals

In systems in which this is possible, ECL creates a signal handling thread to detect and process asynchronous signals (See Signals and Interrupts - Asynchronous signals). This thread is a trivial one and does not process the signals itself: it communicates with, or launches new signal handling threads to act accordingly to the denoted events.

The use of a separate thread has some nice consequences. The first one is that those signals will not interrupt any sensitive code. The second one is that the signal handling thread will be able to execute arbitrary lisp or C code, since it is not being executed in a sensitive context. Most important, this style of signal handling is the recommended one by the POSIX standards, and it is the one that Windows uses.

The installation of the signal handling thread is dictated by a boot time option, ECL_OPT_SIGNAL_HANDLING_THREAD, and it will only be possible in systems that support either POSIX or Windows threads.

Systems which embed ECL as an extension language may wish to deactivate the signal handling thread using the previously mentioned option. If this is the case, then they should take appropriate measures to avoid interrupting the code in ECL when such signals are delivered.

Systems which embed ECL and do not mind having a separate signal handling thread can control the set of asynchronous signals which is handled by this thread. This is done again using the appropriate boot options such as ECL_OPT_TRAP_SIGINT, ECL_OPT_TRAP_SIGTERM, etc. Note that in order to detect and handle those signals, ECL must block them from delivery to any other thread. This means changing the sigprocmask() in POSIX systems or setting up a custom SetConsoleCtrlHandler() in Windows.


Previous: , Up: Signals and Interrupts - Signals and interrupts in ECL   [Contents][Index]

4.5.3.2 Handling of synchronous signals

We have already mentioned that certain synchronous signals and exceptions can not be ignored and yet the corresponding signal handlers are not able to execute arbitrary code. To solve this seemingly impossible contradiction, ECL uses a simple solution, which is to mark the sections of code which are interruptible, and in which it is safe for the handler to run arbitrary code. All other regions would be considered "unsafe" and would be protected from signals and exceptions.

In principle this "marking" of safe areas can be done using POSIX functions such as pthread_sigmask() or sigprocmask(). However in practice this is slow, as it involves at least a function call, resolving thread-local variables, etc, etc, and it will not work in Windows.

Furthermore, sometimes we want signals to be detected but not to be immediately processed. For instance, when reading from the terminal we want to be able to interrupt the process, but we can not execute the code from the handler, since the C function which is used to read from the terminal, read(), may have left the input stream in an inconsistent, or even locked state.

The approach in ECL is more lightweight: we install our own signal handler and use a thread-local variable as a flag that determines whether the thread is executing interrupt safe code or not. More precisely, if the variable ecl_process_env()->disable_interrupts is set, signals and exceptions will be postponed and then the information about the signal is queued. Otherwise the appropriate code is executed: for instance invoking the debugger, jumping to a condition handler, quitting, etc.

Systems that embed ECL may wish to deactivate completely these signal handlers. This is done using the boot options, ECL_OPT_TRAP_SIGFPE, ECL_OPT_TRAP_SIGSEGV, ECL_OPT_TRAP_SIGBUS, ECL_OPT_TRAP_INTERRUPT_SIGNAL.

Systems that embed ECL and want to allow handling of synchronous signals should take care to also trap the associated lisp conditions that may arise. This is automatically taken care of by functions such as si_safe_eval(), and in all other cases it can be solved by enclosing the unsafe code in a CL_CATCH_ALL_BEGIN frame.


Next: , Previous: , Up: Signals and Interrupts   [Contents][Index]

4.5.4 Considerations when embedding ECL

There are several approaches when handling signals and interrupts in a program that uses ECL. One is to install your own signal handlers. This is perfectly fine, but you should respect the same restrictions as ECL. Namely, you may not execute arbitrary code from those signal handlers, and in particular it will not always be safe to execute Common Lisp code from there.

If you want to use your own signal handlers then you should set the appropriate options before invoking cl_boot(), as explained in ecl_set_option. Note that in this case ECL will not always be able to detect floating point exceptions, specially if your compiler does not support C99 and the corresponding floating point flags.

The other option is to let ECL handle signals itself. This would be safer when the dominant part of the code is Common Lisp, but you may need to protect the code that embeds ECL from being interrupted using either the macros ecl_disable_interrupts and ecl_enable_interrupts or the POSIX functions pthread_sigmaksand sigprocmask.


Previous: , Up: Signals and Interrupts   [Contents][Index]

4.5.5 Signals Reference

Macro: ext:with-interrupts &body body

Execute code with interrupts optionally enabled.

Description

Executes the given body with all interrupts enabled. Since interrupts are normally enabled, this macro only makes sense if there is an outer ext:without-interrupts with a corresponding ext:allow-with-interrupts: interrupts are not enabled if any outer mp:without-interrupts is not accompanied by mp:allow-with-interrupts.

Macro: ext:without-interrupts &body body

Execute code without being interrupted.

Description

Executes the given body with all interrupts disabled. This macro is compatible with the one in SBCL and as such it also defines three other local macros ext:allow-with-interrupts, ext:with-local-interrupts and ext:with-restored-interrupts.

Deferrable interrupts include most blockable POSIX signals, and mp:interrupt-process. Does not interfere with garbage collection, and does not inhibit scheduling of other threads.

This macro binds allow-with-interrupts, with-local-interrupts and with-restored-interrupts as a local macros.

ext:with-restored-interrupts executes the body with interrupts enabled if and only if the ext:without-interrupts was in an environment in which interrupts were allowed.

ext:allow-with-interrupts allows the ext:with-interrupts to take effect during the dynamic scope of its body, unless there is an outer ext:without-interrupts without a corresponding ext:allow-with-interrupts.

ext:with-local-interrupts executes its body with interrupts enabled provided that for there is an ext:allow-with-interrupts for every ext:without-interrupts surrounding the current one. ext:with-local-interrupts is equivalent to:

  (allow-with-interrupts (with-interrupts ...))

Care must be taken not to let either ext:allow-with-interrupts or ext:with-local-interrupts appear in a function that escapes from inside the ext:without-interrupts in:

  (without-interrupts
    ;; The body of the lambda would be executed with WITH-INTERRUPTS allowed
    ;; regardless of the interrupt policy in effect when it is called.
    (lambda () (allow-with-interrupts ...)))

  (without-interrupts
    ;; The body of the lambda would be executed with interrupts enabled
    ;; regardless of the interrupt policy in effect when it is called.
    (lambda () (with-local-interrupts ...)))

Next: , Previous: , Up: Extensions   [Contents][Index]

4.6 Memory Management


Next: , Up: Memory Management   [Contents][Index]

4.6.1 Introduction

ECL relies on the Boehm-Weiser garbage collector for handling memory, creating and destroying objects, and handling finalization of objects that are no longer reachable. The use of a garbage collector, and in particular the use of a portable one, imposes certain restrictions that may appear odd for C/C++ programmers.

In this section we will discuss garbage collection, how ECL configures and uses the memory management library, what users may expect, how to handle the memory and how to control the process by which objects are deleted.


Next: , Previous: , Up: Memory Management   [Contents][Index]

4.6.2 Boehm-Weiser garbage collector

First of all, the garbage collector must be able to determine which objects are alive and which are not. In other words, the collector must able to find all references to an object. One possiblity would be to know where all variables of a program reside, and where is the stack of the program and its size, and parse all data there, discriminating references to lisp objects. To do this precisely one would need a very precise control of the data and stack segments, as well as how objects are laid out by the C compiler. This is beyond ECL’s scope and wishes and it can make coexistence with other libraries (C++, Fortran, etc) difficult.

The Boehm-Weiser garbage collector, on the other hand, is a conservative garbage collector. When scanning memory looking for references to live data, it guesses, conservatively, whether a word is a pointer or not. In case of doubt it will consider it to be a pointer and add it to the list of live objects. This may cause certain objects to be retained longer than what an user might expect but, in our experience, this is the best of both worlds and ECL uses certain strategies to minimize the amount of misinterpreted data.

More precisely, ECL uses the garbage collector with the following settings:

Except for finalization, which is a questionable feature, the previous settings are not very relevant for Common Lisp programmers, but are crucial for people interested in embedding in or cooperating with other C, C++ or Fortran libraries. Care should be taken when manipulating directly the GC library to avoid interfering with ECL’s expectations.


Next: , Previous: , Up: Memory Management   [Contents][Index]

4.6.3 Memory limits

Beginning with version 9.2.1, ECL operates a tighter control of the resources it uses. In particular, it features explicit limits in the four stacks and in the amount of live data. These limits are optional, can be changed at run time, but they allow users to better control the evolution of a program, handling memory and stack overflow gracefully via the Common Lisp condition system.

The customizable limits are listed in Table 4.1, but they need a careful description.

If you look at Table 4.1, some of these limits may seem very stringent, but they exist to allow detecting and correcting both stack and memory overflow conditions. Larger values can be set systematically either in the ~/.eclrc initialization file, or using the command line options from the table.


Next: , Previous: , Up: Memory Management   [Contents][Index]

4.6.4 Memory conditions

When ECL surpasses or approaches the memory limits it will signal a Common Lisp condition. There are two types of conditions, ext:stack-overflow and ext:storage-exhausted, for stack and heap overflows, respectively. Both errors are correctable, as the following session shows:

> (defun foo (x) (foo x))

FOO
> (foo 1)
C-STACK overflow at size 1654784. Stack can probably be resized.
Broken at SI:BYTECODES.Available restarts:
1. (CONTINUE) Extend stack size
Broken at FOO.
>> :r1
C-STACK overflow at size 2514944. Stack can probably be resized.
Broken at SI:BYTECODES.Available restarts:
1. (CONTINUE) Extend stack size
Broken at FOO.
>> :q
Top level.

Next: , Previous: , Up: Memory Management   [Contents][Index]

4.6.5 Finalization

As we all know, Common-Lisp relies on garbage collection for deleting unreachable objects. However, it makes no provision for the equivalent of a C++ Destructor function that should be called when the object is eliminated by the garbage collector. The equivalent of such methods in a garbage collected environment is normally called a finalizer.

ECL includes a simple implementation of finalizers which makes the following promises.

The implementation is based on two functions, ext:set-finalizer and ext:get-finalizer, which allow setting and querying the finalizer functions for certain objects.


Previous: , Up: Memory Management   [Contents][Index]

4.6.6 Memory Management Reference

Reference

Condition: ext:stack-overflow

Stack overflow condition

Class Precedence List

ext:stack-overflow, storage-condition, serious-condition, condition, t

Methods

Function: ext:stack-overflow-size condition
returns

A non-negative integer.

Function: ext:stack-overflow-type condition
returns

A symbol from Table 4.1, except ext:heap-size.

Description

This condition is signaled when one of the stack limits in Table 4.1 are violated or dangerously approached. It can be handled by resetting the limits and continuing, or jumping to an outer control point.

Condition: ext:storage-exhausted

Memory overflow condition

Class Precedence List

ext:storage-exhausted, storage-condition, serious-condition, condition, t

Description

This condition is signaled when ECL exhausts the ext:heap-size limit from Table 4.1. In handling this condition ECL follows this logic:

Function: ext:get-finalizer object
object

Any lisp object.

Description

This function returns the finalizer associated to an object, or NIL.

Function: ext:get-limit concept
concept

A symbol.

Description

Queries the different memory and stack limits that condition ECL’s behavior. The value to be queried is denoted by the symbol concept, which should be one from the list: Table 4.1

Function: ext:set-finalizer object function

Associate a finalizer to an object.

object

Any lisp object.

function

A function or closure that takes one argument or NIL.

Description

If function is NIL, no finalizer is associated to the object. Otherwise function must be a function or a closure of one argument, which will be invoked before the object is destroyed.

Example

Close a file associated to an object.

(defclass my-class () ((file :initarg :file :initform nil)))

(defun finalize-my-class (x)
 (let ((s (slot-value x 'file)))
   (when s (format t "~%;;; Closing" s) (close s))))

(defmethod initialize-instance :around ((my-instance my-class) &rest args)
  (ext:set-finalizer my-instance #'finalize-my-class)
  (call-next-method))

(progn
  (make-instance 'my-class :file (open "~/.ecl.old" :direction :input))
  nil)

(si::gc t)
(si::gc t)

;; Closing
Function: ext:set-limit concept value

Set a memory or stack limit.

concept

A symbol.

value

A positive integer.

Changes the different memory and stack limits that condition ECL’s behavior. The value to be changed is denoted by the symbol concept, while the value is the new maximum size. The valid symbols and units are listed in Table 4.1.

Note that the limit has to be positive, but it may be smaller than the previous value of the limit. However, if the supplied value is smaller than what ECL is using at the moment, the new value will be silently ignored.

ConceptUnitsDefaultCommand line
ext:frame-stackNested frames2048--frame-stack
ext:binding-stackBindings8192
ext:c-stackBytes128 kilobytes--c-stack
ext:heap-sizeBytes256 megabytes--heap-size
ext:lisp-stackBytes32 kilobyes--lisp-stack

Table 4.1: Customizable memory limits


Next: , Previous: , Up: Extensions   [Contents][Index]

4.7 Meta-Object Protocol (MOP)


Up: Meta-Object Protocol (MOP)   [Contents][Index]

4.7.1 Introduction

The Meta-Object Protocol is an extension to Common Lisp which provides rules, functions and a type structure to handle the object system. It is a reflective system, where classes are also objects and can be created and manipulated using very well defined procedures.

The Meta-Object Protocol associated to Common Lisp’s object system was introduced in a famous book, The Art of the Metaobject Protocol AMOP (See Bibliography), which was probably intended for the ANSI (See Bibliography) specification but was drop out because of its revolutionary and then not too well tested ideas.

The AMOP is present, in one way or another, in most Common Lisp implementations, eithr using proprietary systems or because their implementation of CLOS descended from PCL (Portable CommonLoops). It has thus become a de facto standard and ECL should not be without it.

Unfortunately ECL’s own implemention originally contained only a subset of the AMOP. This was a clever decision at the time, since the focus was on performance and on producing a stable and lean implementation of Common Lisp. Nowadays it is however not an option, specially given that most of the AMOP can be implemented with little cost for both the implementor and the user.

So ECL has an almost complete implementation of the AMOP. However, since it was written from scratch and progressed according to user’s request and our own innovations, there might still be some missing functionality which we expect to correct in the near future. Please report any feature you miss as a bug through the appropriate channels.

When considering the Metaobject Protocol, the book itself should be the main reference. The following sections contain only further extensions or improvements over the paragraphs which were either conflicting or less specified.


Next: , Previous: , Up: Extensions   [Contents][Index]

4.8 Gray Streams


Next: , Previous: , Up: Extensions   [Contents][Index]

4.9 Tree walker


Next: , Previous: , Up: Extensions   [Contents][Index]

4.10 Local package nicknames

4.10.1 Overview

ECL allows giving packages local nicknames: they allow short and easy-to-use names to be used without fear of name conflict associated with normal nicknames.

A local nickname is valid only when inside the package for which it has been specified. Different packages can use same local nickname for different global names, or different local nickname for same global name.

Symbol :package-local-nicknames in *features* denotes the support for this feature.

4.10.2 Package local nicknames dictionary

Macro: cl:defpackage name [[options]]*

Options are extended to include

:local-nicknames (local-nickname actual-package-name)*

The package has the specified local nicknames for the corresponding actual packages.

Example:

(defpackage :bar (:intern "X"))
(defpackage :foo (:intern "X"))
(defpackage :quux (:use :cl) (:local-nicknames (:bar :foo) (:foo :bar)))
(find-symbol "X" :foo) ; => FOO::X
(find-symbol "X" :bar) ; => BAR::X
(let ((*package* (find-package :quux)))
(find-symbol "X" :foo))               ; => BAR::X
(let ((*package* (find-package :quux)))
(find-symbol "X" :bar))               ; => FOO::X
ext: package-local-nicknames package-designator
C/C++: si_package_local_nicknames package-designator

Returns an alist of (local-nickname . actual-package) describing the nicknames local to the designated package.

When in the designated package, calls to FIND-PACKAGE with the any of the local-nicknames will return the corresponding actual-package instead. This also affects all implied calls to FIND-PACKAGE, including those performed by the reader.

When printing a package prefix for a symbol with a package local nickname, the local nickname is used instead of the real name in order to preserve print-read consistency.

ext: package-locally-nicknamed-by-list package-designator
C/C++: si_package_local_nicknames package-designator

Returns a list of packages which have a local nickname for the designated package.

ext: add-package-local-nickname local-nickname actual-package &optional package-designator
C/C++: si_add_package_local_nickname local-nickname actual-package package-designator

Adds local-nickname for actual-package in the designated package, defaulting to current package. local-nickname must be a string designator, and actual-package must be a package designator.

Returns the designated package.

Signals a continuable error if local-nickname is already a package local nickname for a different package.

When in the designated package, calls to find-package with the local-nickname will return the package the designated actual-package instead. This also affects all implied calls to find-package, including those performed by the reader.

When printing a package prefix for a symbol with a package local nickname, local nickname is used instead of the real name in order to preserve print-read consistency.

ext: remove-package-local-nickname old-nickname &optional package-designator
C/C++: si_remove_package_local_nickname old-nickname package-designator

If the designated package had old-nickname as a local nickname for another package, it is removed. Returns true if the nickname existed and was removed, and NIL otherwise.


Next: , Previous: , Up: Extensions   [Contents][Index]

4.11 Package locks

4.11.1 Package Locking Overview

ECL borrows parts of the protocol and documentation from SBCL for compatibility. Interface is the same except that the home package for locking is ext and that ECL doesn’t implement Implementation Packages and a few constructs. To load the extension you need to require package-locks:

(require '#:package-locks)

Package locks protect against unintentional modifications of a package: they provide similar protection to user packages as is mandated to common-lisp package by the ANSI specification. They are not, and should not be used as, a security measure.

Newly created packages are by default unlocked (see the :lock option to defpackage).

The package common-lisp and ECL internal implementation packages are locked by default, including ext.

It may be beneficial to lock common-lisp-user as well, to ensure that various libraries don’t pollute it without asking, but this is not currently done by default.

4.11.2 Operations Violating Package Locks

The following actions cause a package lock violation if the package operated on is locked, and *package* is not an implementation package of that package, and the action would cause a change in the state of the package (so e.g. exporting already external symbols is never a violation). Package lock violations caused by these operations signal errors of type package-error.

  1. Shadowing a symbol in a package.
  2. Importing a symbol to a package.
  3. Uninterning a symbol from a package.
  4. Exporting a symbol from a package.
  5. Unexporting a symbol from a package.
  6. Changing the packages used by a package.
  7. Renaming a package.
  8. Deleting a package.
  9. Attempting to redefine a function in a locked package.
  10. Adding a new package local nickname to a package.
  11. Removing an existing package local nickname to a package.

4.11.3 Package Lock Dictionary

Function: ext:package-locked-p package

Returns t when package is locked, nil otherwise. Signals an error if package doesn’t designate a valid package.

Function: ext:lock-package package

Locks package and returns t. Has no effect if package was already locked. Signals an error if package is not a valid package designator

Function: ext:unlock-package package

Unlocks package and returns t. Has no effect if package was already unlocked. Signals an error if package is not a valid package designator.

Macro: ext:without-package-locks &body body

Ignores all runtime package lock violations during the execution of body. Body can begin with declarations.

Macro: ext:with-unlocked-packages (&rest packages) &body body

Unlocks packages for the dynamic scope of the body. Signals an error if any of packages is not a valid package designator.

Macro: cl:defpackage name [[option]]* ⇒ package

Options are extended to include the following:

Example:

(defpackage "FOO" (:export "BAR") (:lock t))

;;; is equivalent to

(defpackage "FOO") (:export "BAR"))
(lock-package "FOO")

Previous: , Up: Extensions   [Contents][Index]

4.12 CDR Extensions


Footnotes

(1)

You may also link ECL runtime statically. That is not covered in this walkthrough.


Previous: , Up: Extensions   [Contents][Index]