ECL is a fantastic ANSI Commnon Lisp implementation that aims at embedding. It allows us to build Lisp runtime into C/C++ code as external library. That means you can both call C code from Lisp runtime and call Lisp code form the C/C++ part. In this article we focus on how to call Lisp procedures from the C/C++ side, you may also achieve the reversed process by inlining C/C++ code in ECL, but that's beyond our discussion here. Generally, this one-side work is fairly enough to enable us to exhaust any power of both Lisp and C/C++.

This article shows how you can embed ECL into a Qt project, and serve as the kernel of that program. I hope that can serve as a common example, or tutorial, for the one who want to know about the true power of ECL, and perhaps that awesome guy is you. At least we will show you that the ability to hybrid is what makes ECL different from other implementations. And if one day you need that power, you know where you gonna head for.

I know I should shoot the code and demo quickly, but let the theory come first, just for those who don't understand why we are doing this.

What we are doing is just an instance of mixed-language programming. That means you will be writing codes in different languages and glue them together to combine a program. Usually, the product program would appear to be a peanut, the kernel is written in language A and the nutshell written in language B. The most frequent combination is C++ and Lua, in game programming. This enables us to take advantage of both language A and B. But in this case we are hybridizing C++ and Common Lisp, both aiming at building large software system, an we are sure to gain even more benifits here.

Common Lisp is a masterpiece, but it lacks a good GUI toolkit. Well, if you work on Linux you definitely can take a shot on packages like CL-GTK or CommonQt, and someone will suggest you to use another package called EQL, the Qt4 binding for ECL. But any of these GUI tools look coarse on my Mac, with Retina screen. I don't want to spend my scholarships on Lispworks, so I must find another way to enable Lisp in GUI-related programming. Finally I ended up in here. The hybrid is no compromise, since I can develop fancy GUI interface without losing the power of Lisp.

We can see more benifits that are shown below:

1. Live hotpatching. Common Lisp is a dynamic language, so it allows you to add new components at runtime. That means you can upgrade your program without re-compiling or even restart. In more advanced discussions, you may even recompile your C/C++ functions. So embedded ECL could even change the host language.
2. With the most powerful macro system, Lisp is natively designed for complex system designing. Different from light-weight languages like Python and Lua, Lisp is suitable for huge programs. So you can always focus on Lisp, instead of seeking for help from language B. You can also use Lisp environment as a direct DSL interpreter so that you needn't write one yourself.
3. Efficiency. There are other approaches to combine Lisp runtime with other languages, like using pipes, channels, sockets or temporary files. These are never elegant solutions, since you can only read the output by Lisp runtime manually and there's no direct access to the Lisp runtime memory. And you have to string-parse each Lisp output. So this is neither quick nor efficient. You may also use FFIs (Foreign Language Interfaces) and it's more common with the reverse direction, say call C from Lisp. Now the ECL approach is to save. The Lisp runtime in ECL shares the same part of memory with C/C++ side and there's direct way to fetch the return value of any Lisp function calls. How can they achieve this magic? Well ECL compiles Lisp to C code or bytecode so that they get the same tone.
4. Stable and mature. ECL is currently the best for embedding. You may have heard of other implementations like Clasp, which works on LLVM and is compatible with C++. But it's not yet stable or ANSI-compatible hitherto. Meanwhile ECL has a long history and is already ready to use. When built with C++ compiler like g++ (flag –with-cxx), ECL also enables us to Call C++ functions. So stick to ECL.

I hope this should convince you that this could be a promising hybrid.

The embedding process of ECL can be pretty simple if you understand how it works. Unfortunately the ECL official documentation of this part is not quite clear at the moment, here are some example code in the example/embed directory. Thanks to Daniel Kochmański, he helped me through the way towards my first success of hybridding. I'm still a newbie here.

The example code is enough for understanding the process of hybridizing Lisp and C codes by ECL. There is absolutely a general approach and you can use it as a pattern in your development.

ECL achieves this by compiling itself into C libraries and link it to C runtime. There is two ways to go: static library and shared library. In this article we will take the first approach. For embedding, there are a few steps:

1. Write your Lisp files. (absolutely)
2. Compile your Lisp files.
3. Write a C/C++ file that includes the ECL library and boots the CL environment.
4. Link the runtime and compile the whole project into executables.

Easy enough, isn't it? Let me explain in detail.

The first step is nothing different than general Lisp development. You can either create your own package or not. (Just leave the naked lisp file.)

The second step, well, it's time for ECL to rock. We've got two approaches, which depend on whenever you use ASDF or not. If you do not want to use it, you may follow this code:

(compile-file "<YOUR-LISP-FILE>.lisp" :system-p t)
(c:build-static-library "<LIBRARY-NAME>"
                        :lisp-files '("<THE-OBJECT-FILE>.o")
                        :init-name "<INIT-NAME>")

The first line of code compiles your .lisp file into a .o object file, say, <THE-OBJECT-FILE>.o. This file serves as the input for the next procedure. The c:build-static-library function is defined by ECL, it builds the object file into a static library, say, <LIBRARY-NAME>.a. We should pay attention to the init-name. You can define your init-name here as a string, and this is useful for step 3. We will head back when it happens.

If you choose to use ASDF, you can head for the asdf:make-build function. This can be seen in the Makefile in example:

hello.exe: hello.c hello-lisp.a
        $(CC) `ecl-config --cflags` -o $@ hello.c hello-lisp.a \
              `ecl-config --ldflags` -lecl

hello-lisp.a: hello-lisp.lisp
        ecl -norc \
            -eval '(require :asdf)' \
            -eval '(push "./" asdf:*central-registry*)' \
            -eval '(asdf:make-build :hello-lisp :type :static-library :move-here "./")' \
            -eval '(quit)'

clean:
        -rm -f hello-lisp.a hello.exe

And you may use asdf:defsystem in your lisp code. We will see this closer in my demo.

In the third step, we must dance with some C/C++ code. In your .c file where you want the ECL environment to run, you should #include <ecl/ecl.h> to make sure all the ECL symbols are linked. Then write some simple code to boot the environment:

/* Initialize ECL */
cl_boot(argc, argv);

/* Initialize the library we linked in. Each library
 * has to be initialized. It is best if all libraries
 * are joined using ASDF:MAKE-BUILD.
 */
extern void init_lib_HELLO_LISP(cl_object);
ecl_init_module(NULL, init_lib_HELLO_LISP);

The cl_ boot procedure boots the CL environment, it takes the right args from your main entry. Now take a look at the extern declaration. Remember last time I suggest you to notice the :init-name, now it's time to use it. If you took the first approach of building library and defined your own *init-name*, now the function name should be the same with it. And if you didn't define your init-name, now the name convention should be: init_lib_<FILE_NAME_OF_THE_LIBRARY>. Say, if the static library named "hello-world–all-systems.a", then you write init_lib_HELLO_WORLD__ALL_SYSTEMS for the function name.

Notice: In C++, you should encapsule the extern code in an extern "C" block:

extern "C"{
extern void init_lib_HELLO_LISP(cl_object);
}

To make the link process complete. This has something to do with the naming convention that differs from C to C++. In general ECL exports symbols following C naming convention to allow seamless FFI to it from C and other languages. C++ does some weird name mangling.So if you want to call C functions from C++, you have to declare them in C++ that way indeed. The function is used by procedure ecl_init_module to load all of your user-defined Lisp symbols. Then you are freely to call your Lisp code in C/C++.

The forth step builds the whole project. So it acquires all of your C/C++ files, libraries and the ECL library. All of the work can be easily done if you are familiar with Makefile. See the example above.

"How can I call the Lisp functions I wrote?" This should be the most urgent question you may ask. The ECL manual describes most of the functions in the Standards chapter. Apparently most of the Lisp functions, or macros have been maped into C functions, with some name convention. For example the [[https://common-lisp.net/project/ecl/static/manual/re02.html][cl_ eval]] means the corresponding Lisp code "eval". Most of the ANSI-defined procedure has the naming convention of using cl_ as prefix. So you can easily find the primitive symbol you need.

But perhaps the problem you most concern is:

1. How can I call MY Lisp functions in C/C++?
2. How can I translate the return value into C/C++ objects?

For the first question I suggest you to use the cl_eval function. The reason is it's simple and extensible. For the safety reasons you may use cl_funcall or cl_safe_eval. But none of them is as universal as cl_eval. The cl_funcall, as its name means, can only call functions and cannot be used to call macros. And cl_safe_eval requires more parameters in order to handle potential errors that may occur on the Lisp side. But here since I don't mean to make my code productive so I won't care about the safety or convenience. I wrote a friendlier version of cl_eval and you can call lisp code like this:

cl_eval("mapcar", list_foo, "(lambda (x) (princ x))");

And that's nearly Lisp code in appearance.

So let's head for the cl_eval. Its signature is:

cl_object cl_eval(cl_object);

It receives a cl_object and returns a cl_object. Hmm. Now you should get knowledge of how ECL manipulate Common Lisp Objects before wondering what cl_object is.

Quite simple. ECL encapsules any Lisp object into the same structure cl_object. It's a C union whose definition can be seen in object.h, line 1011. So you don't need to worry about using different types to capture the return value.

Translating C strings to cl_object is trivial: use the c_string_to_object function:

cl_object c_string_to_object (const char * s)

You just write the Lisp form in C string and the function will create Lisp object for you. So you may write

cl_eval(c_string_to_object("(princ \"hello world\")"));

To get your first hybrid-call.

The second question can be a little tough due to lack of documentation. And there's another naming convention.

Generally, you may use the ecl_to_* family to convert the cl_object to primitive C data, here is some regular examples:

char ecl_to_char(cl_object x);
int ecl_to_int(cl_object x);
double ecl_to_double(cl_object x);

I've said that these functions could only help convert cl_object to primitive C data. No array, and no string. The ECL API didn't provide them officially. So we have to implement them manually, sorry to say that. (If I missed something, correct me.)

I would show two trivial yet useful functions that may help you. The first one helps you to traverse Lisp List:

auto cl_list_traverse=[](auto& cl_lst, auto fn){
    while(!Null(cl_lst))
    {
        fn(cl_car(cl_lst));
        cl_lst=cl_cdr(cl_lst);
    }
};

This is implemented in C++ with the convenience of C++14 standard. Can be rewritten in C like this:

void cl_list_traverse(cl_object cl_lst, void(*fn)(cl_object)){
    while(!Null(cl_lst))
    {
        fn(cl_car(cl_lst));
        cl_lst=cl_cdr(cl_lst);
    }
};

Usage example:

void print_element(cl_object obj){
    printf("%d\n", ecl_to_int(obj));
}
list_traverse(foo_list, print_element);

And the second one converts the cl_object into C++ *std::string*.

std::string to_std_string(cl_object obj){
    std::string val;
    auto & str=obj->string;
    for(unsigned long i=0;i<str.fillp;i+=1)
       val+=*(typeof(str.elttype) *)(str.self+i);
    return val;
}

When you are using these functions to convert a cl_object to C/C++ objects, you have to know exactly what the return value is. That means, if you are trying to call ecl_to_int on a cl_object, you should be clear that the cl_object IS an integer. And for some complicate situation, a cl_object could contain more than one type at the same time. For example, if you call a function that returns a list of strings, say '("hello" "lisp") then the corresponding cl_object can both contain a string (on its car position) and a list (on its cdr position). Call cl_car and you get a cl_object containing a string, and you can call to_std_string on that object to get a C++ string. I mean, you should figure out that before you code. The secret is to just think you are still in Lisp.

Now it's time to head for our ultimate goal: let Lisp rock with Qt! We have had enough knowledge of embedding ECL into C++ code in the former chapters and Qt is nothing but C++. So the work should be trivial. Sounds this is true but, there's still many things to be solved. I have stuggled much about them but now I can just write down the final progress and pretend this is simple.

The first one is, how to build a Lisp package system, instead of compiling a naked Lisp file or a single package.

A common question heard from the Common Lisp newcomers is:

How to create my own application with Common Lisp?

Numerous concepts like packages, Quicklisp, modules and ASDF bring the confusion, which is only deepened by a wide range of implementations and foreign to the new programmer developing paradigm of working on a live image in the memory.

This post is a humble attempt to provide a brief tutorial on creating a small application from scratch. Our goal is to build a tool to manage document collection. Due to the introductory nature of the tutorial we will name our application "Clamber".

We will start with a quick description of what should be installed on the programmer's system (assumed operating system is Linux). Later we will create a project boilerplate with the quickproject, define a protocol for the software, write the application prototype (ineffective naive implementation), provide the command line interface with Command Line Option Nuker. This is where the first part ends.

Second part will be published on McCLIM blog and will show how to create a graphical user interface for our application with McCLIM.

Afterwards (in a third part in next ECL Quarterly) we will take a look into some considerations on how to distribute the software in various scenarios:

• Common Lisp developers perspective with Quicklisp,
• ordinary users with system-wide package managers with ql-to-deb,
• source code distribution to clients with Qucklisp Bundles,
• binary distribution (closed source) with ADSF prebuilt-system,
• as a shared library for non-CL applications with ECL.

Obviously a similar result may be achieved using different building blocks and all choices reflect my personal preference regarding the libraries I use.

Before we jump into the project creation and actual development I want to talk a little about the software distribution. We may divide our target audience in two groups – programmers and end users. Sometimes it is hard to tell the difference.

Programmers want to use our software as part of their own software as a dependency. This is a common approach in FOSS applications, where we want to focus on the problem we want to solve, not the building blocks which are freely available (what kind of freedom it is depends on the license). To make it easy to acquire such dependencies the Quicklisp project was born. It is a package manager.

End users aren't much concerned about the underlying technology. They want to use the application in the most convenient way to them. For instance average non-programming Linux user would expect to find the software with theirs system distribution package manager. Commercial client will be interested in source code with all dependencies with whom the application was tested.

Proposed solution is to use Quicklisp during the development and bundle the dependencies (also with Quicklisp) when the application is ready. After that operation our source code doesn't depend on the package manager and we have all the source code available, what simplifies further distribution.

Notion of "system" is unknown to the Common Lisp specification. It is a build-system specific concept. Most widely used build-system in 2016 is ASDF. System definition is meant to contain information essential for building the software – application name, author, license, components and dependencies. Unfortunately ADSF doesn't separate system definitions from the source code and asd format can't be considered declarative. In effect, we can't load all system definitions with certainty that unwanted side-effects will follow.

We will only outline steps which are necessary to configure the development environment. There are various tutorials on how to do that which are more descriptive.

1. Install Emacs and SBCL¹:

   These two packages should be available in your system package manager (if it has one).

2. Install Quicklisp:

   Visit https://www.quicklisp.org/beta/ and follow the instructions. It contains steps to add Quicklisp to Lisp initialization file and to install and configure SLIME. Follow all these instructions.

3. Start Emacs and run Slime:

   To run Slime issue M-x slime in Emacs window.

These steps are arbitrary. We propose Linux + SBCL + Emacs + Quicklisp + SLIME setup, but alternative configurations are possible.

Quickproject is an excellent solution for this task because it is very simple tool with a well defined goal – to simplify creating basic project structure.

The simplest way of creating a new one is loading the quickproject system with Quicklisp and calling the appropriate function. Issue the following in the REPL:

(ql:quickload 'quickproject)
(quickproject:make-project #P"~/quicklisp/local-projects/clamber/"
                           :depends-on '(#:alexandria)
                           :author "Daniel Kochmański <daniel@turtleware.eu>"
                           :license "Public Domain")

That's it. We have created a skeleton for our project. For now, we depend only on alexandria – public domain utility library. List of dependencies will grow during the development to reflect our needs. Go to the clamber directory and examine its contents.

Now we will customize the skeleton. I prefer to have one package per file, so I will squash package.lisp and clamber.lisp into one. Moreover, README.txt will be renamed to README.md, because we will use markdown format for it.

To avoid clobbering the tutorial with unnecessary code we put only interesting parts here. Complete steps are covered in the application GIT repository available here:

https://gitlab.common-lisp.net/dkochmanski/clamber

We propose to clone the repository and track the progress with the subsequent commits and this tutorial.

Here is our application Clamber informal specification:

• Application will be used to maintain a book collection,
• Each book has associated meta-information (disregarding the underlying book file format),
• Books may be organized with tags and shelves,
• Book may be only on one shelf, but may have multiple tags,
• Both CLI and GUI interfaces are a required,
• Displaying the books is not part of the requirements (we may use external programs for that).

There are various solutions which enable creation of standalone binaries. The most appealing to me is Clon: the Command-Line Options Nuker, which has a very complete documentation (end-user manual, user manual and reference manual) , well thought API and works on a wide range of implementations. Additionally, it is easy to use and covers various corner-cases in a very elegant manner.

Our initial CLI (Command Line Interface) will be quite modest:

% clamber --help
% clamber add-book foo \
  --tags a,b,c \
  --shelf "Favourites" \
  --meta author "Bar" title "Quux"  
% clamber del-book bah
% clamber list-books
% clamber list-books --help
% clamber list-books --shelf=bah --tags=drama,psycho
% clamber show-book bah

Recently I ported my Common Lisp game engine "Xelf" to the Android operating system using Embeddable Common Lisp.

Some work remains to be done before I can do a proper beta test release, but ECL Quarterly provides a good opportunity to pause and share the results thus far. This is the first part of a two-part article. The focus of Part 2 will be on performance optimization, testing, and user interface concerns.

Special thanks to Daniel Kochmański, 3-B, oGMo, and the rest of the Lisp Games crew for their inspiration and assistance.

In less than a month we went from "let's do it" to "wow, it works!" What more can you ask for?

This concludes Part 1 of my article on building Lisp games for Android with Embeddable Common Lisp. To read my running commentary and see news and test results as they are posted, you can visit the project README:

https://gitlab.com/dto/ecl-android-games-src/blob/master/README.org

More details and all scripts and configurations can be found in that repository.

Thanks for reading,

–
David O'Toole (dto@xelf.me)
11 June, 2016

Requires

jdk, ant

pushd abcl
ant
cp abcl ${LISPS_DIR}/bin/abcl-dev
popd

Requires

gcc, m4, gnumake

pushd ccl
echo '(ccl:rebuild-ccl :full t)' | ./lx86cl64 -n -Q -b

# installation script is inspired by the AUR's PKGBUILD
mkdir -p ${LISPS_DIR}/ccl-dev
cp -a compiler contrib level-* lib* lisp-kernel objc-bridge \
   tools x86-headers64 xdump lx86cl64* examples doc \
   ${LISPS_DIR}/ccl-dev

find ${LISPS_DIR}/ccl-dev -type d -name .svn -exec rm -rf '{}' +
find ${LISPS_DIR}/ccl-dev -name '*.o' -exec rm -f '{}' +
find ${LISPS_DIR}/ccl-dev -name '*.*fsl' -exec rm -f '{}' +

cat <<EOF > ${LISPS_DIR}/bin/ccl-dev
#!/bin/sh
exec ${LISPS_DIR}/ccl-dev/lx86cl64 "\$@"
EOF
chmod +x ${LISPS_DIR}/bin/ccl-dev
popd

Requires

gcc, make

Notes

don't build with ASDF (it's old and broken)

pushd clisp
./configure --prefix=${LISPS_DIR}/clisp-dev/ \
            --with-threads=POSIX_THREADS \
            build/
cd build
make && make install
ln -s ${LISPS_DIR}/clisp-dev/bin/clisp ${LISPS_DIR}/bin/clisp-dev
popd

Requires

cmucl binary, gcc, make, openmotif

Notes

it needs another CMUCL to bootstrap (release 21a)

pushd cmucl
mkdir -p prebuilt
pushd prebuilt
wget https://common-lisp.net/project/cmucl/downloads/release/21a/cmucl-21a-x86-linux.tar.bz2 \
     https://common-lisp.net/project/cmucl/downloads/release/21a/cmucl-21a-x86-linux.extra.tar.bz2
mkdir ${LISPS_DIR}/cmucl-21a
tar -xf cmucl-21a-x86-linux.tar.bz2 -C ${LISPS_DIR}/cmucl-21a/
tar -xf cmucl-21a-x86-linux.extra.tar.bz2 -C ${LISPS_DIR}/cmucl-21a/
cat <<EOF > ${LISPS_DIR}/bin/cmucl-21a
#!/bin/sh
exec ${LISPS_DIR}/cmucl-21a/bin/lisp "\$@"
EOF
chmod +x ${LISPS_DIR}/bin/cmucl-21a
# Note, that this is already a fully functional lisp now
popd
bin/build.sh -C "" -o "cmucl-21a"
bin/make-dist.sh -I ${LISPS_DIR}/cmucl-dev/ linux-4/
cat <<EOF > ${LISPS_DIR}/bin/cmucl-dev
#!/bin/sh
exec ${LISPS_DIR}/cmucl-dev/bin/lisp "\$@"
EOF
chmod +x ${LISPS_DIR}/bin/cmucl-dev
popd

Requires

gcc, make

./configure --prefix=${LISPS_DIR}/ecl-dev/
make && make install
ln -s $LISPS_DIR/ecl-dev/bin/ecl ${LISPS_DIR}/bin/ecl-dev

Requires

Conforming CL implementation, web browser, nodejs

Notes

Doesn't provide LOAD yet (no filesystem), but author confirmed that this will be implemented (virtual filesystem on the browser and the physical one on the nodejs).

mkdir ${LISPS_DIR}/jscl-dev
pushd jscl
./make.sh

# Run in the console (node-repl)
cp jscl.js repl-node.js ${LISPS_DIR}/bin/jscl-dev
cat <<EOF > ${LISPS_DIR}/bin/jscl-dev
#!/bin/sh
exec node ${LISPS_DIR}/jscl-dev/repl-node.js
EOF
chmod +x ${LISPS_DIR}/bin/jscl-dev

# Run in the web browser (optional)
cp jscl.js repl-web.js jquery.js jqconsole.min.js jscl.html style.css \
   ${LISPS_DIR}/jscl-dev/
# replace surf with your favourite browser supporting JS
cat <<EOF > ${LISPS_DIR}/bin/jscl-dev-browser
#!/bin/sh
exec surf ${LISPS_DIR}/jscl-dev/jscl.html
EOF
chmod +x ${LISPS_DIR}/bin/jscl-dev-browser

popd

Requires

gcc, make

# Doesn't work both with head and the release, luckily it works with
# the next pre-release branch
git checkout Version_2_6_13pre
./configure --prefix=${LISPS_DIR}/gcl-2.6.13-pre
make && make install
ln -s ${LISPS_DIR}/gcl-2.6.13-pre/bin/gcl ${LISPS_DIR}/bin/gcl-2.6.13-pre

Requires

gcc, make

pushd mkcl
./configure --prefix=${LISPS_DIR}/mkcl-dev
make && make install
ln -s ${LISPS_DIR}/mkcl-dev/bin/mkcl ${LISPS_DIR}/bin/mkcl-dev
popd

Requires

ANSI-compliant CL implementation

Notes

• Lisp has to close on EOF in top-level (CMUCL doesn't do that),
• ECL has some bug regarding Lisp-to-C compiler apparently triggered by the SBCL compilation – don't use it here,
• we could use precompiled SBCL like with the CMUCL, but let's exploit the fact, that we can compile from the C-bootstrapped implementation (we'll use already built clisp-dev),
• it is advised to run the script in fast terminal (like xterm) or in the terminal multiplexer and to detach it – SBCL compilation process is very verbose,
• if you build SBCL on Windows, consider using MinGW to preserve POSIX compatibility.

pushd sbcl
export GNUMAKE=make
./make.sh "clisp"
INSTALL_ROOT=${LISPS_DIR}/sbcl-dev ./install.sh
cat <<EOF > ${LISPS_DIR}/bin/sbcl-dev
#!/bin/sh
SBCL_HOME=${LISPS_DIR}/sbcl-dev/lib/sbcl exec ${LISPS_DIR}/sbcl-dev/bin/sbcl "\$@"
EOF
chmod +x ${LISPS_DIR}/bin/sbcl-dev
popd

Requires

tcsh, gcc, git

Notes

very incomplete implementation

pushd wcl
REV=`git rev-parse HEAD`
sed -i -e "s/WCL_VERSION = \"3.0.*$/WCL_VERSION = \"3.0-dev (git-${REV})\"/" CONFIGURATION
LD_LIBRARY_PATH=`pwd`/lib make rebuild
mkdir ${LISPS_DIR}/wcl-dev
cp -a bin/ lib/ doc/ ${LISPS_DIR}/wcl-dev/
cat <<EOF > ${LISPS_DIR}/bin/wcl-dev
#!/bin/sh
LD_LIBRARY_PATH=${LISPS_DIR}/wcl-dev/lib exec ${LISPS_DIR}/wcl-dev/bin/wcl "\$@"
EOF
chmod +x ${LISPS_DIR}/bin/wcl-dev
popd

Requires

gcc

Notes

last commit in 2011

pushd xcl
mkdir ${LISPS_DIR}/xcl-dev
XCL_HOME=${LISPS_DIR}/xcl-dev make
cp -a clos compiler lisp COPYING README xcl ${LISPS_DIR}/xcl-dev
# This will build in XCL_HOME, even if run in source directory
./xcl <<EOF
(rebuild-lisp)
EOF

ln -s ${LISPS_DIR}/xcl-dev/xcl ${LISPS_DIR}/bin/xcl-dev
popd

Here are some great projects helping CL implementations be part of a more usable ecosystem. Many of these are considered being part of the de-facto standard:

bordeaux-threads

Provides thread primitives, locks and conditionals

cl-store

Serializing and deserializing CL objects from streams

cffi

Foreign function interface (accessing foreign libraries)

closer-mop

Meta-object protocol – provides it's own closer-common-lisp-user package (redefines for instance defmethod)

usocket

TCP/IP and UDP/IP socket interface.

osicat

Osicat is a lightweight operating system interface for Common Lisp on POSIX-like systems, including Windows

cl-fad

Portable pathname library

trivial-garbage

trivial-garbage provides a portable API to finalizers, weak hash-tables and weak pointers

trivial-features

trivial-features ensures consistent *FEATURES* across multiple Common Lisp implementations

trivial-gray-streams

trivial-gray-streams system provides an extremely thin compatibility layer for gray streams

external-program

external-program enables running programs outside the Lisp process

There are many other very good libraries which span multiple implementations. Some of them have some drawbacks though.

For instance IOlib is a great library, but piggy-backs heavily on UN*X – if you develop for many platforms you may want to consider other alternatives..

UIOP is also a very nice set of utilities, but isn't documented well, does too many things at once and tries to deprecate other actively maintained projects – that is counterproductive and socially wrong. I'd discourage using it.

There are a few arguments supporting UIOP's state – it is a direct dependency of ASDF, so it can't (or doesn't want to) depend on other libraries, but many utilities are needed by this commonly used system definition library. My reasoning here is as follows: UIOP goes beyond ASDF's requirements and tries to make actively maintained projects obsolete. Additionally it works only on supported implementations even for features which may be implemented portably.

I'm aware that my opinion regarding UIOP may be a bit controversial. I've asked the library author and a few other people for feedback which I'm very grateful for. I'm publishing it here to keep opinions balanced.

I've always had a total mental block when it comes to C pointers. It makes no sense to my brain that * indicates a variable is a pointer when used in a declaration, but retrieves a value when used as an operator. And an array of pointers to a character string makes total sense to me in words but char** str[] causes my mind to go blank. As a consequence any C code I write, or even look at too intently immediately blows up when compiled and run. A big inconvenience when embedding Lisp. Replacing the usual,

(setq inferior-lisp-program "ecl")

with,

(setq inferior-lisp-program
      "gdb --eval-command=run --eval-command=quit --args ecl")

Will run ecl under gdb, which will provide you the normal gdb environment with c runtime errors, while throwing you into the lisp debugger for Lisp errors. Note that gdb by default breaks on SIGPWR and SIGXCPU which ecl uses for internal processing. So, you'll also want to add the following to your .gdbinit file.

handle SIGPWR nostop noprint
handle SIGXCPU nostop noprint

Swank is a Lisp program that provides remote access to a Lisp instance. It started as client/server application layer in CMUCL and the Hemlock editor it ran. It's since been ported to most Lisps. Slime is the Emacs front-end client to Swank. Together the two tools provide a powerful Lisp development environment in Emacs. The easiest way to install Swank and Slime is simply to get it from quicklisp. See:

https://www.quicklisp.org/beta/

Swank and slime work in following way:

+----------+     launch ecl in                +--------------------+ 
| emacs    |---- process buffer, tell ------> | ecl process buffer |
+----------+     ecl to start swank           +-----+--------------+                         
   |                                                       |
   |                                         start swank server:
create slime                                 (swank-loader:init)
buffer                                       (swank:start-server)
   |                                                |
   |                                                |
  \/                                              \/
+--------------+      integrated      +--------------------------------+
| repl:        +<---- lisp repl   --->| swank server listening         |
| slime buffer |      interaction     | on some arbitrary              |
+--------------+                      | TCP/IP port e.g.               |
                                      | "Swank started at port: 46493" |
                                      +--------------------------------+
                                                      /\
+--------------------------+                           |
| edit:                    +<--------------------------+
| buffer with Lisp source  |
+--------------------------+

To embed swank in a C application we need the application to launch Swank and then for Emacs to establish the connection to the swank server using slime-connect. Below is the C code that launches Swank.

Note, the following example is for a GNU/Linux type system. ecl needs to explicitly load load a shared library in order to access binary symbols such as C functions or C variables in the process, this is a hackish way of handling it since the library was already loaded when the applicaiton started, and could cause problems on platforms that put different constraints on loading shared libraries.

/* -*- mode: c;  -*-
   file: main.c
*/

#include "app_main.h"
/* a.out wrapper for call into a shared library. */
int main() {
  return app_main();
}

/* -*- mode: c;  -*-
   file: app_main.h
*/

#ifndef __APP_MAIN_H__
#define __APP_MAIN_H__

#include <ecl/ecl.h>

int app_main();

#endif /* APP_MAIN_H */

The following creates the shared library app_main used by both the C program and ECL for symbols. The embedded ECL code initializes the ECL environment and calls the Common Lisp load function to load a local Lisp file with the code to run swank.

/* -*- mode: c;  -*-
   file: app_main.c
*/

#include <stdlib.h>
#include <math.h>
#include "app_main.h"

void run_swank();

/* TODO: Are embedded quotes really needed? */
char start_swank[] =
  "\"/mnt/pixel-512/dev/stupid-ecl-tricks-1/start-swank-server.lisp\"";

char* argv;
char** pargv;

int app_main() {
  argv = "app";
  pargv = &argv;

  cl_boot(1, pargv);
  atexit(cl_shutdown);

  /* Set up handler for Lisp errors to prevent buggy Lisp (an */
  /* imposibility, I know!) from killing the app. */
  const cl_env_ptr l_env = ecl_process_env();
  CL_CATCH_ALL_BEGIN(l_env) {
    CL_UNWIND_PROTECT_BEGIN(l_env) {
      run_swank();
    }
    CL_UNWIND_PROTECT_EXIT {}
    CL_UNWIND_PROTECT_END;
  }
  CL_CATCH_ALL_END;

  return 0;

}

void run_swank() {
  cl_object cl_start_swank_path = c_string_to_object(start_swank);
  cl_object cl_load =  ecl_make_symbol("LOAD","CL");
  cl_funcall(2, cl_load, cl_start_swank_path);
  return;
}

The following Lisp file, loaded by app_main, contains a couple of snippets of code I copied from the Emacs Slime client that launches the Swank server. When Swank launches it will print out the socket you can use to connect to it, e.g.

;; Swank started at port: 58252.

you can then connect to it in Emacs using Slime:

M-x slime-connect

;;; -*- mode: lisp ; syntax: ansi-common-lisp -*-

;; standard quicklisp init file, since with be launching ecl without ~/.eclrc
(let ((quicklisp-init (merge-pathnames "quicklisp/setup.lisp"
                                       (user-homedir-pathname))))
  (when (probe-file quicklisp-init)
    (load quicklisp-init)))

(when (probe-file  "/tmp/slime.2565")
  (delete-file "/tmp/slime.2565"))

(load
 "~/quicklisp/dists/quicklisp/software/slime-2.14/swank-loader.lisp"
 :verbose t)

(funcall (read-from-string "swank-loader:init"))
(funcall (read-from-string "swank:start-server")
         "/tmp/slime.2565"))

A quick and dirty script file to build a shared library.

# -*- mode: bash;  -*-


rm -f *.o *.so app

export libs="-lm"

# Note, the -Wl,-R flags will make our shared library available to the
# executable app from the location that it was compiled, rather than
# having to be installed globably or adding the build path to
# LD_LIBRARY_PATH.

export ldflags="-L. -Wl,-R -Wl,."
export cflags="-DGC_LINUX_THREADS -D_REENTRANT -fPIC  -g -pipe -Wall"

gcc $cflags -c app_main.c
gcc -shared -Wl,-soname,libapp_main.so $ldflags -lecl -o libapp_main.so *o $libs
gcc main.c $cflags $ldflags -lapp_main -lecl -o app

To build and run

$ ./build_app.sh
$ ./app

ECL provide a facility for embedding C code directly in Lisp code like the following:

(defun c-sin (x)
  (ffi:clines "#include \"ecl/ecl.h\"")
  ;; Whoops!  mathh.h should be math.h
  (ffi:clines "#include <mathh.h>")
  (ffi:clines  "#include \"app_main.h\"")
  (ffi:c-inline (x) (:double) :double "{
@(return 0)= sin(#0);
}" :one-liner nil))

To use this function you need to compile the defun. When you issue the explicit compile,

(compile 'c-sin)

ECL will invoke your underlying C compiler. However, C syntax and header include errors, like we included in the above example, will cause compilation to fail. Unfortunately, ECL doesn't pass along the compilers output. You'll get something like the following:

;;; OPTIMIZE levels: Safety=2, Space=0, Speed=3, Debug=3
;;;
;;; End of Pass 1.
;;; Internal error:
;;;   ** Error code 1 when executing
;;; (RUN-PROGRAM "gcc" ("-I." "-I/usr/local/include/" "-D_GNU_SOURCE" "-D_FILE_OFFSET_BITS=64" "-g" "-O2" "-fPIC" "-D_THREAD_SAFE" "-Dlinux" "-O2" "-c" "/tmp/ecl001QoKf80.c" "-o" "/tmp/ecl001QoKf80.o"))

if you try to recreate the error by invoking the implied shell command:

$ gcc -I. -I/usr/local/include/ -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 \
    -g -O2 -fPIC -D_THREAD_SAFE -Dlinux -O2 -c /tmp/ecl001QoKf8.c \
    -o /tmp/ecl001QoKf80.o

You'll get the error:

gcc: error: /tmp/ecl001QoKf80.c: No such file or directory
gcc: fatal error: no input files
compilation terminated.

Because ECL has already cleaned it from /tmp.

But, ECL has a special variable, compiler::*delete-files* that controls cleaning up c output files. By setting it to nil, (setf compiler::*delete-files* nil) you can troubleshoot compilation errors. Re-running above gcc command on from the Unix shell gives us the following:

In file included from /tmp/ecl001QoKf80.c:6:0:
/tmp/ecl001QoKf80.eclh:8:19: fatal error: mathh.h: No such file or directory
#include <mathh.h>
                ^
compilation terminated.

Swank and ECL's embedded C in Lisp facility seem to have some issues with caching where compiled C snippets and a Swank images don't get refreshed when they should (at least on GNU/Linux). If you start noticing strange issues with changes to ffi:c-inline not taking effect or Swank having the wrong image, try deleting the following cache files:

rm -rf ~/.cache/common-lisp/ecl-15.2.21-ee989b97-linux-x64
rm -rf ~/.slime

/Earl Ducaine/

This document describe my experience with the SBC MIPS Creator CI20 from Imagination and ecl (Embeddable Common Lisp) interpreter/compiler. The goal is to program a distributed diagnostic/supervision system for DVB transmitter equipment's.

The Creator CI20 comes with a linux Debian Jessie distro preinstalled on its 8 GB internal flash drive. After upgrading it and install some packages I was ready to deploy ECL. The version choosen for the prototype was the 16.1.0, compilation was really straightforward (I modified a bash script written for SBCL to download, compile and install the git version of ECL) and after 20/30 minutes of code crunching I've got a shiny working ECL. I must say that problems didn't come from ECL itself, but from other packages. As I installed quicklisp soon came troubles, for example: slime does not add the correct architecture to the *features* variable and some other minor problems with iolib too. The Lisp community is the most active and helpful among those living on irc and with their help I've got fixed and ready in short time. The project I'm currently working on is divided in two main areas: the hardware design and test phase and the software one. The first, at this stage, contemplates the design and production of a limited number of ADIO boards (designed by me). More basic an ADIO board is an insulated analog and digital i/o board, it is interfaced with the Creator CI20 via the i2c bus and externally powered. The software part is the more vast part of the project. The idea is to provide a system that is able to interface to external users and feed them with sensors data regarding the state of the DVB apparatuses plus every ADIO + Creator CI20 hold the configuration for the apparatus they're going to monitor/supervise. In addition to that the system should be responsible to warn about anomalies and take some pre-emptive actions just to not make the whole thing blow up. So the Customer was facing a dilemma: Arduino + RS485 and a lot of amulets or rely on a modern and saner idea? My advice for the landing party was: "why do not use a cheap sbc (80 Eur) with an embedded OS powered by a Lisp interpreter?" here are the benefits:

• ethernet based communication hardware layer is proven to be cost effective and reliable, plus every oses and cheap sbcs support it;
• an embedded os, in most cases, is better than start a baremetal application: e.g. tcp/ip connections and standard network services like http, ftp, ssh, samba and snmp require a lot time to program, debug and test even if you use on the shelf solutions. on the contrary with an embedded os you've got these things already coded and ready to use, shortening the deployment time of about one order of magnitude (in man hours);
• using lisp is useful for almost three aspects: portability, remote debugging and last but not least the possibility to create compiled code for mips;
• possibility to expand hardware and software via external modules.

So lisp at last, the first prototype is now running a test code which provide basic functionalities like: a general i2c library to access the linux i2c device via /dev/i2c-*, a simple i2c gpio expander library for the mcp23017 and another library to cope with hitachi compatible 16x2 lcd and a simple looping program which displays some random characters on it. next improvements are ready to be tested too, like a machine learning algorithm to detect anomalies and so on.

At this stage, ECL was truly the right choice: first of all the time to develop this small set of libraries was incredibly short compared to c/c++ due to that now I've got a test prototype running on my desk happily showing on a 16x2 display some test patterns in just about one month circa from Lisp environment setup. I must thank the friend PJB for his invaluable help and patience and jackdaniel for his work on ECL and the patience too.

/Angelo Rossi/

ECL may be built as a library. This great feature allows us to dynamically link ECL with arbitrary applications capable of using shared objects. We take advantage of this and the fact, that the android platform allows us to embed shared objects with the .apk and use it with the Java Native Interface (JNI).

To use libecl we first need to cross compile it for the target platform. There is some work pending to simplify the cross compilation process, but for a time being you have to build a host ECL compiler and after that the target library.

Before that, however, you have to build a proper android toolchain. If you don't have a prebuilt one you may use android NDK scripts. Both Android SDK and NDK are a prerequisites and installing them is an exercise for the reader.

export PLATFORM_PREFIX=/opt/toolchains/android-ndk9/
export NDK_PATH=/opt/android-ndk/
export NDK_PLATFORM=android-12

mkdir ${PLATFORM_PREFIX}
/opt/android-ndk/build/tools/make-standalone-toolchain.sh \
    --platform=${NDK_PLATFORM} \
    --install-dir=${PLATFORM_PREFIX} \
    --arch=arm

export PATH=${PLATFORM_PREFIX}/bin:${PATH}

Now you have to build the host compiler¹ and the final library for the android. Note, that using the preexisting ECL binary could work if you it is a 32 bit installation with disabled longdouble.

./configure ABI=32 CFLAGS="-m32 -g -O2" LDFLAGS="-m32 -g -O2" \
            --disable-longdouble \
            --prefix=`pwd`/ecl-android-host \
            --enable-libatomic=included

make && make install
export ECL_TO_RUN=`pwd`/ecl-android-host/bin/ecl
rm -r build

./configure --host=arm-linux-androideabi \
            --prefix=`pwd`/ecl-android-target \
            --with-cross-config=`pwd`/src/util/android.cross_config \
            --disable-soname

# You have to adjust build/cross_config to your settings (especially
# set ECL_TO_RUN to your host ecl)
make && make install

You should have libecl.so and the other necessary files in the ecl-android/ directory. Some pre-compiled modules are located in the ecl-android/lib/ecl-16.1.0 directory. You will want them on the target system.

This application isn't stable yet and documentation is still rather scarce. API isn't stable and it is discouraged to base any work on it for now. It is provided on terms of AGPL-3.0+ license, however alternative licensing is possible.

First thing to do is to clone the repository and adjust the project configuration to your local setup.

git clone https://gitlab.common-lisp.net/ecl/ecl-android.git
cd ecl-android
# update the project (sets sdk path and the other android "magic")
android update project -t android-10 -p .
# create symlinks (sets ECL directories). For instance:
ln -s ../ecl-android-target ecl-android
ln -s ecl-android/lib/ecl-*.*.* ecl-libdir

Now it's worth to explain how the application works on the target platform. We put the library libecl.so in the apk file, but rest of the module and other ECL-related files are packed as a resource in the assets/lisp/ directory at which *default-pathname-defaults* points to.

Initialization is performed by assets/lisp/etc/init.lisp file, which loads the user.lisp. The latter contains some sample code loading swank (if it's put in the home/slime-2.14/ directory – swank is not bundled with the repository) and defines auxiliary functions: #'get-quicklisp, #'start-swank and #'stop-swank.

Function #'get-quicklisp will download and install the Quicklisp. It will also replace it's bundled compression tools with a prebuilt one (it is essential for performance – GCC produces faster code then the byte compiler).

Before you build the ecl-android you may want to copy some files for further use and edit the initialization script:

mkdir assets/lisp/home
cp -rf ~/things/slime-2.14 assets/lisp/home
cp ~/things/my-awesome-lisp-app/awesome.lisp assets/lisp/
emacs assets/etc/user.lisp

When you are ready you may build and deploy the application on the phone:

ndk-build
ant debug install

ECL Android launcher should appear on your phone.