Embeddable Common-Lisp

Recent Content

ECL Quarterly Volume IV

posted on 2016-06-15

1 Preface

Hello,

I've managed to assemble the fourth volume of the ECL Quarterly. As always a bit off schedule but I hope you'll find it interesting.

This issue will revovle around ECL news, some current undertakings and plans. Additionally we'll talk about Common Lisp implementations in general and the portability layers. I believe it is important to keep things portable. Why? Keep reading!

Lately we're working with David O'Toole on making support for ECL on Android better. He wants to distribute his games on this platform and was kind enough to write an article for ECL Quarterly. Thanks to his work we've discovered various rough edges and bugs in ECL and gained some invaluable insight into the cross compilation problems of Common Lisp applications.

As the final remark – I've found some time to establish a proper RSS subscription feed for ECL and ECL Quarterly. I hope that this issue will finally land on the Planet Lisp – a well known Lisp-related blog posts aggregator maintained by Zach Beane.

I want to thank for the valuable feedback and proofreading to many people, especially Antoni Grzymała, Javier Olaechea, Michał Posta, Ilya Khaprov and David O'Toole.

Have a nice lecture,


Daniel Kochmański ;; aka jackdaniel | TurtleWare
Poznań, Poland
June 2016

2 ECL's "what's going on"

I've added a milestone with a deadline for the ECL 16.1.3 release with the bugs I want to fix. You may find it here. I'm very happy to receive a lot of positive feedback, merge requests and awesome bug reports. Thank you for that! :-)

Backporting CLOS changes from CLASP was successful but we won't incorporate them in the main branch. The recently resurrected cl-bench has shown that these changes impact performance and consing negatively (check benchmarks). If you are curious about the changes, you may checkout the branch feature-improve-clos in the repository.

I'm slowly working on the new documentation. This is very mundane task which I'm not sure I'll be able to finish. Rewriting DocBook to TexInfo and filling the missing parts is hard. I'm considering giving up and improving the DocBook instead.

In the near future I plan to make a crowdfunding campaign to improve support for cross-compilation, Android and Java interoperability in order to boost development. More details will be probably covered in the next Quarterly issue.

3 Porting Lisp Games to Android with Embeddable Common Lisp, Part 1

3.1 Introduction

Recently I ported my Common Lisp game engine "Xelf" to the Android operating system using Embeddable Common Lisp.

Some work remains to be done before I can do a proper beta test release, but ECL Quarterly provides a good opportunity to pause and share the results thus far. This is the first part of a two-part article. The focus of Part 2 will be on performance optimization, testing, and user interface concerns.

Special thanks to Daniel Kochmański, 3-B, oGMo, and the rest of the Lisp Games crew for their inspiration and assistance.

3.1.1 About the software

Xelf is a simple 2-D game engine written in Common Lisp. It is the basis of all the games I have released since 2008, and can currently be used with SBCL to deliver optimized standalone game executables for GNU/Linux, MS Windows, and Mac OSX.

I've also published a Git repository with all the work-in-progress scripts, patches, and libraries needed to compile Xelf for Android with Embeddable Common Lisp, OpenGL, and SDL.

Please note that this is a pre-alpha release and is mainly intended for Common Lisp developers looking to get a head start in building an Android game. Use with caution.

Xelf is not required; you can substitute your own Lisp libraries and applications and just use the repo as a springboard.

I would like to add support for CL-SDL2 as well, both as a prelude to porting Xelf to SDL 2.0, and as a way to help the majority who use SDL 2.0 for current projects.

3.2 Problems

3.2.1 Choosing an implementation

As I use only Free Software for my projects, I did not consider any proprietary Lisps.

Steel Bank Common Lisp now runs on Android, but SBCL as a whole cannot yet be loaded as a dynamic shared library. This is a show-stopper because Android requires the entry point of a native application to be in a shared library specially embedded in the app.

Xelf works very well with Clozure Common Lisp, but CCL's Android support is not fully functional at present. So I've been quite happy to discover Embeddable Common Lisp. Its technique of translating Common Lisp into plain C has made integration with the Android NDK toolchain relatively simple.

3.2.2 Cross-compilation

For performance reasons the Lisp stack (meaning LISPBUILDER-SDL, CL-OPENGL, CFFI, Xelf, the game, and all their dependencies) must be compiled to native ARM machine code and loaded as shared libraries.

There is a complication in this task as regards ECL. The latter produces native code by translating Common Lisp into plain C, and then invoking the C compiler. But the C compiler toolchain is not typically present on Android, and building one that is properly configured for this task has proved difficult so far.

Therefore we must cross-compile the entire Lisp stack. ECL's Android build procedure already cross-compiles the Lisp contained in ECL, but there were additional difficulties in compiling Lisp libraries which I'll cover below in the "Solutions" section.

3.2.3 Legacy code

Xelf has improved a lot over time and gained new features, but is now outdated in some respects. When I first wrote Xelf in the 2006-2007 period SDL 1.2 was current and OpenGL Immediate mode had not yet been officially deprecated. This hasn't been a terrible problem in practical terms, given that both are still widely supported on PC platforms. But porting to Android would mean I could not procrastinate any longer on updating Xelf's SDL and OpenGL support.

3.3 Solutions

3.3.1 CommanderGenius to the rescue

Help arrived for my SDL woes in the form of Sergii Pylypenko's "CommanderGenius", a fancy port of SDL 1.2/2.0 to Android. I can utilize the existing LISPBUILDER-SDL bindings for SDL, SDL-MIXER, SDL-TTF, SDL-IMAGE, and SDL-GFX. Not only that, there are extra features such as gamepad support, floating virtual joysticks, access to touchscreen gesture data and Android system events, support for the Android TV standard, and much more.

CommanderGenius is actually designed from the start to rebuild existing SDL 1.2 / 2.0 / OpenGL projects as Android applications, and includes dozens of examples to work with. So in mid-May this year I set about splicing together Daniel Kochmański's ECL-ANDROID Java wrapper and startup code (which together load ECL as a shared object from within the app) into the CommanderGenius SDL application code and build procedures.

The result is a fullscreen SDL/OpenGL application with Embeddable Common Lisp, optionally running Swank. There's even a configurable splash screen!

3.3.2 Do a little dance with ASDF

ECL can compile an entire system into one FASL file, but I ran into a snag with the ASDF-based build procedure. The typical way is to compile each Lisp file and then load the resulting compiled file. But on the cross-compiler,

(load (compile-file "myfile.lisp"))

fails because the output of COMPILE-FILE is a binary for the wrong architecture. Likewise, alien shared libraries cannot be loaded during Lisp compilation, which broke CL-OPENGL and LISPBUILDER-SDL.

My temporary solution was to redefine the function ASDF:PERFORM-LISP-LOAD-FASL in my build script. My modified version does something like this instead:

(compile-file "myfile.lisp")
(load "myfile.lisp")

I then invoke ECL's system builder, which spits out a big binary FASB file containing the whole system. But thanks to the LOAD statements, each Lisp file has had access to the macros and other definitions that preceded it in compilation.

I'm sure this is really wrong, but it works, and the resulting FASBs load very quickly. (App startup time went from over 30 seconds when loading byte-compiled FASCs, to about 3.5 seconds.)

In the end, it was simple to deal with CL-OPENGL and LISPBUILDER-SDL wanting to open shared libraries during compilation. I used Grep to find and then comment out calls to CFFI:USE-FOREIGN-LIBRARY, leaving the DEFINE-FOREIGN-LIBRARY definitions intact. This allows cross-compilation to proceed normally.

Then on Android, after the FASBs are loaded I invoke USE-FOREIGN-LIBRARY on each of the required definitions.

So tricking ASDF works. But aside from being a hack, it's not enough for some of the things I'd like to do. The INLINED-GENERIC-FUNCTION technique looks like a highly promising way to increase performance, but my cross-compilation trick led in this case to invalid FASB's with embedded FASC bytecodes. Indeed, to work with ECL in this situation would require actually loading the ARM-architecture compiled INLINED-GENERIC-FUNCTION binary before compiling systems that use inlining—which as mentioned above cannot be done during cross-compilation.

I'm exploring other potential solutions, such as installing a GNU/Linux container on my Android development unit in order to give ECL access to a native C compiler toolchain (see below). I may even attempt to write a custom cross-compilation procedure using Clang and LLVM. But this is less urgent for now, because tweaking ASDF is sufficient to produce a working application.

3.3.3 Use OpenGL ESv1 with CL-OPENGL

Luckily the the path of least resistance could prevail here. OpenGL ES version 1 is widely supported on Android devices, and is easier to port to from Immediate mode than is GLESv2. CL-OPENGL supports it right out of the box. (I'd like to thank 3-B and oGMo for their help in bridging the gap with my own code.)

Some tasks remain to be done here but most of Xelf's drawing functions are now working, including TrueType fonts and vertex coloring.

I've also written some code to partially emulate vertex coloring as a way of increasing render performance, and this will be covered in the forthcoming Part 2 of this article.

3.3.4 ProTip: Use the byte-compiler

One issue has gone unmentioned. How do I interactively redefine functions and set variables in order to develop the running game via SLIME/Swank, if everything must be cross-compiled on an X86 system?

The answer is that ECL's built-in bytecode compiler is used in these cases, and the bytecoded definitions replace the originals. I can freely use COMPILE-FILE, LOAD, and even ASDF:LOAD-SYSTEM during "live" development; under normal circumstances the only real difference is execution speed of the resulting code. The final game app will ship without Swank, of course, and with a fully native Lisp stack.

Now you have a new problem, which is how to edit the Lisp files on your Android device so that Swank can compile and load them.

3.3.5 ProTip: Use Emacs TRAMP with ADB

To make this useful you need a rooted android device.

(add-to-list 'tramp-default-user-alist '("adb" nil "root"))
(find-file "/adb::/")

This can integrate with Emacs' "bookmarks" and "desktop" features for even more convenience.

3.3.6 ProTip: Use Emacs to inspect your APK package

They're just zip files. Missing libraries or assets? Check the APK by opening it as a file in GNU Emacs.

3.3.7 ProTip: Use a GNU/Linux container for SSH and native Emacs with X11!

You can actually install a GNU/Linux "container" with Debian, Ubuntu, or some other distribution on your Android development system in order run the Secure Shell daemon and many other applications. I use it to run a graphical Emacs on the Android box, with Emacs' X11 connection forwarded through SSH so that its windows open on my desktop GNU/Linux PC's X server—right alongside my native Emacs. I use different color themes to avoid mixing them up.

This gives me full access to everything on both systems from a single mouse/keyboard/monitor, and I can cut and paste text freely between applications.

Setting up such a container is beyond the scope of this article, but I highly recommend it. It was pretty easy on a rooted device, and works very well.

3.4 Conclusion

In less than a month we went from "let's do it" to "wow, it works!" What more can you ask for?

This concludes Part 1 of my article on building Lisp games for Android with Embeddable Common Lisp. To read my running commentary and see news and test results as they are posted, you can visit the project README:

https://gitlab.com/dto/ecl-android-games-src/blob/master/README.org

More details and all scripts and configurations can be found in that repository.

Thanks for reading,


David O'Toole (dto@xelf.me)
11 June, 2016

4 Common Lisp implementations

Some time ago I've created with the help of many kind people (most notably from Rainer Joswig and Fare Rideau) a graph presenting Common Lisp implementations and the relations between them. This version is improved over drafts presented on twitter and linkedin. If you find any errors, please contact me.

all-hierarchy.png

It is worth noting that LispWorks and VAX share the code with Spice Lisp which later evolved into Common Lisp implementation CMUCL. Striped lines lead to CMUCL, because I didn't want to add pre-CL implementations.

There is also suspicion that Lucid shares code with Spice Lisp and/or VAX, but I couldn't confirm that, so I'm leaving it as is.

"JavaScript Lisp Implementations" classifies some lisps as CL, but I've added only Acheron and parenscript to the list, because rest is just CL-ish, not even being a subset.

Resources I've found on the internet: CMU FAQ, ALU list, CLiki overview, Wikipedia article, JavaScript Lisp Implementations.

5 Building various implementations

I've built various lisps to perform some benchmarks and to have some material for comparison. Ultimately I've decided to polish it a little and publish. I had some problems with Clasp and Mezzano so I've decided to not include them and leave building these as an exercise for the reader ;-). Also, if you feel adventurous, you may try to build Poplog, which has Common Lisp as one of the supported languages.

If you want to read about various implementations, please consult Daniel's Weinreb Common Lisp Implementations: A Survey (material from 2010, definitely worth reading).

First we create a directory for the lisp implementations (we'll build as an ordinary user) and download the sources. Each implementation has a list of building prerequisites, but it may be not comprehensive.

export LISPS_DIR=${HOME}/lisps
mkdir -p ${LISPS_DIR}/{src,bin}
pushd ${LISPS_DIR}/src

# Obtain sources
svn co http://abcl.org/svn/trunk/abcl/ abcl
# git clone git@github.com:drmeister/clasp.git
svn co http://svn.clozure.com/publicsvn/openmcl/trunk/linuxx86/ccl ccl
hg clone http://hg.code.sf.net/p/clisp/clisp clisp
git clone git@common-lisp.net:cmucl/cmucl.git cmucl
git clone https://gitlab.com/embeddable-common-lisp/ecl.git ecl
git clone git://git.sv.gnu.org/gcl.git gcl
git clone git@github.com:davazp/jscl.git jscl
# git clone https://github.com/froggey/Mezzano.git
git clone https://gitlab.common-lisp.net/mkcl/mkcl.git mkcl
git clone git://git.code.sf.net/p/sbcl/sbcl sbcl
git clone git@github.com:wadehennessey/wcl.git wcl
git clone https://github.com/gnooth/xcl.git

5.0.1 ABCL (Armed Bear Common Lisp)

Requires
jdk, ant
pushd abcl
ant
cp abcl ${LISPS_DIR}/bin/abcl-dev
popd

5.0.2 CCL (Clozure Common Lisp)

Requires
gcc, m4, gnumake
pushd ccl
echo '(ccl:rebuild-ccl :full t)' | ./lx86cl64 -n -Q -b

# installation script is inspired by the AUR's PKGBUILD
mkdir -p ${LISPS_DIR}/ccl-dev
cp -a compiler contrib level-* lib* lisp-kernel objc-bridge \
   tools x86-headers64 xdump lx86cl64* examples doc \
   ${LISPS_DIR}/ccl-dev

find ${LISPS_DIR}/ccl-dev -type d -name .svn -exec rm -rf '{}' +
find ${LISPS_DIR}/ccl-dev -name '*.o' -exec rm -f '{}' +
find ${LISPS_DIR}/ccl-dev -name '*.*fsl' -exec rm -f '{}' +

cat <<EOF > ${LISPS_DIR}/bin/ccl-dev
#!/bin/sh
exec ${LISPS_DIR}/ccl-dev/lx86cl64 "\$@"
EOF
chmod +x ${LISPS_DIR}/bin/ccl-dev
popd

5.0.3 CLISP

Requires
gcc, make
Notes
don't build with ASDF (it's old and broken)
pushd clisp
./configure --prefix=${LISPS_DIR}/clisp-dev/ \
            --with-threads=POSIX_THREADS \
            build/
cd build
make && make install
ln -s ${LISPS_DIR}/clisp-dev/bin/clisp ${LISPS_DIR}/bin/clisp-dev
popd

5.0.4 CMUCL (CMU Common Lisp)

Requires
cmucl binary, gcc, make, openmotif
Notes
it needs another CMUCL to bootstrap (release 21a)
pushd cmucl
mkdir -p prebuilt
pushd prebuilt
wget https://common-lisp.net/project/cmucl/downloads/release/21a/cmucl-21a-x86-linux.tar.bz2 \
     https://common-lisp.net/project/cmucl/downloads/release/21a/cmucl-21a-x86-linux.extra.tar.bz2
mkdir ${LISPS_DIR}/cmucl-21a
tar -xf cmucl-21a-x86-linux.tar.bz2 -C ${LISPS_DIR}/cmucl-21a/
tar -xf cmucl-21a-x86-linux.extra.tar.bz2 -C ${LISPS_DIR}/cmucl-21a/
cat <<EOF > ${LISPS_DIR}/bin/cmucl-21a
#!/bin/sh
exec ${LISPS_DIR}/cmucl-21a/bin/lisp "\$@"
EOF
chmod +x ${LISPS_DIR}/bin/cmucl-21a
# Note, that this is already a fully functional lisp now
popd
bin/build.sh -C "" -o "cmucl-21a"
bin/make-dist.sh -I ${LISPS_DIR}/cmucl-dev/ linux-4/
cat <<EOF > ${LISPS_DIR}/bin/cmucl-dev
#!/bin/sh
exec ${LISPS_DIR}/cmucl-dev/bin/lisp "\$@"
EOF
chmod +x ${LISPS_DIR}/bin/cmucl-dev
popd

5.0.5 ECL (Embeddable Common Lisp)

Requires
gcc, make
./configure --prefix=${LISPS_DIR}/ecl-dev/
make && make install
ln -s $LISPS_DIR/ecl-dev/bin/ecl ${LISPS_DIR}/bin/ecl-dev

5.0.6 JSCL (Java Script Common Lisp)

Requires
Conforming CL implementation, web browser, nodejs
Notes
Doesn't provide LOAD yet (no filesystem), but author confirmed that this will be implemented (virtual filesystem on the browser and the physical one on the nodejs).
mkdir ${LISPS_DIR}/jscl-dev
pushd jscl
./make.sh

# Run in the console (node-repl)
cp jscl.js repl-node.js ${LISPS_DIR}/bin/jscl-dev
cat <<EOF > ${LISPS_DIR}/bin/jscl-dev
#!/bin/sh
exec node ${LISPS_DIR}/jscl-dev/repl-node.js
EOF
chmod +x ${LISPS_DIR}/bin/jscl-dev

# Run in the web browser (optional)
cp jscl.js repl-web.js jquery.js jqconsole.min.js jscl.html style.css \
   ${LISPS_DIR}/jscl-dev/
# replace surf with your favourite browser supporting JS
cat <<EOF > ${LISPS_DIR}/bin/jscl-dev-browser
#!/bin/sh
exec surf ${LISPS_DIR}/jscl-dev/jscl.html
EOF
chmod +x ${LISPS_DIR}/bin/jscl-dev-browser

popd

5.0.7 GCL (GNU Common Lisp)

Requires
gcc, make
# Doesn't work both with head and the release, luckily it works with
# the next pre-release branch
git checkout Version_2_6_13pre
./configure --prefix=${LISPS_DIR}/gcl-2.6.13-pre
make && make install
ln -s ${LISPS_DIR}/gcl-2.6.13-pre/bin/gcl ${LISPS_DIR}/bin/gcl-2.6.13-pre

5.0.8 MKCL (Man-Kai Common Lisp)

Requires
gcc, make
pushd mkcl
./configure --prefix=${LISPS_DIR}/mkcl-dev
make && make install
ln -s ${LISPS_DIR}/mkcl-dev/bin/mkcl ${LISPS_DIR}/bin/mkcl-dev
popd

5.0.9 SBCL (Steel Bank Common Lisp)

Requires
ANSI-compliant CL implementation
Notes
  • Lisp has to close on EOF in top-level (CMUCL doesn't do that),
  • ECL has some bug regarding Lisp-to-C compiler apparently triggered by the SBCL compilation – don't use it here,
  • we could use precompiled SBCL like with the CMUCL, but let's exploit the fact, that we can compile from the C-bootstrapped implementation (we'll use already built clisp-dev),
  • it is advised to run the script in fast terminal (like xterm) or in the terminal multiplexer and to detach it – SBCL compilation process is very verbose,
  • if you build SBCL on Windows, consider using MinGW to preserve POSIX compatibility.
pushd sbcl
export GNUMAKE=make
./make.sh "clisp"
INSTALL_ROOT=${LISPS_DIR}/sbcl-dev ./install.sh
cat <<EOF > ${LISPS_DIR}/bin/sbcl-dev
#!/bin/sh
SBCL_HOME=${LISPS_DIR}/sbcl-dev/lib/sbcl exec ${LISPS_DIR}/sbcl-dev/bin/sbcl "\$@"
EOF
chmod +x ${LISPS_DIR}/bin/sbcl-dev
popd

5.0.10 WCL

Requires
tcsh, gcc, git
Notes
very incomplete implementation
pushd wcl
REV=`git rev-parse HEAD`
sed -i -e "s/WCL_VERSION = \"3.0.*$/WCL_VERSION = \"3.0-dev (git-${REV})\"/" CONFIGURATION
LD_LIBRARY_PATH=`pwd`/lib make rebuild
mkdir ${LISPS_DIR}/wcl-dev
cp -a bin/ lib/ doc/ ${LISPS_DIR}/wcl-dev/
cat <<EOF > ${LISPS_DIR}/bin/wcl-dev
#!/bin/sh
LD_LIBRARY_PATH=${LISPS_DIR}/wcl-dev/lib exec ${LISPS_DIR}/wcl-dev/bin/wcl "\$@"
EOF
chmod +x ${LISPS_DIR}/bin/wcl-dev
popd

5.0.11 XCL

Requires
gcc
Notes
last commit in 2011
pushd xcl
mkdir ${LISPS_DIR}/xcl-dev
XCL_HOME=${LISPS_DIR}/xcl-dev make
cp -a clos compiler lisp COPYING README xcl ${LISPS_DIR}/xcl-dev
# This will build in XCL_HOME, even if run in source directory
./xcl <<EOF
(rebuild-lisp)
EOF

ln -s ${LISPS_DIR}/xcl-dev/xcl ${LISPS_DIR}/bin/xcl-dev
popd

6 Portability libraries

It is important to know the difference between the language standard, implementation-specific extensions and the portability libraries. The language standard is something you can depend on in any conforming implementation.

Sometimes it's just not enough. You may want to do ** serializethreading*, or to *data, which is very hard to express (or even impossible) in the language provided by the standard. That's where the implementation-specific extensions kick in. Why are they called "implementation-specific"? Because the API may be different between implementations – reaching consensus is a hard thing1.

The most straightforward approach I can imagine is to reach for the documentation of the Common Lisp implementation you are currently using and to use the API provided by this implementation. I dare you not to do that! It's definitely the easiest thing to do at first, but mind the consequences. You lock yourself, and your users in the implementation you prefer. What if you want to run it on the JVM or to make it a shared library? Nope, you're locked-in.

"What can I do then?" – you may ask. Before I answer this question, I'll tell you how many people do it (or did it in the past) – they used read-time conditionals directly in the code. Something like the following:

(defun my-baz ()
  #+sbcl                        (sb-foo:do-baz-thing 'quux)
  #+ccl                         (ccl:baz-thing       'quux)
  #+(and ecl :baz-thing)        (ext:baz             'quux)
  #+abcl                        (ext:baz             'quux)
  #+(and clisp :built-with-baz) (ext:baz-thingie     'quux)
  #-(or sbcl ccl ecl abcl clisp)
  (error "Your implementation isn't supported. Fix me!"))

If the creator felt more fancy and had some extra time, they put it in the package my-app-compat. It's all great, now your application works on all supported implementations. If somebody wants theirs implementation to work, send the creator a patch, who incorporates it into the code and voila, everything works as desired.

We have one problem however. Libraries tend to depend on one another. There is also a lot of software which uses features beyond the ANSI specification (it's all good, programmers need these!). Do you see code duplication everywhere? How many times does a snippet above have to be copy-pasted, or rewritten from scratch? It's not black magic after all. APIs between ad-hoc implementations don't exactly match, covered CL implementations differ…

So you quickload your favorite library which depends on 10 other libraries which implement BAZ functionality in theirs own unique way, with a slightly different API on the unsupported implementation – that's why we have my-baz abstraction after all, right? Now, to make it work, a user has to:

  1. Find which of the ten libraries don't work (not trivial!),
  2. find and clone the repositories (we want to use git for patches),
  3. fix each one of them (grep helps!) and commit the changes,
  4. push the changes to your own forked repository and create a pull request (or send a diff to the mailing list) – *ten times*,
  5. voila, you're done, profit, get rich, grab a beer.

It's a lot of work which the user probably won't bothered to do. They will just drop the task, choose another implementation or hack their own code creating the Yet Another Baz Library for the implementations he cares for reinventing the wheel once more. It's a hacker's mortal sin.

I'm going to tell you now what is the Right Thing™ here. Of course you are free to disagree. When you feel that there is a functionality you need which isn't covered by the standard you should

  1. Look if there is a library which provides it.

    You may ask on IRC, the project's mailing list, check out the CLiki, do some research on the web. Names sometimes start with trivial-*, but it's not a rule. In other words: do your homework.

  2. If you can't find such a library, create one.

    And by creating such a library I mean comparing the API proposed by at least two CL implementations (three would be optimal IMHO), carefully designing your own API which covers the functionality (if it's trivial, this should be easy) and implementing it in your library.

    Preferably (if possible) add a fallback implementation for implementations not covered (with the appropriate warning, that it may be inefficient or not complete in one way or another).

    It may be worth reading the Maintaining Portable Lisp Programs paper written by Christophe Rhodes.

  3. Write beautiful documentation.

    A CL implementation docs may be very rough. It takes time to write them and programmers tend to prioritize code over the documentation. It's really bad, but it's very common for the documentation to be incomplete or outdated.

    Document your library, describe what it does, how to use it. Don't be afraid of the greatness! People will praise you, success will come, world will be a better place. And most importantly, your library will be useful to others.

  4. Publish the library.
  5. Make that library your project's dependency.

I know it's not easy, but in the long term it's beneficial. I guarantee you that. That's how the ecosystem grows. Less duplication, more cooperation – pure benefit.

Some people don't follow this path. They didn't think it through, or maybe they did and decided that keeping the dependency list minimal is essential to their project, or were simply lazy and hacked their own solution. There are also some old projects which exported a number of features being a very big portability library and an application at the same time (ACL-compat, McCLIM and others). What to do then?

If it's a conscious decision of the developer (who doesn't want to depend on /anything/), you can do nothing but provide a patch adding your own implementation to the supported list. It's their project, their choice, we have to respect that.

But before doing that you may simply ask if they have something against plugging these hacks with the proper portability library. If they don't – do it, everybody will benefit.

There are a few additional benefits of the presented portability library approach for the implementations itself. Having these internal details in one place makes it more probable that your implementation is already supported. If the library has a bug it's easier to fix it in one place. Also, if the CL implementation changes it's API, it's easy to propagate changes to the corresponding portability libraries. New CL implementation creators have a simplified task of making their work usable with existing libraries.

It is worth noting, that creating such library paves the way to the new quasi-standard functionalities. For instance Bordeaux Threads has added recently CONDITION-WAIT function, which isn't implemented on all implementations. It is a very good stimulus to add it. This is how library creators may have real impact on the implementation creators decisions about what to implement next.

6.1 Portability layer highlights

Here are some great projects helping CL implementations be part of a more usable ecosystem. Many of these are considered being part of the de-facto standard:

bordeaux-threads
Provides thread primitives, locks and conditionals
cl-store
Serializing and deserializing CL objects from streams
cffi
Foreign function interface (accessing foreign libraries)
closer-mop
Meta-object protocol – provides it's own closer-common-lisp-user package (redefines for instance defmethod)
usocket
TCP/IP and UDP/IP socket interface.
osicat
Osicat is a lightweight operating system interface for Common Lisp on POSIX-like systems, including Windows
cl-fad
Portable pathname library
trivial-garbage
trivial-garbage provides a portable API to finalizers, weak hash-tables and weak pointers
trivial-features
trivial-features ensures consistent *FEATURES* across multiple Common Lisp implementations
trivial-gray-streams
trivial-gray-streams system provides an extremely thin compatibility layer for gray streams
external-program
external-program enables running programs outside the Lisp process

There are many other very good libraries which span multiple implementations. Some of them have some drawbacks though.

For instance IOlib is a great library, but piggy-backs heavily on UN*X – if you develop for many platforms you may want to consider other alternatives..

UIOP is also a very nice set of utilities, but isn't documented well, does too many things at once and tries to deprecate other actively maintained projects – that is counterproductive and socially wrong. I'd discourage using it.

There are a few arguments supporting UIOP's state – it is a direct dependency of ASDF, so it can't (or doesn't want to) depend on other libraries, but many utilities are needed by this commonly used system definition library. My reasoning here is as follows: UIOP goes beyond ASDF's requirements and tries to make actively maintained projects obsolete. Additionally it works only on supported implementations even for features which may be implemented portably.

6.2 UIOP discussion

I'm aware that my opinion regarding UIOP may be a bit controversial. I've asked the library author and a few other people for feedback which I'm very grateful for. I'm publishing it here to keep opinions balanced.

6.2.1 Fare Rideau

Dear Daniel,

while there is a variety of valid opinions based on different interests and preferences, I believe your judgment of UIOP is based on incorrect premises.

First, I object to calling UIOP "not well documented". While UIOP isn't the best documented project around, all its exported functions and variables have pretty decent DOCSTRINGs, and there is at least one automatic document extractor, HEΛP, that can deal with the fact that UIOP is made of many packages, and extract the docstrings into a set of web pages, with a public heλp site listed in the UIOP README.md. The fact that some popular docstring extractors such as quickdocs can't deal with the many packages that UIOP creates with its own uiop:define-package doesn't mean that UIOP is less documented than other projects on which these extractors work well, it's a bug in these extractors.

Second, regarding the deprecation of other projects: yes, UIOP does try to deprecate other projects, but (a) it's a good thing, and (b) I don't know that any of the projects being deprecated is "actively maintained". It's a good thing to try to deprecate other lesser libraries, as I've argued in my article Consolidating Common Lisp libraries: whoever writes any library should work hard so it will deprecate all its rivals, or so that a better library will deprecate his and all rivals (such as optima deprecating my fare-matcher). That's what being serious about a library is all about. As for the quality of the libraries I'm deprecating, one widely-used project the functionality of which is completely covered by UIOP is cl-fad. cl-fad was a great improvement in its day, but some of its API is plain broken (e.g. the :directories argument to its walk-directory function has values with bogus names, while its many pathname manipulation functions get things subtly wrong in corner cases), and its implementation not quite as portable as UIOP (that works on all known actively used implementations). There is no reason whatsoever to ever choose cl-fad over UIOP for a new project. Another project is trivial-backtrace. I reproduced most of its functionality, except in a more stable, more portable way (to every single CL implementation). The only interface I didn't reproduce from it is map-backtrace, which is actually not portable in trivial-backtrace (only for SBCL and CCL), whereas serious portable backtrace users will want to use SLIME's or SLY's API, anyway. As for external-program, a good thing it has for it is some support for asynchronous execution of subprocesses; but it fails to abstract much over the discrepancies between implementations and operating systems, and is much less portable than uiop:run-program (as for trivial-shell, it just doesn't compete).

UIOP is also ubiquitous in a way that other libraries aren't: all implementations will let you (require "asdf") out of the box at which point you have UIOP available (exception: mostly dead implementations like Corman Lisp, GCL, Genera, SCL, XCL, may require you to install ASDF 3 on top of their code; still they are all supported by UIOP, whereas most portability libraries don't even bother with any of them). This ubiquity is important when writing scripts. Indeed, all the functionality in UIOP is so basic that ASDF needed it at some point — there is nothing in UIOP that wasn't itself required by some of ASDF's functionality, contrary to your claim that "UIOP goes beyond ASDF's requirements" (exception: I added one function or two to match the functionality in cl-fad, such as delete-directory-tree which BTW has an important safeguard argument :validate; but even those functions are used if not by ASDF itself, at least by the scripts used to release ASDF itself). I never decided "hey, let's make a better portability library, for the heck of it". Instead, I started making ASDF portable and robust, and at some point the portability code became a large chunk of ASDF and I made it into its own library, and because ASDF is targetting 16 different implementations and has to actually work on them, this library soon became much more portable, much more complete and much more robust than any other portability library, and I worked hard to achieve feature parity with all the libraries I was thereby deprecating.

Finally, a lot of the functionality that UIOP offers is just not offered by any other library, much less with any pretense of universal portability.

6.2.2 David Gu

For the documentation thing, I really think Quickdocs could do a better job. The bug #24 stated that problem, however, it's remain to be solved. I will check this out if I have free time recently.

I use UIOP a lot in my previous company, the reason is simple and maybe a little naive: my manager didn't want to involve too many add-ons in the software. UIOP is shipped together with ASDF, it's really "convenient", and its robustness is the final reason why I will stick to it. If people understand how UIOP came out in the history from ASDF2 to ASDF3, I think people will understand why it's acting like deprecating several other projects – that's not the original idea of it.

But anyway, I really learned a lot from this post and also the comments. In my opinion, avoid reinventing the wheels is the right idea and directions for this community. So from that perspective, I support @fare's idea "It's a good thing to try to deprecate other lesser libraries". Including this article and along with Maintaining Portable Lisp Programs and @fare's Consolidating Common Lisp Libraries, we should let more people involved in this topic.

Footnotes:

1

If you are Common Lisp implementer and plan to add a feature beyond ANSI specification, please consider writing a proposal and submitting it to Common Lisp Document Repository. It will make everybody's life easier.

New website look

posted on 2016-04-22

I've imported the old archives and genearated ECL website with help of the coleslaw and the sclp. Now we have a proper RSS feed and posting news is less annoying then before.

For posterity, here is the ugly hack I've used to import archives from JSON:

(defparameter *archives-template*
  ";;;;;
title: ~A
tags:
date: ~A
author: ~A
format: md
;;;;;

~A")

(setf *json-posts*
  (with-open-file (f #P"/home/jack/linki/repo/ecl-website/static/files/misc/news-ecl-backup-2015-08-25.json"
                     :direction :input
                     :external-format '(:line-termination :cr :character-encoding :utf-8))
    (cl-json:decode-json f)))

(mapcar (let ((cnt 0))
          #'(lambda (post)
              (with-open-file (f (format nil "/tmp/archives/archive-~A.post" (incf cnt))
                                 :direction :output
                                 :if-exists :supersede
                                 :external-format (make-external-format :line-termination :unix))
                (format f *archives-template*
                        (cdr (assoc :title post))
                        ;; (cdr (assoc :labels post))
                        (substitute #\- #\/
                                    (subseq (cdr (assoc :url post)) 40 47))
                        (let ((author (cdr (assoc :author post))))
                          (if (string-equal author "dkochmanski")
                              "jackdaniel" author))
                        (remove #\Return (cdr (assoc :text post)))))))
        (cdar *json-posts*))

You may find a guide how to use the Sample Common Lisp Project template for your own project here. The clnet theme is inspired by the css in most of the common-lisp.net projects.

Best regards, Daniel

Bountysource Salt account

posted on 2016-03-04

We have set up an account on the bountysource to simplify the financial contributions to the project.

Additionally you may send money contributions with a PayPal account:

ECL 16.1.2 release

posted on 2016-02-29

We are happy to inform that the official ECL 16.1.2 release is available for download:

ECL Quarterly Volume III

posted on 2016-02-06

1 Preface

Dear all,

I'm proud to publish the third ECL Quarterly volume. I'm aware that it's a bit late, but very happy that delay didn't extend any further. Some exciting stuff is going on. First of all Embeddable Common-Lisp has been successfully ported to the Android platform. Additionally we have untested ports to NaCL and PNaCL (any volunteer?) and an alpha quality Android application - ecl-android (fully fledged swank-accessible Common Lisp running in Dalvik via JNI with all the goodies ECL provides).

I'm very happy to open this volume with a great guide written and contributed by Earl Ducaine – Stupid ECL tricks. This is set of a very useful hints for the development with ECL. We invite everyone to contribute to the repository located at Earl's GitHub repository. Moreover Angelo Rossi has finished his project embodying ECL on the embedded board SBC MIPS Creator CI20 and written very nice summary to share with us.

Third chapter will resolve around the Android port and how to build the ecl-android application.

I want to apology for this delay everyone, who waited for this volume. I'm quite busy lately and despite having some unfinished material I couldn't find time to polish it, so I've decided to skip it for the next volume. I'm very grateful to both Earl and Angelo for providing good material for a readers. I hope the next volume will be published very soon, or at least on schedule :-). Thank you for waiting.

We're testing now a new release of ECL (version number 16.1.2) and if everything goes fine it will be released on February 29th. If you are curious about the changes, you may skim the "Pending changes" section here.

*Please* send all the feedback to the mailing list or directly to me. It is really ensuring to know, that someone reads this. Also if you want to be published here please let me know. Thank you!


Daniel Kochmański ;; aka jackdaniel | TurtleWare
Poznań, Poland
February 2016

2 Stupid ECL tricks

Mostly these are half baked hacks. But hopefully they stimulate the imagination of real programmers by providing a glims of what ELC is cabable of.

2.1 Running ECL in gdb.

I've always had a total mental block when it comes to C pointers. It makes no sense to my brain that * indicates a variable is a pointer when used in a declaration, but retrieves a value when used as an operator. And an array of pointers to a character string makes total sense to me in words but char** str[] causes my mind to go blank. As a consequence any C code I write, or even look at too intently immediately blows up when compiled and run. A big inconvenience when embedding Lisp. Replacing the usual,

(setq inferior-lisp-program "ecl")

with,

(setq inferior-lisp-program
      "gdb --eval-command=run --eval-command=quit --args ecl")

Will run ecl under gdb, which will provide you the normal gdb environment with c runtime errors, while throwing you into the lisp debugger for Lisp errors. Note that gdb by default breaks on SIGPWR and SIGXCPU which ecl uses for internal processing. So, you'll also want to add the following to your .gdbinit file.

handle SIGPWR nostop noprint
handle SIGXCPU nostop noprint

2.2 Embedding Swank in a c application.

Swank is a Lisp program that provides remote access to a Lisp instance. It started as client/server application layer in CMUCL and the Hemlock editor it ran. It's since been ported to most Lisps. Slime is the Emacs front-end client to Swank. Together the two tools provide a powerful Lisp development environment in Emacs. The easiest way to install Swank and Slime is simply to get it from quicklisp. See:

https://www.quicklisp.org/beta/

Swank and slime work in following way:

+----------+     launch ecl in                +--------------------+ 
| emacs    |---- process buffer, tell ------> | ecl process buffer |
+----------+     ecl to start swank           +-----+--------------+                         
   |                                                       |
   |                                         start swank server:
create slime                                 (swank-loader:init)
buffer                                       (swank:start-server)
   |                                                |
   |                                                |
  \/                                              \/
+--------------+      integrated      +--------------------------------+
| repl:        +<---- lisp repl   --->| swank server listening         |
| slime buffer |      interaction     | on some arbitrary              |
+--------------+                      | TCP/IP port e.g.               |
                                      | "Swank started at port: 46493" |
                                      +--------------------------------+
                                                      /\
+--------------------------+                           |
| edit:                    +<--------------------------+
| buffer with Lisp source  |
+--------------------------+

To embed swank in a C application we need the application to launch Swank and then for Emacs to establish the connection to the swank server using slime-connect. Below is the C code that launches Swank.

Note, the following example is for a GNU/Linux type system. ecl needs to explicitly load load a shared library in order to access binary symbols such as C functions or C variables in the process, this is a hackish way of handling it since the library was already loaded when the applicaiton started, and could cause problems on platforms that put different constraints on loading shared libraries.

/* -*- mode: c;  -*-
   file: main.c
*/

#include "app_main.h"
/* a.out wrapper for call into a shared library. */
int main() {
  return app_main();
}
/* -*- mode: c;  -*-
   file: app_main.h
*/

#ifndef __APP_MAIN_H__
#define __APP_MAIN_H__

#include <ecl/ecl.h>

int app_main();

#endif /* APP_MAIN_H */

The following creates the shared library app_main used by both the C program and ECL for symbols. The embedded ECL code initializes the ECL environment and calls the Common Lisp load function to load a local Lisp file with the code to run swank.

/* -*- mode: c;  -*-
   file: app_main.c
*/

#include <stdlib.h>
#include <math.h>
#include "app_main.h"

void run_swank();

/* TODO: Are embedded quotes really needed? */
char start_swank[] =
  "\"/mnt/pixel-512/dev/stupid-ecl-tricks-1/start-swank-server.lisp\"";

char* argv;
char** pargv;

int app_main() {
  argv = "app";
  pargv = &argv;

  cl_boot(1, pargv);
  atexit(cl_shutdown);

  /* Set up handler for Lisp errors to prevent buggy Lisp (an */
  /* imposibility, I know!) from killing the app. */
  const cl_env_ptr l_env = ecl_process_env();
  CL_CATCH_ALL_BEGIN(l_env) {
    CL_UNWIND_PROTECT_BEGIN(l_env) {
      run_swank();
    }
    CL_UNWIND_PROTECT_EXIT {}
    CL_UNWIND_PROTECT_END;
  }
  CL_CATCH_ALL_END;

  return 0;

}

void run_swank() {
  cl_object cl_start_swank_path = c_string_to_object(start_swank);
  cl_object cl_load =  ecl_make_symbol("LOAD","CL");
  cl_funcall(2, cl_load, cl_start_swank_path);
  return;
}

The following Lisp file, loaded by appmain, contains a couple of snippets of code I copied from the Emacs Slime client that launches the Swank server. When Swank launches it will print out the socket you can use to connect to it, e.g.

;; Swank started at port: 58252.

you can then connect to it in Emacs using Slime:

M-x slime-connect

;;; -*- mode: lisp ; syntax: ansi-common-lisp -*-

;; standard quicklisp init file, since with be launching ecl without ~/.eclrc
(let ((quicklisp-init (merge-pathnames "quicklisp/setup.lisp"
                                       (user-homedir-pathname))))
  (when (probe-file quicklisp-init)
    (load quicklisp-init)))

(when (probe-file  "/tmp/slime.2565")
  (delete-file "/tmp/slime.2565"))

(load
 "~/quicklisp/dists/quicklisp/software/slime-2.14/swank-loader.lisp"
 :verbose t)

(funcall (read-from-string "swank-loader:init"))
(funcall (read-from-string "swank:start-server")
         "/tmp/slime.2565"))

A quick and dirty script file to build a shared library.

# -*- mode: bash;  -*-


rm -f *.o *.so app

export libs="-lm"

# Note, the -Wl,-R flags will make our shared library available to the
# executable app from the location that it was compiled, rather than
# having to be installed globably or adding the build path to
# LD_LIBRARY_PATH.

export ldflags="-L. -Wl,-R -Wl,."
export cflags="-DGC_LINUX_THREADS -D_REENTRANT -fPIC  -g -pipe -Wall"

gcc $cflags -c app_main.c
gcc -shared -Wl,-soname,libapp_main.so $ldflags -lecl -o libapp_main.so *o $libs
gcc main.c $cflags $ldflags -lapp_main -lecl -o app

To build and run

$ ./build_app.sh
$ ./app

2.3 Troubleshooting compilation problems with ffi:c-inline

ECL provide a facility for embedding C code directly in Lisp code like the following:

(defun c-sin (x)
  (ffi:clines "#include \"ecl/ecl.h\"")
  ;; Whoops!  mathh.h should be math.h
  (ffi:clines "#include <mathh.h>")
  (ffi:clines  "#include \"app_main.h\"")
  (ffi:c-inline (x) (:double) :double "{
@(return 0)= sin(#0);
}" :one-liner nil))

To use this function you need to compile the defun. When you issue the explicit compile,

(compile 'c-sin)

ECL will invoke your underlying C compiler. However, C syntax and header include errors, like we included in the above example, will cause compilation to fail. Unfortunately, ECL doesn't pass along the compilers output. You'll get something like the following:

;;; OPTIMIZE levels: Safety=2, Space=0, Speed=3, Debug=3
;;;
;;; End of Pass 1.
;;; Internal error:
;;;   ** Error code 1 when executing
;;; (RUN-PROGRAM "gcc" ("-I." "-I/usr/local/include/" "-D_GNU_SOURCE" "-D_FILE_OFFSET_BITS=64" "-g" "-O2" "-fPIC" "-D_THREAD_SAFE" "-Dlinux" "-O2" "-c" "/tmp/ecl001QoKf80.c" "-o" "/tmp/ecl001QoKf80.o"))

if you try to recreate the error by invoking the implied shell command:

$ gcc -I. -I/usr/local/include/ -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 \
    -g -O2 -fPIC -D_THREAD_SAFE -Dlinux -O2 -c /tmp/ecl001QoKf8.c \
    -o /tmp/ecl001QoKf80.o

You'll get the error:

gcc: error: /tmp/ecl001QoKf80.c: No such file or directory
gcc: fatal error: no input files
compilation terminated.

Because ECL has already cleaned it from /tmp.

But, ECL has a special variable, compiler::*delete-files* that controls cleaning up c output files. By setting it to nil, (setf compiler::*delete-files* nil) you can troubleshoot compilation errors. Re-running above gcc command on from the Unix shell gives us the following:

In file included from /tmp/ecl001QoKf80.c:6:0:
/tmp/ecl001QoKf80.eclh:8:19: fatal error: mathh.h: No such file or directory
#include <mathh.h>
                ^
compilation terminated.

2.4 Cache Files

Swank and ECL's embedded C in Lisp facility seem to have some issues with caching where compiled C snippets and a Swank images don't get refreshed when they should (at least on GNU/Linux). If you start noticing strange issues with changes to ffi:c-inline not taking effect or Swank having the wrong image, try deleting the following cache files:

rm -rf ~/.cache/common-lisp/ecl-15.2.21-ee989b97-linux-x64
rm -rf ~/.slime

/Earl Ducaine/

3 ADIO project with ECL and a bunch of electronic devices

3.1 Synopsis

This document describe my experience with the SBC MIPS Creator CI20 from Imagination and ecl (Embeddable Common Lisp) interpreter/compiler. The goal is to program a distributed diagnostic/supervision system for DVB transmitter equipment's.

development-board.jpg

3.2 The experience

The Creator CI20 comes with a linux Debian Jessie distro preinstalled on its 8 GB internal flash drive. After upgrading it and install some packages I was ready to deploy ECL. The version choosen for the prototype was the 16.1.0, compilation was really straightforward (I modified a bash script written for SBCL to download, compile and install the git version of ECL) and after 20/30 minutes of code crunching I've got a shiny working ECL. I must say that problems didn't come from ECL itself, but from other packages. As I installed quicklisp soon came troubles, for example: slime does not add the correct architecture to the *features* variable and some other minor problems with iolib too. The Lisp community is the most active and helpful among those living on irc and with their help I've got fixed and ready in short time. The project I'm currently working on is divided in two main areas: the hardware design and test phase and the software one. The first, at this stage, contemplates the design and production of a limited number of ADIO boards (designed by me). More basic an ADIO board is an insulated analog and digital i/o board, it is interfaced with the Creator CI20 via the i2c bus and externally powered. The software part is the more vast part of the project. The idea is to provide a system that is able to interface to external users and feed them with sensors data regarding the state of the DVB apparatuses plus every ADIO + Creator CI20 hold the configuration for the apparatus they're going to monitor/supervise. In addition to that the system should be responsible to warn about anomalies and take some pre-emptive actions just to not make the whole thing blow up. So the Customer was facing a dilemma: Arduino + RS485 and a lot of amulets or rely on a modern and saner idea? My advice for the landing party was: "why do not use a cheap sbc (80 Eur) with an embedded OS powered by a Lisp interpreter?" here are the benefits:

  • ethernet based communication hardware layer is proven to be cost effective and reliable, plus every oses and cheap sbcs support it;
  • an embedded os, in most cases, is better than start a baremetal application: e.g. tcp/ip connections and standard network services like http, ftp, ssh, samba and snmp require a lot time to program, debug and test even if you use on the shelf solutions. on the contrary with an embedded os you've got these things already coded and ready to use, shortening the deployment time of about one order of magnitude (in man hours);
  • using lisp is useful for almost three aspects: portability, remote debugging and last but not least the possibility to create compiled code for mips;
  • possibility to expand hardware and software via external modules.

lcd-panel.jpg

So lisp at last, the first prototype is now running a test code which provide basic functionalities like: a general i2c library to access the linux i2c device via /dev/i2c-*, a simple i2c gpio expander library for the mcp23017 and another library to cope with hitachi compatible 16x2 lcd and a simple looping program which displays some random characters on it. next improvements are ready to be tested too, like a machine learning algorithm to detect anomalies and so on.

3.3 Last words

At this stage, ECL was truly the right choice: first of all the time to develop this small set of libraries was incredibly short compared to c/c++ due to that now I've got a test prototype running on my desk happily showing on a 16x2 display some test patterns in just about one month circa from Lisp environment setup. I must thank the friend PJB for his invaluable help and patience and jackdaniel for his work on ECL and the patience too.

/Angelo Rossi/

4 ECL Android port

I'm happy to announce, that the android port has been merged to the develop branch of the main ECL repository. This work is mainly based on Sylvain Ageneau's work. I've also adjusted and incorporated NaCL and pNaCL patches but these are not tested (yet).

To use ECL fully on the android, user want's to have an application which may be started from within the phone, not in the terminal. That's why I've created a separate project ECL Android which is based on Sylvain's integration example code (further adjusted by Evrim Ulu). For now it is in alpha state and it's discouraged to use it in applications meant for the end user.

The following sections will explain current compilation process of the ECL and how to build ECL Android application and deploy it to the computer.

4.1 Building ECL

ECL may be built as a library. This great feature allows us to dynamically link ECL with arbitrary applications capable of using shared objects. We take advantage of this and the fact, that the android platform allows us to embed shared objects with the .apk and use it with the Java Native Interface (JNI).

To use libecl we first need to cross compile it for the target platform. There is some work pending to simplify the cross compilation process, but for a time being you have to build a host ECL compiler and after that the target library.

Before that, however, you have to build a proper android toolchain. If you don't have a prebuilt one you may use android NDK scripts. Both Android SDK and NDK are a prerequisites and installing them is an exercise for the reader.

export PLATFORM_PREFIX=/opt/toolchains/android-ndk9/
export NDK_PATH=/opt/android-ndk/
export NDK_PLATFORM=android-12

mkdir ${PLATFORM_PREFIX}
/opt/android-ndk/build/tools/make-standalone-toolchain.sh \
    --platform=${NDK_PLATFORM} \
    --install-dir=${PLATFORM_PREFIX} \
    --arch=arm

export PATH=${PLATFORM_PREFIX}/bin:${PATH}

Now you have to build the host compiler1 and the final library for the android. Note, that using the preexisting ECL binary could work if you it is a 32 bit installation with disabled longdouble.

./configure ABI=32 CFLAGS="-m32 -g -O2" LDFLAGS="-m32 -g -O2" \
            --disable-longdouble \
            --prefix=`pwd`/ecl-android-host \
            --enable-libatomic=included

make && make install
export ECL_TO_RUN=`pwd`/ecl-android-host/bin/ecl
rm -r build

./configure --host=arm-linux-androideabi \
            --prefix=`pwd`/ecl-android-target \
            --with-cross-config=`pwd`/src/util/android.cross_config \
            --disable-soname

# You have to adjust build/cross_config to your settings (especially
# set ECL_TO_RUN to your host ecl)
make && make install

You should have libecl.so and the other necessary files in the ecl-android/ directory. Some pre-compiled modules are located in the ecl-android/lib/ecl-16.1.0 directory. You will want them on the target system.

4.2 ECL Android (application/service)

This application isn't stable yet and documentation is still rather scarce. API isn't stable and it is discouraged to base any work on it for now. It is provided on terms of AGPL-3.0+ license, however alternative licensing is possible.

First thing to do is to clone the repository and adjust the project configuration to your local setup.

git clone https://gitlab.common-lisp.net/ecl/ecl-android.git
cd ecl-android
# update the project (sets sdk path and the other android "magic")
android update project -t android-10 -p .
# create symlinks (sets ECL directories). For instance:
ln -s ../ecl-android-target ecl-android
ln -s ecl-android/lib/ecl-*.*.* ecl-libdir

Now it's worth to explain how the application works on the target platform. We put the library libecl.so in the apk file, but rest of the module and other ECL-related files are packed as a resource in the assets/lisp/ directory at which *default-pathname-defaults* points to.

Initialization is performed by assets/lisp/etc/init.lisp file, which loads the user.lisp. The latter contains some sample code loading swank (if it's put in the home/slime-2.14/ directory – swank is not bundled with the repository) and defines auxiliary functions: #'get-quicklisp, #'start-swank and #'stop-swank.

Function #'get-quicklisp will download and install the Quicklisp. It will also replace it's bundled compression tools with a prebuilt one (it is essential for performance – GCC produces faster code then the byte compiler).

Before you build the ecl-android you may want to copy some files for further use and edit the initialization script:

mkdir assets/lisp/home
cp -rf ~/things/slime-2.14 assets/lisp/home
cp ~/things/my-awesome-lisp-app/awesome.lisp assets/lisp/
emacs assets/etc/user.lisp

When you are ready you may build and deploy the application on the phone:

ndk-build
ant debug install

ECL Android launcher should appear on your phone.

Footnotes:

1

If your host platform is darwin, then the host compiler should be built with the Apple's GCC (not the GCC from Macports). Using the MacPort command:

sudo port select --set gcc none

Hint provided by Pascal J. Bourguignon.

ECL Android 0.0.1

posted on 2015-11-07

ECL Android 0.0.1 has been released. Keep in mind that this is an alpha quality software for the preview purposes. Prebuilt apk is based on the ECL repository, not the 16.0.0 release. More in-depth information will be provided in the upcoming ECL Quarterly. Please send a feedback to the mailing list or directly to me.

Enjoy :-).

ECL 16.0.0 release

posted on 2015-09-28

We are happy to inform that the official ECL 16.0.0 release is available for download:

ECL Quarterly Volume II

posted on 2015-09-16

1 Preface

Hello!

Three months has passed (plus a few days - 25 actually) and a new volume of ECL Quarterly arises — as promised.

Highlights of this volume:

Embeddable Common-Lisp 16.0.0 (new release)
A few words about new ECL release
ECL future
Things we consider implementing in the future
On Lisp (not a book!)
Three short essays about unique ECL C/C++ inlining feature (in the form of a tutorial), the ANSI CL specification in the context of LET, FLET and LABELS block, and non ANSI compliant extensions of the reader syntax

I've recently started my own company – TurtleWare. There is a shameless plug at the end of the volume with information that I'm open for consultancy work. Everyone is free to skip the last section.

I also want to remind everyone that the main project site (containing all further resource pointers) is located at:

https://www.common-lisp.net/project/ecl

If you have any suggestions regarding the Quarterly – you like it? or maybe you hate it? – please tell me either by writing to the mailing list or by writing an e-mail directly to me. If you want your own article or tutorial to be published in this e-zin – please drop me an e-mail (daniel[at]turtleware.eu). Thank you!


Daniel Kochmański ;; aka jackdaniel
Poznań, Poland
August 2015

2 Sourceforge migration

Until now we had a few remaining resources kept on Sourceforge:

  • Mailing lists
  • Release source archive
  • Announcements / blog channel

All these have now been moved to the common-lisp.net servers (https://www.common-lisp.net/project/ecl). All members of the mailing list did get an invitation e-mail to the new one. The obsolete list will be shut down at September 1st. Mailing list archives are already imported, so we won't lost any accumulated threads.

I owe a big thanks to Erik Huelsmann for help with the migration and for his constant work on the common-lisp.net platform. Without him, such migration would be somewhere between a nightmare and the impossible. Thanks!

3 Embeddable Common-Lisp 16.0.0 (new release)

3.1 Announcement

We are happy to announce that the new ECL release has been published. Version 16.0.0 has various improvements over the previous one mostly focused on the interoperability with the existing CL libraries ecosystem, pushing forward ANSI compliance and increasing portability across various platforms.

Significant (ongoing) efforts have been made to improve general code quality – removing dead blocks and interfaces, untabifying sources and refactoring parts of the code base. Also we've refreshed the testing framework. Documentation has been verified and updated.

We owe big "thank you" to many people who helped us with understanding the ANSI spec and pointing us in the right direction when in doubt, and to those who bothered to report issues and provide test cases. These discussions took place mostly on IRC and via the "Issues" tab on the GitLab platform. Unfortunately I don't remember all the nicks and names, so it would be unfair to list only a few I remember.

People who have contributed to this release are (alphabetically): Daniel Kochmański, Philipp Mark, Roger Sen and Evrim Ulu.

Without further ado – the changes:

3.1.1 Known issues

  • In Windows, ECL comes with the bytecode compiler by default, because C compilers are normally not available. Unfortunately several libraries out there are not prepared for this. If you plan to use quicklisp and have a C compiler accessible to ECL, you may use (ext:install-c-compiler) to switch back to the Lisp-to-C compiler.
  • In order to test a package, programmer has to install ECL on a desired destination (specified with "–prefix" parameter given to the configure script).

3.1.2 API changes

  • There is no UFFI nickname for the FFI package - we piggyback on cffi-uffi-compat for UFFI dependent systems (our UFFI wasn't version 2.0 compatible and there were problems with ADSF dependencies on UFFI - it wasn't a system)
  • CLOS has the new nickname "MOP"
  • The new ext:run-program :error argument can automatically create a separate stream if provided with the :stream keyword. The external-process structure also has a new field to hold that stream.
  • ext:run-program accepts new arguments - :if-input-does-not-exist, :if-error-exists and :external-format
  • ext:system no longer binds ** standard-outputstandard-input* and *and now ignores input and output (use ext:run-program for more control)
  • methods can be specialized on both single-float and double-float (built-in classes were added for them)
  • LET/FLET/LABELS will signal an error if the parameter of the same name appears multiple times
  • lambda lists with repeated required parameters name are considered invalid
  • deprecated configure options "–with-system-boehm=x" and "–enable-slow-config" removed

3.1.3 Enhancements

  • Verification if manual is up-to-date, providing corrections for outdated parts
  • Documentation is now included in the main repository under the top-level directory `doc'
  • Update libffi to version 3.2.1
  • Update asdf to version 3.1.5.4
  • Update Boehm-Demers-Weiser garbage collector to version 7.4.2
  • Pathname string-parts internal representation is now character, not base-char
  • Dead code removal, tabulators were replaced by spaces
  • Better quality of generated code (explicit casting when necessary)

3.1.4 Issues fixed

  • Various fixes of bogus declarations
  • Remove deprecated GC calls
  • ROTATEF, SHIFTF, PSETF reworked to conform to the ANSI standard. Places were handled improperly in regard of multiple values.
  • Improved Unicode support in character handling
  • Format handles floats and exponentials correctly (major format rework)
  • Stack limits refinements and cleanup, inconsistency and bug fixes
  • Duplicate large block deallocation with GMP 6.0.0a fixed
  • ECL builds on OpenBSD with threads enabled
  • Closures put in mapcar work as expected in both compiled and interpreted code
  • Improved readtable-case handling (:invert and character literals now conform)
  • Library initialization functions have unique names - no risk of clashing symbol names in object files
  • Format float bug fixed, when width and fdigits were not set, but k was
  • `logical-pathname-translations' now signals an error if logical pathname wasn't defined yet, to conform with ANSI (it used to return NIL)
  • Wildcards in logical pathname translations are replaced correctly
  • Regression testing framework and unit tests cleanup
  • deftype ANSI conformity fix (deftype accepts macro labda-lists)
  • ECL built with MSVC doesn't crash when Control-C is pressed
  • Other minor tweaks

3.2 New version numbering scheme

The data-based version numbering scheme is ceased from this release. From now on all the releases will follow the rules described in https://autotools.io/libtool/version.html.

Basically release numbers will follow the scheme X.Y.Z, where X increases when the API changes, Y if interfaces are added (but not removed) or changed in mostly backward compatible and Z is the "patch level" part, which changes for fixes not affecting API.

All new releases are considered ABI incompatible, so the sources have to be recompiled with each new release (this is the default when using ASDF).

4 ECL future

There is plenty of work to be done and many ideas to realize. We can't do it all at the same time and all the further points are rather loose ideas than plans for the next release. Just a food for thought.

  • Dynamic Foreign Function Interface

    ECL has a decent DFFI for a small set of platforms (x86, x8664 and PPC32) hand coded with assembly. ECL on other platforms doesn't support DFFI with the bytecode compiler and requires the C backend (function calls are inlined in the generated C code). On the other hand, we already depend on libffi with regard to closures. We can use it for DFFI as well – it has a really impressive list of supported platforms and using it for that will be a big win.

  • Cross-compilation framework

    We are able to cross-compile applications with host the ECL and GCC toolchain, but this is somewhat painful and we have no convenient interface for doing that at run-time from the REPL user perspective. Compilation flags and parameters are stored as global variables and API for cross compilation is documented.

    Juan Jose Garcia Ripoll started to make the target machine description separate from the host implementation. On the other hand Sylvain Ageneau created the cross-cmp compiler package. I haven't investigated either work in great detail, but I see an opportunity to combine both these works for greater support for the cross comilation. Sylvain's repository (https://github.com/ageneau/ecl-android) seems to be a place full of real treasures worth merging back to the main repository.

    Cross-compiling applications for Android to native code is definitely something I could put to good use ;-)

  • Documentation improvements

    Many exported functions and interfaces are undocumented or existing documentation is scarce. More usage examples and verification of "examples/" src directory would be a good thing too. The official manual layout could be refined to allow html "one-page" builds (source is in DocBook).

  • C compiler on-board

    Providing an optional C compiler package to be bundled as fallback with ECL would be a nice thing to have. Both GCC and TCC can be built as libraries. Didn't investigate it much yet.

  • More platforms

    The repository mentioned above ("ecl-android") isn't only about Android support. It has iOS and NaCL ports too. Creating a Minix port would be rather easy – only the bdwgc port is missing, and porting it would be as easy as adding two defines since it's advertised as "API compatible" with NetBSD which is already supported.

  • Introspection facilities

    ECL has a few problems when it comes to introspection. While it works nicely with SLIME when it comes to the user-defined functions, many functions defined at ECL build time don't have any hints. It's even worse when it comes to macros – no hints at all. This issues have to be addressed and fixed by fixing ECL interfaces and improving SLIME integration. It will also create a good background for writing simplified (not emacs-dependant) IDE (read below).

  • Graphical integrated development environment

    This should work via Swank backend and must be easy to grasp for a programming beginners. I truly believe that Emacs (while being editor of my choice) doesn't have to be the only solution for CL programmers who use free implementations. It's a big entrance barrier which people often fail to conquer.

    I imagine it being written in EQL and supporting Swank, so any implementation can be attached. Some ideas from Bret Victor's "Learnable Programming" might be incorporated, especially contextual information in the case when documentation is in a specific format (for instance providing description for each parameter).

    Decent support from line-edit would also be nice for quick hacking from the console – auto-completion and hints are a killer feature here. Imagine BPython (it's ncurses).

  • Improving existing interfaces

    Gray streams and MOP has a few remaining issues (implementation isn't fully conforming). For instance: Gray Streams #'close isn't a method but function, MOP's complex methods are terribly slow or even non-functional. This issues has to be addressed.

  • Compiler thread-safety

    ECL compiler isn't thread safe (at least that's what I've been told). Verifying and fixing it is one of the things to be done.

  • Central Document Repository

    We plan to incorporate suitable extensions into ECL. Starting from CDR-14 (we already have CDR-5). It's a nice place for quasi-standard proposals and we think it's a good way to achieve consensus among various CL implementations about a common API for future extensions to allow portability.

    We know many facilities are practically standardized by portability libraries (ASDF, Bordeaux Threads, CFFI etc.), but future extensions might have an actual specification – we all know warts of "reference implementation" standards *cough-python-cough*.

    I would love to see "Extensible Sequences" in the Central Document Repository.

5 On Lisp (not a book!)

5.1 Inlining C/C++ code in Common-Lisp with ECL

Like many compilers ECL offers a facility to inline it's assembly in the source code for a convenience and performance reasons. ECL's assembler is C/C++°.

To achieve inlining, ECL offers three constructs.

(ffi:clines                           &body strings)
(ffi:c-inline args arg-types ret-type &body others)
(ffi:c-progn  args                    &body forms-and-strings)

The most basic is ffi:clines, which allows you to just drop in some C/C++ code (for instance include a header you need for further use).

ffi:c-inline is much more useful, allowing you to pass values from Lisp to the block and receive output in return. This construct allows returning multiple values declared in the third clause. You may also declare whenever it has side effects and if it is one-liner°° (more details in ECL manual). Short example:

(defun ctrunc (number divisor)
  (ffi:c-inline (number divisor) (:int :int) (values :int :int) "{
    int num = #0, div = #1;
    @(return 0) = num/div;
    @(return 1) = num%div;
}"))

It's worth mentioning that the number and divisor types are checked when the function #'ctrunc is called and if incorrect – proper lisp condition is signaled.

ffi:c-progn is the last construct. It allows you to intermix both C and lisp code – it doesn't return any value so it is called purely for it's side-effects (assigning variable with computed result for instance). Don't forget to declare variable types! If you don't, results might be surprising.

To illustrate potential gain of using inlined C language we'll take the trivial Fibbonachi algorithm and benchmark ECL against itself. It's not a proper benchmark, nor a good implementation, but both implementations are comparable and this should prove a point, that ECL might be quite fast when we want it to be°°°.

(defun fib-1 (n)
  "Borrowed from http://www.cliki.net/fibonacci"
  (loop for f1 = 0 then f2
     and f2 = 1 then (+ f1 f2)
     repeat n finally (return f1)))

(defun fib-2 (n)
  (let ((f1 0) (f2 1) (n n))
    (declare (:int f1 f2 n))
    (ffi:c-progn (n f1 f2) "
     int aux = 0;
     for( ; #0>0; #0--, aux=#1) {
         #1 = #2;
         #2 = aux + #2;
     }")
    f1))

(defun bench ()
  (compile 'fib-1)
  (compile 'fib-2)
  (prog1 nil
    (print "Common Lisp:")
    (time (dotimes (x 10000000) (fib-1 20)))
    (print "Common Lisp with inlined C:")
    (time (dotimes (x 10000000) (fib-2 20)))))

On my computer This yields:

"Common Lisp:" real time : 9.583 secs run time : 9.596 secs gc count : 1 times consed : 271531840 bytes

"Common Lisp with inlined C:" real time : 0.657 secs run time : 0.656 secs gc count : 1 times consed : 271567088 bytes

So, as we can see, the speed improvement is pretty decent. Proper declarations would speed up #'fib-1 a little, but the C version will still be faster (note the fact that we are comparing ECL with ECL, other implementations might theoretically outperform ECL's C version using provided #'fib-1 definition).

For more information please consult manual: https://common-lisp.net/project/ecl/static/manual/ch28.html

° None of these constructs will work with the bytecode compiler (it will signal an error).

°° C doesn't treat each statement as a valid R-value. By "one-liner" we mean something, what might be used as such.

°°° We can't say it's fast Common Lisp, since it's no longer Common Lisp – if we use this technique we're using ECL and it's not portable by any means.

5.2 Case on LET / FLET / LABELS (aka Nasal Demons Reborn)

Consider an example

(let ((x 'foo)
      (x 'bar))
  x)

what symbol does this form evaluate to: **BARFOO* or *?

Well, it's not defined – in CCL and CLISP it is **FOOBAR*, ECL and ABCL make it *, while SBCL signals an error. It isn't clear what should happen because it's not specified in the spec. I think that the SBCL approach is the most sane. I can hardly imagine a programmer doing that, not as a typo, but as a conscious decision°.

The same argument applies to **FLET* FLET*FLET* (except the side-effect thingy – all but one definition of the same name can be optimized out) – which function definition is the valid one? It isn't specified – note that the operator name isn't *but *.

The last case – the most dangerous and the least comprehensible. *LABELS* allows mutual recursion and it's far less obvious what would happen even from an implementation point of view. If we write a code:

(labels ((function1 ()
           (im-dangerous 2))
         (im-dangerous (x)
           (format t "FIRST X=~A~%" x)
           (if (zerop x)
               'first
               (im-dangerous (1- x))))
         (function2 ()
           (im-dangerous 2))
         (im-dangerous (x)
           (format t "SECOND X=~A~%" x)
           (if (zerop x)
               'second
               (im-dangerous (1- x))))
         (function3 ()
           (im-dangerous 2)))
  (list (function1)
        (function2)
        (function3)
        (im-dangerous 1)))

CLISP take the first definition of ** IM-DANGEROUSIM-DANGEROUS*, while others the second one. SBCL on the other hand behaves inconsistently – inside function definitions references to *are bound to the first definition, while references from inner block are bound to the second one.

It is important to say that the behavior in these situations is undefined by the spec and each implementation is free to do what it considers most reasonable, easiest to implement, or best to optimize – and none of these is wrong (CCL allows all constructs, but issues a style warning for each – it is the only implementation which does that – bravo).

Curious minds may find the following article amusing (warning, C code involved): http://blogs.msdn.com/b/oldnewthing/archive/2014/06/27/10537746.aspx.

My conclusion is as follows: you can't rely on ** LABELS* FLETLET*, *and *constructs if multiple definitions of the same name exist – they are ambiguous and should be considered being an error. The new release of ECL treats them that way°°.

° Multiple variables can't be optimized out, because the initialization form might have side effects

(let ((x (side-effects1))
      (x (side-effects2)))
  x)

;; More deterministic form:
(let ((x (progn (side-effects1)
                (side-effects2))))
  x)

°° ANSI defines LAMBDA-LIST in terms of LET*, so repeating the parameters of the same name is theoretically correct. There is no valid use-case for two required parameters of the same name (there is no initialization form) though, so ECL signals an error on such situation.

5.3 Case for portability and against unusual reader syntax

Why we shouldn't use all these implementation-specific niceties

Lately, Zach Bane posted a nifty trick for accessing symbols from packages you're not in, literally:

foo::(list 'these 'symbols 'are 'from "foo" 'package)

It's nice, intuitive, practical… and not compliant. It would be a minor problem, if it weren't a syntax hack, which is hard to implement portably as a library (at least I don't see any elegant solution). In my opinion it requires digging into implementation innards and tweaking the reader to support it. Manipulating strings before they are actually read is an option too. Also – unlike the metaobject protocol, gray streams or extensible sequences – it doesn't bring anything new to the table.

"But hey! It's just syntactic sugar for REPL interaction! Don't be such a grumpy guy!"

OK, fine, if you promise me that this "syntactic sugar" won't land in any library *ever* – it's advertised after all. Nobody knows who will pick up this hint. And if critical mass will prevail, then this dubious syntactic sugar will become de-facto standard, so other implementations will be forced to implement it, or fail to load libraries using it.

Such a situation happened once. CLtL had an example of the ** UNTIL* FORLOOP* usage, where a *clause landed after an *clause, which isn't ANSI compliant. The fact that it was supported by a few implementations lead to the situation imagined above.

If you really want some syntactic sugar for using other packages locally – I propose a little uglier, yet portable, solution:

(defun sharp-l (stream char subchar)
  (declare (ignore char subchar))
  (let ((*package* (find-package (read stream))))
    (read stream nil nil t)))

(set-dispatch-macro-character #\# #\l #'sharp-l)

#l foo (list 'these 'symbols 'are 'from "foo" 'package)

It's only five lines of code and works everywhere (unless someone binds #l to his own reader macro). If someone wants to be a little fancier, then he may mimic the SLIME prompt in his syntax:

(defun sharp-x (stream char subchar)
  (declare (ignore char subchar))
  (let ((*readtable* (copy-readtable))
        (right #\))
        (rpar #\>))
    (set-macro-character rpar (get-macro-character right))
    (let ((*package* (find-package (read stream))))
      (peek-char #\> stream)
      (read-char stream)
      (read stream nil nil t))))

(set-dispatch-macro-character #\# #\[ #'sharp-x)

#[foo> (list 'these 'symbols 'are 'from "foo" 'package)

And, honestly I really like proposed syntax, but for my taste it's totally unportable and harmful for reasons I've mentioned above.

6 Advertisement

Here comes the shameless plug – I've recently (officially) launched the company named TurtleWare. I am open for consultancy. If anyone:

  • wants to pay for a dreamed ECL feature,
  • prioritizes his own issues with implementation,
  • likes to pay for support or system maintenance,
  • has some Lisp and/or embedded systems work to do

then reach me with further details at: hello[at]turtleware.eu.

ECL Quarterly Volume I

posted on 2015-05-01

1 Preface

Hello everyone!

From time to time there are misleading signals, that ECL is "floating dead". This FUD is mostly spread by IRC newcomers with little orientation, but it's good! This is a signal, that this particular implementation of Common Lisp needs to make little more noise, instead of silently fixing ** stuffstuff* (however it obviously needs a bit of work on *as well ;-)) .

Some projects make new release each month to prevent such disinformation. It solves this particular problem, but introduces a new one - it's a bit fuzzy, if a new release introduces any significant improvements, or if it's just a release number bump. Yet, updating it requires recompilation of personal projects, which depending on machine might take a while and is surely a bit annoying.

This is where ECL Quarterly idea has born. To show a bit activity from time to time - post will be published every three months. I want to make it an e-zine containing info about development, tutorials, comparisons, benchmarks and anything, what is at least slightly related to ECL. Everyone is welcome to contribute - such materials will be also published on wiki, added to git tree, and if appropriate - incorporated in documentation. If you have such material, don't hesitate to contact me at dkochmanski[at]turtle-solutions[dot]eu.

This chapter highlights:

  • Changes, development status etc.
  • Contributing to ECL

If you have any suggestions regarding ECL Quarterly, you like it, or maybe hate it - please tell me either by commenting this post, or writing an e-mail. Thank you!


Daniel Kochmański
Poznań, Poland
May 2015

2 Changes and development status

After 15.3.7 release there are a few changes going on. We are now hosted at gitlab.com (however it would be nice to move to gitlab.common-lisp.net, just /not now/), a few wiki entries are added, official site is https://common-lisp.net/project/ecl/. Domain embeddable-common-lisp.net is already bought (not configured properly yet).

Until now we have updated libraries, which we depend on. Namely:

  • libffi to version 3.2.1,
  • asdf to version 3.1.4,
  • bdwgc to version 7.4.2.

There was also update of GMP to more recent version, but we won't include it - yes, we are staying LGPLv2, to remain GPLv2 compatible. All changes were tested on Windows with MSVC2012.

For now we have also reworked ROTATEF, SHIFTF and PSETF macros, to conform ANSI standard in regard to places (multiple values weren't held properly).

Experimental branch contains Android (as name suggests - experimental) port, which is preview only for now (this branch is LGPLv3, because included GMP enforces it, and doesn't build on Windows).

Loads of things are to be done, and we're slowly making progress. After all we won't change release versioning scheme. No ETA for next release, but I assure you - if you contribute, it will be sooner ;-). What leads us to the next part.

3 Contributing to ECL

3.1 Documentation, tutorials, wiki, others

You can contribute in numerous ways. One way is to code and fix bugs - and it's very important part. But equally important is to find bugs and report them, so developers can improve codebase. Feature requests are also welcome (however we are focused now on fixing stuff rather than adding functionality, so even good proposition might wait in queue behind something maybe less exciting).

Testing on various platform and architectures is essential. Thanks to Anton Vodonosov and cl-test-grid it is as easy as setting environment, tweaking two configuration files and invoking ./run-agent.sh. Run may take a while (depending on computer) and is limited to operating systems and architectures supported by Clozure Common Lisp.

If ECL doesn't build, crashes, or works non-conforming to specification, please do report at https://gitlab.com/embeddable-common-lisp/ecl/issues. It requires account on GitLab, but setting it up is free. I'm still struggling to produce some time and move all tickets from https://sourceforge.net/p/ecls/_list/tickets to above mentioned issue tracking site - volunteers are more then welcome to do it.

If you encounter problem, please write to mailing list. There are many kind souls there more then willing to help - and usually they do. On the other hand, if someone asks for help and you know the answer - act! :-)

Wiki is a place, where many resources are gathered. It is incomplete and barely usable, because I don't have time to improve it. If you successfully build ECL on Android - great! Please add page to wiki, so others may reproduce your work! Do you have some nifty idea, and you believe it is worth to keep it there - do it. See a typo? Bug? Outdated information? You know what to do! You spot some nice blog post about ECL? Please share with others - wiki is best place to do so. You did successful project using ECL? Share this information! All this, and so much more, may be done here: https://gitlab.com/embeddable-common-lisp/ecl/wikis/home.

You can also write something for ECL Quarterly (my e-mail is mentioned in preface).

3.2 Source code contributions

Development takes place on git repository located at git@gitlab.com:embeddable-common-lisp/ecl.git.

If you want your code in Embeddable Common-Lisp project, please send a patch to mailing list with additional tag [PATCH] in subject. Generally we want to follow convention used by U-Boot development team (http://www.denx.de/wiki/U-Boot/Patches), which borrows a lot from Linux policy. Please read this guide, it's really worthwhile reading. If you submit a significant change, please report it in CHANGELOG located in top directory.

Basically, if you want to contribute to code, you have at least two choices. You may improve C code (which is probably less interesting for lispers). Most of the sources are written in lisp however, and require no knowledge about C. Problems vary from fairly easy even for seasoned developer to hard conceptual riddles, so everyone can find something interesting for himself.. Please consult appropriate directories under src/ sub-directory (i.e. src/lsp, src/clos etc.) - it's pure lisp! And it's fun to hack. Improving both C and Lisp sources might be a great learning experience. To figure what's wrong requires often getting your hands dirty, and then cleaning proposed solution. This, connected with peer review, might be a next step to become a better programmer.

There is also a third part, which is tiresome (for me at least) - improving build system - it is buggy and inconsistent. First person who will fix this has granted a free dinner when we meet in person. Remember however, that we support many operating systems, so it might be tricky to do it properly without introducing *new* problems.

If you are a library or application developer, _please_ test against as many implementations as possible - it's hard, it takes time, but in my humble opinion it is essential for CL ecosystem. Each implementation has it's weak and strong sides, and you never know, when you'll need one, or who and in what purpose is using your code :-).

ECL Quarterly - Volume I

posted on 2015-05

ECL Quarterly - Volume I

Preface

Hello everyone!

From time to time there are misleading signals, that ECL is "floating dead". This FUD is mostly spread by IRC newcomers with little orientation, but it's good! This is a signal, that this particular implementation of Common Lisp needs to make a bit more noise, instead of silently fixing stuff (however it obviously needs a bit of work on stuff as well ;-)).

Some projects make new release each month to prevent such disinformation. It solves this particular problem, but introduces a new one - it's a bit fuzzy, if a new release introduces any significant improvements, or if it's just a release number bump. Yet, updating it requires recompilation of personal projects, which depending on machine might take a while and is surely a bit annoying.

This is how the ECL Quarterly idea was born. To show a bit activity from time to time - post will be published every three months. I want to make it an e-zine containing info about development, tutorials, comparisons, benchmarks and anything what is at least slightly related to ECL. Everyone is welcome to contribute - such material will also be published on wiki, added to git tree, and if appropriate - incorporated in documentation. If you have such material, don't hesitate to contact me at dkochmanski[at]turtle-solutions[dot]eu.

This chapter highlights:

  • Changes, development status etc.
  • Contributing to ECL

If you have any suggestions regarding ECL Quarterly, you like it, or maybe hate it - please tell me either by commenting this post, or writing an e-mail. Thank you!

Daniel Kochmaski Pozna, Poland May 2015

Changes and development status

After the 15.3.7 release there are a few changes going on. We are now hosted at gitlab.com (however it would be nice to move to gitlab.common-lisp.net, just not now), a few wiki entries are added, official site is https://common-lisp.net/project/ecl/. Domain embeddable-common-lisp.net is already bought (not configured properly yet).

Until now we have updated libraries that we depend on. Namely:

  • libffi to version 3.2.1,
  • asdf to version 3.1.4,
  • bdwgc to version 7.4.2.

There was also an update of GMP to more recent version, but we won't include it - yes, we are staying LGPLv2, to remain GPLv2 compatible. All changes were tested on Windows with MSVC2012.

For now we have also reworked the ROTATEF, SHIFTF and PSETF macros, to conform ANSI standard in regard to places (multiple values weren't held properly).

The experimental branch contains Android (as name suggests - experimental) port, which is preview only for now (this branch is LGPLv3, because included GMP enforces it, and doesn't build on Windows).

Loads of things are to be done, and we're slowly making progress. After all we won't change release versioning scheme. No ETA for next release, but I assure you - if you contribute, it will be sooner ;-). What leads us to the next part.

Contributing to ECL

Documentation, tutorials, wiki, others

You can contribute in numerous ways. One way is to code and fix bugs - and it's very important part. But equally important is to find bugs and report them, so developers can improve codebase. Feature requests are also welcome (however we are focused now on fixing stuff rather than adding functionality, so even good proposition might wait in queue behind something maybe less exciting).

Testing on various platform and architectures is essential. Thanks to Anton Vodonosov and cl-test-grid it is as easy as setting environment, tweaking two configuration files and invoking ./run-agent.sh. Run may take a while (depending on computer) and is limited to operating systems and architectures supported by Clozure Common Lisp.

If ECL doesn't build, crashes, or works non-conforming to specification, please do report at https://gitlab.com/embeddable-common-lisp/ecl/issues. It requires account on GitLab, but setting it up is free. I'm still struggling to produce some time and move all tickets from https://sourceforge.net/p/ecls/_list/tickets to above mentioned issue tracking site - volunteers are more then welcome to do it.

If you encounter problem, please write to mailing list. There are many kind souls there more then willing to help - and usually they do. On the other hand, if someone asks for help and you know the answer - act! :-)

Wiki is a place, where many resources are gathered. It is incomplete and barely usable, because I don't have time to improve it. If you successfully build ECL on Android - great! Please add page to wiki, so others may reproduce your work! Do you have some nifty idea, and you believe it is worth to keep it there - do it. See a typo? Bug? Outdated information? You know what to do! You spot some nice blog post about ECL? Please share with others - wiki is best place to do so. You did successful project using ECL? Share this information! All this, and so much more, may be done here: https://gitlab.com/embeddable-common-lisp/ecl/wikis/home.

You can also write something for ECL Quarterly (e-mail me at dkochmanski[at]turtle-solutions[dot]eu).

Source code contributions

Development takes place on git repository located at git@gitlab.com:embeddable-common-lisp/ecl.git.

If you want your code in Embeddable Common-Lisp project, please send a patch to mailing list with additional tag [PATCH] in subject. Generally we want to follow convention used by U-Boot development team (http://www.denx.de/wiki/U-Boot/Patches), which borrows a lot from Linux policy. Please read this guide, it's really worthwhile reading. If you submit a significant change, please report it in CHANGELOG located in top directory.

Basically, if you want to contribute to code, you have at least two choices. You may improve C code (which is probably less interesting for lispers). Most of the sources are written in lisp however, and require no knowledge about C. Problems vary from fairly easy even for seasoned developer to hard conceptual riddles, so everyone can find something interesting for himself.. Please consult appropriate directories under src/ sub-directory (i.e. src/lsp, src/clos etc.) - it's pure lisp! And it's fun to hack. Improving both C and Lisp sources might be a great learning experience. To figure what's wrong requires often getting your hands dirty, and then cleaning proposed solution. This, connected with peer review, might be a next step to become a better programmer.

There is also a third part, which is tiresome (for me at least) - improving build system - it is buggy and inconsistent. First person who will fix this has granted a free dinner when we meet in person. Remember however, that we support many operating systems, so it might be tricky to do it properly without introducing new problems.

If you are a library or application developer, please test against as many implementations as possible - its hard, it takes time, but in my humble opinion it is essential for CL ecosystem. Each implementation has it's weak and strong sides, and you never know, when you'll need one, or who and in what purpose is using your code :-).