Embeddable Common-Lisp

Content from 2015-09

ECL 16.0.0 release

posted on 2015-09-28

We are happy to inform that the official ECL 16.0.0 release is available for download:

ECL Quarterly Volume II

posted on 2015-09-16

1 Preface

Hello!

Three months has passed (plus a few days - 25 actually) and a new volume of ECL Quarterly arises — as promised.

Highlights of this volume:

Embeddable Common-Lisp 16.0.0 (new release)
A few words about new ECL release
ECL future
Things we consider implementing in the future
On Lisp (not a book!)
Three short essays about unique ECL C/C++ inlining feature (in the form of a tutorial), the ANSI CL specification in the context of LET, FLET and LABELS block, and non ANSI compliant extensions of the reader syntax

I've recently started my own company – TurtleWare. There is a shameless plug at the end of the volume with information that I'm open for consultancy work. Everyone is free to skip the last section.

I also want to remind everyone that the main project site (containing all further resource pointers) is located at:

https://www.common-lisp.net/project/ecl

If you have any suggestions regarding the Quarterly – you like it? or maybe you hate it? – please tell me either by writing to the mailing list or by writing an e-mail directly to me. If you want your own article or tutorial to be published in this e-zin – please drop me an e-mail (daniel[at]turtleware.eu). Thank you!


Daniel Kochmański ;; aka jackdaniel
Poznań, Poland
August 2015

2 Sourceforge migration

Until now we had a few remaining resources kept on Sourceforge:

  • Mailing lists
  • Release source archive
  • Announcements / blog channel

All these have now been moved to the common-lisp.net servers (https://www.common-lisp.net/project/ecl). All members of the mailing list did get an invitation e-mail to the new one. The obsolete list will be shut down at September 1st. Mailing list archives are already imported, so we won't lost any accumulated threads.

I owe a big thanks to Erik Huelsmann for help with the migration and for his constant work on the common-lisp.net platform. Without him, such migration would be somewhere between a nightmare and the impossible. Thanks!

3 Embeddable Common-Lisp 16.0.0 (new release)

3.1 Announcement

We are happy to announce that the new ECL release has been published. Version 16.0.0 has various improvements over the previous one mostly focused on the interoperability with the existing CL libraries ecosystem, pushing forward ANSI compliance and increasing portability across various platforms.

Significant (ongoing) efforts have been made to improve general code quality – removing dead blocks and interfaces, untabifying sources and refactoring parts of the code base. Also we've refreshed the testing framework. Documentation has been verified and updated.

We owe big "thank you" to many people who helped us with understanding the ANSI spec and pointing us in the right direction when in doubt, and to those who bothered to report issues and provide test cases. These discussions took place mostly on IRC and via the "Issues" tab on the GitLab platform. Unfortunately I don't remember all the nicks and names, so it would be unfair to list only a few I remember.

People who have contributed to this release are (alphabetically): Daniel Kochmański, Philipp Mark, Roger Sen and Evrim Ulu.

Without further ado – the changes:

3.1.1 Known issues

  • In Windows, ECL comes with the bytecode compiler by default, because C compilers are normally not available. Unfortunately several libraries out there are not prepared for this. If you plan to use quicklisp and have a C compiler accessible to ECL, you may use (ext:install-c-compiler) to switch back to the Lisp-to-C compiler.
  • In order to test a package, programmer has to install ECL on a desired destination (specified with "–prefix" parameter given to the configure script).

3.1.2 API changes

  • There is no UFFI nickname for the FFI package - we piggyback on cffi-uffi-compat for UFFI dependent systems (our UFFI wasn't version 2.0 compatible and there were problems with ADSF dependencies on UFFI - it wasn't a system)
  • CLOS has the new nickname "MOP"
  • The new ext:run-program :error argument can automatically create a separate stream if provided with the :stream keyword. The external-process structure also has a new field to hold that stream.
  • ext:run-program accepts new arguments - :if-input-does-not-exist, :if-error-exists and :external-format
  • ext:system no longer binds ** standard-outputstandard-input* and *and now ignores input and output (use ext:run-program for more control)
  • methods can be specialized on both single-float and double-float (built-in classes were added for them)
  • LET/FLET/LABELS will signal an error if the parameter of the same name appears multiple times
  • lambda lists with repeated required parameters name are considered invalid
  • deprecated configure options "–with-system-boehm=x" and "–enable-slow-config" removed

3.1.3 Enhancements

  • Verification if manual is up-to-date, providing corrections for outdated parts
  • Documentation is now included in the main repository under the top-level directory `doc'
  • Update libffi to version 3.2.1
  • Update asdf to version 3.1.5.4
  • Update Boehm-Demers-Weiser garbage collector to version 7.4.2
  • Pathname string-parts internal representation is now character, not base-char
  • Dead code removal, tabulators were replaced by spaces
  • Better quality of generated code (explicit casting when necessary)

3.1.4 Issues fixed

  • Various fixes of bogus declarations
  • Remove deprecated GC calls
  • ROTATEF, SHIFTF, PSETF reworked to conform to the ANSI standard. Places were handled improperly in regard of multiple values.
  • Improved Unicode support in character handling
  • Format handles floats and exponentials correctly (major format rework)
  • Stack limits refinements and cleanup, inconsistency and bug fixes
  • Duplicate large block deallocation with GMP 6.0.0a fixed
  • ECL builds on OpenBSD with threads enabled
  • Closures put in mapcar work as expected in both compiled and interpreted code
  • Improved readtable-case handling (:invert and character literals now conform)
  • Library initialization functions have unique names - no risk of clashing symbol names in object files
  • Format float bug fixed, when width and fdigits were not set, but k was
  • `logical-pathname-translations' now signals an error if logical pathname wasn't defined yet, to conform with ANSI (it used to return NIL)
  • Wildcards in logical pathname translations are replaced correctly
  • Regression testing framework and unit tests cleanup
  • deftype ANSI conformity fix (deftype accepts macro labda-lists)
  • ECL built with MSVC doesn't crash when Control-C is pressed
  • Other minor tweaks

3.2 New version numbering scheme

The data-based version numbering scheme is ceased from this release. From now on all the releases will follow the rules described in https://autotools.io/libtool/version.html.

Basically release numbers will follow the scheme X.Y.Z, where X increases when the API changes, Y if interfaces are added (but not removed) or changed in mostly backward compatible and Z is the "patch level" part, which changes for fixes not affecting API.

All new releases are considered ABI incompatible, so the sources have to be recompiled with each new release (this is the default when using ASDF).

4 ECL future

There is plenty of work to be done and many ideas to realize. We can't do it all at the same time and all the further points are rather loose ideas than plans for the next release. Just a food for thought.

  • Dynamic Foreign Function Interface

    ECL has a decent DFFI for a small set of platforms (x86, x8664 and PPC32) hand coded with assembly. ECL on other platforms doesn't support DFFI with the bytecode compiler and requires the C backend (function calls are inlined in the generated C code). On the other hand, we already depend on libffi with regard to closures. We can use it for DFFI as well – it has a really impressive list of supported platforms and using it for that will be a big win.

  • Cross-compilation framework

    We are able to cross-compile applications with host the ECL and GCC toolchain, but this is somewhat painful and we have no convenient interface for doing that at run-time from the REPL user perspective. Compilation flags and parameters are stored as global variables and API for cross compilation is documented.

    Juan Jose Garcia Ripoll started to make the target machine description separate from the host implementation. On the other hand Sylvain Ageneau created the cross-cmp compiler package. I haven't investigated either work in great detail, but I see an opportunity to combine both these works for greater support for the cross comilation. Sylvain's repository (https://github.com/ageneau/ecl-android) seems to be a place full of real treasures worth merging back to the main repository.

    Cross-compiling applications for Android to native code is definitely something I could put to good use ;-)

  • Documentation improvements

    Many exported functions and interfaces are undocumented or existing documentation is scarce. More usage examples and verification of "examples/" src directory would be a good thing too. The official manual layout could be refined to allow html "one-page" builds (source is in DocBook).

  • C compiler on-board

    Providing an optional C compiler package to be bundled as fallback with ECL would be a nice thing to have. Both GCC and TCC can be built as libraries. Didn't investigate it much yet.

  • More platforms

    The repository mentioned above ("ecl-android") isn't only about Android support. It has iOS and NaCL ports too. Creating a Minix port would be rather easy – only the bdwgc port is missing, and porting it would be as easy as adding two defines since it's advertised as "API compatible" with NetBSD which is already supported.

  • Introspection facilities

    ECL has a few problems when it comes to introspection. While it works nicely with SLIME when it comes to the user-defined functions, many functions defined at ECL build time don't have any hints. It's even worse when it comes to macros – no hints at all. This issues have to be addressed and fixed by fixing ECL interfaces and improving SLIME integration. It will also create a good background for writing simplified (not emacs-dependant) IDE (read below).

  • Graphical integrated development environment

    This should work via Swank backend and must be easy to grasp for a programming beginners. I truly believe that Emacs (while being editor of my choice) doesn't have to be the only solution for CL programmers who use free implementations. It's a big entrance barrier which people often fail to conquer.

    I imagine it being written in EQL and supporting Swank, so any implementation can be attached. Some ideas from Bret Victor's "Learnable Programming" might be incorporated, especially contextual information in the case when documentation is in a specific format (for instance providing description for each parameter).

    Decent support from line-edit would also be nice for quick hacking from the console – auto-completion and hints are a killer feature here. Imagine BPython (it's ncurses).

  • Improving existing interfaces

    Gray streams and MOP has a few remaining issues (implementation isn't fully conforming). For instance: Gray Streams #'close isn't a method but function, MOP's complex methods are terribly slow or even non-functional. This issues has to be addressed.

  • Compiler thread-safety

    ECL compiler isn't thread safe (at least that's what I've been told). Verifying and fixing it is one of the things to be done.

  • Central Document Repository

    We plan to incorporate suitable extensions into ECL. Starting from CDR-14 (we already have CDR-5). It's a nice place for quasi-standard proposals and we think it's a good way to achieve consensus among various CL implementations about a common API for future extensions to allow portability.

    We know many facilities are practically standardized by portability libraries (ASDF, Bordeaux Threads, CFFI etc.), but future extensions might have an actual specification – we all know warts of "reference implementation" standards *cough-python-cough*.

    I would love to see "Extensible Sequences" in the Central Document Repository.

5 On Lisp (not a book!)

5.1 Inlining C/C++ code in Common-Lisp with ECL

Like many compilers ECL offers a facility to inline it's assembly in the source code for a convenience and performance reasons. ECL's assembler is C/C++°.

To achieve inlining, ECL offers three constructs.

(ffi:clines                           &body strings)
(ffi:c-inline args arg-types ret-type &body others)
(ffi:c-progn  args                    &body forms-and-strings)

The most basic is ffi:clines, which allows you to just drop in some C/C++ code (for instance include a header you need for further use).

ffi:c-inline is much more useful, allowing you to pass values from Lisp to the block and receive output in return. This construct allows returning multiple values declared in the third clause. You may also declare whenever it has side effects and if it is one-liner°° (more details in ECL manual). Short example:

(defun ctrunc (number divisor)
  (ffi:c-inline (number divisor) (:int :int) (values :int :int) "{
    int num = #0, div = #1;
    @(return 0) = num/div;
    @(return 1) = num%div;
}"))

It's worth mentioning that the number and divisor types are checked when the function #'ctrunc is called and if incorrect – proper lisp condition is signaled.

ffi:c-progn is the last construct. It allows you to intermix both C and lisp code – it doesn't return any value so it is called purely for it's side-effects (assigning variable with computed result for instance). Don't forget to declare variable types! If you don't, results might be surprising.

To illustrate potential gain of using inlined C language we'll take the trivial Fibbonachi algorithm and benchmark ECL against itself. It's not a proper benchmark, nor a good implementation, but both implementations are comparable and this should prove a point, that ECL might be quite fast when we want it to be°°°.

(defun fib-1 (n)
  "Borrowed from http://www.cliki.net/fibonacci"
  (loop for f1 = 0 then f2
     and f2 = 1 then (+ f1 f2)
     repeat n finally (return f1)))

(defun fib-2 (n)
  (let ((f1 0) (f2 1) (n n))
    (declare (:int f1 f2 n))
    (ffi:c-progn (n f1 f2) "
     int aux = 0;
     for( ; #0>0; #0--, aux=#1) {
         #1 = #2;
         #2 = aux + #2;
     }")
    f1))

(defun bench ()
  (compile 'fib-1)
  (compile 'fib-2)
  (prog1 nil
    (print "Common Lisp:")
    (time (dotimes (x 10000000) (fib-1 20)))
    (print "Common Lisp with inlined C:")
    (time (dotimes (x 10000000) (fib-2 20)))))

On my computer This yields:

"Common Lisp:" real time : 9.583 secs run time : 9.596 secs gc count : 1 times consed : 271531840 bytes

"Common Lisp with inlined C:" real time : 0.657 secs run time : 0.656 secs gc count : 1 times consed : 271567088 bytes

So, as we can see, the speed improvement is pretty decent. Proper declarations would speed up #'fib-1 a little, but the C version will still be faster (note the fact that we are comparing ECL with ECL, other implementations might theoretically outperform ECL's C version using provided #'fib-1 definition).

For more information please consult manual: https://common-lisp.net/project/ecl/static/manual/ch28.html

° None of these constructs will work with the bytecode compiler (it will signal an error).

°° C doesn't treat each statement as a valid R-value. By "one-liner" we mean something, what might be used as such.

°°° We can't say it's fast Common Lisp, since it's no longer Common Lisp – if we use this technique we're using ECL and it's not portable by any means.

5.2 Case on LET / FLET / LABELS (aka Nasal Demons Reborn)

Consider an example

(let ((x 'foo)
      (x 'bar))
  x)

what symbol does this form evaluate to: **BARFOO* or *?

Well, it's not defined – in CCL and CLISP it is **FOOBAR*, ECL and ABCL make it *, while SBCL signals an error. It isn't clear what should happen because it's not specified in the spec. I think that the SBCL approach is the most sane. I can hardly imagine a programmer doing that, not as a typo, but as a conscious decision°.

The same argument applies to **FLET* FLET*FLET* (except the side-effect thingy – all but one definition of the same name can be optimized out) – which function definition is the valid one? It isn't specified – note that the operator name isn't *but *.

The last case – the most dangerous and the least comprehensible. *LABELS* allows mutual recursion and it's far less obvious what would happen even from an implementation point of view. If we write a code:

(labels ((function1 ()
           (im-dangerous 2))
         (im-dangerous (x)
           (format t "FIRST X=~A~%" x)
           (if (zerop x)
               'first
               (im-dangerous (1- x))))
         (function2 ()
           (im-dangerous 2))
         (im-dangerous (x)
           (format t "SECOND X=~A~%" x)
           (if (zerop x)
               'second
               (im-dangerous (1- x))))
         (function3 ()
           (im-dangerous 2)))
  (list (function1)
        (function2)
        (function3)
        (im-dangerous 1)))

CLISP take the first definition of ** IM-DANGEROUSIM-DANGEROUS*, while others the second one. SBCL on the other hand behaves inconsistently – inside function definitions references to *are bound to the first definition, while references from inner block are bound to the second one.

It is important to say that the behavior in these situations is undefined by the spec and each implementation is free to do what it considers most reasonable, easiest to implement, or best to optimize – and none of these is wrong (CCL allows all constructs, but issues a style warning for each – it is the only implementation which does that – bravo).

Curious minds may find the following article amusing (warning, C code involved): http://blogs.msdn.com/b/oldnewthing/archive/2014/06/27/10537746.aspx.

My conclusion is as follows: you can't rely on ** LABELS* FLETLET*, *and *constructs if multiple definitions of the same name exist – they are ambiguous and should be considered being an error. The new release of ECL treats them that way°°.

° Multiple variables can't be optimized out, because the initialization form might have side effects

(let ((x (side-effects1))
      (x (side-effects2)))
  x)

;; More deterministic form:
(let ((x (progn (side-effects1)
                (side-effects2))))
  x)

°° ANSI defines LAMBDA-LIST in terms of LET*, so repeating the parameters of the same name is theoretically correct. There is no valid use-case for two required parameters of the same name (there is no initialization form) though, so ECL signals an error on such situation.

5.3 Case for portability and against unusual reader syntax

Why we shouldn't use all these implementation-specific niceties

Lately, Zach Bane posted a nifty trick for accessing symbols from packages you're not in, literally:

foo::(list 'these 'symbols 'are 'from "foo" 'package)

It's nice, intuitive, practical… and not compliant. It would be a minor problem, if it weren't a syntax hack, which is hard to implement portably as a library (at least I don't see any elegant solution). In my opinion it requires digging into implementation innards and tweaking the reader to support it. Manipulating strings before they are actually read is an option too. Also – unlike the metaobject protocol, gray streams or extensible sequences – it doesn't bring anything new to the table.

"But hey! It's just syntactic sugar for REPL interaction! Don't be such a grumpy guy!"

OK, fine, if you promise me that this "syntactic sugar" won't land in any library *ever* – it's advertised after all. Nobody knows who will pick up this hint. And if critical mass will prevail, then this dubious syntactic sugar will become de-facto standard, so other implementations will be forced to implement it, or fail to load libraries using it.

Such a situation happened once. CLtL had an example of the ** UNTIL* FORLOOP* usage, where a *clause landed after an *clause, which isn't ANSI compliant. The fact that it was supported by a few implementations lead to the situation imagined above.

If you really want some syntactic sugar for using other packages locally – I propose a little uglier, yet portable, solution:

(defun sharp-l (stream char subchar)
  (declare (ignore char subchar))
  (let ((*package* (find-package (read stream))))
    (read stream nil nil t)))

(set-dispatch-macro-character #\# #\l #'sharp-l)

#l foo (list 'these 'symbols 'are 'from "foo" 'package)

It's only five lines of code and works everywhere (unless someone binds #l to his own reader macro). If someone wants to be a little fancier, then he may mimic the SLIME prompt in his syntax:

(defun sharp-x (stream char subchar)
  (declare (ignore char subchar))
  (let ((*readtable* (copy-readtable))
        (right #\))
        (rpar #\>))
    (set-macro-character rpar (get-macro-character right))
    (let ((*package* (find-package (read stream))))
      (peek-char #\> stream)
      (read-char stream)
      (read stream nil nil t))))

(set-dispatch-macro-character #\# #\[ #'sharp-x)

#[foo> (list 'these 'symbols 'are 'from "foo" 'package)

And, honestly I really like proposed syntax, but for my taste it's totally unportable and harmful for reasons I've mentioned above.

6 Advertisement

Here comes the shameless plug – I've recently (officially) launched the company named TurtleWare. I am open for consultancy. If anyone:

  • wants to pay for a dreamed ECL feature,
  • prioritizes his own issues with implementation,
  • likes to pay for support or system maintenance,
  • has some Lisp and/or embedded systems work to do

then reach me with further details at: hello[at]turtleware.eu.