.. ; -*- mode:rst; -*-

===================================================================
XCVB - an eXtensible Component Verifier and Builder for Common Lisp
===================================================================

The goal of this project is to have a scalable system
to build large software in Common Lisp,
featuring deterministic separate compilation and
enforced locally-declared dependencies.

.. Contents::

..

    1  About XCVB

      1.1  Home page
      1.2  Status
      1.3  Supported Implementations
      1.4  Getting Help
      1.5  Downloading the Code
      1.6  Dependencies
      1.7  Documentation
      1.8  Examples
      1.9  Timeline

    2  Using XCVB

      2.1  Building XCVB

      2.2  Starting a new XCVB project

      2.3  Compiling with XCVB
      2.4  Lisp Setup

      2.5  Interactive update of a running image with XCVB driver

      2.6  Makefile backend

    3  Restrictions on your Lisp Code
      3.1  Compile-time side-effects

      3.2  Full library needed at compile-time

      3.3  Optimization Settings

    4  Combining Multiple Projects
      4.1  Module Full Names

      4.2  Source Registry

      4.3  Module Name Resolution

      4.4  Dumping Images

    5  Additional Features
      5.1  Lisp file generation
      5.2  Data Dependencies
      5.3  Conditional Dependencies

      5.4  Warning Control

      5.5  CL:REQUIRE

      5.6  Creating Executables

      5.7  Around-Compile Hook

    6  Troubleshooting an XCVB build

      6.1  Debugging the built system
      6.2  Debugging the build
      6.3  Profiling the build

    7  XCVB and ASDF
      7.1  XCVB builds depending on ASDF systems
      7.2  ASDF systems depending on XCVB builds
      7.3  Converting XCVB builds into ASDF systems
      7.4  Converting ASDF systems into XCVB builds

      7.5  ASDF extensions without XCVB equivalent
      7.6  Undoing a conversion to XCVB
      7.7  Fast non-enforcing builds using POIU

    8  Frequently Asked Questions
      8.1  Why can't module declarations be moved to a central file?

      8.2  How do I prevent clash between incompatible FASLs?

      8.3  What are CFASLs?
      8.4  Why not just extend ASDF?

      8.5  Why is XCVB so much bigger and more complex than ASDF?

About XCVB
==========

Home page
---------

The home page of the project is at:

	http://common-lisp.net/project/xcvb/

Status
------

Bottom line:

XCVB is available as a working deterministic build system for Common Lisp.

It can already replace ASDF for the better
in many cases and on most implementations that matter.
But is not yet a universal replacement for it.

The current version (in git master branch or the release tarball)

can handle builds of very large and complex systems.
It is not very user-friendly in its failure modes,
though arguably still better than ASDF.

You can either write your project directly as an XCVB build

or automatically convert an existing ASDF project

(manual work required if ASDF extensions are used).

XCVB can then build your project directly (and sequentially),
or you can use it in conjunction with ``make -j``
to build your project in parallel,

which on a SMP machine may end up being faster than ASDF.

XCVB builds can depend on other builds and/or on ASDF systems.

ASDF systems can be wholly migrated to XCVB and still work on ASDF
(on implementations supported by XCVB above).
Or a same piece of software can have both an XCVB build and an ASDF system,
though you'll then have to manually maintain the coherence
between the two system definitions.

Hopefully a version 1.0 will be released at the end of this year (which?)

that can seriously replace ASDF and provide a migration path.

Supported Implementations
-------------------------

Here's a grid summarizing which Common Lisp implementations are supported,
in decreasing order of how much we support them and how often we test them.

============== ======= === ==== =========================
Implementation support exe free Notes
============== ======= === ==== =========================
SBCL           full    Y   Y    Our main platform
CCL            full    Y   Y    Well supported
CLISP          full    Y   Y    most portable
ECL            Y       Y   Y    has per-build .so
CMUCL          Y       Y   Y
LispWorks Pro  Y       Y   N
Allegro        Y       N   N
SCL            Y       N   N
ABCL           limited N   Y    slow; no bundling yet
XCL            limited N   Y    slow; no bundling yet
MCL            N       N   Y    it's obsolete, quaint
Genera         N       N   N    it's obsolete, quainter
GCL            N       N   Y    it's obsolete, incomplete
Cormanlisp     N       N   N    it's obsolete, incomplete
LispWorks Per  N       N   N    it's crippled
============== ======= === ==== =========================

Note that only SBCL, CCL and CLISP
are currently supported to compile XCVB itself.
The same implementations, on Unix only, also support
XCVB's advanced "farmer" mode of compilation through incremental forking.

On the same implementations and also on
ECL, CMUCL, and LispWorks Professional,
you can create executables from your Lisp builds
(column "exe" above).

On Allegro and SCL, you can create images that can then
be used by CL-Launch to wrap into an application,
but XCVB cannot create an application at this point.

While experimental support exists for ABCL and XCL,

it is slow due to long lisp startup time;
moreover, we cannot create images or bundles at this point.

Some implementations are just not supported;
they might never be, but do they really matter?
They are obsolete or crippled.
If one of them does matter to you, you may contact us to add support,
or you can add the support yourself.
Or then again, you can switch to a better implementation.

Getting Help
------------

We have two mailing-lists, one reserved for announcements,

``xcvb-announce`` http://www.common-lisp.net/mailman/listinfo/xcvb-announce

and one open to general discussions,
``xcvb-devel`` http://www.common-lisp.net/mailman/listinfo/xcvb-devel

For reporting bugs, we also have opened a project on https://launchpad.net/xcvb


Downloading the Code
--------------------

You can find a tarball containing XCVB ready to build,
with all its dependencies (except a Lisp implementation)
and a few generated files sufficient to bootstrap the whole thing in:

           http://common-lisp.net/project/xcvb/releases/

From the toplevel directory of the expanded release tarball,
you can afterwards ``make update``

and it will update XCVB and its dependencies to the latest version.

If you want instead to download XCVB as part of
your pre-existing Lisp development environment,

you will have to also download all our `dependencies`_.

You can checkout our public repository with::

	git clone git://common-lisp.net/projects/xcvb/xcvb.git

Or if you are a project member, you can obtain write access using::

	git clone ssh://USER@common-lisp.net/project/xcvb/git/xcvb.git

You can also browse the repository at either of these:

	http://common-lisp.net/gitweb?p=projects/xcvb/xcvb.git

	http://github.com/fare/xcvb/tree/master

Dependencies
------------

As mentioned above, you can find a tarball with all dependencies
sufficient to bootstrap XCVB at:

           http://common-lisp.net/project/xcvb/releases/

You will still need a Lisp implementation to build and run XCVB itself,
and to build and run the target software that you will build with XCVB:

  * We develop mainly with SBCL (CFASL support is included in SBCL 1.0.30.4 or later):
        - homepage: http://sbcl.sourceforge.net/

	- CLiki page: http://cliki.net/SBCL
        - get it from: http://sbcl.sourceforge.net/platform-table.html

  * We also make our best to keep it running with the following other Lisp implementations:

	- Clozure CL: http://ccl.clozure.com/
	- GNU CLISP: http://clisp.cons.org/

  * We unhappily haven't had resources to port it to other implementations yet,
    but porting should hopefully be easy:
    you can grep for e.g. ``clozure`` to find the few spots where
    there are implementation dependencies, and flesh out
    some variant for your favorite implementation there.
    We will assist you, and may even do it for you if we have access
    to the implementation on top of which you are interested in using XCVB.

If you want to automatically download all of XCVB's dependencies,
you may run this command from the ``xcvb/`` directory::

	make -f doc/Makefile.release checkout

If you prefer to collect XCVB's dependencies manually into your system,

you should consult the above file, as it is the authoritative source

for the list of XCVB dependencies.
These dependencies include libraries
that support being compiled with XCVB natively,
and libraries that require patching to support being compiled with XCVB.
If you don't use the patched versions of those libraries,
you may have to compile XCVB using ASDF.

The dependencies that support XCVB natively include:

  * ASDF 2.017 or later, aka ASDF 2:

	- homepage: http://cliki.net/asdf
	- getting it:

		``git clone git://common-lisp.net/projects/asdf/asdf.git``

        - getting it, if you're an ASDF project member:
		``git clone ssh://USER@common-lisp.net/project/asdf/asdf.git``

  * POIU for fast (parallelized) non-enforcing build:
	- homepage: http://common-lisp.net/project/qitab/
	- getting it:
		``git clone git://common-lisp.net/projects/qitab/poiu.git``
        - getting it, if you're a QITAB project member::
		``git clone ssh://USER@common-lisp.net/project/qitab/git/poiu.git``

  * Our fork of ``asdf-dependency-grovel``:

	- homepage: http://cliki.net/asdf-dependency-grovel
	- getting it:

		``git clone git://common-lisp.net/projects/xcvb/asdf-dependency-grovel.git``

        - getting it, if you're an XCVB project member:

		``git clone ssh://USER@common-lisp.net/project/xcvb/git/asdf-dependency-grovel.git``

  * cl-launch 3.011 or later:
	- homepage: http://cliki.net/CL-Launch
	- getting it:
		``git clone git://common-lisp.net/projects/xcvb/cl-launch.git``
        - getting it, if you're an XCVB project member::
        	``git clone ssh://USER@common-lisp.net/project/xcvb/git/cl-launch.git``

  * command-line-arguments:
	- homepage: http://common-lisp.net/project/qitab/
	- getting it:

		``git clone git://common-lisp.net/projects/qitab/command-line-arguments.git``

        - getting it, if you're a QITAB project member::

        	``git clone ssh://USER@common-lisp.net/project/qitab/git/command-line-arguments.git``

  * fare-utils:
	- homepage: http://cliki.net/fare-utils

	- getting it:

		``git clone git://common-lisp.net/users/frideau/fare-utils.git``
        - getting it, if you're frideau::
        	``git clone ssh://frideau@common-lisp.net/home/frideau/git/fare-utils.git``

  * fare-quasiquote:
	- homepage: http://cliki.net/fare-quasiquote

	- getting it:

		``git clone git://common-lisp.net/users/frideau/fare-quasiquote.git``
        - getting it, if you're me::
        	``git clone ssh://frideau@common-lisp.net/home/frideau/git/fare-quasiquote.git``

  * fare-mop:

	- homepage: http://cliki.net/fare-mop

	- getting it:

		``git clone git://common-lisp.net/users/frideau/fare-mop.git``

        - getting it, if you're a QITAB project member::

        	``git clone ssh://frideau@common-lisp.net/home/frideau/git/fare-mop.git``

  * quux-iolib:
	- homepage: http://cliki.net/quux-iolib
	- getting it:
        	``git clone git://common-lisp.net/projects/xcvb/quux-iolib.git``
        - getting it, if you're a XCVB project member::
        	``git clone ssh://USER@common-lisp.net/project/xcvb/git/quux-iolib.git``

  * libfixposix, a dependency of iolib that you'll need if you use our `farmer` extension:
	- homepage: http://common-lisp.net/project/iolib/
	- getting it:
        	``git clone git://gitorious.org/libfixposix/libfixposix.git``

  * iolib, that you'll need if you use our `farmer` extension:
	- homepage: http://common-lisp.net/project/iolib/
	- getting our XCVB-patched version:
        	``git clone git://common-lisp.net/projects/xcvb/iolib.git``
        - getting our XCVB-patched version, if you're a XCVB project member::
        	``git clone ssh://USER@common-lisp.net/project/xcvb/git/iolib.git``
	- getting the upstream repository:
        	``git clone git://gitorious.org/libfixposix/libfixposix.git``

        - see also the Windows port:
                http://src.knowledgetools.de/tomas/winapi/index.html

  * ironclad, for cryptography:
	- homepage: http://www.method-combination.net/lisp/ironclad/

	- getting our XCVB-patched version:
        	``git clone git://common-lisp.net/projects/xcvb/ironclad.git``
        - getting our XCVB-patched version, if you're a XCVB project member::
        	``git clone ssh://USER@common-lisp.net/project/xcvb/git/ironclad.git``
	- getting the upstream repository:

		``git clone git://github.com/froydnj/ironclad.git``

  * binascii for printing cryptographic digests:
	- homepage: http://www.cliki.net/binascii

	- getting our XCVB-patched version:
        	``git clone git://common-lisp.net/projects/xcvb/binascii.git``
        - getting our XCVB-patched version, if you're a XCVB project member::
        	``git clone ssh://USER@common-lisp.net/project/xcvb/git/binascii.git``
	- getting the upstream repository:

		``git clone git://github.com/froydnj/binascii.git``

  * alexandria:
	- homepage: http://common-lisp.net/project/alexandria/
	- getting our XCVB-patched version:
        	``git clone git://common-lisp.net/projects/xcvb/alexandria.git``
        - getting our XCVB-patched version, if you're a XCVB project member::
        	``git clone ssh://USER@common-lisp.net/project/xcvb/git/alexandria.git``
	- getting the upstream repository:
        	``git clone git://common-lisp.net/projects/alexandria/alexandria.git``

  * cffi:
	- homepage: http://common-lisp.net/project/cffi/
	- getting our XCVB-patched version:
        	``git clone git://common-lisp.net/projects/xcvb/cffi.git``
        - getting our XCVB-patched version, if you're a XCVB project member::
        	``git clone ssh://USER@common-lisp.net/project/xcvb/git/cffi.git``
	- getting the upstream repository:
        	``git clone git://common-lisp.net/projects/cffi/cffi.git``

  * Babel:
	- homepage: http://common-lisp.net/project/babel/
	- getting our XCVB-patched version:
        	``git clone git://common-lisp.net/projects/xcvb/babel.git``
        - getting our XCVB-patched version, if you're a XCVB project member::
        	``git clone ssh://USER@common-lisp.net/project/xcvb/git/babel.git``
	- getting the upstream repository:
		``darcs get http://www.common-lisp.net/project/babel/darcs/babel``

  * Closer to MOP:
	- homepage: http://common-lisp.net/project/closer/closer-mop.html
	- getting our XCVB-patched version:
        	``git clone git://common-lisp.net/projects/xcvb/closer-mop.git``
        - getting our XCVB-patched version, if you're a XCVB project member::
        	``git clone ssh://USER@common-lisp.net/project/xcvb/git/closer-mop.git``
	- getting the upstream repository:
		``darcs get http://common-lisp.net/project/closer/repos/closer-mop``

  * named-readtables:
	- homepage: http://common-lisp.net/project/editor-hints/darcs/named-readtables/doc/named-readtables.html
	- getting our XCVB-patched version:
        	``git clone git://common-lisp.net/projects/xcvb/named-readtables.git``
        - getting our XCVB-patched version, if you're a XCVB project member::
        	``git clone ssh://USER@common-lisp.net/project/xcvb/git/named-readtables.git``
	- getting the upstream repository:
		``darcs get http://common-lisp.net/project/editor-hints/darcs/named-readtables``

  * trivial-garbage:
	- homepage: http://www.cliki.net/trivial-garbage
	- getting our XCVB-patched version:
        	``git clone git://common-lisp.net/projects/xcvb/trivial-garbage.git``
        - getting our XCVB-patched version, if you're a XCVB project member::
        	``git clone ssh://USER@common-lisp.net/project/xcvb/git/trivial-garbage.git``
	- getting the upstream repository:
		``darcs get http://common-lisp.net/~loliveira/darcs/trivial-garbage/``

  * trivial-feature:
	- homepage: http://www.cliki.net/trivial-feature
	- getting our XCVB-patched version:
        	``git clone git://common-lisp.net/projects/xcvb/trivial-feature.git``
        - getting our XCVB-patched version, if you're a XCVB project member::
        	``git clone ssh://USER@common-lisp.net/project/xcvb/git/trivial-feature.git``
	- getting the upstream repository:
		``darcs get http://common-lisp.net/~loliveira/darcs/trivial-feature/``

  * bordeaux-threads:
	- homepage: http://common-lisp.net/project/bordeaux-threads/
	- getting our XCVB-patched version:
        	``git clone git://common-lisp.net/projects/xcvb/bordeaux-threads.git``
        - getting our XCVB-patched version, if you're a XCVB project member::
        	``git clone ssh://USER@common-lisp.net/project/xcvb/git/bordeaux-threads.git``
	- getting the upstream repository:
		``git clone git://common-lisp.net/projects/bordeaux-threads/bordeaux-threads.git``

  * rucksack:
	- homepage: http://common-lisp.net/project/rucksack/
	- getting our XCVB-patched version:
        	``git clone git://common-lisp.net/projects/xcvb/rucksack.git``
        - getting our XCVB-patched version, if you're a XCVB project member::
        	``git clone ssh://USER@common-lisp.net/project/xcvb/git/rucksack.git``
	- getting the upstream repository:
		``cvs -d :pserver:anonymous:anonymous@common-lisp.net:/project/rucksack/cvsroot checkout rucksack``

Note that you'll have to checkout these dependencies in a tree
that you register in your Lisp source-registry (see ASDF documentation)
for ASDF and/or XCVB to be able to see them.
If you want to compile XCVB using itself,
you'll also have to ensure you avoid conflicts when setting up your source-registry.

Moreover, cl-launch requires special installation of both its executable
in your ``PATH`` and its (self-extracted) library files
in your ASDF or XCVB configured paths.
See ``00INSTALL`` in the cl-launch source repository for more information.

Note that the release tarball includes an installation procedure that
should take care of all of the above for you. See the ``INSTALL`` file
in your toplevel release directory,
also available as ``doc/INSTALL.release`` in your XCVB source checkout.

Documentation
-------------

``doc/README.rest``

	The present file describes how to currently use XCVB.

``doc/xcvb.rest`` ``doc/ilc09-xcvb.rest`` ``doc/ilc09-xcvb-slides.rest``
	These files should provide with an overview of the project,
	its intended benefits, rationale and general design.

``doc/ilc09-xcvb-paper.tex`` ``doc/xcvb.bib`` ``doc/sigplanconf.cls``
	These source files compile to a PDF for the demonstration
        that was presented at ILC'2009 for XCVB.

``doc/INTERNALS.rest``

	This file is an introduction to the internals of XCVB
        for hackers interested in debugging or extending it.

``doc/TODO.rest``

	The many prioritized TO DO items on our plate to improve XCVB.
        Some already implemented items should be moved to current documentation.

``doc/configure.mk.example``
	example file to copy into ``configure.mk`` (in the main XCVB directory) and edit,
	so as to be able to create the ``xcvb`` executable.

``Makefile.release``

	the ``Makefile`` for XCVB itself is an example
        for how to integrate XCVB in your ``Makefile``.

``INSTALL.release``
	instructions for installing XCVB from the expansion of a release tarball.

Examples
--------

XCVB itself is an example.

Simpler examples are available in directory ``examples``.
The simplest are ``example-1`` and ``example-2``.
A more complete kind of Common Lisp application is ``hello``
See each time the source code and the associated ``Makefile``.

Timeline
--------

The original idea for what became XCVB is due to James Knight,
in 2007 discussions about the failure of POIU and ASDF-DEPENDENCY-GROVEL
to yield a *maintainable* alternative to serial ASDF systems
for the QRes project at ITA Software.

XCVB was originally written by Spencer Brody under the supervision of
Francois-Rene Rideau from late May to mid August 2008.
The result was a working prototype,

but lacking in features and requiring some layer of manual hacks
specific to the target on which XCVB was run.

Francois-Rene Rideau restarted work on it

and released in mid December 2008
a version of the prototype where the above hacks were automated away.

A paper about the prototype was presented at ILC'09.

The prototype can be retrieved in the git branch ``v0.1-prototype``.

Francois-Rene Rideau then undertook a complete rewrite of XCVB

to make its architecture more general and actually extensible
building on the lessons of the prototype.
A semi-usable product was published in May 2009,

and usable release tarballs have been produced since July 2009,
with notable contributions from Joyce Chen and Matthew Steele.

In late August 2009, we released XCVB 0.366
which successfully compiled a large system at ITA.

In 2011 Q2-Q3, Francois-Rene Rideau and Peter Keller have worked

on making XCVB more usable on more implementations,
so that it may be a valid replacement to ASDF,
and notably added support for building executable files.

We are working hard towards making a 1.0 release

usable by random Common Lisp programmer by Spring (which year?).

Subscribe to one of our mailing-lists
if you're interested in learning about our progress
or in influencing our design.

Using XCVB
==========

You may want to checkout the latest XCVB from git.

We'll tag stable releases when there are some.

Building XCVB
-------------

The simple way to build XCVB is to use our release tarballs,
and just follow the instructions in the ``INSTALL`` file.

The hard way to build XCVB is to install all the dependencies as listed above,
then create and edit your own ``configure.mk`` from ``doc/configure.mk.example``.

You'll notably need to configure those dependencies to work with ASDF.
Typically, you'll download all the code under ``~/.local/share/common-lisp/source/``
or another place configured in your ASDF source-registry.
Alternatively, you can provide an appropriate ``--source-registry`` option to ``cl-launch``.

You can now to bootstrap XCVB using ASDF with ``make xcvb-using-asdf``.
If successful, you may finally complete the bootstrap of XCVB using XCVB itself,

by exporting the proper ``CL_SOURCE_REGISTRY`` and then invoking ``make xcvb``.

Assuming you installed ``xcvb`` into some directory in your ``$PATH``,
you can test that XCVB built correctly by using the following query commands::

	xcvb version

	xcvb show-source-registry

Starting a new XCVB project
---------------------------

The general principle is that every Lisp file starts with a form like::

       #+xcvb (module (:depends-on ("packages" "macros" "specials" "foo")))

That form specifies all the dependencies of the Lisp file
and nothing but its dependencies.
Optionally, you can distinguish
between ``:compile-depends-on``

and ``:load-depends-on``
and even ``:cload-depends-on``:

 * ``:load-depends-on`` specifies what must be loaded before the compiled FASL file may be loaded.

 * ``:compile-depends-on`` specifies what must be loaded before the Lisp file may be compiled
   (by default, the compile-time versions of the files listed in ``:load-depends-on`` above;
   where compile-time version is a CFASL if available, or else a FASL).
 * ``:cload-depends-on`` specifies what must be loaded before the compiled CFASL file may be loaded
   when compiling files that depend on it, assuming CFASLs are available
   (by default, the same as ``:compile-depends-on`` above;
   it should probably always be a subset thereof).
 * ``:depends-on`` specifies what must be loaded no matter what in all the above cases.

Finally, a main file called ``build.xcvb``

is the main file that needs to be loaded in the final image.
It is typically empty except for the module form specifying
all its dependencies
and possibly some finalization forms to run at the end of the build.
The dependencies typically include ASDF libraries,
currently specified in an ugly way as dependencies of the build image.

The ``build.xcvb`` file for Exscribe_ is as follows::

	#+xcvb

        (module
          (:fullname "fare.tunes.org/exscribe"

           :nicknames ("exscribe")
           :supersedes-asdf ("exscribe")

           :build-depends-on ("/cl-launch" "/fare-utils"

                              "/fare-quasiquote" "/scribble"

                              (:asdf "cl-typesetting"))
           :depends-on ("scheme-compat" "exscribe"
                        "exscribe-html" "exscribe-txt" "exscribe-typeset")

           :pre-image t

           :build-image t))

You don't need to specify all your files in the ``build.xcvb`` module form,

only those containing the functions you really care about,

since the load dependencies will be automatically identified and loaded.

Compiling with XCVB
-------------------

The most portable way to compile with XCVB is
to use its ``simple-build`` backend::

	xcvb simple-build --build /foo/bar

Where ``/foo/bar`` is the fullname of your target.
The target is usually a build, but
it can be a single Lisp module or an executable.
If you do not specify a build name,
XCVB will attempt to use the build in the current working directory.

Note that you need to properly setup your `Source Registry`_
which can be overridden with ``CL_SOURCE_REGISTRY``
or by specifying option ``--source-registry``::

	--source-registry /home/luser/my/project//:

Also, by default, XCVB will use the same implementation
that it was compiled with. You can override this default with::

        --lisp-implementation sbcl

If the implementation is not in your ``PATH`` or has a non-standard name,
you may override with::

        --lisp-binary-path /home/luser/hacks/bin/sbcl

If needed, make sure you export the proper
``SBCL_HOME`` or ``CCL_DEFAULT_DIRECTORY``, etc.,
or that you use some wrapper script that does
before it invokes your actual Lisp binary.

Finally, object files by default will be stored
in a directory under ``~/.cache/xcvb/`` such as

``/home/luser/.cache/xcvb/sbcl-1.0.49-linux-x64/``.
You can override the base directory by exporting ``XCVB_CACHE``
of passing the argument::

	--cache /tmp/luser/xcvb-cache/

You can override the object directory by exporting ``XCVB_CACHE_OBJECTS``
(relative to the xcvb cache, if a relative pathname), or passing the argument::

	--cache-objects /tmp/luser/xcvb-cache/my-sbcl-r12345/

Once you've build a target, say the executable ``/xcvb/example-1/example-1``
you can install it somewhere you like::

	cp $(xcvb show cache-objects)/xcvb/example-1/example-1 /my/install/bin/

As an example of how to put it all together, see the ``Makefile``
of one of the XCVB examples or of XCVB itself.

Lisp Setup
----------

It is often useful to setup your target Lisp system

before you start to compile files and build images with it.

Such setup may include

proclaiming optimization settings, (see below `Optimization Settings`_),

configuring variables that control your Lisp implementation,

selecting compiler warnings you want to silence (see below `Warning Control`_),

pushing ``*features*`` for conditional compilation
based on e.g. some environment variables,

initializing some pathname translation layer such as logical pathnames,
loading specialized handlers for ASDF-DEPENDENCY-GROVEL when it is used,

configuring or extending the XCVB driver itself,

etc.,
to name a few things that have been done as part of ITA Software's build.

(Also, loading and configuring ASDF 1 was one major such thing,
that's happily not needed anymore.)

The setup file is specified with option ``--setup`` to ``xcvb simple-build``,

followed by the fullname of an XCVB Lisp module,
following usual XCVB naming conventions.
For instance, assuming your project has fullname ``/foo/bar``,
you may create an XCVB module called ``setup.lisp`` under your build directory,

and specify ``--setup /foo/bar/setup`` as arguments to ``xcvb simple-build``.

By default, XCVB will start the build by dumping an image
starting from the specified implementation binary and image,
and loading the XCVB driver followed by any setup you specified.
This is often useful for performance reasons
if your setup is non-trivial, or
if there will be a large number of files being compiled from the initial setup
(because their respective builds do not otherwise specify or inherit a pre-image).
There are cases however, for instance in a small project with a trivial setup,

where you will prefer to eschew the automated dumping of this initial base image

(especially on implementations such as CCL
where dumping is much slower than loading FASLs).
In these cases, you pass to ``xcvb simple-build`` the option

``--no-base-image``.

Interactive update of a running image with XCVB driver

------------------------------------------------------

``/xcvb/driver`` is a small standalone Lisp file.

It allows you to build some Lisp software cleanly in a subprocess
then load the results into the current image.

When you call ``xcvb-driver:build-and-load``

(which also has the short-hand ``xd:bnl``),

it will use ``run-program`` to invoke a slave XCVB process
that will drive all the compilation out-of-image
then reply to the master with a specification of files to load.
The function takes as arguments the name of a build and a flurry of options.

The build specifies what you want to update;

the options precisely match those of ``xcvb simple-build``

(or more precisely, those of ``xcvb make-makefile`` — see below).

The slave will complete the build then reply with a list of files to load,

including a TTH_ digest of each file
so the master can skip the files
of which the very same version has already been loaded.

The slave will do all the cryptographic hashing on its side,

so you don't have to include crypto libraries in your image ---

or anything beside the simple and short ``xcvb-driver``.

.. _TTH: http://en.wikipedia.org/wiki/Hash_tree#Tiger_tree_hash

The XCVB driver includes the interface and initial configuration,
so your Lisp process may call XCVB to build software,
and load this software in its current image.

If you're using ASDF, you can load the XCVB driver with
with ``(asdf:load-system :xcvb-driver)``.
Or you can load it from a bare Lisp image with

``(load "/path/to/xcvb/driver.lisp")``

as an alternative to using ``asdf.lisp`` itself
to drive further compilations.

The XCVB driver can advantageously replace ASDF,
as it has a noticeably smaller footprint,

doesn't pollute the current process with artefacts of compilation
or the compilation with the artefacts of the current process,

doesn't have as many multithreading concurrency issues
as compiling in-process can raise,

and preserves determinism and reusability in the FASL files being built.

Note that if you built your Lisp image using ASDF
or one of our ASDF-based backends (such as our non-enforcing parallel build),
then XCVB master will not be able to assess
which version of which files have already been loaded,
and will basically rebuild and reload everything using our enforcing backend.
However it you used ASDF, you can still use
the usual ASDF mechanisms to update your system.

Also note that the usual restrictions apply

as to any update of a running Common Lisp image,
whether using ASDF, XCVB, SLIME, or any other tool,
or directly using ``compile-file`` and ``load``:

  * errors may happen if you change a symbol's nature
    between lexical, special, constant or symbol-macro as a variable,
    or between simple function, generic function or macro as a function.

  * errors may happen if you change the signature of a generic function,
    the definition of a structure type, if you fail to provide appropriate
    methods on ``update-instance-for-redefined-class``, etc.

  * side-effects may override your variables
    (such as from ``setf`` or ``defparameter``).

  * the order in which side-effects happen or fail to happen
    (as XCVB master skips files an identical copy of which was already loaded)
    can trigger subtle bugs in your code
    (watch any compile-time dictionaries your code may use).

More generally, the correct loading a file may implicitly require
pre-conditions that are valid in a clean build,
but violated in an incremental load.

Some of these issues can be worked around by
manually uninterning specific symbols,

clearing and rebuilding dictionaries or twiddling specific entries,
unregistering and re-registering specific hooks,
deleting and re-creating specific packages, etc.

(see for instance the ``/xcvb/no-asdf`` used with ASDF 1,
or the many hot-upgrade mechanisms used by ASDF 2).

Unhappily, there is no general solution to this problem,

and that we know, no declarative meta-level protocol
was devised to help alleviate it.

Makefile backend
----------------

If you have a multicore machine (most machines are today),
XCVB allows you to compile your project in parallel
with its Makefile backend.

You first need to have XCVB generate a ``Makefile`` for your project
with the ``make-makefile`` command,
then to invoke ``make(1)`` with this ``Makefile``.

A simple example would be::

	xcvb make-makefile --build /foo/bar

Where ``/foo/bar`` is the fullname of your target as usual:
a build, or a single Lisp module, or an executable.
As for the ``simple-build`` backend, you may have to specify various
arguments such as ``--source-registry``, ``--lisp-implementation``,

``--lisp-binary-path``, ``--cache``, ``--cache-objects``, etc.

The default name for said generated ``Makefile``
is actually ``xcvb.mk`` in the current directory,
but you can override either or both of this name and that directory
with the ``--output-path`` option to ``make-makefile``.

For a correct build, it is recommended that you should run
the ``xcvb make-makefile`` command before every time you build,
in case any dependency has changed due to some modification.
XCVB should be fast and this shouldn't be a problem.
If making the ``Makefile`` takes more than a few seconds,
there's probably a performance bug and you should contact us.

After obtaining your ``xcvb.mk``, you may invoke ``make(1)``
using this output file in the following way::

	make -f xcvb.mk -j

``-j`` specifies a parallel build (also see ``make`` option ``-l``).

As an example of how to put it all together, see
the ``Makefile`` of the XCVB examples, of XCVB itself, or of exscribe_.

.. _exscribe: http://www.cliki.net/exscribe

Restrictions on your Lisp Code
==============================

Compile-time side-effects
-------------------------

To conform with the CLHS_ (see `section 3.2.2.3`_),
any ``defun``, ``defvar`` or other side-effect
must be effected in the compilation environment
to be available for use in macros,
while compiling either the current file or any further file.
This means that these effects should be enclosed in an
``(eval-when (:compile-toplevel :load-toplevel :execute) ...)``.
(See on my blog a discussion of `eval-when`_.)

.. _CLHS: http://www.lispworks.com/documentation/HyperSpec/Front/index.htm
.. _section 3.2.2.3: http://www.lispworks.com/documentation/HyperSpec/Body/03_bbc.htm
.. _eval-when: http://fare.livejournal.com/146698.html

Respecting that constraint will allow for
faster recompilation (where CFASLs are available),
since CFASLs may be smaller and faster to load than FASLs,
and subject to less variation which may trigger fewer recompilations.
It will also allow for easier cross-compilation
and more generally situations where the target image
should be different from compilation environment,
which was no doubt a strong motivating factor
behind this constraint being made part of the standard to begin with.
And of course, you can have a stronger claim of standard compliance.

This constraint is both enforced and taken advantage of
when you use the CFASL option of XCVB.
This option is enabled by default on implementations that support it,
currently only SBCL
(but there are plans for Clozure CL to support it if and when XCVB takes off).

See our FAQ `What are CFASLs?`_.

When you use this option, ``(:compile "foo")`` dependencies
turn into ``(:cfasl "foo")`` and specify that the specified CFASL will be loaded.
(This is what ``"foo"`` expands into for ``:compile-depends-on``
as explicitly specified or inherited from ``:depends-on``);
your effects will then be available if and only if you used
``:compile-toplevel`` in your ``eval-when``.
When you don't use this option ``(:compile "foo")`` dependencies
turn into ``(:fasl "foo")`` and specify that the specified FASL will be loaded instead;
your effects will then be available if and only if you used
``:load-toplevel`` in your ``eval-when``.
Finally, some other programs (notably ASDF-DEPENDENCY-GROVEL) will load your source code
at which point your effects will be available if and only if you used
``:execute`` in your ``eval-when``.
Therefore, any form whose effects must be available in the compilation environment
*must* be enclosed in ``(eval-when (:compile-toplevel :load-toplevel :execute) ...)``.
Other forms can be used without ``eval-when``, which is the same as enclosing them in
``(eval-when (:load-toplevel :execute) ...)``.

Note that this differs notably from ASDF.
On the one hand, ASDF always loads the FASL before it compiles more things,
so you don't need an ``eval-when`` around functions that are only used by macros

when these macros are expanded in a subsequent file rather than the same file.
On the other hand, ASDF doesn't support loading CFASLs, so that effects that only
happen at compile-toplevel are lost during an incremental compilation.

Finally, neither FASL nor CFASL includes
read-time or macroexpansion-time side-effects,
except as materialized in the resulting code as read then expanded.
Therefore to preserve incremental compilation with either XCVB or ASDF,
you must not rely on such side-effects to have been effected
outside of the file where they happen,
unless you also include them in the expansion.

Full library needed at compile-time
-----------------------------------

Sometimes, you actually want the full run-time power
of a library to be available at compile-time,
rather than only the definition of its macros and other usual compile-time side-effects.

You can achieve this by including your required dependencies in the

``:build-depends-on`` option of your ``build.xcvb``,

or the ``:compile-depends-on`` option of your Lisp module
(and possibly the ``:cload-depends-on`` option).

Indeed, when you specify a dependency in a ``:depends-on`` clause,
the dependency will be added to the load dependencies of the module
as if by ``:load-depends-on``,
and the compile-time-only version of the dependency (if available)
will be added to the compile dependencies of the module,
as if by ``:compile-depends-on``.

For instance, a ``:depends-on (... "foo" ...)`` will specify

a load-time dependency on ``(:fasl "foo")`` and
a compile-time dependency on ``(:cfasl "foo")``.

Note however that nothing is done to prevent the FASL and CFASL from both being loaded,

in any order implicit in the transitive dependencies of a module.
This can cause interesting surprises if any of those compile-time side-effects
are not idempotent, and/or if loading the CFASL resets some work done by the FASL.
Consider notably the proper use of ``defvar`` vs ``defparameter`` at compile-time.

Optimization Settings

---------------------

Amongst compile-time side-effects, a notable one is the optimization settings.

A default may to be set in your initial Lisp image setup
(see above section `Lisp Setup`_)
by modifying the variable ``xcvb-driver:*optimization-settings*``.
For instance, your setup file may include::

	(setf xcvb-driver:*optimization-settings* '(optimize (speed 1) (safety 3)))

Modules that want to change these settings

from this default specified for the compilation of just one file
can ``declaim`` it, or
``(eval-when (:compile-toplevel :load-toplevel :execute) (proclaim ...))`` it,
or (more portably) ``(locally (declare ...) ...)`` it.

You may define some macro that does that for you,
that you will call at the beginning of relevant files.

Note however that using ``declaim`` or ``proclaim`` may or may not
have a different meaning in other build systems (e.g. ASDF)
depending on implementations.
Some implementations (like SBCL) will confine ``declaim`` settings
and ``:compile-toplevel`` ``proclaim`` settings to the current file,
while other implementations (like CCL, Allegro)
will leak them to subsequent files,
whereas ``:load-toplevel`` ``proclaim`` settings
will usually leak to subsequent files when you load the FASL.

This non-determinism is against the spirit of XCVB, and
XCVB resets optimization settings to the declared default
before each and every command issued to load or compile any file.
When using ASDF, we have similarly been in the habit,
which we recommend others to follow, to similarly reset optimization settings
in a ``asdf:perform`` method ``:before`` or ``:around``
the relevant operations
(and we'll push for such method to be standardized in the upstream ASDF).

Combining Multiple Projects
===========================

Module Full Names
-----------------

XCVB relies on a global namespace for developers to name modules.
Using this namespace allows programmers to name modules independently
from their specific location on any particular machine's filesystem.