ViewVC logotype

Contents of /src/BUILDING

Parent Directory Parent Directory | Revision Log Revision Log

Revision 1.2 - (hide annotations)
Tue May 17 16:14:05 2005 UTC (8 years, 11 months ago) by rtoy
Branch: MAIN
Changes since 1.1: +114 -58 lines
Update with some fixes from src/tools/README.  Correct some other
small items.
1 toy 1.1 -*- Mode: text -*-
2     Building CMU CL
3     ===============
5     This document is intended to give you a general overview of the build
6     process (i.e. what needs to be done, in what order, and what is it
7     generally called). It will also tell you how to set up a suitable
8     build environment, how the individual scripts fit into the general
9     scheme of things, and give you a couple of examples.
11     General Requirements
12     --------------------
14     In order to build CMU CL, you will need:
16     a) A working CMU CL binary. There is no way around this requirement!
18     This binary can either be for the platform you want to target, in
19     that case you can either recompile or cross-compile, or for another
20     supported platform, in that case you must cross-compile, obviously.
22     b) A supported C compiler for the C runtime code.
24     Most of the time, this means GNU gcc, though for some ports it
25     means the vendor-supplied C compiler. The compiler must be
26     available under the name specified by your ports Config file.
28     c) GNU make
30     This has to be available either as gmake or make in your PATH, or
31     the MAKE environment variable has to be set to point to the correct
32     binary.
34     d) The CMU CL source code
36     Here you can either use one of the release source tarballs, or
37     check out the source code directly from the public CMUCL CVS
38     repository.
40     If you want to build CMU CL's Motif interface/toolkit, you'll need a
41     working version of the Motif libraries, either true-blue OSF/Motif, or
42     OpenMotif, or Lesstif. The code was developed against 1.2 Motif,
43     though recompilation against 2.x Motif probably works as well.
45     Setting up a build environment
46     ------------------------------
48     1.) Create a base directory and change to it
50     mkdir cmucl ; cd cmucl
52     2.) Fetch the sources and put them into the base directory
54 rtoy 1.2 tar xzf /tmp/cmucl-source.tar.gz
55 toy 1.1
56     or, if you want to use the CVS sources directly:
58 rtoy 1.2 export CVSROOT=:pserver:anonymous@common-lisp.net:/project/cmucl/cvsroot
59 toy 1.1 cvs login (password is `anonymous')
60     cvs co src
62     Whatever you do, the sources must be in a directory named src
63     inside the base directory. Since the build tools keep all
64     generated files in separate target directories, the src directory
65     can be read-only (e.g. mounted read-only via NFS, etc.)
67     The build tools are all in the src/tools directory.
69     That's it, you are now ready to build CMU CL.
72     A general outline of the build process
73     --------------------------------------
75     Building CMU CL can happen in one of two ways: Normal recompilation,
76     and cross-compilation. We'll first look at normal recompilation:
78     The recompilation process basically consists of 4 phases/parts:
80     a) Compiling the lisp files that make up the standard kernel.
82     This happens in your current CMU CL process, using your current
83     CMU CL's normal file compiler. This phase currently consists of 3
84     sub-phases, namely those controlled by src/tools/worldcom.lisp,
85     which compiles all the runtime files, src/tools/comcom.lisp, which
86     compiles the compiler (including your chosen backend), and finally
87     src/tools/pclcom.lisp, which compiles PCL, CMU CL's CLOS
88     implementation. The whole phase is often called "world-compile",
89     or "compiling up a world", based on the name of the first
90     sub-phase.
92     b) Building a new kernel.core file out of the so created files
94     This process, which is generally called genesis, and which is
95     controlled by src/tools/worldbuild.lisp, uses the newly compiled
96     files in order to build a new, basic core file, which is then used
97     by the last phase to create a fully functional normal core file.
98     It does this by "loading" the compiled files into an in-core
99     representation of a new core file, which is then dumped out to
100     disk, together with lots of fixups that need to happen once the new
101     core is started.
103     As part of this process, it also creates the file internals.h,
104     which contains information about the general memory layout of the
105     new core and its basic types, their type tags, and the location of
106     several important constants and other variables, that are needed by
107     the C runtime code to work with the given core.
109     So going through genesis is needed to create internals.h, which is
110     needed to compile the C runtime code (i.e. the "lisp" binary).
111     However there is a slight circularity here, since genesis needs as
112     one of its inputs the file target:lisp/lisp.nm, which contains the
113     (slightly pre-treated) output of running nm on the new lisp
114     binary. Genesis uses this information to fixup the addresses of C
115     runtime support functions for calls from Lisp code.
117     However the circularity isn't complete, since genesis can work with
118     an empty/bogus lisp.nm file. While the kernel.core it then
119     produces is unusable, it will create a usable internals.h file,
120     which can be used to recompile the C runtime code, producing a
121     usable lisp.nm file, which in turn can be used to restart genesis,
122     producing a working kernel.core file.
124     Genesis also checks whether the newly produced internals.h file
125     differs from a pre-existing internals.h file (this might be caused
126     by an empty internals.h file if you are rebuilding for the first
127     time, or by changes in the lisp sources that cause differences in
128     the memory layout of the kernel.core), and informs you of this, so
129     that you can recompile the C runtime code, and restart genesis.
131     If it doesn't inform you of this, you can skip directly to the last
132     phase d).
134     c) Recompiling the C runtime code, producing the "lisp" binary file
136     This step is only needed if you haven't yet got a suitable lisp
137     binary, or if the internals.h file has changed during genesis (of
138     which genesis informs you), or when you made changes to the C
139     sources that you want to take effect.
141     Recompiling the C runtime code is controlled by a GNU Makefile, and
142     your target's Config file. It depends on a correct internals.h
143     file as produced by genesis.
145     Note that whenever you recompile the runtime code, for whatever
146     reason, you must redo phase b). Note that if you make changes to
147     the C sources and recompile because of this, you can do that before
148     Phase b), so that you don't have to perform that phase twice.
150     d) Populating the kernel.core, and dumping a new lisp.core file.
152     In this phase, which is controlled by src/tools/worldload.lisp, and
153     hence often called world-load, the kernel.core file is started up
154     using the (possibly new) lisp binary, the remaining files which
155     were compiled in phase a) are loaded into it, and a new lisp.core
156     file is dumped out.
158 rtoy 1.2 We're not quite done yet. This produces just a basic lisp.core.
159     To complete the build so that you something similar to what the
160     releases of CMUCL do, there are a few more steps:
162     e) Build the utilities like Gray streams, simple streams, CLX, CLM,
163     and Hemlock. Use the src/tools/build-utils.sh script for this, as
164     described below
166     f) Create tarfiles using the src/tools/make-dist.sh script, as
167     explained below.
169     With these tarfiles, you can install them anywhere. The contents of
170     the tarfiles will be the same as the snapshots and releases of CMUCL.
172 toy 1.1 When cross-compiling, there is additional phase at the beginning, and
173     some of the phases happen with different hosts/platforms. The initial
174     phase is setting up and compiling the cross-compilation backend, using
175     your current compiler. The new backend is then loaded, and all
176     compilation in phase a) happens using this compiler backend. The
177     creation of the kernel.core file in phase b) happens as usual, while
178     phase c) of course happens on the target platform (if that differs
179     from the host platform), as does the final phase d). Another major
180     difference is that you can't compile PCL using the cross-compiler, so
181     one usually does a normal rebuild using the cross-compiled core on the
182     target platform to get a full CMU CL core.
184     So, now you know all about CMU CL compilation, how does that map onto
185     the scripts included with this little text?
187     Overview of the included build scripts
188     --------------------------------------
190 rtoy 1.2 * src/tools/create-target.sh target-directory [lisp-variant [motif-variant]]
191 toy 1.1
192     This script creates a new target directory, which is a shadow of the
193     source directory, that will contain all the files that are created by
194     the build process. Thus, each target's files are completely separate
195     from the src directory, which could, in fact, be read-only. Hence you
196     can simultaneously build CMUCL for different targets from the same
197     source directory.
199     The first argument is the name of the target directory to create. The
200     remaining arguments are optional. If they are not given, the script
201     tries to determine the lisp variant and motif variant from the system
202     the script is running on.
204     The lisp-variant (i.e. the suffix of the src/lisp/Config.* to use as
205     the target's Config file), and optionally the motif-variant (again the
206     suffix of the src/motif/server/Config.* file to use as the Config file
207     for the target's CMUCL/Motif server code). If the lisp-variant is
208     given but the motif-variant is not, the motif-variant is determined
209     from the lisp-variant.
211     The script will generate the target directory tree, link the relevant
212     Config files, and generate place-holder files for various files, in
213     order to ensure proper operation of the other build-scripts. It also
214     creates a sample setenv.lisp file in the target directory, which is
215     used by the build and load processes to set up the correct list of
216     *features* for your target lisp core.
218     IMPORTANT: You will normally NOT have to modify the sample setenv.lisp
219     file, if you are building from a binary that has the desired features.
220     In fact, the sample has all code commented out, If you want to add or
221     remove features, you need to include code that puts at least a minimal
222     set of features onto the list (use PUSHNEW and/or REMOVE). You can
223     use the current set of *features* of your lisp as a first guide. The
224     sample setenv.lisp includes a set of features that should work for the
225     intended configuration. Note also that some adding or removing some
226     features may require a cross-compile instead of a normal compile.
228 rtoy 1.2 * src/tools/clean-target.sh [-l] target-directory [more dirs]
229 toy 1.1
230     Cleans the given target directory, so that all created files will be
231     removed. This is useful to force recompilation. If the -l flag is
232     given, then the C runtime is also removed, including all the lisp
233     executable, any lisp cores, all object files, lisp.nm, internals.h,
234     and the config file.
236 rtoy 1.2 * src/tools/build-world.sh target-directory [build-binary] [build-flags...]
237 toy 1.1
238     Starts a complete world build for the given target, using the lisp
239     binary/core specified as a build host. The recompilation step will
240     only recompile changed files, or files for which the fasl files are
241     missing. It will also not recompile the C runtime code (the lisp
242     binary). If a (re)compilation of that code is needed, the genesis
243     step of the world build will inform you of that fact. In that case,
244     you'll have to use the rebuild-lisp.sh script, and then restart the
245     world build process with build-world.sh
247 rtoy 1.2 * src/tools/rebuild-lisp.sh target-directory
248 toy 1.1
249     This script will force a complete recompilation of the C runtime code
250     of CMU CL (aka the lisp executable). Doing this will necessitate
251     building a new kernel.core file, using build-world.sh.
253 rtoy 1.2 * src/tools/load-world.sh target-directory version
254 toy 1.1
255     This will finish the CMU CL rebuilding process, by loading the
256     remaining compiled files generated in the world build process into the
257     kernel.core file, that also resulted from that process, creating the
258     final lisp.core file.
260     You have to pass the version string as a second argument. The dumped
261     core will anounce itself using that string. Please don't use a string
262     consisting of an official release name only, (e.g. "18d"), since those
263     are reserved for official release builds. Including the build-date in
264     ISO8601 format is often a good idea, e.g. "18d+ 2002-05-06" for a
265     binary that is based on sources current on the 6th May, 2002, which is
266     post the 18d release.
268 rtoy 1.2 * src/tools/build-utils.sh target-directory
269 toy 1.1
270     This script will build auxiliary libraries packaged with CMU CL,
271     including CLX, CMUCL/Motif, the Motif debugger, inspector, and control
272     panel, and the Hemlock editor. It will use the lisp executable and
273     core of the given target.
275 rtoy 1.2 * src/tools/make-dist.sh [-bg] [-G group] [-O owner] target-directory version arch os
276 toy 1.1
277     This script creates both main and extra distribution tarballs from the
278     given target directory, using the make-main-dist.sh and
279 rtoy 1.2 make-extra-dist.sh scripts. The result will be two tar files. One
280     contains the main distribution including the runtime and lisp.core
281     with PCL (CLOS); the second contains the extra libraries such as
282     Gray-streams, simple-streams, CLX, CLM, and Hemlock.
284     Some options that are available:
286     -b Use bzip2 compression
287     -g Use gzip compression
288     -G group Group to use
289     -O owner Owner to use
291     If you specify both -b and -g, you will get two sets of tarfiles. The
292     -G and -O options will attempt to set the owner and group of the files
293     when building the tarfiles. This way, when you extract the tarfiles,
294     the owner and group will be set as specified. You may need to be root
295     to do this because many Unix systems don't normally let you change the
296     owner and group of a file.
298     The remaining arguments used to create the name of the tarfiles. The
299     names will have the form:
301     cmucl-<version>-<arch>-<os>.tar.bz2
302     cmucl-<version>-<arch>-<os>.extras.tar.bz2
304     Of course, the "bz2" will be "gz" if you specified gzip compression
305     instead of bzip.
307     * /src/tools/make-main-dist.sh target-directory version arch os
308 toy 1.1
309 rtoy 1.2 This is script is not normally invoked by the user; make-dist will do
310     it appropriately.
311 toy 1.1
312     This script creates a main distribution tarball (both in gzipped and
313     bzipped variants) from the given target directory. This will include
314     all the stuff that is normally included in official release tarballs
315     such as lisp.core and the PCL libraries, including Gray streams and
316     simple streams.
318     This is intended to be run from make-dist.sh.
320 rtoy 1.2 * src/tools/make-extra-dist.sh target-directory version arch os
322     This is script is not normally invoked by the user; make-dist will do
323     it appropriately.
324 toy 1.1
325     This script creates an extra distribution tarball (both in gzipped and
326     bzipped variants) from the given target directory. This will include
327     all the stuff that is normally included in official extra release
328     tarballs, i.e. the auxiliary libraries such as CLX, CLM, and Hemlock.
330     This is intended to be run from make-dist.sh.
333     * cross-build-world.sh target-directory cross-directory cross-script
334     [build-binary] [build-flags...]
336     This is a script that can be used instead of build-world.sh for
337     cross-compiling CMUCL. In addition to the arguments of build-world.sh
338     it takes two further required arguments: The name of a directory that
339     will contain the cross-compiler backend (the directory is created if
340     it doesn't exist, and must not be the same as the target-directory),
341     and the name of a Lisp cross-compilation script, which is responsible
342     for setting up, compiling, and loading the cross-compiler backend.
343     The latter argument is needed because each host/target combination of
344     platform's needs slightly different code to produce a working
345     cross-compiler.
347     We include a number of working examples of cross-compiler scripts in
348     the cross-scripts directory. You'll have to edit the features section
349     of the given scripts, to specify the features that should be removed
350     from the current set of features in the host lisp, and those that
351     should be added, so that the backend features are correct for the
352     intended target.
354     You can look at Eric Marsden's collection of build scripts for the
355     basis of more cross-compiler scripts.
357     Step-by-Step Example of recompiling CMUCL for OpenBSD
358     -----------------------------------------------------
360     Set up everything as described in the setup section above. Then
361     execute:
363     # Create a new target directory structure/config for OpenBSD:
364     src/tools/create-target.sh openbsd OpenBSD_gencgc OpenBSD
366     # edit openbsd/setenv.lisp to contain what we want:
367     cat <<EOF > openbsd/setenv.lisp
368     ;;; Put code to massage *features* list here...
370     (in-package :user)
372     (pushnew :openbsd *features*)
373     (pushnew :bsd *features*)
374     (pushnew :i486 *features*)
375     (pushnew :mp *features*)
376     (pushnew :hash-new *features*)
377     (pushnew :random-mt19937 *features*)
378     (pushnew :conservative-float-type *features*)
379     (pushnew :gencgc *features*)
381     ;;; Version tags
383     (pushnew :cmu18d *features*)
384     (pushnew :cmu18 *features*)
385     (setf *features* (remove :cmu17 *features*))
386     (setf *features* (remove :cmu18c *features*))
387     EOF
389     # Recompile the lisp world, and dump a new kernel.core:
390     src/tools/build-world.sh openbsd lisp # Or whatever you need to invoke your
391     # current lisp binary+core
393     # If build-world tells you (as it will the first time) that:
394     # "The C header file has changed. Be sure to re-compile the startup
395     # code."
396     # You 'll need to start rebuild-lisp.sh to do that, and then reinvoke
397     # build-world.sh:
399     # Recompile lisp binary itself:
400     src/tools/rebuild-lisp.sh openbsd
402     # Restart build-world.sh now:
403     src/tools/build-world.sh openbsd lisp
405     # Now we populate the kernel.core with further compiled files,
406     # and dump the final lisp.core file:
408     src/tools/load-world.sh openbsd "18d+ 2002-05-06"
410     # The second argument above is the version number that the built
411     # core will announce. Please always put the build-date and some
412     # other information in there, to make it possible to differentiate
413     # those builds from official builds, which only contain the release.
415     Now you should have a new lisp.core, which you can start with
417     ./openbsd/lisp/lisp -core ./openbsd/lisp/lisp.core -noinit -nositeinit
419     Compiling sources that contain disruptive changes
420     -------------------------------------------------
422     The above instructions should always work as-is for recompiling CMU CL
423     using matching binaries and source files. They also work quite often
424     when recompiling newer sources. However, every so often, some change
425     to the CMU CL sources necessitates some form of bootstrapping, so that
426     binaries built from earlier sources can compile the sources containing
427     that change. There are two forms of boostrapping that can be
428     required:
430     a) Bootfiles
432     The maintainers try to make bootfiles available, that allow going
433     from an old release to the next release. These are located in the
434     src/bootfiles/<old-release>/ directory of the CMU CL sources.
436     I.e. if you have binaries that match release 18d, then you'll need
437     to use all the bootfiles in src/bootfiles/18d/ in order to go to
438     the next release (or current sources, if no release has been made
439     yet). If you already used some of the bootstrap files to compile
440     your current lisp, you obviously don't need to use those to get to
441     later versions.
443     You can use the bootfiles by concatenating them into a file called
444     bootstrap.lisp in the target directory (i.e. target:bootstrap.lisp)
445     in the order they are numbered. Be sure to remove the bootstrap
446     file once it is no longer needed.
448 rtoy 1.2 Alternatively, the bootstrap file can just "load" the individual
449     bootfiles as needed.
450 toy 1.1
451     b) Cross-compiling
453     Under some circumstances, bootstrap code will not be sufficient,
454     and a cross-compilation is needed. In that case you will have to
455     use cross-build-world.sh, instead of build-world.sh. Please read
456     the instructions of that script for details of the more complex
457     procedure.
459     << This isn't really true anymore, and we should place a more
460     elaborate description of the cross-compiling process here >>
462     When cross-compiling, there are two sorts of bootscripts that can be
463     used: Those that want to be executed prior to compiling and loading
464     the cross-compiler, which should be placed in the file called
465     target:cross-bootstrap.lisp, and those that should happen after the
466     cross-compiler has been compiled and loaded, just prior to compiling
467     the target, which should be placed in target:bootstrap.lisp, just
468     like when doing a normal recompile.
470     Additionally, sometimes customized cross-compiler setup scripts
471     (to be used in place of e.g. cross-x86-x86.lisp) are required,
472     which are also placed in one of the bootfiles/*/* files. In those
473     cases follow the instructions provided in that file, possibly merging
474     the changed contents thereof with your normal cross-script.
476     Step-by-Step Example of Cross-Compiling
477     ---------------------------------------
479     This gives a step-by-step example of cross-compiling a sparc-v8 build
480     using a sparc-v9 build. (For some unknown reason, you can't just
481     remove the :sparc-v9 feature and add :sparc-v8.)
483     So, first get a recent sparc-v9 build. It's best to get a version
484     that is up-to-date with the sources. Otherwise, you may also need to
485     add a bootstrap file to get any bootfiles to make your lisp
486     up-to-date with the current sources.
488 rtoy 1.2 1. Select a directory for the cross-compiler and compiled target:
490     Create a cross-compiler directory to hold the cross-compiler
491     and a target directory to hold the result:
493     src/tools/create-target.sh xcross
494     src/tools/create-target.sh xtarget
496     2. Adjust cross-compilation script
498     Copy the src/tools/cross-scripts/cross-sparc-sparc.lisp to
499     xtarget/cross.lisp. Edit it appropriately. In this case, it
500     should look something like:
501 toy 1.1
502 rtoy 1.2 (c::new-backend "SPARC"
503     ;; Features to add here
504     '(:sparc :sparc-v8
505     :complex-fp-vops
506     :linkage-table
507     :gencgc
508     :stack-checking
509     :relative-package-names
510     :conservative-float-type
511     :hash-new :random-mt19937
512     :cmu :cmu19 :cmu19a
513     )
514     ;; Features to remove from current *features* here
515     '(:sparc-v9 :sparc-v7 :x86 :x86-bootstrap :alpha :osf1 :mips
516     :propagate-fun-type :propagate-float-type :constrain-float-type
517     :openbsd :freebsd :glibc2 :linux :pentium
518     :long-float :new-random :small))
519 toy 1.1
520 rtoy 1.2 (setf *features* (remove :sparc-v9 *features*))
521     (pushnew :sparc-v8 *features*)
522 toy 1.1
523 rtoy 1.2 It's important to add frob *features* here as well as in the
524     new-backend. If you don't adjust *features*, they won't be
525     set appropriately in the result.
526 toy 1.1
527 rtoy 1.2 3. Build the cross compiler and target
528     Now compile the result:
529 toy 1.1
530 rtoy 1.2 src/tools/cross-build-world.sh xtarget xcross xtarget/cross.lisp [v9 binary]
531 toy 1.1
532 rtoy 1.2 4. Rebuild the lisp files:
533 toy 1.1
534 rtoy 1.2 When this finishes, you need to compile the C code:
535 toy 1.1
536 rtoy 1.2 src/tools/rebuild-lisp.sh xtarget
537 toy 1.1
538 rtoy 1.2 At this point, you may want to run cross-build-world.sh again
539     to generate a new kernel.core. It shouldn't build anything;
540     just loads everything and creates a kernel.core.
541 toy 1.1
542 rtoy 1.2 5. Build the world:
543 toy 1.1
544 rtoy 1.2 With the new kernel.core, we need to create a lisp.core:
545 toy 1.1
546 rtoy 1.2 src/tools/load-world.sh xtarget "new lisp"
547 toy 1.1
548 rtoy 1.2 Test the result with
549 toy 1.1
550 rtoy 1.2 xtarget/lisp/lisp -noinit
551 toy 1.1
552 rtoy 1.2 However, this lisp will be missing some functionality like PCL. You
553     probably now want to use the compiler to rebuild everything once
554     again. Just follow the directions for a normal build, and use
555     xtarget/lisp/lisp as your compiler. Be sure to use create-target.sh
556     to create a new directory where the result can go.
557 toy 1.1

  ViewVC Help
Powered by ViewVC 1.1.5