/[cmucl]/src/BUILDING
ViewVC logotype

Contents of /src/BUILDING

Parent Directory Parent Directory | Revision Log Revision Log


Revision 1.1 - (hide annotations)
Fri Oct 24 16:55:36 2003 UTC (10 years, 6 months ago) by toy
Branch: MAIN
CVS Tags: snapshot-2004-10, snapshot-2004-08, snapshot-2004-09, snapshot-2004-05, snapshot-2004-06, snapshot-2004-07, mod-arith-base, snapshot-2004-12, snapshot-2004-11, amd64-merge-start, prm-before-macosx-merge-tag, snapshot-2003-11, release-19a-base, snapshot-2003-12, release-19a-pre1, release-19a-pre3, release-19a-pre2, release-19a, snapshot-2005-03, snapshot-2004-04, snapshot-2005-01, snapshot-2005-05, snapshot-2005-04, ppc_gencgc_snap_2005-05-14, snapshot-2005-02
Branch point for: mod-arith-branch, ppc_gencgc_branch, release-19a-branch
Initial version.
1 toy 1.1 -*- Mode: text -*-
2     Building CMU CL
3     ===============
4    
5     This document is intended to give you a general overview of the build
6     process (i.e. what needs to be done, in what order, and what is it
7     generally called). It will also tell you how to set up a suitable
8     build environment, how the individual scripts fit into the general
9     scheme of things, and give you a couple of examples.
10    
11     General Requirements
12     --------------------
13    
14     In order to build CMU CL, you will need:
15    
16     a) A working CMU CL binary. There is no way around this requirement!
17    
18     This binary can either be for the platform you want to target, in
19     that case you can either recompile or cross-compile, or for another
20     supported platform, in that case you must cross-compile, obviously.
21    
22     b) A supported C compiler for the C runtime code.
23    
24     Most of the time, this means GNU gcc, though for some ports it
25     means the vendor-supplied C compiler. The compiler must be
26     available under the name specified by your ports Config file.
27    
28     c) GNU make
29    
30     This has to be available either as gmake or make in your PATH, or
31     the MAKE environment variable has to be set to point to the correct
32     binary.
33    
34     d) The CMU CL source code
35    
36     Here you can either use one of the release source tarballs, or
37     check out the source code directly from the public CMUCL CVS
38     repository.
39    
40     If you want to build CMU CL's Motif interface/toolkit, you'll need a
41     working version of the Motif libraries, either true-blue OSF/Motif, or
42     OpenMotif, or Lesstif. The code was developed against 1.2 Motif,
43     though recompilation against 2.x Motif probably works as well.
44    
45     Setting up a build environment
46     ------------------------------
47    
48     1.) Create a base directory and change to it
49    
50     mkdir cmucl ; cd cmucl
51    
52     2.) Fetch the sources and put them into the base directory
53    
54     tar xzf /tmp/cmucl-18d.source.tar.gz
55    
56     or, if you want to use the CVS sources directly:
57    
58     export CVSROOT=:pserver:anonymous@cvs2.cons.org:/home/anoncvs/CVS-cmucl
59     cvs login (password is `anonymous')
60     cvs co src
61    
62     Whatever you do, the sources must be in a directory named src
63     inside the base directory. Since the build tools keep all
64     generated files in separate target directories, the src directory
65     can be read-only (e.g. mounted read-only via NFS, etc.)
66    
67     The build tools are all in the src/tools directory.
68    
69     That's it, you are now ready to build CMU CL.
70    
71    
72     A general outline of the build process
73     --------------------------------------
74    
75     Building CMU CL can happen in one of two ways: Normal recompilation,
76     and cross-compilation. We'll first look at normal recompilation:
77    
78     The recompilation process basically consists of 4 phases/parts:
79    
80     a) Compiling the lisp files that make up the standard kernel.
81    
82     This happens in your current CMU CL process, using your current
83     CMU CL's normal file compiler. This phase currently consists of 3
84     sub-phases, namely those controlled by src/tools/worldcom.lisp,
85     which compiles all the runtime files, src/tools/comcom.lisp, which
86     compiles the compiler (including your chosen backend), and finally
87     src/tools/pclcom.lisp, which compiles PCL, CMU CL's CLOS
88     implementation. The whole phase is often called "world-compile",
89     or "compiling up a world", based on the name of the first
90     sub-phase.
91    
92     b) Building a new kernel.core file out of the so created files
93    
94     This process, which is generally called genesis, and which is
95     controlled by src/tools/worldbuild.lisp, uses the newly compiled
96     files in order to build a new, basic core file, which is then used
97     by the last phase to create a fully functional normal core file.
98     It does this by "loading" the compiled files into an in-core
99     representation of a new core file, which is then dumped out to
100     disk, together with lots of fixups that need to happen once the new
101     core is started.
102    
103     As part of this process, it also creates the file internals.h,
104     which contains information about the general memory layout of the
105     new core and its basic types, their type tags, and the location of
106     several important constants and other variables, that are needed by
107     the C runtime code to work with the given core.
108    
109     So going through genesis is needed to create internals.h, which is
110     needed to compile the C runtime code (i.e. the "lisp" binary).
111     However there is a slight circularity here, since genesis needs as
112     one of its inputs the file target:lisp/lisp.nm, which contains the
113     (slightly pre-treated) output of running nm on the new lisp
114     binary. Genesis uses this information to fixup the addresses of C
115     runtime support functions for calls from Lisp code.
116    
117     However the circularity isn't complete, since genesis can work with
118     an empty/bogus lisp.nm file. While the kernel.core it then
119     produces is unusable, it will create a usable internals.h file,
120     which can be used to recompile the C runtime code, producing a
121     usable lisp.nm file, which in turn can be used to restart genesis,
122     producing a working kernel.core file.
123    
124     Genesis also checks whether the newly produced internals.h file
125     differs from a pre-existing internals.h file (this might be caused
126     by an empty internals.h file if you are rebuilding for the first
127     time, or by changes in the lisp sources that cause differences in
128     the memory layout of the kernel.core), and informs you of this, so
129     that you can recompile the C runtime code, and restart genesis.
130    
131     If it doesn't inform you of this, you can skip directly to the last
132     phase d).
133    
134     c) Recompiling the C runtime code, producing the "lisp" binary file
135    
136     This step is only needed if you haven't yet got a suitable lisp
137     binary, or if the internals.h file has changed during genesis (of
138     which genesis informs you), or when you made changes to the C
139     sources that you want to take effect.
140    
141     Recompiling the C runtime code is controlled by a GNU Makefile, and
142     your target's Config file. It depends on a correct internals.h
143     file as produced by genesis.
144    
145     Note that whenever you recompile the runtime code, for whatever
146     reason, you must redo phase b). Note that if you make changes to
147     the C sources and recompile because of this, you can do that before
148     Phase b), so that you don't have to perform that phase twice.
149    
150     d) Populating the kernel.core, and dumping a new lisp.core file.
151    
152     In this phase, which is controlled by src/tools/worldload.lisp, and
153     hence often called world-load, the kernel.core file is started up
154     using the (possibly new) lisp binary, the remaining files which
155     were compiled in phase a) are loaded into it, and a new lisp.core
156     file is dumped out.
157    
158     When cross-compiling, there is additional phase at the beginning, and
159     some of the phases happen with different hosts/platforms. The initial
160     phase is setting up and compiling the cross-compilation backend, using
161     your current compiler. The new backend is then loaded, and all
162     compilation in phase a) happens using this compiler backend. The
163     creation of the kernel.core file in phase b) happens as usual, while
164     phase c) of course happens on the target platform (if that differs
165     from the host platform), as does the final phase d). Another major
166     difference is that you can't compile PCL using the cross-compiler, so
167     one usually does a normal rebuild using the cross-compiled core on the
168     target platform to get a full CMU CL core.
169    
170     So, now you know all about CMU CL compilation, how does that map onto
171     the scripts included with this little text?
172    
173     Overview of the included build scripts
174     --------------------------------------
175    
176     * create-target.sh target-directory [lisp-variant [motif-variant]]
177    
178     This script creates a new target directory, which is a shadow of the
179     source directory, that will contain all the files that are created by
180     the build process. Thus, each target's files are completely separate
181     from the src directory, which could, in fact, be read-only. Hence you
182     can simultaneously build CMUCL for different targets from the same
183     source directory.
184    
185     The first argument is the name of the target directory to create. The
186     remaining arguments are optional. If they are not given, the script
187     tries to determine the lisp variant and motif variant from the system
188     the script is running on.
189    
190     The lisp-variant (i.e. the suffix of the src/lisp/Config.* to use as
191     the target's Config file), and optionally the motif-variant (again the
192     suffix of the src/motif/server/Config.* file to use as the Config file
193     for the target's CMUCL/Motif server code). If the lisp-variant is
194     given but the motif-variant is not, the motif-variant is determined
195     from the lisp-variant.
196    
197     The script will generate the target directory tree, link the relevant
198     Config files, and generate place-holder files for various files, in
199     order to ensure proper operation of the other build-scripts. It also
200     creates a sample setenv.lisp file in the target directory, which is
201     used by the build and load processes to set up the correct list of
202     *features* for your target lisp core.
203    
204     IMPORTANT: You will normally NOT have to modify the sample setenv.lisp
205     file, if you are building from a binary that has the desired features.
206     In fact, the sample has all code commented out, If you want to add or
207     remove features, you need to include code that puts at least a minimal
208     set of features onto the list (use PUSHNEW and/or REMOVE). You can
209     use the current set of *features* of your lisp as a first guide. The
210     sample setenv.lisp includes a set of features that should work for the
211     intended configuration. Note also that some adding or removing some
212     features may require a cross-compile instead of a normal compile.
213    
214     * clean-target.sh [-l] target-directory [more dirs]
215    
216     Cleans the given target directory, so that all created files will be
217     removed. This is useful to force recompilation. If the -l flag is
218     given, then the C runtime is also removed, including all the lisp
219     executable, any lisp cores, all object files, lisp.nm, internals.h,
220     and the config file.
221    
222     * build-world.sh target-directory [build-binary] [build-flags...]
223    
224     Starts a complete world build for the given target, using the lisp
225     binary/core specified as a build host. The recompilation step will
226     only recompile changed files, or files for which the fasl files are
227     missing. It will also not recompile the C runtime code (the lisp
228     binary). If a (re)compilation of that code is needed, the genesis
229     step of the world build will inform you of that fact. In that case,
230     you'll have to use the rebuild-lisp.sh script, and then restart the
231     world build process with build-world.sh
232    
233     * rebuild-lisp.sh target-directory
234    
235     This script will force a complete recompilation of the C runtime code
236     of CMU CL (aka the lisp executable). Doing this will necessitate
237     building a new kernel.core file, using build-world.sh.
238    
239     * load-world.sh target-directory version
240    
241     This will finish the CMU CL rebuilding process, by loading the
242     remaining compiled files generated in the world build process into the
243     kernel.core file, that also resulted from that process, creating the
244     final lisp.core file.
245    
246     You have to pass the version string as a second argument. The dumped
247     core will anounce itself using that string. Please don't use a string
248     consisting of an official release name only, (e.g. "18d"), since those
249     are reserved for official release builds. Including the build-date in
250     ISO8601 format is often a good idea, e.g. "18d+ 2002-05-06" for a
251     binary that is based on sources current on the 6th May, 2002, which is
252     post the 18d release.
253    
254     * build-utils.sh target-directory
255    
256     This script will build auxiliary libraries packaged with CMU CL,
257     including CLX, CMUCL/Motif, the Motif debugger, inspector, and control
258     panel, and the Hemlock editor. It will use the lisp executable and
259     core of the given target.
260    
261     * make-dist.sh [-bg] [-G group] [-O owner] target-directory version arch os
262    
263     This script creates both main and extra distribution tarballs from the
264     given target directory, using the make-main-dist.sh and
265     make-extra-dist.sh scripts. You can select the compression method via
266     the -b (bzip2) and -g (gzip) flags. The default is gzip compression.
267     When making the distribution, you can also select the group and owner
268     of the files via the -G and -O options.
269    
270     * make-main-dist.sh target-directory version arch os
271    
272     This script creates a main distribution tarball (both in gzipped and
273     bzipped variants) from the given target directory. This will include
274     all the stuff that is normally included in official release tarballs
275     such as lisp.core and the PCL libraries, including Gray streams and
276     simple streams.
277    
278     This is intended to be run from make-dist.sh.
279    
280     * make-extra-dist.sh target-directory version arch os
281    
282     This script creates an extra distribution tarball (both in gzipped and
283     bzipped variants) from the given target directory. This will include
284     all the stuff that is normally included in official extra release
285     tarballs, i.e. the auxiliary libraries such as CLX, CLM, and Hemlock.
286    
287     This is intended to be run from make-dist.sh.
288    
289    
290     * cross-build-world.sh target-directory cross-directory cross-script
291     [build-binary] [build-flags...]
292    
293     This is a script that can be used instead of build-world.sh for
294     cross-compiling CMUCL. In addition to the arguments of build-world.sh
295     it takes two further required arguments: The name of a directory that
296     will contain the cross-compiler backend (the directory is created if
297     it doesn't exist, and must not be the same as the target-directory),
298     and the name of a Lisp cross-compilation script, which is responsible
299     for setting up, compiling, and loading the cross-compiler backend.
300     The latter argument is needed because each host/target combination of
301     platform's needs slightly different code to produce a working
302     cross-compiler.
303    
304     We include a number of working examples of cross-compiler scripts in
305     the cross-scripts directory. You'll have to edit the features section
306     of the given scripts, to specify the features that should be removed
307     from the current set of features in the host lisp, and those that
308     should be added, so that the backend features are correct for the
309     intended target.
310    
311     You can look at Eric Marsden's collection of build scripts for the
312     basis of more cross-compiler scripts.
313    
314     Step-by-Step Example of recompiling CMUCL for OpenBSD
315     -----------------------------------------------------
316    
317     Set up everything as described in the setup section above. Then
318     execute:
319    
320     # Create a new target directory structure/config for OpenBSD:
321     src/tools/create-target.sh openbsd OpenBSD_gencgc OpenBSD
322    
323     # edit openbsd/setenv.lisp to contain what we want:
324     cat <<EOF > openbsd/setenv.lisp
325     ;;; Put code to massage *features* list here...
326    
327     (in-package :user)
328    
329     (pushnew :openbsd *features*)
330     (pushnew :bsd *features*)
331     (pushnew :i486 *features*)
332     (pushnew :mp *features*)
333     (pushnew :hash-new *features*)
334     (pushnew :random-mt19937 *features*)
335     (pushnew :conservative-float-type *features*)
336     (pushnew :gencgc *features*)
337    
338     ;;; Version tags
339    
340     (pushnew :cmu18d *features*)
341     (pushnew :cmu18 *features*)
342     (setf *features* (remove :cmu17 *features*))
343     (setf *features* (remove :cmu18c *features*))
344     EOF
345    
346     # Recompile the lisp world, and dump a new kernel.core:
347     src/tools/build-world.sh openbsd lisp # Or whatever you need to invoke your
348     # current lisp binary+core
349    
350     # If build-world tells you (as it will the first time) that:
351     # "The C header file has changed. Be sure to re-compile the startup
352     # code."
353     # You 'll need to start rebuild-lisp.sh to do that, and then reinvoke
354     # build-world.sh:
355    
356     # Recompile lisp binary itself:
357     src/tools/rebuild-lisp.sh openbsd
358    
359     # Restart build-world.sh now:
360     src/tools/build-world.sh openbsd lisp
361    
362     # Now we populate the kernel.core with further compiled files,
363     # and dump the final lisp.core file:
364    
365     src/tools/load-world.sh openbsd "18d+ 2002-05-06"
366    
367     # The second argument above is the version number that the built
368     # core will announce. Please always put the build-date and some
369     # other information in there, to make it possible to differentiate
370     # those builds from official builds, which only contain the release.
371    
372     Now you should have a new lisp.core, which you can start with
373    
374     ./openbsd/lisp/lisp -core ./openbsd/lisp/lisp.core -noinit -nositeinit
375    
376     Compiling sources that contain disruptive changes
377     -------------------------------------------------
378    
379     The above instructions should always work as-is for recompiling CMU CL
380     using matching binaries and source files. They also work quite often
381     when recompiling newer sources. However, every so often, some change
382     to the CMU CL sources necessitates some form of bootstrapping, so that
383     binaries built from earlier sources can compile the sources containing
384     that change. There are two forms of boostrapping that can be
385     required:
386    
387     a) Bootfiles
388    
389     The maintainers try to make bootfiles available, that allow going
390     from an old release to the next release. These are located in the
391     src/bootfiles/<old-release>/ directory of the CMU CL sources.
392    
393     I.e. if you have binaries that match release 18d, then you'll need
394     to use all the bootfiles in src/bootfiles/18d/ in order to go to
395     the next release (or current sources, if no release has been made
396     yet). If you already used some of the bootstrap files to compile
397     your current lisp, you obviously don't need to use those to get to
398     later versions.
399    
400     You can use the bootfiles by concatenating them into a file called
401     bootstrap.lisp in the target directory (i.e. target:bootstrap.lisp)
402     in the order they are numbered. Be sure to remove the bootstrap
403     file once it is no longer needed.
404    
405    
406     b) Cross-compiling
407    
408     Under some circumstances, bootstrap code will not be sufficient,
409     and a cross-compilation is needed. In that case you will have to
410     use cross-build-world.sh, instead of build-world.sh. Please read
411     the instructions of that script for details of the more complex
412     procedure.
413    
414     << This isn't really true anymore, and we should place a more
415     elaborate description of the cross-compiling process here >>
416    
417     When cross-compiling, there are two sorts of bootscripts that can be
418     used: Those that want to be executed prior to compiling and loading
419     the cross-compiler, which should be placed in the file called
420     target:cross-bootstrap.lisp, and those that should happen after the
421     cross-compiler has been compiled and loaded, just prior to compiling
422     the target, which should be placed in target:bootstrap.lisp, just
423     like when doing a normal recompile.
424    
425     Additionally, sometimes customized cross-compiler setup scripts
426     (to be used in place of e.g. cross-x86-x86.lisp) are required,
427     which are also placed in one of the bootfiles/*/* files. In those
428     cases follow the instructions provided in that file, possibly merging
429     the changed contents thereof with your normal cross-script.
430    
431     Step-by-Step Example of Cross-Compiling
432     ---------------------------------------
433    
434     This gives a step-by-step example of cross-compiling a sparc-v8 build
435     using a sparc-v9 build. (For some unknown reason, you can't just
436     remove the :sparc-v9 feature and add :sparc-v8.)
437    
438     So, first get a recent sparc-v9 build. It's best to get a version
439     that is up-to-date with the sources. Otherwise, you may also need to
440     add a bootstrap file to get any bootfiles to make your lisp
441     up-to-date with the current sources.
442    
443     Create a cross-compiler directory to hold the cross-compiler and a
444     target directory to hold the result:
445    
446     src/tools/create-target.sh xcross
447     src/tools/create-target.sh xtarget
448    
449     Copy the src/tools/cross-scripts/cross-sparc-sparc.lisp to
450     xtarget/cross.lisp. Edit it appropriately. In this case, it should
451     look something like:
452    
453     (c::new-backend "SPARC"
454     ;; Features to add here
455     '(:sparc :sparc-v8
456     :complex-fp-vops
457     :linkage-table
458     :gencgc
459     :stack-checking
460     :relative-package-names
461     :conservative-float-type
462     :hash-new :random-mt19937
463     :cmu :cmu19 :cmu19a
464     )
465     ;; Features to remove from current *features* here
466     '(:sparc-v9 :sparc-v7 :x86 :x86-bootstrap :alpha :osf1 :mips
467     :propagate-fun-type :propagate-float-type :constrain-float-type
468     :openbsd :freebsd :glibc2 :linux :pentium
469     :long-float :new-random :small))
470    
471     (setf *features* (remove :sparc-v9 *features*))
472     (pushnew :sparc-v8 *features*)
473    
474     It's important to add frob *features* here as well as in the
475     new-backend. If you don't adjust *features*, they won't be set
476     appropriately in the result.
477    
478     Now compile the result:
479    
480     src/tools/cross-build-world.sh xtarget xcross xtarget/cross.lisp [v9 binary]
481    
482     When this finishes, you need to compile the C code:
483    
484     src/tools/rebuild-lisp.sh xtarget
485    
486     At this point, you may want to run cross-build-world.sh again to
487     generate a new kernel.core. It shouldn't build anything; just loads
488     everything and creates a kernel.core.
489    
490     With the new kernel.core, we need to create a lisp.core:
491    
492     src/tools/load-world.sh xtarget "new lisp"
493    
494     Test the result with
495    
496     xtarget/lisp/lisp -noinit
497    
498     You may now want to use this cross-compiled lisp to rebuild itself, to
499     make sure you haven't messed up. You can do a normal build as
500     described above, except your build-lisp is xtarget/lisp/lisp.
501    

  ViewVC Help
Powered by ViewVC 1.1.5