/[cmucl]/src/BUILDING
ViewVC logotype

Contents of /src/BUILDING

Parent Directory Parent Directory | Revision Log Revision Log


Revision 1.2 - (hide annotations)
Tue May 17 16:14:05 2005 UTC (8 years, 11 months ago) by rtoy
Branch: MAIN
Changes since 1.1: +114 -58 lines
Update with some fixes from src/tools/README.  Correct some other
small items.
1 toy 1.1 -*- Mode: text -*-
2     Building CMU CL
3     ===============
4    
5     This document is intended to give you a general overview of the build
6     process (i.e. what needs to be done, in what order, and what is it
7     generally called). It will also tell you how to set up a suitable
8     build environment, how the individual scripts fit into the general
9     scheme of things, and give you a couple of examples.
10    
11     General Requirements
12     --------------------
13    
14     In order to build CMU CL, you will need:
15    
16     a) A working CMU CL binary. There is no way around this requirement!
17    
18     This binary can either be for the platform you want to target, in
19     that case you can either recompile or cross-compile, or for another
20     supported platform, in that case you must cross-compile, obviously.
21    
22     b) A supported C compiler for the C runtime code.
23    
24     Most of the time, this means GNU gcc, though for some ports it
25     means the vendor-supplied C compiler. The compiler must be
26     available under the name specified by your ports Config file.
27    
28     c) GNU make
29    
30     This has to be available either as gmake or make in your PATH, or
31     the MAKE environment variable has to be set to point to the correct
32     binary.
33    
34     d) The CMU CL source code
35    
36     Here you can either use one of the release source tarballs, or
37     check out the source code directly from the public CMUCL CVS
38     repository.
39    
40     If you want to build CMU CL's Motif interface/toolkit, you'll need a
41     working version of the Motif libraries, either true-blue OSF/Motif, or
42     OpenMotif, or Lesstif. The code was developed against 1.2 Motif,
43     though recompilation against 2.x Motif probably works as well.
44    
45     Setting up a build environment
46     ------------------------------
47    
48     1.) Create a base directory and change to it
49    
50     mkdir cmucl ; cd cmucl
51    
52     2.) Fetch the sources and put them into the base directory
53    
54 rtoy 1.2 tar xzf /tmp/cmucl-source.tar.gz
55 toy 1.1
56     or, if you want to use the CVS sources directly:
57    
58 rtoy 1.2 export CVSROOT=:pserver:anonymous@common-lisp.net:/project/cmucl/cvsroot
59 toy 1.1 cvs login (password is `anonymous')
60     cvs co src
61    
62     Whatever you do, the sources must be in a directory named src
63     inside the base directory. Since the build tools keep all
64     generated files in separate target directories, the src directory
65     can be read-only (e.g. mounted read-only via NFS, etc.)
66    
67     The build tools are all in the src/tools directory.
68    
69     That's it, you are now ready to build CMU CL.
70    
71    
72     A general outline of the build process
73     --------------------------------------
74    
75     Building CMU CL can happen in one of two ways: Normal recompilation,
76     and cross-compilation. We'll first look at normal recompilation:
77    
78     The recompilation process basically consists of 4 phases/parts:
79    
80     a) Compiling the lisp files that make up the standard kernel.
81    
82     This happens in your current CMU CL process, using your current
83     CMU CL's normal file compiler. This phase currently consists of 3
84     sub-phases, namely those controlled by src/tools/worldcom.lisp,
85     which compiles all the runtime files, src/tools/comcom.lisp, which
86     compiles the compiler (including your chosen backend), and finally
87     src/tools/pclcom.lisp, which compiles PCL, CMU CL's CLOS
88     implementation. The whole phase is often called "world-compile",
89     or "compiling up a world", based on the name of the first
90     sub-phase.
91    
92     b) Building a new kernel.core file out of the so created files
93    
94     This process, which is generally called genesis, and which is
95     controlled by src/tools/worldbuild.lisp, uses the newly compiled
96     files in order to build a new, basic core file, which is then used
97     by the last phase to create a fully functional normal core file.
98     It does this by "loading" the compiled files into an in-core
99     representation of a new core file, which is then dumped out to
100     disk, together with lots of fixups that need to happen once the new
101     core is started.
102    
103     As part of this process, it also creates the file internals.h,
104     which contains information about the general memory layout of the
105     new core and its basic types, their type tags, and the location of
106     several important constants and other variables, that are needed by
107     the C runtime code to work with the given core.
108    
109     So going through genesis is needed to create internals.h, which is
110     needed to compile the C runtime code (i.e. the "lisp" binary).
111     However there is a slight circularity here, since genesis needs as
112     one of its inputs the file target:lisp/lisp.nm, which contains the
113     (slightly pre-treated) output of running nm on the new lisp
114     binary. Genesis uses this information to fixup the addresses of C
115     runtime support functions for calls from Lisp code.
116    
117     However the circularity isn't complete, since genesis can work with
118     an empty/bogus lisp.nm file. While the kernel.core it then
119     produces is unusable, it will create a usable internals.h file,
120     which can be used to recompile the C runtime code, producing a
121     usable lisp.nm file, which in turn can be used to restart genesis,
122     producing a working kernel.core file.
123    
124     Genesis also checks whether the newly produced internals.h file
125     differs from a pre-existing internals.h file (this might be caused
126     by an empty internals.h file if you are rebuilding for the first
127     time, or by changes in the lisp sources that cause differences in
128     the memory layout of the kernel.core), and informs you of this, so
129     that you can recompile the C runtime code, and restart genesis.
130    
131     If it doesn't inform you of this, you can skip directly to the last
132     phase d).
133    
134     c) Recompiling the C runtime code, producing the "lisp" binary file
135    
136     This step is only needed if you haven't yet got a suitable lisp
137     binary, or if the internals.h file has changed during genesis (of
138     which genesis informs you), or when you made changes to the C
139     sources that you want to take effect.
140    
141     Recompiling the C runtime code is controlled by a GNU Makefile, and
142     your target's Config file. It depends on a correct internals.h
143     file as produced by genesis.
144    
145     Note that whenever you recompile the runtime code, for whatever
146     reason, you must redo phase b). Note that if you make changes to
147     the C sources and recompile because of this, you can do that before
148     Phase b), so that you don't have to perform that phase twice.
149    
150     d) Populating the kernel.core, and dumping a new lisp.core file.
151    
152     In this phase, which is controlled by src/tools/worldload.lisp, and
153     hence often called world-load, the kernel.core file is started up
154     using the (possibly new) lisp binary, the remaining files which
155     were compiled in phase a) are loaded into it, and a new lisp.core
156     file is dumped out.
157    
158 rtoy 1.2 We're not quite done yet. This produces just a basic lisp.core.
159     To complete the build so that you something similar to what the
160     releases of CMUCL do, there are a few more steps:
161    
162     e) Build the utilities like Gray streams, simple streams, CLX, CLM,
163     and Hemlock. Use the src/tools/build-utils.sh script for this, as
164     described below
165    
166     f) Create tarfiles using the src/tools/make-dist.sh script, as
167     explained below.
168    
169     With these tarfiles, you can install them anywhere. The contents of
170     the tarfiles will be the same as the snapshots and releases of CMUCL.
171    
172 toy 1.1 When cross-compiling, there is additional phase at the beginning, and
173     some of the phases happen with different hosts/platforms. The initial
174     phase is setting up and compiling the cross-compilation backend, using
175     your current compiler. The new backend is then loaded, and all
176     compilation in phase a) happens using this compiler backend. The
177     creation of the kernel.core file in phase b) happens as usual, while
178     phase c) of course happens on the target platform (if that differs
179     from the host platform), as does the final phase d). Another major
180     difference is that you can't compile PCL using the cross-compiler, so
181     one usually does a normal rebuild using the cross-compiled core on the
182     target platform to get a full CMU CL core.
183    
184     So, now you know all about CMU CL compilation, how does that map onto
185     the scripts included with this little text?
186    
187     Overview of the included build scripts
188     --------------------------------------
189    
190 rtoy 1.2 * src/tools/create-target.sh target-directory [lisp-variant [motif-variant]]
191 toy 1.1
192     This script creates a new target directory, which is a shadow of the
193     source directory, that will contain all the files that are created by
194     the build process. Thus, each target's files are completely separate
195     from the src directory, which could, in fact, be read-only. Hence you
196     can simultaneously build CMUCL for different targets from the same
197     source directory.
198    
199     The first argument is the name of the target directory to create. The
200     remaining arguments are optional. If they are not given, the script
201     tries to determine the lisp variant and motif variant from the system
202     the script is running on.
203    
204     The lisp-variant (i.e. the suffix of the src/lisp/Config.* to use as
205     the target's Config file), and optionally the motif-variant (again the
206     suffix of the src/motif/server/Config.* file to use as the Config file
207     for the target's CMUCL/Motif server code). If the lisp-variant is
208     given but the motif-variant is not, the motif-variant is determined
209     from the lisp-variant.
210    
211     The script will generate the target directory tree, link the relevant
212     Config files, and generate place-holder files for various files, in
213     order to ensure proper operation of the other build-scripts. It also
214     creates a sample setenv.lisp file in the target directory, which is
215     used by the build and load processes to set up the correct list of
216     *features* for your target lisp core.
217    
218     IMPORTANT: You will normally NOT have to modify the sample setenv.lisp
219     file, if you are building from a binary that has the desired features.
220     In fact, the sample has all code commented out, If you want to add or
221     remove features, you need to include code that puts at least a minimal
222     set of features onto the list (use PUSHNEW and/or REMOVE). You can
223     use the current set of *features* of your lisp as a first guide. The
224     sample setenv.lisp includes a set of features that should work for the
225     intended configuration. Note also that some adding or removing some
226     features may require a cross-compile instead of a normal compile.
227    
228 rtoy 1.2 * src/tools/clean-target.sh [-l] target-directory [more dirs]
229 toy 1.1
230     Cleans the given target directory, so that all created files will be
231     removed. This is useful to force recompilation. If the -l flag is
232     given, then the C runtime is also removed, including all the lisp
233     executable, any lisp cores, all object files, lisp.nm, internals.h,
234     and the config file.
235    
236 rtoy 1.2 * src/tools/build-world.sh target-directory [build-binary] [build-flags...]
237 toy 1.1
238     Starts a complete world build for the given target, using the lisp
239     binary/core specified as a build host. The recompilation step will
240     only recompile changed files, or files for which the fasl files are
241     missing. It will also not recompile the C runtime code (the lisp
242     binary). If a (re)compilation of that code is needed, the genesis
243     step of the world build will inform you of that fact. In that case,
244     you'll have to use the rebuild-lisp.sh script, and then restart the
245     world build process with build-world.sh
246    
247 rtoy 1.2 * src/tools/rebuild-lisp.sh target-directory
248 toy 1.1
249     This script will force a complete recompilation of the C runtime code
250     of CMU CL (aka the lisp executable). Doing this will necessitate
251     building a new kernel.core file, using build-world.sh.
252    
253 rtoy 1.2 * src/tools/load-world.sh target-directory version
254 toy 1.1
255     This will finish the CMU CL rebuilding process, by loading the
256     remaining compiled files generated in the world build process into the
257     kernel.core file, that also resulted from that process, creating the
258     final lisp.core file.
259    
260     You have to pass the version string as a second argument. The dumped
261     core will anounce itself using that string. Please don't use a string
262     consisting of an official release name only, (e.g. "18d"), since those
263     are reserved for official release builds. Including the build-date in
264     ISO8601 format is often a good idea, e.g. "18d+ 2002-05-06" for a
265     binary that is based on sources current on the 6th May, 2002, which is
266     post the 18d release.
267    
268 rtoy 1.2 * src/tools/build-utils.sh target-directory
269 toy 1.1
270     This script will build auxiliary libraries packaged with CMU CL,
271     including CLX, CMUCL/Motif, the Motif debugger, inspector, and control
272     panel, and the Hemlock editor. It will use the lisp executable and
273     core of the given target.
274    
275 rtoy 1.2 * src/tools/make-dist.sh [-bg] [-G group] [-O owner] target-directory version arch os
276 toy 1.1
277     This script creates both main and extra distribution tarballs from the
278     given target directory, using the make-main-dist.sh and
279 rtoy 1.2 make-extra-dist.sh scripts. The result will be two tar files. One
280     contains the main distribution including the runtime and lisp.core
281     with PCL (CLOS); the second contains the extra libraries such as
282     Gray-streams, simple-streams, CLX, CLM, and Hemlock.
283    
284     Some options that are available:
285    
286     -b Use bzip2 compression
287     -g Use gzip compression
288     -G group Group to use
289     -O owner Owner to use
290    
291     If you specify both -b and -g, you will get two sets of tarfiles. The
292     -G and -O options will attempt to set the owner and group of the files
293     when building the tarfiles. This way, when you extract the tarfiles,
294     the owner and group will be set as specified. You may need to be root
295     to do this because many Unix systems don't normally let you change the
296     owner and group of a file.
297    
298     The remaining arguments used to create the name of the tarfiles. The
299     names will have the form:
300    
301     cmucl-<version>-<arch>-<os>.tar.bz2
302     cmucl-<version>-<arch>-<os>.extras.tar.bz2
303    
304     Of course, the "bz2" will be "gz" if you specified gzip compression
305     instead of bzip.
306    
307     * /src/tools/make-main-dist.sh target-directory version arch os
308 toy 1.1
309 rtoy 1.2 This is script is not normally invoked by the user; make-dist will do
310     it appropriately.
311 toy 1.1
312     This script creates a main distribution tarball (both in gzipped and
313     bzipped variants) from the given target directory. This will include
314     all the stuff that is normally included in official release tarballs
315     such as lisp.core and the PCL libraries, including Gray streams and
316     simple streams.
317    
318     This is intended to be run from make-dist.sh.
319    
320 rtoy 1.2 * src/tools/make-extra-dist.sh target-directory version arch os
321    
322     This is script is not normally invoked by the user; make-dist will do
323     it appropriately.
324 toy 1.1
325     This script creates an extra distribution tarball (both in gzipped and
326     bzipped variants) from the given target directory. This will include
327     all the stuff that is normally included in official extra release
328     tarballs, i.e. the auxiliary libraries such as CLX, CLM, and Hemlock.
329    
330     This is intended to be run from make-dist.sh.
331    
332    
333     * cross-build-world.sh target-directory cross-directory cross-script
334     [build-binary] [build-flags...]
335    
336     This is a script that can be used instead of build-world.sh for
337     cross-compiling CMUCL. In addition to the arguments of build-world.sh
338     it takes two further required arguments: The name of a directory that
339     will contain the cross-compiler backend (the directory is created if
340     it doesn't exist, and must not be the same as the target-directory),
341     and the name of a Lisp cross-compilation script, which is responsible
342     for setting up, compiling, and loading the cross-compiler backend.
343     The latter argument is needed because each host/target combination of
344     platform's needs slightly different code to produce a working
345     cross-compiler.
346    
347     We include a number of working examples of cross-compiler scripts in
348     the cross-scripts directory. You'll have to edit the features section
349     of the given scripts, to specify the features that should be removed
350     from the current set of features in the host lisp, and those that
351     should be added, so that the backend features are correct for the
352     intended target.
353    
354     You can look at Eric Marsden's collection of build scripts for the
355     basis of more cross-compiler scripts.
356    
357     Step-by-Step Example of recompiling CMUCL for OpenBSD
358     -----------------------------------------------------
359    
360     Set up everything as described in the setup section above. Then
361     execute:
362    
363     # Create a new target directory structure/config for OpenBSD:
364     src/tools/create-target.sh openbsd OpenBSD_gencgc OpenBSD
365    
366     # edit openbsd/setenv.lisp to contain what we want:
367     cat <<EOF > openbsd/setenv.lisp
368     ;;; Put code to massage *features* list here...
369    
370     (in-package :user)
371    
372     (pushnew :openbsd *features*)
373     (pushnew :bsd *features*)
374     (pushnew :i486 *features*)
375     (pushnew :mp *features*)
376     (pushnew :hash-new *features*)
377     (pushnew :random-mt19937 *features*)
378     (pushnew :conservative-float-type *features*)
379     (pushnew :gencgc *features*)
380    
381     ;;; Version tags
382    
383     (pushnew :cmu18d *features*)
384     (pushnew :cmu18 *features*)
385     (setf *features* (remove :cmu17 *features*))
386     (setf *features* (remove :cmu18c *features*))
387     EOF
388    
389     # Recompile the lisp world, and dump a new kernel.core:
390     src/tools/build-world.sh openbsd lisp # Or whatever you need to invoke your
391     # current lisp binary+core
392    
393     # If build-world tells you (as it will the first time) that:
394     # "The C header file has changed. Be sure to re-compile the startup
395     # code."
396     # You 'll need to start rebuild-lisp.sh to do that, and then reinvoke
397     # build-world.sh:
398    
399     # Recompile lisp binary itself:
400     src/tools/rebuild-lisp.sh openbsd
401    
402     # Restart build-world.sh now:
403     src/tools/build-world.sh openbsd lisp
404    
405     # Now we populate the kernel.core with further compiled files,
406     # and dump the final lisp.core file:
407    
408     src/tools/load-world.sh openbsd "18d+ 2002-05-06"
409    
410     # The second argument above is the version number that the built
411     # core will announce. Please always put the build-date and some
412     # other information in there, to make it possible to differentiate
413     # those builds from official builds, which only contain the release.
414    
415     Now you should have a new lisp.core, which you can start with
416    
417     ./openbsd/lisp/lisp -core ./openbsd/lisp/lisp.core -noinit -nositeinit
418    
419     Compiling sources that contain disruptive changes
420     -------------------------------------------------
421    
422     The above instructions should always work as-is for recompiling CMU CL
423     using matching binaries and source files. They also work quite often
424     when recompiling newer sources. However, every so often, some change
425     to the CMU CL sources necessitates some form of bootstrapping, so that
426     binaries built from earlier sources can compile the sources containing
427     that change. There are two forms of boostrapping that can be
428     required:
429    
430     a) Bootfiles
431    
432     The maintainers try to make bootfiles available, that allow going
433     from an old release to the next release. These are located in the
434     src/bootfiles/<old-release>/ directory of the CMU CL sources.
435    
436     I.e. if you have binaries that match release 18d, then you'll need
437     to use all the bootfiles in src/bootfiles/18d/ in order to go to
438     the next release (or current sources, if no release has been made
439     yet). If you already used some of the bootstrap files to compile
440     your current lisp, you obviously don't need to use those to get to
441     later versions.
442    
443     You can use the bootfiles by concatenating them into a file called
444     bootstrap.lisp in the target directory (i.e. target:bootstrap.lisp)
445     in the order they are numbered. Be sure to remove the bootstrap
446     file once it is no longer needed.
447    
448 rtoy 1.2 Alternatively, the bootstrap file can just "load" the individual
449     bootfiles as needed.
450 toy 1.1
451     b) Cross-compiling
452    
453     Under some circumstances, bootstrap code will not be sufficient,
454     and a cross-compilation is needed. In that case you will have to
455     use cross-build-world.sh, instead of build-world.sh. Please read
456     the instructions of that script for details of the more complex
457     procedure.
458    
459     << This isn't really true anymore, and we should place a more
460     elaborate description of the cross-compiling process here >>
461    
462     When cross-compiling, there are two sorts of bootscripts that can be
463     used: Those that want to be executed prior to compiling and loading
464     the cross-compiler, which should be placed in the file called
465     target:cross-bootstrap.lisp, and those that should happen after the
466     cross-compiler has been compiled and loaded, just prior to compiling
467     the target, which should be placed in target:bootstrap.lisp, just
468     like when doing a normal recompile.
469    
470     Additionally, sometimes customized cross-compiler setup scripts
471     (to be used in place of e.g. cross-x86-x86.lisp) are required,
472     which are also placed in one of the bootfiles/*/* files. In those
473     cases follow the instructions provided in that file, possibly merging
474     the changed contents thereof with your normal cross-script.
475    
476     Step-by-Step Example of Cross-Compiling
477     ---------------------------------------
478    
479     This gives a step-by-step example of cross-compiling a sparc-v8 build
480     using a sparc-v9 build. (For some unknown reason, you can't just
481     remove the :sparc-v9 feature and add :sparc-v8.)
482    
483     So, first get a recent sparc-v9 build. It's best to get a version
484     that is up-to-date with the sources. Otherwise, you may also need to
485     add a bootstrap file to get any bootfiles to make your lisp
486     up-to-date with the current sources.
487    
488 rtoy 1.2 1. Select a directory for the cross-compiler and compiled target:
489    
490     Create a cross-compiler directory to hold the cross-compiler
491     and a target directory to hold the result:
492    
493     src/tools/create-target.sh xcross
494     src/tools/create-target.sh xtarget
495    
496     2. Adjust cross-compilation script
497    
498     Copy the src/tools/cross-scripts/cross-sparc-sparc.lisp to
499     xtarget/cross.lisp. Edit it appropriately. In this case, it
500     should look something like:
501 toy 1.1
502 rtoy 1.2 (c::new-backend "SPARC"
503     ;; Features to add here
504     '(:sparc :sparc-v8
505     :complex-fp-vops
506     :linkage-table
507     :gencgc
508     :stack-checking
509     :relative-package-names
510     :conservative-float-type
511     :hash-new :random-mt19937
512     :cmu :cmu19 :cmu19a
513     )
514     ;; Features to remove from current *features* here
515     '(:sparc-v9 :sparc-v7 :x86 :x86-bootstrap :alpha :osf1 :mips
516     :propagate-fun-type :propagate-float-type :constrain-float-type
517     :openbsd :freebsd :glibc2 :linux :pentium
518     :long-float :new-random :small))
519 toy 1.1
520 rtoy 1.2 (setf *features* (remove :sparc-v9 *features*))
521     (pushnew :sparc-v8 *features*)
522 toy 1.1
523 rtoy 1.2 It's important to add frob *features* here as well as in the
524     new-backend. If you don't adjust *features*, they won't be
525     set appropriately in the result.
526 toy 1.1
527 rtoy 1.2 3. Build the cross compiler and target
528     Now compile the result:
529 toy 1.1
530 rtoy 1.2 src/tools/cross-build-world.sh xtarget xcross xtarget/cross.lisp [v9 binary]
531 toy 1.1
532 rtoy 1.2 4. Rebuild the lisp files:
533 toy 1.1
534 rtoy 1.2 When this finishes, you need to compile the C code:
535 toy 1.1
536 rtoy 1.2 src/tools/rebuild-lisp.sh xtarget
537 toy 1.1
538 rtoy 1.2 At this point, you may want to run cross-build-world.sh again
539     to generate a new kernel.core. It shouldn't build anything;
540     just loads everything and creates a kernel.core.
541 toy 1.1
542 rtoy 1.2 5. Build the world:
543 toy 1.1
544 rtoy 1.2 With the new kernel.core, we need to create a lisp.core:
545 toy 1.1
546 rtoy 1.2 src/tools/load-world.sh xtarget "new lisp"
547 toy 1.1
548 rtoy 1.2 Test the result with
549 toy 1.1
550 rtoy 1.2 xtarget/lisp/lisp -noinit
551 toy 1.1
552 rtoy 1.2 However, this lisp will be missing some functionality like PCL. You
553     probably now want to use the compiler to rebuild everything once
554     again. Just follow the directions for a normal build, and use
555     xtarget/lisp/lisp as your compiler. Be sure to use create-target.sh
556     to create a new directory where the result can go.
557 toy 1.1

  ViewVC Help
Powered by ViewVC 1.1.5