/[cmucl]/src/BUILDING
ViewVC logotype

Contents of /src/BUILDING

Parent Directory Parent Directory | Revision Log Revision Log


Revision 1.1 - (show annotations)
Fri Oct 24 16:55:36 2003 UTC (10 years, 5 months ago) by toy
Branch: MAIN
CVS Tags: snapshot-2004-10, snapshot-2004-08, snapshot-2004-09, snapshot-2004-05, snapshot-2004-06, snapshot-2004-07, mod-arith-base, snapshot-2004-12, snapshot-2004-11, amd64-merge-start, prm-before-macosx-merge-tag, snapshot-2003-11, release-19a-base, snapshot-2003-12, release-19a-pre1, release-19a-pre3, release-19a-pre2, release-19a, snapshot-2005-03, snapshot-2004-04, snapshot-2005-01, snapshot-2005-05, snapshot-2005-04, ppc_gencgc_snap_2005-05-14, snapshot-2005-02
Branch point for: mod-arith-branch, ppc_gencgc_branch, release-19a-branch
Initial version.
1 -*- Mode: text -*-
2 Building CMU CL
3 ===============
4
5 This document is intended to give you a general overview of the build
6 process (i.e. what needs to be done, in what order, and what is it
7 generally called). It will also tell you how to set up a suitable
8 build environment, how the individual scripts fit into the general
9 scheme of things, and give you a couple of examples.
10
11 General Requirements
12 --------------------
13
14 In order to build CMU CL, you will need:
15
16 a) A working CMU CL binary. There is no way around this requirement!
17
18 This binary can either be for the platform you want to target, in
19 that case you can either recompile or cross-compile, or for another
20 supported platform, in that case you must cross-compile, obviously.
21
22 b) A supported C compiler for the C runtime code.
23
24 Most of the time, this means GNU gcc, though for some ports it
25 means the vendor-supplied C compiler. The compiler must be
26 available under the name specified by your ports Config file.
27
28 c) GNU make
29
30 This has to be available either as gmake or make in your PATH, or
31 the MAKE environment variable has to be set to point to the correct
32 binary.
33
34 d) The CMU CL source code
35
36 Here you can either use one of the release source tarballs, or
37 check out the source code directly from the public CMUCL CVS
38 repository.
39
40 If you want to build CMU CL's Motif interface/toolkit, you'll need a
41 working version of the Motif libraries, either true-blue OSF/Motif, or
42 OpenMotif, or Lesstif. The code was developed against 1.2 Motif,
43 though recompilation against 2.x Motif probably works as well.
44
45 Setting up a build environment
46 ------------------------------
47
48 1.) Create a base directory and change to it
49
50 mkdir cmucl ; cd cmucl
51
52 2.) Fetch the sources and put them into the base directory
53
54 tar xzf /tmp/cmucl-18d.source.tar.gz
55
56 or, if you want to use the CVS sources directly:
57
58 export CVSROOT=:pserver:anonymous@cvs2.cons.org:/home/anoncvs/CVS-cmucl
59 cvs login (password is `anonymous')
60 cvs co src
61
62 Whatever you do, the sources must be in a directory named src
63 inside the base directory. Since the build tools keep all
64 generated files in separate target directories, the src directory
65 can be read-only (e.g. mounted read-only via NFS, etc.)
66
67 The build tools are all in the src/tools directory.
68
69 That's it, you are now ready to build CMU CL.
70
71
72 A general outline of the build process
73 --------------------------------------
74
75 Building CMU CL can happen in one of two ways: Normal recompilation,
76 and cross-compilation. We'll first look at normal recompilation:
77
78 The recompilation process basically consists of 4 phases/parts:
79
80 a) Compiling the lisp files that make up the standard kernel.
81
82 This happens in your current CMU CL process, using your current
83 CMU CL's normal file compiler. This phase currently consists of 3
84 sub-phases, namely those controlled by src/tools/worldcom.lisp,
85 which compiles all the runtime files, src/tools/comcom.lisp, which
86 compiles the compiler (including your chosen backend), and finally
87 src/tools/pclcom.lisp, which compiles PCL, CMU CL's CLOS
88 implementation. The whole phase is often called "world-compile",
89 or "compiling up a world", based on the name of the first
90 sub-phase.
91
92 b) Building a new kernel.core file out of the so created files
93
94 This process, which is generally called genesis, and which is
95 controlled by src/tools/worldbuild.lisp, uses the newly compiled
96 files in order to build a new, basic core file, which is then used
97 by the last phase to create a fully functional normal core file.
98 It does this by "loading" the compiled files into an in-core
99 representation of a new core file, which is then dumped out to
100 disk, together with lots of fixups that need to happen once the new
101 core is started.
102
103 As part of this process, it also creates the file internals.h,
104 which contains information about the general memory layout of the
105 new core and its basic types, their type tags, and the location of
106 several important constants and other variables, that are needed by
107 the C runtime code to work with the given core.
108
109 So going through genesis is needed to create internals.h, which is
110 needed to compile the C runtime code (i.e. the "lisp" binary).
111 However there is a slight circularity here, since genesis needs as
112 one of its inputs the file target:lisp/lisp.nm, which contains the
113 (slightly pre-treated) output of running nm on the new lisp
114 binary. Genesis uses this information to fixup the addresses of C
115 runtime support functions for calls from Lisp code.
116
117 However the circularity isn't complete, since genesis can work with
118 an empty/bogus lisp.nm file. While the kernel.core it then
119 produces is unusable, it will create a usable internals.h file,
120 which can be used to recompile the C runtime code, producing a
121 usable lisp.nm file, which in turn can be used to restart genesis,
122 producing a working kernel.core file.
123
124 Genesis also checks whether the newly produced internals.h file
125 differs from a pre-existing internals.h file (this might be caused
126 by an empty internals.h file if you are rebuilding for the first
127 time, or by changes in the lisp sources that cause differences in
128 the memory layout of the kernel.core), and informs you of this, so
129 that you can recompile the C runtime code, and restart genesis.
130
131 If it doesn't inform you of this, you can skip directly to the last
132 phase d).
133
134 c) Recompiling the C runtime code, producing the "lisp" binary file
135
136 This step is only needed if you haven't yet got a suitable lisp
137 binary, or if the internals.h file has changed during genesis (of
138 which genesis informs you), or when you made changes to the C
139 sources that you want to take effect.
140
141 Recompiling the C runtime code is controlled by a GNU Makefile, and
142 your target's Config file. It depends on a correct internals.h
143 file as produced by genesis.
144
145 Note that whenever you recompile the runtime code, for whatever
146 reason, you must redo phase b). Note that if you make changes to
147 the C sources and recompile because of this, you can do that before
148 Phase b), so that you don't have to perform that phase twice.
149
150 d) Populating the kernel.core, and dumping a new lisp.core file.
151
152 In this phase, which is controlled by src/tools/worldload.lisp, and
153 hence often called world-load, the kernel.core file is started up
154 using the (possibly new) lisp binary, the remaining files which
155 were compiled in phase a) are loaded into it, and a new lisp.core
156 file is dumped out.
157
158 When cross-compiling, there is additional phase at the beginning, and
159 some of the phases happen with different hosts/platforms. The initial
160 phase is setting up and compiling the cross-compilation backend, using
161 your current compiler. The new backend is then loaded, and all
162 compilation in phase a) happens using this compiler backend. The
163 creation of the kernel.core file in phase b) happens as usual, while
164 phase c) of course happens on the target platform (if that differs
165 from the host platform), as does the final phase d). Another major
166 difference is that you can't compile PCL using the cross-compiler, so
167 one usually does a normal rebuild using the cross-compiled core on the
168 target platform to get a full CMU CL core.
169
170 So, now you know all about CMU CL compilation, how does that map onto
171 the scripts included with this little text?
172
173 Overview of the included build scripts
174 --------------------------------------
175
176 * create-target.sh target-directory [lisp-variant [motif-variant]]
177
178 This script creates a new target directory, which is a shadow of the
179 source directory, that will contain all the files that are created by
180 the build process. Thus, each target's files are completely separate
181 from the src directory, which could, in fact, be read-only. Hence you
182 can simultaneously build CMUCL for different targets from the same
183 source directory.
184
185 The first argument is the name of the target directory to create. The
186 remaining arguments are optional. If they are not given, the script
187 tries to determine the lisp variant and motif variant from the system
188 the script is running on.
189
190 The lisp-variant (i.e. the suffix of the src/lisp/Config.* to use as
191 the target's Config file), and optionally the motif-variant (again the
192 suffix of the src/motif/server/Config.* file to use as the Config file
193 for the target's CMUCL/Motif server code). If the lisp-variant is
194 given but the motif-variant is not, the motif-variant is determined
195 from the lisp-variant.
196
197 The script will generate the target directory tree, link the relevant
198 Config files, and generate place-holder files for various files, in
199 order to ensure proper operation of the other build-scripts. It also
200 creates a sample setenv.lisp file in the target directory, which is
201 used by the build and load processes to set up the correct list of
202 *features* for your target lisp core.
203
204 IMPORTANT: You will normally NOT have to modify the sample setenv.lisp
205 file, if you are building from a binary that has the desired features.
206 In fact, the sample has all code commented out, If you want to add or
207 remove features, you need to include code that puts at least a minimal
208 set of features onto the list (use PUSHNEW and/or REMOVE). You can
209 use the current set of *features* of your lisp as a first guide. The
210 sample setenv.lisp includes a set of features that should work for the
211 intended configuration. Note also that some adding or removing some
212 features may require a cross-compile instead of a normal compile.
213
214 * clean-target.sh [-l] target-directory [more dirs]
215
216 Cleans the given target directory, so that all created files will be
217 removed. This is useful to force recompilation. If the -l flag is
218 given, then the C runtime is also removed, including all the lisp
219 executable, any lisp cores, all object files, lisp.nm, internals.h,
220 and the config file.
221
222 * build-world.sh target-directory [build-binary] [build-flags...]
223
224 Starts a complete world build for the given target, using the lisp
225 binary/core specified as a build host. The recompilation step will
226 only recompile changed files, or files for which the fasl files are
227 missing. It will also not recompile the C runtime code (the lisp
228 binary). If a (re)compilation of that code is needed, the genesis
229 step of the world build will inform you of that fact. In that case,
230 you'll have to use the rebuild-lisp.sh script, and then restart the
231 world build process with build-world.sh
232
233 * rebuild-lisp.sh target-directory
234
235 This script will force a complete recompilation of the C runtime code
236 of CMU CL (aka the lisp executable). Doing this will necessitate
237 building a new kernel.core file, using build-world.sh.
238
239 * load-world.sh target-directory version
240
241 This will finish the CMU CL rebuilding process, by loading the
242 remaining compiled files generated in the world build process into the
243 kernel.core file, that also resulted from that process, creating the
244 final lisp.core file.
245
246 You have to pass the version string as a second argument. The dumped
247 core will anounce itself using that string. Please don't use a string
248 consisting of an official release name only, (e.g. "18d"), since those
249 are reserved for official release builds. Including the build-date in
250 ISO8601 format is often a good idea, e.g. "18d+ 2002-05-06" for a
251 binary that is based on sources current on the 6th May, 2002, which is
252 post the 18d release.
253
254 * build-utils.sh target-directory
255
256 This script will build auxiliary libraries packaged with CMU CL,
257 including CLX, CMUCL/Motif, the Motif debugger, inspector, and control
258 panel, and the Hemlock editor. It will use the lisp executable and
259 core of the given target.
260
261 * make-dist.sh [-bg] [-G group] [-O owner] target-directory version arch os
262
263 This script creates both main and extra distribution tarballs from the
264 given target directory, using the make-main-dist.sh and
265 make-extra-dist.sh scripts. You can select the compression method via
266 the -b (bzip2) and -g (gzip) flags. The default is gzip compression.
267 When making the distribution, you can also select the group and owner
268 of the files via the -G and -O options.
269
270 * make-main-dist.sh target-directory version arch os
271
272 This script creates a main distribution tarball (both in gzipped and
273 bzipped variants) from the given target directory. This will include
274 all the stuff that is normally included in official release tarballs
275 such as lisp.core and the PCL libraries, including Gray streams and
276 simple streams.
277
278 This is intended to be run from make-dist.sh.
279
280 * make-extra-dist.sh target-directory version arch os
281
282 This script creates an extra distribution tarball (both in gzipped and
283 bzipped variants) from the given target directory. This will include
284 all the stuff that is normally included in official extra release
285 tarballs, i.e. the auxiliary libraries such as CLX, CLM, and Hemlock.
286
287 This is intended to be run from make-dist.sh.
288
289
290 * cross-build-world.sh target-directory cross-directory cross-script
291 [build-binary] [build-flags...]
292
293 This is a script that can be used instead of build-world.sh for
294 cross-compiling CMUCL. In addition to the arguments of build-world.sh
295 it takes two further required arguments: The name of a directory that
296 will contain the cross-compiler backend (the directory is created if
297 it doesn't exist, and must not be the same as the target-directory),
298 and the name of a Lisp cross-compilation script, which is responsible
299 for setting up, compiling, and loading the cross-compiler backend.
300 The latter argument is needed because each host/target combination of
301 platform's needs slightly different code to produce a working
302 cross-compiler.
303
304 We include a number of working examples of cross-compiler scripts in
305 the cross-scripts directory. You'll have to edit the features section
306 of the given scripts, to specify the features that should be removed
307 from the current set of features in the host lisp, and those that
308 should be added, so that the backend features are correct for the
309 intended target.
310
311 You can look at Eric Marsden's collection of build scripts for the
312 basis of more cross-compiler scripts.
313
314 Step-by-Step Example of recompiling CMUCL for OpenBSD
315 -----------------------------------------------------
316
317 Set up everything as described in the setup section above. Then
318 execute:
319
320 # Create a new target directory structure/config for OpenBSD:
321 src/tools/create-target.sh openbsd OpenBSD_gencgc OpenBSD
322
323 # edit openbsd/setenv.lisp to contain what we want:
324 cat <<EOF > openbsd/setenv.lisp
325 ;;; Put code to massage *features* list here...
326
327 (in-package :user)
328
329 (pushnew :openbsd *features*)
330 (pushnew :bsd *features*)
331 (pushnew :i486 *features*)
332 (pushnew :mp *features*)
333 (pushnew :hash-new *features*)
334 (pushnew :random-mt19937 *features*)
335 (pushnew :conservative-float-type *features*)
336 (pushnew :gencgc *features*)
337
338 ;;; Version tags
339
340 (pushnew :cmu18d *features*)
341 (pushnew :cmu18 *features*)
342 (setf *features* (remove :cmu17 *features*))
343 (setf *features* (remove :cmu18c *features*))
344 EOF
345
346 # Recompile the lisp world, and dump a new kernel.core:
347 src/tools/build-world.sh openbsd lisp # Or whatever you need to invoke your
348 # current lisp binary+core
349
350 # If build-world tells you (as it will the first time) that:
351 # "The C header file has changed. Be sure to re-compile the startup
352 # code."
353 # You 'll need to start rebuild-lisp.sh to do that, and then reinvoke
354 # build-world.sh:
355
356 # Recompile lisp binary itself:
357 src/tools/rebuild-lisp.sh openbsd
358
359 # Restart build-world.sh now:
360 src/tools/build-world.sh openbsd lisp
361
362 # Now we populate the kernel.core with further compiled files,
363 # and dump the final lisp.core file:
364
365 src/tools/load-world.sh openbsd "18d+ 2002-05-06"
366
367 # The second argument above is the version number that the built
368 # core will announce. Please always put the build-date and some
369 # other information in there, to make it possible to differentiate
370 # those builds from official builds, which only contain the release.
371
372 Now you should have a new lisp.core, which you can start with
373
374 ./openbsd/lisp/lisp -core ./openbsd/lisp/lisp.core -noinit -nositeinit
375
376 Compiling sources that contain disruptive changes
377 -------------------------------------------------
378
379 The above instructions should always work as-is for recompiling CMU CL
380 using matching binaries and source files. They also work quite often
381 when recompiling newer sources. However, every so often, some change
382 to the CMU CL sources necessitates some form of bootstrapping, so that
383 binaries built from earlier sources can compile the sources containing
384 that change. There are two forms of boostrapping that can be
385 required:
386
387 a) Bootfiles
388
389 The maintainers try to make bootfiles available, that allow going
390 from an old release to the next release. These are located in the
391 src/bootfiles/<old-release>/ directory of the CMU CL sources.
392
393 I.e. if you have binaries that match release 18d, then you'll need
394 to use all the bootfiles in src/bootfiles/18d/ in order to go to
395 the next release (or current sources, if no release has been made
396 yet). If you already used some of the bootstrap files to compile
397 your current lisp, you obviously don't need to use those to get to
398 later versions.
399
400 You can use the bootfiles by concatenating them into a file called
401 bootstrap.lisp in the target directory (i.e. target:bootstrap.lisp)
402 in the order they are numbered. Be sure to remove the bootstrap
403 file once it is no longer needed.
404
405
406 b) Cross-compiling
407
408 Under some circumstances, bootstrap code will not be sufficient,
409 and a cross-compilation is needed. In that case you will have to
410 use cross-build-world.sh, instead of build-world.sh. Please read
411 the instructions of that script for details of the more complex
412 procedure.
413
414 << This isn't really true anymore, and we should place a more
415 elaborate description of the cross-compiling process here >>
416
417 When cross-compiling, there are two sorts of bootscripts that can be
418 used: Those that want to be executed prior to compiling and loading
419 the cross-compiler, which should be placed in the file called
420 target:cross-bootstrap.lisp, and those that should happen after the
421 cross-compiler has been compiled and loaded, just prior to compiling
422 the target, which should be placed in target:bootstrap.lisp, just
423 like when doing a normal recompile.
424
425 Additionally, sometimes customized cross-compiler setup scripts
426 (to be used in place of e.g. cross-x86-x86.lisp) are required,
427 which are also placed in one of the bootfiles/*/* files. In those
428 cases follow the instructions provided in that file, possibly merging
429 the changed contents thereof with your normal cross-script.
430
431 Step-by-Step Example of Cross-Compiling
432 ---------------------------------------
433
434 This gives a step-by-step example of cross-compiling a sparc-v8 build
435 using a sparc-v9 build. (For some unknown reason, you can't just
436 remove the :sparc-v9 feature and add :sparc-v8.)
437
438 So, first get a recent sparc-v9 build. It's best to get a version
439 that is up-to-date with the sources. Otherwise, you may also need to
440 add a bootstrap file to get any bootfiles to make your lisp
441 up-to-date with the current sources.
442
443 Create a cross-compiler directory to hold the cross-compiler and a
444 target directory to hold the result:
445
446 src/tools/create-target.sh xcross
447 src/tools/create-target.sh xtarget
448
449 Copy the src/tools/cross-scripts/cross-sparc-sparc.lisp to
450 xtarget/cross.lisp. Edit it appropriately. In this case, it should
451 look something like:
452
453 (c::new-backend "SPARC"
454 ;; Features to add here
455 '(:sparc :sparc-v8
456 :complex-fp-vops
457 :linkage-table
458 :gencgc
459 :stack-checking
460 :relative-package-names
461 :conservative-float-type
462 :hash-new :random-mt19937
463 :cmu :cmu19 :cmu19a
464 )
465 ;; Features to remove from current *features* here
466 '(:sparc-v9 :sparc-v7 :x86 :x86-bootstrap :alpha :osf1 :mips
467 :propagate-fun-type :propagate-float-type :constrain-float-type
468 :openbsd :freebsd :glibc2 :linux :pentium
469 :long-float :new-random :small))
470
471 (setf *features* (remove :sparc-v9 *features*))
472 (pushnew :sparc-v8 *features*)
473
474 It's important to add frob *features* here as well as in the
475 new-backend. If you don't adjust *features*, they won't be set
476 appropriately in the result.
477
478 Now compile the result:
479
480 src/tools/cross-build-world.sh xtarget xcross xtarget/cross.lisp [v9 binary]
481
482 When this finishes, you need to compile the C code:
483
484 src/tools/rebuild-lisp.sh xtarget
485
486 At this point, you may want to run cross-build-world.sh again to
487 generate a new kernel.core. It shouldn't build anything; just loads
488 everything and creates a kernel.core.
489
490 With the new kernel.core, we need to create a lisp.core:
491
492 src/tools/load-world.sh xtarget "new lisp"
493
494 Test the result with
495
496 xtarget/lisp/lisp -noinit
497
498 You may now want to use this cross-compiled lisp to rebuild itself, to
499 make sure you haven't messed up. You can do a normal build as
500 described above, except your build-lisp is xtarget/lisp/lisp.
501

  ViewVC Help
Powered by ViewVC 1.1.5