/[cmucl]/src/BUILDING
ViewVC logotype

Contents of /src/BUILDING

Parent Directory Parent Directory | Revision Log Revision Log


Revision 1.2 - (show annotations)
Tue May 17 16:14:05 2005 UTC (8 years, 11 months ago) by rtoy
Branch: MAIN
Changes since 1.1: +114 -58 lines
Update with some fixes from src/tools/README.  Correct some other
small items.
1 -*- Mode: text -*-
2 Building CMU CL
3 ===============
4
5 This document is intended to give you a general overview of the build
6 process (i.e. what needs to be done, in what order, and what is it
7 generally called). It will also tell you how to set up a suitable
8 build environment, how the individual scripts fit into the general
9 scheme of things, and give you a couple of examples.
10
11 General Requirements
12 --------------------
13
14 In order to build CMU CL, you will need:
15
16 a) A working CMU CL binary. There is no way around this requirement!
17
18 This binary can either be for the platform you want to target, in
19 that case you can either recompile or cross-compile, or for another
20 supported platform, in that case you must cross-compile, obviously.
21
22 b) A supported C compiler for the C runtime code.
23
24 Most of the time, this means GNU gcc, though for some ports it
25 means the vendor-supplied C compiler. The compiler must be
26 available under the name specified by your ports Config file.
27
28 c) GNU make
29
30 This has to be available either as gmake or make in your PATH, or
31 the MAKE environment variable has to be set to point to the correct
32 binary.
33
34 d) The CMU CL source code
35
36 Here you can either use one of the release source tarballs, or
37 check out the source code directly from the public CMUCL CVS
38 repository.
39
40 If you want to build CMU CL's Motif interface/toolkit, you'll need a
41 working version of the Motif libraries, either true-blue OSF/Motif, or
42 OpenMotif, or Lesstif. The code was developed against 1.2 Motif,
43 though recompilation against 2.x Motif probably works as well.
44
45 Setting up a build environment
46 ------------------------------
47
48 1.) Create a base directory and change to it
49
50 mkdir cmucl ; cd cmucl
51
52 2.) Fetch the sources and put them into the base directory
53
54 tar xzf /tmp/cmucl-source.tar.gz
55
56 or, if you want to use the CVS sources directly:
57
58 export CVSROOT=:pserver:anonymous@common-lisp.net:/project/cmucl/cvsroot
59 cvs login (password is `anonymous')
60 cvs co src
61
62 Whatever you do, the sources must be in a directory named src
63 inside the base directory. Since the build tools keep all
64 generated files in separate target directories, the src directory
65 can be read-only (e.g. mounted read-only via NFS, etc.)
66
67 The build tools are all in the src/tools directory.
68
69 That's it, you are now ready to build CMU CL.
70
71
72 A general outline of the build process
73 --------------------------------------
74
75 Building CMU CL can happen in one of two ways: Normal recompilation,
76 and cross-compilation. We'll first look at normal recompilation:
77
78 The recompilation process basically consists of 4 phases/parts:
79
80 a) Compiling the lisp files that make up the standard kernel.
81
82 This happens in your current CMU CL process, using your current
83 CMU CL's normal file compiler. This phase currently consists of 3
84 sub-phases, namely those controlled by src/tools/worldcom.lisp,
85 which compiles all the runtime files, src/tools/comcom.lisp, which
86 compiles the compiler (including your chosen backend), and finally
87 src/tools/pclcom.lisp, which compiles PCL, CMU CL's CLOS
88 implementation. The whole phase is often called "world-compile",
89 or "compiling up a world", based on the name of the first
90 sub-phase.
91
92 b) Building a new kernel.core file out of the so created files
93
94 This process, which is generally called genesis, and which is
95 controlled by src/tools/worldbuild.lisp, uses the newly compiled
96 files in order to build a new, basic core file, which is then used
97 by the last phase to create a fully functional normal core file.
98 It does this by "loading" the compiled files into an in-core
99 representation of a new core file, which is then dumped out to
100 disk, together with lots of fixups that need to happen once the new
101 core is started.
102
103 As part of this process, it also creates the file internals.h,
104 which contains information about the general memory layout of the
105 new core and its basic types, their type tags, and the location of
106 several important constants and other variables, that are needed by
107 the C runtime code to work with the given core.
108
109 So going through genesis is needed to create internals.h, which is
110 needed to compile the C runtime code (i.e. the "lisp" binary).
111 However there is a slight circularity here, since genesis needs as
112 one of its inputs the file target:lisp/lisp.nm, which contains the
113 (slightly pre-treated) output of running nm on the new lisp
114 binary. Genesis uses this information to fixup the addresses of C
115 runtime support functions for calls from Lisp code.
116
117 However the circularity isn't complete, since genesis can work with
118 an empty/bogus lisp.nm file. While the kernel.core it then
119 produces is unusable, it will create a usable internals.h file,
120 which can be used to recompile the C runtime code, producing a
121 usable lisp.nm file, which in turn can be used to restart genesis,
122 producing a working kernel.core file.
123
124 Genesis also checks whether the newly produced internals.h file
125 differs from a pre-existing internals.h file (this might be caused
126 by an empty internals.h file if you are rebuilding for the first
127 time, or by changes in the lisp sources that cause differences in
128 the memory layout of the kernel.core), and informs you of this, so
129 that you can recompile the C runtime code, and restart genesis.
130
131 If it doesn't inform you of this, you can skip directly to the last
132 phase d).
133
134 c) Recompiling the C runtime code, producing the "lisp" binary file
135
136 This step is only needed if you haven't yet got a suitable lisp
137 binary, or if the internals.h file has changed during genesis (of
138 which genesis informs you), or when you made changes to the C
139 sources that you want to take effect.
140
141 Recompiling the C runtime code is controlled by a GNU Makefile, and
142 your target's Config file. It depends on a correct internals.h
143 file as produced by genesis.
144
145 Note that whenever you recompile the runtime code, for whatever
146 reason, you must redo phase b). Note that if you make changes to
147 the C sources and recompile because of this, you can do that before
148 Phase b), so that you don't have to perform that phase twice.
149
150 d) Populating the kernel.core, and dumping a new lisp.core file.
151
152 In this phase, which is controlled by src/tools/worldload.lisp, and
153 hence often called world-load, the kernel.core file is started up
154 using the (possibly new) lisp binary, the remaining files which
155 were compiled in phase a) are loaded into it, and a new lisp.core
156 file is dumped out.
157
158 We're not quite done yet. This produces just a basic lisp.core.
159 To complete the build so that you something similar to what the
160 releases of CMUCL do, there are a few more steps:
161
162 e) Build the utilities like Gray streams, simple streams, CLX, CLM,
163 and Hemlock. Use the src/tools/build-utils.sh script for this, as
164 described below
165
166 f) Create tarfiles using the src/tools/make-dist.sh script, as
167 explained below.
168
169 With these tarfiles, you can install them anywhere. The contents of
170 the tarfiles will be the same as the snapshots and releases of CMUCL.
171
172 When cross-compiling, there is additional phase at the beginning, and
173 some of the phases happen with different hosts/platforms. The initial
174 phase is setting up and compiling the cross-compilation backend, using
175 your current compiler. The new backend is then loaded, and all
176 compilation in phase a) happens using this compiler backend. The
177 creation of the kernel.core file in phase b) happens as usual, while
178 phase c) of course happens on the target platform (if that differs
179 from the host platform), as does the final phase d). Another major
180 difference is that you can't compile PCL using the cross-compiler, so
181 one usually does a normal rebuild using the cross-compiled core on the
182 target platform to get a full CMU CL core.
183
184 So, now you know all about CMU CL compilation, how does that map onto
185 the scripts included with this little text?
186
187 Overview of the included build scripts
188 --------------------------------------
189
190 * src/tools/create-target.sh target-directory [lisp-variant [motif-variant]]
191
192 This script creates a new target directory, which is a shadow of the
193 source directory, that will contain all the files that are created by
194 the build process. Thus, each target's files are completely separate
195 from the src directory, which could, in fact, be read-only. Hence you
196 can simultaneously build CMUCL for different targets from the same
197 source directory.
198
199 The first argument is the name of the target directory to create. The
200 remaining arguments are optional. If they are not given, the script
201 tries to determine the lisp variant and motif variant from the system
202 the script is running on.
203
204 The lisp-variant (i.e. the suffix of the src/lisp/Config.* to use as
205 the target's Config file), and optionally the motif-variant (again the
206 suffix of the src/motif/server/Config.* file to use as the Config file
207 for the target's CMUCL/Motif server code). If the lisp-variant is
208 given but the motif-variant is not, the motif-variant is determined
209 from the lisp-variant.
210
211 The script will generate the target directory tree, link the relevant
212 Config files, and generate place-holder files for various files, in
213 order to ensure proper operation of the other build-scripts. It also
214 creates a sample setenv.lisp file in the target directory, which is
215 used by the build and load processes to set up the correct list of
216 *features* for your target lisp core.
217
218 IMPORTANT: You will normally NOT have to modify the sample setenv.lisp
219 file, if you are building from a binary that has the desired features.
220 In fact, the sample has all code commented out, If you want to add or
221 remove features, you need to include code that puts at least a minimal
222 set of features onto the list (use PUSHNEW and/or REMOVE). You can
223 use the current set of *features* of your lisp as a first guide. The
224 sample setenv.lisp includes a set of features that should work for the
225 intended configuration. Note also that some adding or removing some
226 features may require a cross-compile instead of a normal compile.
227
228 * src/tools/clean-target.sh [-l] target-directory [more dirs]
229
230 Cleans the given target directory, so that all created files will be
231 removed. This is useful to force recompilation. If the -l flag is
232 given, then the C runtime is also removed, including all the lisp
233 executable, any lisp cores, all object files, lisp.nm, internals.h,
234 and the config file.
235
236 * src/tools/build-world.sh target-directory [build-binary] [build-flags...]
237
238 Starts a complete world build for the given target, using the lisp
239 binary/core specified as a build host. The recompilation step will
240 only recompile changed files, or files for which the fasl files are
241 missing. It will also not recompile the C runtime code (the lisp
242 binary). If a (re)compilation of that code is needed, the genesis
243 step of the world build will inform you of that fact. In that case,
244 you'll have to use the rebuild-lisp.sh script, and then restart the
245 world build process with build-world.sh
246
247 * src/tools/rebuild-lisp.sh target-directory
248
249 This script will force a complete recompilation of the C runtime code
250 of CMU CL (aka the lisp executable). Doing this will necessitate
251 building a new kernel.core file, using build-world.sh.
252
253 * src/tools/load-world.sh target-directory version
254
255 This will finish the CMU CL rebuilding process, by loading the
256 remaining compiled files generated in the world build process into the
257 kernel.core file, that also resulted from that process, creating the
258 final lisp.core file.
259
260 You have to pass the version string as a second argument. The dumped
261 core will anounce itself using that string. Please don't use a string
262 consisting of an official release name only, (e.g. "18d"), since those
263 are reserved for official release builds. Including the build-date in
264 ISO8601 format is often a good idea, e.g. "18d+ 2002-05-06" for a
265 binary that is based on sources current on the 6th May, 2002, which is
266 post the 18d release.
267
268 * src/tools/build-utils.sh target-directory
269
270 This script will build auxiliary libraries packaged with CMU CL,
271 including CLX, CMUCL/Motif, the Motif debugger, inspector, and control
272 panel, and the Hemlock editor. It will use the lisp executable and
273 core of the given target.
274
275 * src/tools/make-dist.sh [-bg] [-G group] [-O owner] target-directory version arch os
276
277 This script creates both main and extra distribution tarballs from the
278 given target directory, using the make-main-dist.sh and
279 make-extra-dist.sh scripts. The result will be two tar files. One
280 contains the main distribution including the runtime and lisp.core
281 with PCL (CLOS); the second contains the extra libraries such as
282 Gray-streams, simple-streams, CLX, CLM, and Hemlock.
283
284 Some options that are available:
285
286 -b Use bzip2 compression
287 -g Use gzip compression
288 -G group Group to use
289 -O owner Owner to use
290
291 If you specify both -b and -g, you will get two sets of tarfiles. The
292 -G and -O options will attempt to set the owner and group of the files
293 when building the tarfiles. This way, when you extract the tarfiles,
294 the owner and group will be set as specified. You may need to be root
295 to do this because many Unix systems don't normally let you change the
296 owner and group of a file.
297
298 The remaining arguments used to create the name of the tarfiles. The
299 names will have the form:
300
301 cmucl-<version>-<arch>-<os>.tar.bz2
302 cmucl-<version>-<arch>-<os>.extras.tar.bz2
303
304 Of course, the "bz2" will be "gz" if you specified gzip compression
305 instead of bzip.
306
307 * /src/tools/make-main-dist.sh target-directory version arch os
308
309 This is script is not normally invoked by the user; make-dist will do
310 it appropriately.
311
312 This script creates a main distribution tarball (both in gzipped and
313 bzipped variants) from the given target directory. This will include
314 all the stuff that is normally included in official release tarballs
315 such as lisp.core and the PCL libraries, including Gray streams and
316 simple streams.
317
318 This is intended to be run from make-dist.sh.
319
320 * src/tools/make-extra-dist.sh target-directory version arch os
321
322 This is script is not normally invoked by the user; make-dist will do
323 it appropriately.
324
325 This script creates an extra distribution tarball (both in gzipped and
326 bzipped variants) from the given target directory. This will include
327 all the stuff that is normally included in official extra release
328 tarballs, i.e. the auxiliary libraries such as CLX, CLM, and Hemlock.
329
330 This is intended to be run from make-dist.sh.
331
332
333 * cross-build-world.sh target-directory cross-directory cross-script
334 [build-binary] [build-flags...]
335
336 This is a script that can be used instead of build-world.sh for
337 cross-compiling CMUCL. In addition to the arguments of build-world.sh
338 it takes two further required arguments: The name of a directory that
339 will contain the cross-compiler backend (the directory is created if
340 it doesn't exist, and must not be the same as the target-directory),
341 and the name of a Lisp cross-compilation script, which is responsible
342 for setting up, compiling, and loading the cross-compiler backend.
343 The latter argument is needed because each host/target combination of
344 platform's needs slightly different code to produce a working
345 cross-compiler.
346
347 We include a number of working examples of cross-compiler scripts in
348 the cross-scripts directory. You'll have to edit the features section
349 of the given scripts, to specify the features that should be removed
350 from the current set of features in the host lisp, and those that
351 should be added, so that the backend features are correct for the
352 intended target.
353
354 You can look at Eric Marsden's collection of build scripts for the
355 basis of more cross-compiler scripts.
356
357 Step-by-Step Example of recompiling CMUCL for OpenBSD
358 -----------------------------------------------------
359
360 Set up everything as described in the setup section above. Then
361 execute:
362
363 # Create a new target directory structure/config for OpenBSD:
364 src/tools/create-target.sh openbsd OpenBSD_gencgc OpenBSD
365
366 # edit openbsd/setenv.lisp to contain what we want:
367 cat <<EOF > openbsd/setenv.lisp
368 ;;; Put code to massage *features* list here...
369
370 (in-package :user)
371
372 (pushnew :openbsd *features*)
373 (pushnew :bsd *features*)
374 (pushnew :i486 *features*)
375 (pushnew :mp *features*)
376 (pushnew :hash-new *features*)
377 (pushnew :random-mt19937 *features*)
378 (pushnew :conservative-float-type *features*)
379 (pushnew :gencgc *features*)
380
381 ;;; Version tags
382
383 (pushnew :cmu18d *features*)
384 (pushnew :cmu18 *features*)
385 (setf *features* (remove :cmu17 *features*))
386 (setf *features* (remove :cmu18c *features*))
387 EOF
388
389 # Recompile the lisp world, and dump a new kernel.core:
390 src/tools/build-world.sh openbsd lisp # Or whatever you need to invoke your
391 # current lisp binary+core
392
393 # If build-world tells you (as it will the first time) that:
394 # "The C header file has changed. Be sure to re-compile the startup
395 # code."
396 # You 'll need to start rebuild-lisp.sh to do that, and then reinvoke
397 # build-world.sh:
398
399 # Recompile lisp binary itself:
400 src/tools/rebuild-lisp.sh openbsd
401
402 # Restart build-world.sh now:
403 src/tools/build-world.sh openbsd lisp
404
405 # Now we populate the kernel.core with further compiled files,
406 # and dump the final lisp.core file:
407
408 src/tools/load-world.sh openbsd "18d+ 2002-05-06"
409
410 # The second argument above is the version number that the built
411 # core will announce. Please always put the build-date and some
412 # other information in there, to make it possible to differentiate
413 # those builds from official builds, which only contain the release.
414
415 Now you should have a new lisp.core, which you can start with
416
417 ./openbsd/lisp/lisp -core ./openbsd/lisp/lisp.core -noinit -nositeinit
418
419 Compiling sources that contain disruptive changes
420 -------------------------------------------------
421
422 The above instructions should always work as-is for recompiling CMU CL
423 using matching binaries and source files. They also work quite often
424 when recompiling newer sources. However, every so often, some change
425 to the CMU CL sources necessitates some form of bootstrapping, so that
426 binaries built from earlier sources can compile the sources containing
427 that change. There are two forms of boostrapping that can be
428 required:
429
430 a) Bootfiles
431
432 The maintainers try to make bootfiles available, that allow going
433 from an old release to the next release. These are located in the
434 src/bootfiles/<old-release>/ directory of the CMU CL sources.
435
436 I.e. if you have binaries that match release 18d, then you'll need
437 to use all the bootfiles in src/bootfiles/18d/ in order to go to
438 the next release (or current sources, if no release has been made
439 yet). If you already used some of the bootstrap files to compile
440 your current lisp, you obviously don't need to use those to get to
441 later versions.
442
443 You can use the bootfiles by concatenating them into a file called
444 bootstrap.lisp in the target directory (i.e. target:bootstrap.lisp)
445 in the order they are numbered. Be sure to remove the bootstrap
446 file once it is no longer needed.
447
448 Alternatively, the bootstrap file can just "load" the individual
449 bootfiles as needed.
450
451 b) Cross-compiling
452
453 Under some circumstances, bootstrap code will not be sufficient,
454 and a cross-compilation is needed. In that case you will have to
455 use cross-build-world.sh, instead of build-world.sh. Please read
456 the instructions of that script for details of the more complex
457 procedure.
458
459 << This isn't really true anymore, and we should place a more
460 elaborate description of the cross-compiling process here >>
461
462 When cross-compiling, there are two sorts of bootscripts that can be
463 used: Those that want to be executed prior to compiling and loading
464 the cross-compiler, which should be placed in the file called
465 target:cross-bootstrap.lisp, and those that should happen after the
466 cross-compiler has been compiled and loaded, just prior to compiling
467 the target, which should be placed in target:bootstrap.lisp, just
468 like when doing a normal recompile.
469
470 Additionally, sometimes customized cross-compiler setup scripts
471 (to be used in place of e.g. cross-x86-x86.lisp) are required,
472 which are also placed in one of the bootfiles/*/* files. In those
473 cases follow the instructions provided in that file, possibly merging
474 the changed contents thereof with your normal cross-script.
475
476 Step-by-Step Example of Cross-Compiling
477 ---------------------------------------
478
479 This gives a step-by-step example of cross-compiling a sparc-v8 build
480 using a sparc-v9 build. (For some unknown reason, you can't just
481 remove the :sparc-v9 feature and add :sparc-v8.)
482
483 So, first get a recent sparc-v9 build. It's best to get a version
484 that is up-to-date with the sources. Otherwise, you may also need to
485 add a bootstrap file to get any bootfiles to make your lisp
486 up-to-date with the current sources.
487
488 1. Select a directory for the cross-compiler and compiled target:
489
490 Create a cross-compiler directory to hold the cross-compiler
491 and a target directory to hold the result:
492
493 src/tools/create-target.sh xcross
494 src/tools/create-target.sh xtarget
495
496 2. Adjust cross-compilation script
497
498 Copy the src/tools/cross-scripts/cross-sparc-sparc.lisp to
499 xtarget/cross.lisp. Edit it appropriately. In this case, it
500 should look something like:
501
502 (c::new-backend "SPARC"
503 ;; Features to add here
504 '(:sparc :sparc-v8
505 :complex-fp-vops
506 :linkage-table
507 :gencgc
508 :stack-checking
509 :relative-package-names
510 :conservative-float-type
511 :hash-new :random-mt19937
512 :cmu :cmu19 :cmu19a
513 )
514 ;; Features to remove from current *features* here
515 '(:sparc-v9 :sparc-v7 :x86 :x86-bootstrap :alpha :osf1 :mips
516 :propagate-fun-type :propagate-float-type :constrain-float-type
517 :openbsd :freebsd :glibc2 :linux :pentium
518 :long-float :new-random :small))
519
520 (setf *features* (remove :sparc-v9 *features*))
521 (pushnew :sparc-v8 *features*)
522
523 It's important to add frob *features* here as well as in the
524 new-backend. If you don't adjust *features*, they won't be
525 set appropriately in the result.
526
527 3. Build the cross compiler and target
528 Now compile the result:
529
530 src/tools/cross-build-world.sh xtarget xcross xtarget/cross.lisp [v9 binary]
531
532 4. Rebuild the lisp files:
533
534 When this finishes, you need to compile the C code:
535
536 src/tools/rebuild-lisp.sh xtarget
537
538 At this point, you may want to run cross-build-world.sh again
539 to generate a new kernel.core. It shouldn't build anything;
540 just loads everything and creates a kernel.core.
541
542 5. Build the world:
543
544 With the new kernel.core, we need to create a lisp.core:
545
546 src/tools/load-world.sh xtarget "new lisp"
547
548 Test the result with
549
550 xtarget/lisp/lisp -noinit
551
552 However, this lisp will be missing some functionality like PCL. You
553 probably now want to use the compiler to rebuild everything once
554 again. Just follow the directions for a normal build, and use
555 xtarget/lisp/lisp as your compiler. Be sure to use create-target.sh
556 to create a new directory where the result can go.
557

  ViewVC Help
Powered by ViewVC 1.1.5