1 Design document for the unified scheme data
2 ===========================================
4 How are things connected?
5 -------------------------
7 The unified scheme takes all its data from the build.info files seen
8 throughout the source tree. These files hold the minimum information
9 needed to build end product files from diverse sources. See the
10 section on build.info files below.
12 From the information in build.info files, Configure builds up an
13 information database as a hash table called %unified_info, which is
14 stored in configdata.pm, found at the top of the build tree (which may
15 or may not be the same as the source tree).
17 Configurations/common.tmpl uses the data from %unified_info to
18 generate the rules for building end product files as well as
19 intermediary files with the help of a few functions found in the
20 build-file templates. See the section on build-file templates further
21 down for more information.
26 As mentioned earlier, build.info files are meant to hold the minimum
27 information needed to build output files, and therefore only (with a
28 few possible exceptions [1]) have information about end products (such
29 as scripts, library files and programs) and source files (such as C
30 files, C header files, assembler files, etc). Intermediate files such
31 as object files are rarely directly referred to in build.info files (and
32 when they are, it's always with the file name extension .o), they are
33 inferred by Configure. By the same rule of minimalism, end product
34 file name extensions (such as .so, .a, .exe, etc) are never mentioned
35 in build.info. Their file name extensions will be inferred by the
36 build-file templates, adapted for the platform they are meant for (see
37 sections on %unified_info and build-file templates further down).
39 The variables PROGRAMS, LIBS, ENGINES and SCRIPTS are used to declare
40 end products. There are variants for them with '_NO_INST' as suffix
41 (PROGRAM_NO_INST etc) to specify end products that shouldn't get
44 The variables SOURCE, DEPEND and INCLUDE are indexed by a produced
45 file, and their values are the source used to produce that particular
46 produced file, extra dependencies, and include directories needed.
48 All their values in all the build.info throughout the source tree are
49 collected together and form a set of programs, libraries, engines and
50 scripts to be produced, source files, dependencies, etc etc etc.
52 Let's have a pretend example, a very limited contraption of OpenSSL,
53 composed of the program 'apps/openssl', the libraries 'libssl' and
54 'libcrypto', an engine 'engines/ossltest' and their sources and
59 INCLUDE[libcrypto]=include
60 INCLUDE[libssl]=include
61 DEPEND[libssl]=libcrypto
63 This is the top directory build.info file, and it tells us that two
64 libraries are to be built, the include directory 'include/' shall be
65 used throughout when building anything that will end up in each
66 library, and that the library 'libssl' depend on the library
67 'libcrypto' to function properly.
71 SOURCE[openssl]=openssl.c
72 INCLUDE[openssl]=.. ../include
73 DEPEND[openssl]=../libssl
75 This is the build.info file in 'apps/', one may notice that all file
76 paths mentioned are relative to the directory the build.info file is
77 located in. This one tells us that there's a program to be built
78 called 'apps/openssl' (the file name extension will depend on the
79 platform and is therefore not mentioned in the build.info file). It's
80 built from one source file, 'apps/openssl.c', and building it requires
81 the use of '.' and 'include' include directories (both are declared
82 from the point of view of the 'apps/' directory), and that the program
83 depends on the library 'libssl' to function properly.
87 SOURCE[../libcrypto]=aes.c evp.c cversion.c
88 DEPEND[cversion.o]=buildinf.h
90 GENERATE[buildinf.h]=../util/mkbuildinf.pl "$(CC) $(CFLAGS)" "$(PLATFORM)"
91 DEPEND[buildinf.h]=../Makefile
92 DEPEND[../util/mkbuildinf.pl]=../util/Foo.pm
94 This is the build.info file in 'crypto', and it tells us a little more
95 about what's needed to produce 'libcrypto'. LIBS is used again to
96 declare that 'libcrypto' is to be produced. This declaration is
97 really unnecessary as it's already mentioned in the top build.info
98 file, but can make the info file easier to understand. This is to
99 show that duplicate information isn't an issue.
101 This build.info file informs us that 'libcrypto' is built from a few
102 source files, 'crypto/aes.c', 'crypto/evp.c' and 'crypto/cversion.c'.
103 It also shows us that building the object file inferred from
104 'crypto/cversion.c' depends on 'crypto/buildinf.h'. Finally, it
105 also shows the possibility to declare how some files are generated
106 using some script, in this case a perl script, and how such scripts
107 can be declared to depend on other files, in this case a perl module.
109 Two things are worth an extra note:
111 'DEPEND[cversion.o]' mentions an object file. DEPEND indexes is the
112 only location where it's valid to mention them
114 Lines in 'BEGINRAW'..'ENDRAW' sections must always mention files as
115 seen from the top directory, no exception.
119 SOURCE[../libssl]=tls.c
121 This is the build.info file in 'ssl/', and it tells us that the
122 library 'libssl' is built from the source file 'ssl/tls.c'.
126 SOURCE[dasync]=e_dasync.c
127 DEPEND[dasync]=../libcrypto
128 INCLUDE[dasync]=../include
130 ENGINES_NO_INST=ossltest
131 SOURCE[ossltest]=e_ossltest.c
132 DEPEND[ossltest]=../libcrypto.a
133 INCLUDE[ossltest]=../include
135 This is the build.info file in 'engines/', telling us that two engines
136 called 'engines/dasync' and 'engines/ossltest' shall be built, that
137 dasync's source is 'engines/e_dasync.c' and ossltest's source is
138 'engines/e_ossltest.c' and that the include directory 'include/' may
139 be used when building anything that will be part of these engines.
140 Also, both engines depend on the library 'libcrypto' to function
141 properly. ossltest is explicitly linked with the static variant of
142 the library 'libcrypto'. Finally, only dasync is being installed, as
143 ossltest is only for internal testing.
145 When Configure digests these build.info files, the accumulated
146 information comes down to this:
148 LIBS=libcrypto libssl
149 SOURCE[libcrypto]=crypto/aes.c crypto/evp.c crypto/cversion.c
150 DEPEND[crypto/cversion.o]=crypto/buildinf.h
151 INCLUDE[libcrypto]=include
152 SOURCE[libssl]=ssl/tls.c
153 INCLUDE[libssl]=include
154 DEPEND[libssl]=libcrypto
156 PROGRAMS=apps/openssl
157 SOURCE[apps/openssl]=apps/openssl.c
158 INCLUDE[apps/openssl]=. include
159 DEPEND[apps/openssl]=libssl
161 ENGINES=engines/dasync
162 SOURCE[engines/dasync]=engines/e_dasync.c
163 DEPEND[engines/dasync]=libcrypto
164 INCLUDE[engines/dasync]=include
166 ENGINES_NO_INST=engines/ossltest
167 SOURCE[engines/ossltest]=engines/e_ossltest.c
168 DEPEND[engines/ossltest]=libcrypto.a
169 INCLUDE[engines/ossltest]=include
171 GENERATE[crypto/buildinf.h]=util/mkbuildinf.pl "$(CC) $(CFLAGS)" "$(PLATFORM)"
172 DEPEND[crypto/buildinf.h]=Makefile
173 DEPEND[util/mkbuildinf.pl]=util/Foo.pm
176 A few notes worth mentioning:
178 LIBS may be used to declare routine libraries only.
180 PROGRAMS may be used to declare programs only.
182 ENGINES may be used to declare engines only.
184 The indexes for SOURCE must only be end product files, such as
185 libraries, programs or engines. The values of SOURCE variables must
186 only be source files (possibly generated).
188 INCLUDE and DEPEND shows a relationship between different files
189 (usually produced files) or between files and directories, such as a
190 program depending on a library, or between an object file and some
193 When Configure processes the build.info files, it will take it as
194 truth without question, and will therefore perform very few checks.
195 If the build tree is separate from the source tree, it will assume
196 that all built files and up in the build directory and that all source
197 files are to be found in the source tree, if they can be found there.
198 Configure will assume that source files that can't be found in the
199 source tree (such as 'crypto/bildinf.h' in the example above) are
200 generated and will be found in the build tree.
203 The %unified_info database
204 --------------------------
206 The information in all the build.info get digested by Configure and
207 collected into the %unified_info database, divided into the following
210 depends => a hash table containing 'file' => [ 'dependency' ... ]
211 pairs. These are directly inferred from the DEPEND
212 variables in build.info files.
214 engines => a list of engines. These are directly inferred from
215 the ENGINES variable in build.info files.
217 generate => a hash table containing 'file' => [ 'generator' ... ]
218 pairs. These are directly inferred from the GENERATE
219 variables in build.info files.
221 includes => a hash table containing 'file' => [ 'include' ... ]
222 pairs. These are directly inferred from the INCLUDE
223 variables in build.info files.
225 install => a hash table containing 'type' => [ 'file' ... ] pairs.
226 The types are 'programs', 'libraries', 'engines' and
227 'scripts', and the array of files list the files of
228 that type that should be installed.
230 libraries => a list of libraries. These are directly inferred from
231 the LIBS variable in build.info files.
233 programs => a list of programs. These are directly inferred from
234 the PROGRAMS variable in build.info files.
236 rawlines => a list of build-file lines. These are a direct copy of
237 the BEGINRAW..ENDRAW lines in build.info files. Note:
238 only the BEGINRAW..ENDRAW section for the current
239 platform are copied, the rest are ignored.
241 scripts => a list of scripts. There are directly inferred from
242 the SCRIPTS variable in build.info files.
244 sources => a hash table containing 'file' => [ 'sourcefile' ... ]
245 pairs. These are indirectly inferred from the SOURCE
246 variables in build.info files. Object files are
247 mentioned in this hash table, with source files from
248 SOURCE variables, and AS source files for programs and
252 a hash table just like 'sources', but only as source
253 files (object files) for building shared libraries.
255 As an example, here is how the build.info files example from the
256 section above would be digested into a %unified_info table:
258 our %unified_info = (
265 "crypto/buildinf.h" =>
269 "crypto/cversion.o" =>
277 "engines/ossltest" =>
285 "util/mkbuildinf.pl" =>
297 "crypto/buildinf.h" =>
299 "util/mkbuildinf.pl",
312 "engines/ossltest" =>
324 "util/mkbuildinf.pl" =>
371 "crypto/cversion.o" =>
379 "engines/e_dasync.o" =>
381 "engines/e_dasync.c",
385 "engines/e_dasync.o",
387 "engines/e_ossltest.o" =>
389 "engines/e_ossltest.c",
391 "engines/ossltest" =>
393 "engines/e_ossltest.o",
412 As can be seen, everything in %unified_info is fairly simple suggest
413 of information. Still, it tells us that to build all programs, we
414 must build 'apps/openssl', and to build the latter, we will need to
415 build all its sources ('apps/openssl.o' in this case) and all the
416 other things it depends on (such as 'libssl'). All those dependencies
417 need to be built as well, using the same logic, so to build 'libssl',
418 we need to build 'ssl/tls.o' as well as 'libcrypto', and to build the
425 Build-file templates are essentially build-files (such as Makefile on
426 Unix) with perl code fragments mixed in. Those perl code fragment
427 will generate all the configuration dependent data, including all the
428 rules needed to build end product files and intermediary files alike.
429 At a minimum, there must be a perl code fragment that defines a set of
430 functions that are used to generates specific build-file rules, to
431 build static libraries from object files, to build shared libraries
432 from static libraries, to programs from object files and libraries,
435 generatesrc - function that produces build file lines to generate
436 a source file from some input.
438 It's called like this:
440 generatesrc(src => "PATH/TO/tobegenerated",
441 generator => [ "generatingfile", ... ]
442 generator_incs => [ "INCL/PATH", ... ]
443 generator_deps => [ "dep1", ... ]
444 incs => [ "INCL/PATH", ... ],
445 deps => [ "dep1", ... ],
446 intent => one of "libs", "dso", "bin" );
448 'src' has the name of the file to be generated.
449 'generator' is the command or part of command to
450 generate the file, of which the first item is
451 expected to be the file to generate from.
452 generatesrc() is expected to analyse and figure out
453 exactly how to apply that file and how to capture
454 the result. 'generator_incs' and 'generator_deps'
455 are include directories and files that the generator
456 file itself depends on. 'incs' and 'deps' are
457 include directories and files that are used if $(CC)
458 is used as an intermediary step when generating the
459 end product (the file indicated by 'src'). 'intent'
460 indicates what the generated file is going to be
463 src2obj - function that produces build file lines to build an
464 object file from source files and associated data.
466 It's called like this:
468 src2obj(obj => "PATH/TO/objectfile",
469 srcs => [ "PATH/TO/sourcefile", ... ],
470 deps => [ "dep1", ... ],
471 incs => [ "INCL/PATH", ... ]
472 intent => one of "lib", "dso", "bin" );
474 'obj' has the intended object file *without*
475 extension, src2obj() is expected to add that.
476 'srcs' has the list of source files to build the
477 object file, with the first item being the source
478 file that directly corresponds to the object file.
479 'deps' is a list of explicit dependencies. 'incs'
480 is a list of include file directories. Finally,
481 'intent' indicates what this object file is going
484 obj2lib - function that produces build file lines to build a
485 static library file ("libfoo.a" in Unix terms) from
490 obj2lib(lib => "PATH/TO/libfile",
491 objs => [ "PATH/TO/objectfile", ... ]);
493 'lib' has the intended library file name *without*
494 extension, obj2lib is expected to add that. 'objs'
495 has the list of object files (also *without*
496 extension) to build this library.
498 libobj2shlib - function that produces build file lines to build a
499 shareable object library file ("libfoo.so" in Unix
500 terms) from the corresponding static library file
505 libobj2shlib(shlib => "PATH/TO/shlibfile",
506 lib => "PATH/TO/libfile",
507 objs => [ "PATH/TO/objectfile", ... ],
508 deps => [ "PATH/TO/otherlibfile", ... ]);
510 'lib' has the intended library file name *without*
511 extension, libobj2shlib is expected to add that.
512 'shlib' has the corresponding shared library name
513 *without* extension. 'deps' has the list of other
514 libraries (also *without* extension) this library
515 needs to be linked with. 'objs' has the list of
516 object files (also *without* extension) to build
519 This function has a choice; it can use the
520 corresponding static library as input to make the
521 shared library, or the list of object files.
523 obj2dynlib - function that produces build file lines to build a
524 dynamically loadable library file ("libfoo.so" on
525 Unix) from object files.
529 obj2dynlib(lib => "PATH/TO/libfile",
530 objs => [ "PATH/TO/objectfile", ... ],
531 deps => [ "PATH/TO/otherlibfile",
534 This is almost the same as libobj2shlib, but the
535 intent is to build a shareable library that can be
536 loaded in runtime (a "plugin"...). The differences
537 are subtle, one of the most visible ones is that the
538 resulting shareable library is produced from object
541 obj2bin - function that produces build file lines to build an
542 executable file from object files.
546 obj2bin(bin => "PATH/TO/binfile",
547 objs => [ "PATH/TO/objectfile", ... ],
548 deps => [ "PATH/TO/libfile", ... ]);
550 'bin' has the intended executable file name
551 *without* extension, obj2bin is expected to add
552 that. 'objs' has the list of object files (also
553 *without* extension) to build this library. 'deps'
554 has the list of library files (also *without*
555 extension) that the programs needs to be linked
558 in2script - function that produces build file lines to build a
559 script file from some input.
563 in2script(script => "PATH/TO/scriptfile",
564 sources => [ "PATH/TO/infile", ... ]);
566 'script' has the intended script file name.
567 'sources' has the list of source files to build the
568 resulting script from.
570 Along with the build-file templates is the driving engine
571 Configurations/common.tmpl, which looks through all the information in
572 %unified_info and generates all the rulesets to build libraries,
573 programs and all intermediate files, using the rule generating
574 functions defined in the build-file template.
576 As an example with the smaller build.info set we've seen as an
577 example, producing the rules to build 'libcrypto' would result in the
580 # Note: libobj2shlib will only be called if shared libraries are
582 # Note 2: libobj2shlib gets both the name of the static library
583 # and the names of all the object files that go into it. It's up
584 # to the implementation to decide which to use as input.
585 # Note 3: common.tmpl peals off the ".o" extension from all object
586 # files, as the platform at hand may have a different one.
587 libobj2shlib(shlib => "libcrypto",
589 objs => [ "crypto/aes", "crypto/evp", "crypto/cversion" ],
592 obj2lib(lib => "libcrypto"
593 objs => [ "crypto/aes", "crypto/evp", "crypto/cversion" ]);
595 src2obj(obj => "crypto/aes"
596 srcs => [ "crypto/aes.c" ],
598 incs => [ "include" ],
601 src2obj(obj => "crypto/evp"
602 srcs => [ "crypto/evp.c" ],
604 incs => [ "include" ],
607 src2obj(obj => "crypto/cversion"
608 srcs => [ "crypto/cversion.c" ],
609 deps => [ "crypto/buildinf.h" ],
610 incs => [ "include" ],
613 generatesrc(src => "crypto/buildinf.h",
614 generator => [ "util/mkbuildinf.pl", "\"$(CC)",
615 "$(CFLAGS)\"", "\"$(PLATFORM)\"" ],
616 generator_incs => [ "util" ],
617 generator_deps => [ "util/Foo.pm" ],
622 The returned strings from all those calls are then concatenated
623 together and written to the resulting build-file.