1 Design document for the unified scheme data
2 ===========================================
4 How are things connected?
5 -------------------------
7 The unified scheme takes all its data from the build.info files seen
8 throughout the source tree. These files hold the minimum information
9 needed to build end product files from diverse sources. See the
10 section on build.info files below.
12 From the information in build.info files, Configure builds up an
13 information database as a hash table called %unified_info, which is
14 stored in configdata.pm, found at the top of the build tree (which may
15 or may not be the same as the source tree).
17 Configurations/common.tmpl uses the data from %unified_info to
18 generate the rules for building end product files as well as
19 intermediary files with the help of a few functions found in the
20 build-file templates. See the section on build-file templates further
21 down for more information.
26 As mentioned earlier, build.info files are meant to hold the minimum
27 information needed to build output files, and therefore only (with a
28 few possible exceptions [1]) have information about end products (such
29 as scripts, library files and programs) and source files (such as C
30 files, C header files, assembler files, etc). Intermediate files such
31 as object files are rarely directly referred to in build.info files (and
32 when they are, it's always with the file name extension .o), they are
33 inferred by Configure. By the same rule of minimalism, end product
34 file name extensions (such as .so, .a, .exe, etc) are never mentioned
35 in build.info. Their file name extensions will be inferred by the
36 build-file templates, adapted for the platform they are meant for (see
37 sections on %unified_info and build-file templates further down).
39 The variables PROGRAMS, LIBS, ENGINES and SCRIPTS are used to declare
40 end products. There are variants for them with '_NO_INST' as suffix
41 (PROGRAM_NO_INST etc) to specify end products that shouldn't get
44 The variables SOURCE, DEPEND, INCLUDE and ORDINALS are indexed by a
45 produced file, and their values are the source used to produce that
46 particular produced file, extra dependencies, include directories
47 needed, and ordinal files (explained further below.
49 All their values in all the build.info throughout the source tree are
50 collected together and form a set of programs, libraries, engines and
51 scripts to be produced, source files, dependencies, etc etc etc.
53 Let's have a pretend example, a very limited contraption of OpenSSL,
54 composed of the program 'apps/openssl', the libraries 'libssl' and
55 'libcrypto', an engine 'engines/ossltest' and their sources and
60 ORDINALS[libcrypto]=crypto
62 INCLUDE[libcrypto]=include
63 INCLUDE[libssl]=include
64 DEPEND[libssl]=libcrypto
66 This is the top directory build.info file, and it tells us that two
67 libraries are to be built, there are some ordinals to be used to
68 declare what symbols in those libraries are seen as public, the
69 include directory 'include/' shall be used throughout when building
70 anything that will end up in each library, and that the library
71 'libssl' depend on the library 'libcrypto' to function properly.
75 SOURCE[openssl]=openssl.c
76 INCLUDE[openssl]=.. ../include
77 DEPEND[openssl]=../libssl
79 This is the build.info file in 'apps/', one may notice that all file
80 paths mentioned are relative to the directory the build.info file is
81 located in. This one tells us that there's a program to be built
82 called 'apps/openssl' (the file name extension will depend on the
83 platform and is therefore not mentioned in the build.info file). It's
84 built from one source file, 'apps/openssl.c', and building it requires
85 the use of '.' and 'include' include directories (both are declared
86 from the point of view of the 'apps/' directory), and that the program
87 depends on the library 'libssl' to function properly.
91 SOURCE[../libcrypto]=aes.c evp.c cversion.c
92 DEPEND[cversion.o]=buildinf.h
94 GENERATE[buildinf.h]=../util/mkbuildinf.pl "$(CC) $(CFLAGS)" "$(PLATFORM)"
95 DEPEND[buildinf.h]=../Makefile
96 DEPEND[../util/mkbuildinf.pl]=../util/Foo.pm
98 This is the build.info file in 'crypto', and it tells us a little more
99 about what's needed to produce 'libcrypto'. LIBS is used again to
100 declare that 'libcrypto' is to be produced. This declaration is
101 really unnecessary as it's already mentioned in the top build.info
102 file, but can make the info file easier to understand. This is to
103 show that duplicate information isn't an issue.
105 This build.info file informs us that 'libcrypto' is built from a few
106 source files, 'crypto/aes.c', 'crypto/evp.c' and 'crypto/cversion.c'.
107 It also shows us that building the object file inferred from
108 'crypto/cversion.c' depends on 'crypto/buildinf.h'. Finally, it
109 also shows the possibility to declare how some files are generated
110 using some script, in this case a perl script, and how such scripts
111 can be declared to depend on other files, in this case a perl module.
113 Two things are worth an extra note:
115 'DEPEND[cversion.o]' mentions an object file. DEPEND indexes is the
116 only location where it's valid to mention them
118 Lines in 'BEGINRAW'..'ENDRAW' sections must always mention files as
119 seen from the top directory, no exception.
123 SOURCE[../libssl]=tls.c
125 This is the build.info file in 'ssl/', and it tells us that the
126 library 'libssl' is built from the source file 'ssl/tls.c'.
130 SOURCE[dasync]=e_dasync.c
131 DEPEND[dasync]=../libcrypto
132 INCLUDE[dasync]=../include
134 ENGINES_NO_INST=ossltest
135 SOURCE[ossltest]=e_ossltest.c
136 DEPEND[ossltest]=../libcrypto.a
137 INCLUDE[ossltest]=../include
139 This is the build.info file in 'engines/', telling us that two engines
140 called 'engines/dasync' and 'engines/ossltest' shall be built, that
141 dasync's source is 'engines/e_dasync.c' and ossltest's source is
142 'engines/e_ossltest.c' and that the include directory 'include/' may
143 be used when building anything that will be part of these engines.
144 Also, both engines depend on the library 'libcrypto' to function
145 properly. ossltest is explicitly linked with the static variant of
146 the library 'libcrypto'. Finally, only dasync is being installed, as
147 ossltest is only for internal testing.
149 When Configure digests these build.info files, the accumulated
150 information comes down to this:
152 LIBS=libcrypto libssl
153 ORDINALS[libcrypto]=crypto
154 SOURCE[libcrypto]=crypto/aes.c crypto/evp.c crypto/cversion.c
155 DEPEND[crypto/cversion.o]=crypto/buildinf.h
156 INCLUDE[libcrypto]=include
158 SOURCE[libssl]=ssl/tls.c
159 INCLUDE[libssl]=include
160 DEPEND[libssl]=libcrypto
162 PROGRAMS=apps/openssl
163 SOURCE[apps/openssl]=apps/openssl.c
164 INCLUDE[apps/openssl]=. include
165 DEPEND[apps/openssl]=libssl
167 ENGINES=engines/dasync
168 SOURCE[engines/dasync]=engines/e_dasync.c
169 DEPEND[engines/dasync]=libcrypto
170 INCLUDE[engines/dasync]=include
172 ENGINES_NO_INST=engines/ossltest
173 SOURCE[engines/ossltest]=engines/e_ossltest.c
174 DEPEND[engines/ossltest]=libcrypto.a
175 INCLUDE[engines/ossltest]=include
177 GENERATE[crypto/buildinf.h]=util/mkbuildinf.pl "$(CC) $(CFLAGS)" "$(PLATFORM)"
178 DEPEND[crypto/buildinf.h]=Makefile
179 DEPEND[util/mkbuildinf.pl]=util/Foo.pm
182 A few notes worth mentioning:
184 LIBS may be used to declare routine libraries only.
186 PROGRAMS may be used to declare programs only.
188 ENGINES may be used to declare engines only.
190 The indexes for SOURCE and ORDINALS must only be end product files,
191 such as libraries, programs or engines. The values of SOURCE
192 variables must only be source files (possibly generated)
194 INCLUDE and DEPEND shows a relationship between different files
195 (usually produced files) or between files and directories, such as a
196 program depending on a library, or between an object file and some
199 When Configure processes the build.info files, it will take it as
200 truth without question, and will therefore perform very few checks.
201 If the build tree is separate from the source tree, it will assume
202 that all built files and up in the build directory and that all source
203 files are to be found in the source tree, if they can be found there.
204 Configure will assume that source files that can't be found in the
205 source tree (such as 'crypto/bildinf.h' in the example above) are
206 generated and will be found in the build tree.
209 The %unified_info database
210 --------------------------
212 The information in all the build.info get digested by Configure and
213 collected into the %unified_info database, divided into the following
216 depends => a hash table containing 'file' => [ 'dependency' ... ]
217 pairs. These are directly inferred from the DEPEND
218 variables in build.info files.
220 engines => a list of engines. These are directly inferred from
221 the ENGINES variable in build.info files.
223 generate => a hash table containing 'file' => [ 'generator' ... ]
224 pairs. These are directly inferred from the GENERATE
225 variables in build.info files.
227 includes => a hash table containing 'file' => [ 'include' ... ]
228 pairs. These are directly inferred from the INCLUDE
229 variables in build.info files.
231 install => a hash table containing 'type' => [ 'file' ... ] pairs.
232 The types are 'programs', 'libraries', 'engines' and
233 'scripts', and the array of files list the files of
234 that type that should be installed.
236 libraries => a list of libraries. These are directly inferred from
237 the LIBS variable in build.info files.
239 ordinals => a hash table containing 'file' => [ 'word', 'ordfile' ]
240 pairs. 'file' and 'word' are directly inferred from
241 the ORDINALS variables in build.info files, while the
242 file 'ofile' comes from internal knowledge in
245 programs => a list of programs. These are directly inferred from
246 the PROGRAMS variable in build.info files.
248 rawlines => a list of build-file lines. These are a direct copy of
249 the BEGINRAW..ENDRAW lines in build.info files. Note:
250 only the BEGINRAW..ENDRAW section for the current
251 platform are copied, the rest are ignored.
253 scripts => a list of scripts. There are directly inferred from
254 the SCRIPTS variable in build.info files.
256 sources => a hash table containing 'file' => [ 'sourcefile' ... ]
257 pairs. These are indirectly inferred from the SOURCE
258 variables in build.info files. Object files are
259 mentioned in this hash table, with source files from
260 SOURCE variables, and AS source files for programs and
264 a hash table just like 'sources', but only as source
265 files (object files) for building shared libraries.
267 As an example, here is how the build.info files example from the
268 section above would be digested into a %unified_info table:
270 our %unified_info = (
277 "crypto/buildinf.h" =>
281 "crypto/cversion.o" =>
289 "engines/ossltest" =>
297 "util/mkbuildinf.pl" =>
309 "crypto/buildinf.h" =>
311 "util/mkbuildinf.pl",
324 "engines/ossltest" =>
336 "util/mkbuildinf.pl" =>
367 "util/libcrypto.num",
396 "crypto/cversion.o" =>
404 "engines/e_dasync.o" =>
406 "engines/e_dasync.c",
410 "engines/e_dasync.o",
412 "engines/e_ossltest.o" =>
414 "engines/e_ossltest.c",
416 "engines/ossltest" =>
418 "engines/e_ossltest.o",
437 As can be seen, everything in %unified_info is fairly simple suggest
438 of information. Still, it tells us that to build all programs, we
439 must build 'apps/openssl', and to build the latter, we will need to
440 build all its sources ('apps/openssl.o' in this case) and all the
441 other things it depends on (such as 'libssl'). All those dependencies
442 need to be built as well, using the same logic, so to build 'libssl',
443 we need to build 'ssl/tls.o' as well as 'libcrypto', and to build the
450 Build-file templates are essentially build-files (such as Makefile on
451 Unix) with perl code fragments mixed in. Those perl code fragment
452 will generate all the configuration dependent data, including all the
453 rules needed to build end product files and intermediary files alike.
454 At a minimum, there must be a perl code fragment that defines a set of
455 functions that are used to generates specific build-file rules, to
456 build static libraries from object files, to build shared libraries
457 from static libraries, to programs from object files and libraries,
460 generatesrc - function that produces build file lines to generate
461 a source file from some input.
463 It's called like this:
465 generatesrc(src => "PATH/TO/tobegenerated",
466 generator => [ "generatingfile", ... ]
467 generator_incs => [ "INCL/PATH", ... ]
468 generator_deps => [ "dep1", ... ]
469 incs => [ "INCL/PATH", ... ],
470 deps => [ "dep1", ... ],
471 intent => one of "libs", "dso", "bin" );
473 'src' has the name of the file to be generated.
474 'generator' is the command or part of command to
475 generate the file, of which the first item is
476 expected to be the file to generate from.
477 generatesrc() is expected to analyse and figure out
478 exactly how to apply that file and how to capture
479 the result. 'generator_incs' and 'generator_deps'
480 are include directories and files that the generator
481 file itself depends on. 'incs' and 'deps' are
482 include directories and files that are used if $(CC)
483 is used as an intermediary step when generating the
484 end product (the file indicated by 'src'). 'intent'
485 indicates what the generated file is going to be
488 src2obj - function that produces build file lines to build an
489 object file from source files and associated data.
491 It's called like this:
493 src2obj(obj => "PATH/TO/objectfile",
494 srcs => [ "PATH/TO/sourcefile", ... ],
495 deps => [ "dep1", ... ],
496 incs => [ "INCL/PATH", ... ]
497 intent => one of "lib", "dso", "bin" );
499 'obj' has the intended object file *without*
500 extension, src2obj() is expected to add that.
501 'srcs' has the list of source files to build the
502 object file, with the first item being the source
503 file that directly corresponds to the object file.
504 'deps' is a list of explicit dependencies. 'incs'
505 is a list of include file directories. Finally,
506 'intent' indicates what this object file is going
509 obj2lib - function that produces build file lines to build a
510 static library file ("libfoo.a" in Unix terms) from
515 obj2lib(lib => "PATH/TO/libfile",
516 objs => [ "PATH/TO/objectfile", ... ]);
518 'lib' has the intended library file name *without*
519 extension, obj2lib is expected to add that. 'objs'
520 has the list of object files (also *without*
521 extension) to build this library.
523 libobj2shlib - function that produces build file lines to build a
524 shareable object library file ("libfoo.so" in Unix
525 terms) from the corresponding static library file
530 libobj2shlib(shlib => "PATH/TO/shlibfile",
531 lib => "PATH/TO/libfile",
532 objs => [ "PATH/TO/objectfile", ... ],
533 deps => [ "PATH/TO/otherlibfile", ... ],
534 ordinals => [ "word", "/PATH/TO/ordfile" ]);
536 'lib' has the intended library file name *without*
537 extension, libobj2shlib is expected to add that.
538 'shlib' has the corresponding shared library name
539 *without* extension. 'deps' has the list of other
540 libraries (also *without* extension) this library
541 needs to be linked with. 'objs' has the list of
542 object files (also *without* extension) to build
543 this library. 'ordinals' MAY be present, and when
544 it is, its value is an array where the word is
545 "crypto" or "ssl" and the file is one of the ordinal
546 files util/libcrypto.num or util/libssl.num in the
549 This function has a choice; it can use the
550 corresponding static library as input to make the
551 shared library, or the list of object files.
553 obj2dynlib - function that produces build file lines to build a
554 dynamically loadable library file ("libfoo.so" on
555 Unix) from object files.
559 obj2dynlib(lib => "PATH/TO/libfile",
560 objs => [ "PATH/TO/objectfile", ... ],
561 deps => [ "PATH/TO/otherlibfile",
564 This is almost the same as libobj2shlib, but the
565 intent is to build a shareable library that can be
566 loaded in runtime (a "plugin"...). The differences
567 are subtle, one of the most visible ones is that the
568 resulting shareable library is produced from object
571 obj2bin - function that produces build file lines to build an
572 executable file from object files.
576 obj2bin(bin => "PATH/TO/binfile",
577 objs => [ "PATH/TO/objectfile", ... ],
578 deps => [ "PATH/TO/libfile", ... ]);
580 'bin' has the intended executable file name
581 *without* extension, obj2bin is expected to add
582 that. 'objs' has the list of object files (also
583 *without* extension) to build this library. 'deps'
584 has the list of library files (also *without*
585 extension) that the programs needs to be linked
588 in2script - function that produces build file lines to build a
589 script file from some input.
593 in2script(script => "PATH/TO/scriptfile",
594 sources => [ "PATH/TO/infile", ... ]);
596 'script' has the intended script file name.
597 'sources' has the list of source files to build the
598 resulting script from.
600 Along with the build-file templates is the driving engine
601 Configurations/common.tmpl, which looks through all the information in
602 %unified_info and generates all the rulesets to build libraries,
603 programs and all intermediate files, using the rule generating
604 functions defined in the build-file template.
606 As an example with the smaller build.info set we've seen as an
607 example, producing the rules to build 'libcrypto' would result in the
610 # Note: libobj2shlib will only be called if shared libraries are
612 # Note 2: libobj2shlib gets both the name of the static library
613 # and the names of all the object files that go into it. It's up
614 # to the implementation to decide which to use as input.
615 # Note 3: common.tmpl peals off the ".o" extension from all object
616 # files, as the platform at hand may have a different one.
617 libobj2shlib(shlib => "libcrypto",
619 objs => [ "crypto/aes", "crypto/evp", "crypto/cversion" ],
621 ordinals => [ "crypto", "util/libcrypto.num" ]);
623 obj2lib(lib => "libcrypto"
624 objs => [ "crypto/aes", "crypto/evp", "crypto/cversion" ]);
626 src2obj(obj => "crypto/aes"
627 srcs => [ "crypto/aes.c" ],
629 incs => [ "include" ],
632 src2obj(obj => "crypto/evp"
633 srcs => [ "crypto/evp.c" ],
635 incs => [ "include" ],
638 src2obj(obj => "crypto/cversion"
639 srcs => [ "crypto/cversion.c" ],
640 deps => [ "crypto/buildinf.h" ],
641 incs => [ "include" ],
644 generatesrc(src => "crypto/buildinf.h",
645 generator => [ "util/mkbuildinf.pl", "\"$(CC)",
646 "$(CFLAGS)\"", "\"$(PLATFORM)\"" ],
647 generator_incs => [ "util" ],
648 generator_deps => [ "util/Foo.pm" ],
653 The returned strings from all those calls are then concatenated
654 together and written to the resulting build-file.