5 This tool aims to test U-Boot by executing U-Boot shell commands using the
6 console interface. A single top-level script exists to execute or attach to the
7 U-Boot console, run the entire script of tests against it, and summarize the
8 results. Advantages of this approach are:
10 - Testing is performed in the same way a user or script would interact with
11 U-Boot; there can be no disconnect.
12 - There is no need to write or embed test-related code into U-Boot itself.
13 It is asserted that writing test-related code in Python is simpler and more
14 flexible than writing it all in C.
15 - It is reasonably simple to interact with U-Boot in this way.
19 The test suite is implemented using pytest. Interaction with the U-Boot console
20 involves executing some binary and interacting with its stdin/stdout. You will
21 need to implement various "hook" scripts that are called by the test suite at
24 In order to run the testsuite at a minimum we require that both python3 and
25 pip for python3 be installed. All of the required python modules are
26 described in the requirements.txt file in this directory and can be installed
27 with the command ```pip install -r requirements.txt```
29 In order to execute certain tests on their supported platforms other tools
30 will be required. The following is an incomplete list:
38 | sudo OR guestmount |
50 Please use the apporirate commands for your distribution to match these tools
51 up with the package that provides them.
53 The test script supports either:
55 - Executing a sandbox port of U-Boot on the local machine as a sub-process,
56 and interacting with it over stdin/stdout.
57 - Executing an external "hook" scripts to flash a U-Boot binary onto a
58 physical board, attach to the board's console stream, and reset the board.
59 Further details are described later.
61 ### Using `virtualenv` to provide requirements
63 The recommended way to run the test suite, in order to ensure reproducibility
64 is to use `virtualenv` to set up the necessary environment. This can be done
65 via the following commands:
69 $ sudo apt-get install python3 python3-virtualenv
70 $ virtualenv -p /usr/bin/python3 venv
71 $ . ./venv/bin/activate
72 $ pip install -r test/py/requirements.txt
77 To run the testsuite on the sandbox port (U-Boot built as a native user-space
78 application), simply execute:
81 ./test/py/test.py --bd sandbox --build
84 The `--bd` option tells the test suite which board type is being tested. This
85 lets the test suite know which features the board has, and hence exactly what
88 The `--build` option tells U-Boot to compile U-Boot. Alternatively, you may
89 omit this option and build U-Boot yourself, in whatever way you choose, before
90 running the test script.
92 The test script will attach to U-Boot, execute all valid tests for the board,
93 then print a summary of the test process. A complete log of the test session
94 will be written to `${build_dir}/test-log.html`. This is best viewed in a web
95 browser, but may be read directly as plain text, perhaps with the aid of the
98 ### Testing under a debugger
100 If you need to run sandbox under a debugger, you may pass the command-line
101 option `--gdbserver COMM`. This causes two things to happens:
103 - Instead of running U-Boot directly, it will be run under gdbserver, with
104 debug communication via the channel `COMM`. You can attach a debugger to the
105 sandbox process in order to debug it. See `man gdbserver` and the example
106 below for details of valid values for `COMM`.
107 - All timeouts in tests are disabled, allowing U-Boot an arbitrary amount of
108 time to execute commands. This is useful if U-Boot is stopped at a breakpoint
115 ./test/py/test.py --bd sandbox --gdbserver localhost:1234
120 gdb ./build-sandbox/u-boot -ex 'target remote localhost:1234'
123 Alternatively, you could leave off the `-ex` option and type the command
124 manually into gdb once it starts.
126 You can use any debugger you wish, so long as it speaks the gdb remote
127 protocol, or any graphical wrapper around gdb.
129 Some tests deliberately cause the sandbox process to exit, e.g. to test the
130 reset command, or sandbox's CTRL-C handling. When this happens, you will need
131 to attach the debugger to the new sandbox instance. If these tests are not
132 relevant to your debugging session, you can skip them using pytest's -k
133 command-line option; see the next section.
135 ## Command-line options
137 - `--board-type`, `--bd`, `-B` set the type of the board to be tested. For
138 example, `sandbox` or `seaboard`.
139 - `--board-identity`, `--id` set the identity of the board to be tested.
140 This allows differentiation between multiple instances of the same type of
141 physical board that are attached to the same host machine. This parameter is
142 not interpreted by the test script in any way, but rather is simply passed
143 to the hook scripts described below, and may be used in any site-specific
144 way deemed necessary.
145 - `--build` indicates that the test script should compile U-Boot itself
146 before running the tests. If using this option, make sure that any
147 environment variables required by the build process are already set, such as
149 - `--buildman` indicates that `--build` should use buildman to build U-Boot.
150 There is no need to set $CROSS_COMPILE` in this case since buildman handles
152 - `--build-dir` sets the directory containing the compiled U-Boot binaries.
153 If omitted, this is `${source_dir}/build-${board_type}`.
154 - `--result-dir` sets the directory to write results, such as log files,
155 into. If omitted, the build directory is used.
156 - `--persistent-data-dir` sets the directory used to store persistent test
157 data. This is test data that may be re-used across test runs, such as file-
160 `pytest` also implements a number of its own command-line options. Commonly used
161 options are mentioned below. Please see `pytest` documentation for complete
162 details. Execute `py.test --version` for a brief summary. Note that U-Boot's
163 test.py script passes all command-line arguments directly to `pytest` for
166 - `-k` selects which tests to run. The default is to run all known tests. This
167 option takes a single argument which is used to filter test names. Simple
168 logical operators are supported. For example:
169 - `'ums'` runs only tests with "ums" in their name.
170 - `'ut_dm'` runs only tests with "ut_dm" in their name. Note that in this
171 case, "ut_dm" is a parameter to a test rather than the test name. The full
172 test name is e.g. "test_ut[ut_dm_leak]".
173 - `'not reset'` runs everything except tests with "reset" in their name.
174 - `'ut or hush'` runs only tests with "ut" or "hush" in their name.
175 - `'not (ut or hush)'` runs everything except tests with "ut" or "hush" in
177 - `-s` prevents pytest from hiding a test's stdout. This allows you to see
178 U-Boot's console log in real time on pytest's stdout.
180 ## Testing real hardware
182 The tools and techniques used to interact with real hardware will vary
183 radically between different host and target systems, and the whims of the user.
184 For this reason, the test suite does not attempt to directly interact with real
185 hardware in any way. Rather, it executes a standardized set of "hook" scripts
186 via `$PATH`. These scripts implement certain actions on behalf of the test
187 suite. This keeps the test suite simple and isolated from system variances
188 unrelated to U-Boot features.
192 #### Environment variables
194 The following environment variables are set when running hook scripts:
196 - `UBOOT_BOARD_TYPE` the board type being tested.
197 - `UBOOT_BOARD_IDENTITY` the board identity being tested, or `na` if none was
199 - `UBOOT_SOURCE_DIR` the U-Boot source directory.
200 - `UBOOT_TEST_PY_DIR` the full path to `test/py/` in the source directory.
201 - `UBOOT_BUILD_DIR` the U-Boot build directory.
202 - `UBOOT_RESULT_DIR` the test result directory.
203 - `UBOOT_PERSISTENT_DATA_DIR` the test persistent data directory.
205 #### `u-boot-test-console`
207 This script provides access to the U-Boot console. The script's stdin/stdout
208 should be connected to the board's console. This process should continue to run
209 indefinitely, until killed. The test suite will run this script in parallel
210 with all other hooks.
212 This script may be implemented e.g. by exec()ing `cu`, `kermit`, `conmux`, etc.
214 If you are able to run U-Boot under a hardware simulator such as qemu, then
215 you would likely spawn that simulator from this script. However, note that
216 `u-boot-test-reset` may be called multiple times per test script run, and must
217 cause U-Boot to start execution from scratch each time. Hopefully your
218 simulator includes a virtual reset button! If not, you can launch the
219 simulator from `u-boot-test-reset` instead, while arranging for this console
220 process to always communicate with the current simulator instance.
222 #### `u-boot-test-flash`
224 Prior to running the test suite against a board, some arrangement must be made
225 so that the board executes the particular U-Boot binary to be tested. Often,
226 this involves writing the U-Boot binary to the board's flash ROM. The test
227 suite calls this hook script for that purpose.
229 This script should perform the entire flashing process synchronously; the
230 script should only exit once flashing is complete, and a board reset will
231 cause the newly flashed U-Boot binary to be executed.
233 It is conceivable that this script will do nothing. This might be useful in
236 - Some other process has already written the desired U-Boot binary into the
237 board's flash prior to running the test suite.
238 - The board allows U-Boot to be downloaded directly into RAM, and executed
239 from there. Use of this feature will reduce wear on the board's flash, so
240 may be preferable if available, and if cold boot testing of U-Boot is not
241 required. If this feature is used, the `u-boot-test-reset` script should
242 perform this download, since the board could conceivably be reset multiple
243 times in a single test run.
245 It is up to the user to determine if those situations exist, and to code this
246 hook script appropriately.
248 This script will typically be implemented by calling out to some SoC- or
249 board-specific vendor flashing utility.
251 #### `u-boot-test-reset`
253 Whenever the test suite needs to reset the target board, this script is
254 executed. This is guaranteed to happen at least once, prior to executing the
255 first test function. If any test fails, the test infra-structure will execute
256 this script again to restore U-Boot to an operational state before running the
259 This script will likely be implemented by communicating with some form of
260 relay or electronic switch attached to the board's reset signal.
262 The semantics of this script require that when it is executed, U-Boot will
263 start running from scratch. If the U-Boot binary to be tested has been written
264 to flash, pulsing the board's reset signal is likely all this script need do.
265 However, in some scenarios, this script may perform other actions. For
266 example, it may call out to some SoC- or board-specific vendor utility in order
267 to download the U-Boot binary directly into RAM and execute it. This would
268 avoid the need for `u-boot-test-flash` to actually write U-Boot to flash, thus
269 saving wear on the flash chip(s).
273 https://github.com/swarren/uboot-test-hooks contains some working example hook
274 scripts, and may be useful as a reference when implementing hook scripts for
275 your platform. These scripts are not considered part of U-Boot itself.
277 ### Board-type-specific configuration
279 Each board has a different configuration and behaviour. Many of these
280 differences can be automatically detected by parsing the `.config` file in the
281 build directory. However, some differences can't yet be handled automatically.
283 For each board, an optional Python module `u_boot_board_${board_type}` may exist
284 to provide board-specific information to the test script. Any global value
285 defined in these modules is available for use by any test function. The data
286 contained in these scripts must be purely derived from U-Boot source code.
287 Hence, these configuration files are part of the U-Boot source tree too.
289 ### Execution environment configuration
291 Each user's hardware setup may enable testing different subsets of the features
292 implemented by a particular board's configuration of U-Boot. For example, a
293 U-Boot configuration may support USB device mode and USB Mass Storage, but this
294 can only be tested if a USB cable is connected between the board and the host
295 machine running the test script.
297 For each board, optional Python modules `u_boot_boardenv_${board_type}` and
298 `u_boot_boardenv_${board_type}_${board_identity}` may exist to provide
299 board-specific and board-identity-specific information to the test script. Any
300 global value defined in these modules is available for use by any test
301 function. The data contained in these is specific to a particular user's
302 hardware configuration. Hence, these configuration files are not part of the
303 U-Boot source tree, and should be installed outside of the source tree. Users
304 should set `$PYTHONPATH` prior to running the test script to allow these
305 modules to be loaded.
307 ### Board module parameter usage
309 The test scripts rely on the following variables being defined by the board
314 ### U-Boot `.config` feature usage
316 The test scripts rely on various U-Boot `.config` features, either directly in
317 order to test those features, or indirectly in order to query information from
318 the running U-Boot instance in order to test other features.
320 One example is that testing of the `md` command requires knowledge of a RAM
321 address to use for the test. This data is parsed from the output of the
322 `bdinfo` command, and hence relies on CONFIG_CMD_BDI being enabled.
324 For a complete list of dependencies, please search the test scripts for
327 - `buildconfig.get(...`
328 - `@pytest.mark.buildconfigspec(...`
329 - `@pytest.mark.notbuildconfigspec(...`
331 ### Complete invocation example
333 Assuming that you have installed the hook scripts into $HOME/ubtest/bin, and
334 any required environment configuration Python modules into $HOME/ubtest/py,
335 then you would likely invoke the test script as follows:
337 If U-Boot has already been built:
340 PATH=$HOME/ubtest/bin:$PATH \
341 PYTHONPATH=${HOME}/ubtest/py/${HOSTNAME}:${PYTHONPATH} \
342 ./test/py/test.py --bd seaboard
345 If you want the test script to compile U-Boot for you too, then you likely
346 need to set `$CROSS_COMPILE` to allow this, and invoke the test script as
350 CROSS_COMPILE=arm-none-eabi- \
351 PATH=$HOME/ubtest/bin:$PATH \
352 PYTHONPATH=${HOME}/ubtest/py/${HOSTNAME}:${PYTHONPATH} \
353 ./test/py/test.py --bd seaboard --build
356 or, using buildman to handle it:
359 PATH=$HOME/ubtest/bin:$PATH \
360 PYTHONPATH=${HOME}/ubtest/py/${HOSTNAME}:${PYTHONPATH} \
361 ./test/py/test.py --bd seaboard --build --buildman
366 Please refer to the pytest documentation for details of writing pytest tests.
367 Details specific to the U-Boot test suite are described below.
369 A test fixture named `u_boot_console` should be used by each test function. This
370 provides the means to interact with the U-Boot console, and retrieve board and
371 environment configuration information.
373 The function `u_boot_console.run_command()` executes a shell command on the
374 U-Boot console, and returns all output from that command. This allows
375 validation or interpretation of the command output. This function validates
376 that certain strings are not seen on the U-Boot console. These include shell
377 error messages and the U-Boot sign-on message (in order to detect unexpected
378 board resets). See the source of `u_boot_console_base.py` for a complete list of
379 "bad" strings. Some test scenarios are expected to trigger these strings. Use
380 `u_boot_console.disable_check()` to temporarily disable checking for specific
381 strings. See `test_unknown_cmd.py` for an example.
383 Board- and board-environment configuration values may be accessed as sub-fields
384 of the `u_boot_console.config` object, for example
385 `u_boot_console.config.ram_base`.
387 Build configuration values (from `.config`) may be accessed via the dictionary
388 `u_boot_console.config.buildconfig`, with keys equal to the Kconfig variable