blob: a234e8867d59af07610f5fcedb8ed2c64a32aa6c [file] [log] [blame]
Jianliang Shen5f7b1b72022-05-03 19:01:27 +08001#################################
2Adding TF-M Regression Test Suite
3#################################
4
5.. contents:: Table of Contents
6
7*************************************
8Introduction of TF-M regression tests
9*************************************
10
11TF-M regression tests test whether changes to TF-M code work as expected.
12A new regression test can consist of following 3 components:
13
141. ``test suite``: A series of tests of a certain function.
152. ``test case``: A specific test instance in test suites.
163. ``test service or partition``: Optional secure services or partitions to
17 support related test suites.
18
19Source structure
20================
21
22+---------------------------------------+---------------------------------------------------------------+
23| Folder name | Description |
24+=======================================+===============================================================+
25| test/bl1 | TF-M bl1 test suites source code. |
26+---------------------------------------+---------------------------------------------------------------+
27| test/bl2 | TF-M bL2 test suites source code. |
28+---------------------------------------+---------------------------------------------------------------+
29| test/config | The CMAKE test configurations files. |
30+---------------------------------------+---------------------------------------------------------------+
31| test/framework | Source code for test framework code, managing test suites. |
32+---------------------------------------+---------------------------------------------------------------+
33| test/secure_fw/suites | Test suites divided into subdirectories. |
34+---------------------------------------+---------------------------------------------------------------+
35| test/secure_fw/suites/<suite>/service | Test service divided into corresponding suite subdirectories. |
36+---------------------------------------+---------------------------------------------------------------+
37| test/secure_fw/common_test_services | Common test services. |
38+---------------------------------------+---------------------------------------------------------------+
39
40***********************
41Adding a new test suite
42***********************
43
44This section introduces how to add a new test suite and control its compilation
45with a test configuration in ``tf-m-tests`` repository.
46
47Source code
48===========
49
50The test suite example subdirectory named ``<test_name>`` is located under the
51path ``tf-m-tests/test/secure_fw/suites``. If the new test suite includes both
52non-secure and secure parts, the source code shall be divided shared code and
53specific code. An example test suite folder can be organized as the figure
54below.
55
56.. code-block:: bash
57
58 .
59 ├── CMakeLists.txt
60 ├── <test_name>_tests_common.c
61 ├── non_secure
62 ├── <test_name>_ns_interface_testsuite.c
63 └── <test_name>_ns_tests.h
64 └── secure
65 ├── <test_name>_s_interface_testsuite.c
66 └── <test_name>_s_tests.h
67
68Creating test configurations
69============================
70
71A test configuration controls whether one or multiple test suites are enabled.
BohdanHunko48eedb32022-10-19 19:01:25 +030072The doc `TF-M Build Instructions <https://tf-m-user-guide.trustedfirmware.org/technical_references/instructions/tfm_build_instruction.html>`_.
Jianliang Shen5f7b1b72022-05-03 19:01:27 +080073shows some test configurations which are already supported in current TF-M.
74An example usage of test configuration shows below.
75
76.. code-block:: bash
77
78 cmake -S . -B cmake_build -DTFM_PLATFORM=arm/mps2/an521 \
79 -DTFM_TOOLCHAIN_FILE=toolchain_GNUARM.cmake \
80 -DCMAKE_BUILD_TYPE=Release \
81 -DTEST_NS_ATTESTATION=ON
82
83With this command, only non-secure initial attestation test suite will be built.
84
85Developers shall assign corresponding test configurations to control the test
86suites.
87
88Naming test configurations
89--------------------------
90
91The test configurations of example test suites are ``TEST_NS_<TEST_NAME>``
92in non-secure and ``TEST_S_<TEST_NAME>`` in secure.
93
94.. Note::
95 The test configurations must be named with the prefixes ``TEST_S_`` and
96 ``TEST_NS_``, for secure regression tests and non-secure regression tests
97 respectively. Otherwise, tf-m-tests build system may not recognize it.
98
99Setting test configurations
100---------------------------
101
102When the test configurations have dependences, the default value need to be set.
103The setting is performed in following four steps.
104
105#. Command line input: The configuration can be enabled or disabled by the
106 command ``-DTEST_NS_<TEST_NAME>=ON/OFF -DTEST_S_<TEST_NAME>=ON/OFF``, and
107 the value cannot be changed throughout the whole compilation of TF-M tests.
108
109#. ``tf-m-tests/config/set_config.cmake``: The test configurations shall be
110 OFF if its dependences are not supported. The dependences are probably
111 from:
112
113 #. TF-M partitions configurations like ``TFM_PARTITION_CRYPTO``,
114 ``TFM_PARTITION_INITIAL_ATTESTATION``, etc.
Summer Qinab4e42d2022-10-09 17:36:39 +0800115 #. TF-M build mode configuration like ``CONFIG_TFM_SPM_BACKEND``.
Mark Horvath2f072582022-09-09 16:15:30 +0200116 #. TF-M other configurations like ``TFM_PARTITION_FIRMWARE_UPDATE``, etc.
Jianliang Shen5f7b1b72022-05-03 19:01:27 +0800117
118#. ``tf-m-tests/config/default_ns_test_config.cmake`` or
119 ``tf-m-tests/config/default_s_test_config.cmake``: It is required to give
120 the default value of the new test configuration in these two files when
121 ``TEST_NS`` or ``TEST_S`` is ON. The recommended value is ON unless the
122 single test's code or data size is very large.
123
124#. ``tf-m-tests/config/default_test_config.cmake``: It is required to give the
125 default value of the new test configuration in the file when both
126 ``TEST_NS`` and ``TEST_S`` are OFF. The default value must be OFF.
127
128.. Note::
129 The test configurations must be set as CACHE value in CMAKE files. The CACHE
130 set cannot replace the value from command line, see
131 `Set Cache Entry <https://cmake.org/cmake/help/latest/command/set.html#set-cache-entry>`_.
132
133Checking test configurations
134----------------------------
135
136The new test configurations must be checked by function ``tfm_invalid_config()``
137if they have any dependence. The value comes from command line may be wrong when
138the dependences are conflicting. In addition to the dependences quoted in
139``tf-m-tests/config/set_config.cmake``, some other test configurations may be
140necessary.
141
BohdanHunko48eedb32022-10-19 19:01:25 +0300142Applying test configurations
143============================
Jianliang Shen5f7b1b72022-05-03 19:01:27 +0800144
145The mission of test configurations is to control the build. They are applied
146in ``test/secure_fw/suites/<test_name>/CMakeLists.txt`` like the example below.
147
148.. code-block:: cmake
149
150 cmake_policy(SET CMP0079 NEW)
151
152 if (NOT TEST_NS_<TEST_NAME> AND NOT TEST_S_<TEST_NAME>)
153 return()
154 endif()
155
156 ####################### Non Secure #########################################
157
158 if (TEST_NS_<TEST_NAME>)
159 add_library(tfm_test_suite_<test_name>_ns STATIC EXCLUDE_FROM_ALL)
160 # target_sources()
161 # target_include_directories()
162 target_compile_definitions(tfm_test_suite_<test_name>_ns
163 INTERFACE
164 TEST_NS_<TEST_NAME>
165 )
166 # target_link_libraries()
167 endif()
168
169 ####################### Secure #############################################
170
171 if (TEST_S_<TEST_NAME>)
172 add_library(tfm_test_suite_<test_name>_s STATIC EXCLUDE_FROM_ALL)
173 # target_sources()
174 # target_include_directories()
175 target_compile_definitions(tfm_test_suite_<test_name>_s
176 INTERFACE
177 TEST_S_<TEST_NAME>
178 )
179 # target_link_libraries()
180 endif()
181
182The function ``target_compile_definitions`` will export the macros
183``TEST_NS_<TEST_NAME>`` or ``TEST_S_<TEST_NAME>`` into source code. and in the
184file ``tf-m-tests/framework/non_secure_suites.c`` or
185``tests/framework/secure_suites.c``, the definitions of these macros will be
186checked, and then the head file will be included and test cases will be
187registered if the macro is defined.
188
189.. code-block:: c
190
191 #ifdef TEST_NS_<TEST_NAME>
192 #include "<test_name>_ns_tests.h"
193 #endif
194
195 static struct test_suite_t test_suites[] = {
196 /* Non-secure example test cases */
197 // ......
198 #ifdef TEST_NS_<TEST_NAME>
199 {&register_testsuite_ns_<test_name>_interface, 0, 0, 0},
200 #endif
201 };
202
203.. code-block:: c
204
205 #ifdef TEST_S_<TEST_NAME>
206 #include "<test_name>_s_tests.h"
207 #endif
208
209 static struct test_suite_t test_suites[] = {
210 /* Secure example test cases */
211 // ......
212 #ifdef TEST_S_<TEST_NAME>
213 {&register_testsuite_s_<test_name>_interface, 0, 0, 0},
214 #endif
215 };
216
217.. Note::
218 On most platforms non-secure tests and secure tests run on the same CPU
219 core, but dual-core platform is an exception. So secure test library and
BohdanHunko48eedb32022-10-19 19:01:25 +0300220 secure services shall be linked together in the file
Jianliang Shen5f7b1b72022-05-03 19:01:27 +0800221 ``tf-m-tests/test/secure_fw/secure_tests.cmake``. Thus they can be built on
222 secure CPU core and non-secure tests library and RTOS are built on
223 non-secure CPU core.
224
225.. code-block:: cmake
226
227 if (TEST_FRAMEWORK_S)
228 # ...
229 if (TEST_S_<TEST_NAME>)
230 add_library(tfm_test_suite_<test_name>_s STATIC EXCLUDE_FROM_ALL)
231 endif()
232 endif()
233
234************************************
235Adding a new test case in test suite
236************************************
237
238The test cases usually express as a function in source code. They will be added
239into an array with structure type called ``test_t`` defined in
240``tf-m-tests/test/framework/test_framework.h``.
241
242.. code-block:: c
243
244 struct test_t {
245 TEST_FUN * const test; /*!< Test function to call */
246 const char *name; /*!< Test name */
247 const char *desc; /*!< Test description */
248 };
249
250For example, a new test case called ``TFM_NS_<TEST_NAME>_TEST_1001`` is created
251and the function ``tfm_<test_name>_test_1001`` needs to be defined in file
252``<test_name>_ns_interface_testsuite.c``. Then the function shall be appended
253into the array which will be quoted in function
254``register_testsuite_ns_<test_name>_interface``. See the reference code below.
255
256.. code-block:: c
257
258 /* List of test cases */
259 static void tfm_<test_name>_test_1001(struct test_result_t *ret);
260
261 /* Append test cases */
262 static struct test_t <test_name>_tests[] = {
263 {&tfm_<test_name>_test_1001, "TFM_NS_<TEST_NAME>_TEST_1001",
264 "Example test case"},
265 };
266
267 /* Register test case into test suites */
268 void register_testsuite_ns_<test_name>_interface(struct test_suite_t *p_test_suite)
269 {
270 uint32_t list_size;
271
272 list_size = (sizeof(<test_name>_tests) / sizeof(<test_name>_tests[0]));
273
274 set_testsuite("<TEST_NAME> non-secure interface test (TFM_NS_<TEST_NAME>_TEST_1XXX)",
275 <test_name>_tests, list_size, p_test_suite);
276 }
277
278 static void tfm_<test_name>_test_1001(struct test_result_t *ret)
279 {
280 /* test case code */
281 }
282
283********************
284Adding test services
285********************
286
287Some test group may need specific test services. These test services may support
288one or more groups thus developers shall determine the proper test scope. Refer
289to
290`Adding partitions for regression tests <https://git.trustedfirmware.org/TF-M/tf-m-tests.git/tree/docs/tfm_test_services_addition.rst>`_
291to get more information.
292
293**********************************
294Out-of-tree regression test suites
295**********************************
296
297TF-M supports out-of-tree regression test suites build, whose source code
298folders are maintained outside tf-m-tests repo. There are two configurations
299for developers to include the source code.
300
301- ``EXTRA_NS_TEST_SUITES_PATHS``
302
303 A list of the absolute directories of the out-of-tree non-secure test suites
304 source code folder(s). TF-M build system searches ``CMakeLists.txt`` of
305 non-secure test suites in the source code folder(s).
306 Use semicolons ``;`` to separate multiple out-of-tree non-secure test suites
BohdanHunko48eedb32022-10-19 19:01:25 +0300307 directories.
Jianliang Shen5f7b1b72022-05-03 19:01:27 +0800308
309- ``EXTRA_S_TEST_SUITES_PATHS``
310
311 A list of the absolute directories of the out-of-tree secure test suites
312 source code folder(s).
313
314Example usage
315=============
316
317Take non-secure test as an example in
318`tf-m-extras <https://git.trustedfirmware.org/TF-M/tf-m-extras.git/>`_.
319A single out-of-tree test suite folder can be organized as the figure below:
320
321.. code-block:: bash
322
323 extra_ns
324 ├── CMakeLists.txt
325 ├── ns_test.c
326 ├── ns_test_config.cmake
327 └── ns_test.h
328
329In the example above, ``EXTRA_NS_TEST_SUITES_PATHS`` in the build command can be
330specified as listed below.
331
332.. code-block:: bash
333
334 -DEXTRA_NS_TEST_SUITES_PATHS=<Absolute-path-extra-test-folder>
335
336Coding instructions
337===================
338
339This is a demo of source code so the structure has been simplified. Files like
340``ns_test.c`` and ``ns_test.h`` can be expanded to ``src`` and ``include``
341folders. The ``CMakeLists.txt`` is required in the root path and
342``ns_test_config.cmake`` is optional.
343
344Header Files
345------------
346
347The header file ``extra_ns_tests.h`` must be included by out-of-tree source
348code. This file contains the definition of ``struct extra_tests_t``,
349``int32_t register_extra_tests()`` and declaration of
350``int32_t extra_ns_tests_init()``.
351
352Source code
353-----------
354
355To connect the out-of-tree source code and tf-m-tests framework, the high-level
356test function must be defined first. An example format is:
357
358.. code-block:: c
359
360 int32_t ns_test(void)
361 {
362 /* Add platform specific non-secure test suites code here. */
363
364 return EXTRA_TEST_SUCCESS;
365 }
366
367This function is the main entry to test framework. All the out-of-tree's test
368cases can be added into it. The returned error code of this function is
369specified as ``int32_t``. This function shall return an expected value which is
370same as the part ``expected_ret`` set in ``plat_ns_t`` below, take the macro
371EXTRA_TEST_SUCCESS as an example.
372
373After ``ns_test()`` is defined, a structure variable need to be created like:
374
375.. code-block:: c
376
377 const struct extra_tests_t plat_ns_t = {
378 .test_entry = ns_test,
379 .expected_ret = EXTRA_TEST_SUCCESS
380 };
381
382It will be used by function ``extra_ns_tests_init()`` to register the test by
383function ``register_extra_tests()``:
384
385.. code-block:: c
386
387 int32_t extra_ns_tests_init(struct extra_tests_t *internal_test_t)
388 {
389 /* Add platform init code here. */
390
391 return register_extra_tests(internal_test_t, &plat_ns_t);
392 }
393
394The platform initialization code can be added in this function because it runs
395before ``ns_test()``.
396
397.. Note::
398 Function ``extra_ns_tests_init()`` is declared in tf-m-tests repository
399 without definition. It is supplied to out-of-tree source code and need to be
400 defined with no change of its format, like returns error code and parameter
401 name.
402
403
404CMakeLists.txt
405--------------
406
407In addition to the implementation of CMAKE target like ``example_test_ns`` of
408out-of-tree source code, the configuration below needs to be appended:
409
410.. code-block:: cmake
411
412 # Example test must link tfm_test_suite_extra_common to use related interface
413 target_link_libraries(example_test_ns
414 PRIVATE
415 tfm_test_suite_extra_common
416 )
417
418 # The example_test_ns library must be linked by tfm_test_suite_extra_common
419 target_link_libraries(tfm_test_suite_extra_ns
420 PRIVATE
421 example_test_ns
422 )
423
424To use the interfaces come from tf-m-tests repository, library
425``tfm_test_suite_extra_common`` must be linked by ``example_test_ns``.
426To add out-of-tree test into TF-M, library ``example_test_ns`` must be linked
427by ``tfm_test_suite_extra_ns``.
428
429ns_test_config.cmake
430--------------------
431
432The CMAKE configuration file is optional. If out-of-tree source already exists
433another configuration file, a new one can be ignored.
434
435--------------
436
437*Copyright (c) 2021-2022, Arm Limited. All rights reserved.*