Initial version of unit testing documentation
This commit includes the 'User guide' and 'Implementing tests' sections
for helping people building, running and writing unit tests. A more
detailed version of documentation is available under the 'Component
user manuals section'.
Change-Id: I67e93ac805d1f4e7727964f3d95a70436ae34733
Signed-off-by: Imre Kis <imre.kis@arm.com>
diff --git a/docs/implementing_tests.rst b/docs/implementing_tests.rst
new file mode 100644
index 0000000..cc107d2
--- /dev/null
+++ b/docs/implementing_tests.rst
@@ -0,0 +1,366 @@
+Implementing tests
+==================
+
+Concept of unit testing
+-----------------------
+
+First of all unit tests exercise the C code on a function level. The tests
+should call functions directly from the code under tests and verify if their
+return values are matching the expected ones and the functions are behaving
+according to the specification.
+
+Because of the function level testing the dependencies of the tested functions
+should be detached. This is done by mocking the underlying layer. This provides
+an additional advantage of controlling and verifying all the call to the lower
+layer.
+
+
+Adding new unit test suite
+--------------------------
+
+The first step is to define a new unit test suite. If a completely new module is
+being test the test suite definition should be created in a separate ``.cmake``
+file which is placed in the test files' directory. Otherwise the test
+definition can be added to an existing ``.cmake`` file. These files should be
+included in the root ``CMakeLists.txt``.
+
+The ``UnitTest`` CMake module defines the ``unit_test_add_suite`` function so
+before using this function the module must be included in the ``.cmake`` file.
+The function first requires a unique test name which will be test binary's name.
+The test sources, include directories and macro definition are passed to the
+function in the matching arguments. CMake variables can be used to reference
+files relative to common directories:
+
+- ``CMAKE_CURRENT_LIST_DIR`` - Relative to the ``.cmake`` file
+- :cmake:variable:`TF_A_PATH` - Relative to the Trusted Firmware-A root directory
+- :cmake:variable:`TF_A_UNIT_TESTS_PATH` - Relative to the unit test root directory
+
+.. code-block:: cmake
+
+ # tests/new_module/new_test_suite.cmake
+ include(UnitTest)
+
+ unit_test_add_suite(
+ NAME [unique test name]
+ SOURCES
+ [source files]
+ INCLUDE_DIRECTORIES
+ [include directories]
+ COMPILE_DEFINITIONS
+ [defines]
+ )
+
+.. code-block:: cmake
+
+ # Root CMakeLists.txt
+ include(tests/new_module/new_test_suite.cmake)
+
+Example test definition
+^^^^^^^^^^^^^^^^^^^^^^^
+
+.. code-block:: cmake
+
+ unit_test_add_suite(
+ NAME memcmp
+ SOURCES
+ ${CMAKE_CURRENT_LIST_DIR}/test_memcmp.cpp
+ ${CMAKE_CURRENT_LIST_DIR}/memcmp.yml
+ INCLUDE_DIRECTORIES
+ ${TF_A_PATH}/include
+ ${TF_A_PATH}/include/lib/libc/aarch64/
+ )
+
+
+Using c-picker
+--------------
+
+c-picker is a simple tool used for detaching dependencies of the code under
+test. It can copy elements (i.e. functions, variables, etc.) from the original
+source code into generated files. This way the developer can pick functions from
+compilation units and surround them with a mocked environment.
+
+If a ``.yml`` file listed among source files the build system invokes c-picker
+and the generated ``.c`` file is implicitly added to the source file list.
+
+Example .yml file
+^^^^^^^^^^^^^^^^^
+
+In this simple example c-picker is instructed to copy the include directives and
+the ``memcmp`` function from the ``lib/libc/memcmp.c`` file. The root directory
+of the source files referenced by c-picker is the Trusted Firmware-A root
+directory.
+
+.. code-block:: yaml
+
+ elements:
+ - file: lib/libc/memcmp.c
+ type: include
+ - file: lib/libc/memcmp.c
+ type: function
+ name: memcmp
+
+
+Writing unit tests
+------------------
+
+Unit test code should be placed in ``.cpp`` files.
+
+Four-phase test pattern
+^^^^^^^^^^^^^^^^^^^^^^^
+
+All tests cases should follow the four-phase test pattern. This consists of four
+simple steps that altogether ensure the isolation between test cases. These
+steps follows below.
+
+- Setup
+- Exercise
+- Verify
+- Teardown
+
+After the teardown step all global states should be the same as they were at the
+beginning of the setup step.
+
+CppUTest
+^^^^^^^^
+
+CppUTest is an open source unit testing framework for C/C++. It is written in
+C++ so all the useful features of the language is available while testing. It
+automatically collects and runs the defined ``TEST_GROUPS`` and provides an
+interface for implementing the four-phase test pattern. Furthermore the
+framework has assertion macros for many variable types and test scenarios.
+
+Include
+'''''''
+
+The unit test source files should include the CppUTest header after all other
+headers to avoid conflicts.
+
+.. code-block:: C++
+
+ // Other headers
+ // [...]
+
+ #include "CppUTest/TestHarness.h"
+
+Test group
+''''''''''
+
+The next step is to define a test group. When multiple tests cases are written
+around testing the same function or couple related functions these tests cases
+should be part of the same test group. Basically test cases in a test group
+share have same setup/teardown sequence. In CppUTest the ``TEST_GROUP`` macro
+defines a new class which can contain member variables and functions. Special
+setup/teardown function are defined using ``TEST_SETUP`` and ``TEST_TEARDOWN``
+macros. These functions are called before/after running each test case of the
+group so all the common initilization and cleanup code should go into these
+functions.
+
+.. code-block:: C++
+
+ TEST_GROUP(List) {
+ TEST_SETUP() {
+ list = list_alloc();
+ }
+
+ TEST_TEARDOWN() {
+ list_cleanup(list);
+ }
+
+ bool has_element(int value) {
+ for (int i = 0; i < list_count (list); i++) {
+ if (list_get(i) == value) { return true; }
+ }
+ return false;
+ }
+
+ List* list;
+ };
+
+
+Test case
+'''''''''
+
+Test cases are defined by the ``TEST`` macro. This macro defines a child class
+of the test group's class so it can access the member functions and variables of
+the test group. The test case's block itself is the body of the function of the
+child class.
+
+.. code-block:: C++
+
+ TEST(List, add_one) {
+ // Exercise
+ const int test_value = 5;
+ list_add(list, test_value);
+
+ // Verify using CHECK_TRUE assertion and TEST_GROUP member function
+ CHECK_TRUE(has_element(test_value));
+ }
+
+ TEST(List, add_two) {
+ // Exercise
+ const int test_value1 = 5;
+ const int test_value2 = 6;
+ list_add(list, test_value1);
+ list_add(list, test_value2);
+
+ // Verify
+ CHECK_TRUE(has_element(test_value1));
+ CHECK_TRUE(has_element(test_value2));
+ }
+
+CppUMock
+^^^^^^^^
+
+During unit testing the dependencies of the tested functions should be replaced
+by stubs or mocks. When using mocks the developer would usually like to check if
+the function was called with corrent parameters and would like to return
+controlled values from the function. When a mocked function is called multiple
+times from the tested function maybe it should check or return different values
+on each call. This is where CppUMock comes handy.
+
+All CppUMock interactions start with calling the ``mock()`` function. This
+returs a reference to the mocking system. At this point the developer either
+wants to define expected or actual calls. This is achiveable by calling
+``expectOneCall(functionName)`` or ``expectNCalls(amount, functionName)`` or
+``actualCall(functionName)`` member functions of ``mock()`` call's return value.
+Registering expected calls are done in the test case before exercising the code
+and actual calls happen from the mocked functions.
+
+After this point the following functions can be chained:
+
+- ``onObject(object)`` - In C++ it is usually the ``this`` pointer but it can be
+ useful in C too.
+- ``with[type]Parameter(name, value)`` - Specifying and checking call parameters
+- ``andReturnValue(result)`` - Specifying return value when defining expected
+ call
+- ``return[type]Value()`` - Returning value from function
+
+The mocking system has two important functions. ``mock().checkExpectation()``
+checks if all the expected calls have been fulfilled and and the
+``mock().clear()`` removes all the expected calls from CppUMock's registry.
+These two functions are usually called from the ``TEST_TEARDOWN`` function
+because there should not be any crosstalk between test cases through the mocking
+system.
+
+CppUMock's typical use-case is shown below by a simple example of the
+``print_to_eeprom`` function.
+
+.. code-block:: C++
+
+ int eeprom_write(const char* str); /* From eeprom.h */
+
+ int printf_to_eeprom(const char* format, ...) {
+ char buffer[256];
+ int length, written_bytes = 0, eeprom_write_result;
+ va_list args;
+
+ va_start(args, format);
+ length = vsnprintf(buffer, sizeof(buffer), format, args);
+ va_end(args);
+
+ if (length < 0) {
+ return length;
+ }
+
+ while(written_bytes < length) {
+ eeprom_write_result = eeprom_write(&buffer[written_bytes]);
+ if (eeprom_write_result < 0) {
+ return eeprom_write_result;
+ }
+ written_bytes += eeprom_write_result;
+ }
+
+ return written_bytes;
+ }
+
+Having the code snipped above a real life usage of the function would look like
+something shown in the following sequence diagram.
+
+.. uml:: resources/sequence_print_to_eeprom.puml
+
+It would be really hard to test unit this whole system so all the lower layers
+should be separated and mock on the first possible level. In the following
+example the ``print_to_eeprom`` function is being tested and the
+``eeprom_write`` function is mocked. In test cases where ``eeprom_write``
+function is expected to be called the test case should first call the
+``expect_write`` function. This registers an expected call to CppUMocks internal
+database and when the call actually happens it matches the call parameters with
+the entry in the database. It also returns the previously specified value.
+
+.. code-block:: C++
+
+ TEST_GROUP(printf_to_eeprom) {
+ TEST_TEARDOWN() {
+ mock().checkExpectations();
+ mock().clear();
+ }
+
+ void expect_write(const char* str, int result) {
+ mock().expectOneCall("eeprom_write").withStringParameter("str", str).
+ andReturnValue(result);
+ }
+ };
+
+ /* Mocked function */
+ int eeprom_write(const char* str) {
+ return mock().actualCall("eeprom_write").withStringParameter("str", str).
+ returnIntValue();
+ }
+
+ TEST(printf_to_eeprom, empty) {
+ LONGS_EQUAL(0, printf_to_eeprom(""))
+ }
+
+ TEST(printf_to_eeprom, two_writes) {
+ expect_write("hello1hello2", 6);
+ expect_write("hello2", 6);
+ LONGS_EQUAL(12, printf_to_eeprom("hello%dhello%d", 1, 2))
+ }
+
+ TEST(printf_to_eeprom, error) {
+ expect_write("hello", -1);
+ LONGS_EQUAL(-1, printf_to_eeprom("hello"))
+ }
+
+This how the ``printf_to_eeprom/two_writes`` test case's sequence diagram looks
+like after mocking ``eeprom_write``. The test case became able to check the
+parameters of multiple calls and it could return controlled values.
+
+.. uml:: resources/sequence_print_to_eeprom_mock.puml
+
+
+Analyzing code coverage
+-----------------------
+
+The code coverage reports can be easily used for finding untested parts of the
+code. The two main parts of the coverage report are the line coverage and the
+branch coverage. Line coverage shows that how many times the tests ran the given
+line of the source code. It is beneficial to increase the line coverage however
+100% line coverage is still not enough to consider the code fully tested.
+
+Let's have a look on the following example.
+
+.. code-block:: C++
+
+ void set_pointer_value(unsigned int id, unsigned int value) {
+ unsigned int *pointer;
+
+ if (id < MAX_ID) {
+ pointer = get_pointer(id);
+ }
+
+ *pointer = value;
+ }
+
+The 100% line coverage is achievable by testing the function with an ``id``
+value smaller than ``MAX_ID``. However if an ``id`` larger than or equal to
+``MAX_ID`` is used as a parameter of this function it will try to write to a
+memory address pointed by an uninitialized variable. To catch untested
+conditions like this the branch coverage comes handy. It will show that only one
+branch of the ``if`` statement has been tested as the condition was always true
+in the tests.
+
+
+--------------
+
+*Copyright (c) 2019-2020, Arm Limited. All rights reserved.*