Initial version of unit testing documentation
This commit includes the 'User guide' and 'Implementing tests' sections
for helping people building, running and writing unit tests. A more
detailed version of documentation is available under the 'Component
user manuals section'.
Change-Id: I67e93ac805d1f4e7727964f3d95a70436ae34733
Signed-off-by: Imre Kis <imre.kis@arm.com>
diff --git a/docs/components/build_system.rst b/docs/components/build_system.rst
new file mode 100644
index 0000000..672681f
--- /dev/null
+++ b/docs/components/build_system.rst
@@ -0,0 +1,25 @@
+Build system
+============
+
+.. cmake-module:: ../../CMakeLists.txt
+
+.. cmake-module:: ../../cmake/UnitTest.cmake
+
+.. cmake-module:: ../../cmake/FindLibClang.cmake
+
+Generating documentation
+------------------------
+
+``sphinx-builder`` is used for building the documentation which is required to
+be installed on the machine. The documentation can be built by executing the
+following commands.
+
+::
+
+ cd docs
+ make html
+
+
+--------------
+
+*Copyright (c) 2019-2020, Arm Limited. All rights reserved.*
diff --git a/docs/components/c_picker.rst b/docs/components/c_picker.rst
new file mode 100644
index 0000000..673d2ce
--- /dev/null
+++ b/docs/components/c_picker.rst
@@ -0,0 +1,63 @@
+c-picker
+========
+
+c-picker uses ``libclang``'s Python interface for parsing source files.
+
+Command line options
+--------------------
+
+- ``-h, --help`` - Showing help message
+- ``--root ROOT`` - Root source directory
+- ``--config CONFIG`` - Configuration file (``.json|.yml``)
+- ``--output OUTPUT`` - Output file
+- ``--print-dependencies`` - Prints the dependencies
+- ``--version`` - Shows the program's version number and exit
+- ``--args ...`` - clang arguments
+
+
+Configuration file format
+-------------------------
+
+c-picker configuration can be described in ``JSON`` or ``YAML`` format using the
+same object structure.
+
+- ``elements`` - List of elements to pick from the original code
+
+ - ``name`` - Name of the element
+ - ``type`` - Type of the element: ``include``, ``function``, ``variable``
+ - ``args`` - Arguments for clang used during picking the element
+ - ``options`` - Currenly the only options is ``remove_static`` which removes
+ the ``static`` qualifier from the element's definition.
+
+- ``options`` - Global options for all elements
+- ``args`` - Global clang arguments for all elements
+
+YAML example
+------------
+
+YAML format is preferred because it can contain metadata like comments and
+licence information.
+
+.. code-block:: YAML
+
+ elements:
+ - file: bl1/bl1_fwu.c
+ type: variable
+ name: bl1_fwu_loaded_ids
+ options: [remove_static]
+ - file: bl1/bl1_fwu.c
+ type: function
+ name: bl1_fwu_add_loaded_id
+ options: [remove_static]
+ - file: bl1/bl1_fwu.c
+ type: function
+ name: bl1_fwu_remove_loaded_id
+ options: [remove_static]
+ args:
+ - -DFWU_MAX_SIMULTANEOUS_IMAGES=10
+ - -DINVALID_IMAGE_ID=0xffffffff
+
+
+--------------
+
+*Copyright (c) 2019-2020, Arm Limited. All rights reserved.*
diff --git a/docs/components/code_coverage.rst b/docs/components/code_coverage.rst
new file mode 100644
index 0000000..6ac6d9f
--- /dev/null
+++ b/docs/components/code_coverage.rst
@@ -0,0 +1,31 @@
+Code coverage
+=============
+
+Coverage processing flow
+------------------------
+
+1. Prerequisites
+
+ 1. Having :cmake:variable:`COVERAGE`` CMake variable set to ``ON``
+
+ 2. Building all or selected test binaries with coverage flags
+
+ 3. Running all or selected test binaries
+
+2. Collecting coverage data from ``.gcda`` and ``.gcno`` file into ``lcov``
+ coverage info file
+
+3. Mapping c-picker generated files' coverage to the original source lines
+
+4. Filtering coverage data for separating the coverage of the code under tests
+ and the coverage of the test code
+
+5. Generating HTML coverage report from the filtered lcov info files
+
+
+.. cmake-module:: ../../cmake/Coverage.cmake
+
+
+--------------
+
+*Copyright (c) 2019-2020, Arm Limited. All rights reserved.*
diff --git a/docs/components/cppumock.rst b/docs/components/cppumock.rst
new file mode 100644
index 0000000..145c4ad
--- /dev/null
+++ b/docs/components/cppumock.rst
@@ -0,0 +1,212 @@
+CppUMock
+========
+
+CppUMock is the built-in mocking system of CppUTest. This chapter describes the
+basic features of the system. For details please refer the `official CppUMock
+manual`_.
+
+System functions
+----------------
+
+Checking expectations
+^^^^^^^^^^^^^^^^^^^^^
+
+After defining expected calls an invoking actual call the test should check if
+all the expected calls have happened. This can be done by calling
+``mock().checkExpectations()``. The common place to put this function call is
+the ``TEST_TEARDOWN`` function of the test group.
+
+
+Resetting mocking system
+^^^^^^^^^^^^^^^^^^^^^^^^
+
+After the last interaction with the mocking system the developer should reset
+the state of the system by calling ``mock().clear()``. The common place to put
+this function call is the ``TEST_TEARDOWN`` function of the test group after
+calling ``mock().checkExpectations()``.
+
+
+Namespaces
+^^^^^^^^^^
+
+All interactions with the mocking system start by calling ``mock()``. This
+function has an option ``name`` string parameter which can be used for limiting
+the scope of the mocking operation.
+
+.. code-block:: C++
+
+ mock("bl1").expectOneCall("bl1_main");
+
+
+Enable/disable
+^^^^^^^^^^^^^^
+
+The mocking system can be enabled/disabled runtime using the functions below.
+This causes call expected and actual call to be ignored. Default settings are
+restored by ``mock().clear()``.
+
+- ``enable()``
+- ``disable()``
+
+.. code-block:: C++
+
+ mock().disable();
+ // All CppUMock calls are ignored after this point
+ mock().enable();
+ // CppUMock calls are working again
+
+ mock().disable();
+ // All CppUMock calls are ignored after this point
+ [...]
+
+ mock().clear();
+ // Default settings are restored
+
+
+String order
+^^^^^^^^^^^^
+
+After defining multiple expected function call the mocking system always
+The mocking system always uses the next matching function when an actual call
+happens but by default it doesn't check if different function calls happen in
+the order of definition. This behaviour can be turned on using the
+``mock().strictOrder()`` function. This option is also set to default by the
+``mock().clear()`` function.
+
+.. code-block:: C++
+
+ mock().expectOneCall("A");
+ mock().expectOneCall("B");
+
+ mock().actualCall("B");
+ mock().actualCall("A");
+ // No error
+
+ mock().strictOrder();
+ mock().expectOneCall("A");
+ mock().expectOneCall("B");
+
+ mock().actualCall("B"); // Error generated here
+ mock().actualCall("A");
+
+
+Ignoring unspecified calls
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+If there are addition actual calls happening in the test case which are
+unrelated to the test case (i.e. log or other messages) the unspecified calls
+can be ignored by adding ``mock().ignoreOtherCalls()`` to the test case. This
+function should be used carefully because it can hide unexpected call which are
+indicating errors in the code. The affect of this call ends by calling
+``mock().clear()``.
+
+
+Specifying object
+-----------------
+
+In object oriented environment member function should validate if the function
+call happened on the suitable object. This is done by adding the following
+function to the mock specification.
+
+- ``onObject(objectPtr)``
+
+
+Validating parameters
+---------------------
+
+Each supported parameter type has a corresponding function. These are the same
+in the expected and actual calls.
+
+- ``withBoolParameter(name, bool value)``
+- ``withIntParameter(name, int value)``
+- ``withUnsignedIntParameter(name, unsigned int value)``
+- ``withLongIntParameter(name, long int value)``
+- ``withUnsignedLongIntParameter(name, unsigned long int value)``
+- ``withDoubleParameter(name, double value)``
+- ``withStringParameter(name, const char* value)``
+- ``withPointerParameter(name, void* value)``
+- ``withFunctionPointerParameter(name, void (*value)())``
+- ``withConstPointerParameter(name, const void* value)``
+- ``withMemoryBufferParameter(name, const unsigned char* value, size_t size)``
+
+If custum types are defined and copier/comparator objects were installed the
+following function can handle these parameters.
+
+- ``withParameterOfType(typeName, name, value)``
+
+There's an option to copying data from the test environment into the mock
+function. When setting expectations the following function can be used to set
+the pointer and the address of the data. **The mocking system will not create a
+copy of this data** so the original data should be kept intact until the actual
+call happens.
+
+- ``withOutputParameterReturning(name, value, size)``
+- ``withOutputParameterOfTypeReturning(typeName, name, value)``
+
+In the actual call the pair of these function are shown below.
+
+- ``withOutputParameter(name, output)``
+- ``withOutputParameterOfType(typeName, name, output)``
+
+
+Ignoring parameters
+^^^^^^^^^^^^^^^^^^^
+
+There are cases when the developer doesn't want to specify all parameters. The
+following function can set this behaviour in the expected call.
+
+- ``ignoreOtherParameters()``
+
+
+Specifying return values
+------------------------
+
+Using function name overloading the return values are specified by calling
+``andReturnValue`` and the parameter type will determine the exact function.
+
+- ``andReturnValue(bool value)``
+- ``andReturnValue(int value)``
+- ``andReturnValue(unsigned int value)``
+- ``andReturnValue(long int value)``
+- ``andReturnValue(unsigned long int value)``
+- ``andReturnValue(double value)``
+- ``andReturnValue(const char* value)``
+- ``andReturnValue(void* value)``
+- ``andReturnValue(const void* value)``
+- ``andReturnValue(void (*value)())``
+
+
+Returning value in actual calls
+-------------------------------
+
+All of these function have version with ``OrDefault(type default_value)``
+suffix. These version return a default value if the return value was not
+specified in the expected call.
+
+- ``bool returnBoolValue()``
+- ``int returnIntValue()``
+- ``unsigned int returnUnsignedIntValue()``
+- ``long int returnLongIntValue()``
+- ``unsigned long int returnUnsignedLongIntValue()``
+- ``double returnDoubleValue()``
+- ``const char * returnStringValue()``
+- ``void * returnPointerValue()``
+- ``const void * returnConstPointerValue()``
+- ``void (*returnFunctionPointerValue())()``
+
+
+Debugging CppUMock errors
+-------------------------
+
+Debugging CppUMock errors can be hard unlike assertion errors because a mocking
+failure can happen multiple layers of function calls under the test case. The
+mocking system has a very similar feature to CppUTest's ``UT_CRASH()`` which is
+``mock().crashOnFailure()``. By enabling this feature the code will crash on
+mocking errors and the developer could easily catch it with the debugger.
+
+
+--------------
+
+*Copyright (c) 2019-2020, Arm Limited. All rights reserved.*
+
+.. _`official CppUMock manual`: https://cpputest.github.io/mocking_manual.html
\ No newline at end of file
diff --git a/docs/components/cpputest.rst b/docs/components/cpputest.rst
new file mode 100644
index 0000000..4963b76
--- /dev/null
+++ b/docs/components/cpputest.rst
@@ -0,0 +1,291 @@
+CppUTest
+========
+
+This document is based on CppUTest v3.8. CppUtest is a unit testing framework
+for testing C and C++ code. This document introduces the basic features of the
+framework. For further information check the `official manual of CppUTest`_.
+
+
+Why CppUTest?
+-------------
+
+First of all it was not our goal to develop a new unit testing framework while
+plenty of open source solutions are already available. There were no special
+requirements agains the unit testing framework that would rule out all existing
+frameworks so we only had to choose a suitable one for our current and possible
+future needs.
+
+We ended up selecting CppUTest because of its small footprint and easy
+portability. It also goes along with the standard xUnit frameworks' principles
+and provides a standard interface to the outside world. Some details are listed
+below.
+
+- C/C++ support
+- Small footprint (compared to Google Test)
+- Easy to use on embedded systems
+- Built-in mocking system (CppUMock)
+- Implements four-phase testing pattern (setup, exercise, verify, teardown)
+- Selective run of test cases
+- Standard output format (JUnit, TeamCity)
+
+
+Assertions
+----------
+
+Generally is a good practice to use more specific assertions because it can
+output more informative error messages in case of a failure. The generic form or
+assertions is ``assert(expected, actual)``. Each assert type has a _TEXT variant
+for user defined error messages as last parameter.
+
+- Boolean
+
+ - ``CHECK(condition)`` - Same as ``CHECK_TRUE``
+ - ``CHECK_TRUE(condition)``
+ - ``CHECK_FALSE(condition)``
+
+- C string
+
+ - ``STRCMP_EQUAL(expected, actual)``
+ - ``STRNCMP_EQUAL(expected, actual, length)``
+ - ``STRCMP_NOCASE_EQUAL(expected, actual)``
+ - ``STRCMP_CONTAINS(expected, actual)``
+ - ``STRCMP_NOCASE_CONTAINS(expected, actual)``
+
+- Integer
+
+ - ``LONGS_EQUAL(expected, actual)``
+ - ``UNSIGNED_LONGS_EQUAL(expected, actual)``
+ - ``LONGLONGS_EQUAL(expected, actual)``
+ - ``UNSIGNED_LONGLONGS_EQUAL(expected, actual)``
+ - ``BYTES_EQUAL(expected, actual)``
+ - ``SIGNED_BYTES_EQUAL(expected, actual)``
+ - ``POINTERS_EQUAL(expected, actual)``
+ - ``FUNCTIONPOINTERS_EQUAL(expected, actual)``
+
+- Enums
+
+ - ``ENUMS_EQUAL_INT(expected, actual)``
+ - ``ENUMS_EQUAL_TYPE(underlying_type, expected, actual)``
+
+- Other assertions
+
+ - ``CHECK_EQUAL(expected, actual)`` - Requires ``operator=`` and
+ ``StringFrom(type)`` to be implemented
+ - ``CHECK_COMPARE(first, relop, second)`` - Same as ``CHECK_EQUAL`` but with
+ any type of compare
+ - ``DOUBLES_EQUAL(expected, actual, threshold)``
+ - ``MEMCMP_EQUAL(expected, actual, size)``
+ - ``BITS_EQUAL(expected, actual, mask)``
+ - ``FAIL()`` or ``FAIL_TEST()`` - Test case fails if called
+ - ``CHECK_THROWS(expected, expression)`` - Catching C++ exceptions
+
+- Miscellaneous macros
+
+ - ``IGNORE_TEST`` - Same as ``TEST`` but it’s not called
+ - ``TEST_EXIT`` - Exists test
+ - ``UT_CRASH()`` - Crashes the test which is easy to catch with debugger
+ - ``UT_PRINT(text)`` - Generic print function
+
+
+Test runner
+-----------
+
+Test cases are collected automatically. Under the hood the ``TEST`` macros are
+creating global instances of classes and their constructor registers the test
+cases into the test registry. This happens before entering the ``main``
+function. In the ``main`` function only the ``RUN_ALL_TESTS`` macro needs to be
+placed with the command line arguments passed to it. On executing the binary the
+command line arguments will control the behaviour of the test process.
+
+.. code-block:: C++
+
+ #include "CppUTest/CommandLineTestRunner.h"
+
+ int main(int argc, char* argv[]) {
+ return RUN_ALL_TESTS(argc, argv);
+ }
+
+The default ``main`` implementation is added to all unit test suites by the
+build system.
+
+
+Command line options
+--------------------
+
+Command line options are available mainly for configuring the output format of
+the test binary and for filtering test groups or cases.
+
+- Output
+
+ - ``-v`` - Prints each test name before running them
+ - ``-c`` - Colorized output
+ - ``-o{normal, junit, teamcity}`` - Output format, junit can be processed by
+ most CIs
+ - ``-k packageName`` - Package name for junit output
+ - ``-lg`` - List test groups
+ - ``-ln`` - List test cases
+
+- Other
+
+ - ``-p`` - Runs each test case in separate processes
+ - ``-ri`` - Runs ignored test cases
+ - ``-r#`` - Repeats testing ``#`` times
+ - ``-s seed`` - Shuffles tests
+
+- Filtering test cases
+
+ - ``"TEST(groupName, testName)"`` - Running single test
+ - ``"IGNORE_TEST(groupName, testName)"`` -- Running single ignored test
+ - ``-g text`` - Runing groups containing text
+ - ``-n text`` - Runing tests containing text
+ - ``-sg text`` - Runing groups matching text
+ - ``-sn text`` - Runing tests matching text
+ - ``-xg text`` - Excluding groups containing text
+ - ``-xn text`` - Excluding tests containing text
+ - ``-xsg text`` - Excluding groups matching text
+ - ``-xsn text`` - Excluding tests matching text
+
+
+Troubleshooting
+---------------
+
+Output messages
+^^^^^^^^^^^^^^^
+
+When one of tests fails the first step is to run it separately and check its
+output message. Usually it shows the exact line of the file where the error
+happened.
+
+::
+
+ test_memcmp.cpp:17: error: Failure in TEST(memcmp, empty)
+ expected <1 0x1>
+ but was <0 0x0>
+
+The executed tests can be followed by adding ``-v`` command line option.
+
+::
+
+ ./memcmp -v
+ TEST(memcmp, different) - 0 ms
+ TEST(memcmp, same) - 0 ms
+ TEST(memcmp, empty) - 0 ms
+
+ OK (3 tests, 3 ran, 1 checks, 0 ignored, 0 filtered out, 0 ms)
+
+
+Catching failure with debugger
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+If a failure happens in a helper function or in a loop where the assertion
+is called multiple times it is harder to get the exact environment of a failure.
+In this case it's a good practice to put a ``UT_CRASH()`` call into a
+conditional block which hits when the failure happens. This way the debugger can
+stop on failure because the code emits a signal.
+
+.. code-block:: C++
+
+ TEST(magic, iterate) {
+ int result;
+
+ for(int i = 0; i < 1000; i++) {
+ result = magic_function(i);
+
+ // Debug code
+ if (result) {
+ UT_CRASH();
+ }
+
+ LONGS_EQUAL(0, result);
+ }
+ }
+
+
+Using ``FAIL`` macro
+^^^^^^^^^^^^^^^^^^^^
+
+It's recommended to use ``FAIL`` macro in conditions that should never occur in
+tests. For example if a test case loads test data from an external file but the
+file could not be opened the ``FAIL`` macro should be used with an informative
+message.
+
+.. code-block:: C++
+
+ fd = open("test.bin", O_RDONLY);
+ if (fd < 0) {
+ FAIL("test.bin open failed");
+ }
+
+
+Interference between test cases
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Test cases can interfere if there's a global resource which was not restored to
+its original state after leaving a test case. This can be hard to find but at
+least the it's easy to make sure that this is root case of an error. Let's
+assume there's a global variable which is set during the test case but it
+original value is not restore at the end. CppUTest has an command line option
+for running each test case in separate process. This makes the global variable
+to have its original value at the beginning of the test cases. Basically if the
+test works by passing argument ``-p`` when running but fails without it, there's
+a good chance for having an interference between test cases.
+
+.. code-block:: C++
+
+ int x = 0;
+
+ TEST_GROUP(crosstalk) {
+ };
+
+ TEST(crosstalk, a) {
+ LONGS_EQUAL(0, x);
+ x = 1;
+ }
+
+ TEST(crosstalk, b) {
+ LONGS_EQUAL(0, x);
+ x = 1;
+ }
+
+ TEST(crosstalk, c) {
+ LONGS_EQUAL(0, x);
+ x = 1;
+ }
+
+By running the test executable with different command line arguments it produces
+a different result.
+
+.. code-block::
+
+ ./crosstalk -v
+
+ TEST(crosstalk, c) - 0 ms
+ TEST(crosstalk, b)
+ test_crosstalk.cpp:37: error:
+ Failure in TEST(crosstalk, b)
+ expected <0 0x0>
+ but was <1 0x1>
+
+ - 0 ms
+ TEST(crosstalk, a)
+ test_crosstalk.cpp:32: error: Failure in TEST(crosstalk, a)
+ expected <0 0x0>
+ but was <1 0x1>
+
+ - 0 ms
+
+ Errors (2 failures, 3 tests, 3 ran, 3 checks, 0 ignored, 0 filtered out, 0 ms)
+
+ ./crosstalk -v -p
+ TEST(crosstalk, c) - 1 ms
+ TEST(crosstalk, b) - 0 ms
+ TEST(crosstalk, a) - 0 ms
+
+ OK (3 tests, 0 ran, 0 checks, 0 ignored, 0 filtered out, 2 ms)
+
+
+--------------
+
+*Copyright (c) 2019, Arm Limited. All rights reserved.*
+
+.. _`official manual of CppUTest`: https://cpputest.github.io/
\ No newline at end of file
diff --git a/docs/components/index.rst b/docs/components/index.rst
new file mode 100644
index 0000000..3e8a68b
--- /dev/null
+++ b/docs/components/index.rst
@@ -0,0 +1,14 @@
+Component user manuals
+======================
+
+.. toctree::
+ cpputest
+ cppumock
+ c_picker
+ build_system
+ code_coverage
+
+
+--------------
+
+*Copyright (c) 2019, Arm Limited. All rights reserved.*