Trusted Firmware-A Tests, version 2.0
This is the first public version of the tests for the Trusted
Firmware-A project. Please see the documentation provided in the
source tree for more details.
Change-Id: I6f3452046a1351ac94a71b3525c30a4ca8db7867
Signed-off-by: Sandrine Bailleux <sandrine.bailleux@arm.com>
Co-authored-by: amobal01 <amol.balasokamble@arm.com>
Co-authored-by: Antonio Nino Diaz <antonio.ninodiaz@arm.com>
Co-authored-by: Asha R <asha.r@arm.com>
Co-authored-by: Chandni Cherukuri <chandni.cherukuri@arm.com>
Co-authored-by: David Cunado <david.cunado@arm.com>
Co-authored-by: Dimitris Papastamos <dimitris.papastamos@arm.com>
Co-authored-by: Douglas Raillard <douglas.raillard@arm.com>
Co-authored-by: dp-arm <dimitris.papastamos@arm.com>
Co-authored-by: Jeenu Viswambharan <jeenu.viswambharan@arm.com>
Co-authored-by: Jonathan Wright <jonathan.wright@arm.com>
Co-authored-by: Kévin Petit <kevin.petit@arm.com>
Co-authored-by: Roberto Vargas <roberto.vargas@arm.com>
Co-authored-by: Sathees Balya <sathees.balya@arm.com>
Co-authored-by: Shawon Roy <Shawon.Roy@arm.com>
Co-authored-by: Soby Mathew <soby.mathew@arm.com>
Co-authored-by: Thomas Abraham <thomas.abraham@arm.com>
Co-authored-by: Vikram Kanigiri <vikram.kanigiri@arm.com>
Co-authored-by: Yatharth Kochar <yatharth.kochar@arm.com>
diff --git a/docs/change-log.rst b/docs/change-log.rst
new file mode 100644
index 0000000..912a35a
--- /dev/null
+++ b/docs/change-log.rst
@@ -0,0 +1,241 @@
+
+.. section-numbering::
+ :suffix: .
+
+.. contents::
+
+Please note that the Trusted Firmware-A Tests version follows the Trusted
+Firmware-A version for simplicity. At any point in time, TF-A Tests version
+`x.y` aims at testing TF-A version `x.y`. Different versions of TF-A and TF-A
+Tests are not guaranteed to be compatible. This also means that a version
+upgrade on the TF-A-Tests side might not necessarily introduce any new feature.
+
+Trusted Firmware-A Tests - version 2.0
+======================================
+
+New features
+------------
+
+This is the first public release of the Trusted Firmware-A Tests source code.
+
+TFTF
+````
+
+- Provides a baremetal test framework to exercise TF-A features through its
+ ``SMC`` interface.
+
+- Integrates easily with TF-A: the TFTF binary is packaged in the FIP image
+ as a ``BL33`` component.
+
+- Standalone binary that runs on the target without human intervention (except
+ for some specific tests that require a manual target reset).
+
+- Designed for multi-core testing. The various sub-frameworks allow maximum
+ parallelism in order to stress the firmware.
+
+- Displays test results on the UART output. This may then be parsed by an
+ external tool and integrated in a continuous integration system.
+
+- Supports running in AArch64 (NS-EL2 or NS-EL1) and AArch32 states.
+
+- Supports parsing a tests manifest (XML file) listing the tests to include in
+ the binary.
+
+- Detects most platform features at run time (e.g. topology, GIC version, ...).
+
+- Provides a topology enumeration framework. Allows tests to easily go through
+ affinity levels and power domain nodes.
+
+- Provides an event framework to synchronize CPU operations in a multi-core
+ context.
+
+- Provides a timer framework. Relies on a single global timer to generate
+ interrupts for all CPUs in the system. This allows tests to easily program
+ interrupts on demand to use as a wake-up event source to come out of CPU
+ suspend state for example.
+
+- Provides a power-state enumeration framework. Abstracts the valid power
+ states supported on the platform.
+
+- Provides helper functions for power management operations (CPU hotplug,
+ CPU suspend, system suspend, ...) with proper saving of the hardware state.
+
+- Supports rebooting the platform at the end of each test for greater
+ independence between tests.
+
+- Supports interrupting and resuming a test session. This relies on storing
+ test results in non-volatile memory (e.g. flash).
+
+FWU images
+``````````
+
+- Provides example code to exercise the Firmware Update feature of TF-A.
+
+- Tests the robustness of the FWU state machine implemented in the TF-A by
+ sending valid and invalid authentication, copy and image execution requests
+ to the TF-A BL1 image.
+
+EL3 test payload
+````````````````
+
+- Tests the ability of TF-A to load an EL3 payload.
+
+Cactus test secure partition
+````````````````````````````
+
+- Tests that TF-A has correctly setup the secure partition environment: it
+ should be allowed to perform cache maintenance operations, access floating
+ point registers, etc.
+
+- Tests the ability of a secure partition to request changing data access
+ permissions and instruction permissions of memory regions it owns.
+
+- Tests the ability of a secure partition to handle StandaloneMM requests.
+
+Known issues and limitations
+----------------------------
+
+The sections below lists the known issues and limitations of each test image
+provided in this repository.
+
+TFTF
+````
+
+The TFTF test image might be conceptually sub-divided further in 2 parts: the
+tests themselves, and the test framework they are based upon.
+
+Test framework
+''''''''''''''
+
+- Some stability issues.
+
+- No mechanism to abort tests when they time out (e.g. this could be
+ implemented using a watchdog).
+
+- No convenient way to include or exclude tests on a per-platform basis.
+
+- Power domains and affinity levels are considered equivalent but they may
+ not necessarily be.
+
+- Need to provide better support to alleviate duplication of test code. There
+ are some recurrent test patterns for which helper functions should be
+ provided. For example, bringing up all CPUs on the platform and executing the
+ same function on all of them, or programming an interrupt and waiting for it
+ to trigger.
+
+- Every CPU that participates in a test must return from the test function. If
+ it does not - e.g. because it powered itself off for testing purposes - then
+ the test framework will wait forever for this CPU. This limitation is too
+ restrictive for some tests.
+
+- No protection against interrupted flash operations. If the target is reset
+ while some data is written to flash, the test framework might behave
+ incorrectly on reset.
+
+- When compiling the code, if the generation of the tests_list.c and/or
+ tests_list.h files fails, the build process is not aborted immediately and
+ will only fail later on.
+
+- The directory layout is confusing. Most of the test framework code has been
+ moved under the ``tftf/`` directory to better isolate it but this effort is
+ not complete. As a result, there are still some TFTF files scattered around.
+
+Tests
+'''''
+
+- Some tests are implemented for AArch64 only and are skipped on AArch32.
+
+- Some tests are not robust enough:
+
+ - Some tests might hang in some circumstances. For example, they might wait
+ forever for a condition to become true.
+
+ - Some tests rely on arbitrary time delays instead of proper synchronization
+ when executing order-sensitive steps.
+
+ - Some tests have been implemented in a practical manner: they seem to work
+ on actual hardware but they make assumptions that are not guaranteed by
+ the Arm architecture. Therefore, they might fail on some other platforms.
+
+- PSCI stress tests are very unreliable and will often hang. The root cause is
+ not known for sure but this might be due to bad synchronization between CPUs.
+
+- The GICv2 spurious IRQ test is Juno-specific. In reality, it should only be
+ GICv2-specific. It should be reworked to remove any platform-specific
+ assumption.
+
+- The GICv2 spurious IRQ test sometimes fails with the following error message:
+
+ ``SMC @ lead CPU returned 0xFFFFFFFF 0x8 0xC``
+
+ The root cause is unknown.
+
+- The manual tests in AArch32 mode do not work properly. They save some state
+ information into non-volatile memory in order to detect the reset reason but
+ this state does not appear to be retained. As a result, these tests keep
+ resetting infinitely.
+
+- The FWU tests take a long time to complete. This is because they wait for the
+ watchdog to reset the system. On FVP, TF-A configures the watchdog period to
+ about 4 min. This is way too long in an automated testing context. Besides,
+ the user gets not feedback, which may let them think that the tests are not
+ working properly.
+
+- The test "Target timer to a power down cpu" sometimes fails with the
+ following error message:
+
+ ``Expected timer switch: 4 Actual: 3``
+
+ The root cause is unknown.
+
+FWU images
+``````````
+
+- NS-BL1U and NS-BL2U images reuse TFTF-specific code for legacy reasons. This
+ is not a clean design and may cause confusion.
+
+Cactus test secure partition
+````````````````````````````
+
+- Cactus is experimental code. It's likely to change a lot as the secure
+ partition software architecture evolves.
+
+- Fails to link with GNU toolchain 7.3.1.
+
+- Cactus is supported on AArch64 FVP platform only.
+
+All test images
+```````````````
+
+- TF-A Tests are derived from a fork of TF-A so:
+
+ - they've got some code in common but lag behind on some features.
+
+ - there might still be some irrelevant references to TF-A.
+
+- Some design issues.
+ E.g. TF-A Tests inherited from the I/O layer of TF-A, which still needs a
+ major rework.
+
+- Cannot build TF-A Tests with Clang. Only GCC is supported.
+
+- The build system does not cope well with parallel building. The user should
+ not attempt to run multiple jobs in parallel with the ``-j`` option of `GNU
+ make`.
+
+- The build system does not properly track build options. A clean build must be
+ performed every time a build option changes.
+
+- SMCCC v2 is not properly supported.
+
+- UUIDs are not compliant to RFC 4122.
+
+- No floating point support. The code is compiled with GCC flag
+ ``-mgeneral-regs-only``, which prevents the compiler from generating code
+ that accesses floating point registers. This might limit some test scenarios.
+
+- The documentation is too lightweight.
+
+--------------
+
+*Copyright (c) 2018, Arm Limited. All rights reserved.*