Document how to interpret negative reports
The abi_check script has common false positives. Document the intent of each
family of checks and typical cases of false positives that can be overridden.
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
diff --git a/scripts/abi_check.py b/scripts/abi_check.py
index 0a80d98..c228843 100755
--- a/scripts/abi_check.py
+++ b/scripts/abi_check.py
@@ -3,6 +3,8 @@
for backward incompatibilities between two different Git revisions within
an Mbed TLS repository. It must be run from the root of a Git working tree.
+### How the script works ###
+
For the source (API) and runtime (ABI) interface compatibility, this script
is a small wrapper around the abi-compliance-checker and abi-dumper tools,
applying them to compare the header and library files.
@@ -19,6 +21,66 @@
Returns 0 on success, 1 on non-compliance, and 2 if there is an error
while running the script.
+### How to interpret non-compliance ###
+
+This script has relatively common false positives. In many scenarios, it only
+reports a pass if there is a strict textual match between the old version and
+the new version, and it reports problems where there is a sufficient semantic
+match but not a textual match. This section lists some common false positives.
+This is not an exhaustive list: in the end what matters is whether we are
+breaking a backward compatibility goal.
+
+**API**: the goal is that if an application works with the old version of the
+library, it can be recompiled against the new version and will still work.
+This is normally validated by comparing the declarations in `include/*/*.h`.
+A failure is a declaration that has disappeared or that now has a different
+type.
+
+ * It's ok to change or remove macros and functions that are documented as
+ for internal use only or as experimental.
+ * It's ok to rename function or macro parameters as long as the semantics
+ has not changed.
+ * It's ok to change or remove structure fields that are documented as
+ private.
+ * It's ok to add fields to a structure that already had private fields
+ or was documented as extensible.
+
+**ABI**: the goal is that if an application was built against the old version
+of the library, the same binary will work when linked against the new version.
+This is normally validated by comparing the symbols exported by `libmbed*.so`.
+A failure is a symbol that is no longer exported by the same library or that
+now has a different type.
+
+ * All ABI changes are acceptable if the library version is bumped
+ (see `scripts/bump_version.sh`).
+ * ABI changes that concern functions which are declared only inside the
+ library directory, and not in `include/*/*.h`, are acceptable only if
+ the function was only ever used inside the same library (libmbedcrypto,
+ libmbedx509, libmbedtls). As a counter example, if the old version
+ of libmbedtls calls mbedtls_foo() from libmbedcrypto, and the new version
+ of libmbedcrypto no longer has a compatible mbedtls_foo(), this does
+ require a version bump for libmbedcrypto.
+
+**Storage format**: the goal is to check that persistent keys stored by the
+old version can be read by the new version. This is normally validated by
+comparing the `*read*` test cases in `test_suite*storage_format*.data`.
+A failure is a storage read test case that is no longer present with the same
+function name and parameter list.
+
+ * It's ok if the same test data is present, but its presentation has changed,
+ for example if a test function is renamed or has different parameters.
+ * It's ok if redundant tests are removed.
+
+**Generated test coverage**: the goal is to check that automatically
+generated tests have as much coverage as before. This is normally validated
+by comparing the test cases that are automatically generated by a script.
+A failure is a generated test case that is no longer present with the same
+function name and parameter list.
+
+ * It's ok if the same test data is present, but its presentation has changed,
+ for example if a test function is renamed or has different parameters.
+ * It's ok if redundant tests are removed.
+
"""
# Copyright The Mbed TLS Contributors