expect-post: Introduce post-expect scripts to match logs post-factum
Currently LAVA cannot peform test matching on any UARTs but the main
one (UART0). However, existing expect scripts for other UARTs are
simple and just passively parse the output. So, it's possible to
just capture the UART output to a log file (what LAVA already does)
and then parse and analyze this log file. This is exactly the idea
behind post-expect scripts: where e.g. expect/tsp.exp would be run
by ArmCI (in parallel with other expect scripts), OpenCI will run
expect-lava/tsp.exp - after LAVA tests finished and logs are captured.
Currently, post-expect scripts are intended to handle all UARTs but
UART0 (UART0 is handled by lava-expect/ instead).
The directory is named `expect-post` to sort together with `expect`
and so relationship between them is clearer.
Signed-off-by: Paul Sokolovsky <paul.sokolovsky@linaro.org>
Change-Id: I8659d2e4740dcd8050bf72c035bf31481bd69b16
diff --git a/expect-post/README.md b/expect-post/README.md
new file mode 100644
index 0000000..157231b
--- /dev/null
+++ b/expect-post/README.md
@@ -0,0 +1,40 @@
+# Post-expect scripts
+
+Post-expect scripts perform checking and validation on the captured logs
+of DUT interaction (usually, output from different UARTs). The original
+reason for introducing of this type of scripts was the fact that LAVA
+couldn't perform test matching on additional communication channels
+(e.g. other UARTs than UART0). Instead, it just records output from
+them to the logs. And post-expect scripts then can parse these logs
+and perform required validation.
+
+From the above description it should be clear that such tests are
+passive, i.e. can't inject any input into DUT. However, at the time
+of writing, TrustedFirmware worked in exactly this way: the primary
+UART could be connected to an interactive shell (Linux command line,
+etc.), where input can be accepted, while other UARTs are used for
+logging from different subsystems.
+
+Note that these tests are contained in the subdirectory named
+`expect-post`, to make that sort together with the original `expect`
+directory, to make all variants of expect scripts immediately visible.
+
+## Scripts format
+
+Post-expect scripts are just arbitrary executables (usually shell
+or Python scripts), which are passed the name of the log file they
+should check (which contain output from a particular UART). A script
+should exit with 0 status if the test was successful, and non-zero
+otherwise.
+
+Specific execution model is:
+
+For each artefacts/debug/uart<num>/run/expect file (where N is 1, 2,
+etc.) the expect script name is read from that file. In ArmCI, that
+script would be looked up in tf-a-ci-scripts/expect/ , but in
+OpenCI, it's instead looked up tf-a-ci-scripts/expect-post/ . Each
+test script is executed in Jenkins, after LAVA job finished execution,
+and its log was fetched (and split into per-UART output). A short
+summary of total number of executed post-expect scripts is printed at
+the end, together with number of failed tests. If any test is failed,
+Jenkins job is failing too.
diff --git a/expect-post/hold_uart.exp b/expect-post/hold_uart.exp
new file mode 100755
index 0000000..6035aaf
--- /dev/null
+++ b/expect-post/hold_uart.exp
@@ -0,0 +1,8 @@
+#!/bin/sh
+#
+# Copyright (c) 2021 Arm Limited. All rights reserved.
+#
+# SPDX-License-Identifier: BSD-3-Clause
+#
+
+# Do nothing
diff --git a/expect-post/tsp.exp b/expect-post/tsp.exp
new file mode 100755
index 0000000..3148a77
--- /dev/null
+++ b/expect-post/tsp.exp
@@ -0,0 +1,25 @@
+#!/usr/bin/env python
+#
+# Copyright (c) 2021, Linaro Limited
+#
+# SPDX-License-Identifier: BSD-3-Clause
+#
+
+import sys
+
+# FIXME: At the time of writing, due to STG-2668, logs for UARTs != UART0
+# are not complete, so match fewer lines.
+#REQUIRED_NUM = 1000
+REQUIRED_NUM = 30
+
+cnt = 0
+
+with open(sys.argv[1]) as f:
+ for l in f:
+ if "TSP: cpu" in l:
+ cnt += 1
+
+if cnt >= REQUIRED_NUM:
+ sys.exit(0)
+else:
+ sys.exit(1)
diff --git a/script/expect-post-runner.sh b/script/expect-post-runner.sh
new file mode 100755
index 0000000..c41851e
--- /dev/null
+++ b/script/expect-post-runner.sh
@@ -0,0 +1,45 @@
+#!/usr/bin/env bash
+#
+# Copyright (c) 2021, Linaro Limited
+#
+# SPDX-License-Identifier: BSD-3-Clause
+#
+# Runner for scripts in expect-post/ directory. This script is intended
+# to be run from Jenkins build, with $WORKSPACE set and per-UART test
+# plans prepare in artefacts/debug/run/. See expect-post/README.md for
+# more info about post-expect scripts.
+
+set -e
+
+if [ -z "$WORKSPACE" ]; then
+ echo "Error: WORKSPACE is not set. This script is intended to be run from Jenkins build. (Or suitably set up local env)."
+ exit 1
+fi
+
+total=0
+failed=0
+
+for uartdir in $WORKSPACE/artefacts/debug/run/uart*; do
+ uart=$(basename $uartdir)
+ if [ $uart == "uart0" ]; then
+ continue
+ fi
+ expscript=$(cat $uartdir/expect)
+ if [ ! -f $WORKSPACE/tf-a-ci-scripts/expect-post/$expscript ]; then
+ echo "expect-post/$expscript: MISS"
+ continue
+ fi
+ if ! $WORKSPACE/tf-a-ci-scripts/expect-post/$expscript $WORKSPACE/lava-$uart.log; then
+ echo "expect-post/$expscript($uart): FAIL"
+ failed=$((failed + 1))
+ else
+ echo "expect-post/$expscript($uart): pass"
+ fi
+ total=$((total + 1))
+done
+
+echo "Post expect scripts: total=$total failed=$failed"
+
+if [ $failed -gt 0 ]; then
+ exit 1
+fi