feat: use original Expect scripts for UART validation
To avoid having to implement custom Python post-expect scripts for every
payload that might use a non-primary UART, and to make use of as much
prior art as possible, this change replaces the OpenCI-only post-expect
scripts with the same one uses used by both the local and on-premises CI.
The original Expect scripts have been amended to identify whether they're
being run in the postprocessing step and, if so, load the UART log from a
file rather than connecting to the FVP over Telnet.
Change-Id: I61a5859de4f3f62b50deaffe84580206ec411c99
Signed-off-by: Chris Kay <chris.kay@arm.com>
diff --git a/expect-post/README.md b/expect-post/README.md
deleted file mode 100644
index 157231b..0000000
--- a/expect-post/README.md
+++ /dev/null
@@ -1,40 +0,0 @@
-# Post-expect scripts
-
-Post-expect scripts perform checking and validation on the captured logs
-of DUT interaction (usually, output from different UARTs). The original
-reason for introducing of this type of scripts was the fact that LAVA
-couldn't perform test matching on additional communication channels
-(e.g. other UARTs than UART0). Instead, it just records output from
-them to the logs. And post-expect scripts then can parse these logs
-and perform required validation.
-
-From the above description it should be clear that such tests are
-passive, i.e. can't inject any input into DUT. However, at the time
-of writing, TrustedFirmware worked in exactly this way: the primary
-UART could be connected to an interactive shell (Linux command line,
-etc.), where input can be accepted, while other UARTs are used for
-logging from different subsystems.
-
-Note that these tests are contained in the subdirectory named
-`expect-post`, to make that sort together with the original `expect`
-directory, to make all variants of expect scripts immediately visible.
-
-## Scripts format
-
-Post-expect scripts are just arbitrary executables (usually shell
-or Python scripts), which are passed the name of the log file they
-should check (which contain output from a particular UART). A script
-should exit with 0 status if the test was successful, and non-zero
-otherwise.
-
-Specific execution model is:
-
-For each artefacts/debug/uart<num>/run/expect file (where N is 1, 2,
-etc.) the expect script name is read from that file. In ArmCI, that
-script would be looked up in tf-a-ci-scripts/expect/ , but in
-OpenCI, it's instead looked up tf-a-ci-scripts/expect-post/ . Each
-test script is executed in Jenkins, after LAVA job finished execution,
-and its log was fetched (and split into per-UART output). A short
-summary of total number of executed post-expect scripts is printed at
-the end, together with number of failed tests. If any test is failed,
-Jenkins job is failing too.
diff --git a/expect-post/hold_uart.exp b/expect-post/hold_uart.exp
deleted file mode 100755
index 6035aaf..0000000
--- a/expect-post/hold_uart.exp
+++ /dev/null
@@ -1,8 +0,0 @@
-#!/bin/sh
-#
-# Copyright (c) 2021 Arm Limited. All rights reserved.
-#
-# SPDX-License-Identifier: BSD-3-Clause
-#
-
-# Do nothing
diff --git a/expect-post/spm-cactus-sp-uart1.exp b/expect-post/spm-cactus-sp-uart1.exp
deleted file mode 100755
index 1a657c3..0000000
--- a/expect-post/spm-cactus-sp-uart1.exp
+++ /dev/null
@@ -1,21 +0,0 @@
-#!/usr/bin/env python3
-#
-# Copyright (c) 2021, Linaro Limited
-#
-# SPDX-License-Identifier: BSD-3-Clause
-#
-
-import sys
-
-
-found = False
-
-with open(sys.argv[1]) as f:
- for l in f:
- if "Booting Secure Partition" in l:
- found = True
-
-if found:
- sys.exit(0)
-else:
- sys.exit(1)
diff --git a/expect-post/spm-edk2-uart2.exp b/expect-post/spm-edk2-uart2.exp
deleted file mode 100755
index 0a9168d..0000000
--- a/expect-post/spm-edk2-uart2.exp
+++ /dev/null
@@ -1,24 +0,0 @@
-#!/usr/bin/env python3
-#
-# Copyright (c) 2021, Linaro Limited
-#
-# SPDX-License-Identifier: BSD-3-Clause
-#
-# Expect script for Standalone MM partition UART2
-#
-
-import sys
-
-
-def expect(f, what):
- for l in f:
- if what in l:
- return True
- assert False, "'%s' not found in output" % what
-
-
-with open(sys.argv[1]) as f:
- expect(f, "SPM Version")
- expect(f, "MmMain Done!")
- expect(f, "Received event - 0xC4000041 on cpu")
- expect(f, "MmEntryPoint Done")
diff --git a/expect-post/spm-uart2.exp b/expect-post/spm-uart2.exp
deleted file mode 100755
index 1a657c3..0000000
--- a/expect-post/spm-uart2.exp
+++ /dev/null
@@ -1,21 +0,0 @@
-#!/usr/bin/env python3
-#
-# Copyright (c) 2021, Linaro Limited
-#
-# SPDX-License-Identifier: BSD-3-Clause
-#
-
-import sys
-
-
-found = False
-
-with open(sys.argv[1]) as f:
- for l in f:
- if "Booting Secure Partition" in l:
- found = True
-
-if found:
- sys.exit(0)
-else:
- sys.exit(1)
diff --git a/expect-post/tpm-logs.exp b/expect-post/tpm-logs.exp
deleted file mode 100755
index 17bf4a4..0000000
--- a/expect-post/tpm-logs.exp
+++ /dev/null
@@ -1,45 +0,0 @@
-#!/usr/bin/env python3
-#
-# Copyright (c) 2021, Linaro Limited
-# Copyright (c) 2022, Arm Limited. All rights reserved.
-#
-# SPDX-License-Identifier: BSD-3-Clause
-#
-
-import re
-import sys
-from subprocess import check_call
-
-
-PATTERN = "\
-Digest(\\s|\\w)*:\\s(\\w{2}\\s){16}|\
- : (\\w{2}\\s){16}|\
- Event\\w*\\s*:\\s\\w+\\s"
-
-
-# Extract pertinent information from TPM side.
-with open(sys.argv[1]) as f, open("ftpm_event_log", "w") as out:
- for l in f:
- m = re.search(PATTERN, l)
- if m:
- print(m.group(), file=out)
-
-# Extract pertinent information from TF side.
-with open(sys.argv[1].replace("uart1", "uart0")) as f, open("tfa_event_log", "w") as out:
- # Wait for the start of the event log dump.
- for l in f:
- if re.search("TCG_EfiSpecIDEvent:", l):
- break
-
- for l in f:
- # Look for the end of the event log dump.
- if re.search("Booting BL31", l):
- break
-
- # Capture relevant lines in between start and end strings
- m = re.search(PATTERN, l)
- if m:
- print(m.group(), file=out)
-
-# Compare it to match.
-check_call("diff -s -u tfa_event_log ftpm_event_log", shell=True)
diff --git a/expect-post/tsp.exp b/expect-post/tsp.exp
deleted file mode 100755
index 86e44f4..0000000
--- a/expect-post/tsp.exp
+++ /dev/null
@@ -1,22 +0,0 @@
-#!/usr/bin/env python
-#
-# Copyright (c) 2021, Linaro Limited
-#
-# SPDX-License-Identifier: BSD-3-Clause
-#
-
-import sys
-
-REQUIRED_NUM = 1000
-
-cnt = 0
-
-with open(sys.argv[1]) as f:
- for l in f:
- if "TSP: cpu" in l:
- cnt += 1
-
-if cnt >= REQUIRED_NUM:
- sys.exit(0)
-else:
- sys.exit(1)
diff --git a/expect/handle-arguments.inc b/expect/handle-arguments.inc
index f475302..6a730a5 100644
--- a/expect/handle-arguments.inc
+++ b/expect/handle-arguments.inc
@@ -1,5 +1,5 @@
#
-# Copyright (c) 2019-2020 Arm Limited. All rights reserved.
+# Copyright (c) 2019-2022, Arm Limited. All rights reserved.
#
# SPDX-License-Identifier: BSD-3-Clause
#
@@ -10,8 +10,14 @@
source [file join [file dirname [info script]] utils.inc]
# Store environment variables into local variables
-set uart_port [get_param uart_port]
-set timeout [get_param timeout]
+set timeout [get_param timeout 30]
-# Open a telnet connection on the required UART port
-set telnet_pid [spawn telnet localhost $uart_port]
+if { [postprocessing] != 1 } {
+ # Open a Telnet connection to the required UART port
+ set uart_port [get_param uart_port]
+ set telnet_pid [spawn telnet localhost $uart_port]
+} else {
+ # Read directly from the UART log file
+ set uart_log_file [get_param uart_log_file]
+ set telnet_pid [spawn cat $uart_log_file]
+}
diff --git a/expect/spm-optee-sp-uart1.exp b/expect/spm-optee-sp-uart1.exp
index c874551..6908609 100644
--- a/expect/spm-optee-sp-uart1.exp
+++ b/expect/spm-optee-sp-uart1.exp
@@ -1,5 +1,5 @@
#
-# Copyright (c) 2020, Arm Limited. All rights reserved.
+# Copyright (c) 2020-2022, Arm Limited. All rights reserved.
#
# SPDX-License-Identifier: BSD-3-Clause
#
@@ -7,7 +7,7 @@
source [file join [file dirname [info script]] handle-arguments.inc]
expect {
- "OP-TEE version: 3.3" {
+ "OP-TEE version: " {
puts "<<OP-TEE version>>"
}
timeout {
diff --git a/expect/utils.inc b/expect/utils.inc
index 0100534..3d5a7f5 100644
--- a/expect/utils.inc
+++ b/expect/utils.inc
@@ -1,8 +1,19 @@
#
-# Copyright (c) 2019-2020 Arm Limited. All rights reserved.
+# Copyright (c) 2019-2022, Arm Limited. All rights reserved.
#
# SPDX-License-Identifier: BSD-3-Clause
#
+
+# Determine whether the script is being run on a live FVP UART terminal or
+# postprocessing a UART log file (probably dumped by LAVA).
+proc postprocessing {} {
+ if { [info exists ::env(uart_log_file)] } {
+ return 1
+ } else {
+ return 0
+ }
+}
+
# Retrieve script parameters from environment variables. If they don't exist,
# return empty string
proc get_param {name {default ""}} {
@@ -14,10 +25,13 @@
}
proc exit_uart {status} {
- # Allow UART output to flush
- sleep 1
- send "\x1b"
- send "close\r"
+ if { [postprocessing] != 1 } {
+ # Allow UART output to flush
+ sleep 1
+ send "\x1b"
+ send "close\r"
+ }
+
exit $status
}
diff --git a/script/expect-post-runner.sh b/script/expect-post-runner.sh
index ab22562..ea9a3ca 100755
--- a/script/expect-post-runner.sh
+++ b/script/expect-post-runner.sh
@@ -1,6 +1,6 @@
#!/usr/bin/env bash
#
-# Copyright (c) 2021, Linaro Limited
+# Copyright (c) 2021-2022, Linaro Limited
#
# SPDX-License-Identifier: BSD-3-Clause
#
@@ -9,8 +9,6 @@
# plans prepare in artefacts/debug/run/. See expect-post/README.md for
# more info about post-expect scripts.
-set -e
-
if [ -z "$WORKSPACE" ]; then
echo "Error: WORKSPACE is not set. This script is intended to be run from Jenkins build. (Or suitably set up local env)."
exit 1
@@ -19,28 +17,49 @@
total=0
failed=0
+# TODO: move dependency installation to the Dockerfile
+sudo DEBIAN_FRONTEND=noninteractive apt update && \
+ sudo DEBIAN_FRONTEND=noninteractive TZ=Etc/UTC apt install -y expect ||
+ exit 1
+
for uartdir in $WORKSPACE/artefacts/debug/run/uart*; do
# In case no dirs exist and the glob above isn't expanded at all.
if [ ! -d "$uartdir" ]; then
break
fi
+ total=$((total + 1))
+
+ expscript_fragment=$(cat ${uartdir}/expect)
+ expscript=${WORKSPACE}/tf-a-ci-scripts/expect/${expscript_fragment}
+
+ if [ ! -f "${expscript}" ]; then
+ echo "expect/${expscript_fragment}: MISS"
+ failed=$((failed + 1))
+
+ continue
+ fi
+
uart=$(basename $uartdir)
- if [ $uart == "uart0" ]; then
- continue
- fi
- expscript=$(cat $uartdir/expect)
- if [ ! -f $WORKSPACE/tf-a-ci-scripts/expect-post/$expscript ]; then
- echo "expect-post/$expscript: MISS"
- continue
- fi
- if ! $WORKSPACE/tf-a-ci-scripts/expect-post/$expscript $WORKSPACE/lava-$uart.log; then
- echo "expect-post/$expscript($uart): FAIL"
+
+ (
+ if [ -f "${uartdir}/env" ]; then
+ set -a
+ source "${uartdir}/env"
+ set +a
+ fi
+
+ export uart_log_file="${WORKSPACE}/lava-${uart}.log"
+
+ expect "${expscript}"
+ )
+
+ if [ $? != 0 ]; then
+ echo "expect/${expscript_fragment}(${uart}): FAIL"
failed=$((failed + 1))
else
- echo "expect-post/$expscript($uart): pass"
+ echo "expect/${expscript_fragment}(${uart}): pass"
fi
- total=$((total + 1))
done
echo "Post expect scripts: total=$total failed=$failed"