lava-expect: enable LAVA expect support
This is the first patch of a series to enable 'expect' in the Open CI
framework [1]. Under the Open CI enviroment, the expect support is done
in LAVA and not using `expect` tool, through LAVA Interactive Actions
[2], where expect strings are transformed into one or more LAVA
Interactive Actions, matching either successes or failures
strings. All dynamically created LAVA content is finally located in
the LAVA job definition, where ultimatelly, LAVA matches the
corresponding strings.
[1] https://projects.linaro.org/browse/TFC-36
[2] https://validation.linaro.org/static/docs/v2/actions-test.html?highlight=interactive#interactive-test-action
Signed-off-by: Leonardo Sandoval <leonardo.sandoval@linaro.org>
Change-Id: Iff454bd3714a38a729022a98a862ea08e8f440fc
diff --git a/fvp_utils.sh b/fvp_utils.sh
index fb9cc2d..5545a95 100644
--- a/fvp_utils.sh
+++ b/fvp_utils.sh
@@ -609,6 +609,9 @@
done < "$lava_model_params"
sed -i -e '/{BOOT_ARGUMENTS}/d' "$yaml_file"
+ # Append expect commands into the job definition through test-interactive commands
+ gen_fvp_yaml_expect >> "$yaml_file"
+
# create job.yaml
cp "$yaml_file" "$yaml_job_file"
@@ -617,6 +620,133 @@
archive_file "$yaml_job_file"
}
+gen_fvp_yaml_expect() {
+ # Loop through all uarts expect files
+ run_root="$archive/run"
+ for expect_file in $(find $run_root -name expect); do
+
+ # TODO: currently, only handle UART 0
+ case $expect_file in
+ *uart0* )
+ uart_number=uart0
+ ;;
+ *)
+ continue
+ ;;
+ esac
+
+ # Array containing expect strings and populated during run config execution
+ expect_string=()
+
+ # Get the real name of the expect file
+ expect_file=$(cat $expect_file)
+
+ # Source the run_config enviroment variables
+ env=$run_root/$uart_number/env
+ if [ -e $env ]; then
+ source $env
+ fi
+
+ # Get all expect strings
+ expect_file=$ci_root/lava-expect/${expect_file}
+ source $expect_file
+
+ # Append the corresponding test-interactive commands
+ if [ ${#expect_string[@]} -gt 0 ]; then
+
+ cat << EOF
+- test:
+ timeout:
+ minutes: 15
+ interactive:
+EOF
+ for key in "${!expect_string[@]}"; do
+
+ # Expect strings have the format '<prompt>;<successes>;<failures>', so use these
+ # arrays to store them separately
+ prompts=()
+ successes=()
+ failures=()
+ commands=()
+
+ # Parse expect string values and split into the arrays
+ for es in "${expect_string[@]}"; do
+ prompts+=("$(echo "${es}" | awk -F ';' '{print $1}')")
+ successes+=("$(echo "${es}" | awk -F ';' '{print $2}')")
+ failures+=("$(echo "${es}" | awk -F ';' '{print $3}')")
+ commands+=("$(echo "${es}" | awk -F ';' '{print $4}')")
+ done
+
+ OLD_IFS=$IFS; IFS=$'@'
+
+ if [[ -n "${prompts[$key]}" && -n "${successes[$key]}" && -n "${failures[$key]}" ]]; then
+ cat << EOF
+ - name: ${uart_number}_${key}
+ prompts: ['${prompts[$key]}']
+ script:
+ - name: result
+ command: ${commands[$key]}
+EOF
+ cat << EOF
+ successes:
+EOF
+ for s in ${successes[$key]}; do
+ cat <<EOF
+ - message: '$s'
+EOF
+ done
+ cat << EOF
+ failures:
+EOF
+ for f in ${failures[$key]}; do
+ cat <<EOF
+ - message: '$f'
+EOF
+ done
+ elif [[ -n "${prompts[$key]}" && -n "${successes[$key]}" ]]; then
+ cat << EOF
+ - name: ${uart_number}_${key}
+ prompts: ['${prompts[$key]}']
+ script:
+ - name: result
+ command: ${commands[$key]}
+ successes:
+EOF
+ for s in ${successes[$key]}; do
+ cat <<EOF
+ - message: '$s'
+EOF
+ done
+ elif [[ -n "${prompts[$key]}" && -n "${failures[$key]}" ]]; then
+ cat << EOF
+ - name: ${uart_number}_${key}
+ prompts: ['${prompts[$key]}']
+ script:
+ - name: result
+ command: ${commands[$key]}
+ failures:
+EOF
+ for f in ${failures[$key]}; do
+ cat <<EOF
+ - message: '$f'
+EOF
+ done
+
+ else
+ cat << EOF
+ - name: ${uart_number}_${key}
+ prompts: ['${prompts[$key]}']
+ script:
+ - name: result
+ command: ${commands[$key]}
+EOF
+ fi
+ IFS=$OLD_IFS
+ done
+ fi
+ done
+}
+
docker_registry_append() {
# if docker_registry is empty, just use local docker registry
[ -z "$docker_registry" ] && return
diff --git a/lava-expect/README.md b/lava-expect/README.md
new file mode 100644
index 0000000..4aad2f9
--- /dev/null
+++ b/lava-expect/README.md
@@ -0,0 +1,67 @@
+# Expect Scripts
+
+The project tracks two set of expect scripts under two different folders, `expect` and
+`lava-expect`, the former required for local (non-LAVA) or Internal CI (Arm CI) and
+the latter for Open CI (LAVA). Note that any contribution into the expect scripts
+**must be done in both folders**, otherwise expect test coverage will differ.
+
+# LAVA Expect Scripts
+
+The `lava-expect` script does exactly the same as its counterpart under the `expect`
+folder. However, LAVA uses [Test Interactive Actions](https://validation.linaro.org/static/docs/v2/actions-test.html#interactive-test-action)
+where it expects success or failure strings. The CI would take these `lava-expect`
+scripts and converted into LAVA Test Interactive Actions, so the only task on these files is to define either successes
+or failures strings and probably the order of appearance.
+
+To better understand how `expect` scripts translates into `lava-expect`, we can compare similar
+scripts, i.e. `expect/cactus.exp` vesus `lava-expect/cactus.exp`. Let's compare these two:
+
+* `expect/cactus.exp`
+
+```
+.
+.
+source [file join [file dirname [info script]] handle-arguments.inc]
+
+expect_string "Booting test Secure Partition Cactus"
+
+source [file join [file dirname [info script]] uart-hold.inc]
+```
+
+* and its counterpart `lava-expect/cactus.exp` (note, the same filename but different folder)
+
+```
+.
+.
+expect_string+=("Booting test Secure Partition Cactus")
+
+source $ci_root/lava-expect/uart-hold.inc
+```
+
+As seen, the same *expect string* appears in both, but in case
+of `lava-expect/cactus.exp`, which is written in bash, the variable to be
+used is `expect_string` and its *expect string* **must be** appended (`+=`) as an
+**array element**. Appending is important, otherwise previous *expect strings* would be
+replaced by the assigment (`=`).
+
+In case we want to indicate the **order** of expected scripts and the **pass/fail
+criteria**, the *expect string* syntax is a bit more complex but with a few examples,
+it can be quickly understood.
+
+For example, in an hypothetical scenario, we want to first match A, then after A,
+match B or C for success and match D or E for failures. This would be the `lava-expect`
+definition script (to be located under `lava-expect` folder)
+
+```
+prompt='A'
+successes='B@C'
+failures='D@E'
+expect_string+=("${prompt};${successes};${failures}")
+```
+
+As you can see, `prompt` defines the first match, in this case `A`, then `successes` defines
+possible success strings, in this case `B` and `C`, separated by `@` sign, and the same
+pattern applies for failure strings, but in this case, matching these would tell LAVA
+to fail the job.
+
+For a real examples, compare similar scripts (the same filename) in both expect folders.
diff --git a/script/gen_fvp_linux_yaml.sh b/script/gen_fvp_linux_yaml.sh
index cbd8d72..9d85cdb 100755
--- a/script/gen_fvp_linux_yaml.sh
+++ b/script/gen_fvp_linux_yaml.sh
@@ -126,7 +126,4 @@
minutes: 30
arguments:
{BOOT_ARGUMENTS}
- prompts:
- - '{PROMPT1}'
- - '{PROMPT2}'
EOF
diff --git a/script/gen_fvp_tftf_yaml.sh b/script/gen_fvp_tftf_yaml.sh
index 99295f9..72b4d51 100755
--- a/script/gen_fvp_tftf_yaml.sh
+++ b/script/gen_fvp_tftf_yaml.sh
@@ -144,24 +144,4 @@
arguments:
{BOOT_ARGUMENTS}
-- test:
-
- monitors:
- - name: TFTF
- # LAVA looks for a testsuite start string...
- start: 'Booting trusted firmware test framework'
- # ...and a testsuite end string.
- end: 'Exiting tests.'
-
- # For each test case, LAVA looks for a string which includes the testcase
- # name and result.
- pattern: "(?s)> Executing '(?P<test_case_id>.+?(?='))'(.*) TEST COMPLETE\\\s+(?P<result>(Skipped|Passed|Failed|Crashed))"
-
- # Teach to LAVA how to interpret the TFTF Tests results.
- fixupdict:
- Passed: pass
- Failed: fail
- Crashed: fail
- Skipped: skip
-
EOF