Update TF-M CI job description
This patch updates TF-M CI job description to introduce the detailed
workflows of different jobs. And add instruction on how to read CI job
logs and artifacts.
Signed-off-by: Xinyu Zhang <xinyu.zhang@arm.com>
Change-Id: Ia10ce1df9df5178fbd4e39276749f5917a051257
diff --git a/docs/images/Per-patch-diagram.png b/docs/images/Per-patch-diagram.png
deleted file mode 100644
index 41db788..0000000
--- a/docs/images/Per-patch-diagram.png
+++ /dev/null
Binary files differ
diff --git a/docs/images/TFM-images/TFM-FVP-test.png b/docs/images/TFM-images/TFM-FVP-test.png
new file mode 100644
index 0000000..ea131c3
--- /dev/null
+++ b/docs/images/TFM-images/TFM-FVP-test.png
Binary files differ
diff --git a/docs/images/TFM-images/TFM-build-config-log.png b/docs/images/TFM-images/TFM-build-config-log.png
new file mode 100644
index 0000000..e652c0e
--- /dev/null
+++ b/docs/images/TFM-images/TFM-build-config-log.png
Binary files differ
diff --git a/docs/images/TFM-images/TFM-cc-workflow.png b/docs/images/TFM-images/TFM-cc-workflow.png
new file mode 100644
index 0000000..0a512a8
--- /dev/null
+++ b/docs/images/TFM-images/TFM-cc-workflow.png
Binary files differ
diff --git a/docs/images/TFM-images/TFM-console-output.png b/docs/images/TFM-images/TFM-console-output.png
new file mode 100644
index 0000000..17686ce
--- /dev/null
+++ b/docs/images/TFM-images/TFM-console-output.png
Binary files differ
diff --git a/docs/images/TFM-images/TFM-extra-build-workflow.png b/docs/images/TFM-images/TFM-extra-build-workflow.png
new file mode 100644
index 0000000..64dac0d
--- /dev/null
+++ b/docs/images/TFM-images/TFM-extra-build-workflow.png
Binary files differ
diff --git a/docs/images/TFM-images/TFM-failure-log.png b/docs/images/TFM-images/TFM-failure-log.png
new file mode 100644
index 0000000..a5dbcec
--- /dev/null
+++ b/docs/images/TFM-images/TFM-failure-log.png
Binary files differ
diff --git a/docs/images/TFM-images/TFM-nightly-workflow.png b/docs/images/TFM-images/TFM-nightly-workflow.png
new file mode 100644
index 0000000..2d155c0
--- /dev/null
+++ b/docs/images/TFM-images/TFM-nightly-workflow.png
Binary files differ
diff --git a/docs/images/TFM-images/TFM-per-patch-vote.png b/docs/images/TFM-images/TFM-per-patch-vote.png
new file mode 100644
index 0000000..3980d3b
--- /dev/null
+++ b/docs/images/TFM-images/TFM-per-patch-vote.png
Binary files differ
diff --git a/docs/images/TFM-images/TFM-per-patch-workflow.png b/docs/images/TFM-images/TFM-per-patch-workflow.png
new file mode 100644
index 0000000..82cdcf9
--- /dev/null
+++ b/docs/images/TFM-images/TFM-per-patch-workflow.png
Binary files differ
diff --git a/docs/images/TFM-images/TFM-release-workflow.png b/docs/images/TFM-images/TFM-release-workflow.png
new file mode 100644
index 0000000..3c4cf23
--- /dev/null
+++ b/docs/images/TFM-images/TFM-release-workflow.png
Binary files differ
diff --git a/docs/images/TFM-images/TFM-result-summary.png b/docs/images/TFM-images/TFM-result-summary.png
new file mode 100644
index 0000000..e2e7cbd
--- /dev/null
+++ b/docs/images/TFM-images/TFM-result-summary.png
Binary files differ
diff --git a/docs/images/TFM-images/TFM-toolchain-version.png b/docs/images/TFM-images/TFM-toolchain-version.png
new file mode 100644
index 0000000..906f385
--- /dev/null
+++ b/docs/images/TFM-images/TFM-toolchain-version.png
Binary files differ
diff --git a/docs/images/tf-m-cppcheck.png b/docs/images/tf-m-cppcheck.png
deleted file mode 100644
index 1892004..0000000
--- a/docs/images/tf-m-cppcheck.png
+++ /dev/null
Binary files differ
diff --git a/docs/images/tf-m-nightly.png b/docs/images/tf-m-nightly.png
deleted file mode 100644
index 9bd4838..0000000
--- a/docs/images/tf-m-nightly.png
+++ /dev/null
Binary files differ
diff --git a/docs/index.md b/docs/index.md
index bc58092..8bb5c87 100644
--- a/docs/index.md
+++ b/docs/index.md
@@ -294,48 +294,106 @@
## TF-M CI pipeline description
-TF-M jobs are found at https://ci.trustedfirmware.org/ and can be classified depending on the code coverage
+### Job classification
+TF-M jobs can be found at https://ci.trustedfirmware.org/view/TF-M/ and classified by use cases.
-* Release job: active during release stage, manually triggered. XL size
-* Nightly job: active everyday to cover the latest HEAD; in case of failure, notification is done through the tf-m mailing list. M size
-* Per-patch job: gerrit patch verification before merge. S size
+#### Per-patch job
-Below is a diagram that shows their relationship and the amount of code coverage targeted.
+Per-patch job is triggered by option *Allow-CI* in the _Reply_ button of each Gerrit review item to verify TF-M patches before merge. The job votes to Gerrit with a Code Review +1 if it passed and Code Review -1 once failed.
-
+<div style="text-align: center"><img src="images/TFM-images/TFM-per-patch-workflow.png" alt="alt text" title="Per-patch workflow" width=100%/><div style="text-align: left">
-Jobs can also be classified depending on their specific task:
+Following checks are covered in per-patch job:
+* __Static check__: Include tf-m-cppcheck (using [CppCheck](https://cppcheck.sourceforge.io/)), tf-m-checkpatch (using [CheckPatch](https://github.com/torvalds/linux/blob/v5.9/scripts/checkpatch.pl)) and tf-m-static-checks (checking trailing spaces, etc.), which are three independent jobs.
+* __Build configs__: Build TF-M images with tf-m-build-config jobs in parallel. Each job builds one config. Configs built by per-patch job are listed [here](https://ci.trustedfirmware.org/view/TF-M/job/tf-m-build-and-test/lastSuccessfulBuild/artifact/build_links.html).
+* __Build docs__: Build TF-M [user guide](https://ci.trustedfirmware.org/job/tf-m-build-docs/lastSuccessfulBuild/artifact/trusted-firmware-m/build/docs/user_guide/html/index.html) and [reference manual](https://ci.trustedfirmware.org/job/tf-m-build-docs/lastSuccessfulBuild/artifact/trusted-firmware-m/build/docs/reference_manual/html/index.html) based on the current patch.
+* __Test images__: Run TF-M images on target devices which are managed on [LAVA server](https://tf.validation.linaro.org/). Tf-m-build-config job will trigger tf-m-lava-submit job to start a LAVA test on the supported device, to verify the image built by tf-m-build-config job. Tests on following devices are covered in per-patch job: AN519 FVP, AN521 FVP, AN552 FVP, Corstone1000 FVP, STM32L562E_DK and Cypress/PSOC64.
-* Production jobs
- * tf-m-builds-docs-nightly
- * tf-m-build-and-test
- * tf-m-coverity
- * tf-m-static-checks
- * tf-m-nightly (scheduled)
- * tf-m-static (per-patch)
- * tf-m-build-docs
- * tf-m-build-config
- * tf-m-lava-submit
- * tf-m-cppcheck
- * tf-m-checkpatch
-* Release jobs
- * tf-m-release (release)
- * tf-m-code-coverage
-* Infra jobs
- * tf-m-infra-health
- * tf-m-build-config-infra-health
+Every check job generates a vote to Gerrit with corresponding job link. There wil be a final __Verified +1__ from _TrustedFirmware Code Review_ once all checks passed. Otherwise a __Verfied -1__ will be voted.
-### TF-M Job dependencies
+<div style="text-align: center"><img src="images/TFM-images/TFM-per-patch-vote.png" alt="alt text" title="Per-patch vote +1 to Gerrit" width=65%/><div style="text-align: left">
-When a patch arrives at https://review.trustedfirmware.org/ and reviewed, a maintainer may allow the CI to be executed, which in turn triggers the `tf-m-static` job. This is exactly the same CI workflow as for TF-A. In case of failure, the patch cannot be merged into the stable branch. The tf-m-static triggers many more jobs as seen in the picture below
+#### Nightly job
-
+Nightly job is daily scheduled to check the health of latest HEAD on TF-M master branch. Email notification will be sent to [tf-m-ci-notifications](https://lists.trustedfirmware.org/mailman3/lists/tf-m-ci-notifications.lists.trustedfirmware.org/) mailing list once the job failed. [TF-M user guide](https://tf-m-user-guide.trustedfirmware.org/) posted to [trustedfirmware.org](https://www.trustedfirmware.org/) is also daily generated by tf-m-build-docs-nightly in nightly job.
-The job tf-m-nigthly is a more extensive job, triggered everyday and tests the latest code (HEAD) at the project
+<div style="text-align: center"><img src="images/TFM-images/TFM-nightly-workflow.png" alt="alt text" title="Nighty workflow" width=100%/><div style="text-align: left">
-
+Following checks are covered in nightly job:
+* __Build configs__: Configs built by nightly job are listed [here](https://ci.trustedfirmware.org/view/TF-M/job/tf-m-nightly/lastSuccessfulBuild/artifact/build_links.html).
+* __Build docs__: Build TF-M [user guide](https://tf-m-user-guide.trustedfirmware.org/) and [reference manual](https://ci.trustedfirmware.org/job/tf-m-build-docs-nightly/lastSuccessfulBuild/artifact/trusted-firmware-m/build/docs/reference_manual/html/index.html) based on the latest TF-M master branch and posted to [trustedfirmware.org](https://www.trustedfirmware.org/).
+* __Test images__: Tests on following devices are covered in nightly job: AN519 FVP, AN521 FVP, AN521 FPGA, AN521 QEMU, AN552 FVP, Corstone1000 FVP, STM32L562E_DK, NXP/lpcxpresso55s69 and Cypress/PSOC64.
-In case the nightly job fails, an email notification is sent through the mailing list https://lists.trustedfirmware.org/mailman/listinfo/tf-m-ci-notifications . The maintainer is responsible for looking at the failed errors and identifying the (commit) culprit then reporting it to the developer.
+#### Release job
+
+Release job is scheduled bi-weekly to make sure the latest TF-M on master branch could pass all release tests. It will also be manually triggered for release branch during release stage.
+
+<div style="text-align: center"><img src="images/TFM-images/TFM-release-workflow.png" alt="alt text" title="Release workflow" width=100%/><div style="text-align: left">
+
+Following checks are covered in release job:
+* __Build configs__: Configs built by release job are listed [here](https://ci.trustedfirmware.org/view/TF-M/job/tf-m-release/lastSuccessfulBuild/artifact/build_links.html).
+* __Build docs__: Build TF-M [user guide](https://ci.trustedfirmware.org/job/tf-m-build-docs/lastSuccessfulBuild/artifact/trusted-firmware-m/build/docs/user_guide/html/index.html) and [reference manual](https://ci.trustedfirmware.org/job/tf-m-build-docs/lastSuccessfulBuild/artifact/trusted-firmware-m/build/docs/reference_manual/html/index.html) based on the current HEAD of master/release branch.
+* __Test images__: Tests on following devices are covered in release job: AN519 FVP, AN521 FVP, AN521 FPGA, AN521 QEMU, AN552 FVP, Corstone1000 FVP, STM32L562E_DK, NXP/lpcxpresso55s69 and Cypress/PSOC64.
+
+#### Code coverage job
+
+Code coverage job is scheduled weekly to monitor the code coverage of tests in the latest TF-M on master branch.
+
+<div style="text-align: center"><img src="images/TFM-images/TFM-cc-workflow.png" alt="alt text" title="Code coverage workflow" width=100%/><div style="text-align: left">
+
+Following configs are covered in code coverage job:
+* __Build configs__: Configs built by code coverage job are listed [here](https://ci.trustedfirmware.org/job/tf-m-code-coverage/lastSuccessfulBuild/artifact/build_links.html).
+* __Test images__: Tests on AN521 FVP will be traced by [qa-tool](https://git.trustedfirmware.org/ci/qa-tools.git) to get the code coverage data.
+
+The final code coverage data is available in the job artifacts at [merged_report/index.html](https://ci.trustedfirmware.org/job/tf-m-code-coverage/lastSuccessfulBuild/artifact/merged_report/index.html)
+
+#### Extra build job
+
+Extra build job is daily scheduled to check whether the latest TF-M could be correctly built for platforms supported by TF-M. Images are only built but not tested in this job. Configs built by extra build job are listed [here](https://ci.trustedfirmware.org/view/TF-M/job/tf-m-extra-build/lastSuccessfulBuild/artifact/build_links.html).
+
+<div style="text-align: center"><img src="images/TFM-images/TFM-extra-build-workflow.png" alt="alt text" title="Extra build workflow" width=50%/><div style="text-align: left">
+
+### Job logs
+
+For detailed logs, please go to the job link and choose __Console Output__ at the left column of the page.
+
+<div style="text-align: center"><img src="images/TFM-images/TFM-console-output.png" alt="alt text" title="Console Output" width=30%/><div style="text-align: left">
+
+And then search for the key word needs to be checked.
+
+For example, search for "fail" to see the error cause.
+
+<div style="text-align: center"><img src="images/TFM-images/TFM-failure-log.png" alt="alt text" title="Failure Log" width=80%/><div style="text-align: left">
+
+Or seach for config name that needs to be checked.
+
+<div style="text-align: center"><img src="images/TFM-images/TFM-build-config-log.png" alt="alt text" title="Build Config Log" width=90%/><div style="text-align: left">
+
+* __NOTE__: If LAVA test is run on FVP, LAVA link is only accessible to authorized users. Please check __TFM LOG__ to find the test result. If access to LAVA job is necessary, please contact tf-openci@lists.trustedfirmware.org for help.
+
+<div style="text-align: center"><img src="images/TFM-images/TFM-FVP-test.png" alt="alt text" title="FVP test" width=60%/><div style="text-align: left">
+
+### Job artifacts
+
+Artifacts archived by each job can be found in __Build Artifacts__.
+
+* __Build binary__: TF-M build binary of each config is available in corresponding tf-m-build-config job.
+* __Result summary__: Summary of all build and test results in both csv and html formats are available in artifacts of tf-m-build-and-test, tf-m-nightly and tf-m-release correspondingly for per-patch, nightly and release jobs.
+
+<div style="text-align: center"><img src="images/TFM-images/TFM-result-summary.png" alt="alt text" title="Result Summary" width=40%/><div style="text-align: left">
+
+### Build environment
+
+#### Toolchain
+
+There are different versions of toolchain installed in CI workspace. Each tf-m-build-config job selects the needed version automatically. To check the chosen toolchain, search for the name of toolchain (arm-none-eabi-gcc, armclang) in the job console.
+
+<div style="text-align: center"><img src="images/TFM-images/TFM-toolchain-version.png" alt="alt text" title="Toolchain Version" width=50%/><div style="text-align: left">
+
+#### Dependencies
+
+ * Dependency tools installed in TF-M CI workspace are listed [here](https://git.trustedfirmware.org/ci/dockerfiles.git/tree/bionic-amd64-tf-m-build/tf-dependencies.install).
+ * Python packages installed in TF-M CI workspace are listed [here](https://git.trustedfirmware.org/ci/dockerfiles.git/tree/bionic-amd64-tf-m-build/requirements_python3.txt).
## Code coverage support
