The test runner can now repeat tests in two ways:
- Via --repeat to run tests multiple times and fail if any one of them
fails. Which will help us find flakes in post-submit.
- Via --retry, which will retry a test if it has failed. Which makes
pre-submit tests more resilient to flakes.
Both can be configured by builder configs to adjust as needed.
Also slighly changes the repeat behavior to repeat tests in batches, so
we do not run the same test multiple times simultaneously, which can
cause some of them to fail. It's also easier to read the results.
BUG=b:238232551
TEST=added a random flake into some tests and used --repeat and --retry
Change-Id: I78e0ff0751da40a99a56080d9baf207307c9e93a
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/3751835
Tested-by: Dennis Kempin <denniskempin@google.com>
Reviewed-by: Daniel Verkamp <dverkamp@chromium.org>
This will help us find flaky test. Runtime increase is not that
significant (~10min).
BUG=b:238232551
TEST=Builder config needs to be tested in luci
Change-Id: Ie75cc9620177115623500fe2206f33464f4c8580
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/3751831
Reviewed-by: Daniel Verkamp <dverkamp@chromium.org>
Tested-by: kokoro <noreply+kokoro@google.com>
Then 'builder' cache dir is set up by default to be a cache directory
for each builder, so luci will try to schedule a builder to run on
the same bot as before.
We can optimize that by sharing the cache among all linux builders.
BUG=b:238333690
TEST=lucicfg validate main.star
Change-Id: If4639658271881f1bf1ad001ae54f50645e8890a
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/3751828
Reviewed-by: Daniel Verkamp <dverkamp@chromium.org>
Tested-by: kokoro <noreply+kokoro@google.com>
An early return snuck in during a rebase.
BUG=b:233230027
TEST=./recipes.py test run
Change-Id: I152281dd55148ca56fc7300a66b6bdcc1eef0ada
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/3751830
Reviewed-by: Daniel Verkamp <dverkamp@chromium.org>
Tested-by: kokoro <noreply+kokoro@google.com>
- Use vpython to call crovm tooling (This provides a predictable python
environment and allow us to use argh).
- Add build_context() to provide a simple API that readies source and
container.
- Nest preparation steps to reduce noise on builder page.
BUG=b:233913643
TEST=recipes.py test run
Change-Id: I6800e55f7311b32c6ef61918d13795af1fb3b588
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/3674217
Reviewed-by: Daniel Verkamp <dverkamp@chromium.org>
Tested-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: Anton Romanov <romanton@google.com>
This allows us to run multiple steps with the same intermediate
build data.
To make sure we start with a clean slate, any existing containers are
deleted at the beginning of each recipe.
BUG=b:233913455
TEST=./recipes.py test run
Change-Id: Ieda684d40bf08937183cba34242360c025724af8
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/3668813
Tested-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: Daniel Verkamp <dverkamp@chromium.org>
The builders use the existing dev_container to run tests
for the 3 currently supported architectures on linux:
x86_64, aarch64 and armhf.
A new recipe 'build_linux' has been added runs the
dev_container to build and execute tests.
To share code with other upcoming recipes, some of the
recipe code has been extracted into a shared crosvm
module.
Basic tests for the recipes have been added.
BUG=b:233230344
TEST=./recipes.py test run
./recipes.py run build_linux
Change-Id: I17ecb25c0a0eabdce56537831454ac22d4dc7021
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/3654196
Tested-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: Daniel Verkamp <dverkamp@chromium.org>
