Changes how unit tests are executed. Instead of running all tests on
a selected target (i.e. host or a VM), we will select a platform to test
and run_tests will use a separate target for unit tests than for
integration tests.
e.g. instead of running all tests in the aarch64 vm:
./tools/run_tests --target=vm:aarch64
We can now run
./tools/run_tests --platform=aarch64
to run unit tests via user-space emulation on the host, and only
integration tests on the VM.
This will eventually apply to x86 testing as well, so tests can be
run in an unprivileged environment (e.g. podman).
A new TestOption has been added to mark unit tests that have system
interactions that require them to be run like integration tests in a VM.
These should be fixed.
You can also use --unit-tests and --integration-tests to run just one
type of tests. Running unit tests only takes a few seconds on most
platforms since we can use user-space emulation.
BUG=b:247139912
TEST=./tools/run_tests
./tools/run_tests -p aarch64
./tools/run_tests -p mingw64
./tools/run_tests -p armhf
Change-Id: Icd0c502623f1889906d199e752b3eccb7de76dc0
Reviewed-on: https://chromium-review.googlesource.com/c/crosvm/crosvm/+/3902688
Reviewed-by: Zihan Chen <zihanchen@google.com>
Commit-Queue: Dennis Kempin <denniskempin@google.com>
This search/replace updates all copyright notices to drop the
"All rights reserved", Use "ChromiumOS" instead of "Chromium OS"
and drops the trailing dots.
This fulfills the request from legal and unifies our notices.
./tools/health-check has been updated to only accept this style.
BUG=b:246579983
TEST=./tools/health-check
Change-Id: I87a80701dc651f1baf4820e5cc42469d7c5f5bf7
Reviewed-on: https://chromium-review.googlesource.com/c/crosvm/crosvm/+/3894243
Reviewed-by: Daniel Verkamp <dverkamp@chromium.org>
Commit-Queue: Dennis Kempin <denniskempin@google.com>
The option will generate an lcov file in the standard name lcov.info
and print a coverage report after running tests.
The lcov.info file is understood by many IDE plugins, e.g. "Coverage
Gutters" for vscode.
BUG=b:239255082
TEST=./tools/run_tests --cov
Change-Id: I475c62ada73f100a984ad863e511083c69807aff
Reviewed-on: https://chromium-review.googlesource.com/c/crosvm/crosvm/+/3840956
Commit-Queue: Dennis Kempin <denniskempin@google.com>
Reviewed-by: Daniel Verkamp <dverkamp@chromium.org>
Tested-by: Dennis Kempin <denniskempin@google.com>
Tests may spawn multiple processes (e.g. for running crosvm in
integration_tests). We want those to generate a separate profile
file to collect them as well.
This should add coverage from integration tests into the coverage
data.
Fixed coverage collection with repeated test runs
BUG=b:239255082
TEST=./tools/run_tests --generate-lcov coverage.lcov --repeat 2
Change-Id: I63e812620b79182a5be7e981b8f524b36b596ce2
Reviewed-on: https://chromium-review.googlesource.com/c/crosvm/crosvm/+/3840484
Reviewed-by: Daniel Verkamp <dverkamp@chromium.org>
Tested-by: Dennis Kempin <denniskempin@google.com>
Commit-Queue: Dennis Kempin <denniskempin@google.com>
This reverts commit 9ec4afc0ae.
Reason for revert: Breaks post-submit
https://ci.chromium.org/ui/p/crosvm/builders/ci/linux_x86_64/b8805461167965542065/overview
Original change's description:
> test_runner: Support multiple profiles per test
>
> Tests may spawn multiple processes (e.g. for running crosvm in
> integration_tests). We want those to generate a separate profile
> file to collect them as well.
>
> This should add coverage from integration tests into the coverage
> data.
>
> BUG=b:239255082
> TEST=./tools/run_tests --generate-lcov coverage.lcov
>
> Change-Id: Ic6c6b0801b676c96c4692069c1cd6111edea6fc4
> Reviewed-on: https://chromium-review.googlesource.com/c/crosvm/crosvm/+/3840311
> Reviewed-by: Daniel Verkamp <dverkamp@chromium.org>
> Commit-Queue: Dennis Kempin <denniskempin@google.com>
> Tested-by: Dennis Kempin <denniskempin@google.com>
Bug: b:239255082
Change-Id: I958792f86a5d5bc24910c057c61242fe405db20d
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://chromium-review.googlesource.com/c/crosvm/crosvm/+/3840169
Tested-by: Dennis Kempin <denniskempin@google.com>
Commit-Queue: Dennis Kempin <denniskempin@google.com>
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Auto-Submit: Dennis Kempin <denniskempin@google.com>
Tests may spawn multiple processes (e.g. for running crosvm in
integration_tests). We want those to generate a separate profile
file to collect them as well.
This should add coverage from integration tests into the coverage
data.
BUG=b:239255082
TEST=./tools/run_tests --generate-lcov coverage.lcov
Change-Id: Ic6c6b0801b676c96c4692069c1cd6111edea6fc4
Reviewed-on: https://chromium-review.googlesource.com/c/crosvm/crosvm/+/3840311
Reviewed-by: Daniel Verkamp <dverkamp@chromium.org>
Commit-Queue: Dennis Kempin <denniskempin@google.com>
Tested-by: Dennis Kempin <denniskempin@google.com>
This will keep colorized output for interactive usage, but disables
it for Luci logs.
BUG=b:239990667
TEST=tools/run_tests vs tools/run_tests > foo
Change-Id: Ie8667c1c2c1421ab45d0b98f42ce4df3d971d531
Reviewed-on: https://chromium-review.googlesource.com/c/crosvm/crosvm/+/3818250
Reviewed-by: Daniel Verkamp <dverkamp@chromium.org>
Commit-Queue: Dennis Kempin <denniskempin@google.com>
Tested-by: Dennis Kempin <denniskempin@google.com>
- Compiles out pvclock, sandbox and slirp (thereby net device)
- Disables default features on windows (audio and gpu are not ready yet)
- Enables unit tests for crosvm
BUG=b:213146388
TEST=presubmit
Change-Id: I5f4b3d8bf44d9c571c3a191a632e09550a4f1869
Reviewed-on: https://chromium-review.googlesource.com/c/crosvm/crosvm/+/3807045
Tested-by: Vikram Auradkar <auradkar@google.com>
Commit-Queue: Vikram Auradkar <auradkar@google.com>
Reviewed-by: Daniel Verkamp <dverkamp@chromium.org>
Auto-Submit: Vikram Auradkar <auradkar@google.com>
Use the recently stabilized coverage instrumentation feature of
rust to generate coverage profiles of our test runs.
These can then be used by a recipe to upload coverage to covecov
or use tools locally to generate coverage reports.
BUG=b:239255082
TEST=./tools/run_tests --generate-lcov coverage.lcov
Change-Id: Ifc64d11f3ae19a2eb7fdce36172d67bf3f7e6d17
Reviewed-on: https://chromium-review.googlesource.com/c/crosvm/crosvm/+/3805831
Tested-by: Dennis Kempin <denniskempin@google.com>
Reviewed-by: Daniel Verkamp <dverkamp@chromium.org>
Commit-Queue: Dennis Kempin <denniskempin@google.com>
BUG=None
TEST=tools/presubmit --all # with a failed test
Change-Id: I67262353cbd732b1394cdbe60b38d04c3a6e817b
Reviewed-on: https://chromium-review.googlesource.com/c/crosvm/crosvm/+/3763359
Reviewed-by: Dennis Kempin <denniskempin@google.com>
Tested-by: Dennis Kempin <denniskempin@google.com>
Commit-Queue: Daniel Verkamp <dverkamp@chromium.org>
Normally we run test binaries in parallel, this change allows some to be
executed exclusively.
This fixes some of the odd issues and flakes we have been seeing.
integration_tests can hang if run in parallel with other tests and
cros_async has the odd habit of throwing the test_runner process into
the background.
Neither happens when the tests are the only ones being executed.
Also adds the capability to specify which test binaries to run,
which was helpful in debugging this issue.
BUG=b:238232551
TEST=./tools/run_tests --repeat 10
Change-Id: Ie5b82e329007399e95bf38111ec6e7cb6761ec0c
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/3751027
Tested-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: Daniel Verkamp <dverkamp@chromium.org>
The test runner can now repeat tests in two ways:
- Via --repeat to run tests multiple times and fail if any one of them
fails. Which will help us find flakes in post-submit.
- Via --retry, which will retry a test if it has failed. Which makes
pre-submit tests more resilient to flakes.
Both can be configured by builder configs to adjust as needed.
Also slighly changes the repeat behavior to repeat tests in batches, so
we do not run the same test multiple times simultaneously, which can
cause some of them to fail. It's also easier to read the results.
BUG=b:238232551
TEST=added a random flake into some tests and used --repeat and --retry
Change-Id: I78e0ff0751da40a99a56080d9baf207307c9e93a
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/3751835
Tested-by: Dennis Kempin <denniskempin@google.com>
Reviewed-by: Daniel Verkamp <dverkamp@chromium.org>
Updates run_tests to use cargo style target triples for specifying
build targets. A simple 'aarch64' or 'armhf' was nice while we just
had linux builds. We now are looking at windows and possibly
different toolchain options (e.g. msvc vs gnu), so our old system
was getting confusing and inconsistent.
We used to have some special handling for adding wrappers to test
runs for emulation (e.g. wine, qemu). That logic has been moved
into TestTarget which now contains not just where to run the test
but also how.
Supported are armhf/aarch64 qemu as well as wine64.
The CLI has been updated to match and now uses the build-target
argument instead of arch.
The following combinations have been tested (though not all
combinations actually pass all tests, which is a separate issue).
./tools/run_tests
./tools/run_tests --target=host --build-target=x86_64-unknown-linux-gnu
./tools/run_tests --target=host --build-target=armhf
./tools/run_tests --target=host --build-target=aarch64
./tools/run_tests --target=host --build-target=mingw64
./tools/run_tests --target=vm:aarch64
./tools/run_tests --target=vm:aarch64 --build-target=armhf
BUG=b:233914170
TEST=See above
Change-Id: Ic6dbb5b39788e2573714606d3bb0e7c712032d91
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/3739240
Tested-by: kokoro <noreply+kokoro@google.com>
Commit-Queue: Dennis Kempin <denniskempin@google.com>
Reviewed-by: Daniel Verkamp <dverkamp@chromium.org>
One more script that needs to run on python 3.8 so we can run test
for windows on luci.
BUG=b:234173142
TEST=vpython3 tools/run_tests
Change-Id: I0348a567b2edec3b5e3fca77f316f39d1e924adc
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/3723037
Reviewed-by: Daniel Verkamp <dverkamp@chromium.org>
Tested-by: kokoro <noreply+kokoro@google.com>
Fixes cases when we use target vm:aarch64 but arch armhf and test_runner
picks up tests for aarch64
BUG=none
TEST=run and made sure armhf-disabled tests are not picked up
Change-Id: I4d0734bb6320fc698bf16adc59de5e03b00a4a68
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/3617081
Tested-by: kokoro <noreply+kokoro@google.com>
Commit-Queue: Anton Romanov <romanton@google.com>
Reviewed-by: Dennis Kempin <denniskempin@google.com>
Auto-Submit: Anton Romanov <romanton@google.com>
This change adds python type and formatting checks and
consolidates code health checks in ./tools/health-check.
Dealing with relative imports in python is tricky, so
we are making ./tools/impl a proper package with no
directly executable files.
Some of the bash shorthands in ./tools had to be converted
to python for this.
To make the new checks pass, we run the formatter and fix
some mypy type checks.
TEST=./tools/health-check
BUG=b:218559722,b:219965702
Change-Id: Ie18d3d6dd2f5a033141e167a6e1aa762791941d6
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/3558592
Reviewed-by: Daniel Verkamp <dverkamp@chromium.org>
Tested-by: kokoro <noreply+kokoro@google.com>
Commit-Queue: Dennis Kempin <denniskempin@google.com>
Also fixes few warning introduced during upstream which linux clippy
run does not catch.
test: py tools\clippy
bug: 226966790
Change-Id: I6979f87be5d6d34727afabafcaccb57631975567
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/3553401
Tested-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: Dennis Kempin <denniskempin@google.com>
Commit-Queue: Vikram Auradkar <auradkar@google.com>
This is a prerequisite for sys_util* to move into the base
crate. The crate should never have been in common in the
first place since it is not shared.
BUG=b:223206469
TEST=python tools/contrib/cargo_refactor.py
Change-Id: I60115e0418c6980d7bf7ff624a3cc0a24e71a57d
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/3530502
Reviewed-by: Anton Romanov <romanton@google.com>
Tested-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: Vikram Auradkar <auradkar@google.com>
With crosvm adding support for windows, we want to make sure that
the line endings are not crlf.
git hooks and configs will be added as a part of another change.
Bug: b:213146393
Test: Modified Cargo.toml to have crlf endings and ran the script.
Change-Id: Ie6f53a7c0687578546e63770bb98706e4315a520
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/3495708
Reviewed-by: Daniel Verkamp <dverkamp@chromium.org>
Tested-by: kokoro <noreply+kokoro@google.com>
Commit-Queue: Vikram Auradkar <auradkar@google.com>
As we make progress on upstreaming sys_util, we do not want more
dependencies on sys_util added. This check helps in that cause.
With this change only files that are inside common can depend directly
on sys_util.
Test: python3 ./tools/impl/check_code_hygiene.py common/sys_util_core
Bug: b:213149154
Change-Id: I57a8cb9189d3263a4031338b302cb27da799f4de
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/3473344
Tested-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: Dennis Kempin <denniskempin@google.com>
Auto-Submit: Vikram Auradkar <auradkar@google.com>
Commit-Queue: Vikram Auradkar <auradkar@google.com>
Run tests for sys_util_core, poll_token_derive and balloon_control on
windows.
Using dotfiles to disable/serialize test runs of a subset of crates does
not work well with third party crates as it forces us to commit the dot
file to the crate.
The patch modifies and uses the script that runs linux tests.
This patch also allows us to
- build/test child crate even if parent crate has disabled build/test.
- avoid building crosvm if it is not explicitly specified.
RIP short lived .windows_build_test_skip. You allowed us to run noop
kokoro tests.
Test: py .\tools\impl\test_runner.py --arch x86_64
Bug: b:215610772
Change-Id: Icc6d04ffd7c0c33d4f60aeac16fc7d23881c387d
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/3459809
Reviewed-by: Dennis Kempin <denniskempin@google.com>
Tested-by: kokoro <noreply+kokoro@google.com>
Commit-Queue: Vikram Auradkar <auradkar@google.com>
Tests can now be marked as large to increase the timeout from it's
original 60s to 120s.
The test runner is also updated to detect if the test is running in
an emulated environment, and doubles the timeout for those.
BUG=b:218681921
TEST=./tools/presubmit --all
Change-Id: I95fc238dcd23a5035826f422752eaf0ca215706c
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/3469496
Tested-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: Anton Romanov <romanton@google.com>
Commit-Queue: Dennis Kempin <denniskempin@google.com>
While we can run armhf binaries on aarch64 or use user-space emulation
to run aarch64 tests on the x86 host, some tests will fail in that
scenario as they will interact with kernel APIs directly (e.g. kvm).
This will make it easier to distinguish 'do not run on armhf because
we haven't fixed the tests' from 'this test is not supported on
a foreign arch kernel and that is expected'.
It also enables a quicker method of running aarch64 tests with
user-space emulation.
BUG=b:218374759
TEST=./tools/dev_container ./tools/run_tests --target=host
--arch=aarch64
Change-Id: I81871e11d3b02ee1ed3bac88bada28fbba8a3f31
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/3447292
Reviewed-by: Daniel Verkamp <dverkamp@chromium.org>
Tested-by: kokoro <noreply+kokoro@google.com>
Commit-Queue: Dennis Kempin <denniskempin@google.com>
Enable VVU feature unconditionally.
Since kokoro will start checking VVU code from this CL, clippy errors
are fixed.
In addition, the time limit of kokoro unit testing needed to be
increased because some time-consuming unit tests will start running on
aarch64 QEMU, which is really slow.
BUG=none
TEST=kokoro
Change-Id: I67437c060ff5ba9f690dcfc6e5c3dd155b95a785
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/3450014
Tested-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: Dennis Kempin <denniskempin@google.com>
Commit-Queue: Keiichi Watanabe <keiichiw@chromium.org>
This will still run the same tests as before, the options are just
more consistent now.
BUG=b:218374759
TEST=./tools/presubmit
Change-Id: I41a8730f6cfaf65a62aaa26c45e2c284f0356d33
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/3447541
Reviewed-by: Daniel Verkamp <dverkamp@chromium.org>
Tested-by: kokoro <noreply+kokoro@google.com>
Commit-Queue: Dennis Kempin <denniskempin@google.com>
The patch disallows any platform specific code in sys_util_core and
doesn't let sys_util compile on windows platform.
This ensure to some extent that we keep sys_util and it's dependency
sys_util_core independent of windows code.
check_code_hygiene is not foolproof.
Test: Ran the script against a modified file in sys_util_core containing
string "target_os = "
Tried to compile sys_util on windows.
Bug: b:215610772
Upstream-Crate: common/win_sys_util
Cq-Depend: chromium:3433709
Change-Id: Ideb45092a959dd347d966633c3bd4e82f842b1a8
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/3438709
Reviewed-by: Dennis Kempin <denniskempin@google.com>
Tested-by: kokoro <noreply+kokoro@google.com>
proc-macro tests run on the host, adding logic into the test runner
to accomodate for that.
BUG=b:218374759
TEST=./tools/presubmit
Change-Id: I83c853b9e28dae725d299509f7e3a07f56bd6ad8
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/3445531
Reviewed-by: Daniel Verkamp <dverkamp@chromium.org>
Tested-by: kokoro <noreply+kokoro@google.com>
Commit-Queue: Dennis Kempin <denniskempin@google.com>
Instead of enumerating crosvm crates separately, we can now use
the workspace to build all tests for crosvm.
This enables the vmm_vhost tests.
BUG=b:206026060
TEST=./tools/presubmit
Change-Id: I2af84ad8111e874ddea5e438bda41599caf1fd30
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/3287463
Tested-by: kokoro <noreply+kokoro@google.com>
Commit-Queue: Dennis Kempin <denniskempin@google.com>
Reviewed-by: Daniel Verkamp <dverkamp@chromium.org>
Each of these is a separate workspace and can be built in
parallel.
BUG=b:195126527
TEST=./tools/run_tests
Change-Id: I9ad61b5d5d83903fda36c948ba470bba8f608d46
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/3248128
Commit-Queue: Dennis Kempin <denniskempin@google.com>
Tested-by: Dennis Kempin <denniskempin@google.com>
Reviewed-by: Daniel Verkamp <dverkamp@chromium.org>
When looking up crate options from test_options.py we used their
path relative to the crosvm root (e.g. common/p9). But when looking
up options for binaries, the name is parsed from cargo output and
just the crate name (e.g. p9). Fix this by just using the crate
name.
BUG=b:195126527
TEST=./tools/run_tests
Change-Id: I09a3469daa071dc66295d777449dc101a6564162
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/3248127
Commit-Queue: Dennis Kempin <denniskempin@google.com>
Tested-by: Dennis Kempin <denniskempin@google.com>
Reviewed-by: Daniel Verkamp <dverkamp@chromium.org>
The new runner makes use of the simplified crosvm build and greatly
improves execution speed by gathering test binaries from cargo output
and executes them directly in parallel.
This allows all of our tests to execute in ~5 seconds when run locally.
The new test runner also makes use of the new testvm tools to make it
easy to switch between testing on the host, in a VM or via SSH on a
remote device.
See ./tools/run_tests --help for usage instructions.
To allow more iterative testing with the same test targets, this CL
includes a set_test_target script to write env vars that instruct cargo
to build for the target arch and run on the test target.
Note: The test runner can build for armhf, but we need build file fixed
to allow armhf to build successfully.
BUG=b:199951064
TEST=Inside ./tools/dev_container:
./tools/run_tests --target=host
./tools/run_tests --target=vm:aarch64
./tools/run_tests --aarch=armhf --build-only
./tools/set_test_target vm:aarch64 && source .envrc
cargo test
Change-Id: I60291fa726fb2298f62d1f032e55ed6e7ab1c4ac
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/3221779
Tested-by: kokoro <noreply+kokoro@google.com>
Commit-Queue: Dennis Kempin <denniskempin@google.com>
Reviewed-by: Daniel Verkamp <dverkamp@chromium.org>
