Commit graph

47 commits

Author SHA1 Message Date
Keiichi Watanabe
3e7c388efa tools: Add cargo-doc script
BUG=none
TEST=Run GitHub Action on my personal GitHub repository

Change-Id: I62de2693f3a654e0bbd327eff5931380a153395d
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/3305958
Tested-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: Dennis Kempin <denniskempin@google.com>
Commit-Queue: Keiichi Watanabe <keiichiw@chromium.org>
2021-12-09 14:34:31 +00:00
Dennis Kempin
cd0c2b846a kokoro: Add user name / email to commits
This prevents gerrit from claiming we are forging email addresses.

BUG=b:209034086
TEST=None

Change-Id: I9f5bab0684f40ad5220d8c88d89a34e23facab99
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/3321734
Reviewed-by: Daniel Verkamp <dverkamp@chromium.org>
Tested-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: Dennis Kempin <denniskempin@google.com>
Commit-Queue: Dennis Kempin <denniskempin@google.com>
2021-12-08 18:50:50 +00:00
Dennis Kempin
f18de38a02 kokoro: clean repo before merging
There can be leftover files from a previous build.

BUG=b:209034086
TEST=./ci/kokoro/simulate build-merge-into-chromeos.sh

Change-Id: I0f7223bc6dd1769db04408f595764931c3ff2af8
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/3321730
Reviewed-by: Daniel Verkamp <dverkamp@chromium.org>
Tested-by: Dennis Kempin <denniskempin@google.com>
2021-12-07 21:03:03 +00:00
Dennis Kempin
fbb2de0188 kokoro: Various fixes to merge-into-chromeos job
- Adds the needed job config
- Makes a copy of the script before git checkout
- Follow naming scheme of other build scripts
- Add reviewer when uploading

BUG=b:209034086
TEST=./ci/kokoro/simulate build-merge-into-chromeos.sh

Change-Id: I279ff6aca7c3c304354a238ebd5c40b4dd272385
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/3321726
Reviewed-by: Daniel Verkamp <dverkamp@chromium.org>
Tested-by: Dennis Kempin <denniskempin@google.com>
2021-12-07 20:50:29 +00:00
Dennis Kempin
e061930706 Add Kokoro build script for merging main into chromeos
The Kokoro build script sets up access to the chromium
gerrit, creates the merge commit and uploads it.

BUG=b:209034086
TEST=./ci/kokoro/simulate merge-to-chromeos.sh

Change-Id: Ia686b937c3d6b0b744d5c2162eb2847c5952e977
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/3318793
Tested-by: kokoro <noreply+kokoro@google.com>
Commit-Queue: Dennis Kempin <denniskempin@google.com>
Reviewed-by: Daniel Verkamp <dverkamp@chromium.org>
2021-12-06 23:28:26 +00:00
Dennis Kempin
95b80d13b2 dev_container: Preserve container between calls
This vastly improves iterative build times and enables more flexible
usage of the container.

BUG=None
TEST=./tools/dev_container cargo build
First run will build everything. Second run will finish right away.

Change-Id: I9b4eeee0689f0e9d07f0a32f846d21ab42f689f7
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/3292100
Tested-by: kokoro <noreply+kokoro@google.com>
Commit-Queue: Dennis Kempin <denniskempin@google.com>
Reviewed-by: Junichi Uekawa <uekawa@chromium.org>
2021-11-19 20:09:39 +00:00
Keiichi Watanabe
e538834fe1 ci: kokoro: Run mdbook command
BUG=none
TEST=kokoro

Change-Id: I61f80da1c6d82979201aa1b3c11b60d9da6ba705
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/3255195
Auto-Submit: Keiichi Watanabe <keiichiw@chromium.org>
Commit-Queue: Dennis Kempin <denniskempin@google.com>
Tested-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: Dennis Kempin <denniskempin@google.com>
2021-11-01 19:26:51 +00:00
Keiichi Watanabe
610158aa55 ci: kokoro: push-to-github: Use git push --all instead of --mirror
Unlike git push --mirror, git push --all won't delete remote branch so
that gh-pages branch won't be removed by kokoro, I think.

BUG=b:199692764
TEST=none

Change-Id: I2a5ce2f59e966775d24c25b996a1091af1d6efc9
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/3234557
Auto-Submit: Keiichi Watanabe <keiichiw@chromium.org>
Commit-Queue: Dennis Kempin <denniskempin@google.com>
Tested-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: Dennis Kempin <denniskempin@google.com>
2021-10-20 16:06:04 +00:00
Dennis Kempin
8a1277de1a Kokoro: Rebase changes to ToT before testing
By default Kokoro will use changes as they come from gerrit, which may
have an outdated parent. We want to make sure that we are always testing
against tip of tree so we are confident the presubmit test result will
reflect the postsubmit results.

BUG=b:202275156
TEST=Tested in Kokoro

Change-Id: I2a1d2860a361eee741dc522c9859e32890ff15bf
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/3229947
Tested-by: kokoro <noreply+kokoro@google.com>
Commit-Queue: Dennis Kempin <denniskempin@google.com>
Reviewed-by: Daniel Verkamp <dverkamp@chromium.org>
2021-10-19 17:10:04 +00:00
Dennis Kempin
76e6fd051b Fix push-to-github Kokoro builder
One more fix. Everything seems to work, but the remote is rejecting
an update of the active main branch. Hopefully the last fix needed.

BUG=b:199692764
TEST=Testing in prod.

Change-Id: I4f341cc374080454317cb3ac29cade59fec6aeee
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/3229958
Tested-by: Dennis Kempin <denniskempin@google.com>
Tested-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: Daniel Verkamp <dverkamp@chromium.org>
2021-10-18 23:22:27 +00:00
Dennis Kempin
2d50ccb4a7 Fix push-to-github kokoro build
We were in the wrong directory.

BUG=b:199692764
TEST=Testing in prod...

Change-Id: Idbc60f7f6084109220186e56cd7fad2e6553da9a
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/3229950
Tested-by: Dennis Kempin <denniskempin@google.com>
Reviewed-by: Daniel Verkamp <dverkamp@chromium.org>
2021-10-18 22:00:22 +00:00
Dennis Kempin
49c169a410 Add push-to-github kokoro config and script
This is hard to test. I verified that kokoro can access
the github token and use it to access github by adding
a git clone into the x86 presubmit build file in patchset 3.

But we can only really test the full process by submitting this
CL and have it pass through the continuous builder.

BUG=b:199692764
TEST=Needs to be tested in prod...

Change-Id: Ifdcd385552c61ecea75a43fb71b08950e8b036ca
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/3229941
Tested-by: Dennis Kempin <denniskempin@google.com>
Reviewed-by: Daniel Verkamp <dverkamp@chromium.org>
2021-10-18 21:37:58 +00:00
Dennis Kempin
d1dbefc9dc Integrate new container and test runner with Kokoro
This replaces the existing tooling at ./ci/ with the new tooling
at ./tools.

BUG=b:199951064
TEST=./ci/kokoro/simulate_all

Change-Id: Id0296d5a46ca0f3eb0f655d85d1aed801ee3b434
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/3226275
Tested-by: kokoro <noreply+kokoro@google.com>
Commit-Queue: Dennis Kempin <denniskempin@google.com>
Reviewed-by: Daniel Verkamp <dverkamp@chromium.org>
2021-10-18 20:10:42 +00:00
Junichi Uekawa
ce6cf4efe2 ci: Quote underscores.
My editor doesn't like them, and probably they should be.

BUG=None
TEST=read it.

Change-Id: I40041e41841e47cb386fcd12a4622efeca33474d
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/3215932
Reviewed-by: Dennis Kempin <denniskempin@google.com>
Commit-Queue: Dennis Kempin <denniskempin@google.com>
Tested-by: Dennis Kempin <denniskempin@google.com>
2021-10-11 18:28:42 +00:00
Dennis Kempin
e9aa5f9598 Fix ./ci/kokoro/simulate
Somehow, after rsyncing the whole directory, git will no longer work in the
copy. So instead, we are now using 'git clone'.
This changes the behavior to not include local workspace modifications in
kokoro simulations. Which is ok, we have test_all for that and the behavior
will more closely match what's happening in kokoro presubmits.

BUG=None
TEST=./ci/simulate_all

Change-Id: I439b9eadcac65d99782e2b0eb869519abb2ada37
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/3152425
Reviewed-by: Daniel Verkamp <dverkamp@chromium.org>
Tested-by: kokoro <noreply+kokoro@google.com>
Commit-Queue: Dennis Kempin <denniskempin@google.com>
2021-09-15 17:06:48 +00:00
Dennis Kempin
3b3545b2c3 Upgrade ci containers to bullseye and clean up
We no longer need to build dependencies from source and provide them
via pkg-config, this simplifies much of our builder container setup.

As debian bullseye got promoted to stable, we can now also simplify
further by just using bullseye stable instead of mixing stable and
testing packages.

BUG=b:181359683
TEST=./test_all

Change-Id: I2ce61992d5cfe6eb5dc3f0ec61920dcc5455ca40
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/3141772
Tested-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: Daniel Verkamp <dverkamp@chromium.org>
Commit-Queue: Dennis Kempin <denniskempin@google.com>
2021-09-15 17:06:46 +00:00
Dennis Kempin
ab0ad4caa0 Move virglrenderer/minigbm build into build.rs
Both libraries have previously been built as part of
ci/build_environment/Makefile. This CL moves that behavior into the
build.rs file of rutabaga_gfx.

This is the last third party dependency that we need to build from
source, and allows us to build/test on the host machine instead of
requiring the builder container.

It also allows us to greatly simplify the builder containers, which
I will do in a follow-up CL as we also need to upgrade them to bullseye.

This CL uprevs virglrenderer to include:
https://gitlab.freedesktop.org/virgl/virglrenderer/-/merge_requests/609

and minigbm to include: https://crrev.com/c/3141018

BUG=b:196059146
TEST=./test_all && ./run_tests --run-privileged

Change-Id: I4442ccc991d13a3fcfa224de50e916b3926f0cb4
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/3141771
Commit-Queue: Dennis Kempin <denniskempin@google.com>
Tested-by: Dennis Kempin <denniskempin@google.com>
Reviewed-by: Gurchetan Singh <gurchetansingh@chromium.org>
2021-09-09 23:13:24 +00:00
Dennis Kempin
3407843bd4 Build minijail through build.rs file in CI containers
This removes the minijail build from build_environment, which will
cause the minijail-sys crate to build it from source.

Minijail is upreved to include https://r.android.com/1815277

BUG=b:198305518
TEST=./test_all

Change-Id: I38c46c2a7df43e3d3a94ae0c5f8a9aae2abd3555
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/3141770
Tested-by: kokoro <noreply+kokoro@google.com>
Commit-Queue: Dennis Kempin <denniskempin@google.com>
Reviewed-by: Daniel Verkamp <dverkamp@chromium.org>
2021-09-07 17:42:19 +00:00
Dennis Kempin
00bfcab3e8 Update tpm2-sys build.rs to support cross-compilation
Also removes the use of the hermetic flag in favor of not failing the
build if the submodule is not checked out.
This allows us to remove the tpm2 build from the build_environment
Makefile.

BUG=b:198293072
TEST=./test_all

Change-Id: Ide81e78efe0da3a1b64d4b8ef094a2e901f99ccf
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/3133623
Tested-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: Daniel Verkamp <dverkamp@chromium.org>
Commit-Queue: Dennis Kempin <denniskempin@google.com>
2021-09-01 01:59:09 +00:00
Daniel Verkamp
dcfb7de431 ci: prevent creation of virglrenderer __pycache__
Prevent the dependency build process from leaving untracked files in the
working copy when running test_all.

BUG=None
TEST=./test_all

Change-Id: Ia9dca114ffe98e73f9858795657a3864deab9d3b
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/3119699
Reviewed-by: Dennis Kempin <denniskempin@google.com>
Tested-by: kokoro <noreply+kokoro@google.com>
Commit-Queue: Daniel Verkamp <dverkamp@chromium.org>
2021-08-31 20:12:29 +00:00
Dennis Kempin
b3ccebbf3d Improve build / test documentation
A lot of the information was outdated. The new guide prioritizes
building for linux and testing on linux as the default workflow
for crosvm.

BUG=b:194323235
TEST=None

Change-Id: Idb7cdb07151fc7d7d5466075e316015cf5f4c615
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/3098391
Commit-Queue: Dennis Kempin <denniskempin@google.com>
Tested-by: Dennis Kempin <denniskempin@google.com>
Reviewed-by: Daniel Verkamp <dverkamp@chromium.org>
2021-08-16 20:43:22 +00:00
Junichi Uekawa
74b18b8ccb vm_tools: Use /var/tmp instead of /tmp.
/tmp is mounted as tmpfs, and tmpfs does not provide O_DIRECT capabilities. Try
using a ext4 file system.

BUG=b:190435784
TEST=time ./ci/builder --vm #    boot_test_vm_odirect passes

Change-Id: I3f8245052ed06c703cc3aa320d300d5f21254e90
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/3083206
Tested-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: Daniel Verkamp <dverkamp@chromium.org>
Commit-Queue: Junichi Uekawa <uekawa@chromium.org>
2021-08-11 02:22:40 +00:00
Dennis Kempin
42b825f740 Switch to submodules based workflow
Instead of requiring crosvm to be checked out via the chromeos manifest
to access dependencies in the ChromeOS monorepo, this change adds
git submodules to third_party/.

The CI scripts and Cargo.toml are updated to use the new paths.

BUG=b:194336213
TEST=git clone --recursive https://chromium.googlesource.com/chromiumos/platform/crosvm
     cd crosvm
     cargo test
     ./test_all
     ./ci/kokoro/simulate_all

Change-Id: I9859d18176e21909ac3a140976fbd67cc14129bb
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/3049003
Tested-by: kokoro <noreply+kokoro@google.com>
Commit-Queue: Dennis Kempin <denniskempin@google.com>
Reviewed-by: Chirantan Ekbote <chirantan@chromium.org>
2021-08-05 18:32:32 +00:00
Allen Webb
c6fa73d76f Fix clippy warnings and Cargo.lock
This fixes:
* version mismatches in Cargo.lock
* style issues
* implementations of Into that should be From
* deprecated protobuf APIs

It also adds RUST_BACKTRACE=1 to the kokoro tests.

BUG=None
TEST=./bin/preupload-clippy

Change-Id: I8e9157c903f2080a5fdcc4d3e4ed72fbad41c64f
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/3024427
Auto-Submit: Allen Webb <allenwebb@google.com>
Tested-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: Daniel Verkamp <dverkamp@chromium.org>
Commit-Queue: Allen Webb <allenwebb@google.com>
2021-07-15 03:33:17 +00:00
Dennis Kempin
75eef5f065 Docker: Do not curl | bash
Instead, download the rustup-init binary directly and verify it's sha
before executing.
Also update debian image used in the bundled VM. The previous image
file was no longer available.

TEST=Rebuilt docker images
BUG=b:192355221

Change-Id: I8a2d8091b9cecdf40ecbdd4bb6cef7ca161421d9
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/3011343
Reviewed-by: Daniel Verkamp <dverkamp@chromium.org>
Tested-by: kokoro <noreply+kokoro@google.com>
Commit-Queue: Dennis Kempin <denniskempin@google.com>
2021-07-07 23:55:27 +00:00
Zach Reizner
7a96510dcd book: add mermaid-js support
TEST=mdbook docs/book
BUG=None

Change-Id: I63c487e3718ed47f7084b211fa398422b8b571fa
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/2991090
Reviewed-by: Dennis Kempin <denniskempin@google.com>
Tested-by: Dennis Kempin <denniskempin@google.com>
Tested-by: kokoro <noreply+kokoro@google.com>
Commit-Queue: Dennis Kempin <denniskempin@google.com>
2021-07-01 16:28:32 +00:00
Junichi Uekawa
dc2f396277 kokoro: Remove mention of manifest.xml.
We are no longer pinned to a specific version since
https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/2840057

BUG=None
TEST=read it.

Change-Id: I23750d5404f776c21de4ba11e17b5704751b247a
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/2902915
Reviewed-by: Dennis Kempin <denniskempin@google.com>
Reviewed-by: Chirantan Ekbote <chirantan@chromium.org>
Commit-Queue: Junichi Uekawa <uekawa@chromium.org>
Tested-by: kokoro <noreply+kokoro@google.com>
Auto-Submit: Junichi Uekawa <uekawa@chromium.org>
2021-05-21 08:07:02 +00:00
Junichi Uekawa
b6ae6517ae kokoro: format README.md with Google style.
BUG=None
TEST=Read it.

Change-Id: I477a4d18255c8cc5ed2e4eb903d6651615473ad6
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/2902914
Reviewed-by: Dennis Kempin <denniskempin@google.com>
Reviewed-by: Chirantan Ekbote <chirantan@chromium.org>
Commit-Queue: Junichi Uekawa <uekawa@chromium.org>
Tested-by: kokoro <noreply+kokoro@google.com>
2021-05-20 08:38:23 +00:00
Dennis Kempin
acdf8de5c6 Kokoro: Add debian bullseye version of libwayland/protocols
This is required by the latest gfx changes.

BUG=None
TEST=./ci/kokoro/simulate_all

Change-Id: I13280a3c5213d12730bcb154514b7864663a1682
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/2880148
Tested-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: Gurchetan Singh <gurchetansingh@chromium.org>
Reviewed-by: Daniel Verkamp <dverkamp@chromium.org>
Commit-Queue: Dennis Kempin <denniskempin@google.com>
2021-05-10 18:27:07 +00:00
Dennis Kempin
e6216f926c Uprev rust-toolchain for Kokoro to 1.51.0
Change-Id: I27b30762207021bbaf307bb02118992442021a93
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/2877257
Reviewed-by: Zach Reizner <zachr@chromium.org>
Reviewed-by: Daniel Verkamp <dverkamp@chromium.org>
Commit-Queue: Zach Reizner <zachr@chromium.org>
Tested-by: Zach Reizner <zachr@chromium.org>
2021-05-07 00:49:19 +00:00
Zach Reizner
b0ae5378c6 ci: install bindgen and required header files
The cras client library has switch to build time generation for their
Rust bindings and requires the bindgen binary and the usual standard
header files to run the build script. This change adds those needed
components to the Docker images so that kokoro may work again.

BUG=None
TEST=./test_all

Change-Id: Ib16b164f44d536be84f9ecca6b5187eaba7d57ca
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/2863167
Reviewed-by: Dennis Kempin <denniskempin@google.com>
Reviewed-by: Daniel Verkamp <dverkamp@chromium.org>
Reviewed-by: Abhishek Pandit-Subedi <abhishekpandit@chromium.org>
Tested-by: kokoro <noreply+kokoro@google.com>
Commit-Queue: Zach Reizner <zachr@chromium.org>
2021-05-04 20:59:54 +00:00
Dennis Kempin
98d2f0294b Kokoro: Always build against ToT instead of a fixed manifest
This will introduce a little instability into our test infrastructure
since tests will no longer be hermetic, but build against the latest
ToT version of our chromeos dependencies.

It will however, make our day-to-day workflows a lot easier. A lot of
changes are made across repository boundaries and require
annoying extra steps since Kokoro does not understand Cq-Depend.

BUG=b:185917395
TEST=./ci/kokoro/simulate_all passes

Change-Id: I3c3cae20d418574a3db53cc74d81245b87c228db
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/2840057
Tested-by: kokoro <noreply+kokoro@google.com>
Commit-Queue: Dennis Kempin <denniskempin@google.com>
Reviewed-by: Daniel Verkamp <dverkamp@chromium.org>
2021-04-21 04:51:34 +00:00
Dennis Kempin
78145acd86 Make Kokoro happy
Update Kokoro manifest and Cargo.lock.

This change does not affect ebuild builds, so it can be safely chumped
to get Kokoro green.

BUG=None
TEST=./ci/kokoro/simulate_all passes

Change-Id: If4e2b849f12d4e223dcfe6dd5bb48e034067b466
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/2840055
Commit-Queue: Dennis Kempin <denniskempin@google.com>
Commit-Queue: Zach Reizner <zachr@chromium.org>
Tested-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: Daniel Verkamp <dverkamp@chromium.org>
Reviewed-by: Zach Reizner <zachr@chromium.org>
2021-04-20 23:05:26 +00:00
Zach Reizner
d09392e37e uprev rust-toolchain and fix clippy warnings
Some judgement calls were made about unnecessary wrapping. Usually they
would get resolved by removing the wrapping or returning a convenient
error, but the ones that returned results for consistency with other
functions were added to the allow list.

The error handling in the usb code had a lot of unit error types which
is now a clippy lint. This was resolved by either removing the result
entirely or returning a convenient error.

The field_reassign_with_default lint is faulty and was added to the list
of supressions. This affected virtio-wayland code.

BUG=b:179277332
TEST=cargo clippy with rustc 1.50+

Change-Id: Ie812cdeaf7c42f4f2b47b1dc87f05a7c87a60f8f
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/2757510
Tested-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: Zach Reizner <zachr@chromium.org>
Reviewed-by: Gurchetan Singh <gurchetansingh@chromium.org>
Auto-Submit: Zach Reizner <zachr@chromium.org>
Commit-Queue: Dennis Kempin <denniskempin@google.com>
2021-03-22 21:41:07 +00:00
Dennis Kempin
3f63661afc Run integration_tests by calling crosvm binary
Instead of instanciating crosvm directly, we can start the binary as
a sub-process.
This includes parsing of crosvm options in the tests, and makes the
test cases closer to real-world usage.

To make make this possible, we need to make sure that the crosvm binary
is uploaded to the VM before running the test, which is done by the
sync_so script, which is baked into the builder container.
We prevent future container re-builds for just maintaining the script,
I have removed them from the container, and call the scripts from the
local source directly.

The test runner is also updated to ensure all package binaries are
built (currently only tests are built).

BUG=b:182841358
TEST=./test_all passes

Change-Id: I7dfd21abcb2b90fe125eb43f85572fbf645b888a
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/2744280
Tested-by: Dennis Kempin <denniskempin@google.com>
Commit-Queue: Dennis Kempin <denniskempin@google.com>
Reviewed-by: Daniel Verkamp <dverkamp@chromium.org>
2021-03-19 20:35:53 +00:00
Keiichi Watanabe
5dc0f60d44 kokoro: Add rust-vmm/vhost in manifest.xml
BUG=b:181227406
TEST=ci/kokoro/simulate_all

Change-Id: I65065963daa9e96457ed9a64f01e34ad93bd8667
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/2762488
Tested-by: Keiichi Watanabe <keiichiw@chromium.org>
Tested-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: Dennis Kempin <denniskempin@google.com>
Reviewed-by: Dylan Reid <dgreid@chromium.org>
Reviewed-by: Daniel Verkamp <dverkamp@chromium.org>
Commit-Queue: Dennis Kempin <denniskempin@google.com>
2021-03-17 19:12:08 +00:00
Dennis Kempin
dcf540be92 Add SEPARATE_WORKSPACE requirement for run_tests
The io_uring and fuzz crate cannot be tested in the same workspace
via `cargo test -p io_uring -p fuzz`.
This change allows test_runner.py to run tests independently by
cd'ing into the crates directory and running cargo test in there.

BUG=b:181672910
BUG=b:181673923
TEST=Tests for the above mentioned crates can be enabled and run, but
test failures prevent them from being enabled so far.

Change-Id: Ia03868d53e508549fe2f071da399b982359a8834
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/2749772
Tested-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: Daniel Verkamp <dverkamp@chromium.org>
Commit-Queue: Dennis Kempin <denniskempin@google.com>
2021-03-12 23:36:13 +00:00
Dennis Kempin
f248a8ec2b Fix Kokoro build with manifest uprev
Kokoro was failing since http://crrev.com/c/2744968 depends on recent
cras changes that were not included in the current Kokoro manifest.

BUG=None
TEST=./ci/kokoro/simulate_all passes

Change-Id: I470844af73b9f89ac8cbf9f5fcf8706d01924279
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/2746014
Reviewed-by: Zach Reizner <zachr@chromium.org>
Reviewed-by: Dylan Reid <dgreid@chromium.org>
Tested-by: kokoro <noreply+kokoro@google.com>
Commit-Queue: Dennis Kempin <denniskempin@google.com>
2021-03-10 06:16:25 +00:00
Dennis Kempin
3682372764 ci: Upgrade test-vm to debian bullseye
This change updates the test VM to debian bullseye, primarily to enable
io_uring features that are not available in the 4.16 kernel shipping
with debian buster.

We are not updating the builder containers to bullseye, as newer
versions of qemu fail to start on kokoro. See b/181359683

Since it was needed for debugging this CL, it includes a feature to
debug kokoro builders via SSH. It can be enabled by setting
DEBUG_SSH_KEY from the fusion2 UI.

BUG=b:178228512
TEST=./test_all passes

Change-Id: I22f1678577183a0aace80245a87c31907221cc09
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/2721062
Reviewed-by: Daniel Verkamp <dverkamp@chromium.org>
Reviewed-by: Zach Reizner <zachr@chromium.org>
Tested-by: kokoro <noreply+kokoro@google.com>
Commit-Queue: Dennis Kempin <denniskempin@google.com>
2021-03-10 03:37:00 +00:00
Daniel Verkamp
0a91c96437 arch: rewrite FDT writer in native rust
This removes some unsafe code, improving the interface so that it cannot
be misused (e.g. previously, different Vec<u8> instances could be passed
to fdt functions that did not validate the contents).

The new implementation also adds some extra error checking to catch
invalid string values in all API entry points that accept strings, as
well as out-of-order node nesting that would result in DTB data that did
not conform to the spec.

BUG=b:175729255
TEST=cargo test -p arch
TEST=Boot Crostini on kevin (arm)
TEST=diff old and new dts from `dtc -I fs -O dts /proc/device-tree`

Change-Id: I567d562ed1b447aa0d282b68c71758edea13ffc0
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/2713569
Reviewed-by: Zach Reizner <zachr@chromium.org>
Reviewed-by: Keiichi Watanabe <keiichiw@chromium.org>
Reviewed-by: Dylan Reid <dgreid@chromium.org>
Tested-by: kokoro <noreply+kokoro@google.com>
Commit-Queue: Daniel Verkamp <dverkamp@chromium.org>
2021-03-03 01:29:15 +00:00
Dennis Kempin
4d2253b529 Kokoro: Extensive polishing and bugfixing
This CL fixes some of the issues that prevented the
test system from running on other workstations.

- Builders run by Kokoro will not use a scratch mount.
  It's not needed, and will cause issues if stored on
  /tmp with noexec set.
- Running podman with label=disable to prevent selinux
  issues.

The run_tests implementation has been moved to a separate
file and updated with parsing of cargo test output. This
allows simpler test output and integration with the
test results UI in sponge.

The sponge test UI can make it much easier to see which
tests failed, and to find the log of that test case.

This CL also includes an ./ci/kokoro/uprev script to
uprev the manifest versions. And runs the uprev.

BUG=b:174861002
TEST=Tested by forcing a kokoro build with this CL

Change-Id: I0cba9bb68915e2558a4ea6061dd9ba0a7050421b
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/2669712
Reviewed-by: Zach Reizner <zachr@chromium.org>
Tested-by: kokoro <noreply+kokoro@google.com>
Tested-by: Dennis Kempin <denniskempin@google.com>
Commit-Queue: Dennis Kempin <denniskempin@google.com>
2021-02-10 22:04:43 +00:00
Dennis Kempin
26cc51e588 Make Kokoro v2 happy with new config
Fixes some nits that cause kokoro builds to fail and updates
the Kokoro configs to match the new Kokoro-side configs
submitted in cl/355894530.

With this CL Kokoro v2 should be passing and can be enabled as
a presubmit.

BUG=b:179289824
TEST=Kokoro passes

Change-Id: I8c9259407bf1eb932b05376b35cb88d2e7d6058c
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/2678792
Tested-by: Dennis Kempin <denniskempin@google.com>
Tested-by: kokoro <noreply+kokoro@google.com>
Commit-Queue: Dennis Kempin <denniskempin@google.com>
Reviewed-by: Stephen Barber <smbarber@chromium.org>
Reviewed-by: Daniel Verkamp <dverkamp@chromium.org>
2021-02-06 05:26:04 +00:00
Dennis Kempin
0dbb9808a6 Add Kokoro for ci/builders
Adds the crosvm-side infrastructure to build and test
in kokoro.
There is a build script for testing on x86, aarch64
and a separte script for analysis (clippy, fmt).
These will run in parallel on Kokoro. To test the
scripts locally, a simulate script is provided.

Runtime on my workstation:
- aarch64: 10m
- x86: 2:30m
- analysis: 1:40m

BUG=b:177951955
TEST=./ci/kokoro/simulate_all

Change-Id: I2f666ec768e6c3391a258dc7f0cbd999ad9b2fb1
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/2654413
Tested-by: Dennis Kempin <denniskempin@google.com>
Commit-Queue: Dennis Kempin <denniskempin@google.com>
Reviewed-by: Stephen Barber <smbarber@chromium.org>
2021-02-05 03:01:55 +00:00
Dennis Kempin
aa0717443e Add VM to builders
The VM allows both the x86 and aarch64 builders to run tests which
require access to kernel devices that docker would prevent us
from accessing.
The VM is automatically started in the endpoint script, and the
environment is set up to execute `cargo test` binaries inside the VM.

This allows for convenient interactive usage.
The VM is also integrated into the run_tests scripts, executing tests
that do not require special privileges first, while the VM is still
booting up, then execuing the remaining tests in the VM.

BUG=b:177228167,b:177951276
TEST=These test calls pass:
./ci/builder --vm ./run_tests
./ci/aarch64_builder --vm ./run_tests
./run_tests

Change-Id: I20bf2cd848e944d411b97f1f8356279e312d8583
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/2642766
Tested-by: Dennis Kempin <denniskempin@google.com>
Commit-Queue: Dennis Kempin <denniskempin@google.com>
Reviewed-by: Zach Reizner <zachr@chromium.org>
2021-02-05 03:01:53 +00:00
Dennis Kempin
b40ec47309 Make Kokoro happy
A few things happened:

- A recent minigbm change broke compilation for debian, so
  I am pinning the commit SHA for the Dockerfile.
- Newer rust versions come with new clippy errors, which I will
  have to look at in a separate change. For now I am pinning our
  rust version.
- Some repos have transitioned to a main branch. Fixed our uprev
  script to handle those.
- Renamed SYSROOT so we won't confuse pkg-config.

Now Kokoro is finally happy again!

BUG=None
TEST=wrapped_smoke_tests

Change-Id: I76294abe35fd84b305124158b3942a321651982d
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/2643779
Tested-by: Dennis Kempin <denniskempin@google.com>
Tested-by: kokoro <noreply+kokoro@google.com>
Commit-Queue: Dennis Kempin <denniskempin@google.com>
Reviewed-by: Daniel Verkamp <dverkamp@chromium.org>
2021-01-22 22:32:37 +00:00
Dennis Kempin
2118d6faa3 Test Runner for crosvm
The ./run_tests script will select which tests to run on the host.
Unfortunately a lot of our tests require dependencies or access
to certain kernel devices. We cannot run all tests on all hosts.

The test runner works with the CI builders:

To run all x86 tests:
./ci/builder ./run_tests --all

To run tests on aarch64:
./ci/aarch64_builder ./run_tests --all

Or to just run tests that can be run on a standard debian buster
system without obscure chromeos dependencies:

./run_tests

Eventually, the test runner should run tests via the VM of the
aarch64_builder, currently only those tests that can be run with
user-space emulation are executed.

Currently some tests remain disabled in the runner, as they do not
pass in the CI builders yet. These will be enabled as the builders
are set up to allow for them to be run.

BUG=b:173833661
TEST=Ran the above commands.

Change-Id: Id69027793348f4d3f20adf5333d8bdfff1aa9c8b
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/2633889
Tested-by: Dennis Kempin <denniskempin@google.com>
Reviewed-by: Stephen Barber <smbarber@chromium.org>
Reviewed-by: Zach Reizner <zachr@chromium.org>
Commit-Queue: Dennis Kempin <denniskempin@google.com>
2021-01-22 11:05:57 +00:00
Dennis Kempin
191b95b5d7 native and aarch64 cross-compile containers
This CL adds the foundation for running tests consistently
in Kokoro and locally, for both x86 and aarch64.

The crosvm_builder is similar to the original image from
docker/crosvm.Dockerfile.
The main difference is that ChromeOS dependencies are not
compiled into the container, but built at runtime.

The crosvm_aarch64_builder installs the build enviornment
to cross-compile crosvm for aarch64. The tests are run
with user-space emulation using qemu-aarch64-static.

See ci/README.md for instructions on how to use these
builders.

Tests on aarch64 cannot all be run using user-space
emulation. We will need a VM to pass all smoke tests,
this work is tracked in b/177228167.

BUG=b:177133814
TEST=Tested by running
./ci/builder bin/smoke_test
./ci/builder cargo test
./ci/aarch64_builder cargo build
./ci/aarch64_builder cargo test -p tempfile

Change-Id: Iffffcf48894787dd72fff894af351fdaced0b429
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/2621994
Reviewed-by: Zach Reizner <zachr@chromium.org>
Reviewed-by: Stephen Barber <smbarber@chromium.org>
Tested-by: Dennis Kempin <denniskempin@google.com>
Commit-Queue: Dennis Kempin <denniskempin@google.com>
2021-01-20 17:41:27 +00:00