Add a new builder to build crosvm in crOS tree, and all the
depencies of this new builder.
BUG=b:240692674
TESTED=led get-builder luci.crosvm.ci:chromeos_amd64-generic | led edit-cr-cl https://chromium-review.googlesource.com/c/crosvm/crosvm/+/3966928 | led edit-recipe-bundle | led edit -r build_chromeos_hatch | led launch
Change-Id: Id2f284139922916edd2dd584f576da9fb3445518
Reviewed-on: https://chromium-review.googlesource.com/c/crosvm/crosvm/+/3966928
Reviewed-by: Dennis Kempin <denniskempin@google.com>
Commit-Queue: Zihan Chen <zihanchen@google.com>
reland note: Added wineboot for Dockerfile.user, since it prepares
directories for wine on a per-user basis.
To enable podman, the Dockerfile has been split into a root
run Dockerfile and one that adds a non-root user.
The following combinations have been tested:
./tools/dev_container -v --clean --podman --unprivileged
./tools/dev_container -v --clean --podman
./tools/dev_container -v --clean --unprivileged
And warnings have been added to ensure users are aware that
the only fully supported variant is running a privileged
docker container:
./tools/dev_container -v --clean
The unprivileged containers will allow us to validate if
unit tests require privileged system access.
BUG=None
TEST=See above
Change-Id: Ifd70c1e30ef266e39bf517e315dc88fccecc8a62
Reviewed-on: https://chromium-review.googlesource.com/c/crosvm/crosvm/+/3983255
Auto-Submit: Dennis Kempin <denniskempin@google.com>
Reviewed-by: Zihan Chen <zihanchen@google.com>
Commit-Queue: Zihan Chen <zihanchen@google.com>
This reverts commit 6a2b1fda85.
Reason for revert: Broke mingw64 builds
Original change's description:
> dev_container: Fix podman and enable unprivileged containers
>
> To enable podman, the Dockerfile has been split into a root
> run Dockerfile and one that adds a non-root user.
>
> The following combinations have been tested:
>
> ./tools/dev_container -v --clean --podman --unprivileged
> ./tools/dev_container -v --clean --podman
> ./tools/dev_container -v --clean --unprivileged
>
> And warnings have been added to ensure users are aware that
> the only fully supported variant is running a privileged
> docker container:
>
> ./tools/dev_container -v --clean
>
> The unprivileged containers will allow us to validate if
> unit tests require privileged system access.
>
> BUG=None
> TEST=See above
>
> Change-Id: I185b1d9c3829674986305b0e72a39b1a4ba11b98
> Reviewed-on: https://chromium-review.googlesource.com/c/crosvm/crosvm/+/3971029
> Reviewed-by: Zihan Chen <zihanchen@google.com>
> Commit-Queue: Dennis Kempin <denniskempin@google.com>
> Reviewed-by: Dennis Kempin <denniskempin@google.com>
Bug: None
Change-Id: Id57686ed869abcfb54431aa328c54234b9465eb7
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://chromium-review.googlesource.com/c/crosvm/crosvm/+/3979385
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Commit-Queue: Dennis Kempin <denniskempin@google.com>
Auto-Submit: Dennis Kempin <denniskempin@google.com>
To enable podman, the Dockerfile has been split into a root
run Dockerfile and one that adds a non-root user.
The following combinations have been tested:
./tools/dev_container -v --clean --podman --unprivileged
./tools/dev_container -v --clean --podman
./tools/dev_container -v --clean --unprivileged
And warnings have been added to ensure users are aware that
the only fully supported variant is running a privileged
docker container:
./tools/dev_container -v --clean
The unprivileged containers will allow us to validate if
unit tests require privileged system access.
BUG=None
TEST=See above
Change-Id: I185b1d9c3829674986305b0e72a39b1a4ba11b98
Reviewed-on: https://chromium-review.googlesource.com/c/crosvm/crosvm/+/3971029
Reviewed-by: Zihan Chen <zihanchen@google.com>
Commit-Queue: Dennis Kempin <denniskempin@google.com>
Reviewed-by: Dennis Kempin <denniskempin@google.com>
This allows programs to use colors and other tty features. It also
fixes issues with stdout being buffered instead of printed in real
time.
BUG=b:246623045
TEST=./tools/dev_container ./tools/health-check --all -v
Change-Id: I09dd7523a15fbd8b3c5c653471ef091eac75cf6d
Reviewed-on: https://chromium-review.googlesource.com/c/crosvm/crosvm/+/3892522
Commit-Queue: Dennis Kempin <denniskempin@google.com>
Reviewed-by: Daniel Verkamp <dverkamp@chromium.org>
While we are tweaking all of the copyright headers, let's take the
opportunity to ensure there is always a blank line after the copyright
header for consistency. (Almost all files already follow this style.)
This includes a slightly ugly regex to allow the end of a C-style
comment block after the end of the copyright:
/*
* Example comment block
*/ <-- this line
Change-Id: Idfd0855861e5ecb3d33afae942fdba908af0dcff
Reviewed-on: https://chromium-review.googlesource.com/c/crosvm/crosvm/+/3892521
Commit-Queue: Daniel Verkamp <dverkamp@chromium.org>
Reviewed-by: Dennis Kempin <denniskempin@google.com>
This search/replace updates all copyright notices to drop the
"All rights reserved", Use "ChromiumOS" instead of "Chromium OS"
and drops the trailing dots.
This fulfills the request from legal and unifies our notices.
./tools/health-check has been updated to only accept this style.
BUG=b:246579983
TEST=./tools/health-check
Change-Id: I87a80701dc651f1baf4820e5cc42469d7c5f5bf7
Reviewed-on: https://chromium-review.googlesource.com/c/crosvm/crosvm/+/3894243
Reviewed-by: Daniel Verkamp <dverkamp@chromium.org>
Commit-Queue: Dennis Kempin <denniskempin@google.com>
Makes a temporary copy of the git repo for each test so we can
modify the repo for testing purposes.
It's not the fastest test (~12s), but is only run when python files
have been modified. In contrast to other developer tooling, tools/cl
is not used by Luci, so it needs a dedicated test.
BUG=b:244185215
TEST=./tools/health-check
Change-Id: I06c90a580aa8ed0fa267a41ca40895710121767f
Reviewed-on: https://chromium-review.googlesource.com/c/crosvm/crosvm/+/3866692
Tested-by: Dennis Kempin <denniskempin@google.com>
Reviewed-by: Daniel Verkamp <dverkamp@chromium.org>
Commit-Queue: Dennis Kempin <denniskempin@google.com>
Container shell now defaults to user `crosvmdev` which will have
same UID & GID as outside user. VS Code Remote Containers extension
has the same behavior when loading config from devcontainer.json.
TEST=all tests pass in container, new files in container have same
permission as files created outside.
FIXED=b:239476013
Change-Id: Ib3c696509e9ea45edd5f02ff025d9477576ec765
Reviewed-on: https://chromium-review.googlesource.com/c/crosvm/crosvm/+/3814094
Auto-Submit: Zihan Chen <zihanchen@google.com>
Reviewed-by: Dennis Kempin <denniskempin@google.com>
Tested-by: Zihan Chen <zihanchen@google.com>
Commit-Queue: Dennis Kempin <denniskempin@google.com>
See go/crosvm/infra for instructions on how to get access to uploading
them.
Added a helper script to install dependencies needed to build the
guest_under_test.
BUG=b:235269312
TEST=Kokoro
Change-Id: I78387a33ddbf3ab199b36e76ba617acb1250c7e5
Reviewed-on: https://chromium-review.googlesource.com/c/crosvm/crosvm/+/3783011
Reviewed-by: Daniel Verkamp <dverkamp@chromium.org>
Commit-Queue: Dennis Kempin <denniskempin@google.com>
Tested-by: Dennis Kempin <denniskempin@google.com>
CRC32 of checkout's path is now part of dev-container's instance
name.
TEST=`docker ps` shows crc of script's absolute path in checkout
FIXED=b:237592974
Change-Id: I7179c40e5c8436a5866d5d75b01696dc788725c9
Reviewed-on: https://chromium-review.googlesource.com/c/crosvm/crosvm/+/3790038
Reviewed-by: Dennis Kempin <denniskempin@google.com>
Commit-Queue: Dennis Kempin <denniskempin@google.com>
Tested-by: Zihan Chen <zihanchen@google.com>
Auto-Submit: Zihan Chen <zihanchen@google.com>
Mount the cros root instead so the git worktree or symlink can be
resolved.
BUG=b:234322274
TEST=./tools/dev_container
Change-Id: I52d0cbe5e9a38cb6febcf1f328970e13209bcf9f
Reviewed-on: https://chromium-review.googlesource.com/c/crosvm/crosvm/+/3768048
Tested-by: Dennis Kempin <denniskempin@google.com>
Reviewed-by: Daniel Verkamp <dverkamp@chromium.org>
Commit-Queue: Dennis Kempin <denniskempin@google.com>
Auto-Submit: Dennis Kempin <denniskempin@google.com>
This Allows Luci builders to cache cargo home between builds. So we
do not have to download so many third party crates with each build.
CARGO_HOME is specifically intended to be cached in CI systems.
BUG=b:233230027
TEST=CROSVM_CONTAINER_CACHE=/tmp/test ./tools/dev_container --clean
cargo build
Change-Id: I11580c5ed3151519ece4a651cb22d059c7c3eb87
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/3739368
Reviewed-by: Daniel Verkamp <dverkamp@chromium.org>
Tested-by: kokoro <noreply+kokoro@google.com>
Commit-Queue: Dennis Kempin <denniskempin@google.com>
Some recent changes introduced a bug that breaks reusage of
containers. Instead of trying to restart a stopped container,
we delete it instead and start a fresh one.
The code was refactored to allow for testing. The tests run
against the real docker service and are run as part of the health
checks in kokoro / luci.
This CL also gets rid of the custon command line parsing since
luci can now run it with argh available.
BUG=b:234402839
TEST=./tools/dev_containes --self-test [--podman]
Change-Id: Id5eaf0ea83ff07433f8f57cb652b5c393dcb9da3
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/3681399
Reviewed-by: Anton Romanov <romanton@google.com>
Tested-by: kokoro <noreply+kokoro@google.com>
Otherwise we may exit thinking we are trying to run an interactive
command in a non-interactive terminal.
BUG=None
TEST=./tools/dev_container --stop </dev/null
Change-Id: Ie34cf662b82090b100533306572552c5a550211e
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/3671132
Reviewed-by: Daniel Verkamp <dverkamp@chromium.org>
Tested-by: kokoro <noreply+kokoro@google.com>
We run a neverending command in the container to keep it alive in the
background.
By default, this would be bash, which does the job when --tty is
specified. But we no longer do that all the time.
Instead we can use sleep, which does not rely on such tricks.
BUG=b:233913455
TEST=./tools/dev_container true
Change-Id: I2d04aabbef6b670daf6b94fdc2c65c7bd759c61b
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/3669682
Tested-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: Daniel Verkamp <dverkamp@chromium.org>
They have been dropped with the recent refactoring to not use argh.
BUG=None
TEST=./tools/dev_container -v
Change-Id: Ifa65b14d316e27dbb5cef9714fbd3adf110b79e6
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/3664051
Commit-Queue: Dennis Kempin <denniskempin@google.com>
Reviewed-by: Daniel Verkamp <dverkamp@chromium.org>
Tested-by: kokoro <noreply+kokoro@google.com>
By passing main(..., podman=args.podman), we mask the function's default
value for that parameter which is False when --podman isn't passed. This
doesn't match what 34811a88ee ("dev_container: Add --podman command
line parameter") says about the flag:
The parameter [--podman] defaults to true if docker is not installed
Instead, use the computed default for the argument itself.
BUG=b:233346153
TEST=tools/dev_container bash -c 'echo Hello'
Change-Id: I2e5f9855853279a8ceeb9347931d59fd334f0004
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/3655968
Reviewed-by: Dennis Kempin <denniskempin@google.com>
Auto-Submit: Pierre-Clément Tosi <ptosi@google.com>
Tested-by: kokoro <noreply+kokoro@google.com>
Commit-Queue: Dennis Kempin <denniskempin@google.com>
This script is naturally used outside of the container and we
want to avoid having to install dependencies or special python
versions to run it.
This CL reduces the python requirement from 3.9 to 3.8 and
allows the script to run without any dependencies.
BUG=b:233230344
TEST=pyenv global 3.8.15
./tools/dev_container
Change-Id: Ieeb868f7fb275a8e6e39e577b5e984895307765e
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/3657816
Tested-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: Anton Romanov <romanton@google.com>
Sometimes we are left with an existing container that has been stopped.
Usually happens after reboots.
Make sure we start the container again before trying to execute.
BUG=None
TEST=docker stop $USER_dev_container
./tools/dev_container
Change-Id: Ib68091a09546cfefa452ddb8e6f268e495f2167c
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/3657815
Tested-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: Anton Romanov <romanton@google.com>
The parameter defaults to true if docker is not installed. This also
resolves previous issues with resolving which docker command to use.
A warning is added when executing dev_container with podman.
BUG=b:233346153
TEST=./tools/dev_container --podman
Change-Id: I1d3dad6a657c0273c82634e409a0991245f56f4d
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/3657814
Reviewed-by: Anton Romanov <romanton@google.com>
Tested-by: kokoro <noreply+kokoro@google.com>
This fixes problems with freezing when running luci recipes locally.
An interactive process can still be forced by flag -i or --interactive.
BUG=b:233230344
TEST=cd infra && ./recipes.py run build_linux
Change-Id: I261f0ffddcc3795e17bcbe023fa920f48769fdb4
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/3657813
Reviewed-by: Anton Romanov <romanton@google.com>
Tested-by: kokoro <noreply+kokoro@google.com>
This change adds python type and formatting checks and
consolidates code health checks in ./tools/health-check.
Dealing with relative imports in python is tricky, so
we are making ./tools/impl a proper package with no
directly executable files.
Some of the bash shorthands in ./tools had to be converted
to python for this.
To make the new checks pass, we run the formatter and fix
some mypy type checks.
TEST=./tools/health-check
BUG=b:218559722,b:219965702
Change-Id: Ie18d3d6dd2f5a033141e167a6e1aa762791941d6
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/3558592
Reviewed-by: Daniel Verkamp <dverkamp@chromium.org>
Tested-by: kokoro <noreply+kokoro@google.com>
Commit-Queue: Dennis Kempin <denniskempin@google.com>
When the container image is updated, developers may not notice and
continue running a different version. Detect this and restart the
container.
BUG=b:217465164
TEST=./tools/dev_container while changing image_version
Change-Id: Ib1df871c5c6fba9421c436d39ab7065be966f41e
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/3469050
Reviewed-by: Daniel Verkamp <dverkamp@chromium.org>
Tested-by: kokoro <noreply+kokoro@google.com>
Commit-Queue: Dennis Kempin <denniskempin@google.com>
Use the new common.py utilities to make turn this bash script into
python. The behavior should be the same.
BUG=b:218559770
TEST=./tools/presubmit
Change-Id: Id1412c2076089fd21f280959d6cde1f4cb64e163
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/3469049
Reviewed-by: Daniel Verkamp <dverkamp@chromium.org>
Tested-by: kokoro <noreply+kokoro@google.com>
Commit-Queue: Dennis Kempin <denniskempin@google.com>
This works around quirkiness of sparse file support in overlayfs.
Revert "sys_util: ignore seek_hole tests for kokoro uprev"
This reverts commit 3d0e51f71c.
BUG=b:208901617
TEST=./tools/dev_container --hermetic bash -c "cd common/sys_util; cargo test seek_"
Change-Id: Id4d809f09a71b5cd134b5eb9bf6a5f970e5503c8
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/3319404
Tested-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: Anton Romanov <romanton@google.com>
Reviewed-by: Dennis Kempin <denniskempin@google.com>
Commit-Queue: Anton Romanov <romanton@google.com>
Commit 95b80d1 made the dev container persistent between invocations,
but hard-coded `docker` again. This change fixes that and also adds
further improvements:
- Do not use `--privileged` with podman. If the rootless user has
permissions to access `/dev/kvm`, so will the container.
- Map `/dev/vhost-net` and `/dev/vhost-vsock` as well.
- Use `BASH_SOURCE` to find this script's directory. As we're using Bash
to start with, this is more robust than using plain `$0`.
BUG=None
TEST=Run `./tools/dev_container cargo build` with Podman and Docker
Change-Id: I05c699f327c8e1c4f3c4df9679ee92bf7e609e2e
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/3295372
Reviewed-by: Dennis Kempin <denniskempin@google.com>
Tested-by: kokoro <noreply+kokoro@google.com>
Commit-Queue: Christian Blichmann <cblichmann@google.com>
This vastly improves iterative build times and enables more flexible
usage of the container.
BUG=None
TEST=./tools/dev_container cargo build
First run will build everything. Second run will finish right away.
Change-Id: I9b4eeee0689f0e9d07f0a32f846d21ab42f689f7
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/3292100
Tested-by: kokoro <noreply+kokoro@google.com>
Commit-Queue: Dennis Kempin <denniskempin@google.com>
Reviewed-by: Junichi Uekawa <uekawa@chromium.org>
This change allows to override the command line tool that is used to
start the dev container in the `tools/dev_container` script. The
Makefile for creating the dev container uses the `DOCKER` env var to
implement this, so this is used here as well.
Rationale: Googlers are advised to not install Docker
(go/dont-install-docker) and to use Podman instead.
If the `DOCKER` variable is unset, the script will try to use
Docker first and podman.
Change-Id: I33bc8e4af632fa982406e838e762572407ccdc22
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/3259939
Reviewed-by: Dennis Kempin <denniskempin@google.com>
Tested-by: kokoro <noreply+kokoro@google.com>
Commit-Queue: Christian Blichmann <cblichmann@google.com>
The new container just provides a standard debian box with a
rust toolchain.
Installation of dependencies has been extracted into scripts
which can also be used to set up a machine for development.
This will replace the current ./ci/ containers.
BUG=b:199950887,b:199950423
TEST=./tools/dev_container ./tools/run_tests
Change-Id: I832bc5b129246923df937a34614b4d74955304dc
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/3221781
Tested-by: kokoro <noreply+kokoro@google.com>
Commit-Queue: Dennis Kempin <denniskempin@google.com>
Reviewed-by: Junichi Uekawa <uekawa@chromium.org>
