infra: Initial setup of linux pre/post-submit builders

The builders use the existing dev_container to run tests
for the 3 currently supported architectures on linux:
x86_64, aarch64 and armhf.

A new recipe 'build_linux' has been added runs the
dev_container to build and execute tests.

To share code with other upcoming recipes, some of the
recipe code has been extracted into a shared crosvm
module.

Basic tests for the recipes have been added.

BUG=b:233230344
TEST=./recipes.py test run
./recipes.py run build_linux

Change-Id: I17ecb25c0a0eabdce56537831454ac22d4dc7021
Reviewed-on: https://chromium-review.googlesource.com/c/chromiumos/platform/crosvm/+/3654196
Tested-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: Daniel Verkamp <dverkamp@chromium.org>
This commit is contained in:
Dennis Kempin 2022-05-18 17:06:46 +00:00
parent 86939a1337
commit 0948bc8d3a
23 changed files with 761 additions and 423 deletions

1
.gitignore vendored
View file

@ -4,6 +4,7 @@ target/
**/*.sw[po] **/*.sw[po]
**/*.orig **/*.orig
**/Cargo.lock **/Cargo.lock
**/*.pyc
lcov.info lcov.info
.idea .idea
.vscode .vscode

View file

@ -7,3 +7,16 @@ See [Kokoro](../ci/kokoro) configs for the actively used presubmit system.
Note: Luci applies config and recipes changes asynchronously. Do not submit changes to this Note: Luci applies config and recipes changes asynchronously. Do not submit changes to this
directory in the same commit as changes to other crosvm source. directory in the same commit as changes to other crosvm source.
## Recipe Documentation
A few links to relevant documentation needed to write recipes:
- [Recipe Engine](https://chromium.googlesource.com/infra/luci/recipes-py.git/+/HEAD/README.recipes.md)
- [Depot Tools Recipes](https://chromium.googlesource.com/chromium/tools/depot_tools.git/+/HEAD/README.recipes.md))
- [ChromiumOS Recipes](https://chromium.googlesource.com/chromiumos/infra/recipes.git/+/HEAD/README.recipes.md)
Luci also provides a
[User Guide](https://chromium.googlesource.com/infra/luci/recipes-py/+/master/doc/user_guide.md) and
[Walkthrough](https://chromium.googlesource.com/infra/luci/recipes-py/+/refs/heads/main/doc/walkthrough.md)
for getting started with recipes.

View file

@ -2,29 +2,73 @@
# Repo documentation for [crosvm](https://chromium.googlesource.com/crosvm/crosvm.git) # Repo documentation for [crosvm](https://chromium.googlesource.com/crosvm/crosvm.git)
## Table of Contents ## Table of Contents
**[Recipe Modules](#Recipe-Modules)**
* [crosvm](#recipe_modules-crosvm) (Python3 ✅)
**[Recipes](#Recipes)** **[Recipes](#Recipes)**
* [hello_world](#recipes-hello_world) * [build_linux](#recipes-build_linux) (Python3 ✅)
* [verify_cl](#recipes-verify_cl) (Python3 ✅) * [crosvm:examples/prepare_source](#recipes-crosvm_examples_prepare_source) (Python3 ✅)
## Recipes * [crosvm:examples/step_in_container](#recipes-crosvm_examples_step_in_container) (Python3 ✅)
## Recipe Modules
### *recipes* / [hello\_world](/infra/recipes/hello_world.py) ### *recipe_modules* / [crosvm](/infra/recipe_modules/crosvm)
[DEPS](/infra/recipes/hello_world.py#7): [recipe\_engine/step][recipe_engine/recipe_modules/step] [DEPS](/infra/recipe_modules/crosvm/__init__.py#7): [depot\_tools/bot\_update][depot_tools/recipe_modules/bot_update], [depot\_tools/gclient][depot_tools/recipe_modules/gclient], [recipe\_engine/buildbucket][recipe_engine/recipe_modules/buildbucket], [recipe\_engine/context][recipe_engine/recipe_modules/context], [recipe\_engine/file][recipe_engine/recipe_modules/file], [recipe\_engine/path][recipe_engine/recipe_modules/path], [recipe\_engine/step][recipe_engine/recipe_modules/step]
PYTHON_VERSION_COMPATIBILITY: PY2
&mdash; **def [RunSteps](/infra/recipes/hello_world.py#12)(api):**
### *recipes* / [verify\_cl](/infra/recipes/verify_cl.py)
[DEPS](/infra/recipes/verify_cl.py#3): [depot\_tools/bot\_update][depot_tools/recipe_modules/bot_update], [depot\_tools/gclient][depot_tools/recipe_modules/gclient], [recipe\_engine/buildbucket][recipe_engine/recipe_modules/buildbucket], [recipe\_engine/context][recipe_engine/recipe_modules/context], [recipe\_engine/path][recipe_engine/recipe_modules/path]
PYTHON_VERSION_COMPATIBILITY: PY3 PYTHON_VERSION_COMPATIBILITY: PY3
&mdash; **def [RunSteps](/infra/recipes/verify_cl.py#12)(api):** #### **class [CrosvmApi](/infra/recipe_modules/crosvm/api.py#10)([RecipeApi][recipe_engine/wkt/RecipeApi]):**
Crosvm specific functionality shared between recipes.
&emsp; **@property**<br>&mdash; **def [builder\_dir](/infra/recipe_modules/crosvm/api.py#17)(self):**
&mdash; **def [prepare\_source](/infra/recipe_modules/crosvm/api.py#21)(self):**
Prepares the local crosvm source for testing in `self.source_dir`
CI jobs will check out the revision to be tested, try jobs will check out the gerrit
change to be tested.
&emsp; **@property**<br>&mdash; **def [source\_dir](/infra/recipe_modules/crosvm/api.py#13)(self):**
&mdash; **def [step\_in\_container](/infra/recipe_modules/crosvm/api.py#41)(self, step_name, command):**
Runs a luci step inside the crosvm dev container.
## Recipes
### *recipes* / [build\_linux](/infra/recipes/build_linux.py)
[DEPS](/infra/recipes/build_linux.py#11): [crosvm](#recipe_modules-crosvm), [recipe\_engine/buildbucket][recipe_engine/recipe_modules/buildbucket], [recipe\_engine/context][recipe_engine/recipe_modules/context], [recipe\_engine/properties][recipe_engine/recipe_modules/properties], [recipe\_engine/step][recipe_engine/recipe_modules/step]
PYTHON_VERSION_COMPATIBILITY: PY3
&mdash; **def [RunSteps](/infra/recipes/build_linux.py#35)(api, properties):**
&mdash; **def [get\_test\_args](/infra/recipes/build_linux.py#22)(api, test_arch):**
Returns architecture specific arguments for ./tools/run_tests
### *recipes* / [crosvm:examples/prepare\_source](/infra/recipe_modules/crosvm/examples/prepare_source.py)
[DEPS](/infra/recipe_modules/crosvm/examples/prepare_source.py#12): [crosvm](#recipe_modules-crosvm), [recipe\_engine/buildbucket][recipe_engine/recipe_modules/buildbucket]
PYTHON_VERSION_COMPATIBILITY: PY3
&mdash; **def [RunSteps](/infra/recipe_modules/crosvm/examples/prepare_source.py#18)(api):**
### *recipes* / [crosvm:examples/step\_in\_container](/infra/recipe_modules/crosvm/examples/step_in_container.py)
[DEPS](/infra/recipe_modules/crosvm/examples/step_in_container.py#7): [crosvm](#recipe_modules-crosvm)
PYTHON_VERSION_COMPATIBILITY: PY3
&mdash; **def [RunSteps](/infra/recipe_modules/crosvm/examples/step_in_container.py#12)(api):**
[depot_tools/recipe_modules/bot_update]: https://chromium.googlesource.com/chromium/tools/depot_tools.git/+/8a87603683bda769d437e48cc1a7494a2e237ead/recipes/README.recipes.md#recipe_modules-bot_update [depot_tools/recipe_modules/bot_update]: https://chromium.googlesource.com/chromium/tools/depot_tools.git/+/8a87603683bda769d437e48cc1a7494a2e237ead/recipes/README.recipes.md#recipe_modules-bot_update
[depot_tools/recipe_modules/gclient]: https://chromium.googlesource.com/chromium/tools/depot_tools.git/+/8a87603683bda769d437e48cc1a7494a2e237ead/recipes/README.recipes.md#recipe_modules-gclient [depot_tools/recipe_modules/gclient]: https://chromium.googlesource.com/chromium/tools/depot_tools.git/+/8a87603683bda769d437e48cc1a7494a2e237ead/recipes/README.recipes.md#recipe_modules-gclient
[recipe_engine/recipe_modules/buildbucket]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/7b42800366a15f34b28e62f6bcb1cddcb2206db0/README.recipes.md#recipe_modules-buildbucket [recipe_engine/recipe_modules/buildbucket]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/7b42800366a15f34b28e62f6bcb1cddcb2206db0/README.recipes.md#recipe_modules-buildbucket
[recipe_engine/recipe_modules/context]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/7b42800366a15f34b28e62f6bcb1cddcb2206db0/README.recipes.md#recipe_modules-context [recipe_engine/recipe_modules/context]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/7b42800366a15f34b28e62f6bcb1cddcb2206db0/README.recipes.md#recipe_modules-context
[recipe_engine/recipe_modules/file]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/7b42800366a15f34b28e62f6bcb1cddcb2206db0/README.recipes.md#recipe_modules-file
[recipe_engine/recipe_modules/path]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/7b42800366a15f34b28e62f6bcb1cddcb2206db0/README.recipes.md#recipe_modules-path [recipe_engine/recipe_modules/path]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/7b42800366a15f34b28e62f6bcb1cddcb2206db0/README.recipes.md#recipe_modules-path
[recipe_engine/recipe_modules/properties]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/7b42800366a15f34b28e62f6bcb1cddcb2206db0/README.recipes.md#recipe_modules-properties
[recipe_engine/recipe_modules/step]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/7b42800366a15f34b28e62f6bcb1cddcb2206db0/README.recipes.md#recipe_modules-step [recipe_engine/recipe_modules/step]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/7b42800366a15f34b28e62f6bcb1cddcb2206db0/README.recipes.md#recipe_modules-step
[recipe_engine/wkt/RecipeApi]: https://chromium.googlesource.com/infra/luci/recipes-py.git/+/7b42800366a15f34b28e62f6bcb1cddcb2206db0/recipe_engine/recipe_api.py#883

View file

@ -6,7 +6,7 @@
cq_status_host: "chromium-cq-status.appspot.com" cq_status_host: "chromium-cq-status.appspot.com"
config_groups { config_groups {
name: "main_repo" name: "main"
gerrit { gerrit {
url: "https://chromium-review.googlesource.com" url: "https://chromium-review.googlesource.com"
projects { projects {
@ -20,7 +20,13 @@ config_groups {
} }
tryjob { tryjob {
builders { builders {
name: "crosvm/try/Verify CL" name: "crosvm/try/crosvm_linux_aarch64"
}
builders {
name: "crosvm/try/crosvm_linux_armhf"
}
builders {
name: "crosvm/try/crosvm_linux_x86_64"
} }
retry_config { retry_config {
single_quota: 1 single_quota: 1

View file

@ -15,25 +15,7 @@ buckets {
} }
swarming { swarming {
builders { builders {
name: "Example Builder" name: "crosvm_linux_aarch64"
swarming_host: "chromium-swarm.appspot.com"
exe {
cipd_package: "infra/recipe_bundles/chromium.googlesource.com/crosvm/crosvm"
cipd_version: "refs/heads/main"
cmd: "luciexe"
}
properties:
'{'
' "recipe": "hello_world"'
'}'
service_account: "crosvm-luci-ci-builder@crosvm-infra.iam.gserviceaccount.com"
experiments {
key: "luci.recipes.use_python3"
value: 100
}
}
builders {
name: "Verify CL"
swarming_host: "chromium-swarm.appspot.com" swarming_host: "chromium-swarm.appspot.com"
dimensions: "cpu:x86-64" dimensions: "cpu:x86-64"
dimensions: "os:Ubuntu" dimensions: "os:Ubuntu"
@ -45,7 +27,52 @@ buckets {
} }
properties: properties:
'{' '{'
' "recipe": "verify_cl"' ' "recipe": "build_linux",'
' "test_arch": "aarch64"'
'}'
service_account: "crosvm-luci-ci-builder@crosvm-infra.iam.gserviceaccount.com"
experiments {
key: "luci.recipes.use_python3"
value: 100
}
}
builders {
name: "crosvm_linux_armhf"
swarming_host: "chromium-swarm.appspot.com"
dimensions: "cpu:x86-64"
dimensions: "os:Ubuntu"
dimensions: "pool:luci.crosvm.ci"
exe {
cipd_package: "infra/recipe_bundles/chromium.googlesource.com/crosvm/crosvm"
cipd_version: "refs/heads/main"
cmd: "luciexe"
}
properties:
'{'
' "recipe": "build_linux",'
' "test_arch": "armhf"'
'}'
service_account: "crosvm-luci-ci-builder@crosvm-infra.iam.gserviceaccount.com"
experiments {
key: "luci.recipes.use_python3"
value: 100
}
}
builders {
name: "crosvm_linux_x86_64"
swarming_host: "chromium-swarm.appspot.com"
dimensions: "cpu:x86-64"
dimensions: "os:Ubuntu"
dimensions: "pool:luci.crosvm.ci"
exe {
cipd_package: "infra/recipe_bundles/chromium.googlesource.com/crosvm/crosvm"
cipd_version: "refs/heads/main"
cmd: "luciexe"
}
properties:
'{'
' "recipe": "build_linux",'
' "test_arch": "x86_64"'
'}' '}'
service_account: "crosvm-luci-ci-builder@crosvm-infra.iam.gserviceaccount.com" service_account: "crosvm-luci-ci-builder@crosvm-infra.iam.gserviceaccount.com"
experiments { experiments {
@ -68,7 +95,7 @@ buckets {
} }
swarming { swarming {
builders { builders {
name: "Verify CL" name: "crosvm_linux_aarch64"
swarming_host: "chromium-swarm.appspot.com" swarming_host: "chromium-swarm.appspot.com"
dimensions: "cpu:x86-64" dimensions: "cpu:x86-64"
dimensions: "os:Ubuntu" dimensions: "os:Ubuntu"
@ -80,7 +107,52 @@ buckets {
} }
properties: properties:
'{' '{'
' "recipe": "verify_cl"' ' "recipe": "build_linux",'
' "test_arch": "aarch64"'
'}'
service_account: "crosvm-luci-try-builder@crosvm-infra.iam.gserviceaccount.com"
experiments {
key: "luci.recipes.use_python3"
value: 100
}
}
builders {
name: "crosvm_linux_armhf"
swarming_host: "chromium-swarm.appspot.com"
dimensions: "cpu:x86-64"
dimensions: "os:Ubuntu"
dimensions: "pool:luci.crosvm.try"
exe {
cipd_package: "infra/recipe_bundles/chromium.googlesource.com/crosvm/crosvm"
cipd_version: "refs/heads/main"
cmd: "luciexe"
}
properties:
'{'
' "recipe": "build_linux",'
' "test_arch": "armhf"'
'}'
service_account: "crosvm-luci-try-builder@crosvm-infra.iam.gserviceaccount.com"
experiments {
key: "luci.recipes.use_python3"
value: 100
}
}
builders {
name: "crosvm_linux_x86_64"
swarming_host: "chromium-swarm.appspot.com"
dimensions: "cpu:x86-64"
dimensions: "os:Ubuntu"
dimensions: "pool:luci.crosvm.try"
exe {
cipd_package: "infra/recipe_bundles/chromium.googlesource.com/crosvm/crosvm"
cipd_version: "refs/heads/main"
cmd: "luciexe"
}
properties:
'{'
' "recipe": "build_linux",'
' "test_arch": "x86_64"'
'}' '}'
service_account: "crosvm-luci-try-builder@crosvm-infra.iam.gserviceaccount.com" service_account: "crosvm-luci-try-builder@crosvm-infra.iam.gserviceaccount.com"
experiments { experiments {

View file

@ -5,12 +5,21 @@
# https://luci-config.appspot.com/schemas/projects:luci-milo.cfg # https://luci-config.appspot.com/schemas/projects:luci-milo.cfg
consoles { consoles {
id: "CI builders" id: "CI Console"
name: "CI builders" name: "CI Console"
repo_url: "https://chromium.googlesource.com/crosvm/crosvm" repo_url: "https://chromium.googlesource.com/crosvm/crosvm"
refs: "regexp:refs/heads/main" refs: "regexp:refs/heads/main"
manifest_name: "REVISION" manifest_name: "REVISION"
builders { builders {
name: "buildbucket/luci.crosvm.ci/Verify CL" name: "buildbucket/luci.crosvm.ci/crosvm_linux_x86_64"
category: "linux"
}
builders {
name: "buildbucket/luci.crosvm.ci/crosvm_linux_aarch64"
category: "linux"
}
builders {
name: "buildbucket/luci.crosvm.ci/crosvm_linux_armhf"
category: "linux"
} }
} }

View file

@ -4,10 +4,43 @@
# For the schema of this file, see ProjectConfig message: # For the schema of this file, see ProjectConfig message:
# https://luci-config.appspot.com/schemas/projects:luci-scheduler.cfg # https://luci-config.appspot.com/schemas/projects:luci-scheduler.cfg
job {
id: "crosvm_linux_aarch64"
realm: "ci"
acl_sets: "ci"
buildbucket {
server: "cr-buildbucket.appspot.com"
bucket: "ci"
builder: "crosvm_linux_aarch64"
}
}
job {
id: "crosvm_linux_armhf"
realm: "ci"
acl_sets: "ci"
buildbucket {
server: "cr-buildbucket.appspot.com"
bucket: "ci"
builder: "crosvm_linux_armhf"
}
}
job {
id: "crosvm_linux_x86_64"
realm: "ci"
acl_sets: "ci"
buildbucket {
server: "cr-buildbucket.appspot.com"
bucket: "ci"
builder: "crosvm_linux_x86_64"
}
}
trigger { trigger {
id: "main source" id: "main source"
realm: "ci" realm: "ci"
acl_sets: "ci" acl_sets: "ci"
triggers: "crosvm_linux_aarch64"
triggers: "crosvm_linux_armhf"
triggers: "crosvm_linux_x86_64"
gitiles { gitiles {
repo: "https://chromium.googlesource.com/crosvm/crosvm" repo: "https://chromium.googlesource.com/crosvm/crosvm"
refs: "regexp:refs/heads/main" refs: "regexp:refs/heads/main"

View file

@ -114,66 +114,91 @@ luci.bucket(name = "prod")
# as the recipe bundler compiles them from your refs/heads/main branch. # as the recipe bundler compiles them from your refs/heads/main branch.
cipd_version = "refs/heads/main" cipd_version = "refs/heads/main"
# Example builder to verify configuration
luci.builder(
name = "Example Builder",
bucket = "ci",
executable = luci.recipe(
name = "hello_world",
),
schedule = None,
service_account = "crosvm-luci-ci-builder@crosvm-infra.iam.gserviceaccount.com",
)
# Create builders for change verifier
def verify_builder(bucket):
luci.builder(
name = "Verify CL",
bucket = bucket,
executable = luci.recipe(
name = "verify_cl",
),
service_account = "crosvm-luci-%s-builder@crosvm-infra.iam.gserviceaccount.com" % bucket,
dimensions = {
"os": "Ubuntu",
"cpu": "x86-64",
"pool": "luci.crosvm.%s" % bucket,
},
)
verify_builder("try")
verify_builder("ci")
# Configure Change Verifier to watch crosvm # Configure Change Verifier to watch crosvm
luci.cq( luci.cq(
status_host = "chromium-cq-status.appspot.com", status_host = "chromium-cq-status.appspot.com",
) )
luci.cq_group( luci.cq_group(
name = "main_repo", name = "main",
watch = cq.refset( watch = cq.refset(
repo = "https://chromium.googlesource.com/crosvm/crosvm", repo = "https://chromium.googlesource.com/crosvm/crosvm",
refs = ["refs/heads/.+"], # will watch all branches refs = ["refs/heads/.+"], # will watch all branches
), ),
) )
# Attach our "Verify CL" builder to this CQ group. # Configure postsubmit tests running in ci pool
luci.cq_tryjob_verifier( luci.console_view(
builder = "try/Verify CL", name = "CI Console",
cq_group = "main_repo", repo = "https://chromium.googlesource.com/crosvm/crosvm",
) )
# Configure postsubmit tests running in ci pool def verify_builder(name, dimensions, **args):
luci.gitiles_poller( """Creates both a CI and try builder with the same properties.
name = "main source",
bucket = "ci", The CI builder is attached to the gitlies poller and console view, and the try builder
repo = "https://chromium.googlesource.com/crosvm/crosvm", is added to the change verifier.
# triggers = ["ci/Verify CL"],
) Args:
luci.console_view( name: Name of the builder
name = "CI builders", dimensions: Passed to luci.builder
repo = "https://chromium.googlesource.com/crosvm/crosvm", **args: Passed to luci.builder
entries = [ """
luci.console_view_entry(builder = "ci/Verify CL"),
], # CI builder
) luci.builder(
name = name,
bucket = "ci",
service_account = "crosvm-luci-ci-builder@crosvm-infra.iam.gserviceaccount.com",
dimensions = dict(pool = "luci.crosvm.ci", **dimensions),
**args
)
luci.gitiles_poller(
name = "main source",
bucket = "ci",
repo = "https://chromium.googlesource.com/crosvm/crosvm",
triggers = ["ci/%s" % name],
)
luci.console_view_entry(
console_view = "CI Console",
builder = "ci/%s" % name,
category = "linux",
)
# Try builder
luci.builder(
name = name,
bucket = "try",
service_account = "crosvm-luci-try-builder@crosvm-infra.iam.gserviceaccount.com",
dimensions = dict(pool = "luci.crosvm.try", **dimensions),
**args
)
# Attach try builder to Change Verifier
luci.cq_tryjob_verifier(
builder = "try/%s" % name,
cq_group = "main",
)
def verify_linux_builder(arch):
"""Creates a verify builder that builds crosvm on linux
Args:
arch: Architecture to build and test
"""
verify_builder(
name = "crosvm_linux_%s" % arch,
dimensions = {
"os": "Ubuntu",
"cpu": "x86-64",
},
executable = luci.recipe(
name = "build_linux",
),
properties = {
"test_arch": arch,
},
)
verify_linux_builder("x86_64")
verify_linux_builder("aarch64")
verify_linux_builder("armhf")

View file

@ -0,0 +1,15 @@
# Copyright 2022 The Chromium OS Authors. All rights reserved.
# Use of this source code is governed by a BSD-style license that can be
# found in the LICENSE file.
PYTHON_VERSION_COMPATIBILITY = "PY3"
DEPS = [
"depot_tools/bot_update",
"depot_tools/gclient",
"recipe_engine/buildbucket",
"recipe_engine/context",
"recipe_engine/file",
"recipe_engine/path",
"recipe_engine/step",
]

View file

@ -0,0 +1,53 @@
# Copyright 2022 The Chromium OS Authors. All rights reserved.
# Use of this source code is governed by a BSD-style license that can be
# found in the LICENSE file.
from recipe_engine import recipe_api
CROSVM_REPO_URL = "https://chromium.googlesource.com/crosvm/crosvm"
class CrosvmApi(recipe_api.RecipeApi):
"Crosvm specific functionality shared between recipes."
@property
def source_dir(self):
return self.builder_dir.join("crosvm")
@property
def builder_dir(self):
return self.m.path["cache"].join("builder")
def prepare_source(self):
"""
Prepares the local crosvm source for testing in `self.source_dir`
CI jobs will check out the revision to be tested, try jobs will check out the gerrit
change to be tested.
"""
self.m.file.ensure_directory("Ensure builder_dir exists", self.builder_dir)
with self.m.context(cwd=self.builder_dir):
gclient_config = self.m.gclient.make_config()
s = gclient_config.solutions.add()
s.url = CROSVM_REPO_URL
s.name = "crosvm"
gclient_config.got_revision_mapping[s.name] = "got_revision"
self.m.bot_update.ensure_checkout(gclient_config=gclient_config)
with self.m.context(cwd=self.source_dir):
self.m.step("Sync Submodules", ["git", "submodule", "update", "--init"])
def step_in_container(self, step_name, command):
"""
Runs a luci step inside the crosvm dev container.
"""
self.m.step(
step_name,
[
"./tools/dev_container",
"--verbose",
"--hermetic",
]
+ command,
)

View file

@ -0,0 +1,41 @@
# Copyright 2022 The Chromium OS Authors. All rights reserved.
# Use of this source code is governed by a BSD-style license that can be
# found in the LICENSE file.
from recipe_engine.post_process import (
DropExpectation,
StepCommandContains,
)
PYTHON_VERSION_COMPATIBILITY = "PY3"
DEPS = [
"crosvm",
"recipe_engine/buildbucket",
]
def RunSteps(api):
api.crosvm.prepare_source()
def GenTests(api):
REPO = "https://chromium.googlesource.com/crosvm/crosvm"
REVISION = "2d72510e447ab60a9728aeea2362d8be2cbd7789"
yield (
api.test(
"prepare_source_for_try",
api.buildbucket.try_build(project="crosvm", git_repo=REPO),
)
+ api.post_process(StepCommandContains, "bot_update", ["--patch_ref"])
+ api.post_process(DropExpectation)
)
yield (
api.test(
"prepare_source_for_ci",
api.buildbucket.ci_build(project="crosvm", git_repo=REPO, revision=REVISION),
)
+ api.post_process(StepCommandContains, "bot_update", ["--revision", "crosvm@" + REVISION])
+ api.post_process(DropExpectation)
)

View file

@ -0,0 +1,15 @@
[
{
"cmd": [
"./tools/dev_container",
"--verbose",
"--hermetic",
"echo",
"success"
],
"name": "Foo"
},
{
"name": "$result"
}
]

View file

@ -0,0 +1,17 @@
# Copyright 2022 The Chromium OS Authors. All rights reserved.
# Use of this source code is governed by a BSD-style license that can be
# found in the LICENSE file.
PYTHON_VERSION_COMPATIBILITY = "PY3"
DEPS = [
"crosvm",
]
def RunSteps(api):
api.crosvm.step_in_container("Foo", ["echo", "success"])
def GenTests(api):
yield api.test("simple_step")

View file

@ -0,0 +1,73 @@
[
{
"cmd": [
"./tools/dev_container",
"--verbose",
"--hermetic",
"true"
],
"cwd": "[CACHE]/builder/crosvm",
"luci_context": {
"realm": {
"name": "crosvm/crosvm:ci"
},
"resultdb": {
"current_invocation": {
"name": "invocations/build:8945511751514863184",
"update_token": "token"
},
"hostname": "rdbhost"
}
},
"name": "Download container image"
},
{
"cmd": [
"./tools/dev_container",
"--verbose",
"--hermetic",
"./tools/run_tests",
"--verbose",
"--build-only",
"--target=vm:aarch64"
],
"cwd": "[CACHE]/builder/crosvm",
"luci_context": {
"realm": {
"name": "crosvm/crosvm:ci"
},
"resultdb": {
"current_invocation": {
"name": "invocations/build:8945511751514863184",
"update_token": "token"
},
"hostname": "rdbhost"
}
},
"name": "Build crosvm tests"
},
{
"cmd": [
"./tools/dev_container",
"--verbose",
"--hermetic",
"./tools/run_tests",
"--verbose",
"--target=vm:aarch64"
],
"cwd": "[CACHE]/builder/crosvm",
"luci_context": {
"realm": {
"name": "crosvm/crosvm:ci"
},
"resultdb": {
"current_invocation": {
"name": "invocations/build:8945511751514863184",
"update_token": "token"
},
"hostname": "rdbhost"
}
},
"name": "Run crosvm tests"
}
]

View file

@ -0,0 +1,75 @@
[
{
"cmd": [
"./tools/dev_container",
"--verbose",
"--hermetic",
"true"
],
"cwd": "[CACHE]/builder/crosvm",
"luci_context": {
"realm": {
"name": "crosvm/crosvm:ci"
},
"resultdb": {
"current_invocation": {
"name": "invocations/build:8945511751514863184",
"update_token": "token"
},
"hostname": "rdbhost"
}
},
"name": "Download container image"
},
{
"cmd": [
"./tools/dev_container",
"--verbose",
"--hermetic",
"./tools/run_tests",
"--verbose",
"--build-only",
"--target=vm:aarch64",
"--arch=armhf"
],
"cwd": "[CACHE]/builder/crosvm",
"luci_context": {
"realm": {
"name": "crosvm/crosvm:ci"
},
"resultdb": {
"current_invocation": {
"name": "invocations/build:8945511751514863184",
"update_token": "token"
},
"hostname": "rdbhost"
}
},
"name": "Build crosvm tests"
},
{
"cmd": [
"./tools/dev_container",
"--verbose",
"--hermetic",
"./tools/run_tests",
"--verbose",
"--target=vm:aarch64",
"--arch=armhf"
],
"cwd": "[CACHE]/builder/crosvm",
"luci_context": {
"realm": {
"name": "crosvm/crosvm:ci"
},
"resultdb": {
"current_invocation": {
"name": "invocations/build:8945511751514863184",
"update_token": "token"
},
"hostname": "rdbhost"
}
},
"name": "Run crosvm tests"
}
]

View file

@ -0,0 +1,73 @@
[
{
"cmd": [
"./tools/dev_container",
"--verbose",
"--hermetic",
"true"
],
"cwd": "[CACHE]/builder/crosvm",
"luci_context": {
"realm": {
"name": "crosvm/crosvm:ci"
},
"resultdb": {
"current_invocation": {
"name": "invocations/build:8945511751514863184",
"update_token": "token"
},
"hostname": "rdbhost"
}
},
"name": "Download container image"
},
{
"cmd": [
"./tools/dev_container",
"--verbose",
"--hermetic",
"./tools/run_tests",
"--verbose",
"--build-only",
"--target=host"
],
"cwd": "[CACHE]/builder/crosvm",
"luci_context": {
"realm": {
"name": "crosvm/crosvm:ci"
},
"resultdb": {
"current_invocation": {
"name": "invocations/build:8945511751514863184",
"update_token": "token"
},
"hostname": "rdbhost"
}
},
"name": "Build crosvm tests"
},
{
"cmd": [
"./tools/dev_container",
"--verbose",
"--hermetic",
"./tools/run_tests",
"--verbose",
"--target=host"
],
"cwd": "[CACHE]/builder/crosvm",
"luci_context": {
"realm": {
"name": "crosvm/crosvm:ci"
},
"resultdb": {
"current_invocation": {
"name": "invocations/build:8945511751514863184",
"update_token": "token"
},
"hostname": "rdbhost"
}
},
"name": "Run crosvm tests"
}
]

View file

@ -0,0 +1,12 @@
// Copyright 2022 The Chromium OS Authors. All rights reserved.
// Use of this source code is governed by a BSD-style license that can be
// found in the LICENSE file.
syntax = "proto3";
package recipes.crosvm.build_linux;
message BuildLinuxProperties {
// Architecture to test. See `crosvm/tools/run_tests`
optional string test_arch = 1;
}

View file

@ -0,0 +1,94 @@
# Copyright 2022 The Chromium OS Authors. All rights reserved.
# Use of this source code is governed by a BSD-style license that can be
# found in the LICENSE file.
from recipe_engine.recipe_api import Property
from recipe_engine.post_process import DropExpectation, StatusFailure, Filter
from PB.recipes.crosvm.build_linux import BuildLinuxProperties
PYTHON_VERSION_COMPATIBILITY = "PY3"
DEPS = [
"crosvm",
"recipe_engine/buildbucket",
"recipe_engine/context",
"recipe_engine/properties",
"recipe_engine/step",
]
PROPERTIES = BuildLinuxProperties
def get_test_args(api, test_arch):
"Returns architecture specific arguments for ./tools/run_tests"
# TODO(denniskempin): Move this logic into ./tools/presubmit
if test_arch == "" or test_arch == "x86_64":
return ["--target=host"]
elif test_arch == "aarch64":
return ["--target=vm:aarch64"]
elif test_arch == "armhf":
return ["--target=vm:aarch64", "--arch=armhf"]
else:
raise api.step.StepFailure("Unknown test_arch " + test_arch)
def RunSteps(api, properties):
api.crosvm.prepare_source()
with api.context(cwd=api.crosvm.source_dir):
# Run a no-op command in the container which will ensure the image is downloaded.
api.crosvm.step_in_container("Download container image", ["true"])
api.crosvm.step_in_container(
"Build crosvm tests",
[
"./tools/run_tests",
"--verbose",
"--build-only",
]
+ get_test_args(api, properties.test_arch),
)
api.crosvm.step_in_container(
"Run crosvm tests",
[
"./tools/run_tests",
"--verbose",
]
+ get_test_args(api, properties.test_arch),
)
def GenTests(api):
filter_steps = Filter("Download container image", "Build crosvm tests", "Run crosvm tests")
yield (
api.test(
"build_x86_64",
api.buildbucket.ci_build(project="crosvm/crosvm"),
)
+ api.properties(BuildLinuxProperties(test_arch="x86_64"))
+ api.post_process(filter_steps)
)
yield (
api.test(
"build_aarch64",
api.buildbucket.ci_build(project="crosvm/crosvm"),
)
+ api.properties(BuildLinuxProperties(test_arch="aarch64"))
+ api.post_process(filter_steps)
)
yield (
api.test(
"build_armhf",
api.buildbucket.ci_build(project="crosvm/crosvm"),
)
+ api.properties(BuildLinuxProperties(test_arch="armhf"))
+ api.post_process(filter_steps)
)
yield (
api.test(
"build_unknown",
api.buildbucket.ci_build(project="crosvm/crosvm"),
)
+ api.properties(BuildLinuxProperties(test_arch="foobar"))
+ api.post_process(StatusFailure)
+ api.post_process(DropExpectation)
)

View file

@ -1,9 +0,0 @@
[
{
"cmd": [],
"name": "Hello world"
},
{
"name": "$result"
}
]

View file

@ -1,20 +0,0 @@
# Enumerates the recipe modules that this recipe uses.
#
# "recipe_engine" is the "repo_name" for the recipes-py repo, and "step"
# is the name of the "step" recipe module within that repo. The
# "recipe_engine/step" module will be the most frequently-used module in your
# recipes as it allows you to run executables within your build.
DEPS = [
"recipe_engine/step",
]
def RunSteps(api):
# Creates an 'empty' (i.e. no-op) step in the UI with the name "Hello world".
api.step.empty("Hello world")
def GenTests(api):
# Tells the recipe engine to generate an expectation file (JSON simulation
# output) for this recipe when it is run without any input properties.
yield api.test("basic")

View file

@ -1,91 +0,0 @@
[
{
"cmd": [
"vpython3",
"-u",
"RECIPE_MODULE[depot_tools::bot_update]/resources/bot_update.py",
"--spec-path",
"cache_dir = '[CACHE]/git'\nsolutions = [{'deps_file': '.DEPS.git', 'managed': True, 'name': 'src', 'url': 'http://chromium.googlesource.com/crosvm/crosvm'}]",
"--revision_mapping_file",
"{\"got_revision\": \"src\"}",
"--git-cache-dir",
"[CACHE]/git",
"--cleanup-dir",
"[CLEANUP]/bot_update",
"--output_json",
"/path/to/tmp/json",
"--revision",
"src@2d72510e447ab60a9728aeea2362d8be2cbd7789",
"--refs",
"refs/heads/main"
],
"cwd": "[CACHE]/builder",
"env": {
"DEPOT_TOOLS_REPORT_BUILD": "crosvm/ci/builder/8945511751514863184",
"GIT_HTTP_LOW_SPEED_LIMIT": "102400",
"GIT_HTTP_LOW_SPEED_TIME": "1800"
},
"env_suffixes": {
"DEPOT_TOOLS_UPDATE": [
"0"
],
"PATH": [
"RECIPE_REPO[depot_tools]"
]
},
"infra_step": true,
"luci_context": {
"realm": {
"name": "crosvm:ci"
},
"resultdb": {
"current_invocation": {
"name": "invocations/build:8945511751514863184",
"update_token": "token"
},
"hostname": "rdbhost"
}
},
"name": "bot_update",
"~followup_annotations": [
"@@@STEP_TEXT@Some step text@@@",
"@@@STEP_LOG_LINE@json.output@{@@@",
"@@@STEP_LOG_LINE@json.output@ \"did_run\": true, @@@",
"@@@STEP_LOG_LINE@json.output@ \"fixed_revisions\": {@@@",
"@@@STEP_LOG_LINE@json.output@ \"src\": \"2d72510e447ab60a9728aeea2362d8be2cbd7789\"@@@",
"@@@STEP_LOG_LINE@json.output@ }, @@@",
"@@@STEP_LOG_LINE@json.output@ \"manifest\": {@@@",
"@@@STEP_LOG_LINE@json.output@ \"src\": {@@@",
"@@@STEP_LOG_LINE@json.output@ \"repository\": \"https://fake.org/src.git\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"revision\": \"2d72510e447ab60a9728aeea2362d8be2cbd7789\"@@@",
"@@@STEP_LOG_LINE@json.output@ }@@@",
"@@@STEP_LOG_LINE@json.output@ }, @@@",
"@@@STEP_LOG_LINE@json.output@ \"patch_failure\": false, @@@",
"@@@STEP_LOG_LINE@json.output@ \"patch_root\": \"src\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"properties\": {@@@",
"@@@STEP_LOG_LINE@json.output@ \"got_revision\": \"2d72510e447ab60a9728aeea2362d8be2cbd7789\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"got_revision_cp\": \"refs/heads/main@{#170242}\"@@@",
"@@@STEP_LOG_LINE@json.output@ }, @@@",
"@@@STEP_LOG_LINE@json.output@ \"root\": \"src\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"source_manifest\": {@@@",
"@@@STEP_LOG_LINE@json.output@ \"directories\": {@@@",
"@@@STEP_LOG_LINE@json.output@ \"src\": {@@@",
"@@@STEP_LOG_LINE@json.output@ \"git_checkout\": {@@@",
"@@@STEP_LOG_LINE@json.output@ \"repo_url\": \"https://fake.org/src.git\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"revision\": \"2d72510e447ab60a9728aeea2362d8be2cbd7789\"@@@",
"@@@STEP_LOG_LINE@json.output@ }@@@",
"@@@STEP_LOG_LINE@json.output@ }@@@",
"@@@STEP_LOG_LINE@json.output@ }, @@@",
"@@@STEP_LOG_LINE@json.output@ \"version\": 0@@@",
"@@@STEP_LOG_LINE@json.output@ }, @@@",
"@@@STEP_LOG_LINE@json.output@ \"step_text\": \"Some step text\"@@@",
"@@@STEP_LOG_LINE@json.output@}@@@",
"@@@STEP_LOG_END@json.output@@@",
"@@@SET_BUILD_PROPERTY@got_revision@\"2d72510e447ab60a9728aeea2362d8be2cbd7789\"@@@",
"@@@SET_BUILD_PROPERTY@got_revision_cp@\"refs/heads/main@{#170242}\"@@@"
]
},
{
"name": "$result"
}
]

View file

@ -1,151 +0,0 @@
[
{
"cmd": [
"vpython",
"-u",
"RECIPE_REPO[depot_tools]/gerrit_client.py",
"changes",
"--host",
"https://chromium-review.googlesource.com",
"--json_file",
"/path/to/tmp/json",
"--limit",
"1",
"-p",
"change=123456",
"-o",
"ALL_REVISIONS",
"-o",
"DOWNLOAD_COMMANDS"
],
"cwd": "[CACHE]/builder",
"env": {
"PATH": "<PATH>:RECIPE_REPO[depot_tools]"
},
"infra_step": true,
"luci_context": {
"realm": {
"name": "crosvm:try"
},
"resultdb": {
"current_invocation": {
"name": "invocations/build:8945511751514863184",
"update_token": "token"
},
"hostname": "rdbhost"
}
},
"name": "gerrit fetch current CL info",
"timeout": 60,
"~followup_annotations": [
"@@@STEP_LOG_LINE@json.output@[@@@",
"@@@STEP_LOG_LINE@json.output@ {@@@",
"@@@STEP_LOG_LINE@json.output@ \"branch\": \"main\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"owner\": {@@@",
"@@@STEP_LOG_LINE@json.output@ \"name\": \"John Doe\"@@@",
"@@@STEP_LOG_LINE@json.output@ }, @@@",
"@@@STEP_LOG_LINE@json.output@ \"revisions\": {@@@",
"@@@STEP_LOG_LINE@json.output@ \"184ebe53805e102605d11f6b143486d15c23a09c\": {@@@",
"@@@STEP_LOG_LINE@json.output@ \"_number\": \"7\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"ref\": \"refs/changes/56/123456/7\"@@@",
"@@@STEP_LOG_LINE@json.output@ }@@@",
"@@@STEP_LOG_LINE@json.output@ }@@@",
"@@@STEP_LOG_LINE@json.output@ }@@@",
"@@@STEP_LOG_LINE@json.output@]@@@",
"@@@STEP_LOG_END@json.output@@@"
]
},
{
"cmd": [
"vpython3",
"-u",
"RECIPE_MODULE[depot_tools::bot_update]/resources/bot_update.py",
"--spec-path",
"cache_dir = '[CACHE]/git'\nsolutions = [{'deps_file': '.DEPS.git', 'managed': True, 'name': 'src', 'url': 'http://chromium.googlesource.com/crosvm/crosvm'}]",
"--patch_root",
"src",
"--revision_mapping_file",
"{\"got_revision\": \"src\"}",
"--git-cache-dir",
"[CACHE]/git",
"--cleanup-dir",
"[CLEANUP]/bot_update",
"--output_json",
"/path/to/tmp/json",
"--patch_ref",
"https://chromium.googlesource.com/crosvm/crosvm@refs/heads/main:refs/changes/56/123456/7",
"--revision",
"src@refs/heads/main",
"--refs",
"refs/heads/main"
],
"cwd": "[CACHE]/builder",
"env": {
"DEPOT_TOOLS_REPORT_BUILD": "crosvm/try/builder/8945511751514863184",
"GIT_HTTP_LOW_SPEED_LIMIT": "102400",
"GIT_HTTP_LOW_SPEED_TIME": "1800"
},
"env_suffixes": {
"DEPOT_TOOLS_UPDATE": [
"0"
],
"PATH": [
"RECIPE_REPO[depot_tools]"
]
},
"infra_step": true,
"luci_context": {
"realm": {
"name": "crosvm:try"
},
"resultdb": {
"current_invocation": {
"name": "invocations/build:8945511751514863184",
"update_token": "token"
},
"hostname": "rdbhost"
}
},
"name": "bot_update",
"~followup_annotations": [
"@@@STEP_TEXT@Some step text@@@",
"@@@STEP_LOG_LINE@json.output@{@@@",
"@@@STEP_LOG_LINE@json.output@ \"did_run\": true, @@@",
"@@@STEP_LOG_LINE@json.output@ \"fixed_revisions\": {@@@",
"@@@STEP_LOG_LINE@json.output@ \"src\": \"HEAD\"@@@",
"@@@STEP_LOG_LINE@json.output@ }, @@@",
"@@@STEP_LOG_LINE@json.output@ \"manifest\": {@@@",
"@@@STEP_LOG_LINE@json.output@ \"src\": {@@@",
"@@@STEP_LOG_LINE@json.output@ \"repository\": \"https://fake.org/src.git\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"revision\": \"f27fede2220bcd326aee3e86ddfd4ebd0fe58cb9\"@@@",
"@@@STEP_LOG_LINE@json.output@ }@@@",
"@@@STEP_LOG_LINE@json.output@ }, @@@",
"@@@STEP_LOG_LINE@json.output@ \"patch_failure\": false, @@@",
"@@@STEP_LOG_LINE@json.output@ \"patch_root\": \"src\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"properties\": {@@@",
"@@@STEP_LOG_LINE@json.output@ \"got_revision\": \"f27fede2220bcd326aee3e86ddfd4ebd0fe58cb9\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"got_revision_cp\": \"refs/heads/main@{#170242}\"@@@",
"@@@STEP_LOG_LINE@json.output@ }, @@@",
"@@@STEP_LOG_LINE@json.output@ \"root\": \"src\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"source_manifest\": {@@@",
"@@@STEP_LOG_LINE@json.output@ \"directories\": {@@@",
"@@@STEP_LOG_LINE@json.output@ \"src\": {@@@",
"@@@STEP_LOG_LINE@json.output@ \"git_checkout\": {@@@",
"@@@STEP_LOG_LINE@json.output@ \"repo_url\": \"https://fake.org/src.git\", @@@",
"@@@STEP_LOG_LINE@json.output@ \"revision\": \"f27fede2220bcd326aee3e86ddfd4ebd0fe58cb9\"@@@",
"@@@STEP_LOG_LINE@json.output@ }@@@",
"@@@STEP_LOG_LINE@json.output@ }@@@",
"@@@STEP_LOG_LINE@json.output@ }, @@@",
"@@@STEP_LOG_LINE@json.output@ \"version\": 0@@@",
"@@@STEP_LOG_LINE@json.output@ }, @@@",
"@@@STEP_LOG_LINE@json.output@ \"step_text\": \"Some step text\"@@@",
"@@@STEP_LOG_LINE@json.output@}@@@",
"@@@STEP_LOG_END@json.output@@@",
"@@@SET_BUILD_PROPERTY@got_revision@\"f27fede2220bcd326aee3e86ddfd4ebd0fe58cb9\"@@@",
"@@@SET_BUILD_PROPERTY@got_revision_cp@\"refs/heads/main@{#170242}\"@@@"
]
},
{
"name": "$result"
}
]

View file

@ -1,62 +0,0 @@
PYTHON_VERSION_COMPATIBILITY = "PY3"
DEPS = [
"depot_tools/bot_update",
"depot_tools/gclient",
"recipe_engine/buildbucket",
"recipe_engine/context",
"recipe_engine/path",
]
def RunSteps(api):
# TODO(denniskempin): Consider using git directly for simpler config
gitilies = api.buildbucket.build.input.gitiles_commit
if gitilies.host:
url = "http://%s/%s" % (gitilies.host, gitilies.project)
else:
cl = api.buildbucket.build.input.gerrit_changes[0]
gs_suffix = "-review.googlesource.com"
host = cl.host
if host.endswith(gs_suffix):
host = "%s.googlesource.com" % host[: -len(gs_suffix)]
url = "http://%s/%s" % (host, cl.project)
gclient_config = api.gclient.make_config()
s = gclient_config.solutions.add()
s.url = url
s.name = "src"
gclient_config.got_revision_mapping[s.name] = "got_revision"
with api.context(cwd=api.path["cache"].join("builder")):
update_result = api.bot_update.ensure_checkout(gclient_config=gclient_config)
# At this point the code for the Gerrit CL is checked out at
# `api.path['cache'].join('builder')`, which by default is preserved locally
# on the bot machine and re-used between different builds for the same
# builder.
# TODO(denniskempin): Add some sort of build and/or verification step(s).
def GenTests(api):
yield api.test(
"basic try",
# These are just to make the JSON expectation file data look closer to
# reality. Project and git_repo will be filled in "for real" by the LUCI
# Change Verifier service when it creates your build.
api.buildbucket.try_build(
project="crosvm",
git_repo="https://chromium.googlesource.com/crosvm/crosvm",
),
)
yield api.test(
"basic ci",
# These are just to make the JSON expectation file data look closer to
# reality. Project and git_repo will be filled in "for real" by the LUCI
# Change Verifier service when it creates your build.
api.buildbucket.ci_build(
project="crosvm",
git_repo="https://chromium.googlesource.com/crosvm/crosvm",
),
)