mirror of
https://chromium.googlesource.com/crosvm/crosvm
synced 2024-10-23 04:46:29 +00:00
230dc4fd84
This reverts commite19959ab3c
. Reason for revert: Broke bot update. See https://ci.chromium.org/ui/p/crosvm/builders/try/health_check/b8808633426831578849/overview Original change's description: > Roll recipe dependencies (trivial). > > This is an automated CL created by the recipe roller. This CL rolls > recipe changes from upstream projects (depot_tools, recipe_engine) into this repository. > > The build that created this CL was > https://ci.chromium.org/b/8808635415894971729 > > depot_tools: >9a7b7675ba
~..2f988479fb9e31a59f935054d5f230748ed526f4 > 9a7b767 (vapier@chromium.org) > isort: make available to users > fb8cf9c (aravindvasudev@google.com) > Add --protocol-override to fetch.py > 13acea3 (brucedawson@chromium.org) > Let git cl presubmit work with no branch > 23ddab2 (aravindvasudev@google.com) > Reland "Break make_encoded_file into two functions" > 08a30b2 (bpastene@chromium.org) > Add examples to the help string for git-cl-try's -B arg > 443d913 (dpapad@chromium.org) > Include TypeScript .ts files in DEFAULT_FILES_TO_CHECK. > b9dca86 (meiring@google.com) > [depot_tools] Roll rdb. > 6a1494e (tikuta@chromium.org) > Reland "autoninja: increase parallelism for non-HT cpu" > 1bc3cd2 (akhoroshilov@brave.com) > Add a global core limit support for goma/rbe builds. > cb2cef9 (sdefresne@chromium.org) > [mac] Restore an upper limit on the number of jobs on macOS > 31bfd51 (brucedawson@chromium.org) > Actually add a trailing slash to dir_with_slash > 9997ceb (vapier@chromium.org) > black: add helper for formatting Python > aa7d7e4 (dmercadier@chromium.org) > [cpplint] Fix bug with "if constexpr" > bfefaa8 (chanli@chromium.org) > [depot_tools] roll led > 50c631e (jojwang@google.com) > Skip 'ls' when downloading from gs. > f8300fb (jochen@chromium.org) > Remove myself from depot_tools OWNERS > 73218ec (vadimsh@chromium.org) > [cipd] Update to v2.6.4. > 772f3dc (ukai@google.com) > Roll out goma client VERSION=246 > 632bbc0 (sokcevic@google.com) > Skip Python 2 presubmit step when unneeded > cf9a776 (lpromero@chromium.org) > Typo when countdown is more than 1. > 471b549 (chanli@chromium.org) > roll dirmd > 0e9a7d2 (sokcevic@google.com) > Revert "Add iannucci to depot_tools owners" > 2f98847 (dsanders11@ucsbalum.com) > cpplint: improve root path for header guard behavior > > recipe_engine: >f588cc00b9
~..04b3acea0bf45c9e1076079998b6378caa5570d2 > f588cc0 (chromium-autoroll@skia-public.iam.gserviceaccount.com) > Roll CAS Client from 595e0f7a2869 to cb424e70e751 > 52ef306 (iannucci@chromium.org) > [recipe_engine] Implicit CANCELED final build status. > cde011f (machenbach@chromium.org) > [led] Add common logic to trigger a builder > 087afb8 (chanli@chromium.org) > [recipe-py] buildbucket: enable buildbucket tracking parent/chi... > c827c1b (chanli@chromium.org) > Revert "[recipe-py] buildbucket: enable buildbucket tracking pa... > 9eef321 (gbeaty@chromium.org) > Prevent creation of candidates for dependencies of dependencies. > 3db9e1b (chanli@chromium.org) > Reland "[recipe-py] buildbucket: enable buildbucket tracking pa... > b788d0a (vadimsh@chromium.org) > [scheduler] Expose job ID and invocation ID. > 6d2e271 (gredelston@google.com) > walkthrough.md: Fix incorrect kwarg "output" > 14202da (chromium-autoroll@skia-public.iam.gserviceaccount.com) > Roll CAS Client from addd54e4fcef to 6da0608e4fa8 > 826d446 (kimstephanie@google.com) > Add new post_process.PropertiesContain to check that key exists > f15df27 (chanli@chromium.org) > Extend placeholder.py to schedule child builds > 83eab46 (whesse@google.com) > [buildbucket] Relax type check on buildbucket ids > d3504b5 (mohrr@google.com) > [path] Add __lt__ to Path > 9e3ca47 (chromium-autoroll@skia-public.iam.gserviceaccount.com) > Roll CAS Client from a0672c27eaaa to 5fa853f88d79 > 2522322 (gbeaty@chromium.org) > Rewrite candidate_algorithm. > b4882c1 (chanli@chromium.org) > [time] make sleep a timed wait on global_shutdown > 1d661d3 (chromium-autoroll@skia-public.iam.gserviceaccount.com) > Roll CAS Client from b927ccff49e4 to b2aa2d72c22b > a6fd0d2 (chanli@chromium.org) > [time] set the step to canceled if GLOBAL_SHUTDOWN is triggered > 0f5f374 (alexschulze@chromium.org) > [cipd] Fix `instances` command with limit = 0 > 048f6f2 (chanli@chromium.org) > [recipe-py] buildbucket: set `luci.buildbucket.parent_tracking`... > d63eb67 (chanli@chromium.org) > Revert "[recipe-py] buildbucket: set `luci.buildbucket.parent_t... > e783f88 (chanli@chromium.org) > [recipe-py] (2 attempt) buildbucket: set `luci.buildbucket.pare... > 2239191 (gbeaty@chromium.org) > Prevent spurious GitFetchErrors in CommitList.from_backend. > efaa1e2 (iannucci@chromium.org) > [json] Stop using python.inline. > dc013a8 (iannucci@chromium.org) > [python] Mark python.inline with a warning > 6345fb2 (chromium-autoroll@skia-public.iam.gserviceaccount.com) > Roll CAS Client from 4f40f26f3102 to 2aa3d7e5e866 > 36be946 (iannucci@chromium.org) > [python] Deprecate api.python.__call__ > 357b1cd (martiniss@google.com) > Add timing information option to the test command > e174a86 (martiniss@google.com) > Add exponential_backoff to the time module. > ce1d368 (iannucci@chromium.org) > [warnings] Omit warnings which are attributed to other recipe r... > c002009 (akashmukherjee@google.com) > [snoopy] Create luci's bcid recipe module dir. > 1e08dae (akashmukherjee@google.com) > [snoopy] Create a luci's bcid recipe_module. > f2754bc (akashmukherjee@google.com) > [snoopy] Update the cipd for reporter module. > 4cb1bf0 (gbeaty@chromium.org) > Don't set the -u flag when running python unbuffered. > 50cbd44 (meiring@google.com) > [recipe-py] roll recipe_proto > 82fb654 (meiring@google.com) > [recipes-py] Update ResultDB exoneration example to populate re... > bb1fb5a (iannucci@chromium.org) > [python] Set warning for entire python module. > d047727 (gbeaty@chromium.org) > Inhibit displaying coverage information on python3 failures wit... > 57ac2ff (chromium-autoroll@skia-public.iam.gserviceaccount.com) > Roll CAS Client from e361505b12f1 to 3bd002442390 > 253ac82 (chromium-autoroll@skia-public.iam.gserviceaccount.com) > Roll CAS Client from 91c1b9ec335a to cad461b65231 > 2684e3c (chromium-autoroll@skia-public.iam.gserviceaccount.com) > Roll CAS Client from 4137ed7879ac to 93b14d0f4216 > ac03648 (chromium-autoroll@skia-public.iam.gserviceaccount.com) > Roll CAS Client from 35b05be75bc4 to 29b908616bc9 > 0a09360 (kimstephanie@google.com) > Add post_process.LogDoesNotContain > 08c51a1 (chromium-autoroll@skia-public.iam.gserviceaccount.com) > Roll CAS Client from c25db5a11beb to d3db74920e35 > e3cd9eb (kuanhuang@chromium.org) > Add recipes API swarming.show_request(name, task) -> TaskRequest > 04b3ace (chromium-autoroll@skia-public.iam.gserviceaccount.com) > Roll CAS Client from 8b08f5e9f609 to 0ef9351a5b73 > > Please check the following references for more information: > - autoroller, https://chromium.googlesource.com/infra/luci/recipes-py/+/main/doc/workflow.md#autoroller > - rollback, https://chromium.googlesource.com/infra/luci/recipes-py/+/main/doc/workflow.md#rollback > - cross-repo dependencies, https://chromium.googlesource.com/infra/luci/recipes-py/+/main/doc/cross_repo.md > > Use https://goo.gl/noib3a to file a bug. > > R=denniskempin@google.com > > Recipe-Tryjob-Bypass-Reason: Autoroller > Ignore-Freeze: Autoroller > Bugdroid-Send-Email: False > Change-Id: Ic16796c2981fe7b3373146a55921cfee5e64eedc > Reviewed-on: https://chromium-review.googlesource.com/c/crosvm/crosvm/+/3764420 > Commit-Queue: Dennis Kempin <denniskempin@google.com> > Bot-Commit: Recipe Roller <recipe-mega-autoroller@chops-service-accounts.iam.gserviceaccount.com> Change-Id: I56cd86430a8d5707e1c0a9720ac85ebe0cbc89f0 No-Presubmit: true No-Tree-Checks: true No-Try: true Reviewed-on: https://chromium-review.googlesource.com/c/crosvm/crosvm/+/3764439 Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com> Auto-Submit: Dennis Kempin <denniskempin@google.com>
209 lines
8.4 KiB
Python
Executable file
209 lines
8.4 KiB
Python
Executable file
#!/bin/sh
|
|
# Copyright 2019 The LUCI Authors. All rights reserved.
|
|
# Use of this source code is governed under the Apache License, Version 2.0
|
|
# that can be found in the LICENSE file.
|
|
# We want to run python in unbuffered mode; however shebangs on linux grab the
|
|
# entire rest of the shebang line as a single argument, leading to errors like:
|
|
#
|
|
# /usr/bin/env: 'python3 -u': No such file or directory
|
|
#
|
|
# This little shell hack is a triple-quoted noop in python, but in sh it
|
|
# evaluates to re-exec'ing this script in unbuffered mode.
|
|
# pylint: disable=pointless-string-statement
|
|
''''exec python3 -u -- "$0" ${1+"$@"} # '''
|
|
# vi: syntax=python
|
|
"""Bootstrap script to clone and forward to the recipe engine tool.
|
|
*******************
|
|
** DO NOT MODIFY **
|
|
*******************
|
|
This is a copy of https://chromium.googlesource.com/infra/luci/recipes-py/+/main/recipes.py.
|
|
To fix bugs, fix in the googlesource repo then run the autoroller.
|
|
"""
|
|
# pylint: disable=wrong-import-position
|
|
import argparse
|
|
import errno
|
|
import json
|
|
import logging
|
|
import os
|
|
import subprocess
|
|
import sys
|
|
from collections import namedtuple
|
|
from io import open # pylint: disable=redefined-builtin
|
|
try:
|
|
import urllib.parse as urlparse
|
|
except ImportError:
|
|
import urlparse
|
|
# The dependency entry for the recipe_engine in the client repo's recipes.cfg
|
|
#
|
|
# url (str) - the url to the engine repo we want to use.
|
|
# revision (str) - the git revision for the engine to get.
|
|
# branch (str) - the branch to fetch for the engine as an absolute ref (e.g.
|
|
# refs/heads/main)
|
|
EngineDep = namedtuple('EngineDep', 'url revision branch')
|
|
class MalformedRecipesCfg(Exception):
|
|
def __init__(self, msg, path):
|
|
full_message = 'malformed recipes.cfg: %s: %r' % (msg, path)
|
|
super(MalformedRecipesCfg, self).__init__(full_message)
|
|
def parse(repo_root, recipes_cfg_path):
|
|
"""Parse is a lightweight a recipes.cfg file parser.
|
|
Args:
|
|
repo_root (str) - native path to the root of the repo we're trying to run
|
|
recipes for.
|
|
recipes_cfg_path (str) - native path to the recipes.cfg file to process.
|
|
Returns (as tuple):
|
|
engine_dep (EngineDep|None): The recipe_engine dependency, or None, if the
|
|
current repo IS the recipe_engine.
|
|
recipes_path (str) - native path to where the recipes live inside of the
|
|
current repo (i.e. the folder containing `recipes/` and/or
|
|
`recipe_modules`)
|
|
"""
|
|
with open(recipes_cfg_path, 'r') as fh:
|
|
pb = json.load(fh)
|
|
try:
|
|
if pb['api_version'] != 2:
|
|
raise MalformedRecipesCfg('unknown version %d' % pb['api_version'],
|
|
recipes_cfg_path)
|
|
# If we're running ./recipes.py from the recipe_engine repo itself, then
|
|
# return None to signal that there's no EngineDep.
|
|
repo_name = pb.get('repo_name')
|
|
if not repo_name:
|
|
repo_name = pb['project_id']
|
|
if repo_name == 'recipe_engine':
|
|
return None, pb.get('recipes_path', '')
|
|
engine = pb['deps']['recipe_engine']
|
|
if 'url' not in engine:
|
|
raise MalformedRecipesCfg(
|
|
'Required field "url" in dependency "recipe_engine" not found',
|
|
recipes_cfg_path)
|
|
engine.setdefault('revision', '')
|
|
engine.setdefault('branch', 'refs/heads/main')
|
|
recipes_path = pb.get('recipes_path', '')
|
|
# TODO(iannucci): only support absolute refs
|
|
if not engine['branch'].startswith('refs/'):
|
|
engine['branch'] = 'refs/heads/' + engine['branch']
|
|
recipes_path = os.path.join(repo_root,
|
|
recipes_path.replace('/', os.path.sep))
|
|
return EngineDep(**engine), recipes_path
|
|
except KeyError as ex:
|
|
raise MalformedRecipesCfg(str(ex), recipes_cfg_path)
|
|
IS_WIN = sys.platform.startswith(('win', 'cygwin'))
|
|
_BAT = '.bat' if IS_WIN else ''
|
|
GIT = 'git' + _BAT
|
|
VPYTHON = ('vpython' +
|
|
('3' if os.getenv('RECIPES_USE_PY3') == 'true' else '') +
|
|
_BAT)
|
|
CIPD = 'cipd' + _BAT
|
|
REQUIRED_BINARIES = {GIT, VPYTHON, CIPD}
|
|
def _is_executable(path):
|
|
return os.path.isfile(path) and os.access(path, os.X_OK)
|
|
# TODO: Use shutil.which once we switch to Python3.
|
|
def _is_on_path(basename):
|
|
for path in os.environ['PATH'].split(os.pathsep):
|
|
full_path = os.path.join(path, basename)
|
|
if _is_executable(full_path):
|
|
return True
|
|
return False
|
|
def _subprocess_call(argv, **kwargs):
|
|
logging.info('Running %r', argv)
|
|
return subprocess.call(argv, **kwargs)
|
|
def _git_check_call(argv, **kwargs):
|
|
argv = [GIT] + argv
|
|
logging.info('Running %r', argv)
|
|
subprocess.check_call(argv, **kwargs)
|
|
def _git_output(argv, **kwargs):
|
|
argv = [GIT] + argv
|
|
logging.info('Running %r', argv)
|
|
return subprocess.check_output(argv, **kwargs)
|
|
def parse_args(argv):
|
|
"""This extracts a subset of the arguments that this bootstrap script cares
|
|
about. Currently this consists of:
|
|
* an override for the recipe engine in the form of `-O recipe_engine=/path`
|
|
* the --package option.
|
|
"""
|
|
PREFIX = 'recipe_engine='
|
|
p = argparse.ArgumentParser(add_help=False)
|
|
p.add_argument('-O', '--project-override', action='append')
|
|
p.add_argument('--package', type=os.path.abspath)
|
|
args, _ = p.parse_known_args(argv)
|
|
for override in args.project_override or ():
|
|
if override.startswith(PREFIX):
|
|
return override[len(PREFIX):], args.package
|
|
return None, args.package
|
|
def checkout_engine(engine_path, repo_root, recipes_cfg_path):
|
|
dep, recipes_path = parse(repo_root, recipes_cfg_path)
|
|
if dep is None:
|
|
# we're running from the engine repo already!
|
|
return os.path.join(repo_root, recipes_path)
|
|
url = dep.url
|
|
if not engine_path and url.startswith('file://'):
|
|
engine_path = urlparse.urlparse(url).path
|
|
if not engine_path:
|
|
revision = dep.revision
|
|
branch = dep.branch
|
|
# Ensure that we have the recipe engine cloned.
|
|
engine_path = os.path.join(recipes_path, '.recipe_deps', 'recipe_engine')
|
|
with open(os.devnull, 'w') as NUL:
|
|
# Note: this logic mirrors the logic in recipe_engine/fetch.py
|
|
_git_check_call(['init', engine_path], stdout=NUL)
|
|
try:
|
|
_git_check_call(['rev-parse', '--verify',
|
|
'%s^{commit}' % revision],
|
|
cwd=engine_path,
|
|
stdout=NUL,
|
|
stderr=NUL)
|
|
except subprocess.CalledProcessError:
|
|
_git_check_call(['fetch', '--quiet', url, branch],
|
|
cwd=engine_path,
|
|
stdout=NUL)
|
|
try:
|
|
_git_check_call(['diff', '--quiet', revision], cwd=engine_path)
|
|
except subprocess.CalledProcessError:
|
|
index_lock = os.path.join(engine_path, '.git', 'index.lock')
|
|
try:
|
|
os.remove(index_lock)
|
|
except OSError as exc:
|
|
if exc.errno != errno.ENOENT:
|
|
logging.warn('failed to remove %r, reset will fail: %s', index_lock,
|
|
exc)
|
|
_git_check_call(['reset', '-q', '--hard', revision], cwd=engine_path)
|
|
# If the engine has refactored/moved modules we need to clean all .pyc files
|
|
# or things will get squirrely.
|
|
_git_check_call(['clean', '-qxf'], cwd=engine_path)
|
|
return engine_path
|
|
def main():
|
|
for required_binary in REQUIRED_BINARIES:
|
|
if not _is_on_path(required_binary):
|
|
return 'Required binary is not found on PATH: %s' % required_binary
|
|
if '--verbose' in sys.argv:
|
|
logging.getLogger().setLevel(logging.INFO)
|
|
args = sys.argv[1:]
|
|
engine_override, recipes_cfg_path = parse_args(args)
|
|
if recipes_cfg_path:
|
|
# calculate repo_root from recipes_cfg_path
|
|
repo_root = os.path.dirname(
|
|
os.path.dirname(os.path.dirname(recipes_cfg_path)))
|
|
else:
|
|
# find repo_root with git and calculate recipes_cfg_path
|
|
repo_root = (
|
|
_git_output(['rev-parse', '--show-toplevel'],
|
|
cwd=os.path.abspath(os.path.dirname(__file__))).strip())
|
|
repo_root = os.path.abspath(repo_root).decode()
|
|
recipes_cfg_path = os.path.join(repo_root, 'infra', 'config', 'recipes.cfg')
|
|
args = ['--package', recipes_cfg_path] + args
|
|
engine_path = checkout_engine(engine_override, repo_root, recipes_cfg_path)
|
|
argv = (
|
|
[VPYTHON, '-u',
|
|
os.path.join(engine_path, 'recipe_engine', 'main.py')] + args)
|
|
if IS_WIN:
|
|
# No real 'exec' on windows; set these signals to ignore so that they
|
|
# propagate to our children but we still wait for the child process to quit.
|
|
import signal
|
|
signal.signal(signal.SIGBREAK, signal.SIG_IGN)
|
|
signal.signal(signal.SIGINT, signal.SIG_IGN)
|
|
signal.signal(signal.SIGTERM, signal.SIG_IGN)
|
|
return _subprocess_call(argv)
|
|
else:
|
|
os.execvp(argv[0], argv)
|
|
if __name__ == '__main__':
|
|
sys.exit(main())
|