mirror of
https://github.com/sourcegraph/sourcegraph.git
synced 2026-02-06 19:21:50 +00:00
14123fcc42
This PR is a second attempt at improving push_all.sh, continuing on from (and inspired by) https://github.com/sourcegraph/sourcegraph/pull/63391. As a recap, that PR uses [--script_path](https://bazel.build/reference/command-line-reference#flag--script_path) to emit a short bash script for every `oci_push` target, which essentially does minor setup + invokes the executable as if running `bazel run`. While the idea in https://github.com/sourcegraph/sourcegraph/pull/63391 was good, it trades concurrent server locking with an equal amount of overhead in sequentially building the scripts. By observing the scripts<b>[1]</b> that it would emit, we can notice a few things: - The path `/home/noah/.cache/bazel/_bazel_noah/8fd1d20666a46767e7f29541678514a0/execroot/__main__/bazel-out/k8-fastbuild/bin/` shows up twice, which is the same path that `./bazel-bin` points at - The script only `cd`'s to a path, unsets some environment variables, and then executes the underlying script of the target. The path can be observed to be a combination of bazel-bin, the target's package (`//cmd/batcheshelper` in this case), as well as the target with some extra static strings (`candidate_push` with `push_` prefix and `.sh{,.runfiles}` suffixes for the script & its runfiles respectively). Knowing this, and assuming that this is reliably so, we can opt to recreate this manually instead, saving on the hefty overhead of `bazel run --script_path`. The current average times for `Push candidate images` and `Push final images` are ~7m50s and ~8m30s respectively. While the example main-dry-run build [here](https://buildkite.com/sourcegraph/sourcegraph/builds/284041#0190e54a-9aaa-471a-81bf-623fce6ffa45) isnt fully representative of how much rebuilding is required, it sets a pretty solid 3m20s baseline. Note this may break with rules_oci changes, but imo thats a small and very infrequent cost to pay for cleaner log output + shaving a good piece of time off. <details><summary><b>[1]</b> A <code>--script_path</code> example</summary> ``` #!/nix/store/mqc7dqwp046lh41dhs7r7q7192zbliwd-bash/bin/bash cd /home/noah/.cache/bazel/_bazel_noah/8fd1d20666a46767e7f29541678514a0/execroot/__main__/bazel-out/k8-fastbuild/bin/cmd/batcheshelper/push_candidate_push.sh.runfiles/__main__ && \ exec env \ -u JAVA_RUNFILES \ -u RUNFILES_DIR \ -u RUNFILES_MANIFEST_FILE \ -u RUNFILES_MANIFEST_ONLY \ -u TEST_SRCDIR \ BUILD_WORKING_DIRECTORY=/home/noah/Sourcegraph/sourcegraph \ BUILD_WORKSPACE_DIRECTORY=/home/noah/Sourcegraph/sourcegraph \ /home/noah/.cache/bazel/_bazel_noah/8fd1d20666a46767e7f29541678514a0/execroot/__main__/bazel-out/k8-fastbuild/bin/cmd/batcheshelper/push_candidate_push.sh "$@" ``` </details> ## Test plan Observe a `sg ci build main-dry-run` [here](https://buildkite.com/sourcegraph/sourcegraph/builds/284041#0190e54a-9aaa-471a-81bf-623fce6ffa45). ## Changelog <!-- OPTIONAL; info at https://www.notion.so/sourcegraph/Writing-a-changelog-entry-dd997f411d524caabf0d8d38a24a878c --> |
||
|---|---|---|
| .. | ||
| auth-provider | ||
| authtest | ||
| backcompat | ||
| bkstats | ||
| build-tracker | ||
| buildchecker | ||
| check | ||
| ci | ||
| clustering | ||
| codehost_testing | ||
| codeintel-qa | ||
| db | ||
| depgraph | ||
| deployment-lag-notifier | ||
| deployment-notifier | ||
| gitserverintegration | ||
| go-mockgen-gazelle | ||
| go-mockgen-transformer | ||
| gqltest | ||
| grafana | ||
| insight-data-gen | ||
| internal/cmd | ||
| linearhooks | ||
| linters | ||
| managedservicesplatform | ||
| nix | ||
| perforce | ||
| phabricator | ||
| pr-auditor | ||
| prometheus | ||
| scaletesting | ||
| sg | ||
| src-expose | ||
| team | ||
| tilt | ||
| tools | ||
| update-embeddings-list | ||
| zoekt | ||
| .gitignore | ||
| add_https_domain_to_hosts.sh | ||
| bazel_buildkite_stamp_vars.sh | ||
| bazel_stamp_vars.sh | ||
| BUILD.bazel | ||
| caddy.sh | ||
| Caddyfile | ||
| check-test-ownership.sh | ||
| check-tokens.sh | ||
| CLA.txt | ||
| codeinsights-db.sh | ||
| comby-install-or-upgrade.sh | ||
| defs.bzl | ||
| dev-sourcegraph-server.sh | ||
| docsite.sh | ||
| drop-entire-local-database-and-redis.sh | ||
| esbuild.bzl | ||
| eslint-report-test.sh | ||
| eslint.bzl | ||
| forbidcommit.sh | ||
| foreach-non-web-client-project.sh | ||
| git-stats | ||
| global-settings.json | ||
| go_defs.bzl | ||
| go_mockgen_rule.bzl | ||
| go_mockgen.bzl | ||
| go_stringer.bzl | ||
| go-mod-update.sh | ||
| licenses.sh | ||
| migrations.bzl | ||
| mocha.bzl | ||
| msp_delivery.bzl | ||
| oci_defs.bzl | ||
| oci_deps.bzl | ||
| p4-fusion-dev | ||
| proto.bzl | ||
| prune-pick.sh | ||
| redis-postgres.yml | ||
| redis.conf | ||
| remote_cache_local_env.sh | ||
| run-server-image.sh | ||
| sass.bzl | ||
| schema_migrations.bzl | ||
| scip-ctags-dev | ||
| scip-ctags-install.sh | ||
| scip-syntax-dev | ||
| scip-syntax-install.sh | ||
| site-config.json | ||
| src-prof-services.json | ||
| src-search-meta.sh | ||
| tool_deps.bzl | ||
| tools.go | ||
| universal-ctags-dev | ||
| write_generated_to_source_files.bzl | ||