We're currently evaluating only pushing on Dockerhub when releasing, as
everything else uses GCR, but until then, we lower the concurrency to 4
when pushing there, and we keep 8 on the others.
Follow-up to https://github.com/sourcegraph/devx-support/issues/1163
## Test plan
CI
Give a more descriptive error when we are unable to find a merge base
and can't find the files that have changed.
Closes DINF-162
## Test plan
1. `git fetch -v --prune --depth=100 -- origin
ccbaba24b72f3c6f4524b3f560ca839143ea463b`
2. `git merge-base HEAD origin/main` --- you'll get nothing since there
is no merge base in the current history
Increase the repo history
1. `git fetch --unshallow`
2. `git merge-base HEAD origin/main`
```
0a6e509af3
```
## Changelog
<!-- OPTIONAL; info at
https://www.notion.so/sourcegraph/Writing-a-changelog-entry-dd997f411d524caabf0d8d38a24a878c
-->
Co-authored-by: Bolaji Olajide <bolaji.olajide@sourcegraph.com>
2nd attempt of #63111, a follow up
https://github.com/sourcegraph/sourcegraph/pull/63085
rules_oci 2.0 brings a lot of performance improvement around oci_image
and oci_pull, which will benefit Sourcegraph. It will also make RBE
faster and have less load on remote cache.
However, 2.0 makes some breaking changes like
- oci_tarball's default output is no longer a tarball
- oci_image no longer compresses layers that are uncompressed, somebody
has to make sure all `pkg_tar` targets have a `compression` attribute
set to compress it beforehand.
- there is no curl fallback, but this is fine for sourcegraph as it
already uses bazel 7.1.
I checked all targets that use oci_tarball as much as i could to make
sure nothing depends on the default tarball output of oci_tarball. there
was one target which used the default output which i put a TODO for
somebody else (somebody who is more on top of the repo) to tackle
**later**.
## Test plan
CI. Also run delivery on this PR (don't land those changes)
---------
Co-authored-by: Noah Santschi-Cooney <noah@santschi-cooney.ch>
Closes DINF-89
Gazelle sometimes have trouble processing glob expressions, and this
isn't reported as a failure even though it ultimately results in the
`BUILD.bazel` not being correctly updated.
## Test plan
* Manual testing
In `client/web/BUILD.bazel`, add a new `src` to the `web_lib` ts_project
target that includes a glob pattern.
```
...
ts_project(
name = "web_lib",
srcs = glob(["!src/playwright/*.spec.ts"]) + [
"src/Index.tsx",
"src/LegacyLayout.tsx",
"src/LegacyRouteContext.tsx",
"src/LegacySourcegraphWebApp.tsx",
"src/PageError.tsx",
"src/SearchQueryStateObserver.tsx",
"src/SourcegraphWebApp.tsx",
...
```
When you run `go run ./dev/sg bazel configure`, the command should fail
with an error message instead of returning exit 0.
## Changelog
<!-- OPTIONAL; info at
https://www.notion.so/sourcegraph/Writing-a-changelog-entry-dd997f411d524caabf0d8d38a24a878c
-->
---------
Co-authored-by: Jean-Hadrien Chabran <jean-hadrien.chabran@sourcegraph.com>
Typo when copying the flags from somewhere else, names mismatched and
didnt upload 🤦
Also adding in proper BEP emitting, it appears the build_event_log.bin
that was being output was actually for the honeyvent bazel invocation
## Test plan
CI main dry-run
https://buildkite.com/sourcegraph/sourcegraph/builds/285375
## Changelog
For more insights into whats happening during those damn build commands
Also removes emitting to honeycomb, that was my experiment that has
concluded.
## Test plan
CI
## Changelog
This is a register call that is easy to forget. When forgotten, all queries against the repo store will block forever.
In addition, this adds a hard-dependency on conf to every services startup, plus a busy loop. With multi-tenant, this will not work great because authz providers would be a global, and we instead want most things to be ephemeral so they're per-provider. This is a step toward that, but doesn't yet remove the providers global variable.
Good news, it turns out that we don't actually need to register the providers in every service! The reason they were required was to check if zero providers are configured, or if authzbypass mode is enabled.
Authz bypass mode is usually ON, except when there are problems with the authz providers, meaning some authz providers might not be able to sync permissions. Bypassing of permissions is only ever happening if there are ALSO zero providers configured.
So this is basically an optimization for the case where an instance has zero authz configured so that the SQL queries are a bit simpler. This also helps in tests because with bypass mode on and no providers configured, authz enforcement is effectively off in the repo store.
This makes it so that in tests we need to do slightly more work, but also makes for a more realistic test vs at runtime setup. Also, it's highly recommended to use mocks for DB wherever possible in more high-level components to keep tests fast.
To never have a scenario where we accidentally mess up here and enable bypass mode erroneously, this PR drops that entirely. Authz is always enforced, but when a code host connection is unrestricted (i.e., will not spawn a provider) the repos are still visible, so this should be no change over before.
## Test plan
The stack starts and works, and all CI tests are still passing. Code review should help as well.
If a build is triggered from the web the variable BUILDKITE_BUILD_AUTHOR
is not set which the msp_deploy.sh script requires. This PR uses
BUILDKITE_BUILD_CREATOR as a fallback if _AUTHOR is missing
## Test plan
Tested locally
With the https://github.com/sourcegraph/sourcegraph/pull/63985/files
PatchRelease is matched before InternalRelease leading to the wrong
build being generated.
We therefore move the Promote and Internal Release runtypes higher in
priority so that they get matched first.
## Test plan
```
export RELEASE_INTERNAL=true
export VERSION="5.5.2463"
go run ./dev/sg ci preview
```
👇🏼
```
go run ./dev/sg ci preview
⚠️ Running sg with a dev build, following flags have different default value unless explictly set: skip-auto-update, disable-analytics
If the current branch were to be pushed, the following pipeline would be run:
Parsed diff:
changed files: [WORKSPACE client/web-sveltekit/BUILD.bazel client/web-sveltekit/playwright.config.ts client/web-sveltekit/src/lib/navigation/GlobalHeader.svelte client/web-
sveltekit/src/routes/[...repo=reporev]/(validrev)/(code)/page.spec.ts client/web/src/cody/chat/new-chat/NewCodyChatPage.tsx client/web/src/cody/sidebar/new-cody-sidebar/NewCodySidebar.tsx
client/web/src/cody/sidebar/new-cody-sidebar/NewCodySidebarWebChat.tsx client/web/src/enterprise/batches/settings/AddCredentialModal.tsx
client/web/src/enterprise/batches/settings/BatchChangesCreateGitHubAppPage.tsx client/web/src/repo/blame/hooks.ts client/web/src/repo/blame/shared.ts cmd/frontend/auth/user.go
cmd/frontend/auth/user_test.go cmd/frontend/internal/codycontext/context.go cmd/frontend/internal/codycontext/context_test.go deps.bzl dev/ci/push_all.sh dev/ci/runtype/runtype.go go.mod go.sum
internal/codeintel/uploads/BUILD.bazel internal/codeintel/uploads/internal/background/backfiller/BUILD.bazel internal/codeintel/uploads/internal/background/backfiller/mocks_test.go
internal/codeintel/uploads/internal/background/commitgraph/BUILD.bazel internal/codeintel/uploads/internal/background/commitgraph/job_commitgraph.go
internal/codeintel/uploads/internal/background/expirer/BUILD.bazel internal/codeintel/uploads/internal/background/expirer/mocks_test.go
internal/codeintel/uploads/internal/background/processor/BUILD.bazel internal/codeintel/uploads/internal/background/processor/mocks_test.go internal/codeintel/uploads/internal/store/BUILD.bazel
internal/codeintel/uploads/internal/store/commitdate.go internal/codeintel/uploads/internal/store/commitdate_test.go internal/codeintel/uploads/internal/store/observability.go
internal/codeintel/uploads/internal/store/store.go internal/codeintel/uploads/mocks_test.go internal/database/migration/shared/data/cmd/generator/consts.go
internal/database/migration/shared/data/stitched-migration-graph.json package.json pnpm-lock.yaml schema/schema.go schema/site.schema.json]
diff changes: "Go, Client, pnpm, Docs, Shell"
The generated build pipeline will now follow, see you next time!
• Detected run type: Internal release
• Detected diffs: Go, Client, pnpm, Docs, Shell
• Computed variables:
• VERSION=5.5.2463
• Computed build steps:
• Aspect Workflow specific steps
• 🤖 Generated steps that include Buildifier, Gazelle, Test and Integration/E2E tests
• Image builds
• :bazel::packer: 🚧 Build executor image
• :bazel: Bazel prechecks & build sg
• :bazel:⏳ BackCompat Tests
• :bazel:🧹 Go mod tidy
• Linters and static analysis
• 🍍:lint-roller: Run sg lint → depends on bazel-prechecks
• Client checks
• :java: Build (client/jetbrains)
• :vscode: Tests for VS Code extension
• :stylelint: Stylelint (all)
• Security Scanning
• Semgrep SAST Scan
• Publish candidate images
• :bazel::docker: Push candidate Images
• End-to-end tests
• :bazel::docker::packer: Executors E2E → depends on bazel-push-images-candidate
• Publish images
• :bazel::packer: ✅ Publish executor image → depends on executor-vm-image:candidate
• :bazel:⤴️ Publish executor binary
• :bazel::docker: Push final images → depends on main::test main::test_2
• Release
• Release tests → depends on bazel-push-images
• Finalize internal release
```
<!-- REQUIRED; info at
https://docs-legacy.sourcegraph.com/dev/background-information/testing_principles
-->
## Changelog
<!-- OPTIONAL; info at
https://www.notion.so/sourcegraph/Writing-a-changelog-entry-dd997f411d524caabf0d8d38a24a878c
-->
This PR is a second attempt at improving push_all.sh, continuing on from
(and inspired by) https://github.com/sourcegraph/sourcegraph/pull/63391.
As a recap, that PR uses
[--script_path](https://bazel.build/reference/command-line-reference#flag--script_path)
to emit a short bash script for every `oci_push` target, which
essentially does minor setup + invokes the executable as if running
`bazel run`.
While the idea in https://github.com/sourcegraph/sourcegraph/pull/63391
was good, it trades concurrent server locking with an equal amount of
overhead in sequentially building the scripts. By observing the
scripts<b>[1]</b> that it would emit, we can notice a few things:
- The path
`/home/noah/.cache/bazel/_bazel_noah/8fd1d20666a46767e7f29541678514a0/execroot/__main__/bazel-out/k8-fastbuild/bin/`
shows up twice, which is the same path that `./bazel-bin` points at
- The script only `cd`'s to a path, unsets some environment variables,
and then executes the underlying script of the target.
The path can be observed to be a combination of bazel-bin, the target's
package (`//cmd/batcheshelper` in this case), as well as the target with
some extra static strings (`candidate_push` with `push_` prefix and
`.sh{,.runfiles}` suffixes for the script & its runfiles respectively).
Knowing this, and assuming that this is reliably so, we can opt to
recreate this manually instead, saving on the hefty overhead of `bazel
run --script_path`.
The current average times for `Push candidate images` and `Push final
images` are ~7m50s and ~8m30s respectively. While the example
main-dry-run build
[here](https://buildkite.com/sourcegraph/sourcegraph/builds/284041#0190e54a-9aaa-471a-81bf-623fce6ffa45)
isnt fully representative of how much rebuilding is required, it sets a
pretty solid 3m20s baseline.
Note this may break with rules_oci changes, but imo thats a small and
very infrequent cost to pay for cleaner log output + shaving a good
piece of time off.
<details><summary><b>[1]</b> A <code>--script_path</code>
example</summary>
```
#!/nix/store/mqc7dqwp046lh41dhs7r7q7192zbliwd-bash/bin/bash
cd /home/noah/.cache/bazel/_bazel_noah/8fd1d20666a46767e7f29541678514a0/execroot/__main__/bazel-out/k8-fastbuild/bin/cmd/batcheshelper/push_candidate_push.sh.runfiles/__main__ && \
exec env \
-u JAVA_RUNFILES \
-u RUNFILES_DIR \
-u RUNFILES_MANIFEST_FILE \
-u RUNFILES_MANIFEST_ONLY \
-u TEST_SRCDIR \
BUILD_WORKING_DIRECTORY=/home/noah/Sourcegraph/sourcegraph \
BUILD_WORKSPACE_DIRECTORY=/home/noah/Sourcegraph/sourcegraph \
/home/noah/.cache/bazel/_bazel_noah/8fd1d20666a46767e7f29541678514a0/execroot/__main__/bazel-out/k8-fastbuild/bin/cmd/batcheshelper/push_candidate_push.sh "$@"
```
</details>
## Test plan
Observe a `sg ci build main-dry-run`
[here](https://buildkite.com/sourcegraph/sourcegraph/builds/284041#0190e54a-9aaa-471a-81bf-623fce6ffa45).
## Changelog
<!-- OPTIONAL; info at
https://www.notion.so/sourcegraph/Writing-a-changelog-entry-dd997f411d524caabf0d8d38a24a878c
-->
Closes [#1110](https://github.com/sourcegraph/devx-support/issues/1110)
Closes DINF-96
We don't print the stdErr when a command fails … in particular when git
fails. Therefore we see very little in the panic of what went wrong.
Explanation:
> There's a weird behavior that occurs where an error isn't accessible
in the err variable
// from a *Cmd executing a git command after calling CombinedOutput().
// This occurs due to how Git handles errors and how the exec package in
Go interprets the command's output.
// Git often writes error messages to stderr, but it might still exit
with a status code of 0 (indicating success).
// In this case, CombinedOutput() won't return an error, but the error
message will be in the out variable.
## Test plan
Manual testing
```go
func main() {
ctx := context.Background()
cmd := exec.CommandContext(ctx, "git", "rev-parse", "--is-inside-work-tree")
out, err := handleGitCommandExec(cmd)
if err != nil {
// er := errors.Wrap(err, fmt.Sprintf("idsdsd: %s", string(out)))
panic(err)
}
fmt.Println("hello", string(out))
}
```
## Changelog
<!-- OPTIONAL; info at
https://www.notion.so/sourcegraph/Writing-a-changelog-entry-dd997f411d524caabf0d8d38a24a878c
-->
Delivery Manifest step has started to run `bazel build` commands, in them clobbering our execlog artifacts. We should only emit it for the test buildkite jobs (at least for the time being), as it currently doesnt make sense for e.g. the image push jobs which contain multiple invocations
## Test plan
CI
## Changelog
We currently don't publish images from the new-style patch release
branches like `5.4.5099`, as this is all performed using the new release
tooling.
In order to improve the release process, we (Security) would like to run
a daily scan of the current set of images built from the patch release
branch. Currently we only scan images built from `main`, but these
slowly diverge from the patch release branch in the 2 week window
between a monthly release and the patch release.
To give a specific example, we currently have no easy/automated way to
scan images from the `5.4.5099` branch that a release will be cut from
this afternoon until the release team run the internal release process.
This PR updates the pipeline so that whenever a new commit is pushed to
the patch release branch, it will publish a new set of images and
include the tag `<branch>-insiders`. Currently just pushing to
us.gcr.io, but equally could push to dockerhub.
Example of the jobfile for a matching branch after this PR:
`bazel --bazelrc=/tmp/aspect-generated.bazelrc
--bazelrc=.aspect/bazelrc/ci.sourcegraph.bazelrc run
//cmd/batcheshelper:candidate_push --stamp
--workspace_status_command=./dev/bazel_stamp_vars.sh -- --tag
dc438648b0 --tag dc438648b0cc_2024-06-20 --tag dc438648b0cc_279230
--tag will/5.4.9999-insiders --repository
us.gcr.io/sourcegraph-dev/batcheshelper && echo -e
'<tr><td>batcheshelper</td><td><code>us.gcr.io/sourcegraph-dev</code></td><td><code>dc438648b0cc</code>,
<code>dc438648b0cc_2024-06-20</code>, <code>dc438648b0cc_279230</code>,
<code>will/5.4.9999-insiders</code></td></tr>'
>>./annotations/pushed_images.md`
[Example buildkite
run](https://buildkite.com/sourcegraph/sourcegraph/builds/279230#_)
where the pattern was updated to match this branch, and pushing
non-candidate images was disabled.
This resolves one part of
[SEC-1734](https://linear.app/sourcegraph/issue/SEC-1734/scan-images-from-patch-release-branches)
<!-- 💡 To write a useful PR description, make sure that your description
covers:
- WHAT this PR is changing:
- How was it PREVIOUSLY.
- How it will be from NOW on.
- WHY this PR is needed.
- CONTEXT, i.e. to which initiative, project or RFC it belongs.
The structure of the description doesn't matter as much as covering
these points, so use
your best judgement based on your context.
Learn how to write good pull request description:
https://www.notion.so/sourcegraph/Write-a-good-pull-request-description-610a7fd3e613496eb76f450db5a49b6e?pvs=4
-->
## Test plan
- Manual testing of buildkite pipeline
<!-- All pull requests REQUIRE a test plan:
https://docs-legacy.sourcegraph.com/dev/background-information/testing_principles
-->
## Changelog
<!--
1. Ensure your pull request title is formatted as: $type($domain): $what
2. Add bullet list items for each additional detail you want to cover
(see example below)
3. You can edit this after the pull request was merged, as long as
release shipping it hasn't been promoted to the public.
4. For more information, please see this how-to
https://www.notion.so/sourcegraph/Writing-a-changelog-entry-dd997f411d524caabf0d8d38a24a878c?
Audience: TS/CSE > Customers > Teammates (in that order).
Cheat sheet: $type = chore|fix|feat $domain:
source|search|ci|release|plg|cody|local|...
-->
<!--
Example:
Title: fix(search): parse quotes with the appropriate context
Changelog section:
## Changelog
- When a quote is used with regexp pattern type, then ...
- Refactored underlying code.
-->
the executor image and docker mirror image should now follow the
following naming convention:
Image family:
`sourcegraph-executors-[nightly|internal|'']-<MAJOR>-<MINOR>`
Image name:
`sourcegraph-executor-[nightly|internal|'']-<MAJOR>-<MINOR>-<BUILD_NUMBER>`
example:
Image family: `sourcegraph-executors-5-4`
Image name: `sourcegraph-executor-5-4-277666`
## What happens during releases and _not_ releases?
#### Nightly
**`nightly` suffix**
Image family: `sourcegraph-executors-nightly-<MAJOR>-<MINOR>`
Image name:
`sourcegraph-executor-nightly-<MAJOR>-<MINOR>-<BUILD_NUMBER>`
#### Internal
**`internal` suffix**
Image family: `sourcegraph-executors-internal-<MAJOR>-<MINOR>`
Image name:
`sourcegraph-executor-internal-<MAJOR>-<MINOR>-<BUILD_NUMBER>`
#### Public / Promote to public
** No suffix **
Image family: `sourcegraph-executors-<MAJOR>-<MINOR>`
Image name: `sourcegraph-executor-<MAJOR>-<MINOR>-<BUILD_NUMBER>`
> [!IMPORTANT]
> Should we keep the imagine name stable at
`sourcegraph-executor-<MAJOR>-<MINOR>-<BUILD_NUMBER>`
> and only change the family name?
>
> **Why?**
>
> The Image family dictates the collection of images and that changes
each major minor and or release phase so there is really no use in
changing the image name too, except at a glance you can see from the
name what image family it belongs to?
## Test plan
<!-- All pull requests REQUIRE a test plan:
https://docs-legacy.sourcegraph.com/dev/background-information/testing_principles
-->
## Changelog
<!--
1. Ensure your pull request title is formatted as: $type($domain): $what
2. Add bullet list items for each additional detail you want to cover
(see example below)
3. You can edit this after the pull request was merged, as long as
release shipping it hasn't been promoted to the public.
4. For more information, please see this how-to
https://www.notion.so/sourcegraph/Writing-a-changelog-entry-dd997f411d524caabf0d8d38a24a878c?
Audience: TS/CSE > Customers > Teammates (in that order).
Cheat sheet: $type = chore|fix|feat $domain:
source|search|ci|release|plg|cody|local|...
-->
<!--
Example:
Title: fix(search): parse quotes with the appropriate context
Changelog section:
## Changelog
- When a quote is used with regexp pattern type, then ...
- Refactored underlying code.
-->
These functions are used in test code, which makes it tricker to
pass in loggers for checking requests and responses. Instead, make
the API method-based. This introduces slightly more boilerplate
since Client construction is fallible, but it allows calling code
to pass in loggers more easily for debugging test failures.
