Closes
https://linear.app/sourcegraph/issue/GRAPH-765/use-surroundingcontent-returned-from-the-search-client
The primary reason for doing this is to avoid sending a bunch of extra
calls for file contents to gitserver, which dominate the profile/trace
after implementing the remaining opts from
https://github.com/sourcegraph/sourcegraph/pull/63971
Also limits the API to only return the line containing the match for now
as discussed with Camden.
We might need to bring back the `lineGetter` for precise usages, but it
should be easy enough to revive at that point /cc @varungandhi-src
## Test plan
Updated unit tests, some manual testing in the API console
With the https://github.com/sourcegraph/sourcegraph/pull/63985/files
PatchRelease is matched before InternalRelease leading to the wrong
build being generated.
We therefore move the Promote and Internal Release runtypes higher in
priority so that they get matched first.
## Test plan
```
export RELEASE_INTERNAL=true
export VERSION="5.5.2463"
go run ./dev/sg ci preview
```
👇🏼
```
go run ./dev/sg ci preview
⚠️ Running sg with a dev build, following flags have different default value unless explictly set: skip-auto-update, disable-analytics
If the current branch were to be pushed, the following pipeline would be run:
Parsed diff:
changed files: [WORKSPACE client/web-sveltekit/BUILD.bazel client/web-sveltekit/playwright.config.ts client/web-sveltekit/src/lib/navigation/GlobalHeader.svelte client/web-
sveltekit/src/routes/[...repo=reporev]/(validrev)/(code)/page.spec.ts client/web/src/cody/chat/new-chat/NewCodyChatPage.tsx client/web/src/cody/sidebar/new-cody-sidebar/NewCodySidebar.tsx
client/web/src/cody/sidebar/new-cody-sidebar/NewCodySidebarWebChat.tsx client/web/src/enterprise/batches/settings/AddCredentialModal.tsx
client/web/src/enterprise/batches/settings/BatchChangesCreateGitHubAppPage.tsx client/web/src/repo/blame/hooks.ts client/web/src/repo/blame/shared.ts cmd/frontend/auth/user.go
cmd/frontend/auth/user_test.go cmd/frontend/internal/codycontext/context.go cmd/frontend/internal/codycontext/context_test.go deps.bzl dev/ci/push_all.sh dev/ci/runtype/runtype.go go.mod go.sum
internal/codeintel/uploads/BUILD.bazel internal/codeintel/uploads/internal/background/backfiller/BUILD.bazel internal/codeintel/uploads/internal/background/backfiller/mocks_test.go
internal/codeintel/uploads/internal/background/commitgraph/BUILD.bazel internal/codeintel/uploads/internal/background/commitgraph/job_commitgraph.go
internal/codeintel/uploads/internal/background/expirer/BUILD.bazel internal/codeintel/uploads/internal/background/expirer/mocks_test.go
internal/codeintel/uploads/internal/background/processor/BUILD.bazel internal/codeintel/uploads/internal/background/processor/mocks_test.go internal/codeintel/uploads/internal/store/BUILD.bazel
internal/codeintel/uploads/internal/store/commitdate.go internal/codeintel/uploads/internal/store/commitdate_test.go internal/codeintel/uploads/internal/store/observability.go
internal/codeintel/uploads/internal/store/store.go internal/codeintel/uploads/mocks_test.go internal/database/migration/shared/data/cmd/generator/consts.go
internal/database/migration/shared/data/stitched-migration-graph.json package.json pnpm-lock.yaml schema/schema.go schema/site.schema.json]
diff changes: "Go, Client, pnpm, Docs, Shell"
The generated build pipeline will now follow, see you next time!
• Detected run type: Internal release
• Detected diffs: Go, Client, pnpm, Docs, Shell
• Computed variables:
• VERSION=5.5.2463
• Computed build steps:
• Aspect Workflow specific steps
• 🤖 Generated steps that include Buildifier, Gazelle, Test and Integration/E2E tests
• Image builds
• :bazel::packer: 🚧 Build executor image
• :bazel: Bazel prechecks & build sg
• :bazel:⏳ BackCompat Tests
• :bazel:🧹 Go mod tidy
• Linters and static analysis
• 🍍:lint-roller: Run sg lint → depends on bazel-prechecks
• Client checks
• :java: Build (client/jetbrains)
• :vscode: Tests for VS Code extension
• :stylelint: Stylelint (all)
• Security Scanning
• Semgrep SAST Scan
• Publish candidate images
• :bazel::docker: Push candidate Images
• End-to-end tests
• :bazel::docker::packer: Executors E2E → depends on bazel-push-images-candidate
• Publish images
• :bazel::packer: ✅ Publish executor image → depends on executor-vm-image:candidate
• :bazel:⤴️ Publish executor binary
• :bazel::docker: Push final images → depends on main::test main::test_2
• Release
• Release tests → depends on bazel-push-images
• Finalize internal release
```
<!-- REQUIRED; info at
https://docs-legacy.sourcegraph.com/dev/background-information/testing_principles
-->
## Changelog
<!-- OPTIONAL; info at
https://www.notion.so/sourcegraph/Writing-a-changelog-entry-dd997f411d524caabf0d8d38a24a878c
-->
This PR is a second attempt at improving push_all.sh, continuing on from
(and inspired by) https://github.com/sourcegraph/sourcegraph/pull/63391.
As a recap, that PR uses
[--script_path](https://bazel.build/reference/command-line-reference#flag--script_path)
to emit a short bash script for every `oci_push` target, which
essentially does minor setup + invokes the executable as if running
`bazel run`.
While the idea in https://github.com/sourcegraph/sourcegraph/pull/63391
was good, it trades concurrent server locking with an equal amount of
overhead in sequentially building the scripts. By observing the
scripts<b>[1]</b> that it would emit, we can notice a few things:
- The path
`/home/noah/.cache/bazel/_bazel_noah/8fd1d20666a46767e7f29541678514a0/execroot/__main__/bazel-out/k8-fastbuild/bin/`
shows up twice, which is the same path that `./bazel-bin` points at
- The script only `cd`'s to a path, unsets some environment variables,
and then executes the underlying script of the target.
The path can be observed to be a combination of bazel-bin, the target's
package (`//cmd/batcheshelper` in this case), as well as the target with
some extra static strings (`candidate_push` with `push_` prefix and
`.sh{,.runfiles}` suffixes for the script & its runfiles respectively).
Knowing this, and assuming that this is reliably so, we can opt to
recreate this manually instead, saving on the hefty overhead of `bazel
run --script_path`.
The current average times for `Push candidate images` and `Push final
images` are ~7m50s and ~8m30s respectively. While the example
main-dry-run build
[here](https://buildkite.com/sourcegraph/sourcegraph/builds/284041#0190e54a-9aaa-471a-81bf-623fce6ffa45)
isnt fully representative of how much rebuilding is required, it sets a
pretty solid 3m20s baseline.
Note this may break with rules_oci changes, but imo thats a small and
very infrequent cost to pay for cleaner log output + shaving a good
piece of time off.
<details><summary><b>[1]</b> A <code>--script_path</code>
example</summary>
```
#!/nix/store/mqc7dqwp046lh41dhs7r7q7192zbliwd-bash/bin/bash
cd /home/noah/.cache/bazel/_bazel_noah/8fd1d20666a46767e7f29541678514a0/execroot/__main__/bazel-out/k8-fastbuild/bin/cmd/batcheshelper/push_candidate_push.sh.runfiles/__main__ && \
exec env \
-u JAVA_RUNFILES \
-u RUNFILES_DIR \
-u RUNFILES_MANIFEST_FILE \
-u RUNFILES_MANIFEST_ONLY \
-u TEST_SRCDIR \
BUILD_WORKING_DIRECTORY=/home/noah/Sourcegraph/sourcegraph \
BUILD_WORKSPACE_DIRECTORY=/home/noah/Sourcegraph/sourcegraph \
/home/noah/.cache/bazel/_bazel_noah/8fd1d20666a46767e7f29541678514a0/execroot/__main__/bazel-out/k8-fastbuild/bin/cmd/batcheshelper/push_candidate_push.sh "$@"
```
</details>
## Test plan
Observe a `sg ci build main-dry-run`
[here](https://buildkite.com/sourcegraph/sourcegraph/builds/284041#0190e54a-9aaa-471a-81bf-623fce6ffa45).
## Changelog
<!-- OPTIONAL; info at
https://www.notion.so/sourcegraph/Writing-a-changelog-entry-dd997f411d524caabf0d8d38a24a878c
-->
Adds better observability, and follows more standard patterns. I don't
know why I did it that way a year ago when I implemented this 🤦
Test plan: Code review to find behavior changes that CI cannot cover;
also CI.
Monikers carry a bunch of redundant fields which basically consist of
information parsed out of the symbol name. This patch cleans up
some APIs to work with symbol names directly instead of monikers.
This moves the job from `frontend` to `worker`.
Note:
I believe there was a subtle bug in the original code. We only evaluated
the
feature flag on startup, which means that changes of the feature flag
only took effect after a restart.
## Test plan:
new unit test
Automatically generated PR to update package lockfiles for Sourcegraph
base images.
Built from Buildkite run
[#283904](https://buildkite.com/sourcegraph/sourcegraph/builds/283904).
## Test Plan
- CI build verifies image functionality
Co-authored-by: Buildkite <buildkite@sourcegraph.com>
This patch changes the location querying code so that:
1. We're populating structures corresponding to SCIP instead of LSIF
(with "scheme" and "identifier" inside "MonikerData")
2. Avoid repeatedly allocating a constant string 'scip' for the scheme
only to throw it away later.
3. Makes the two queries and their scanning code more similar for easier
comparison. When I land precise usagesForSymbol, I will de-duplicate
some of the scanning code between these two queries.
I have avoided renaming all of the local variables to avoid creating
more noise.
## Test plan
Covered by existing tests.
Also consolidates mocks to avoid wasting time generating
and compiling them repeatedly, and reducing pollution of
Find Usages results in the editor (yes, I know these can be
filtered out, but it's silly to have multiple copies).
Closes
https://linear.app/sourcegraph/issue/GRAPH-742/simplify-gittreetranslator-implementation
- use unified diff format and 0 context lines to allow translation by
just using hunk headers
- only request hunks per file once, and sync follow-up requests for the
same file
- don't bother LRU caching the full hunk contents, just store 4 int32's
per hunk
- replace linear search through sorted hunks with binary search
Current PR keeps the old PR and just proxies to the new implementation.
Once this PR is reviewed and merged I'll remove the old API and update
all callsites.
## Test plan
New/existing tests
This makes it easier to chart this metric on a daily basis. Transmitting
the count in a rolling 24-hour window is harder.
Note that this new usagestats schema for saved searches has not yet been
released in a patch or stable release.
## Test plan
CI
We need to ensure prod data does not end up in the dev Enterprise Portal
instance once we start syncing. This change adds support for the DevOnly
option in ListSubscriptions and strictly enforces it there (i.e. dev
subscriptions cannot end up in the prod instance either).
I've tried to minimize the changes elsewhere to reduce the impact of
this change for now.
## Test plan
Tests
Implements insert and retrieval of subscription conditions, similar to
#63792
Unlike for licenses, which have a more limited lifecycle (create and
revoke), subscriptions are longer-lived and may be updated frequently.
So the storage layer allows the caller to provide the conditions that
are of interest for recording.
Part of https://linear.app/sourcegraph/issue/CORE-156, but doesn't yet
use it from the API.
Part of https://linear.app/sourcegraph/issue/CORE-100
## Test plan
Integration tests
As we start adding more stuff here, we need a bit more flexibility to
add tests without it becoming overwhelming. The current copy-pasta is
quite difficult to read, this standardizes the expected behaviour for
the Update and List tests.
## Test plan
CI
Fixes [inc-313
](https://sourcegraph.slack.com/archives/C07D8ETFAQP/p1721746352871739?thread_ts=1721745191.614829&cid=C07D8ETFAQP)
It should solve problems with the old auth flow in the new Svelte
version. Currently, the go router doesn't know anything about
post-sign-up flow, which is used in the JB Cody extension. Because of
this, it passes it to the Svelte version which doesn't implement this
route on the client (but the react version did have it on the client
which is why it worked before, before we rolled out Svelte version by
default)
## Test plan
- There is no good way to check this properly, but after this PR you
have to see a react (old) version of the App when you go to the
/post-sigh-up router
Closes [#1110](https://github.com/sourcegraph/devx-support/issues/1110)
Closes DINF-96
We don't print the stdErr when a command fails … in particular when git
fails. Therefore we see very little in the panic of what went wrong.
Explanation:
> There's a weird behavior that occurs where an error isn't accessible
in the err variable
// from a *Cmd executing a git command after calling CombinedOutput().
// This occurs due to how Git handles errors and how the exec package in
Go interprets the command's output.
// Git often writes error messages to stderr, but it might still exit
with a status code of 0 (indicating success).
// In this case, CombinedOutput() won't return an error, but the error
message will be in the out variable.
## Test plan
Manual testing
```go
func main() {
ctx := context.Background()
cmd := exec.CommandContext(ctx, "git", "rev-parse", "--is-inside-work-tree")
out, err := handleGitCommandExec(cmd)
if err != nil {
// er := errors.Wrap(err, fmt.Sprintf("idsdsd: %s", string(out)))
panic(err)
}
fmt.Println("hello", string(out))
}
```
## Changelog
<!-- OPTIONAL; info at
https://www.notion.so/sourcegraph/Writing-a-changelog-entry-dd997f411d524caabf0d8d38a24a878c
-->
We don't appear to make use of the stamp values for any rust_binary
targets, so lets not stamp it to save on cache invalidations
## Test plan
```
$ bazel build --stamp --workspace_status_command=./dev/bazel_stamp_vars.sh $(bazel query 'kind("rust_binary", //...)')
...
$ sha256sum bazel-bin/docker-images/syntax-highlighter/scip-ctags bazel-bin/docker-images/syntax-highlighter/crates/scip-syntax/scip-syntax bazel-bin/docker-images/syntax-highlighter/syntect_server
...
$ bazel build $(bazel query 'kind("rust_binary", //...)')
...
$ sha256sum bazel-bin/docker-images/syntax-highlighter/scip-ctags bazel-bin/docker-images/syntax-highlighter/crates/scip-syntax/scip-syntax bazel-bin/docker-images/syntax-highlighter/syntect_server
...
```
observe identical checksums
## Changelog
<!-- OPTIONAL; info at
https://www.notion.so/sourcegraph/Writing-a-changelog-entry-dd997f411d524caabf0d8d38a24a878c
-->
Implements server-side part of AI-132, exposing a single-repo context
endpoint backed by Zoekt.
Differences from the current Enterprise endpoint (that is left
untouched):
- supports a single repo only
- supports passing a repo name, not Sourcegraph-assigned repo ID
- has a shorter, 5s timeout
- reports partial errors (if we hit issues converting search results to
git results, or if some retrievers time out)
## Test plan
- tested locally with
```
{
chatContext(
query: "squirrel"
repo: "github.com/sourcegraph/sourcegraph"
interactionId: "foo"
) {
contextItems {
score
retriever
item {
... on FileChunkContext {
blob {
path
repository {
id
name
}
commit {
oid
}
url
}
startLine
endLine
chunkContent
}
}
}
partialErrors
}
}
```
Pass down semantic information down as much as possible,
delaying lossy conversion to table names till the very end.
I will be introducing the `exhaustive` linter in a future PR
(https://golangci-lint.run/usage/linters/#exhaustive)
so that we can get a static error if you don't switch exhaustively.
For certain types, we do not want the zero-value initialization for structs.
This means we need to trade off readability vs exhaustive initialization
checking, as the Go syntax `Foo{Bar: bar}` is more readable, but doesn't do
exhaustiveness checking, and `Foo{bar}` does check for exhaustiveness but can be less
readable depending on context.
For now, the check is only introduced for one type, and is meant to be
opt-in so that code authors may choose for stricter checking.
Relates to
https://linear.app/sourcegraph/issue/REL-81/service-definition-otel-collector
but does not close it.
Some of the StandardConfig features (notably PodTemplateConfig) assume
that there is only one pod template "thing" per service. The otel
service contains an agent daemonset and a collector deployment, which
have quite different characteristics. We're less likely to paint
ourselves into a corner if we split these services at the config level.
This patch moves the mocks for `codenav/../lsifstore` package to a
separate package `lsifstore/mocks` instead of it living in `codenav`.
The problem is that if you update the `LsifStore` interface, then you
get an error when trying to regenerate mocks, as regeneration happens
in the `codenav` package, and that package's old mocks no longer
satisfy the new interface (so there is an error when looking at other test
files). Moving the codegen to a separate package avoids this problem.
The current description doesn't do a good job of explaining why Search
Jobs is useful and how to create a job.
## Test plan:
visual inspection
Co-authored-by: Erik Seliger <erikseliger@me.com>
This moves the janitor jobs for event logs from `frontend` to `worker`.
Next up: move the other recurring jobs as well.
## Test plan
- new unit tests
- manual testing: the worker dashboard showed the job is running
