Searcher doesn't speak to the database nor has it for a long time. See
https://github.com/sourcegraph/sourcegraph/pull/61463
Test Plan: The following command is empty
go run ./dev/depgraph/ summary internal/database | grep 'cmd/searcher'
The `getSourceRange` function can have a range translation failure,
in which case it would be wrong to propagate the range directly
without the commit information. So skip the ranges in that case.
Introduces basic (and incomplete) UI support for perforce, including displaying changelist ids,
listing changelists, and removing references to commits and branches.
The URL path for streaming blame is ambiguous when the file path
contains `/stream/`. The pattern looks like
`/blame/<repo_name>@<revision>/stream/file/path.txt`. However, if the
file contains `/stream/`, the revision is matched greedily up until the
last stream, so we end up with a revision that looks like
`81af3g/stream/my/path`, which is in invalid revision that results in a
404.
This makes the URL pattern unambiguous by adding a `/-/` element after
the revision, which is not allowed in a revision name and acts as a path
separator. So now, the pattern looks like
`/blame/<repo_name>@<revision>/-/stream/file/path.txt`.
Note, this is a public-facing breaking change, but I don't think it
really matters since I don't expect any customers use our streaming
blame API
Automatically generated PR to update package lockfiles for Sourcegraph
base images.
Built from Buildkite run
[#285676](https://buildkite.com/sourcegraph/sourcegraph/builds/285676).
## Test Plan
- CI build verifies image functionality
Co-authored-by: Buildkite <buildkite@sourcegraph.com>
With SRCH-802 we saw error messages when batch changes may still retry
and resolve the problem. To give a better distinction between resolvable
and non-resolvable errors, I'm changing the colour variation from error
to warning if the changeset has not been marked as `FAILED` yet.
Before:
<img width="913" alt="Screenshot 2024-08-02 at 12 44 27"
src="https://github.com/user-attachments/assets/b192c5e9-d23b-460c-82a7-a039edbca3f5">
After:
<img width="1355" alt="Screenshot 2024-08-02 at 12 36 23"
src="https://github.com/user-attachments/assets/02e231a7-168a-4fe9-bd3b-014d810fd236">
## Test plan
Manual testing
## Changelog
- Batch changes that are still retrying now show a warning instead of an
error.
ResolveRevision additionally adds a "^0" to the spec to make sure we
actually check if it exists. This is important, since most uses of the
ChangedFiles API pass in commit shas which do not get resolved by git
unless the "^0" is present.
I noticed this since hybrid search in searcher started to fail if the
commit was missing. In particular Zoekt for empty repositories uses a
fake SHA of 404aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa so when a repo
started getting cloned hybrid search would fail until it was indexed.
Hybrid search explicitly checks for the NotFound error, so hybrid search
failing was a a regression from moving away from the SymbolDiff API.
I noticed we were using a much older version, so code nav
is a bit janky on S2 as the old version had index data from
lsif-go and not scip-go, causing cross-repo nav from
sg/sg to sg/log to not work.
When determining the set of visible uploads at a given commit,
we group uploads from scip-K and lsif-K for ranking purposes
to allow them to shadow each other. Generally, scip-K will end
up shadowing lsif-K, avoiding sending the client a mix of data
from a much older lsif-K index when newer data from a scip-K
index is available.
Fixes https://linear.app/sourcegraph/issue/GRAPH-787
The logic requires some final tie-breaking to avoid non-determinism
in case when there is a scip-K and lsif-K index at the same distance,
which I've done based on the upload_id in descending order
(upload IDs are monotonically increasing). Earlier, some code was
doing tie-breaking based on upload_id in _ascending_ order,
so I've changed that for consistency. This caused some test failures,
so I've updated the corresponding struct literals.
I've left some PR review comments to describe each modification
to the existing test cases.
## Test plan
Added a new dedicated test checking for shadowing behavior.
## Changelog
Fixes a bug where old LSIF uploads would also be used for code
navigation even when newer SCIP uploads were available for the
same language, potentially leading to duplicate results in the reference
panel. With this change, scip-go uploads shadow the uploads
for lsif-go, and similarly for other indexers following the
scip-X/lsif-X naming
convention.
- Update the "getting started" link in the quickstart guide to point to
the correct URL
- Remove the outdated developer help links as they are no longer
relevant
## Test plan
Test the links manually
When the `modelconfig` system was introduced, the code was written in a
way to go out of its way to ensure that the configuration data was
correct. Unfortunately, this doesn't interact well with the reality of
the Sourcegraph instance and how site configuration is set.
- Failing to initialize the `modelconfig` component if the site config
is invalid prevents the `frontend` binary from starting up. Leading to
it crash looping / an outage. Not good.
- The site config can be forced-in, regardless of any validators that
are registered. So it's definitely possible for the model configuration
to be invalid.
- Some of our validation checks were a bit too onerous anyways.
This PR refactors the `modelconfig/init.go` file to try really hard to
not fail on application start up. And instead just do its best to
initialize the `modelconfig` service.
- If the site configuration data is invalid, it will just be ignored.
And therefor default to the Sourcegraph instance using Cody Gateway /
Sourcegraph-supplied models.
- If _rendering_ the LLM model configuration fails (e.g. merging the
static configuration and admin-supplied site config), then we just
initialize the modelconfig service with 0 providers, and models. (But
still let things start up.)
These changes should fix
[PRIME-449](https://linear.app/sourcegraph/issue/PRIME-449), but this
still isn't enough for a good user experience. (Since we are now just
swallowing/ignoring modelconfig errors.)
So my next PR will be to add the site configuration validation logic, so
that we surface any of the configuration errors to site admins.
## Test plan
Tested this manually, and could no longer get my `frontend` job to fail
on startup.
## Changelog
NA
Due to CORS rules, cody web cannot make requests to arbitrary URLs.
Because of this, mentions for URLs do not currently work in Cody Web.
This adds a GraphQL endpoint to the cody context resolvers that resolves
the contents of a mentioned URL.
Closes
https://linear.app/sourcegraph/issue/SRCH-561/add-support-for-file-ranges-in-cody-web
## Test plan
- Check that you can mention files with ranges
- Check that as you open Cody chat on the blob UI page and you have
selected lines in blob UI then chat should have an initial context
mention with line range
- Check that as you change line selection it updates mention file chip
range (only if you don't have any other text)
In order to reduce the cost of calls to auth-gated endpoints, cache
valid admin passwords in-memory. The appliance's frontend calls
auth-gated endpoints in a tight loop, and bcrypt checking is
intentionally an expensive operation.
This could occasionally cause the appliance-frontend to disconnect from
the backend. We observed frontend's nginx reporting an upstream
connection close, and exec'ing into its pod and curling the backend
regularly hung.
When the admin has first installed Code Search Suite, the appliance
waits for the admin to click an "I'm ready" button. This causes the
appliance to unblock a background thread that periodically checks the
health of sg-frontend. When it is healthy, it ensures that the
ingress-facing frontend is pointed to sg-frontend. And when it is not,
it points to the appliance-frontend. Pointing to the appliance-frontend
is its initial state pre-install, and given that we've just installed
sg, the appliance switches the service over quickly.
Meanwhile, clicking this button transitions the frontend to a "refresh"
state (this being one of the states in its state machine). This causes
the UI to reload the web page. The reason we have to do this is that it
is a way to "redirect to yourself". If the ingress-facing service has
been repointed, refreshing like this will show site-admin, which is the
desired behavior. The issue this commit fixes, is that this is racy:
upon refresh, the browser tab queries the appliance (via an nginx proxy
hosted on the same domain serving appliance-frontend) for its state. We
have to store state on the backend (specifically, we use a ConfigMap
annotation), so that the appliance can do the right thing if it has been
rebooted at any time. This will help power future features such as
UI-driven upgrades. The race occurs if, upon refresh, the ingress-facing
service has been flipped over to sg-frontend. The appliance API that
answered the state questions is no longer available!
In general, we can't tell the difference between this expected turn of
events, and a state in which the backend can't be reached. This commit
mitigates the race by setting the appliance UI to refresh if it cannot
reach the appliance API. This looks no different to a "disconnected"
state if things really are broken, but in the expected path, it will
resolve the race by retrying.
This commit reliably causes the appliance-driven installation flow to
redirect to site-admin after clicking "ready", according to my
experimentation in minikube. I suspect that this would be the case even
without https://github.com/sourcegraph/sourcegraph/pull/64213, which
fixes an unrelated performance issue. I suspect we need both, otherwise
the appliance UI will regularly disconnect for prolonged periods of
time, which is confusing.
Closes
https://linear.app/sourcegraph/issue/REL-308/appliance-frontend-seems-to-disconnect-the-backend-during-installation
This change removes the backend smart search logic. After this, searches
with smart search enabled (`sm=1`) will be executed in the default
'precise' mode (`sm=0`). For old searches that use `sm=1` and
`patterntype=standard`, it's possible that they will now return no
results.
Looking at telemetry, only 0.1% of searches on dot com trigger any smart
search rule. So this change should only affect a small percentage of
usage. To mitigate the impact on these rare cases, this PR adds an alert
whenever there are no results and smart search is enabled, suggesting
users switch to keyword search. (This will help in the majority of
cases, since the most frequent smart search rule rewrites literal
queries to use 'AND' between terms).
Closes SPLF-92
This PR brings the most recent version of @sourcegraph/cody-web package
- Improvements in how we fetch providers and debounce logic for their
query
- Fix with switching-chat actions and providers list
## Test plan
- Manual checks over cody web (React and Svelte version)
Part of
[GRAPH-759](https://linear.app/sourcegraph/issue/GRAPH-759/issue-with-apex-extension-not-appearing-for-langapex)
Linguist only supports a subset of the file extensions often used for
the Apex programming languages. This PR adds support for the main set
commonly used.
**Key changes**
1. Adds all extensions for Apex
2. Update our logic to handle multiple extensions for one language
3. Update tests to ensure we only manually map languages if they don't
exist OR have different extensions in go-enry (prevents us from
duplicating entries completely from go-enry)
## Test plan
- [x] Update unit tests
- [x] Validate locally by testing the language filter
Fixes SPLF-119
This adds a background job to Search Jobs that periodically scans for
finished jobs to aggregate the status, upload logs, and clean up the
tables. This drastically reduces the size of the tables and improves the
performance of the Search Jobs GQL API.
For example, with this change, a finished search job on .com only has 1
entry in the database, whereas before it could have several millions if
we searched each repository.
Notes:
- the diff seems larger than it actually is. I left a couple of comments
to help the reviewers.
## Test plan:
- new unit tests
- manual testing:
I ran a couple of search jobs locally (with the janitor job interval set
to 1 min) and checked that
- logs are uploaded to `blobstore-go/buckets/search-jobs`
- repo jobs are deleted from `exhaustive_repo_jobs`
- logs are served from the blobstore after the janitor ran
- downloading logs while the job is running still works
## Changelog
The new background job drastically reduces the size of the
`exhaustive_*` tables and improves performance of the Search Jobs GQL
API.
To ease the transition from older search semantics to keyword search, we
[boost matches on
phrases](https://github.com/sourcegraph/sourcegraph/pull/59940). For
example if you search for `// The vararg opts parameter can include`, we
ensure the match in the method comment is highest, even though there
were no explicit quotes in the query.
We decided to limit this boosting to searches with 3 or more terms. With
2 terms, it's possible there are a ton of phrase matches, which can
drown out high quality 'AND' matches. However, we've seen a few dogfood
examples with fewer terms where boosting would have been really useful.
This PR removes the 3 term restriction on boosting. To mitigate noise,
it updates the phrase matching to only match on word boundaries. It also
switches to matching on the original query string instead of
reconstructing the phrase from terms. That lets us match even when
there's special characters, like `return "disabled"`.
Relates to SPLF-168
## Test plan
Adapted unit tests. Lots of manual testing:
* `invalid ModelID`
* `return "disabled"`
* `func TestClient_do(`
* `// The vararg opts parameter can include functions to configure the`
* `test server` -- this still works quite well, it's a bit fragile but
restricting the matches to word boundaries helped reduce noise
* `bytes buffer`
A large sub-query was duplicated between two queries, making
the code harder to understand. This patch factors that out.
Ancillary changes:
- Added more doc comments
- Tweak tests to make failures easier to follow.
1. Make `sg cloud eph` instance status Reason simple string, as CloudAPI
will take over and return Reason with job URL - no need to parse Reason
anymore.
```sh
go run ./dev/sg cloud eph status --name ff-eph56
⚠️ Running sg with a dev build, following flags have different default value unless explictly set: skip-auto-update, disable-analytics
✅ Ephemeral instance "ff-eph56" status retrieved
Ephemeral instance details:
Name Expires In Instance status Details
ff-eph56 4h37m4.79031s unspecified creation task is already running: https://console.cloud.google.com/workflows/workflow/us-central1/create-instance-c31bd1a4ea84/executions?project=michael-test-03
```
2. Allow re-run create if CloudAPI returns status with Reason - it means
instance is not fully created yet, so user might re-try create -
CloudAPI will ensure more than one create is not running at the same
time.
3. Updated printers with GOTO action for each instance details:
```sh
go run ./dev/sg cloud eph list --all
⚠️ Running sg with a dev build, following flags have different default value unless explictly set: skip-auto-update, disable-analytics
☁️ Fetched 10 instances
Name Expires In Instance status Details
andre-eph-1 30h10m42.989163s unspecified invoke: sg cloud eph status --name andre-eph-1
ff-eph56 4h34m43.989154s unspecified invoke: sg cloud eph status --name ff-eph56
```
## Test plan
Unit tests simplified.
E2e against old and new CloudAPI.
---------
Co-authored-by: William Bezuidenhout <william.bezuidenhout@sourcegraph.com>
Fixes GRAPH-782
Specifically this allows us avoid updating syntactic_indexing_enabled
when we don't have the value available.
This is done to solve the problem where not providing that field crashes
the resolver. I confirmed it by both via tests and via manual testing of
the API
## Test plan
- Added a new test for resolvers specifically to test the extra logic
around this field
Frontend assets are not embedded into the binary anymore instead they're
added to the final container image at `/assets-dist`.
In this PR we check the directory inside the image for the marker
instead of the binary
Closes DINF-176
## Test plan
Tested locally
```
sg release run test --version 1.2.3
👉 [ setup] Finding release manifest in "."
[ setup] No explicit branch name was provided, assuming current branch is the target: main
[ setup] Found manifest for "sourcegraph" (github.com/sourcegraph/sourcegraph)
[ meta] Owners: @sourcegraph/release
[ meta] Repository: github.com/sourcegraph/sourcegraph
👉 [ vars] Variables
[ vars] tag="1.2.3"
[ vars] config="{\"version\":\"v1.2.3\",\"inputs\":\"\",\"type\":\"patch\"}"
[ vars] git.branch="main"
[ vars] is_development="false"
[ vars] version="v1.2.3"
👉 [ reqs] Checking requirements...
[ reqs] ✅ jq
[ reqs] 🔕 Buidkite access token (excluded for test)
[ reqs] 🔕 GitHub Token to submit changelogs (excluded for test)
👉 [ test] Running testing steps for v1.2.3
👉 [ step] Running step "check:frontend and server image contain bundle"
[check:frontend and server image contain bundle] pulling frontend image us.gcr.io/sourcegraph-dev/frontend:insiders
[check:frontend and server image contain bundle] insiders: Pulling from sourcegraph-dev/frontend
[check:frontend and server image contain bundle] Digest: sha256:1256bfb7c64bee0f11a3d0b82af6899f1d3fe22c0d6f3875a877c5f8f8b0e963
[check:frontend and server image contain bundle] Status: Image is up to date for us.gcr.io/sourcegraph-dev/frontend:insiders
[check:frontend and server image contain bundle] us.gcr.io/sourcegraph-dev/frontend:insiders
[check:frontend and server image contain bundle] checking frontend has web-bundle at /assets-dist inside the container
[check:frontend and server image contain bundle] WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
[check:frontend and server image contain bundle] pulling server image us.gcr.io/sourcegraph-dev/server:insiders
[check:frontend and server image contain bundle] insiders: Pulling from sourcegraph-dev/server
[check:frontend and server image contain bundle] Digest: sha256:592c4e94ced4990a3b461eb474d5e7fee9c408d93ba4df44220b22f7d39ea645
[check:frontend and server image contain bundle] Status: Image is up to date for us.gcr.io/sourcegraph-dev/server:insiders
[check:frontend and server image contain bundle] us.gcr.io/sourcegraph-dev/server:insiders
[check:frontend and server image contain bundle] checking server has web-bundle at /assets-dist inside the container
[check:frontend and server image contain bundle] WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
[ step] Step "check:frontend and server image contain bundle" succeeded
```
<!-- REQUIRED; info at
https://docs-legacy.sourcegraph.com/dev/background-information/testing_principles
-->
## Changelog
* release: check contiainer directory `/assets-dist` for marker instead
of frontend binary
<!-- OPTIONAL; info at
https://www.notion.so/sourcegraph/Writing-a-changelog-entry-dd997f411d524caabf0d8d38a24a878c
-->
Some commands like the
[`batcheshelper-builder`](https://sourcegraph.sourcegraph.com/github.com/sourcegraph/sourcegraph/-/blob/sg.config.yaml?L821)
aren't long running commands.
This command is used to build and load an image into docker. The `cmd`
section returns an `exit 0`. This behavior combined with
`continueWatchOnExit` results in an infinite loop where the process is
continually restarted because `sg` doesn't know that the process has
finished executing and isn't a long-running process.
https://github.com/user-attachments/assets/e7a027a1-6f93-403f-9240-6a791255fba9
An example of the behavior is shown below as running `sg start batches`
results in the `batcheshelper-builder` command continually restarted.
The fix is quite simple, we return an empty receiver channel when the
process is done executing so that `sg` knows it's done and doesn't
restart the command unless there's a change.
## Test plan
* Manual testing with `go run ./dev/sg start batches` doesn't result in
an infinite loop anymore.
* Add unit tests
## Changelog
During an investigation, we saw that Rockskip was not using scip-ctags
for symbol parsing when applicable. This means that
1. Rockskip is getting less than optimal symbols for certain languages
(like Go)
2. Rockskip is getting no symbols for languages not in universal ctags
(Magik)
This PR attempts to solve this problem but updating Rockskip to re-use
the ctags parser pool logic from symbol service.
### Key Changes
- Update parser pool to be re-usable
- Push common logic for parser type detection into the parser pool
module
- Update rockskip service config to take a parser pool
- Update and add unit/integration tests
## Questions
- What performance impact will using this pooled parser have compared to
its previous behavior of spawning a new ctags process each time?
## Test plan
- [x] Add unit tests
- [x] Update integration tests
- [x] Manually test rockskip
- [x] Manually test symbolservice (in case of regression)
---------
Co-authored-by: Keegan Carruthers-Smith <keegan.csmith@gmail.com>
When exactly 1 auth provider is configured, Sourcegraph redirects users
automatically to the IdP to speed up the sign-in process, so that users
don't have to make an extra click to select the one and only sign-in
provider.
SOAP is a special case though because it is hidden by default, but
enabled on all cloud instances. That caused this auto redirect to never
fire for Cloud, since there are technically two auth providers.
This PR fixes it by checking for the sourcegraph-operator query
parameter which tells the UI to show the magic SOAP auth provider in the
list.
Closes SRC-500
Test plan: Tested on a cloud instance that indeed there is no auto
redirect. Then tested locally with SOAP configured that auto redirects
happen after this PR, and that there is no auto redirect when the
?sourcegraph-operator query parameter is set.
## Changelog
When only a single auth provider is configured, users are again
redirected correctly to the identity provider on Sourcegraph Cloud.
