This PR fixes an important bug in #62976, where we didn't properly
map the
symbol line match to the return type. Instead, we accidentally treated
symbol
matches like file matches and returned the start of the file.
## Test plan
Add new unit test for symbol match conversion. Extensive manual testing.
<br> Backport 004eb0fd83 from #63773
Co-authored-by: Julie Tibshirani <julietibs@apache.org>
The OTEL upgrade https://github.com/sourcegraph/sourcegraph/pull/63171
bumps the `prometheus/common` package too far via transitive deps,
causing us to generate configuration for alertmanager that altertmanager
doesn't accept, at least until the alertmanager project cuts a new
release with a newer version of `promethues/common`.
For now we forcibly downgrade with a replace. Everything still builds,
so we should be good to go.
## Test plan
`sg start` and `sg run prometheus`. On `main`, editing
`observability.alerts` will cause Alertmanager to refuse to accept the
generated configuration. With this patch, all is well it seems - config
changes go through as expected. This is a similar test plan for
https://github.com/sourcegraph/sourcegraph/pull/63329
## Changelog
- Fix Prometheus Alertmanager configuration failing to apply
`observability.alerts` from site config <br> Backport
ffa873f3ad from #63790
Co-authored-by: Robert Lin <robert@bobheadxi.dev>
This will correct6 upgrade path for mvu plan creation
## Test plan
CI test
## Changelog
<br> Backport cb19d6f0a9 from #63764
Co-authored-by: Warren Gifford <warren@sourcegraph.com>
Missing bit for the minor release version bump
## Test plan
CI
<br> Backport 087ad83995 from #63767
Co-authored-by: Jean-Hadrien Chabran <jean-hadrien.chabran@sourcegraph.com>
We created a decoder that was never used, but the package is otherwise
unused. It recently had a CVE, so this just removes it so it's no longer
part of our security surface area.
Make SetupEnvtest slightly lower-level by asking callers to construct
their own client from the returned k8s REST config. This is because
there are 2 kinds of official kubernetes client in Go - a
kubernetes.Clientset and a client.Client. The latter is
more-traditionally used in operators, because it's what a
ControllerManager.GetClient() returns, and indeed this is what our
reconciler uses.
We ended up using a kubernetes.Clientset in the envtest-using golden
tests for the reconciler, because its mechanics for listing resources
were simpler. Now, I want to reuse SetupEnvtest somewhere that needs a
client.Client. We could undertake work to use only one flavor of
kubernetes client, but this commit seems like a decent low-cost first
step.
Docker images executor, executor-kubernetes, bundled-executor has
reported high/critical CVE-2024-24790 , CVE-2023-45288 reported on
golang stdlib. Upon testing, src version 5.3.0 was using `1.20.x` as per
e8e79e0311
This pull request attempts to upgrade src version to 5.4.0
## Test plan
- CI 🟢
- src version should report 5.4.0 (I built the image locally and tested
it)
`docker run --platform linux/amd64 -it --entrypoint /bin/sh
executor:candidate`
## Changelog
<!-- OPTIONAL; info at
https://www.notion.so/sourcegraph/Writing-a-changelog-entry-dd997f411d524caabf0d8d38a24a878c
-->
Upgrade src-cli version to 5.4.0 to address CVE-2024-24790 ,
CVE-2023-45288
Currently if a cloud ephemeral build is trigger it is triggered on the
`main` sourcegraph pipeline. Once a build a triggered and a commit is
subsequently pushed the previous build is cancelled - which means the
Cloud Ephemeral build is cancelled leading to a failed deployment.
In this PR, we instead trigger a build on the Cloud Ephemeral pipeline.
Which is the _exact_ pipeline as `sourcegraph` main but:
- sets the pipeline env to always have `CLOUD_EPHEMERAL=true`
- does not cancel previous builds
## Test plan
https://buildkite.com/sourcegraph/cloud-ephemeral/builds/1
## Changelog
* `sg cloud eph` will now trigger builds on the `cloud-ephemeral`
pipeline
Automatically generated PR to update package lockfiles for Sourcegraph
base images.
Built from Buildkite run
[#281769](https://buildkite.com/sourcegraph/sourcegraph/builds/281769).
## Test Plan
- CI build verifies image functionality
Co-authored-by: Buildkite <buildkite@sourcegraph.com>
Since we removed on-demand cloning, the scheduler is now expected to always contain all repositories. Thus, we no longer need to constrain the set of uncloned repos to a ginormous ID list.
Test plan:
CI still passes.
This syncer doesn't depend on anything in repo updater, so we're moving it to worker instead, where it can selectively be disabled and is properly monitored.
Test plan:
CI passes, code review.
We have been using v2 data since >5 years now, this should be safe to
remove.
As a side-effect, we have one less background task running in frontend,
which means it ran N times in horizontally scaled environments, which
isn't exactly useful.
Test plan:
Code review.
Currently, nothing really tells that Cody Gateway needs redis, the env
var for finding the address is hidden somewhere deep in the redispool
package.
In practice, we only use one redis instance, but at some point we
started using both redispool.Cache and redispool.Store, which means we
maintain two connection pools, leading to more than expected
connections.
Test plan:
Code review and CI.
This PR restructures the packages to move all symbols-only code into the
symbols service. This helps to reason better about which service is
accessing what datastores.
Test plan:
Just moved code, compiler and CI are happy.
Recently, this was refactored to also allow using the redispool.Store.
However, that makes it very implicit to know where something is being
written, so instead we pass down the pool instance at instantiation.
This also gives a slightly better overview of where redispool is
actually required.
Test plan: CI passes.
This PR has a single goal: pass a new `ModelConfigInfo` type, which has
the `Provider` and `Model` we should use to serve a completions request,
down into `client.Get()`
This PR explicitly only handles the case we care about for Self-hosted
models, and all pieces of logic that I expect will be
replaced/superseded by your work @chrsmith are annotated with `//
TODO(slimsag): self-hosted-models:` comments so we can easily find and
remove them when your work is ready.
Every location I have modified has been carefully wrapped in an if
statement like `if conf.Get().SiteConfig().ModelConfiguration != nil` to
ensure that this change _only_ affects people who set the new
`"modelConfiguration"` site config property.
## Test plan
1. Configured `"modelConfiguration"` and removed `"completions"` in my
dev instances' site config.
2. Used VS Code chat to confirm I see this codepath is connected
end-to-end and I get a `TODO` error back:
<img width="679" alt="image"
src="https://github.com/sourcegraph/sourcegraph/assets/3173176/d3c07ffe-7ec2-4ad9-9251-2272ce36d44b">
## Changelog
N/A
---------
Signed-off-by: Stephen Gutekanst <stephen@sourcegraph.com>
Closes srch-103
Currently we don't show the global context filter on the search home
page or the search results page (global context is the default context).
This commit does two things:
- It prefills the search input on the search homepage with the user's
default context (like the React app)
- It the logic that pre-processed the search query and removed the
context filter if it was global.
In other words we simplify the query logic by showing/submitting the
search query as is. Notably this doesn't affect how the search input
works on repo pages.
## Test plan
- Opening the search home page pre-fills the query input with the
default search context
- Submitting a query without a `context:` filter does not add a
`context:` filter to the URL or the search input
- If a query contains `context:global` that filter is preserved in the
query input (it wasn't before)
We don't like the style/alignment of the Lucide case sensitivity and
regex icons (the structural search icon seems fine). This commit
switches them back to the MDI icons that we used before.
Solution or temporary workaround (tbd) for srch-544
## Test plan
Manual testing.
Implements upsert for all the subscriptions fields in the DB client. As
part of this I generalized the logic for building upsert DB interactions
into a new `upsert` package, because this pattern is a common one we'll
need to implement to maintain various AIP-update-compliant endpoints,
which specifies various upsert behaviours: https://google.aip.dev/134
Part of CORE-216
Part of CORE-156
## Test plan
Integration tests against DB
This PR bundles several small refactoring to make it easier to review
the pending "refactor completion APIs to read from modelconfig".
Going commit-by-commit:
[Export the ValidateModelRef
function](3c949a88d0)
This just exports a validation function that was previously private from
the `internal/modelconfig` package. We'll be using this to sanity check
the `ModelRef` that is returned from the "getModelsFn".
[Add modelconfig.{InitMock,
ResetMock}](f807f40bf5)
In order to update the `frontend/internal/httpapi/completions` unit
tests, we need to have changes to the site configuration data _ALSO_
update the global `modelconfig.Service`. Typically this would happen by
a "config watcher" that gets registered, but we don't do that as part of
unit tests.
So instead, I just added some clunky `InitMock() error` and `ResetMock()
error` functions to support unit testing. If you have a better idea for
how to expose this behavior, I'm all ears.
[Move fireworks-specific API types into
types.go](bb02ccba45)
Moves the API data types for the `fireworks` client into their own file,
just to keep things tidy.
[Refactor completion
clients](02a5e3bfc9)
This is a bit involved, but from a high-level, we are just passing the
`types.CompletionRequest` object "lower" into the call stack. Rather
than only return the `.Parameters` field
(`types.CompletionRequestParameters`).
This is necessary, because I plan on adding another field to
`types.CompletionRequest` (to include the `modelconfig.Provider` and
`modelconfig.Model` data). So this change just does some of the work for
that now, resulting in a smaller and easier to review diff later.
Also, in some cases I moved the calls to
`tokenManager.UpdateTokenCountsFromModelUsage` into a dedicated function
to simplify the callsites.
## Test plan
Existing unit tests
## Changelog
NA
This adds a line index utility. Frequently, I want to be able to
efficiently index a file to extract a specific line or range of lines,
but it's surprisingly tricky to get exactly right given weird
definitions of "what even is a line" and edge conditions around
out-of-bounds and such.
So this adds a general-purpose utility to pre-calculate the locations of
lines in the file, making extracting a line range a zero-allocation,
`O(1)` operation.
Not implemented: the same index can also be used to find the line that
contains an offset, which I've also needed to do before. But I'll save
that for when I actually have an immediate use for it.
Closes srch-612
This commit adds a link next to the blame commit message that allows
reblaming to a prior commit if available.
It extends the existing blame extension. This won't have any affect on
the React app because it doesn't pass the configuration option needed to
add the extra gutter.
Some notes:
- I originally used the tooltip component instead of `title` but somehow
it starts to break when scrolling the document (tooltips don't show up
anymore). I don't know if CodeMirror does anything to the DOM elements
that causes this to fail.
- The reblame URL also selected the corresponding line so that the
correct line is scrolled into view.
## Test plan
Manual testing
This PR does a bit of cleanup / alignment with other services, most
notably:
- Loads config via the config method of the shared routine package plus
tests for default config values
- Listens on HTTP using the existing httpserver package
- Makes it clear what gitserver methods are used by calling them at the
entry function instead of passing the whole client down
- Adds exhaustive request logging for better auditability (and found an
unused field which I removed)
- Rename log to logger
- Add the common stack of HTTP middlewares for actor, client etc
propagation
Test plan:
Unit tests still pass. CI is green and I was able to run unindexed
searches locally.
This causes the `modelconfig` package to actually begin converting the
new `"modelConfiguration"` site config which was introduced in #63654
into the internal data types we use for model configuration.
This is all pretty straightforward / lame type conversion code, but
carefully written to preserve all the semantics we care about.
## Test plan
Written very carefully, and confirmed that this query:
```
curl -H "Authorization: token $(cat ~/local-token)" https://sourcegraph.test:3443/.api/modelconfig/supported-models.json
```
returns what I would expect with this site configuration:
<details>
<summary>site config</summary>
```
// Setting this field means we are opting into the new Cody model configuration system which is in beta.
"modelConfiguration": {
// Disable use of Sourcegraph's servers for model discovery
"sourcegraph": null,
// Configure the OpenAI-compatible API endpoints that Cody should use to provide
// mistral and bigcode (starcoder) models.
"providerOverrides": [
{
"displayName": "Mistral",
"id": "mistral",
"serverSideConfig": {
"type": "openaicompatible",
"endpoint": "...",
"accessToken": "...",
},
},
{
"displayName": "Bigcode",
"id": "bigcode",
"serverSideConfig": {
"type": "openaicompatible",
"endpoint": "...",
"accessToken": "...",
},
},
],
// Configure which exact mistral and starcoder models we want available
"modelOverridesRecommendedSettings": [
"bigcode::v1::starcoder2-7b",
"mistral::v1::mixtral-8x7b-instruct"
],
// Configure which models Cody will use by default
"defaultModels": {
"chat": "mistral::v1::mixtral-8x7b-instruct",
"fastChat": "mistral::v1::mixtral-8x7b-instruct",
"codeCompletion": "bigcode::v1::starcoder2-7b",
}
}
```
</details>
We could certainly use more unit tests here to prevent potential future
regressions, I can add those in a future PR.
## Changelog
Has no effect on users unless they opt into the early-access
`"modelConfiguration"` site config feature.
---------
Signed-off-by: Stephen Gutekanst <stephen@sourcegraph.com>
Co-authored-by: Chris Smith <chrsmith@users.noreply.github.com>
https://linear.app/sourcegraph/issue/SPLF-116/perforce-searching-by-perforce-changelist-id
## Details
We have had requests from our customers using Perforce to be able to
search inside of a changelist id. For a commit sha we support doing this
```
context:global repo:^perforce-sgdev-org/rhia-depot-test$@345c17c` some text
```
But for perforce they want to do the same thing but with the change list
ids. Which would look like this
```
context:global repo:^perforce-sgdev-org/rhia-depot-test$@changelist/83732` some text
```
To support this, I am attempting to smartly detect when we should do a
DB round trip and look up the proper mapping. I built a simple heuristic
that is
1. Is perforce changelist mapping feature flag enabled
2. Is this a perforce repo?
3. Is the revision request a integer ?
This mapping is just a best effort, if it fails it just falls back on
current behavior.
We are doing with a syntax of `@changelist/CL_ID` instead of supporting
`@CL_ID` to future proof us. This lookup focuses on finding the mapping
in the DB but in the future we may want to pre-create these refs in the
db duing mapping of perforce CLs to git commits.
## Limitations
This works well in testing however, the repo name@changelist/rev we
return contains the sha
I investigated changing this but it would required a larger change in
resolving the stream results. While that would be nice to have, I
decided to keep this minimal for now and add that later if needed
## Test plan
<!-- REQUIRED; info at
https://docs-legacy.sourcegraph.com/dev/background-information/testing_principles
-->
## Changelog
- For perforce depots, support searching within a specific changelist by
specifying a ref like `context:global repo:^repo/name$@changelist/83854`
Closes https://linear.app/sourcegraph/issue/CORE-213
The existing overwrite stripping `window.context.authProviders` sourced
from a proxied API down to only built-in providers makes it impossible
to reliably run `sg start web-standalone` against Sourcegraph.com (and
likely other Sourcegraph instances). The docstring says:
> Only username/password auth-provider provider is supported with the
standalone server.
However, you _can_ log in with other OAuth2 providers locally, so it
doesn't make sense to remove them from the context. This PR removes the
overwrite so that we can log in to Sourcegraph.com with a locally
running web app, and talk to Cody and all that good stuff.
## How to log in to OAuth providers with `sg start web-standalone`
IIRC what happens:
1. Click "Sign in" -> go to OAuth2 provider -> log in
2. OAuth2 provider issues callback to the Sourcegraph instance, telling
it that you logged in
3. OAuth2 provider redirects you to the instance to confirm your session
- for SAMS in dotcom:
```
https://sourcegraph.com/.auth/callback?code=sams_ac_...
```
This redirect will surface an error, like so:
> Authentication failed. Try signing in again (and clearing cookies for
the current site). The error was: state parameter did not match the
expected value (possible request forgery).
This happens because you get redirected to an absolute URL, which will
be `sourcegraph.com` (the upstream Sourcegraph instance), which won't
match the cookie you get issued (`sourcegraph.test`). We just need to
make sure it matches and send the code to the callback through our local
proxy:
```diff
- https://sourcegraph.com/.auth/callback?code=sams_ac_...
+ https://sourcegraph.test:3443/.auth/callback?code=sams_ac_...
```
And everything will work :) I confirmed this works for e.g. logging into
S2 with GitHub as well, using the callback issued by GitHub.
We cannot make this the default behaviour, because the redirect URL is
usually configured in the OAuth provider. An example GitHub OAuth
application configuration:
Note their guidance on valid values:
https://docs.github.com/en/apps/oauth-apps/building-oauth-apps/authorizing-oauth-apps#redirect-urls
- we must provide a single, absolute URL
This is very similar to the process used to log in to private Cloud
instances, e.g.
https://cloud-ops.sgdev.org/dashboard/environments/prod/instances/src-bd02273f6b90d1d1beee#log-in-to-the-instance-ui
(scroll to bottom of this section), which indicates we've thought about
similar cases before and found that this was the best way
## Test plan
https://www.loom.com/share/6cb3b3ca475b4b9392aa4b11938e76e6?sid=d99fcb59-4308-45c7-9f68-af9ac44c4e7e
<!-- PR description tips:
https://www.notion.so/sourcegraph/Write-a-good-pull-request-description-610a7fd3e613496eb76f450db5a49b6e
-->
Patches CVE-2024-24790 by upgrading to 27-0-3 tag. However, the patched
version has CVE-2024-24791 😟 and it doesnt have patch.
## Test plan
<!-- REQUIRED; info at
https://docs-legacy.sourcegraph.com/dev/background-information/testing_principles
-->
Build and test image locally.
### Instruction to build and test locally
- Go to `dev/oci_deps.bzl`
- Find the current tag example `docker:26.1.3-dind`
- Go to docker registry and search for updated tag and grab one example:
`docker:27.0.3-dind`
- docker pull --platform linux/amd64 docker:27.0.3-dind
- Add `platforms = ["linux/amd64"],` to the oci_pull for building and
testing locally
```bzl
oci_pull(
name = "upstream_dind_base",
digest = "sha256:2632da0d24924b179adf1c2e6f4ea6fb866747e84baea6b2ffaa8bff982ce102",
platforms = ["linux/amd64"],
)
```
- Run `sg images build dind`
- For testing, run `docker run --rm -it --entrypoint /bin/sh -v
/var/run/docker.sock:/var/run/docker.sock dind:candidate`
- Test docker commands and pull and run image for testing
## Changelog
- Upgraded dind to 27.0.3 to patch CVE-2024-24790 vulnerability
<!-- OPTIONAL; info at
https://www.notion.so/sourcegraph/Writing-a-changelog-entry-dd997f411d524caabf0d8d38a24a878c
-->
