Docker images executor, executor-kubernetes, bundled-executor has
reported high/critical CVE-2024-24790 , CVE-2023-45288 reported on
golang stdlib. Upon testing, src version 5.3.0 was using `1.20.x` as per
e8e79e0311
This pull request attempts to upgrade src version to 5.4.0
## Test plan
- CI 🟢
- src version should report 5.4.0 (I built the image locally and tested
it)
`docker run --platform linux/amd64 -it --entrypoint /bin/sh
executor:candidate`
## Changelog
<!-- OPTIONAL; info at
https://www.notion.so/sourcegraph/Writing-a-changelog-entry-dd997f411d524caabf0d8d38a24a878c
-->
Upgrade src-cli version to 5.4.0 to address CVE-2024-24790 ,
CVE-2023-45288
Currently if a cloud ephemeral build is trigger it is triggered on the
`main` sourcegraph pipeline. Once a build a triggered and a commit is
subsequently pushed the previous build is cancelled - which means the
Cloud Ephemeral build is cancelled leading to a failed deployment.
In this PR, we instead trigger a build on the Cloud Ephemeral pipeline.
Which is the _exact_ pipeline as `sourcegraph` main but:
- sets the pipeline env to always have `CLOUD_EPHEMERAL=true`
- does not cancel previous builds
## Test plan
https://buildkite.com/sourcegraph/cloud-ephemeral/builds/1
## Changelog
* `sg cloud eph` will now trigger builds on the `cloud-ephemeral`
pipeline
Automatically generated PR to update package lockfiles for Sourcegraph
base images.
Built from Buildkite run
[#281769](https://buildkite.com/sourcegraph/sourcegraph/builds/281769).
## Test Plan
- CI build verifies image functionality
Co-authored-by: Buildkite <buildkite@sourcegraph.com>
Since we removed on-demand cloning, the scheduler is now expected to always contain all repositories. Thus, we no longer need to constrain the set of uncloned repos to a ginormous ID list.
Test plan:
CI still passes.
This syncer doesn't depend on anything in repo updater, so we're moving it to worker instead, where it can selectively be disabled and is properly monitored.
Test plan:
CI passes, code review.
We have been using v2 data since >5 years now, this should be safe to
remove.
As a side-effect, we have one less background task running in frontend,
which means it ran N times in horizontally scaled environments, which
isn't exactly useful.
Test plan:
Code review.
Currently, nothing really tells that Cody Gateway needs redis, the env
var for finding the address is hidden somewhere deep in the redispool
package.
In practice, we only use one redis instance, but at some point we
started using both redispool.Cache and redispool.Store, which means we
maintain two connection pools, leading to more than expected
connections.
Test plan:
Code review and CI.
This PR restructures the packages to move all symbols-only code into the
symbols service. This helps to reason better about which service is
accessing what datastores.
Test plan:
Just moved code, compiler and CI are happy.
Recently, this was refactored to also allow using the redispool.Store.
However, that makes it very implicit to know where something is being
written, so instead we pass down the pool instance at instantiation.
This also gives a slightly better overview of where redispool is
actually required.
Test plan: CI passes.
This PR has a single goal: pass a new `ModelConfigInfo` type, which has
the `Provider` and `Model` we should use to serve a completions request,
down into `client.Get()`
This PR explicitly only handles the case we care about for Self-hosted
models, and all pieces of logic that I expect will be
replaced/superseded by your work @chrsmith are annotated with `//
TODO(slimsag): self-hosted-models:` comments so we can easily find and
remove them when your work is ready.
Every location I have modified has been carefully wrapped in an if
statement like `if conf.Get().SiteConfig().ModelConfiguration != nil` to
ensure that this change _only_ affects people who set the new
`"modelConfiguration"` site config property.
## Test plan
1. Configured `"modelConfiguration"` and removed `"completions"` in my
dev instances' site config.
2. Used VS Code chat to confirm I see this codepath is connected
end-to-end and I get a `TODO` error back:
<img width="679" alt="image"
src="https://github.com/sourcegraph/sourcegraph/assets/3173176/d3c07ffe-7ec2-4ad9-9251-2272ce36d44b">
## Changelog
N/A
---------
Signed-off-by: Stephen Gutekanst <stephen@sourcegraph.com>
Closes srch-103
Currently we don't show the global context filter on the search home
page or the search results page (global context is the default context).
This commit does two things:
- It prefills the search input on the search homepage with the user's
default context (like the React app)
- It the logic that pre-processed the search query and removed the
context filter if it was global.
In other words we simplify the query logic by showing/submitting the
search query as is. Notably this doesn't affect how the search input
works on repo pages.
## Test plan
- Opening the search home page pre-fills the query input with the
default search context
- Submitting a query without a `context:` filter does not add a
`context:` filter to the URL or the search input
- If a query contains `context:global` that filter is preserved in the
query input (it wasn't before)
We don't like the style/alignment of the Lucide case sensitivity and
regex icons (the structural search icon seems fine). This commit
switches them back to the MDI icons that we used before.
Solution or temporary workaround (tbd) for srch-544
## Test plan
Manual testing.
Implements upsert for all the subscriptions fields in the DB client. As
part of this I generalized the logic for building upsert DB interactions
into a new `upsert` package, because this pattern is a common one we'll
need to implement to maintain various AIP-update-compliant endpoints,
which specifies various upsert behaviours: https://google.aip.dev/134
Part of CORE-216
Part of CORE-156
## Test plan
Integration tests against DB
This PR bundles several small refactoring to make it easier to review
the pending "refactor completion APIs to read from modelconfig".
Going commit-by-commit:
[Export the ValidateModelRef
function](3c949a88d0)
This just exports a validation function that was previously private from
the `internal/modelconfig` package. We'll be using this to sanity check
the `ModelRef` that is returned from the "getModelsFn".
[Add modelconfig.{InitMock,
ResetMock}](f807f40bf5)
In order to update the `frontend/internal/httpapi/completions` unit
tests, we need to have changes to the site configuration data _ALSO_
update the global `modelconfig.Service`. Typically this would happen by
a "config watcher" that gets registered, but we don't do that as part of
unit tests.
So instead, I just added some clunky `InitMock() error` and `ResetMock()
error` functions to support unit testing. If you have a better idea for
how to expose this behavior, I'm all ears.
[Move fireworks-specific API types into
types.go](bb02ccba45)
Moves the API data types for the `fireworks` client into their own file,
just to keep things tidy.
[Refactor completion
clients](02a5e3bfc9)
This is a bit involved, but from a high-level, we are just passing the
`types.CompletionRequest` object "lower" into the call stack. Rather
than only return the `.Parameters` field
(`types.CompletionRequestParameters`).
This is necessary, because I plan on adding another field to
`types.CompletionRequest` (to include the `modelconfig.Provider` and
`modelconfig.Model` data). So this change just does some of the work for
that now, resulting in a smaller and easier to review diff later.
Also, in some cases I moved the calls to
`tokenManager.UpdateTokenCountsFromModelUsage` into a dedicated function
to simplify the callsites.
## Test plan
Existing unit tests
## Changelog
NA
This adds a line index utility. Frequently, I want to be able to
efficiently index a file to extract a specific line or range of lines,
but it's surprisingly tricky to get exactly right given weird
definitions of "what even is a line" and edge conditions around
out-of-bounds and such.
So this adds a general-purpose utility to pre-calculate the locations of
lines in the file, making extracting a line range a zero-allocation,
`O(1)` operation.
Not implemented: the same index can also be used to find the line that
contains an offset, which I've also needed to do before. But I'll save
that for when I actually have an immediate use for it.
Closes srch-612
This commit adds a link next to the blame commit message that allows
reblaming to a prior commit if available.
It extends the existing blame extension. This won't have any affect on
the React app because it doesn't pass the configuration option needed to
add the extra gutter.
Some notes:
- I originally used the tooltip component instead of `title` but somehow
it starts to break when scrolling the document (tooltips don't show up
anymore). I don't know if CodeMirror does anything to the DOM elements
that causes this to fail.
- The reblame URL also selected the corresponding line so that the
correct line is scrolled into view.
## Test plan
Manual testing
This PR does a bit of cleanup / alignment with other services, most
notably:
- Loads config via the config method of the shared routine package plus
tests for default config values
- Listens on HTTP using the existing httpserver package
- Makes it clear what gitserver methods are used by calling them at the
entry function instead of passing the whole client down
- Adds exhaustive request logging for better auditability (and found an
unused field which I removed)
- Rename log to logger
- Add the common stack of HTTP middlewares for actor, client etc
propagation
Test plan:
Unit tests still pass. CI is green and I was able to run unindexed
searches locally.
This causes the `modelconfig` package to actually begin converting the
new `"modelConfiguration"` site config which was introduced in #63654
into the internal data types we use for model configuration.
This is all pretty straightforward / lame type conversion code, but
carefully written to preserve all the semantics we care about.
## Test plan
Written very carefully, and confirmed that this query:
```
curl -H "Authorization: token $(cat ~/local-token)" https://sourcegraph.test:3443/.api/modelconfig/supported-models.json
```
returns what I would expect with this site configuration:
<details>
<summary>site config</summary>
```
// Setting this field means we are opting into the new Cody model configuration system which is in beta.
"modelConfiguration": {
// Disable use of Sourcegraph's servers for model discovery
"sourcegraph": null,
// Configure the OpenAI-compatible API endpoints that Cody should use to provide
// mistral and bigcode (starcoder) models.
"providerOverrides": [
{
"displayName": "Mistral",
"id": "mistral",
"serverSideConfig": {
"type": "openaicompatible",
"endpoint": "...",
"accessToken": "...",
},
},
{
"displayName": "Bigcode",
"id": "bigcode",
"serverSideConfig": {
"type": "openaicompatible",
"endpoint": "...",
"accessToken": "...",
},
},
],
// Configure which exact mistral and starcoder models we want available
"modelOverridesRecommendedSettings": [
"bigcode::v1::starcoder2-7b",
"mistral::v1::mixtral-8x7b-instruct"
],
// Configure which models Cody will use by default
"defaultModels": {
"chat": "mistral::v1::mixtral-8x7b-instruct",
"fastChat": "mistral::v1::mixtral-8x7b-instruct",
"codeCompletion": "bigcode::v1::starcoder2-7b",
}
}
```
</details>
We could certainly use more unit tests here to prevent potential future
regressions, I can add those in a future PR.
## Changelog
Has no effect on users unless they opt into the early-access
`"modelConfiguration"` site config feature.
---------
Signed-off-by: Stephen Gutekanst <stephen@sourcegraph.com>
Co-authored-by: Chris Smith <chrsmith@users.noreply.github.com>
https://linear.app/sourcegraph/issue/SPLF-116/perforce-searching-by-perforce-changelist-id
## Details
We have had requests from our customers using Perforce to be able to
search inside of a changelist id. For a commit sha we support doing this
```
context:global repo:^perforce-sgdev-org/rhia-depot-test$@345c17c` some text
```
But for perforce they want to do the same thing but with the change list
ids. Which would look like this
```
context:global repo:^perforce-sgdev-org/rhia-depot-test$@changelist/83732` some text
```
To support this, I am attempting to smartly detect when we should do a
DB round trip and look up the proper mapping. I built a simple heuristic
that is
1. Is perforce changelist mapping feature flag enabled
2. Is this a perforce repo?
3. Is the revision request a integer ?
This mapping is just a best effort, if it fails it just falls back on
current behavior.
We are doing with a syntax of `@changelist/CL_ID` instead of supporting
`@CL_ID` to future proof us. This lookup focuses on finding the mapping
in the DB but in the future we may want to pre-create these refs in the
db duing mapping of perforce CLs to git commits.
## Limitations
This works well in testing however, the repo name@changelist/rev we
return contains the sha
I investigated changing this but it would required a larger change in
resolving the stream results. While that would be nice to have, I
decided to keep this minimal for now and add that later if needed
## Test plan
<!-- REQUIRED; info at
https://docs-legacy.sourcegraph.com/dev/background-information/testing_principles
-->
## Changelog
- For perforce depots, support searching within a specific changelist by
specifying a ref like `context:global repo:^repo/name$@changelist/83854`
Closes https://linear.app/sourcegraph/issue/CORE-213
The existing overwrite stripping `window.context.authProviders` sourced
from a proxied API down to only built-in providers makes it impossible
to reliably run `sg start web-standalone` against Sourcegraph.com (and
likely other Sourcegraph instances). The docstring says:
> Only username/password auth-provider provider is supported with the
standalone server.
However, you _can_ log in with other OAuth2 providers locally, so it
doesn't make sense to remove them from the context. This PR removes the
overwrite so that we can log in to Sourcegraph.com with a locally
running web app, and talk to Cody and all that good stuff.
## How to log in to OAuth providers with `sg start web-standalone`
IIRC what happens:
1. Click "Sign in" -> go to OAuth2 provider -> log in
2. OAuth2 provider issues callback to the Sourcegraph instance, telling
it that you logged in
3. OAuth2 provider redirects you to the instance to confirm your session
- for SAMS in dotcom:
```
https://sourcegraph.com/.auth/callback?code=sams_ac_...
```
This redirect will surface an error, like so:
> Authentication failed. Try signing in again (and clearing cookies for
the current site). The error was: state parameter did not match the
expected value (possible request forgery).
This happens because you get redirected to an absolute URL, which will
be `sourcegraph.com` (the upstream Sourcegraph instance), which won't
match the cookie you get issued (`sourcegraph.test`). We just need to
make sure it matches and send the code to the callback through our local
proxy:
```diff
- https://sourcegraph.com/.auth/callback?code=sams_ac_...
+ https://sourcegraph.test:3443/.auth/callback?code=sams_ac_...
```
And everything will work :) I confirmed this works for e.g. logging into
S2 with GitHub as well, using the callback issued by GitHub.
We cannot make this the default behaviour, because the redirect URL is
usually configured in the OAuth provider. An example GitHub OAuth
application configuration:
Note their guidance on valid values:
https://docs.github.com/en/apps/oauth-apps/building-oauth-apps/authorizing-oauth-apps#redirect-urls
- we must provide a single, absolute URL
This is very similar to the process used to log in to private Cloud
instances, e.g.
https://cloud-ops.sgdev.org/dashboard/environments/prod/instances/src-bd02273f6b90d1d1beee#log-in-to-the-instance-ui
(scroll to bottom of this section), which indicates we've thought about
similar cases before and found that this was the best way
## Test plan
https://www.loom.com/share/6cb3b3ca475b4b9392aa4b11938e76e6?sid=d99fcb59-4308-45c7-9f68-af9ac44c4e7e
<!-- PR description tips:
https://www.notion.so/sourcegraph/Write-a-good-pull-request-description-610a7fd3e613496eb76f450db5a49b6e
-->
Patches CVE-2024-24790 by upgrading to 27-0-3 tag. However, the patched
version has CVE-2024-24791 😟 and it doesnt have patch.
## Test plan
<!-- REQUIRED; info at
https://docs-legacy.sourcegraph.com/dev/background-information/testing_principles
-->
Build and test image locally.
### Instruction to build and test locally
- Go to `dev/oci_deps.bzl`
- Find the current tag example `docker:26.1.3-dind`
- Go to docker registry and search for updated tag and grab one example:
`docker:27.0.3-dind`
- docker pull --platform linux/amd64 docker:27.0.3-dind
- Add `platforms = ["linux/amd64"],` to the oci_pull for building and
testing locally
```bzl
oci_pull(
name = "upstream_dind_base",
digest = "sha256:2632da0d24924b179adf1c2e6f4ea6fb866747e84baea6b2ffaa8bff982ce102",
platforms = ["linux/amd64"],
)
```
- Run `sg images build dind`
- For testing, run `docker run --rm -it --entrypoint /bin/sh -v
/var/run/docker.sock:/var/run/docker.sock dind:candidate`
- Test docker commands and pull and run image for testing
## Changelog
- Upgraded dind to 27.0.3 to patch CVE-2024-24790 vulnerability
<!-- OPTIONAL; info at
https://www.notion.so/sourcegraph/Writing-a-changelog-entry-dd997f411d524caabf0d8d38a24a878c
-->
Updating the `cody-web-experimental` package to the latest version which
- Fixed problems with telemetry (now we send agent level of telemetry
events with `Web.Cody` client name)
- Fixed remote repository context as you switch between chats
- Added support for cody context filters in file and symbols mentions
- Improved link rendering for remote files (no more "remote-file://"
protocol prefixes.
Also, this PR adds some missing CSS token overrides for the LLM picker
on dotcom.
Note: I, by accident, pushed the `0.2.2` version with the same build as
I have published already in `0.2.1`. NPM doesn't allow to override the
build with the same version, so I had to publish 0.2.3 to make it
through to the Sourcegraph repo. This is why we jump from 0.2.1 to 0.2.3
## Test plan
-General manual checks for Cody Web
We missed during the review that we are not using the `open` helper that
wraps using the right method depending on the OS, which means that `sg
analytics` doesn't work on Linux as is.
## Test plan
Locally tested.
Closes SRCH-706
Previously the success notice would appear in the commit signing area of
the site admin, even if we create a github app for regular code host
stuff. This PR fixes it.
It also udpates the partial storage key of the notice, so that it will
reappear if we create more apps.
## Test plan
Manual testing
## Changelog
<!-- OPTIONAL; info at
https://www.notion.so/sourcegraph/Writing-a-changelog-entry-dd997f411d524caabf0d8d38a24a878c
-->
Closes SRCH-685
When a GitHub app credential is used to push a commit, the commits
aren't signed.
With this PR, we re-use the `DuplicateCommit` method to create a signed
commit so users who don't have a standalone Commit Signing app installed
still get a signed commit when the commit is created with a GitHub app.
To avoid cyclic imports, I had to move the `AuthenticationStrategy`
const to the `internal/types` package and rename it to
`SourceAuthenticationStrategy` so it's clear.
## Test plan
Add a GitHub app as a credential, then create a changeset targeting the
repo that the GitHub app is installed on.
The final commit should be signed.
## Changelog
<!-- OPTIONAL; info at
https://www.notion.so/sourcegraph/Writing-a-changelog-entry-dd997f411d524caabf0d8d38a24a878c
-->
We now have the `gitHubApp` field that can be used instead.
`isGitHubApp` was only introduced as part of the batch changes
integration with github app, and is therefore not released yet.
Closes SRCH-704
## Test plan
Manual testing
## Changelog
<!-- OPTIONAL; info at
https://www.notion.so/sourcegraph/Writing-a-changelog-entry-dd997f411d524caabf0d8d38a24a878c
-->
---------
Co-authored-by: Bolaji Olajide <25608335+BolajiOlajide@users.noreply.github.com>
Removes existing `sg analytics` command and replaces it with a
one-per-invocation sqlite backed approach. This is a local storage for
invocation events before theyre pushed to bigquery
## Test plan
```
sqlite> select * from analytics;
0190792e-af38-751a-b93e-8481290a18b6|1|{"args":[],"command":"sg help","flags":{"help":null,"sg":null},"nargs":0,"end_time":"2024-07-03T15:20:21.069837706Z","success":true}
0190792f-4e2b-7c35-98d6-ad73cab82391|1|{"args":["dotcom"],"command":"sg live","flags":{"live":null,"sg":null},"nargs":1,"end_time":"2024-07-03T15:21:04.563232429Z","success":true}
```
## Changelog
<!-- OPTIONAL; info at
https://www.notion.so/sourcegraph/Writing-a-changelog-entry-dd997f411d524caabf0d8d38a24a878c
-->
---------
Co-authored-by: William Bezuidenhout <william.bezuidenhout@sourcegraph.com>
