mirror of
https://github.com/sourcegraph/sourcegraph.git
synced 2026-02-06 16:51:55 +00:00
544d261e66
Previously, for providing self-hosted model' configuration (the models
we've tested and believe work well), a site admin would use
configuration like this:
```
"modelConfiguration": {
...
"modelOverridesRecommendedSettings": [
"mistral::v1::mixtral-8x7b-instruct",
"bigcode::v1::starcoder2-7b"
],
}
```
A few problems with this:
1. If you are NOT self-hosting models, you probably really should not be
using this option, as it would set `serverSideConfig` options specific
to self-hosting, but it's naming "recommended settings" which kind of
suggests otherwise!
2. When self-hosting models, there is almost a 1:1 correlation of
`provider` to actual API endpoint (because you have a single endpoint
per model) - so not being able to configure the `mistral` or `bigcode`
parts of the modelref above is problematic (restricts you to hosting
'only one model per provider'). The only escape for this currently is to
abandon the defaults we provide with `modelOverridesRecommendedSettings`
and rewrite it using `modelOverrides` fully yourself.
3. When self-hosting models, needing to configure the
`serverSideConfig.openaicompatible.apiModel` is a really common need -
the most common option probably - but again there's no way to configure
it here, only option is to abandon defaults and rewrite it yourself.
4. If we improve the default values - such as if we learn that a higher
context window size for `mixtral-8x7b-instruct` is better - we currently
don't have a good way to 'release a new version of the defaults' because
the string is a model ref `mistral::v1::mixtral-8x7b-instruct` we'd have
to do this by appending `-v2` to the model name or something. Having
versioning here is important because there are both:
* Breaking changes: if we increase the context window at all, site
admins hosting these models may need to increase limits in their hosted
model deployment - or else the API may just return a hard error ('you
sent me too many tokens')
* Non-breaking changes: if we _decrease_ the context window, Cody
responses will get faster, and it's fine to do. Similarly, adding new
stop sequences may be fine for example.
This PR fixes all of these^ issues by deprecating
`modelOverridesRecommendedSettings` and introducing a new
`selfHostedModels` field which looks like:
```
"modelConfiguration": {
...
"selfHostedModels": [
{
"provider": "mistral",
"model": "mixtral-8x7b-instruct@v1",
"override": {
"serverSideConfig": {
"type": "openaicompatible",
"apiModel": "mixtral-8x7b-instruct-custom!"
}
}
},
{
"provider": "bigcode",
"model": "starcoder2-7b@v1",
"override": {
"serverSideConfig": {
"type": "openaicompatible",
"apiModel": "starcoder2-7b-custom!"
}
}
}
],
}
```
Notably:
* The `provider` part of the model ref is now configurable, enabling
self-hosting more than one model per provider while still benefitting
from our default model configurations.
* `"model": "starcoder2-7b@v1",` is no longer a model ref, but rather a
'default model configuration name' - and has a version associated with
it.
* `override` allows overriding properties of the default `"model":
"starcoder2-7b@v1",` configuration, like the
`serverSideConfig.apiModel`.
## Importance
I'm hoping to ship this to a few customers asap;
* Unblocks customer https://linear.app/sourcegraph/issue/PRIME-447
* Fixes https://linear.app/sourcegraph/issue/PRIME-454 (you can see some
alternatives I considered here before settling on this approach.)
## Test plan
Manually tested for now. Regression tests will come in the near future
and are being tracked on Linear.
## Changelog
Improved configuration functionality for Cody Enterprise with
Self-hosted models.
---------
Signed-off-by: Stephen Gutekanst <stephen@sourcegraph.com>
|
||
|---|---|---|
| .. | ||
| aws_codecommit.schema.json | ||
| azuredevops.schema.json | ||
| batch_spec.schema.json | ||
| bitbucket_cloud.schema.json | ||
| bitbucket_server_util.go | ||
| bitbucket_server.schema.json | ||
| bitbucketcloud_util.go | ||
| BUILD.bazel | ||
| changeset_spec.schema.json | ||
| gerrit.schema.json | ||
| github_util.go | ||
| github.schema.json | ||
| gitlab_util.go | ||
| gitlab.schema.json | ||
| gitolite.schema.json | ||
| go-modules.schema.json | ||
| json-schema-draft-07.schema.json | ||
| jvm-packages.schema.json | ||
| npm-packages.schema.json | ||
| onboardingtour.schema.json | ||
| opencodegraph-protocol.schema.json | ||
| opencodegraph.schema.json | ||
| other_external_service.schema.json | ||
| package.json | ||
| pagure.schema.json | ||
| perforce.schema.json | ||
| phabricator.schema.json | ||
| python-packages.schema.json | ||
| README.md | ||
| ruby-packages.schema.json | ||
| rust-packages.schema.json | ||
| schema.go | ||
| settings.schema.json | ||
| site.schema.json | ||
| stringdata.go | ||
| tsconfig.json | ||
| validation_test.go | ||
Sourcegraph JSON Schemas
JSON Schema is a way to define the structure of a JSON document. It enables typechecking and code intelligence on JSON documents.
Sourcegraph uses the following JSON Schemas:
Modifying a schema
- Edit the
*.schema.jsonfile in this directory. - Run
bazel run //schema:write_generated_schema. - Commit the changes to both files.
- Run
sg startto automatically update TypeScript schema files.
Known issues
- The JSON Schema IDs (URIs) are of the form
https://sourcegraph.com/v1/*.schema.json#, but these are not actually valid URLs. This means you generally need to supply them to JSON Schema validation libraries manually instead of having the validator fetch the schema from the web.