mirror of
https://github.com/sourcegraph/sourcegraph.git
synced 2026-02-06 17:51:57 +00:00
Code AI platform with Code Search & Cody
544d261e66
Previously, for providing self-hosted model' configuration (the models
we've tested and believe work well), a site admin would use
configuration like this:
```
"modelConfiguration": {
...
"modelOverridesRecommendedSettings": [
"mistral::v1::mixtral-8x7b-instruct",
"bigcode::v1::starcoder2-7b"
],
}
```
A few problems with this:
1. If you are NOT self-hosting models, you probably really should not be
using this option, as it would set `serverSideConfig` options specific
to self-hosting, but it's naming "recommended settings" which kind of
suggests otherwise!
2. When self-hosting models, there is almost a 1:1 correlation of
`provider` to actual API endpoint (because you have a single endpoint
per model) - so not being able to configure the `mistral` or `bigcode`
parts of the modelref above is problematic (restricts you to hosting
'only one model per provider'). The only escape for this currently is to
abandon the defaults we provide with `modelOverridesRecommendedSettings`
and rewrite it using `modelOverrides` fully yourself.
3. When self-hosting models, needing to configure the
`serverSideConfig.openaicompatible.apiModel` is a really common need -
the most common option probably - but again there's no way to configure
it here, only option is to abandon defaults and rewrite it yourself.
4. If we improve the default values - such as if we learn that a higher
context window size for `mixtral-8x7b-instruct` is better - we currently
don't have a good way to 'release a new version of the defaults' because
the string is a model ref `mistral::v1::mixtral-8x7b-instruct` we'd have
to do this by appending `-v2` to the model name or something. Having
versioning here is important because there are both:
* Breaking changes: if we increase the context window at all, site
admins hosting these models may need to increase limits in their hosted
model deployment - or else the API may just return a hard error ('you
sent me too many tokens')
* Non-breaking changes: if we _decrease_ the context window, Cody
responses will get faster, and it's fine to do. Similarly, adding new
stop sequences may be fine for example.
This PR fixes all of these^ issues by deprecating
`modelOverridesRecommendedSettings` and introducing a new
`selfHostedModels` field which looks like:
```
"modelConfiguration": {
...
"selfHostedModels": [
{
"provider": "mistral",
"model": "mixtral-8x7b-instruct@v1",
"override": {
"serverSideConfig": {
"type": "openaicompatible",
"apiModel": "mixtral-8x7b-instruct-custom!"
}
}
},
{
"provider": "bigcode",
"model": "starcoder2-7b@v1",
"override": {
"serverSideConfig": {
"type": "openaicompatible",
"apiModel": "starcoder2-7b-custom!"
}
}
}
],
}
```
Notably:
* The `provider` part of the model ref is now configurable, enabling
self-hosting more than one model per provider while still benefitting
from our default model configurations.
* `"model": "starcoder2-7b@v1",` is no longer a model ref, but rather a
'default model configuration name' - and has a version associated with
it.
* `override` allows overriding properties of the default `"model":
"starcoder2-7b@v1",` configuration, like the
`serverSideConfig.apiModel`.
## Importance
I'm hoping to ship this to a few customers asap;
* Unblocks customer https://linear.app/sourcegraph/issue/PRIME-447
* Fixes https://linear.app/sourcegraph/issue/PRIME-454 (you can see some
alternatives I considered here before settling on this approach.)
## Test plan
Manually tested for now. Regression tests will come in the near future
and are being tracked on Linear.
## Changelog
Improved configuration functionality for Cody Enterprise with
Self-hosted models.
---------
Signed-off-by: Stephen Gutekanst <stephen@sourcegraph.com>
|
||
|---|---|---|
| .apko | ||
| .aspect | ||
| .buildkite | ||
| .github | ||
| .vscode | ||
| client | ||
| cmd | ||
| dev | ||
| doc | ||
| docker-images | ||
| internal | ||
| lib | ||
| migrations | ||
| monitoring | ||
| schema | ||
| testing | ||
| third_party | ||
| third-party-licenses | ||
| tools | ||
| ui/assets | ||
| wolfi-images | ||
| wolfi-packages | ||
| .bazel_fix_commands.json | ||
| .bazelignore | ||
| .bazeliskrc | ||
| .bazelrc | ||
| .bazelversion | ||
| .dockerignore | ||
| .editorconfig | ||
| .eslintrc.js | ||
| .gitattributes | ||
| .gitignore | ||
| .graphqlrc.yml | ||
| .hadolint.yaml | ||
| .mailmap | ||
| .mocharc.js | ||
| .npmrc | ||
| .pre-commit-config.yaml | ||
| .prettierignore | ||
| .stylelintignore | ||
| .stylelintrc.json | ||
| .swcrc | ||
| .tool-versions | ||
| .trivyignore | ||
| BUILD.bazel | ||
| CHANGELOG.md | ||
| CODENOTIFY | ||
| CONTRIBUTING.md | ||
| deps.bzl | ||
| doc.go | ||
| eslint-relative-formatter.js | ||
| flake.lock | ||
| flake.nix | ||
| gen.go | ||
| go.mod | ||
| go.sum | ||
| graphql-schema-linter.config.js | ||
| LICENSE | ||
| LICENSE.enterprise | ||
| linter_deps.bzl | ||
| mockgen.temp.yaml | ||
| mockgen.test.yaml | ||
| mockgen.yaml | ||
| nogo_config.json | ||
| package.json | ||
| pnpm-lock.yaml | ||
| pnpm-workspace.yaml | ||
| postcss.config.js | ||
| prettier.config.js | ||
| README.md | ||
| release.yaml | ||
| renovate.json | ||
| SECURITY.md | ||
| service-catalog.yaml | ||
| sg.config.yaml | ||
| shell.nix | ||
| stamp_tags.bzl | ||
| tsconfig.base.json | ||
| tsconfig.json | ||
| vitest.shared.ts | ||
| vitest.workspace.ts | ||
| WORKSPACE | ||
Docs •
Contributing •
Twitter •
Discord
Sourcegraph makes it easy to read, write, and fix code—even in big, complex codebases.
- Code search: Search all of your repositories across all branches and all code hosts.
- Code intelligence: Navigate code, find references, see code owners, trace history, and more.
- Fix and refactor: Roll out large-scale changes to many repositories at once and track big migrations.
Getting started
Development
Refer to the Developing Sourcegraph guide to get started.
Documentation
The doc directory has additional documentation for developing and understanding Sourcegraph:
- Architecture: high-level architecture
- Database setup: database best practices
- Go style guide
- Documentation style guide
- GraphQL API: useful tips when modifying the GraphQL API
- Contributing
License
This repository contains primarily non-OSS-licensed files. See LICENSE.
Copyright (c) 2018-present Sourcegraph Inc.