mirror of
https://github.com/sourcegraph/sourcegraph.git
synced 2026-02-06 14:31:56 +00:00
dca1b9694d
This PR is stacked on top of all the prior work @chrsmith has done for
shuffling configuration data around; it implements the new "Self hosted
models" functionality.
## Configuration
Configuring a Sourcegraph instance to use self-hosted models basically
involves adding some configuration like this to the site config (if you
set `modelConfiguration`, you are opting in to the new system which is
in early access):
```
// Setting this field means we are opting into the new Cody model configuration system.
"modelConfiguration": {
// Disable use of Sourcegraph's servers for model discovery
"sourcegraph": null,
// Create two model providers
"providerOverrides": [
{
// Our first model provider "mistral" will be a Huggingface TGI deployment which hosts our
// mistral model for chat functionality.
"id": "mistral",
"displayName": "Mistral",
"serverSideConfig": {
"type": "huggingface-tgi",
"endpoints": [{"url": "https://mistral.example.com/v1"}]
},
},
{
// Our second model provider "bigcode" will be a Huggingface TGI deployment which hosts our
// bigcode/starcoder model for code completion functionality.
"id": "bigcode",
"displayName": "Bigcode",
"serverSideConfig": {
"type": "huggingface-tgi",
"endpoints": [{"url": "http://starcoder.example.com/v1"}]
}
}
],
// Make these two models available to Cody users
"modelOverridesRecommendedSettings": [
"mistral::v1::mixtral-8x7b-instruct",
"bigcode::v1::starcoder2-7b"
],
// Configure which models Cody will use by default
"defaultModels": {
"chat": "mistral::v1::mixtral-8x7b-instruct",
"fastChat": "mistral::v1::mixtral-8x7b-instruct",
"codeCompletion": "bigcode::v1::starcoder2-7b"
}
}
```
More advanced configurations are possible, the above is our blessed
configuration for today.
## Hosting models
Another major component of this work is starting to build up
recommendations around how to self-host models, which ones to use, how
to configure them, etc.
For now, we've been testing with these two on a machine with dual A100s:
* Huggingface TGI (this is a Docker container for model inference, which
provides an OpenAI-compatible API - and is widely popular)
* Two models:
* Starcoder2 for code completion; specifically `bigcode/starcoder2-15b`
with `eetq` 8-bit quantization.
* Mixtral 8x7b instruct for chat; specifically
`casperhansen/mixtral-instruct-awq` which uses `awq` 4-bit quantization.
This is our 'starter' configuration. Other models - specifically other
starcoder 2, and mixtral instruct models - certainly work too, and
higher parameter versions may of course provide better results.
Documentation for how to deploy Huggingface TGI, suggested configuration
and debugging tips - coming soon.
## Advanced configuration
As part of this effort, I have added a quite extensive set of
configuration knobs to to the client side model configuration (see `type
ClientSideModelConfigOpenAICompatible` in this PR)
Some of these configuration options are needed for things to work at a
basic level, while others (e.g. prompt customization) are not needed for
basic functionality, but are very important for customers interested in
self-hosting their own models.
Today, Cody clients have a number of different _autocomplete provider
implementations_ which tie model-specific logic to enable autocomplete,
to a provider. For example, if you use a GPT model through Azure OpenAI,
the autocomplete provider for that is entirely different from what you'd
get if you used a GPT model through OpenAI officially. This can lead to
some subtle issues for us, and so it is worth exploring ways to have a
_generalized autocomplete provider_ - and since with self-hosted models
we _must_ address this problem, these configuration knobs fed to the
client from the server are a pathway to doing that - initially just for
self-hosted models, but in the future possibly generalized to other
providers.
## Debugging facilities
Working with customers in the past to use OpenAI-compatible APIs, we've
learned that debugging can be quite a pain. If you can't see what
requests the Sourcegraph backend is making, and what it is getting
back.. it can be quite painful to debug.
This PR implements quite extensive logging, and a `debugConnections`
flag which can be turned on to enable logging of the actual request
payloads and responses. This is critical when a customer is trying to
add support for a new model, their own custom OpenAI API service, etc.
## Robustness
Working with customers in the past, we also learned that various parts
of our backend `openai` provider were not super robust. For example, [if
more than one message was present it was a fatal
error](https://github.com/sourcegraph/sourcegraph/blob/main/internal/completions/client/openai/openai.go#L305),
or if the SSE stream yielded `{"error"}` payloads, they would go
ignored. Similarly, the SSE event stream parser we use is heavily
tailored towards [the exact response
structure](https://github.com/sourcegraph/sourcegraph/blob/main/internal/completions/client/openai/decoder.go#L15-L19)
which OpenAI's official API returns, and is therefor quite brittle if
connecting to a different SSE stream.
For this work, I have _started by forking_ our
`internal/completions/client/openai` - and made a number of major
improvements to it to make it more robust, handle errors better, etc.
I have also replaced the usage of a custom SSE event stream parser -
which was not spec compliant and brittle - with a proper SSE event
stream parser that recently popped up in the Go community:
https://github.com/tmaxmax/go-sse
My intention is that after more extensive testing, this new
`internal/completions/client/openaicompatible` provider will be more
robust, more correct, and all around better than
`internal/completions/client/openai` (and possibly the azure one) so
that we can just supersede those with this new `openaicompatible` one
entirely.
## Client implementation
Much of the work done in this PR is just "let the site admin configure
things, and broadcast that config to the client through the new model
config system."
Actually getting the clients to respect the new configuration, is a task
I am tackling in future `sourcegraph/cody` PRs.
## Test plan
1. This change currently lacks any unit/regression tests, that is a
major noteworthy point. I will follow-up with those in a future PR.
* However, these changes are **incredibly** isolated, clearly only
affecting customers who opt-in to this new self-hosted models
configuration.
* Most of the heavy lifting (SSE streaming, shuffling data around) is
done in other well-tested codebases.
2. Manual testing has played a big role here, specifically:
* Running a dev instance with the new configuration, actually connected
to Huggingface TGI deployed on a remote server.
* Using the new `debugConnections` mechanism (which customers would use)
to directly confirm requests are going to the right places, with the
right data and payloads.
* Confirming with a new client (changes not yet landed) that
autocomplete and chat functionality work.
Can we use more testing? Hell yeah, and I'm going to add it soon. Does
it work quite well and have small room for error? Also yes.
## Changelog
Cody Enterprise: added a new configuration for self-hosting models.
Reach out to support if you would like to use this feature as it is in
early access.
---------
Signed-off-by: Stephen Gutekanst <stephen@sourcegraph.com>
|
||
|---|---|---|
| .. | ||
| accesstoken | ||
| actor | ||
| adminanalytics | ||
| api | ||
| appliance | ||
| audit | ||
| auth | ||
| authbearer | ||
| authz | ||
| batches | ||
| binary | ||
| bytesize | ||
| byteutils | ||
| clientconfig | ||
| cloud | ||
| cmd | ||
| codeintel | ||
| codemonitors | ||
| codygateway | ||
| collections | ||
| comby | ||
| completions | ||
| compute | ||
| conf | ||
| cookie | ||
| ctags_config | ||
| database | ||
| debugserver | ||
| deviceid | ||
| diskcache | ||
| diskusage | ||
| dotcom | ||
| download | ||
| embeddings | ||
| encryption | ||
| endpoint | ||
| env | ||
| errcode | ||
| eventlogger | ||
| executor | ||
| extsvc | ||
| featureflag | ||
| fileutil | ||
| github_apps | ||
| gitserver | ||
| goroutine | ||
| gosyntect | ||
| gqltestutil | ||
| gqlutil | ||
| grpc | ||
| guardrails | ||
| hashutil | ||
| highlight | ||
| honey | ||
| hostmatcher | ||
| hostname | ||
| htmlutil | ||
| httpcli | ||
| httpserver | ||
| httptestutil | ||
| insights | ||
| instrumentation | ||
| ipynb | ||
| jsonc | ||
| k8s/resource | ||
| lazyregexp | ||
| license | ||
| licensing | ||
| limiter | ||
| logging | ||
| luasandbox | ||
| mapfs | ||
| maps | ||
| markdown | ||
| memcmd | ||
| memo | ||
| metrics | ||
| modelconfig | ||
| notebooks | ||
| oauthtoken | ||
| oauthutil | ||
| observation | ||
| oobmigration | ||
| opencodegraph | ||
| otlpenv | ||
| own | ||
| packagefilters | ||
| paths | ||
| pbt | ||
| perforce | ||
| productsubscription | ||
| profiler | ||
| pubsub | ||
| randstring | ||
| ratelimit | ||
| rbac | ||
| rcache | ||
| redislock | ||
| redispool | ||
| releaseregistry | ||
| repos | ||
| repoupdater | ||
| requestclient | ||
| requestinteraction | ||
| sams | ||
| sanitycheck | ||
| search | ||
| searcher | ||
| security | ||
| service | ||
| session | ||
| settings | ||
| siteid | ||
| slack | ||
| sourcegraphoperator | ||
| src-cli | ||
| src-prometheus | ||
| suspiciousnames | ||
| symbols | ||
| sysreq | ||
| telemetry | ||
| telemetrygateway | ||
| temporarysettings | ||
| testutil | ||
| timeutil | ||
| trace | ||
| tracer | ||
| ttlcache | ||
| txemail | ||
| types | ||
| unpack | ||
| updatecheck | ||
| uploadhandler | ||
| uploadstore | ||
| usagestats | ||
| users | ||
| vcs | ||
| version | ||
| webhooks/outbound | ||
| workerutil | ||
| wrexec | ||
| buf.yaml | ||
| BUILD.bazel | ||