artin/sourcegraph
1
0
Fork 0
mirror of https://github.com/sourcegraph/sourcegraph.git synced 2026-02-06 17:31:43 +00:00
Code Issues Packages Projects Releases Wiki Activity

site config: minor fixes, make modelConfiguration enable new backend models API, initial self-hosted model config (#63697)

Browse Source 
These commits do a few things:

---

46b1303e62ea7e01ba6a441cc55bbe4c166ef5ce corrects a few minor mistakes
with the new site config which I introduced in #63654 - namely fixing
`examples` entries and nullability in a few cases. Nothing controversial
here, just bug fixes.

---

750b61e7dfa661338c9b40042087aed8e795f900 makes it so that the
`/.api/client-config` endpoint returns `"modelsAPIEnabled": true,` if
`"modelConfiguration"` is set in the site config. For context,
`"modelConfiguration"` is a new site config field, which is not used
anywhere before this PR, and has this description:

> BETA FEATURE, only enable if you know what you are doing. If set, Cody
will use the new model configuration system and ignore the old
'completions' site configuration entirely.

I will send a change to the client logic next so that it uses this
`modelsAPIEnabled` field instead of the client-side feature flag
`dev.useServerDefinedModels`.

---

Finally, f52fba342dd2e62a606b885802f7f6bc37f4f4ac and
bde67d57c39f4566dc9287f8793cb5ffd25955b3 make a few site config changes
that @chrsmith and I discussed to enable Self-hosted models support.
Specifically, it makes it possible to specify the following
configuration in the site config:

```
  // Setting this field means we are opting into the new Cody model configuration system which is in beta.
  "modelConfiguration": {
    // Disable use of Sourcegraph's servers for model discovery
    "sourcegraph": null,

    // Configure the OpenAI-compatible API endpoints that Cody should use to provide
    // mistral and bigcode (starcoder) models.
    "providerOverrides": [
      {
        "displayName": "Mistral",
        "id": "mistral",
        "serverSideConfig": {
          "type": "openaicompatible",
          "endpoint": "...",
          "accessToken": "...",
        },
      },
      {
        "displayName": "Bigcode",
        "id": "bigcode",
        "serverSideConfig": {
          "type": "openaicompatible",
          "endpoint": "...",
          "accessToken": "...",
        },
      },
    ],

    // Configure which exact mistral and starcoder models we want available
    "modelOverridesRecommendedSettings": [
      "bigcode::v1::starcoder2-7b",
      "mistral::v1::mixtral-8x7b-instruct"
    ],

    // Configure which models Cody will use by default
    "defaultModels": {
      "chat": "mistral::v1::mixtral-8x7b-instruct",
      "fastChat": "mistral::v1::mixtral-8x7b-instruct",
      "codeCompletion": "bigcode::v1::starcoder2-7b",
    }
  }
```

Currently this site config is not actually used, so configuring
Sourcegraph like this should not be done today, but this will be in a
future PR by me.

@chrsmith one divergence from what we discussed.. me and you had planned
to support this:

```
        "modelOverrides": [
            {
                "bigcode::v1::starcoder2-7b"": {
                    "useRecommendSettings": true,
                },
                "mistral::v1::mixtral-8x22b-instruct": {
                    "useRecommendSettings": true,
                },
             }
        ],
```

However, being able to specify `"useRecommendSettings": true,` inside of
a `ModelOverride` in the site configuration means that all other
`ModelOverride` fields (the ones we are accepting as recommended
settings) must be optional, which seems quite bad and opens up a number
of misconfiguration possibilities.

Instead, I opted to introduce a new top-level field for model overrides
_with recommended settings_, so the above becomes this instead:

```
    "modelOverridesRecommendedSettings": [
      "bigcode::v1::starcoder2-7b",
      "mistral::v1::mixtral-8x7b-instruct"
    ],
```

This has the added benefit of making it impossible to set both
`"useRecommendSettings": true,` and other fields.

I will make it a site config error (prevents admins from saving
configuration) to specify the same model in both `modelOverrides` and
`modelOverridesRecommendedSettings` in a future PR.

---

## Test plan

Doesn't affect users yet. Careful review.

## Changelog

N/A

---------

Signed-off-by: Stephen Gutekanst <stephen@sourcegraph.com>
This commit is contained in:
Stephen Gutekanst 2024-07-08 16:53:05 -07:00 committed by GitHub
parent 58a320a611
commit 12b0e4e233
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
4 changed files with 90 additions and 24 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
client/web/src/site-admin/SiteAdminReportBugPage.tsx
View File

@ -39,7 +39,7 @@ import { fetchAllConfigAndSettings } from './backend'
*/
interface JSONSchema {
$id: string
definitions?: Record<string, { type: string }>
definitions?: Record<string, { type: string | string[] }>
}
const externalServices: Record<ExternalServiceKind, JSONSchema> = {

7
cmd/frontend/internal/clientconfig/clientconfig.go
View File

@ -14,9 +14,10 @@ import (
func GetForActor(ctx context.Context, logger log.Logger, db database.DB, actor *actor.Actor) (*clientconfig.ClientConfig, error) {
c := clientconfig.ClientConfig{
// TODO(chrsmith): TODO(slimsag): Set this to `true` when and only when clients should use
// the new LLM models httpapi endpoint being added in e.g. https://github.com/sourcegraph/sourcegraph/pull/63507
ModelsAPIEnabled: false,
// If the site config has "modelConfiguration" specified / non-null, then the site admin
// has opted into the new model configuration system, wants to use the new /.api/supported-llms
// endpoint for models, etc.
ModelsAPIEnabled: conf.Get().SiteConfig().ModelConfiguration != nil,
}
// 🚨 SECURITY: This code lets site admins restrict who has access to Cody at all via RBAC.

33
schema/schema.go
View File

@ -2492,14 +2492,15 @@ type ServerSideModelConfigAwsBedrockProvisionedThroughput struct {
Type string `json:"type"`
}
type ServerSideProviderConfig struct {
AwsBedrock *ServerSideProviderConfigAWSBedrock
AzureOpenAI *ServerSideProviderConfigAzureOpenAI
Anthropic *ServerSideProviderConfigAnthropicProvider
Fireworks *ServerSideProviderConfigFireworksProvider
Google *ServerSideProviderConfigGoogleProvider
Openai *ServerSideProviderConfigOpenAIProvider
Sourcegraph *ServerSideProviderConfigSourcegraphProvider
Unused *DoNotUsePhonyDiscriminantType
AwsBedrock *ServerSideProviderConfigAWSBedrock
AzureOpenAI *ServerSideProviderConfigAzureOpenAI
Anthropic *ServerSideProviderConfigAnthropicProvider
Fireworks *ServerSideProviderConfigFireworksProvider
Google *ServerSideProviderConfigGoogleProvider
Openai *ServerSideProviderConfigOpenAIProvider
Openaicompatible *ServerSideProviderConfigOpenAICompatibleProvider
Sourcegraph *ServerSideProviderConfigSourcegraphProvider
Unused *DoNotUsePhonyDiscriminantType
}
func (v ServerSideProviderConfig) MarshalJSON() ([]byte, error) {
@ -2521,6 +2522,9 @@ func (v ServerSideProviderConfig) MarshalJSON() ([]byte, error) {
if v.Openai != nil {
return json.Marshal(v.Openai)
}
if v.Openaicompatible != nil {
return json.Marshal(v.Openaicompatible)
}
if v.Sourcegraph != nil {
return json.Marshal(v.Sourcegraph)
}
@ -2549,12 +2553,14 @@ func (v *ServerSideProviderConfig) UnmarshalJSON(data []byte) error {
return json.Unmarshal(data, &v.Google)
case "openai":
return json.Unmarshal(data, &v.Openai)
case "openaicompatible":
return json.Unmarshal(data, &v.Openaicompatible)
case "sourcegraph":
return json.Unmarshal(data, &v.Sourcegraph)
case "unused":
return json.Unmarshal(data, &v.Unused)
}
return fmt.Errorf("tagged union type must have a %q property whose value is one of %s", "type", []string{"awsBedrock", "azureOpenAI", "anthropic", "fireworks", "google", "openai", "sourcegraph", "unused"})
return fmt.Errorf("tagged union type must have a %q property whose value is one of %s", "type", []string{"awsBedrock", "azureOpenAI", "anthropic", "fireworks", "google", "openai", "openaicompatible", "sourcegraph", "unused"})
}
type ServerSideProviderConfigAWSBedrock struct {
@ -2592,6 +2598,11 @@ type ServerSideProviderConfigGoogleProvider struct {
Endpoint string `json:"endpoint"`
Type string `json:"type"`
}
type ServerSideProviderConfigOpenAICompatibleProvider struct {
AccessToken string `json:"accessToken"`
Endpoint string `json:"endpoint"`
Type string `json:"type"`
}
type ServerSideProviderConfigOpenAIProvider struct {
AccessToken string `json:"accessToken"`
Endpoint string `json:"endpoint"`
@ -3398,6 +3409,10 @@ type SiteModelConfiguration struct {
DefaultModels *DefaultModels `json:"defaultModels,omitempty"`
// ModelOverrides description: Override, or add to, the list of models Cody is aware of and how they are configured to work
ModelOverrides []*ModelOverride `json:"modelOverrides,omitempty"`
// ModelOverridesRecommendedSettings description: Override, or add to, the list of models Cody is aware of - but let Sourcegraph configure how the model should work. Only available for select models.
//
// Specifying the same model both here and in 'modelOverrides' is not allowed.
ModelOverridesRecommendedSettings []string `json:"modelOverridesRecommendedSettings,omitempty"`
// ProviderOverrides description: Configures model providers. Here you can override how Cody connects to model providers and e.g. bring your own API keys or self-hosted models.
ProviderOverrides []*ProviderOverride `json:"providerOverrides,omitempty"`
Sourcegraph *SourcegraphModelConfig `json:"sourcegraph,omitempty"`

72
schema/site.schema.json
View File

@ -3075,6 +3075,9 @@
"definitions": {
"SiteModelConfiguration": {
"type": "object",
"!go": {
"pointer": true
},
"description": "BETA FEATURE, only enable if you know what you are doing. If set, Cody will use the new model configuration system and ignore the old 'completions' site configuration entirely.",
"properties": {
"sourcegraph": {
@ -3096,6 +3099,25 @@
"$ref": "#/definitions/ModelOverride"
}
},
"modelOverridesRecommendedSettings": {
"description": "Override, or add to, the list of models Cody is aware of - but let Sourcegraph configure how the model should work. Only available for select models.\n\nSpecifying the same model both here and in 'modelOverrides' is not allowed.",
"type": "array",
"default": [],
"items": {
"type": "string",
"enum": [
"bigcode::v1::starcoder2-3b",
"bigcode::v1::starcoder2-7b",
"bigcode::v1::starcoder2-15b",
"mistral::v1::mistral-7b",
"mistral::v1::mistral-7b-instruct",
"mistral::v1::mixtral-8x7b",
"mistral::v1::mixtral-8x22b",
"mistral::v1::mixtral-8x7b-instruct",
"mistral::v1::mixtral-8x22b-instruct"
]
}
},
"defaultModels": {
"$ref": "#/definitions/DefaultModels"
}
@ -3103,7 +3125,7 @@
},
"SourcegraphModelConfig": {
"description": "If null, Cody will not use Sourcegraph's servers for model discovery.",
"type": "object",
"type": ["object", "null"],
"!go": {
"pointer": true
},
@ -3155,7 +3177,7 @@
"type": "string",
"enum": ["experimental", "beta", "stable", "deprecated"]
},
"examples": [["beta", "stable"]],
"examples": ["beta", "stable"],
"default": ["stable"]
},
"allow": {
@ -3164,7 +3186,7 @@
"items": {
"type": "string"
},
"examples": [["anthropic::*", "openai::2024-02-01::*"]],
"examples": ["anthropic::*", "openai::2024-02-01::*"],
"default": ["*"]
},
"deny": {
@ -3185,12 +3207,12 @@
"id": {
"description": "provider ID",
"type": "string",
"examples": [["anthropic", "google", "acme-corp-custom"]]
"examples": ["anthropic", "google", "acme-corp-custom"]
},
"displayName": {
"description": "display name",
"type": "string",
"examples": [["Anthropic", "Google", "ACME Corp Custom"]]
"examples": ["Anthropic", "Google", "ACME Corp Custom"]
},
"clientSideConfig": {
"$ref": "#/definitions/ClientSideProviderConfig"
@ -3210,17 +3232,17 @@
"modelRef": {
"description": "The qualified name of the model in '${ProviderID}::${APIVersionID}::${ModelID}' format",
"type": "string",
"examples": [["anthropic::2023-06-01::claude-3-sonnet", "openai::2024-02-01::gpt-4-turbo"]]
"examples": ["anthropic::2023-06-01::claude-3-sonnet", "openai::2024-02-01::gpt-4-turbo"]
},
"displayName": {
"description": "display name",
"type": "string",
"examples": [["Claude 3 Sonnet", "GPT-4 Turbo"]]
"examples": ["Claude 3 Sonnet", "GPT-4 Turbo"]
},
"modelName": {
"description": "model name used when sending requests to the LLM provider's backend API.",
"type": "string",
"examples": [["claude-3-sonnet-20240229", "gpt-4-turbo"]]
"examples": ["claude-3-sonnet-20240229", "gpt-4-turbo"]
},
"capabilities": {
"description": "Whether the model can be used for chat, just autocomplete, etc.",
@ -3234,12 +3256,12 @@
"category": {
"type": "string",
"enum": ["accuracy", "balanced", "speed"],
"examples": [["balanced"]]
"examples": ["balanced"]
},
"status": {
"type": "string",
"enum": ["experimental", "beta", "stable", "deprecated"],
"examples": [["stable"]]
"examples": ["stable"]
},
"contextWindow": {
"$ref": "#/definitions/ContextWindow"
@ -3352,7 +3374,16 @@
"properties": {
"type": {
"type": "string",
"enum": ["awsBedrock", "azureOpenAI", "anthropic", "fireworks", "google", "openai", "sourcegraph"]
"enum": [
"awsBedrock",
"azureOpenAI",
"anthropic",
"fireworks",
"google",
"openai",
"openaicompatible",
"sourcegraph"
]
}
},
"oneOf": [
@ -3374,6 +3405,9 @@
{
"$ref": "#/definitions/ServerSideProviderConfigOpenAIProvider"
},
{
"$ref": "#/definitions/ServerSideProviderConfigOpenAICompatibleProvider"
},
{
"$ref": "#/definitions/ServerSideProviderConfigSourcegraphProvider"
},
@ -3494,6 +3528,22 @@
}
}
},
"ServerSideProviderConfigOpenAICompatibleProvider": {
"type": "object",
"required": ["type", "accessToken", "endpoint"],
"properties": {
"type": {
"type": "string",
"const": "openaicompatible"
},
"accessToken": {
"type": "string"
},
"endpoint": {
"type": "string"
}
}
},
"ServerSideProviderConfigSourcegraphProvider": {
"type": "object",
"required": ["type", "accessToken", "endpoint"],