sourcegraph/schema
Stephen Gutekanst 12b0e4e233
site config: minor fixes, make modelConfiguration enable new backend models API, initial self-hosted model config (#63697)
These commits do a few things:

---

46b1303e62ea7e01ba6a441cc55bbe4c166ef5ce corrects a few minor mistakes
with the new site config which I introduced in #63654 - namely fixing
`examples` entries and nullability in a few cases. Nothing controversial
here, just bug fixes.

---

750b61e7dfa661338c9b40042087aed8e795f900 makes it so that the
`/.api/client-config` endpoint returns `"modelsAPIEnabled": true,` if
`"modelConfiguration"` is set in the site config. For context,
`"modelConfiguration"` is a new site config field, which is not used
anywhere before this PR, and has this description:

> BETA FEATURE, only enable if you know what you are doing. If set, Cody
will use the new model configuration system and ignore the old
'completions' site configuration entirely.

I will send a change to the client logic next so that it uses this
`modelsAPIEnabled` field instead of the client-side feature flag
`dev.useServerDefinedModels`.

---

Finally, f52fba342dd2e62a606b885802f7f6bc37f4f4ac and
bde67d57c39f4566dc9287f8793cb5ffd25955b3 make a few site config changes
that @chrsmith and I discussed to enable Self-hosted models support.
Specifically, it makes it possible to specify the following
configuration in the site config:

```
  // Setting this field means we are opting into the new Cody model configuration system which is in beta.
  "modelConfiguration": {
    // Disable use of Sourcegraph's servers for model discovery
    "sourcegraph": null,

    // Configure the OpenAI-compatible API endpoints that Cody should use to provide
    // mistral and bigcode (starcoder) models.
    "providerOverrides": [
      {
        "displayName": "Mistral",
        "id": "mistral",
        "serverSideConfig": {
          "type": "openaicompatible",
          "endpoint": "...",
          "accessToken": "...",
        },
      },
      {
        "displayName": "Bigcode",
        "id": "bigcode",
        "serverSideConfig": {
          "type": "openaicompatible",
          "endpoint": "...",
          "accessToken": "...",
        },
      },
    ],

    // Configure which exact mistral and starcoder models we want available
    "modelOverridesRecommendedSettings": [
      "bigcode::v1::starcoder2-7b",
      "mistral::v1::mixtral-8x7b-instruct"
    ],

    // Configure which models Cody will use by default
    "defaultModels": {
      "chat": "mistral::v1::mixtral-8x7b-instruct",
      "fastChat": "mistral::v1::mixtral-8x7b-instruct",
      "codeCompletion": "bigcode::v1::starcoder2-7b",
    }
  }
```

Currently this site config is not actually used, so configuring
Sourcegraph like this should not be done today, but this will be in a
future PR by me.

@chrsmith one divergence from what we discussed.. me and you had planned
to support this:

```
        "modelOverrides": [
            {
                "bigcode::v1::starcoder2-7b"": {
                    "useRecommendSettings": true,
                },
                "mistral::v1::mixtral-8x22b-instruct": {
                    "useRecommendSettings": true,
                },
             }
        ],
```

However, being able to specify `"useRecommendSettings": true,` inside of
a `ModelOverride` in the site configuration means that all other
`ModelOverride` fields (the ones we are accepting as recommended
settings) must be optional, which seems quite bad and opens up a number
of misconfiguration possibilities.

Instead, I opted to introduce a new top-level field for model overrides
_with recommended settings_, so the above becomes this instead:

```
    "modelOverridesRecommendedSettings": [
      "bigcode::v1::starcoder2-7b",
      "mistral::v1::mixtral-8x7b-instruct"
    ],
```

This has the added benefit of making it impossible to set both
`"useRecommendSettings": true,` and other fields.

I will make it a site config error (prevents admins from saving
configuration) to specify the same model in both `modelOverrides` and
`modelOverridesRecommendedSettings` in a future PR.

---

## Test plan

Doesn't affect users yet. Careful review.

## Changelog

N/A

---------

Signed-off-by: Stephen Gutekanst <stephen@sourcegraph.com>
2024-07-08 16:53:05 -07:00
..
aws_codecommit.schema.json
azuredevops.schema.json Docs: update links to point to new site (#60381) 2024-02-13 00:23:47 +00:00
batch_spec.schema.json batches: add fork attribute to changeset template (#51572) 2023-05-31 13:19:07 -04:00
bitbucket_cloud.schema.json Add support for naming repo explicitly for Bitbucket Cloud (#61536) 2024-04-08 19:03:53 +02:00
bitbucket_server_util.go authz/github: validate provider against default github URL if not set (#24598) 2021-09-06 12:37:33 -04:00
bitbucket_server.schema.json fix(Source): Fix documentation URLs for code hosts help pages (#63274) 2024-06-17 14:32:46 -04:00
bitbucketcloud_util.go Add Bitbucket Cloud as an auth provider with Perms syncing (#46309) 2023-01-16 14:20:35 +02:00
BUILD.bazel bazel: transcribe test ownership to bazel tags (#62664) 2024-05-16 15:51:16 +01:00
changeset_spec.schema.json code-search: handle changeset fork when creating a batch change via src-cli (#58156) 2023-11-08 09:55:05 +01:00
extension_schema.go remove extension registry UI and related GraphQL API (#45891) 2022-12-22 00:10:56 -08:00
gerrit.schema.json gerrit: Add support for SSH cloning (#61537) 2024-04-04 15:56:51 +02:00
github_util.go authz/github: validate provider against default github URL if not set (#24598) 2021-09-06 12:37:33 -04:00
github.schema.json dotcom: Remove on-demand cloning of repositories (#63321) 2024-06-26 14:53:14 -07:00
gitlab_util.go authz/github: validate provider against default github URL if not set (#24598) 2021-09-06 12:37:33 -04:00
gitlab.schema.json dotcom: Remove on-demand cloning of repositories (#63321) 2024-06-26 14:53:14 -07:00
gitolite.schema.json Unremoving phabricator integration fields, adding lines to changelog (#32573) 2022-03-15 10:01:39 -04:00
go-modules.schema.json extsvc: Change default rate limits of npm and Go external services (#34042) 2022-04-19 11:50:46 +00:00
json-schema-draft-07.schema.json
jvm-packages.schema.json packages: improve and expand docs (#49774) 2023-03-21 17:47:57 +00:00
npm-packages.schema.json npm: Bump rate limit. (#37018) 2022-06-10 15:00:51 +00:00
onboardingtour.schema.json user onboarding: Use server side configuration and improve admin experience (#56768) 2023-09-19 22:10:45 +02:00
opencodegraph-protocol.schema.json OpenCodeGraph prototype (#58675) 2023-12-06 21:39:33 -08:00
opencodegraph.schema.json OpenCodeGraph prototype (#58675) 2023-12-06 21:39:33 -08:00
other_external_service.schema.json Remove App from codebase (#59115) 2023-12-21 01:07:05 +01:00
package.json web: sync TS project refenreces (#46407) 2023-01-16 18:55:10 -08:00
pagure.schema.json repos: add Pagure code host support (#28084) 2021-11-23 18:03:35 +01:00
perforce.schema.json Remove unused rateLimit on perforce connections (#58188) 2023-11-15 03:27:14 +01:00
phabricator.schema.json
python-packages.schema.json repos: Introduce Python dependency repos integration (#34886) 2022-05-05 13:24:25 +02:00
README.md site-config: Make symbols not required in syntaxHighlighting (#57276) 2023-10-16 19:53:19 -04:00
ruby-packages.schema.json Packages: add RubyGems support (#42817) 2022-10-17 09:48:18 +02:00
rust-packages.schema.json Remove experimental indexRepositoryName for rust packages (#59176) 2024-01-08 17:42:36 +01:00
schema.go site config: minor fixes, make modelConfiguration enable new backend models API, initial self-hosted model config (#63697) 2024-07-08 16:53:05 -07:00
settings.schema.json Search: remove keyword toggle (#63584) 2024-07-03 10:53:33 -07:00
site.schema.json site config: minor fixes, make modelConfiguration enable new backend models API, initial self-hosted model config (#63697) 2024-07-08 16:53:05 -07:00
stringdata.go Remove App from codebase (#59115) 2023-12-21 01:07:05 +01:00
tsconfig.json web: fix pnpm-lock issue (#47478) 2023-02-09 22:04:31 -08:00
validation_test.go schema: remove non-determinism from TestSchemaValidationUUID (#61728) 2024-04-09 15:50:30 +00:00

Sourcegraph JSON Schemas

JSON Schema is a way to define the structure of a JSON document. It enables typechecking and code intelligence on JSON documents.

Sourcegraph uses the following JSON Schemas:

Modifying a schema

  1. Edit the *.schema.json file in this directory.
  2. Run bazel run //schema:write_generated_schema.
  3. Commit the changes to both files.
  4. Run sg start to automatically update TypeScript schema files.

Known issues

  • The JSON Schema IDs (URIs) are of the form https://sourcegraph.com/v1/*.schema.json#, but these are not actually valid URLs. This means you generally need to supply them to JSON Schema validation libraries manually instead of having the validator fetch the schema from the web.