mirror of
https://github.com/sourcegraph/sourcegraph.git
synced 2026-02-06 19:51:50 +00:00
Code AI platform with Code Search & Cody
d19aa106f9
<!-- PR description tips: https://www.notion.so/sourcegraph/Write-a-good-pull-request-description-610a7fd3e613496eb76f450db5a49b6e --> Closes https://linear.app/sourcegraph/issue/CODY-2758/[autocomplete-latency]-add-circuit-breaker-in-cody-gateway-to-handle This PR introduces an in-memory model availability tracker to ensure we do not send consequent requests to the currently unavailable LLM providers. The tracker maintains a history of error records for each model. It evaluates these records to determine whether the model is available for new requests. The evaluation follows these steps: 1. For every request to an upstream provider, the tracker records any errors that occur. Specifically, it logs timeout errors (when a request exceeds its deadline) and responses with a 429 status code (Too Many Requests). 2. These error records are stored in a circular buffer for each model. This buffer holds a fixed number of records, ensuring efficient memory usage. 3. The tracker calculates the failure ratio by analyzing the stored records. It checks the percentage of errors within a specified evaluation window and compares this against the total number of recent requests. 4. Based on the calculated failure ratio, the tracker decides whether the model is available: - Model Unavailable: If the ratio of failures (timeouts or 429 status codes) exceeds a predefined threshold (X%), the model is marked as unavailable. In this state, the system does not send new requests to the upstream provider. When a model is unavailable, the system immediately returns an error status code, typically a 503 Service Unavailable, to the client. This informs the client that the service is temporarily unavailable due to upstream issues. - Model Available: If the failure ratio is within acceptable limits, the system proceeds with sending the request to the upstream provider. This PR suggests considering a model unavailable if **95% of the last 100 requests within the past minute** either time out or return a 429 status code. I am not sure about these exact values and suggest them as a starting point for discussion ## Test plan - Added unit tests - CI <!-- REQUIRED; info at https://docs-legacy.sourcegraph.com/dev/background-information/testing_principles --> ## Changelog <!-- OPTIONAL; info at https://www.notion.so/sourcegraph/Writing-a-changelog-entry-dd997f411d524caabf0d8d38a24a878c --> |
||
|---|---|---|
| .apko | ||
| .aspect | ||
| .buildkite | ||
| .github | ||
| .vscode | ||
| client | ||
| cmd | ||
| dev | ||
| doc | ||
| docker-images | ||
| internal | ||
| lib | ||
| migrations | ||
| monitoring | ||
| schema | ||
| testing | ||
| third_party | ||
| third-party-licenses | ||
| tools | ||
| ui/assets | ||
| wolfi-images | ||
| wolfi-packages | ||
| .bazel_fix_commands.json | ||
| .bazelignore | ||
| .bazeliskrc | ||
| .bazelrc | ||
| .bazelversion | ||
| .dockerignore | ||
| .editorconfig | ||
| .eslintrc.js | ||
| .gitattributes | ||
| .gitignore | ||
| .graphqlrc.yml | ||
| .hadolint.yaml | ||
| .mailmap | ||
| .mocharc.js | ||
| .npmrc | ||
| .pre-commit-config.yaml | ||
| .prettierignore | ||
| .stylelintignore | ||
| .stylelintrc.json | ||
| .swcrc | ||
| .tool-versions | ||
| .trivyignore | ||
| BUILD.bazel | ||
| CHANGELOG.md | ||
| CODENOTIFY | ||
| CONTRIBUTING.md | ||
| deps.bzl | ||
| doc.go | ||
| eslint-relative-formatter.js | ||
| flake.lock | ||
| flake.nix | ||
| gen.go | ||
| go.mod | ||
| go.sum | ||
| graphql-schema-linter.config.js | ||
| LICENSE | ||
| LICENSE.enterprise | ||
| linter_deps.bzl | ||
| mockgen.temp.yaml | ||
| mockgen.test.yaml | ||
| mockgen.yaml | ||
| nogo_config.json | ||
| package.json | ||
| pnpm-lock.yaml | ||
| pnpm-workspace.yaml | ||
| postcss.config.js | ||
| prettier.config.js | ||
| README.md | ||
| release.yaml | ||
| renovate.json | ||
| SECURITY.md | ||
| service-catalog.yaml | ||
| sg.config.yaml | ||
| shell.nix | ||
| stamp_tags.bzl | ||
| tsconfig.base.json | ||
| tsconfig.json | ||
| vitest.shared.ts | ||
| vitest.workspace.ts | ||
| WORKSPACE | ||
Docs •
Contributing •
Twitter •
Discord
Sourcegraph makes it easy to read, write, and fix code—even in big, complex codebases.
- Code search: Search all of your repositories across all branches and all code hosts.
- Code intelligence: Navigate code, find references, see code owners, trace history, and more.
- Fix and refactor: Roll out large-scale changes to many repositories at once and track big migrations.
Getting started
Development
Refer to the Developing Sourcegraph guide to get started.
Documentation
The doc directory has additional documentation for developing and understanding Sourcegraph:
- Architecture: high-level architecture
- Database setup: database best practices
- Go style guide
- Documentation style guide
- GraphQL API: useful tips when modifying the GraphQL API
- Contributing
License
This repository contains primarily non-OSS-licensed files. See LICENSE.
Copyright (c) 2018-present Sourcegraph Inc.