mirror of
https://github.com/sourcegraph/sourcegraph.git
synced 2026-02-06 17:31:43 +00:00
Code AI platform with Code Search & Cody
2c7d217730
Closes https://linear.app/sourcegraph/issue/SRC-369/finalize-memory-tracking-in-gitserver-on-linux This PR adds a memory tracking feature to all gitserver command, using the utilities introduced in https://app.graphite.dev/github/pr/sourcegraph/sourcegraph/62803. When the GITSERVER_MEMORY_OBSERVATION_ENABLED env var is set, an observer will be spawned that keeps track of the memory usage of the git invocation. This information is available in traces. Additionally, we'll print WARN logs if the commands uses more than 100 MB. ## Test plan Run sg start, and manually tweak the high memory usage threshold down from 500 MB to 1MB. ```diff diff --git a/cmd/gitserver/internal/git/gitcli/command.go b/cmd/gitserver/internal/git/gitcli/command.go index 393fa3c91e6..cf2630fc830 100644 --- a/cmd/gitserver/internal/git/gitcli/command.go +++ b/cmd/gitserver/internal/git/gitcli/command.go @@ -297,7 +297,7 @@ func (rc *cmdReader) waitCmd() error { return rc.err } -const highMemoryUsageThreshold = 500 * 1024 * 1024 // 500 MiB +const highMemoryUsageThreshold = 1 * 1024 * 1024 // 500 MiB func (rc *cmdReader) trace() { duration := time.Since(rc.cmdStart) ``` See log lines similar to the following when I run `sg start` (note the memory usage information in the log fields): ``` [ gitserver-1] WARN gitserver gitcli/command.go:363 High memory usage exec request {"TraceId": "5881f40e3bbbe4d26ec0f32b7f64f535", "SpanId": "a68818dc9f1f9ee2", "ev.Fields": {"cmd": "config", "cmd_ru_maxrss_kib": "5056", "cmd_ru_maxrss_human_readable": "5.2 MB", "traceID": "5881f40e3bbbe4d26ec0f32b7f64f535", "cmd_ru_inblock": "0", "args": "[git config --unset-all gc.auto]", "actor": "0", "cmd_duration_ms": "6", "user_time_ms": "2", "system_time_ms": "2", "repo": "github.com/sgtest/typescript-deps", "error": "git command [git config --unset-all gc.auto] failed with status code 5 (output: \"\")", "cmd_ru_majflt": "10", "cmd_ru_oublock": "0", "trace": "https://sourcegraph.test:3443/-/debug/jaeger/trace/5881f40e3bbbe4d26ec0f32b7f64f535", "exit_status": "5", "cmd_ru_minflt": "494"}} [ gitserver-1] WARN gitserver gitcli/command.go:363 High memory usage exec request {"TraceId": "93cb7e9b3a42cfc085f570b8ad4a2ded", "SpanId": "c35ecddb6ef89e24", "ev.Fields": {"cmd_duration_ms": "6", "system_time_ms": "2", "cmd_ru_maxrss_kib": "5072", "cmd_ru_maxrss_human_readable": "5.2 MB", "cmd_ru_minflt": "495", "cmd_ru_majflt": "10", "actor": "0", "exit_status": "5", "user_time_ms": "2", "cmd_ru_oublock": "0", "traceID": "93cb7e9b3a42cfc085f570b8ad4a2ded", "trace": "https://sourcegraph.test:3443/-/debug/jaeger/trace/93cb7e9b3a42cfc085f570b8ad4a2ded", "repo": "github.com/sgtest/update-ids-test-base", "cmd": "config", "args": "[git config --unset-all gc.auto]", "error": "git command [git config --unset-all gc.auto] failed with status code 5 (output: \"\")", "cmd_ru_inblock": "0"}} [ gitserver-1] WARN gitserver gitcli/command.go:363 High memory usage exec request {"TraceId": "346f1d04dc869f069b04bcabadec0665", "SpanId": "ec43228229e5f531", "ev.Fields": {"actor": "0", "error": "git command [git config --unset-all gc.auto] failed with status code 5 (output: \"\")", "cmd_ru_maxrss_kib": "4960", "trace": "https://sourcegraph.test:3443/-/debug/jaeger/trace/346f1d04dc869f069b04bcabadec0665", "args": "[git config --unset-all gc.auto]", "exit_status": "5", "cmd_duration_ms": "7", "system_time_ms": "2", "repo": "github.com/sgtest/utf8_test", "user_time_ms": "2", "cmd_ru_maxrss_human_readable": "5.1 MB", "cmd_ru_minflt": "487", "cmd_ru_inblock": "0", "cmd_ru_oublock": "0", "traceID": "346f1d04dc869f069b04bcabadec0665", "cmd": "config", "cmd_ru_majflt": "10"}} [ gitserver-1] WARN gitserver gitcli/command.go:363 High memory usage exec request {"TraceId": "1b6a65835e3f75e7e83c8fe7355baeb2", "SpanId": "d525ac1f8c077184", "ev.Fields": {"cmd_ru_oublock": "0", "repo": "github.com/sourcegraph/about", "args": "[git config --unset-all gc.auto]", "cmd_duration_ms": "7", "system_time_ms": "2", "cmd_ru_inblock": "0", "exit_status": "5", "error": "git command [git config --unset-all gc.auto] failed with status code 5 (output: \"\")", "cmd_ru_maxrss_kib": "4912", "cmd_ru_maxrss_human_readable": "5.0 MB", "cmd_ru_minflt": "483", "cmd": "config", "user_time_ms": "2", "trace": "https://sourcegraph.test:3443/-/debug/jaeger/trace/1b6a65835e3f75e7e83c8fe7355baeb2", "actor": "0", "cmd_ru_majflt": "10", "traceID": "1b6a65835e3f75e7e83c8fe7355baeb2"}} [ gitserver-1] WARN gitserver gitcli/command.go:363 High memory usage exec request {"TraceId": "39bbc0cc8b99292e6d3b6dfc067d4049", "SpanId": "60e4d69a25a3074b", "ev.Fields": {"args": "[git config --unset-all gc.auto]", "cmd_duration_ms": "7", "cmd_ru_maxrss_kib": "5056", "cmd_ru_minflt": "492", "trace": "https://sourcegraph.test:3443/-/debug/jaeger/trace/39bbc0cc8b99292e6d3b6dfc067d4049", "actor": "0", "system_time_ms": "2", "cmd_ru_inblock": "0", "cmd": "config", "exit_status": "5", "error": "git command [git config --unset-all gc.auto] failed with status code 5 (output: \"\")", "user_time_ms": "2", "cmd_ru_maxrss_human_readable": "5.2 MB", "cmd_ru_oublock": "0", "repo": "github.com/sourcegraph-testing/etcd", "cmd_ru_majflt": "10", "traceID": "39bbc0cc8b99292e6d3b6dfc067d4049"}} [ ``` Note that I have not tested this e2e in a linux system, but I think it's fine to test it on sourcegraph.com since: 1) we have the benchmarks / integration tests from the previous PR 2) it's behind a environment variable that is off by default ## Changelog Adds a new experimental feature to enable track of `git` command memory invocations when the `GITSERVER_MEMORY_OBSERVATION_ENABLED` environment variable is true (off by default). |
||
|---|---|---|
| .apko | ||
| .aspect | ||
| .buildkite | ||
| .github | ||
| .vscode | ||
| client | ||
| cmd | ||
| dev | ||
| doc | ||
| docker-images | ||
| internal | ||
| lib | ||
| migrations | ||
| monitoring | ||
| schema | ||
| testing | ||
| third_party | ||
| third-party-licenses | ||
| tools | ||
| ui/assets | ||
| wolfi-images | ||
| wolfi-packages | ||
| .bazel_fix_commands.json | ||
| .bazelignore | ||
| .bazeliskrc | ||
| .bazelrc | ||
| .bazelversion | ||
| .dockerignore | ||
| .editorconfig | ||
| .eslintrc.js | ||
| .gitattributes | ||
| .gitignore | ||
| .graphqlrc.yml | ||
| .hadolint.yaml | ||
| .mailmap | ||
| .mocharc.js | ||
| .npmrc | ||
| .percy.yml | ||
| .pre-commit-config.yaml | ||
| .prettierignore | ||
| .stylelintignore | ||
| .stylelintrc.json | ||
| .swcrc | ||
| .tool-versions | ||
| .trivyignore | ||
| BUILD.bazel | ||
| CHANGELOG.md | ||
| CODENOTIFY | ||
| CONTRIBUTING.md | ||
| deps.bzl | ||
| doc.go | ||
| eslint-relative-formatter.js | ||
| flake.lock | ||
| flake.nix | ||
| gen.go | ||
| go.mod | ||
| go.sum | ||
| graphql-schema-linter.config.js | ||
| LICENSE | ||
| LICENSE.enterprise | ||
| linter_deps.bzl | ||
| mockgen.temp.yaml | ||
| mockgen.test.yaml | ||
| mockgen.yaml | ||
| nogo_config.json | ||
| package.json | ||
| pnpm-lock.yaml | ||
| pnpm-workspace.yaml | ||
| postcss.config.js | ||
| prettier.config.js | ||
| README.md | ||
| release.yaml | ||
| renovate.json | ||
| SECURITY.md | ||
| service-catalog.yaml | ||
| sg.config.yaml | ||
| shell.nix | ||
| stamp_tags.bzl | ||
| tsconfig.base.json | ||
| tsconfig.json | ||
| vitest.shared.ts | ||
| vitest.workspace.ts | ||
| WORKSPACE | ||
Docs •
Contributing •
Twitter •
Discord
Sourcegraph makes it easy to read, write, and fix code—even in big, complex codebases.
- Code search: Search all of your repositories across all branches and all code hosts.
- Code intelligence: Navigate code, find references, see code owners, trace history, and more.
- Fix and refactor: Roll out large-scale changes to many repositories at once and track big migrations.
Getting started
Development
Refer to the Developing Sourcegraph guide to get started.
Documentation
The doc directory has additional documentation for developing and understanding Sourcegraph:
- Project FAQ
- Architecture: high-level architecture
- Database setup: database best practices
- Go style guide
- Documentation style guide
- GraphQL API: useful tips when modifying the GraphQL API
- Contributing
License
This repository contains primarily non-OSS-licensed files. See LICENSE.
Copyright (c) 2018-present Sourcegraph Inc.