frontend-internal API error responses every 5m by route
+ +**Descriptions** + +- warning worker: 2%+ frontend-internal API error responses every 5m by route for 5m0s + +**Possible solutions** + +- **Single-container deployments:** Check `docker logs $CONTAINER_ID` for logs starting with `repo-updater` that indicate requests to the frontend service are failing. +- **Kubernetes:** + - Confirm that `kubectl get pods` shows the `frontend` pods are healthy. + - Check `kubectl logs worker` for logs indicate request failures to `frontend` or `frontend-internal`. +- **Docker Compose:** + - Confirm that `docker ps` shows the `frontend-internal` container is healthy. + - Check `docker logs worker` for logs indicating request failures to `frontend` or `frontend-internal`. +- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: + +```json +"observability.silenceAlerts": [ + "warning_worker_frontend_internal_api_error_responses" +] +``` + +*Managed by the [Sourcegraph Code-intelligence team](https://about.sourcegraph.com/handbook/engineering/code-intelligence).* + +container cpu usage total (1m average) across all cores by instance
+ +**Descriptions** + +- warning worker: 99%+ container cpu usage total (1m average) across all cores by instance + +**Possible solutions** + +- **Kubernetes:** Consider increasing CPU limits in the the relevant `Deployment.yaml`. +- **Docker Compose:** Consider increasing `cpus:` of the worker container in `docker-compose.yml`. +- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: + +```json +"observability.silenceAlerts": [ + "warning_worker_container_cpu_usage" +] +``` + +*Managed by the [Sourcegraph Code-intelligence team](https://about.sourcegraph.com/handbook/engineering/code-intelligence).* + +container memory usage by instance
+ +**Descriptions** + +- warning worker: 99%+ container memory usage by instance + +**Possible solutions** + +- **Kubernetes:** Consider increasing memory limit in relevant `Deployment.yaml`. +- **Docker Compose:** Consider increasing `memory:` of worker container in `docker-compose.yml`. +- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: + +```json +"observability.silenceAlerts": [ + "warning_worker_container_memory_usage" +] +``` + +*Managed by the [Sourcegraph Code-intelligence team](https://about.sourcegraph.com/handbook/engineering/code-intelligence).* + +container cpu usage total (90th percentile over 1d) across all cores by instance
+ +**Descriptions** + +- warning worker: 80%+ container cpu usage total (90th percentile over 1d) across all cores by instance for 336h0m0s + +**Possible solutions** + +- **Kubernetes:** Consider increasing CPU limits in the `Deployment.yaml` for the worker service. +- **Docker Compose:** Consider increasing `cpus:` of the worker container in `docker-compose.yml`. +- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: + +```json +"observability.silenceAlerts": [ + "warning_worker_provisioning_container_cpu_usage_long_term" +] +``` + +*Managed by the [Sourcegraph Code-intelligence team](https://about.sourcegraph.com/handbook/engineering/code-intelligence).* + +container memory usage (1d maximum) by instance
+ +**Descriptions** + +- warning worker: 80%+ container memory usage (1d maximum) by instance for 336h0m0s + +**Possible solutions** + +- **Kubernetes:** Consider increasing memory limits in the `Deployment.yaml` for the worker service. +- **Docker Compose:** Consider increasing `memory:` of the worker container in `docker-compose.yml`. +- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: + +```json +"observability.silenceAlerts": [ + "warning_worker_provisioning_container_memory_usage_long_term" +] +``` + +*Managed by the [Sourcegraph Code-intelligence team](https://about.sourcegraph.com/handbook/engineering/code-intelligence).* + +container cpu usage total (5m maximum) across all cores by instance
+ +**Descriptions** + +- warning worker: 90%+ container cpu usage total (5m maximum) across all cores by instance for 30m0s + +**Possible solutions** + +- **Kubernetes:** Consider increasing CPU limits in the the relevant `Deployment.yaml`. +- **Docker Compose:** Consider increasing `cpus:` of the worker container in `docker-compose.yml`. +- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: + +```json +"observability.silenceAlerts": [ + "warning_worker_provisioning_container_cpu_usage_short_term" +] +``` + +*Managed by the [Sourcegraph Code-intelligence team](https://about.sourcegraph.com/handbook/engineering/code-intelligence).* + +container memory usage (5m maximum) by instance
+ +**Descriptions** + +- warning worker: 90%+ container memory usage (5m maximum) by instance + +**Possible solutions** + +- **Kubernetes:** Consider increasing memory limit in relevant `Deployment.yaml`. +- **Docker Compose:** Consider increasing `memory:` of worker container in `docker-compose.yml`. +- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: + +```json +"observability.silenceAlerts": [ + "warning_worker_provisioning_container_memory_usage_short_term" +] +``` + +*Managed by the [Sourcegraph Code-intelligence team](https://about.sourcegraph.com/handbook/engineering/code-intelligence).* + +maximum active goroutines
+ +**Descriptions** + +- warning worker: 10000+ maximum active goroutines for 10m0s + +**Possible solutions** + +- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: + +```json +"observability.silenceAlerts": [ + "warning_worker_go_goroutines" +] +``` + +> NOTE: More help interpreting this metric is available in the [dashboards reference](./dashboards.md#worker-go-goroutines). + +*Managed by the [Sourcegraph Code-intelligence team](https://about.sourcegraph.com/handbook/engineering/code-intelligence).* + +maximum go garbage collection duration
+ +**Descriptions** + +- warning worker: 2s+ maximum go garbage collection duration + +**Possible solutions** + +- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: + +```json +"observability.silenceAlerts": [ + "warning_worker_go_gc_duration_seconds" +] +``` + +*Managed by the [Sourcegraph Code-intelligence team](https://about.sourcegraph.com/handbook/engineering/code-intelligence).* + +percentage pods available
+ +**Descriptions** + +- critical worker: less than 90% percentage pods available for 10m0s + +**Possible solutions** + +- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: + +```json +"observability.silenceAlerts": [ + "critical_worker_pods_available_percentage" +] +``` + +*Managed by the [Sourcegraph Code-intelligence team](https://about.sourcegraph.com/handbook/engineering/code-intelligence).* + +frontend-internal API error responses every 5m by route
diff --git a/doc/admin/observability/dashboards.md b/doc/admin/observability/dashboards.md index 94aa7103166..827d553fedd 100644 --- a/doc/admin/observability/dashboards.md +++ b/doc/admin/observability/dashboards.md @@ -1847,6 +1847,140 @@ This panel indicates percentage pods available.Manages background processes.
+ +### Worker: Internal service requests + +#### worker: frontend_internal_api_error_responses + +This panel indicates frontend-internal API error responses every 5m by route. + +> NOTE: Alerts related to this panel are documented in the [alert solutions reference](./alert_solutions.md#worker-frontend-internal-api-error-responses). + +*Managed by the [Sourcegraph Code-intelligence team](https://about.sourcegraph.com/handbook/engineering/code-intelligence).* + +Manages interaction with code hosts, instructs Gitserver to update repositories.
diff --git a/enterprise/cmd/server/build.sh b/enterprise/cmd/server/build.sh index 5bf771c64e2..ee8bd11f4ae 100755 --- a/enterprise/cmd/server/build.sh +++ b/enterprise/cmd/server/build.sh @@ -8,5 +8,6 @@ export SERVER_PKG=${SERVER_PKG:-github.com/sourcegraph/sourcegraph/enterprise/cm ./cmd/server/build.sh \ github.com/sourcegraph/sourcegraph/enterprise/cmd/frontend \ + github.com/sourcegraph/sourcegraph/enterprise/cmd/worker \ github.com/sourcegraph/sourcegraph/enterprise/cmd/repo-updater \ github.com/sourcegraph/sourcegraph/enterprise/cmd/precise-code-intel-worker diff --git a/enterprise/cmd/worker/CODENOTIFY b/enterprise/cmd/worker/CODENOTIFY new file mode 100644 index 00000000000..16cb819f7ff --- /dev/null +++ b/enterprise/cmd/worker/CODENOTIFY @@ -0,0 +1,3 @@ +# See https://github.com/sourcegraph/codenotify for documentation. + +**/* @efritz diff --git a/enterprise/cmd/worker/Dockerfile b/enterprise/cmd/worker/Dockerfile new file mode 100644 index 00000000000..850902028c7 --- /dev/null +++ b/enterprise/cmd/worker/Dockerfile @@ -0,0 +1,19 @@ +FROM sourcegraph/alpine:3.12@sha256:ce099fbcd3cf70b338fc4cb2a4e1fa9ae847de21afdb0a849a393b87d94fb174 + +ARG COMMIT_SHA="unknown" +ARG DATE="unknown" +ARG VERSION="unknown" + +LABEL org.opencontainers.image.revision=${COMMIT_SHA} +LABEL org.opencontainers.image.created=${DATE} +LABEL org.opencontainers.image.version=${VERSION} +LABEL com.sourcegraph.github.url=https://github.com/sourcegraph/sourcegraph/commit/${COMMIT_SHA} + +# hadolint ignore=DL3018 +RUN apk update && apk add --no-cache \ + tini + +USER sourcegraph +EXPOSE 3189 +ENTRYPOINT ["/sbin/tini", "--", "/usr/local/bin/worker"] +COPY worker /usr/local/bin/ diff --git a/enterprise/cmd/worker/README.md b/enterprise/cmd/worker/README.md new file mode 100644 index 00000000000..20ef9fcced2 --- /dev/null +++ b/enterprise/cmd/worker/README.md @@ -0,0 +1,3 @@ +# Enterprise worker + +The enterprise worker supplements the OSS worker with additional enterprise-relevant background tasks. diff --git a/enterprise/cmd/worker/build.sh b/enterprise/cmd/worker/build.sh new file mode 100755 index 00000000000..f9ad1e09d93 --- /dev/null +++ b/enterprise/cmd/worker/build.sh @@ -0,0 +1,29 @@ +#!/usr/bin/env bash + +# This script builds the enterprise worker docker image. + +cd "$(dirname "${BASH_SOURCE[0]}")/../../.." +set -eu + +OUTPUT=$(mktemp -d -t sgdockerbuild_XXXXXXX) +cleanup() { + rm -rf "$OUTPUT" +} +trap cleanup EXIT + +# Environment for building linux binaries +export GO111MODULE=on +export GOARCH=amd64 +export GOOS=linux +export CGO_ENABLED=0 + +echo "--- go build" +pkg="github.com/sourcegraph/sourcegraph/enterprise/cmd/worker" +go build -trimpath -ldflags "-X github.com/sourcegraph/sourcegraph/internal/version.version=$VERSION -X github.com/sourcegraph/sourcegraph/internal/version.timestamp=$(date +%s)" -buildmode exe -tags dist -o "$OUTPUT/$(basename $pkg)" "$pkg" + +echo "--- docker build" +docker build -f enterprise/cmd/worker/Dockerfile -t "$IMAGE" "$OUTPUT" \ + --progress=plain \ + --build-arg COMMIT_SHA \ + --build-arg DATE \ + --build-arg VERSION diff --git a/enterprise/cmd/worker/main.go b/enterprise/cmd/worker/main.go new file mode 100644 index 00000000000..486bab1b42f --- /dev/null +++ b/enterprise/cmd/worker/main.go @@ -0,0 +1,18 @@ +package main + +import ( + "log" + "os" + "strconv" + + "github.com/sourcegraph/sourcegraph/cmd/worker/shared" +) + +func main() { + debug, _ := strconv.ParseBool(os.Getenv("DEBUG")) + if debug { + log.Println("enterprise edition") + } + + shared.Main() +} diff --git a/enterprise/dev/ci/images/images.go b/enterprise/dev/ci/images/images.go index fbb6f3d0561..dc810315894 100644 --- a/enterprise/dev/ci/images/images.go +++ b/enterprise/dev/ci/images/images.go @@ -37,6 +37,7 @@ var SourcegraphDockerImages = []string{ "gitserver", "query-runner", "repo-updater", + "worker", "searcher", "symbols", "precise-code-intel-worker", diff --git a/enterprise/dev/src-prof-services.json b/enterprise/dev/src-prof-services.json index e273e3af716..5d23c827b50 100644 --- a/enterprise/dev/src-prof-services.json +++ b/enterprise/dev/src-prof-services.json @@ -5,5 +5,6 @@ { "Name": "symbols", "Host": "127.0.0.1:6071" }, { "Name": "repo-updater", "Host": "127.0.0.1:6074" }, { "Name": "query-runner", "Host": "127.0.0.1:6067" }, - { "Name": "precise-code-intel-worker", "Host": "127.0.0.1:6088" } + { "Name": "precise-code-intel-worker", "Host": "127.0.0.1:6088" }, + { "Name": "worker", "Host": "127.0.0.1:6089" } ] diff --git a/monitoring/definitions/worker.go b/monitoring/definitions/worker.go new file mode 100644 index 00000000000..bc3f349d28a --- /dev/null +++ b/monitoring/definitions/worker.go @@ -0,0 +1,71 @@ +package definitions + +import ( + "github.com/sourcegraph/sourcegraph/monitoring/definitions/shared" + "github.com/sourcegraph/sourcegraph/monitoring/monitoring" +) + +func Worker() *monitoring.Container { + return &monitoring.Container{ + Name: "worker", + Title: "Worker", + Description: "Manages background processes.", + Groups: []monitoring.Group{ + { + Title: "Internal service requests", + Hidden: true, + Rows: []monitoring.Row{ + { + shared.FrontendInternalAPIErrorResponses("worker", monitoring.ObservableOwnerCodeIntel).Observable(), + }, + }, + }, + { + Title: shared.TitleContainerMonitoring, + Hidden: true, + Rows: []monitoring.Row{ + { + shared.ContainerCPUUsage("worker", monitoring.ObservableOwnerCodeIntel).Observable(), + shared.ContainerMemoryUsage("worker", monitoring.ObservableOwnerCodeIntel).Observable(), + }, + { + shared.ContainerMissing("worker", monitoring.ObservableOwnerCodeIntel).Observable(), + }, + }, + }, + { + Title: shared.TitleProvisioningIndicators, + Hidden: true, + Rows: []monitoring.Row{ + { + shared.ProvisioningCPUUsageLongTerm("worker", monitoring.ObservableOwnerCodeIntel).Observable(), + shared.ProvisioningMemoryUsageLongTerm("worker", monitoring.ObservableOwnerCodeIntel).Observable(), + }, + { + shared.ProvisioningCPUUsageShortTerm("worker", monitoring.ObservableOwnerCodeIntel).Observable(), + shared.ProvisioningMemoryUsageShortTerm("worker", monitoring.ObservableOwnerCodeIntel).Observable(), + }, + }, + }, + { + Title: shared.TitleGolangMonitoring, + Hidden: true, + Rows: []monitoring.Row{ + { + shared.GoGoroutines("worker", monitoring.ObservableOwnerCodeIntel).Observable(), + shared.GoGcDuration("worker", monitoring.ObservableOwnerCodeIntel).Observable(), + }, + }, + }, + { + Title: shared.TitleKubernetesMonitoring, + Hidden: true, + Rows: []monitoring.Row{ + { + shared.KubernetesPodsAvailable("worker", monitoring.ObservableOwnerCodeIntel).Observable(), + }, + }, + }, + }, + } +} diff --git a/monitoring/main.go b/monitoring/main.go index 4ee705b295b..ec0ba41fda2 100644 --- a/monitoring/main.go +++ b/monitoring/main.go @@ -38,6 +38,7 @@ func main() { definitions.Postgres(), definitions.PreciseCodeIntelWorker(), definitions.QueryRunner(), + definitions.Worker(), definitions.RepoUpdater(), definitions.Searcher(), definitions.Symbols(), diff --git a/sg.config.yaml b/sg.config.yaml index 92c8ececb12..f287c93be9c 100644 --- a/sg.config.yaml +++ b/sg.config.yaml @@ -139,6 +139,24 @@ commands: - internal - cmd/github-proxy + worker: + cmd: ulimit -n 10000 && .bin/worker + install: go build -o .bin/worker github.com/sourcegraph/sourcegraph/cmd/worker + watch: + - lib + - internal + - cmd/worker + + enterprise-worker: + cmd: ulimit -n 10000 && .bin/worker + install: go build -o .bin/worker github.com/sourcegraph/sourcegraph/enterprise/cmd/worker + watch: + - lib + - internal + - enterprise/internal + - cmd/worker + - enterprise/cmd/worker + repo-updater: cmd: .bin/repo-updater install: go build -o .bin/repo-updater github.com/sourcegraph/sourcegraph/cmd/repo-updater @@ -442,6 +460,7 @@ commandsets: default: - frontend + - worker - repo-updater - gitserver - searcher @@ -459,6 +478,7 @@ commandsets: enterprise: - enterprise-frontend + - enterprise-worker - enterprise-repo-updater - enterprise-web - gitserver @@ -476,6 +496,7 @@ commandsets: enterprise-codeintel: - enterprise-frontend + - enterprise-worker - enterprise-repo-updater - enterprise-web - gitserver @@ -504,6 +525,7 @@ commandsets: # DISABLE_CODE_INSIGHTS: false # - enterprise-frontend + - enterprise-worker - enterprise-repo-updater - enterprise-web - gitserver @@ -522,6 +544,7 @@ commandsets: api-only: - enterprise-frontend + - enterprise-worker - enterprise-repo-updater - gitserver - searcher