artin/sourcegraph
1
0
Fork 0
mirror of https://github.com/sourcegraph/sourcegraph.git synced 2026-02-06 15:51:43 +00:00
Code Issues Packages Projects Releases Wiki Activity
main
sourcegraph/doc/admin/observability/alerts.md

8223 lines
328 KiB
Markdown
Raw Permalink Normal View History

monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
# Alerts reference
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
monitoring: assign obvious owners to DatabaseConnectionsMonitoringGroup (#58474) updates `DatabaseConnectionsMonitoringGroup` to accept an owner - before it was just hardcoded to `ObservableOwnerDevOps`, which is not very helpful. This assigns some of the obvious service owners: 1. Source: gitserver, repo-updater 2. Cody: embeddings (but should eventually be @sourcegraph/search-platform, along with all embeddings alerts: https://github.com/sourcegraph/sourcegraph/pull/58474#issuecomment-1821505062) Source is an active owner based on [thread](https://sourcegraph.slack.com/archives/C0652SSUA20/p1700592165408089?thread_ts=1700549423.860019&cid=C0652SSUA20), and Cody is a fairly recent addition so hopefully it's valid. I'm not sure the Search one is still up-to-date, so I didn't change some of the obvious search services - for now, these still point to DevOps as they did before. If it becomes problematic we can revisit later.
2023-11-22 01:06:53 +00:00
<!-- DO NOT EDIT: generated via: bazel run //doc/admin/observability:write_monitoring_docs -->
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
This document contains a complete reference of all alerts in Sourcegraph's monitoring, and next steps for when you find alerts that are firing.
If your alert isn't mentioned here, or if the next steps don't help, [contact us](mailto:support@sourcegraph.com) for assistance.
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
monitoring: dashboards reference, refresh observability docs (#16939) - New monitoring definition field, 'Interpretation' - New generated dashboards reference - Grafana panels now link to alert solutions and dashboards reference - Refreshed observability docs Co-authored-by: uwedeportivo <534011+uwedeportivo@users.noreply.github.com>
2020-12-23 02:36:41 +00:00
To learn more about Sourcegraph's alerting and how to set up alerts, see [our alerting guide](https://docs.sourcegraph.com/admin/observability/alerting).
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
## frontend: 99th_percentile_search_request_duration
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
<p class="subtitle">99th percentile successful search request duration over 5m</p>
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
monitoring: dashboards reference, refresh observability docs (#16939) - New monitoring definition field, 'Interpretation' - New generated dashboards reference - Grafana panels now link to alert solutions and dashboards reference - Refreshed observability docs Co-authored-by: uwedeportivo <534011+uwedeportivo@users.noreply.github.com>
2020-12-23 02:36:41 +00:00
- <span class="badge badge-warning">warning</span> frontend: 20s+ 99th percentile successful search request duration over 5m
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
- **Get details on the exact queries that are slow** by configuring `"observability.logSlowSearches": 20,` in the site configuration and looking for `frontend` warning logs prefixed with `slow search request` for additional details.
- **Check that most repositories are indexed** by visiting https://sourcegraph.example.com/site-admin/repositories?filter=needs-index (it should show few or no results.)
- **Kubernetes:** Check CPU usage of zoekt-webserver in the indexed-search pod, consider increasing CPU limits in the `indexed-search.Deployment.yaml` if regularly hitting max CPU utilization.
- **Docker Compose:** Check CPU usage on the Zoekt Web Server dashboard, consider increasing `cpus:` of the zoekt-webserver container in `docker-compose.yml` if regularly hitting max CPU utilization.
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#frontend-99th-percentile-search-request-duration).
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"warning_frontend_99th_percentile_search_request_duration"
]
```
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
Monitoring: update owners for code insights and batches (#59313) This updates ownership for search, code insights, and batch changes. --------- Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2024-01-04 20:18:23 +00:00
<sub>*Managed by the [Sourcegraph Code Search team](https://handbook.sourcegraph.com/departments/engineering/teams/code-search).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
monitoring: extract into a submodule (#45786) This change extracts `monitoring` into a submodule for import in `sourcegraph/controller` (https://github.com/sourcegraph/controller/pull/195) so that we can generate dashboards for Cloud instances. These steps were required: 1. Initialize a `go.mod` in `monitoring` 2. Extract `dev/sg/internal/cliutil` into `lib` to avoid illegal imports from `monitoring` 3. Add local replaces to both `sourcegraph/sourcegraph` and `monitoring` 4. `go mod tidy` on all submodules 5. Update `go generate ./monitoring` commands to use `sg`, since the `go generate` command no longer works 6. Update `grafana/build.sh`, `prometheus/build.sh` to build the submodule 7. Amend linters to check for multiple `go.mod` files and ban imports of `github.com/sourcegraph/sourcegraph` 8. Update `sg generate go` to run in directories rather than from root The only caveat is that if you use VS code, you will now need to open `monitoring` in a separate workspace or similar, like with `lib`. Co-authored-by: Joe Chen <joe@sourcegraph.com>
2022-12-19 17:49:25 +00:00
Generated query for warning alert: `max((histogram_quantile(0.99, sum by (le) (rate(src_search_streaming_latency_seconds_bucket{source="browser"}[5m])))) >= 20)`
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
</details>
monitoring: improve solutions formatting, add team information (#14878)
2020-10-20 05:11:20 +00:00
<br />
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
## frontend: 90th_percentile_search_request_duration
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
<p class="subtitle">90th percentile successful search request duration over 5m</p>
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
monitoring: dashboards reference, refresh observability docs (#16939) - New monitoring definition field, 'Interpretation' - New generated dashboards reference - Grafana panels now link to alert solutions and dashboards reference - Refreshed observability docs Co-authored-by: uwedeportivo <534011+uwedeportivo@users.noreply.github.com>
2020-12-23 02:36:41 +00:00
- <span class="badge badge-warning">warning</span> frontend: 15s+ 90th percentile successful search request duration over 5m
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
- **Get details on the exact queries that are slow** by configuring `"observability.logSlowSearches": 15,` in the site configuration and looking for `frontend` warning logs prefixed with `slow search request` for additional details.
- **Check that most repositories are indexed** by visiting https://sourcegraph.example.com/site-admin/repositories?filter=needs-index (it should show few or no results.)
- **Kubernetes:** Check CPU usage of zoekt-webserver in the indexed-search pod, consider increasing CPU limits in the `indexed-search.Deployment.yaml` if regularly hitting max CPU utilization.
- **Docker Compose:** Check CPU usage on the Zoekt Web Server dashboard, consider increasing `cpus:` of the zoekt-webserver container in `docker-compose.yml` if regularly hitting max CPU utilization.
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#frontend-90th-percentile-search-request-duration).
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"warning_frontend_90th_percentile_search_request_duration"
]
```
Monitoring: update owners for code insights and batches (#59313) This updates ownership for search, code insights, and batch changes. --------- Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2024-01-04 20:18:23 +00:00
<sub>*Managed by the [Sourcegraph Code Search team](https://handbook.sourcegraph.com/departments/engineering/teams/code-search).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
monitoring: extract into a submodule (#45786) This change extracts `monitoring` into a submodule for import in `sourcegraph/controller` (https://github.com/sourcegraph/controller/pull/195) so that we can generate dashboards for Cloud instances. These steps were required: 1. Initialize a `go.mod` in `monitoring` 2. Extract `dev/sg/internal/cliutil` into `lib` to avoid illegal imports from `monitoring` 3. Add local replaces to both `sourcegraph/sourcegraph` and `monitoring` 4. `go mod tidy` on all submodules 5. Update `go generate ./monitoring` commands to use `sg`, since the `go generate` command no longer works 6. Update `grafana/build.sh`, `prometheus/build.sh` to build the submodule 7. Amend linters to check for multiple `go.mod` files and ban imports of `github.com/sourcegraph/sourcegraph` 8. Update `sg generate go` to run in directories rather than from root The only caveat is that if you use VS code, you will now need to open `monitoring` in a separate workspace or similar, like with `lib`. Co-authored-by: Joe Chen <joe@sourcegraph.com>
2022-12-19 17:49:25 +00:00
Generated query for warning alert: `max((histogram_quantile(0.9, sum by (le) (rate(src_search_streaming_latency_seconds_bucket{source="browser"}[5m])))) >= 15)`
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
</details>
monitoring: improve solutions formatting, add team information (#14878)
2020-10-20 05:11:20 +00:00
<br />
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
## frontend: hard_timeout_search_responses
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
<p class="subtitle">hard timeout search responses every 5m</p>
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
monitoring: dashboards reference, refresh observability docs (#16939) - New monitoring definition field, 'Interpretation' - New generated dashboards reference - Grafana panels now link to alert solutions and dashboards reference - Refreshed observability docs Co-authored-by: uwedeportivo <534011+uwedeportivo@users.noreply.github.com>
2020-12-23 02:36:41 +00:00
- <span class="badge badge-warning">warning</span> frontend: 2%+ hard timeout search responses every 5m for 15m0s
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#frontend-hard-timeout-search-responses).
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
monitoring: forbid critical alerts with no solutions (#36423) Adds the restriction that critical alerts must have solutions defined as a hard requirement within the generator, and fixes all the outstanding issues: - Add PossibleSolutions for the pods_available_percentage and pg_up critical alerts (we have a similar solution doc in the container_missing alert) - Removes all the other critical alerts without possible solutions Co-authored-by: Michael Lin <mlzc@hey.com>
2022-06-02 15:51:43 +00:00
"warning_frontend_hard_timeout_search_responses"
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
]
```
Monitoring: update owners for code insights and batches (#59313) This updates ownership for search, code insights, and batch changes. --------- Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2024-01-04 20:18:23 +00:00
<sub>*Managed by the [Sourcegraph Code Search team](https://handbook.sourcegraph.com/departments/engineering/teams/code-search).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
Generated query for warning alert: `max(((sum(increase(src_graphql_search_response{request_name!="CodeIntelSearch",source="browser",status="timeout"}[5m])) + sum(increase(src_graphql_search_response{alert_type="timed_out",request_name!="CodeIntelSearch",source="browser",status="alert"}[5m]))) / sum(increase(src_graphql_search_response{request_name!="CodeIntelSearch",source="browser"}[5m])) * 100) >= 2)`
</details>
monitoring: improve solutions formatting, add team information (#14878)
2020-10-20 05:11:20 +00:00
<br />
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
## frontend: hard_error_search_responses
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
<p class="subtitle">hard error search responses every 5m</p>
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
monitoring: dashboards reference, refresh observability docs (#16939) - New monitoring definition field, 'Interpretation' - New generated dashboards reference - Grafana panels now link to alert solutions and dashboards reference - Refreshed observability docs Co-authored-by: uwedeportivo <534011+uwedeportivo@users.noreply.github.com>
2020-12-23 02:36:41 +00:00
- <span class="badge badge-warning">warning</span> frontend: 2%+ hard error search responses every 5m for 15m0s
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#frontend-hard-error-search-responses).
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
monitoring: forbid critical alerts with no solutions (#36423) Adds the restriction that critical alerts must have solutions defined as a hard requirement within the generator, and fixes all the outstanding issues: - Add PossibleSolutions for the pods_available_percentage and pg_up critical alerts (we have a similar solution doc in the container_missing alert) - Removes all the other critical alerts without possible solutions Co-authored-by: Michael Lin <mlzc@hey.com>
2022-06-02 15:51:43 +00:00
"warning_frontend_hard_error_search_responses"
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
]
```
Monitoring: update owners for code insights and batches (#59313) This updates ownership for search, code insights, and batch changes. --------- Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2024-01-04 20:18:23 +00:00
<sub>*Managed by the [Sourcegraph Code Search team](https://handbook.sourcegraph.com/departments/engineering/teams/code-search).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
monitoring: extract into a submodule (#45786) This change extracts `monitoring` into a submodule for import in `sourcegraph/controller` (https://github.com/sourcegraph/controller/pull/195) so that we can generate dashboards for Cloud instances. These steps were required: 1. Initialize a `go.mod` in `monitoring` 2. Extract `dev/sg/internal/cliutil` into `lib` to avoid illegal imports from `monitoring` 3. Add local replaces to both `sourcegraph/sourcegraph` and `monitoring` 4. `go mod tidy` on all submodules 5. Update `go generate ./monitoring` commands to use `sg`, since the `go generate` command no longer works 6. Update `grafana/build.sh`, `prometheus/build.sh` to build the submodule 7. Amend linters to check for multiple `go.mod` files and ban imports of `github.com/sourcegraph/sourcegraph` 8. Update `sg generate go` to run in directories rather than from root The only caveat is that if you use VS code, you will now need to open `monitoring` in a separate workspace or similar, like with `lib`. Co-authored-by: Joe Chen <joe@sourcegraph.com>
2022-12-19 17:49:25 +00:00
Generated query for warning alert: `max((sum by (status) (increase(src_graphql_search_response{request_name!="CodeIntelSearch",source="browser",status=~"error"}[5m])) / ignoring (status) group_left () sum(increase(src_graphql_search_response{request_name!="CodeIntelSearch",source="browser"}[5m])) * 100) >= 2)`
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
</details>
monitoring: improve solutions formatting, add team information (#14878)
2020-10-20 05:11:20 +00:00
<br />
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
## frontend: partial_timeout_search_responses
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
<p class="subtitle">partial timeout search responses every 5m</p>
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
monitoring: dashboards reference, refresh observability docs (#16939) - New monitoring definition field, 'Interpretation' - New generated dashboards reference - Grafana panels now link to alert solutions and dashboards reference - Refreshed observability docs Co-authored-by: uwedeportivo <534011+uwedeportivo@users.noreply.github.com>
2020-12-23 02:36:41 +00:00
- <span class="badge badge-warning">warning</span> frontend: 5%+ partial timeout search responses every 5m for 15m0s
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#frontend-partial-timeout-search-responses).
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"warning_frontend_partial_timeout_search_responses"
]
```
Monitoring: update owners for code insights and batches (#59313) This updates ownership for search, code insights, and batch changes. --------- Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2024-01-04 20:18:23 +00:00
<sub>*Managed by the [Sourcegraph Code Search team](https://handbook.sourcegraph.com/departments/engineering/teams/code-search).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
monitoring: extract into a submodule (#45786) This change extracts `monitoring` into a submodule for import in `sourcegraph/controller` (https://github.com/sourcegraph/controller/pull/195) so that we can generate dashboards for Cloud instances. These steps were required: 1. Initialize a `go.mod` in `monitoring` 2. Extract `dev/sg/internal/cliutil` into `lib` to avoid illegal imports from `monitoring` 3. Add local replaces to both `sourcegraph/sourcegraph` and `monitoring` 4. `go mod tidy` on all submodules 5. Update `go generate ./monitoring` commands to use `sg`, since the `go generate` command no longer works 6. Update `grafana/build.sh`, `prometheus/build.sh` to build the submodule 7. Amend linters to check for multiple `go.mod` files and ban imports of `github.com/sourcegraph/sourcegraph` 8. Update `sg generate go` to run in directories rather than from root The only caveat is that if you use VS code, you will now need to open `monitoring` in a separate workspace or similar, like with `lib`. Co-authored-by: Joe Chen <joe@sourcegraph.com>
2022-12-19 17:49:25 +00:00
Generated query for warning alert: `max((sum by (status) (increase(src_graphql_search_response{request_name!="CodeIntelSearch",source="browser",status="partial_timeout"}[5m])) / ignoring (status) group_left () sum(increase(src_graphql_search_response{request_name!="CodeIntelSearch",source="browser"}[5m])) * 100) >= 5)`
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
</details>
monitoring: improve solutions formatting, add team information (#14878)
2020-10-20 05:11:20 +00:00
<br />
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
## frontend: search_alert_user_suggestions
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
<p class="subtitle">search alert user suggestions shown every 5m</p>
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
monitoring: dashboards reference, refresh observability docs (#16939) - New monitoring definition field, 'Interpretation' - New generated dashboards reference - Grafana panels now link to alert solutions and dashboards reference - Refreshed observability docs Co-authored-by: uwedeportivo <534011+uwedeportivo@users.noreply.github.com>
2020-12-23 02:36:41 +00:00
- <span class="badge badge-warning">warning</span> frontend: 5%+ search alert user suggestions shown every 5m for 15m0s
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
- This indicates your user`s are making syntax errors or similar user errors.
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#frontend-search-alert-user-suggestions).
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"warning_frontend_search_alert_user_suggestions"
]
```
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
Monitoring: update owners for code insights and batches (#59313) This updates ownership for search, code insights, and batch changes. --------- Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2024-01-04 20:18:23 +00:00
<sub>*Managed by the [Sourcegraph Code Search team](https://handbook.sourcegraph.com/departments/engineering/teams/code-search).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
monitoring: extract into a submodule (#45786) This change extracts `monitoring` into a submodule for import in `sourcegraph/controller` (https://github.com/sourcegraph/controller/pull/195) so that we can generate dashboards for Cloud instances. These steps were required: 1. Initialize a `go.mod` in `monitoring` 2. Extract `dev/sg/internal/cliutil` into `lib` to avoid illegal imports from `monitoring` 3. Add local replaces to both `sourcegraph/sourcegraph` and `monitoring` 4. `go mod tidy` on all submodules 5. Update `go generate ./monitoring` commands to use `sg`, since the `go generate` command no longer works 6. Update `grafana/build.sh`, `prometheus/build.sh` to build the submodule 7. Amend linters to check for multiple `go.mod` files and ban imports of `github.com/sourcegraph/sourcegraph` 8. Update `sg generate go` to run in directories rather than from root The only caveat is that if you use VS code, you will now need to open `monitoring` in a separate workspace or similar, like with `lib`. Co-authored-by: Joe Chen <joe@sourcegraph.com>
2022-12-19 17:49:25 +00:00
Generated query for warning alert: `max((sum by (alert_type) (increase(src_graphql_search_response{alert_type!~"timed_out|no_results__suggest_quotes",request_name!="CodeIntelSearch",source="browser",status="alert"}[5m])) / ignoring (alert_type) group_left () sum(increase(src_graphql_search_response{request_name!="CodeIntelSearch",source="browser"}[5m])) * 100) >= 5)`
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
</details>
monitoring: improve solutions formatting, add team information (#14878)
2020-10-20 05:11:20 +00:00
<br />
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
## frontend: page_load_latency
monitoring: migrate out-of-band alerts to the generator (#12391) Migrates all out-of-band alert rules defined by deploy-sourcegraph to the generator, such that each alert now has a panel as part of a service dashboard like all our other alerts, with a few changes and additions.
2020-07-28 02:17:56 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
<p class="subtitle">90th percentile page load latency over all routes over 10m</p>
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
monitoring: migrate out-of-band alerts to the generator (#12391) Migrates all out-of-band alert rules defined by deploy-sourcegraph to the generator, such that each alert now has a panel as part of a service dashboard like all our other alerts, with a few changes and additions.
2020-07-28 02:17:56 +00:00
monitoring(frontend): downgrade page_load_latency to warning, update next steps (#36684)
2022-06-07 14:54:29 +00:00
- <span class="badge badge-warning">warning</span> frontend: 2s+ 90th percentile page load latency over all routes over 10m
monitoring: migrate out-of-band alerts to the generator (#12391) Migrates all out-of-band alert rules defined by deploy-sourcegraph to the generator, such that each alert now has a panel as part of a service dashboard like all our other alerts, with a few changes and additions.
2020-07-28 02:17:56 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: migrate out-of-band alerts to the generator (#12391) Migrates all out-of-band alert rules defined by deploy-sourcegraph to the generator, such that each alert now has a panel as part of a service dashboard like all our other alerts, with a few changes and additions.
2020-07-28 02:17:56 +00:00
- Confirm that the Sourcegraph frontend has enough CPU/memory using the provisioning panels.
monitoring(frontend): downgrade page_load_latency to warning, update next steps (#36684)
2022-06-07 14:54:29 +00:00
- Investigate potential sources of latency by selecting Explore and modifying the `sum by(le)` section to include additional labels: for example, `sum by(le, job)` or `sum by (le, instance)`.
Docs: update links to point to new site (#60381) We have a number of docs links in the product that point to the old doc site. Method: - Search the repo for `docs.sourcegraph.com` - Exclude the `doc/` dir, all test fixtures, and `CHANGELOG.md` - For each, replace `docs.sourcegraph.com` with `sourcegraph.com/docs` - Navigate to the resulting URL ensuring it's not a dead link, updating the URL if necessary Many of the URLs updated are just comments, but since I'm doing a manual audit of each URL anyways, I felt it was worth it to update these while I was at it.
2024-02-13 00:23:47 +00:00
- Trace a request to see what the slowest part is: https://sourcegraph.com/docs/admin/observability/tracing
monitoring(frontend): downgrade page_load_latency to warning, update next steps (#36684)
2022-06-07 14:54:29 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#frontend-page-load-latency).
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
monitoring: sync dot-com alerts to generator (#12581) Co-authored-by: Stephen Gutekanst <stephen.gutekanst@gmail.com>
2020-08-04 00:40:58 +00:00
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
```json
"observability.silenceAlerts": [
monitoring(frontend): downgrade page_load_latency to warning, update next steps (#36684)
2022-06-07 14:54:29 +00:00
"warning_frontend_page_load_latency"
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
]
```
Update o11y ownership from IAM/repo-mgmt to Source (#54368) Does what it says in the title. ## Test plan - Ran `sg generate`
2023-06-28 13:49:16 +00:00
<sub>*Managed by the [Sourcegraph Source team](https://handbook.sourcegraph.com/departments/engineering/teams/source).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
monitoring: extract into a submodule (#45786) This change extracts `monitoring` into a submodule for import in `sourcegraph/controller` (https://github.com/sourcegraph/controller/pull/195) so that we can generate dashboards for Cloud instances. These steps were required: 1. Initialize a `go.mod` in `monitoring` 2. Extract `dev/sg/internal/cliutil` into `lib` to avoid illegal imports from `monitoring` 3. Add local replaces to both `sourcegraph/sourcegraph` and `monitoring` 4. `go mod tidy` on all submodules 5. Update `go generate ./monitoring` commands to use `sg`, since the `go generate` command no longer works 6. Update `grafana/build.sh`, `prometheus/build.sh` to build the submodule 7. Amend linters to check for multiple `go.mod` files and ban imports of `github.com/sourcegraph/sourcegraph` 8. Update `sg generate go` to run in directories rather than from root The only caveat is that if you use VS code, you will now need to open `monitoring` in a separate workspace or similar, like with `lib`. Co-authored-by: Joe Chen <joe@sourcegraph.com>
2022-12-19 17:49:25 +00:00
Generated query for warning alert: `max((histogram_quantile(0.9, sum by (le) (rate(src_http_request_duration_seconds_bucket{route!="blob",route!="raw",route!~"graphql.*"}[10m])))) >= 2)`
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
</details>
monitoring: improve solutions formatting, add team information (#14878)
2020-10-20 05:11:20 +00:00
<br />
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
## frontend: 99th_percentile_search_codeintel_request_duration
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
<p class="subtitle">99th percentile code-intel successful search request duration over 5m</p>
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
monitoring: dashboards reference, refresh observability docs (#16939) - New monitoring definition field, 'Interpretation' - New generated dashboards reference - Grafana panels now link to alert solutions and dashboards reference - Refreshed observability docs Co-authored-by: uwedeportivo <534011+uwedeportivo@users.noreply.github.com>
2020-12-23 02:36:41 +00:00
- <span class="badge badge-warning">warning</span> frontend: 20s+ 99th percentile code-intel successful search request duration over 5m
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
- **Get details on the exact queries that are slow** by configuring `"observability.logSlowSearches": 20,` in the site configuration and looking for `frontend` warning logs prefixed with `slow search request` for additional details.
- **Check that most repositories are indexed** by visiting https://sourcegraph.example.com/site-admin/repositories?filter=needs-index (it should show few or no results.)
- **Kubernetes:** Check CPU usage of zoekt-webserver in the indexed-search pod, consider increasing CPU limits in the `indexed-search.Deployment.yaml` if regularly hitting max CPU utilization.
- **Docker Compose:** Check CPU usage on the Zoekt Web Server dashboard, consider increasing `cpus:` of the zoekt-webserver container in `docker-compose.yml` if regularly hitting max CPU utilization.
Corrects earliar merge where alertSolutions weren't added correctly (#28699) * add some alerts that are common when symbols is le struggle * add a bit to the alert solutions * Update doc/admin/how-to/monorepo-issues.md Co-authored-by: Chris Wendt <chrismwendt@gmail.com> * forgot to go generate * correctly add to alerts Co-authored-by: Chris Wendt <chrismwendt@gmail.com>
2021-12-07 21:27:41 +00:00
- This alert may indicate that your instance is struggling to process symbols queries on a monorepo, [learn more here](../how-to/monorepo-issues.md).
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#frontend-99th-percentile-search-codeintel-request-duration).
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
```json
"observability.silenceAlerts": [
"warning_frontend_99th_percentile_search_codeintel_request_duration"
]
```
Monitoring: update owners for code insights and batches (#59313) This updates ownership for search, code insights, and batch changes. --------- Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2024-01-04 20:18:23 +00:00
<sub>*Managed by the [Sourcegraph Code Search team](https://handbook.sourcegraph.com/departments/engineering/teams/code-search).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
monitoring: extract into a submodule (#45786) This change extracts `monitoring` into a submodule for import in `sourcegraph/controller` (https://github.com/sourcegraph/controller/pull/195) so that we can generate dashboards for Cloud instances. These steps were required: 1. Initialize a `go.mod` in `monitoring` 2. Extract `dev/sg/internal/cliutil` into `lib` to avoid illegal imports from `monitoring` 3. Add local replaces to both `sourcegraph/sourcegraph` and `monitoring` 4. `go mod tidy` on all submodules 5. Update `go generate ./monitoring` commands to use `sg`, since the `go generate` command no longer works 6. Update `grafana/build.sh`, `prometheus/build.sh` to build the submodule 7. Amend linters to check for multiple `go.mod` files and ban imports of `github.com/sourcegraph/sourcegraph` 8. Update `sg generate go` to run in directories rather than from root The only caveat is that if you use VS code, you will now need to open `monitoring` in a separate workspace or similar, like with `lib`. Co-authored-by: Joe Chen <joe@sourcegraph.com>
2022-12-19 17:49:25 +00:00
Generated query for warning alert: `max((histogram_quantile(0.99, sum by (le) (rate(src_graphql_field_seconds_bucket{error="false",field="results",request_name="CodeIntelSearch",source="browser",type="Search"}[5m])))) >= 20)`
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
</details>
monitoring: improve solutions formatting, add team information (#14878)
2020-10-20 05:11:20 +00:00
<br />
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
## frontend: 90th_percentile_search_codeintel_request_duration
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
<p class="subtitle">90th percentile code-intel successful search request duration over 5m</p>
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
monitoring: dashboards reference, refresh observability docs (#16939) - New monitoring definition field, 'Interpretation' - New generated dashboards reference - Grafana panels now link to alert solutions and dashboards reference - Refreshed observability docs Co-authored-by: uwedeportivo <534011+uwedeportivo@users.noreply.github.com>
2020-12-23 02:36:41 +00:00
- <span class="badge badge-warning">warning</span> frontend: 15s+ 90th percentile code-intel successful search request duration over 5m
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
- **Get details on the exact queries that are slow** by configuring `"observability.logSlowSearches": 15,` in the site configuration and looking for `frontend` warning logs prefixed with `slow search request` for additional details.
- **Check that most repositories are indexed** by visiting https://sourcegraph.example.com/site-admin/repositories?filter=needs-index (it should show few or no results.)
- **Kubernetes:** Check CPU usage of zoekt-webserver in the indexed-search pod, consider increasing CPU limits in the `indexed-search.Deployment.yaml` if regularly hitting max CPU utilization.
- **Docker Compose:** Check CPU usage on the Zoekt Web Server dashboard, consider increasing `cpus:` of the zoekt-webserver container in `docker-compose.yml` if regularly hitting max CPU utilization.
Corrects earliar merge where alertSolutions weren't added correctly (#28699) * add some alerts that are common when symbols is le struggle * add a bit to the alert solutions * Update doc/admin/how-to/monorepo-issues.md Co-authored-by: Chris Wendt <chrismwendt@gmail.com> * forgot to go generate * correctly add to alerts Co-authored-by: Chris Wendt <chrismwendt@gmail.com>
2021-12-07 21:27:41 +00:00
- This alert may indicate that your instance is struggling to process symbols queries on a monorepo, [learn more here](../how-to/monorepo-issues.md).
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#frontend-90th-percentile-search-codeintel-request-duration).
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"warning_frontend_90th_percentile_search_codeintel_request_duration"
]
```
Monitoring: update owners for code insights and batches (#59313) This updates ownership for search, code insights, and batch changes. --------- Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2024-01-04 20:18:23 +00:00
<sub>*Managed by the [Sourcegraph Code Search team](https://handbook.sourcegraph.com/departments/engineering/teams/code-search).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
monitoring: extract into a submodule (#45786) This change extracts `monitoring` into a submodule for import in `sourcegraph/controller` (https://github.com/sourcegraph/controller/pull/195) so that we can generate dashboards for Cloud instances. These steps were required: 1. Initialize a `go.mod` in `monitoring` 2. Extract `dev/sg/internal/cliutil` into `lib` to avoid illegal imports from `monitoring` 3. Add local replaces to both `sourcegraph/sourcegraph` and `monitoring` 4. `go mod tidy` on all submodules 5. Update `go generate ./monitoring` commands to use `sg`, since the `go generate` command no longer works 6. Update `grafana/build.sh`, `prometheus/build.sh` to build the submodule 7. Amend linters to check for multiple `go.mod` files and ban imports of `github.com/sourcegraph/sourcegraph` 8. Update `sg generate go` to run in directories rather than from root The only caveat is that if you use VS code, you will now need to open `monitoring` in a separate workspace or similar, like with `lib`. Co-authored-by: Joe Chen <joe@sourcegraph.com>
2022-12-19 17:49:25 +00:00
Generated query for warning alert: `max((histogram_quantile(0.9, sum by (le) (rate(src_graphql_field_seconds_bucket{error="false",field="results",request_name="CodeIntelSearch",source="browser",type="Search"}[5m])))) >= 15)`
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
</details>
monitoring: improve solutions formatting, add team information (#14878)
2020-10-20 05:11:20 +00:00
<br />
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
## frontend: hard_timeout_search_codeintel_responses
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
<p class="subtitle">hard timeout search code-intel responses every 5m</p>
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
monitoring: dashboards reference, refresh observability docs (#16939) - New monitoring definition field, 'Interpretation' - New generated dashboards reference - Grafana panels now link to alert solutions and dashboards reference - Refreshed observability docs Co-authored-by: uwedeportivo <534011+uwedeportivo@users.noreply.github.com>
2020-12-23 02:36:41 +00:00
- <span class="badge badge-warning">warning</span> frontend: 2%+ hard timeout search code-intel responses every 5m for 15m0s
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#frontend-hard-timeout-search-codeintel-responses).
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
monitoring: forbid critical alerts with no solutions (#36423) Adds the restriction that critical alerts must have solutions defined as a hard requirement within the generator, and fixes all the outstanding issues: - Add PossibleSolutions for the pods_available_percentage and pg_up critical alerts (we have a similar solution doc in the container_missing alert) - Removes all the other critical alerts without possible solutions Co-authored-by: Michael Lin <mlzc@hey.com>
2022-06-02 15:51:43 +00:00
"warning_frontend_hard_timeout_search_codeintel_responses"
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
]
```
Monitoring: update owners for code insights and batches (#59313) This updates ownership for search, code insights, and batch changes. --------- Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2024-01-04 20:18:23 +00:00
<sub>*Managed by the [Sourcegraph Code Search team](https://handbook.sourcegraph.com/departments/engineering/teams/code-search).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
Generated query for warning alert: `max(((sum(increase(src_graphql_search_response{request_name="CodeIntelSearch",source="browser",status="timeout"}[5m])) + sum(increase(src_graphql_search_response{alert_type="timed_out",request_name="CodeIntelSearch",source="browser",status="alert"}[5m]))) / sum(increase(src_graphql_search_response{request_name="CodeIntelSearch",source="browser"}[5m])) * 100) >= 2)`
</details>
monitoring: improve solutions formatting, add team information (#14878)
2020-10-20 05:11:20 +00:00
<br />
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
## frontend: hard_error_search_codeintel_responses
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
<p class="subtitle">hard error search code-intel responses every 5m</p>
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
monitoring: dashboards reference, refresh observability docs (#16939) - New monitoring definition field, 'Interpretation' - New generated dashboards reference - Grafana panels now link to alert solutions and dashboards reference - Refreshed observability docs Co-authored-by: uwedeportivo <534011+uwedeportivo@users.noreply.github.com>
2020-12-23 02:36:41 +00:00
- <span class="badge badge-warning">warning</span> frontend: 2%+ hard error search code-intel responses every 5m for 15m0s
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#frontend-hard-error-search-codeintel-responses).
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
monitoring: forbid critical alerts with no solutions (#36423) Adds the restriction that critical alerts must have solutions defined as a hard requirement within the generator, and fixes all the outstanding issues: - Add PossibleSolutions for the pods_available_percentage and pg_up critical alerts (we have a similar solution doc in the container_missing alert) - Removes all the other critical alerts without possible solutions Co-authored-by: Michael Lin <mlzc@hey.com>
2022-06-02 15:51:43 +00:00
"warning_frontend_hard_error_search_codeintel_responses"
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
]
```
Monitoring: update owners for code insights and batches (#59313) This updates ownership for search, code insights, and batch changes. --------- Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2024-01-04 20:18:23 +00:00
<sub>*Managed by the [Sourcegraph Code Search team](https://handbook.sourcegraph.com/departments/engineering/teams/code-search).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
monitoring: extract into a submodule (#45786) This change extracts `monitoring` into a submodule for import in `sourcegraph/controller` (https://github.com/sourcegraph/controller/pull/195) so that we can generate dashboards for Cloud instances. These steps were required: 1. Initialize a `go.mod` in `monitoring` 2. Extract `dev/sg/internal/cliutil` into `lib` to avoid illegal imports from `monitoring` 3. Add local replaces to both `sourcegraph/sourcegraph` and `monitoring` 4. `go mod tidy` on all submodules 5. Update `go generate ./monitoring` commands to use `sg`, since the `go generate` command no longer works 6. Update `grafana/build.sh`, `prometheus/build.sh` to build the submodule 7. Amend linters to check for multiple `go.mod` files and ban imports of `github.com/sourcegraph/sourcegraph` 8. Update `sg generate go` to run in directories rather than from root The only caveat is that if you use VS code, you will now need to open `monitoring` in a separate workspace or similar, like with `lib`. Co-authored-by: Joe Chen <joe@sourcegraph.com>
2022-12-19 17:49:25 +00:00
Generated query for warning alert: `max((sum by (status) (increase(src_graphql_search_response{request_name="CodeIntelSearch",source="browser",status=~"error"}[5m])) / ignoring (status) group_left () sum(increase(src_graphql_search_response{request_name="CodeIntelSearch",source="browser"}[5m])) * 100) >= 2)`
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
</details>
monitoring: improve solutions formatting, add team information (#14878)
2020-10-20 05:11:20 +00:00
<br />
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
## frontend: partial_timeout_search_codeintel_responses
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
<p class="subtitle">partial timeout search code-intel responses every 5m</p>
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
monitoring: dashboards reference, refresh observability docs (#16939) - New monitoring definition field, 'Interpretation' - New generated dashboards reference - Grafana panels now link to alert solutions and dashboards reference - Refreshed observability docs Co-authored-by: uwedeportivo <534011+uwedeportivo@users.noreply.github.com>
2020-12-23 02:36:41 +00:00
- <span class="badge badge-warning">warning</span> frontend: 5%+ partial timeout search code-intel responses every 5m for 15m0s
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#frontend-partial-timeout-search-codeintel-responses).
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"warning_frontend_partial_timeout_search_codeintel_responses"
]
```
Monitoring: update owners for code insights and batches (#59313) This updates ownership for search, code insights, and batch changes. --------- Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2024-01-04 20:18:23 +00:00
<sub>*Managed by the [Sourcegraph Code Search team](https://handbook.sourcegraph.com/departments/engineering/teams/code-search).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
monitoring: extract into a submodule (#45786) This change extracts `monitoring` into a submodule for import in `sourcegraph/controller` (https://github.com/sourcegraph/controller/pull/195) so that we can generate dashboards for Cloud instances. These steps were required: 1. Initialize a `go.mod` in `monitoring` 2. Extract `dev/sg/internal/cliutil` into `lib` to avoid illegal imports from `monitoring` 3. Add local replaces to both `sourcegraph/sourcegraph` and `monitoring` 4. `go mod tidy` on all submodules 5. Update `go generate ./monitoring` commands to use `sg`, since the `go generate` command no longer works 6. Update `grafana/build.sh`, `prometheus/build.sh` to build the submodule 7. Amend linters to check for multiple `go.mod` files and ban imports of `github.com/sourcegraph/sourcegraph` 8. Update `sg generate go` to run in directories rather than from root The only caveat is that if you use VS code, you will now need to open `monitoring` in a separate workspace or similar, like with `lib`. Co-authored-by: Joe Chen <joe@sourcegraph.com>
2022-12-19 17:49:25 +00:00
Generated query for warning alert: `max((sum by (status) (increase(src_graphql_search_response{request_name="CodeIntelSearch",source="browser",status="partial_timeout"}[5m])) / ignoring (status) group_left () sum(increase(src_graphql_search_response{request_name="CodeIntelSearch",source="browser",status="partial_timeout"}[5m])) * 100) >= 5)`
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
</details>
monitoring: improve solutions formatting, add team information (#14878)
2020-10-20 05:11:20 +00:00
<br />
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
## frontend: search_codeintel_alert_user_suggestions
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
<p class="subtitle">search code-intel alert user suggestions shown every 5m</p>
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
monitoring: dashboards reference, refresh observability docs (#16939) - New monitoring definition field, 'Interpretation' - New generated dashboards reference - Grafana panels now link to alert solutions and dashboards reference - Refreshed observability docs Co-authored-by: uwedeportivo <534011+uwedeportivo@users.noreply.github.com>
2020-12-23 02:36:41 +00:00
- <span class="badge badge-warning">warning</span> frontend: 5%+ search code-intel alert user suggestions shown every 5m for 15m0s
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
- This indicates a bug in Sourcegraph, please [open an issue](https://github.com/sourcegraph/sourcegraph/issues/new/choose).
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#frontend-search-codeintel-alert-user-suggestions).
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
```json
"observability.silenceAlerts": [
"warning_frontend_search_codeintel_alert_user_suggestions"
]
```
Monitoring: update owners for code insights and batches (#59313) This updates ownership for search, code insights, and batch changes. --------- Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2024-01-04 20:18:23 +00:00
<sub>*Managed by the [Sourcegraph Code Search team](https://handbook.sourcegraph.com/departments/engineering/teams/code-search).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
monitoring: extract into a submodule (#45786) This change extracts `monitoring` into a submodule for import in `sourcegraph/controller` (https://github.com/sourcegraph/controller/pull/195) so that we can generate dashboards for Cloud instances. These steps were required: 1. Initialize a `go.mod` in `monitoring` 2. Extract `dev/sg/internal/cliutil` into `lib` to avoid illegal imports from `monitoring` 3. Add local replaces to both `sourcegraph/sourcegraph` and `monitoring` 4. `go mod tidy` on all submodules 5. Update `go generate ./monitoring` commands to use `sg`, since the `go generate` command no longer works 6. Update `grafana/build.sh`, `prometheus/build.sh` to build the submodule 7. Amend linters to check for multiple `go.mod` files and ban imports of `github.com/sourcegraph/sourcegraph` 8. Update `sg generate go` to run in directories rather than from root The only caveat is that if you use VS code, you will now need to open `monitoring` in a separate workspace or similar, like with `lib`. Co-authored-by: Joe Chen <joe@sourcegraph.com>
2022-12-19 17:49:25 +00:00
Generated query for warning alert: `max((sum by (alert_type) (increase(src_graphql_search_response{alert_type!~"timed_out",request_name="CodeIntelSearch",source="browser",status="alert"}[5m])) / ignoring (alert_type) group_left () sum(increase(src_graphql_search_response{request_name="CodeIntelSearch",source="browser"}[5m])) * 100) >= 5)`
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
</details>
monitoring: improve solutions formatting, add team information (#14878)
2020-10-20 05:11:20 +00:00
<br />
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
## frontend: 99th_percentile_search_api_request_duration
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
<p class="subtitle">99th percentile successful search API request duration over 5m</p>
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
monitoring: dashboards reference, refresh observability docs (#16939) - New monitoring definition field, 'Interpretation' - New generated dashboards reference - Grafana panels now link to alert solutions and dashboards reference - Refreshed observability docs Co-authored-by: uwedeportivo <534011+uwedeportivo@users.noreply.github.com>
2020-12-23 02:36:41 +00:00
- <span class="badge badge-warning">warning</span> frontend: 50s+ 99th percentile successful search API request duration over 5m
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
- **Get details on the exact queries that are slow** by configuring `"observability.logSlowSearches": 20,` in the site configuration and looking for `frontend` warning logs prefixed with `slow search request` for additional details.
- **Check that most repositories are indexed** by visiting https://sourcegraph.example.com/site-admin/repositories?filter=needs-index (it should show few or no results.)
- **Kubernetes:** Check CPU usage of zoekt-webserver in the indexed-search pod, consider increasing CPU limits in the `indexed-search.Deployment.yaml` if regularly hitting max CPU utilization.
- **Docker Compose:** Check CPU usage on the Zoekt Web Server dashboard, consider increasing `cpus:` of the zoekt-webserver container in `docker-compose.yml` if regularly hitting max CPU utilization.
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#frontend-99th-percentile-search-api-request-duration).
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"warning_frontend_99th_percentile_search_api_request_duration"
]
```
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
Monitoring: update owners for code insights and batches (#59313) This updates ownership for search, code insights, and batch changes. --------- Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2024-01-04 20:18:23 +00:00
<sub>*Managed by the [Sourcegraph Code Search team](https://handbook.sourcegraph.com/departments/engineering/teams/code-search).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
monitoring: extract into a submodule (#45786) This change extracts `monitoring` into a submodule for import in `sourcegraph/controller` (https://github.com/sourcegraph/controller/pull/195) so that we can generate dashboards for Cloud instances. These steps were required: 1. Initialize a `go.mod` in `monitoring` 2. Extract `dev/sg/internal/cliutil` into `lib` to avoid illegal imports from `monitoring` 3. Add local replaces to both `sourcegraph/sourcegraph` and `monitoring` 4. `go mod tidy` on all submodules 5. Update `go generate ./monitoring` commands to use `sg`, since the `go generate` command no longer works 6. Update `grafana/build.sh`, `prometheus/build.sh` to build the submodule 7. Amend linters to check for multiple `go.mod` files and ban imports of `github.com/sourcegraph/sourcegraph` 8. Update `sg generate go` to run in directories rather than from root The only caveat is that if you use VS code, you will now need to open `monitoring` in a separate workspace or similar, like with `lib`. Co-authored-by: Joe Chen <joe@sourcegraph.com>
2022-12-19 17:49:25 +00:00
Generated query for warning alert: `max((histogram_quantile(0.99, sum by (le) (rate(src_graphql_field_seconds_bucket{error="false",field="results",source="other",type="Search"}[5m])))) >= 50)`
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
</details>
monitoring: improve solutions formatting, add team information (#14878)
2020-10-20 05:11:20 +00:00
<br />
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
## frontend: 90th_percentile_search_api_request_duration
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
<p class="subtitle">90th percentile successful search API request duration over 5m</p>
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
monitoring: dashboards reference, refresh observability docs (#16939) - New monitoring definition field, 'Interpretation' - New generated dashboards reference - Grafana panels now link to alert solutions and dashboards reference - Refreshed observability docs Co-authored-by: uwedeportivo <534011+uwedeportivo@users.noreply.github.com>
2020-12-23 02:36:41 +00:00
- <span class="badge badge-warning">warning</span> frontend: 40s+ 90th percentile successful search API request duration over 5m
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
- **Get details on the exact queries that are slow** by configuring `"observability.logSlowSearches": 15,` in the site configuration and looking for `frontend` warning logs prefixed with `slow search request` for additional details.
- **Check that most repositories are indexed** by visiting https://sourcegraph.example.com/site-admin/repositories?filter=needs-index (it should show few or no results.)
- **Kubernetes:** Check CPU usage of zoekt-webserver in the indexed-search pod, consider increasing CPU limits in the `indexed-search.Deployment.yaml` if regularly hitting max CPU utilization.
- **Docker Compose:** Check CPU usage on the Zoekt Web Server dashboard, consider increasing `cpus:` of the zoekt-webserver container in `docker-compose.yml` if regularly hitting max CPU utilization.
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#frontend-90th-percentile-search-api-request-duration).
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"warning_frontend_90th_percentile_search_api_request_duration"
]
```
Monitoring: update owners for code insights and batches (#59313) This updates ownership for search, code insights, and batch changes. --------- Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2024-01-04 20:18:23 +00:00
<sub>*Managed by the [Sourcegraph Code Search team](https://handbook.sourcegraph.com/departments/engineering/teams/code-search).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
monitoring: extract into a submodule (#45786) This change extracts `monitoring` into a submodule for import in `sourcegraph/controller` (https://github.com/sourcegraph/controller/pull/195) so that we can generate dashboards for Cloud instances. These steps were required: 1. Initialize a `go.mod` in `monitoring` 2. Extract `dev/sg/internal/cliutil` into `lib` to avoid illegal imports from `monitoring` 3. Add local replaces to both `sourcegraph/sourcegraph` and `monitoring` 4. `go mod tidy` on all submodules 5. Update `go generate ./monitoring` commands to use `sg`, since the `go generate` command no longer works 6. Update `grafana/build.sh`, `prometheus/build.sh` to build the submodule 7. Amend linters to check for multiple `go.mod` files and ban imports of `github.com/sourcegraph/sourcegraph` 8. Update `sg generate go` to run in directories rather than from root The only caveat is that if you use VS code, you will now need to open `monitoring` in a separate workspace or similar, like with `lib`. Co-authored-by: Joe Chen <joe@sourcegraph.com>
2022-12-19 17:49:25 +00:00
Generated query for warning alert: `max((histogram_quantile(0.9, sum by (le) (rate(src_graphql_field_seconds_bucket{error="false",field="results",source="other",type="Search"}[5m])))) >= 40)`
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
</details>
monitoring: improve solutions formatting, add team information (#14878)
2020-10-20 05:11:20 +00:00
<br />
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
## frontend: hard_error_search_api_responses
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
<p class="subtitle">hard error search API responses every 5m</p>
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
monitoring: dashboards reference, refresh observability docs (#16939) - New monitoring definition field, 'Interpretation' - New generated dashboards reference - Grafana panels now link to alert solutions and dashboards reference - Refreshed observability docs Co-authored-by: uwedeportivo <534011+uwedeportivo@users.noreply.github.com>
2020-12-23 02:36:41 +00:00
- <span class="badge badge-warning">warning</span> frontend: 2%+ hard error search API responses every 5m for 15m0s
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#frontend-hard-error-search-api-responses).
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
monitoring: forbid critical alerts with no solutions (#36423) Adds the restriction that critical alerts must have solutions defined as a hard requirement within the generator, and fixes all the outstanding issues: - Add PossibleSolutions for the pods_available_percentage and pg_up critical alerts (we have a similar solution doc in the container_missing alert) - Removes all the other critical alerts without possible solutions Co-authored-by: Michael Lin <mlzc@hey.com>
2022-06-02 15:51:43 +00:00
"warning_frontend_hard_error_search_api_responses"
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
]
```
Monitoring: update owners for code insights and batches (#59313) This updates ownership for search, code insights, and batch changes. --------- Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2024-01-04 20:18:23 +00:00
<sub>*Managed by the [Sourcegraph Code Search team](https://handbook.sourcegraph.com/departments/engineering/teams/code-search).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
monitoring: extract into a submodule (#45786) This change extracts `monitoring` into a submodule for import in `sourcegraph/controller` (https://github.com/sourcegraph/controller/pull/195) so that we can generate dashboards for Cloud instances. These steps were required: 1. Initialize a `go.mod` in `monitoring` 2. Extract `dev/sg/internal/cliutil` into `lib` to avoid illegal imports from `monitoring` 3. Add local replaces to both `sourcegraph/sourcegraph` and `monitoring` 4. `go mod tidy` on all submodules 5. Update `go generate ./monitoring` commands to use `sg`, since the `go generate` command no longer works 6. Update `grafana/build.sh`, `prometheus/build.sh` to build the submodule 7. Amend linters to check for multiple `go.mod` files and ban imports of `github.com/sourcegraph/sourcegraph` 8. Update `sg generate go` to run in directories rather than from root The only caveat is that if you use VS code, you will now need to open `monitoring` in a separate workspace or similar, like with `lib`. Co-authored-by: Joe Chen <joe@sourcegraph.com>
2022-12-19 17:49:25 +00:00
Generated query for warning alert: `max((sum by (status) (increase(src_graphql_search_response{source="other",status=~"error"}[5m])) / ignoring (status) group_left () sum(increase(src_graphql_search_response{source="other"}[5m]))) >= 2)`
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
</details>
monitoring: improve solutions formatting, add team information (#14878)
2020-10-20 05:11:20 +00:00
<br />
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
## frontend: partial_timeout_search_api_responses
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
<p class="subtitle">partial timeout search API responses every 5m</p>
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
monitoring: dashboards reference, refresh observability docs (#16939) - New monitoring definition field, 'Interpretation' - New generated dashboards reference - Grafana panels now link to alert solutions and dashboards reference - Refreshed observability docs Co-authored-by: uwedeportivo <534011+uwedeportivo@users.noreply.github.com>
2020-12-23 02:36:41 +00:00
- <span class="badge badge-warning">warning</span> frontend: 5%+ partial timeout search API responses every 5m for 15m0s
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#frontend-partial-timeout-search-api-responses).
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"warning_frontend_partial_timeout_search_api_responses"
]
```
Monitoring: update owners for code insights and batches (#59313) This updates ownership for search, code insights, and batch changes. --------- Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2024-01-04 20:18:23 +00:00
<sub>*Managed by the [Sourcegraph Code Search team](https://handbook.sourcegraph.com/departments/engineering/teams/code-search).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
Generated query for warning alert: `max((sum(increase(src_graphql_search_response{source="other",status="partial_timeout"}[5m])) / sum(increase(src_graphql_search_response{source="other"}[5m]))) >= 5)`
</details>
monitoring: improve solutions formatting, add team information (#14878)
2020-10-20 05:11:20 +00:00
<br />
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
## frontend: search_api_alert_user_suggestions
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
<p class="subtitle">search API alert user suggestions shown every 5m</p>
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
monitoring: dashboards reference, refresh observability docs (#16939) - New monitoring definition field, 'Interpretation' - New generated dashboards reference - Grafana panels now link to alert solutions and dashboards reference - Refreshed observability docs Co-authored-by: uwedeportivo <534011+uwedeportivo@users.noreply.github.com>
2020-12-23 02:36:41 +00:00
- <span class="badge badge-warning">warning</span> frontend: 5%+ search API alert user suggestions shown every 5m
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
- This indicates your user`s search API requests have syntax errors or a similar user error. Check the responses the API sends back for an explanation.
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#frontend-search-api-alert-user-suggestions).
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"warning_frontend_search_api_alert_user_suggestions"
]
```
Monitoring: update owners for code insights and batches (#59313) This updates ownership for search, code insights, and batch changes. --------- Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2024-01-04 20:18:23 +00:00
<sub>*Managed by the [Sourcegraph Code Search team](https://handbook.sourcegraph.com/departments/engineering/teams/code-search).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
monitoring: extract into a submodule (#45786) This change extracts `monitoring` into a submodule for import in `sourcegraph/controller` (https://github.com/sourcegraph/controller/pull/195) so that we can generate dashboards for Cloud instances. These steps were required: 1. Initialize a `go.mod` in `monitoring` 2. Extract `dev/sg/internal/cliutil` into `lib` to avoid illegal imports from `monitoring` 3. Add local replaces to both `sourcegraph/sourcegraph` and `monitoring` 4. `go mod tidy` on all submodules 5. Update `go generate ./monitoring` commands to use `sg`, since the `go generate` command no longer works 6. Update `grafana/build.sh`, `prometheus/build.sh` to build the submodule 7. Amend linters to check for multiple `go.mod` files and ban imports of `github.com/sourcegraph/sourcegraph` 8. Update `sg generate go` to run in directories rather than from root The only caveat is that if you use VS code, you will now need to open `monitoring` in a separate workspace or similar, like with `lib`. Co-authored-by: Joe Chen <joe@sourcegraph.com>
2022-12-19 17:49:25 +00:00
Generated query for warning alert: `max((sum by (alert_type) (increase(src_graphql_search_response{alert_type!~"timed_out|no_results__suggest_quotes",source="other",status="alert"}[5m])) / ignoring (alert_type) group_left () sum(increase(src_graphql_search_response{source="other",status="alert"}[5m]))) >= 5)`
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
</details>
codeintel: Add additional metrics around resolver layer (#15825)
2020-11-17 20:28:42 +00:00
<br />
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
conf: add metric and associated alert if clients fail to update site configuration within 5 minutes (#57682)
2023-10-18 23:53:55 +00:00
## frontend: frontend_site_configuration_duration_since_last_successful_update_by_instance
<p class="subtitle">maximum duration since last successful site configuration update (all "frontend" instances)</p>
**Descriptions**
- <span class="badge badge-critical">critical</span> frontend: 300s+ maximum duration since last successful site configuration update (all "frontend" instances)
**Next steps**
- This indicates that one or more "frontend" instances have not successfully updated the site configuration in over 5 minutes. This could be due to networking issues between services or problems with the site configuration service itself.
- Check for relevant errors in the "frontend" logs, as well as frontend`s logs.
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#frontend-frontend-site-configuration-duration-since-last-successful-update-by-instance).
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"critical_frontend_frontend_site_configuration_duration_since_last_successful_update_by_instance"
]
```
monitoring: test owners for valid Opsgenie teams and handbook pages (#59251) In INC-264 it seems that certain alerts - such as [zoekt: less than 90% percentage pods available for 10m0s](https://opsg.in/a/i/sourcegraph/178a626f-0f28-4295-bee9-84da988bb473-1703759057681) - don't seem to end up going anywhere because the ObservableOwner is defunct. This change adds _opt-in_ testing to report: 1. How many owners have valid Opsgenie teams 2. How many owners have valid handbook pages In addition, we collect ObservableOwners that pass the test and use it to generate configuration for `site.json` in Sourcegraph.com: https://github.com/sourcegraph/deploy-sourcegraph-cloud/pull/18338 - this helps ensure the list is valid and not deceptively high-coverage. The results are not great, but **enforcing** that owners are valid isn't currently in scope: ``` 6/10 ObservableOwners do not have valid Opsgenie teams 3/10 ObservableOwners do not point to valid handbook pages ``` I also removed some defunct/unused functionality/owners. ## Test plan To run these tests: ``` export OPSGENIE_API_KEY="..." go test -timeout 30s github.com/sourcegraph/sourcegraph/monitoring/monitoring -update -online ```
2023-12-29 22:07:35 +00:00
<sub>*Managed by the [Sourcegraph Infrastructure Org team](https://handbook.sourcegraph.com/departments/engineering/infrastructure).*</sub>
conf: add metric and associated alert if clients fail to update site configuration within 5 minutes (#57682)
2023-10-18 23:53:55 +00:00
<details>
<summary>Technical details</summary>
monitoring: fix alert definition for site configuration by adding scrape job label (#59687) We discovered recently that the definition for the alert that fires if the site configuration hasn't been fetched within 5 minutes strips out the regex that targets individual services (since it uses a grafana variable). This means that every instance of this alert will fire if any individual service trips over this threshold. This PR fixes the issue by adding a new `job` filter for this alert that targets only the services that have that Prometheus scrape target name. This works around the previous issue by using a fixed value for the `job` value instead of a dynamic grafana value. The value of the job filter generally looks like `job=~.*$container_name` (following the strategy from https://sourcegraph.com/github.com/sourcegraph/sourcegraph@9a780f2e694238b5326e3e121d6a1828463001b9/-/blob/monitoring/monitoring/monitoring.go?L161 ) unless I noticed that there was different logic in the existing dashboard for the services. Ex: - `frontend`: already used `job=~"(sourcegraph-)?frontend"` for some metrics, so I used it again here - `worker`: `already used `job=~"^worker.*"` in some metrics, so I used it again and standarized the other existing panels to use the same shared variable ## Test plan I eyeballed the generated alert.md and dashboards.md to verify that my changes looked correct (that is, my refactors resulted in either no diff, or that the diff I generated still looked like valid regex).
2024-01-17 23:19:54 +00:00
Generated query for critical alert: `max((max(max_over_time(src_conf_client_time_since_last_successful_update_seconds{job=~"(sourcegraph-)?frontend"}[1m]))) >= 300)`
conf: add metric and associated alert if clients fail to update site configuration within 5 minutes (#57682)
2023-10-18 23:53:55 +00:00
</details>
<br />
monitoring: Update dashboards (#23091)
2021-07-25 16:25:33 +00:00
## frontend: internal_indexed_search_error_responses
codeintel: Add additional metrics around resolver layer (#15825)
2020-11-17 20:28:42 +00:00
monitoring: Update dashboards (#23091)
2021-07-25 16:25:33 +00:00
<p class="subtitle">internal indexed search error responses every 5m</p>
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
codeintel: Add additional metrics around resolver layer (#15825)
2020-11-17 20:28:42 +00:00
monitoring: Update dashboards (#23091)
2021-07-25 16:25:33 +00:00
- <span class="badge badge-warning">warning</span> frontend: 5%+ internal indexed search error responses every 5m for 15m0s
codeintel: Add additional metrics around resolver layer (#15825)
2020-11-17 20:28:42 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
codeintel: Add additional metrics around resolver layer (#15825)
2020-11-17 20:28:42 +00:00
monitoring: Update dashboards (#23091)
2021-07-25 16:25:33 +00:00
- Check the Zoekt Web Server dashboard for indications it might be unhealthy.
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#frontend-internal-indexed-search-error-responses).
codeintel: Add additional metrics around resolver layer (#15825)
2020-11-17 20:28:42 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
monitoring: Update dashboards (#23091)
2021-07-25 16:25:33 +00:00
"warning_frontend_internal_indexed_search_error_responses"
codeintel: Add additional metrics around resolver layer (#15825)
2020-11-17 20:28:42 +00:00
]
```
Monitoring: update owners for code insights and batches (#59313) This updates ownership for search, code insights, and batch changes. --------- Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2024-01-04 20:18:23 +00:00
<sub>*Managed by the [Sourcegraph Code Search team](https://handbook.sourcegraph.com/departments/engineering/teams/code-search).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
monitoring: extract into a submodule (#45786) This change extracts `monitoring` into a submodule for import in `sourcegraph/controller` (https://github.com/sourcegraph/controller/pull/195) so that we can generate dashboards for Cloud instances. These steps were required: 1. Initialize a `go.mod` in `monitoring` 2. Extract `dev/sg/internal/cliutil` into `lib` to avoid illegal imports from `monitoring` 3. Add local replaces to both `sourcegraph/sourcegraph` and `monitoring` 4. `go mod tidy` on all submodules 5. Update `go generate ./monitoring` commands to use `sg`, since the `go generate` command no longer works 6. Update `grafana/build.sh`, `prometheus/build.sh` to build the submodule 7. Amend linters to check for multiple `go.mod` files and ban imports of `github.com/sourcegraph/sourcegraph` 8. Update `sg generate go` to run in directories rather than from root The only caveat is that if you use VS code, you will now need to open `monitoring` in a separate workspace or similar, like with `lib`. Co-authored-by: Joe Chen <joe@sourcegraph.com>
2022-12-19 17:49:25 +00:00
Generated query for warning alert: `max((sum by (code) (increase(src_zoekt_request_duration_seconds_count{code!~"2.."}[5m])) / ignoring (code) group_left () sum(increase(src_zoekt_request_duration_seconds_count[5m])) * 100) >= 5)`
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
</details>
codeintel: Add additional metrics around resolver layer (#15825)
2020-11-17 20:28:42 +00:00
<br />
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: Update dashboards (#23091)
2021-07-25 16:25:33 +00:00
## frontend: internal_unindexed_search_error_responses
codeintel: Add additional metrics around resolver layer (#15825)
2020-11-17 20:28:42 +00:00
monitoring: Update dashboards (#23091)
2021-07-25 16:25:33 +00:00
<p class="subtitle">internal unindexed search error responses every 5m</p>
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
codeintel: Add additional metrics around resolver layer (#15825)
2020-11-17 20:28:42 +00:00
monitoring: Update dashboards (#23091)
2021-07-25 16:25:33 +00:00
- <span class="badge badge-warning">warning</span> frontend: 5%+ internal unindexed search error responses every 5m for 15m0s
codeintel: Add additional metrics around resolver layer (#15825)
2020-11-17 20:28:42 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
codeintel: Add additional metrics around resolver layer (#15825)
2020-11-17 20:28:42 +00:00
monitoring: Update dashboards (#23091)
2021-07-25 16:25:33 +00:00
- Check the Searcher dashboard for indications it might be unhealthy.
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#frontend-internal-unindexed-search-error-responses).
codeintel: Add additional metrics around resolver layer (#15825)
2020-11-17 20:28:42 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
monitoring: Update dashboards (#23091)
2021-07-25 16:25:33 +00:00
"warning_frontend_internal_unindexed_search_error_responses"
codeintel: Add additional metrics around resolver layer (#15825)
2020-11-17 20:28:42 +00:00
]
```
Monitoring: update owners for code insights and batches (#59313) This updates ownership for search, code insights, and batch changes. --------- Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2024-01-04 20:18:23 +00:00
<sub>*Managed by the [Sourcegraph Code Search team](https://handbook.sourcegraph.com/departments/engineering/teams/code-search).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
monitoring: extract into a submodule (#45786) This change extracts `monitoring` into a submodule for import in `sourcegraph/controller` (https://github.com/sourcegraph/controller/pull/195) so that we can generate dashboards for Cloud instances. These steps were required: 1. Initialize a `go.mod` in `monitoring` 2. Extract `dev/sg/internal/cliutil` into `lib` to avoid illegal imports from `monitoring` 3. Add local replaces to both `sourcegraph/sourcegraph` and `monitoring` 4. `go mod tidy` on all submodules 5. Update `go generate ./monitoring` commands to use `sg`, since the `go generate` command no longer works 6. Update `grafana/build.sh`, `prometheus/build.sh` to build the submodule 7. Amend linters to check for multiple `go.mod` files and ban imports of `github.com/sourcegraph/sourcegraph` 8. Update `sg generate go` to run in directories rather than from root The only caveat is that if you use VS code, you will now need to open `monitoring` in a separate workspace or similar, like with `lib`. Co-authored-by: Joe Chen <joe@sourcegraph.com>
2022-12-19 17:49:25 +00:00
Generated query for warning alert: `max((sum by (code) (increase(searcher_service_request_total{code!~"2.."}[5m])) / ignoring (code) group_left () sum(increase(searcher_service_request_total[5m])) * 100) >= 5)`
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
</details>
monitoring: improve solutions formatting, add team information (#14878)
2020-10-20 05:11:20 +00:00
<br />
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: Update dashboards (#23091)
2021-07-25 16:25:33 +00:00
## frontend: 99th_percentile_gitserver_duration
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
monitoring: Update dashboards (#23091)
2021-07-25 16:25:33 +00:00
<p class="subtitle">99th percentile successful gitserver query duration over 5m</p>
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
monitoring: Update dashboards (#23091)
2021-07-25 16:25:33 +00:00
- <span class="badge badge-warning">warning</span> frontend: 20s+ 99th percentile successful gitserver query duration over 5m
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#frontend-99th-percentile-gitserver-duration).
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
monitoring: Update dashboards (#23091)
2021-07-25 16:25:33 +00:00
"warning_frontend_99th_percentile_gitserver_duration"
codeintel: Update monitoring (#14695)
2020-10-14 14:19:15 +00:00
]
```
Update o11y ownership from IAM/repo-mgmt to Source (#54368) Does what it says in the title. ## Test plan - Ran `sg generate`
2023-06-28 13:49:16 +00:00
<sub>*Managed by the [Sourcegraph Source team](https://handbook.sourcegraph.com/departments/engineering/teams/source).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
monitoring: extract into a submodule (#45786) This change extracts `monitoring` into a submodule for import in `sourcegraph/controller` (https://github.com/sourcegraph/controller/pull/195) so that we can generate dashboards for Cloud instances. These steps were required: 1. Initialize a `go.mod` in `monitoring` 2. Extract `dev/sg/internal/cliutil` into `lib` to avoid illegal imports from `monitoring` 3. Add local replaces to both `sourcegraph/sourcegraph` and `monitoring` 4. `go mod tidy` on all submodules 5. Update `go generate ./monitoring` commands to use `sg`, since the `go generate` command no longer works 6. Update `grafana/build.sh`, `prometheus/build.sh` to build the submodule 7. Amend linters to check for multiple `go.mod` files and ban imports of `github.com/sourcegraph/sourcegraph` 8. Update `sg generate go` to run in directories rather than from root The only caveat is that if you use VS code, you will now need to open `monitoring` in a separate workspace or similar, like with `lib`. Co-authored-by: Joe Chen <joe@sourcegraph.com>
2022-12-19 17:49:25 +00:00
Generated query for warning alert: `max((histogram_quantile(0.99, sum by (le, category) (rate(src_gitserver_request_duration_seconds_bucket{job=~"(sourcegraph-)?frontend"}[5m])))) >= 20)`
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
</details>
monitoring: improve solutions formatting, add team information (#14878)
2020-10-20 05:11:20 +00:00
<br />
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: Update dashboards (#23091)
2021-07-25 16:25:33 +00:00
## frontend: gitserver_error_responses
codeintel: Update metrics dashboards (#16806)
2020-12-18 17:57:28 +00:00
monitoring: Update dashboards (#23091)
2021-07-25 16:25:33 +00:00
<p class="subtitle">gitserver error responses every 5m</p>
codeintel: Update metrics dashboards (#16806)
2020-12-18 17:57:28 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
codeintel: Update metrics dashboards (#16806)
2020-12-18 17:57:28 +00:00
monitoring: Update dashboards (#23091)
2021-07-25 16:25:33 +00:00
- <span class="badge badge-warning">warning</span> frontend: 5%+ gitserver error responses every 5m for 15m0s
codeintel: Update metrics dashboards (#16806)
2020-12-18 17:57:28 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
codeintel: Update metrics dashboards (#16806)
2020-12-18 17:57:28 +00:00
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#frontend-gitserver-error-responses).
codeintel: Update metrics dashboards (#16806)
2020-12-18 17:57:28 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
monitoring: Update dashboards (#23091)
2021-07-25 16:25:33 +00:00
"warning_frontend_gitserver_error_responses"
codeintel: Update metrics dashboards (#16806)
2020-12-18 17:57:28 +00:00
]
```
Update o11y ownership from IAM/repo-mgmt to Source (#54368) Does what it says in the title. ## Test plan - Ran `sg generate`
2023-06-28 13:49:16 +00:00
<sub>*Managed by the [Sourcegraph Source team](https://handbook.sourcegraph.com/departments/engineering/teams/source).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
monitoring: extract into a submodule (#45786) This change extracts `monitoring` into a submodule for import in `sourcegraph/controller` (https://github.com/sourcegraph/controller/pull/195) so that we can generate dashboards for Cloud instances. These steps were required: 1. Initialize a `go.mod` in `monitoring` 2. Extract `dev/sg/internal/cliutil` into `lib` to avoid illegal imports from `monitoring` 3. Add local replaces to both `sourcegraph/sourcegraph` and `monitoring` 4. `go mod tidy` on all submodules 5. Update `go generate ./monitoring` commands to use `sg`, since the `go generate` command no longer works 6. Update `grafana/build.sh`, `prometheus/build.sh` to build the submodule 7. Amend linters to check for multiple `go.mod` files and ban imports of `github.com/sourcegraph/sourcegraph` 8. Update `sg generate go` to run in directories rather than from root The only caveat is that if you use VS code, you will now need to open `monitoring` in a separate workspace or similar, like with `lib`. Co-authored-by: Joe Chen <joe@sourcegraph.com>
2022-12-19 17:49:25 +00:00
Generated query for warning alert: `max((sum by (category) (increase(src_gitserver_request_duration_seconds_count{code!~"2..",job=~"(sourcegraph-)?frontend"}[5m])) / ignoring (code) group_left () sum by (category) (increase(src_gitserver_request_duration_seconds_count{job=~"(sourcegraph-)?frontend"}[5m])) * 100) >= 5)`
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
</details>
codeintel: Update metrics dashboards (#16806)
2020-12-18 17:57:28 +00:00
<br />
monitoring: Update dashboards (#23091)
2021-07-25 16:25:33 +00:00
## frontend: observability_test_alert_warning
codeintel: Update metrics dashboards (#16806)
2020-12-18 17:57:28 +00:00
monitoring: Update dashboards (#23091)
2021-07-25 16:25:33 +00:00
<p class="subtitle">warning test alert metric</p>
codeintel: Update metrics dashboards (#16806)
2020-12-18 17:57:28 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
codeintel: Update metrics dashboards (#16806)
2020-12-18 17:57:28 +00:00
monitoring: Update dashboards (#23091)
2021-07-25 16:25:33 +00:00
- <span class="badge badge-warning">warning</span> frontend: 1+ warning test alert metric
codeintel: Update metrics dashboards (#16806)
2020-12-18 17:57:28 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
codeintel: Update metrics dashboards (#16806)
2020-12-18 17:57:28 +00:00
monitoring: Update dashboards (#23091)
2021-07-25 16:25:33 +00:00
- This alert is triggered via the `triggerObservabilityTestAlert` GraphQL endpoint, and will automatically resolve itself.
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#frontend-observability-test-alert-warning).
codeintel: Update metrics dashboards (#16806)
2020-12-18 17:57:28 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
monitoring: Update dashboards (#23091)
2021-07-25 16:25:33 +00:00
"warning_frontend_observability_test_alert_warning"
codeintel: Update metrics dashboards (#16806)
2020-12-18 17:57:28 +00:00
]
```
monitoring: test owners for valid Opsgenie teams and handbook pages (#59251) In INC-264 it seems that certain alerts - such as [zoekt: less than 90% percentage pods available for 10m0s](https://opsg.in/a/i/sourcegraph/178a626f-0f28-4295-bee9-84da988bb473-1703759057681) - don't seem to end up going anywhere because the ObservableOwner is defunct. This change adds _opt-in_ testing to report: 1. How many owners have valid Opsgenie teams 2. How many owners have valid handbook pages In addition, we collect ObservableOwners that pass the test and use it to generate configuration for `site.json` in Sourcegraph.com: https://github.com/sourcegraph/deploy-sourcegraph-cloud/pull/18338 - this helps ensure the list is valid and not deceptively high-coverage. The results are not great, but **enforcing** that owners are valid isn't currently in scope: ``` 6/10 ObservableOwners do not have valid Opsgenie teams 3/10 ObservableOwners do not point to valid handbook pages ``` I also removed some defunct/unused functionality/owners. ## Test plan To run these tests: ``` export OPSGENIE_API_KEY="..." go test -timeout 30s github.com/sourcegraph/sourcegraph/monitoring/monitoring -update -online ```
2023-12-29 22:07:35 +00:00
<sub>*Managed by the [Sourcegraph Infrastructure Org team](https://handbook.sourcegraph.com/departments/engineering/infrastructure).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
monitoring: extract into a submodule (#45786) This change extracts `monitoring` into a submodule for import in `sourcegraph/controller` (https://github.com/sourcegraph/controller/pull/195) so that we can generate dashboards for Cloud instances. These steps were required: 1. Initialize a `go.mod` in `monitoring` 2. Extract `dev/sg/internal/cliutil` into `lib` to avoid illegal imports from `monitoring` 3. Add local replaces to both `sourcegraph/sourcegraph` and `monitoring` 4. `go mod tidy` on all submodules 5. Update `go generate ./monitoring` commands to use `sg`, since the `go generate` command no longer works 6. Update `grafana/build.sh`, `prometheus/build.sh` to build the submodule 7. Amend linters to check for multiple `go.mod` files and ban imports of `github.com/sourcegraph/sourcegraph` 8. Update `sg generate go` to run in directories rather than from root The only caveat is that if you use VS code, you will now need to open `monitoring` in a separate workspace or similar, like with `lib`. Co-authored-by: Joe Chen <joe@sourcegraph.com>
2022-12-19 17:49:25 +00:00
Generated query for warning alert: `max((max by (owner) (observability_test_metric_warning)) >= 1)`
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
</details>
codeintel: Update metrics dashboards (#16806)
2020-12-18 17:57:28 +00:00
<br />
monitoring: Update dashboards (#23091)
2021-07-25 16:25:33 +00:00
## frontend: observability_test_alert_critical
codeintel: Update metrics dashboards (#16806)
2020-12-18 17:57:28 +00:00
monitoring: Update dashboards (#23091)
2021-07-25 16:25:33 +00:00
<p class="subtitle">critical test alert metric</p>
codeintel: Update metrics dashboards (#16806)
2020-12-18 17:57:28 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
codeintel: Update metrics dashboards (#16806)
2020-12-18 17:57:28 +00:00
monitoring: Update dashboards (#23091)
2021-07-25 16:25:33 +00:00
- <span class="badge badge-critical">critical</span> frontend: 1+ critical test alert metric
codeintel: Update metrics dashboards (#16806)
2020-12-18 17:57:28 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
codeintel: Update metrics dashboards (#16806)
2020-12-18 17:57:28 +00:00
monitoring: Update dashboards (#23091)
2021-07-25 16:25:33 +00:00
- This alert is triggered via the `triggerObservabilityTestAlert` GraphQL endpoint, and will automatically resolve itself.
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#frontend-observability-test-alert-critical).
codeintel: Update metrics dashboards (#16806)
2020-12-18 17:57:28 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
monitoring: Update dashboards (#23091)
2021-07-25 16:25:33 +00:00
"critical_frontend_observability_test_alert_critical"
codeintel: Update metrics dashboards (#16806)
2020-12-18 17:57:28 +00:00
]
```
monitoring: test owners for valid Opsgenie teams and handbook pages (#59251) In INC-264 it seems that certain alerts - such as [zoekt: less than 90% percentage pods available for 10m0s](https://opsg.in/a/i/sourcegraph/178a626f-0f28-4295-bee9-84da988bb473-1703759057681) - don't seem to end up going anywhere because the ObservableOwner is defunct. This change adds _opt-in_ testing to report: 1. How many owners have valid Opsgenie teams 2. How many owners have valid handbook pages In addition, we collect ObservableOwners that pass the test and use it to generate configuration for `site.json` in Sourcegraph.com: https://github.com/sourcegraph/deploy-sourcegraph-cloud/pull/18338 - this helps ensure the list is valid and not deceptively high-coverage. The results are not great, but **enforcing** that owners are valid isn't currently in scope: ``` 6/10 ObservableOwners do not have valid Opsgenie teams 3/10 ObservableOwners do not point to valid handbook pages ``` I also removed some defunct/unused functionality/owners. ## Test plan To run these tests: ``` export OPSGENIE_API_KEY="..." go test -timeout 30s github.com/sourcegraph/sourcegraph/monitoring/monitoring -update -online ```
2023-12-29 22:07:35 +00:00
<sub>*Managed by the [Sourcegraph Infrastructure Org team](https://handbook.sourcegraph.com/departments/engineering/infrastructure).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
monitoring: extract into a submodule (#45786) This change extracts `monitoring` into a submodule for import in `sourcegraph/controller` (https://github.com/sourcegraph/controller/pull/195) so that we can generate dashboards for Cloud instances. These steps were required: 1. Initialize a `go.mod` in `monitoring` 2. Extract `dev/sg/internal/cliutil` into `lib` to avoid illegal imports from `monitoring` 3. Add local replaces to both `sourcegraph/sourcegraph` and `monitoring` 4. `go mod tidy` on all submodules 5. Update `go generate ./monitoring` commands to use `sg`, since the `go generate` command no longer works 6. Update `grafana/build.sh`, `prometheus/build.sh` to build the submodule 7. Amend linters to check for multiple `go.mod` files and ban imports of `github.com/sourcegraph/sourcegraph` 8. Update `sg generate go` to run in directories rather than from root The only caveat is that if you use VS code, you will now need to open `monitoring` in a separate workspace or similar, like with `lib`. Co-authored-by: Joe Chen <joe@sourcegraph.com>
2022-12-19 17:49:25 +00:00
Generated query for critical alert: `max((max by (owner) (observability_test_metric_critical)) >= 1)`
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
</details>
codeintel: Update metrics dashboards (#16806)
2020-12-18 17:57:28 +00:00
<br />
[CLOUD-105] cloudkms: add metrics for number of cryptographic requests (#28005)
2021-11-24 03:54:50 +00:00
## frontend: cloudkms_cryptographic_requests
<p class="subtitle">cryptographic requests to Cloud KMS every 1m</p>
**Descriptions**
- <span class="badge badge-warning">warning</span> frontend: 15000+ cryptographic requests to Cloud KMS every 1m for 5m0s
- <span class="badge badge-critical">critical</span> frontend: 30000+ cryptographic requests to Cloud KMS every 1m for 5m0s
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
[CLOUD-105] cloudkms: add metrics for number of cryptographic requests (#28005)
2021-11-24 03:54:50 +00:00
- Revert recent commits that cause extensive listing from "external_services" and/or "user_external_accounts" tables.
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#frontend-cloudkms-cryptographic-requests).
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"warning_frontend_cloudkms_cryptographic_requests",
"critical_frontend_cloudkms_cryptographic_requests"
]
```
Update o11y ownership from IAM/repo-mgmt to Source (#54368) Does what it says in the title. ## Test plan - Ran `sg generate`
2023-06-28 13:49:16 +00:00
<sub>*Managed by the [Sourcegraph Source team](https://handbook.sourcegraph.com/departments/engineering/teams/source).*</sub>
[CLOUD-105] cloudkms: add metrics for number of cryptographic requests (#28005)
2021-11-24 03:54:50 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
Generated query for warning alert: `max((sum(increase(src_cloudkms_cryptographic_total[1m]))) >= 15000)`
Generated query for critical alert: `max((sum(increase(src_cloudkms_cryptographic_total[1m]))) >= 30000)`
</details>
[CLOUD-105] cloudkms: add metrics for number of cryptographic requests (#28005)
2021-11-24 03:54:50 +00:00
<br />
monitoring: Update dashboards (#23091)
2021-07-25 16:25:33 +00:00
## frontend: mean_blocked_seconds_per_conn_request
codeintel: Update metrics dashboards (#16806)
2020-12-18 17:57:28 +00:00
monitoring: Update dashboards (#23091)
2021-07-25 16:25:33 +00:00
<p class="subtitle">mean blocked seconds per conn request</p>
codeintel: Update metrics dashboards (#16806)
2020-12-18 17:57:28 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
codeintel: Update metrics dashboards (#16806)
2020-12-18 17:57:28 +00:00
monitoring: relax mean_blocked_seconds_per_conn_request alerts (#59507) https://github.com/sourcegraph/sourcegraph/pull/59284 dramatically reduced the `mean_blocked_seconds_per_conn_request` issues we've been seeing, but overall delays are still higher, even with generally healthy Cloud SQL resource utilization. <img width="1630" alt="image" src="https://github.com/sourcegraph/sourcegraph/assets/23356519/91615471-5187-4d15-83e7-5cc94595303c"> Spot-checking the spikes in load in Cloud SQL, it seems that there is a variety of causes for each spike (analytics workloads, Cody Gateway syncs, code intel workloads, gitserver things, `ListSourcegraphDotComIndexableRepos` etc) so I'm chalking this up to "expected". Since this alert is seen firing on a Cloud instance, let's just relax it for now so that it only fires a critical alert on very significant delays.
2024-01-11 01:14:28 +00:00
- <span class="badge badge-warning">warning</span> frontend: 0.1s+ mean blocked seconds per conn request for 10m0s
- <span class="badge badge-critical">critical</span> frontend: 0.5s+ mean blocked seconds per conn request for 10m0s
codeintel: Update metrics dashboards (#16806)
2020-12-18 17:57:28 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
codeintel: Update metrics dashboards (#16806)
2020-12-18 17:57:28 +00:00
Add possible solutions and make docs discoverable (#22945) Add possible solutions and make docs discoverable
2021-07-30 00:30:31 +00:00
- Increase SRC_PGSQL_MAX_OPEN together with giving more memory to the database if needed
Docs: update links to point to new site (#60381) We have a number of docs links in the product that point to the old doc site. Method: - Search the repo for `docs.sourcegraph.com` - Exclude the `doc/` dir, all test fixtures, and `CHANGELOG.md` - For each, replace `docs.sourcegraph.com` with `sourcegraph.com/docs` - Navigate to the resulting URL ensuring it's not a dead link, updating the URL if necessary Many of the URLs updated are just comments, but since I'm doing a manual audit of each URL anyways, I felt it was worth it to update these while I was at it.
2024-02-13 00:23:47 +00:00
- Scale up Postgres memory/cpus - [see our scaling guide](https://sourcegraph.com/docs/admin/config/postgres-conf)
monitoring: relax mean_blocked_seconds_per_conn_request alerts (#59507) https://github.com/sourcegraph/sourcegraph/pull/59284 dramatically reduced the `mean_blocked_seconds_per_conn_request` issues we've been seeing, but overall delays are still higher, even with generally healthy Cloud SQL resource utilization. <img width="1630" alt="image" src="https://github.com/sourcegraph/sourcegraph/assets/23356519/91615471-5187-4d15-83e7-5cc94595303c"> Spot-checking the spikes in load in Cloud SQL, it seems that there is a variety of causes for each spike (analytics workloads, Cody Gateway syncs, code intel workloads, gitserver things, `ListSourcegraphDotComIndexableRepos` etc) so I'm chalking this up to "expected". Since this alert is seen firing on a Cloud instance, let's just relax it for now so that it only fires a critical alert on very significant delays.
2024-01-11 01:14:28 +00:00
- If using GCP Cloud SQL, check for high lock waits or CPU usage in query insights
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#frontend-mean-blocked-seconds-per-conn-request).
codeintel: Update metrics dashboards (#16806)
2020-12-18 17:57:28 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
monitoring: Update dashboards (#23091)
2021-07-25 16:25:33 +00:00
"warning_frontend_mean_blocked_seconds_per_conn_request",
"critical_frontend_mean_blocked_seconds_per_conn_request"
codeintel: Update metrics dashboards (#16806)
2020-12-18 17:57:28 +00:00
]
```
monitoring: test owners for valid Opsgenie teams and handbook pages (#59251) In INC-264 it seems that certain alerts - such as [zoekt: less than 90% percentage pods available for 10m0s](https://opsg.in/a/i/sourcegraph/178a626f-0f28-4295-bee9-84da988bb473-1703759057681) - don't seem to end up going anywhere because the ObservableOwner is defunct. This change adds _opt-in_ testing to report: 1. How many owners have valid Opsgenie teams 2. How many owners have valid handbook pages In addition, we collect ObservableOwners that pass the test and use it to generate configuration for `site.json` in Sourcegraph.com: https://github.com/sourcegraph/deploy-sourcegraph-cloud/pull/18338 - this helps ensure the list is valid and not deceptively high-coverage. The results are not great, but **enforcing** that owners are valid isn't currently in scope: ``` 6/10 ObservableOwners do not have valid Opsgenie teams 3/10 ObservableOwners do not point to valid handbook pages ``` I also removed some defunct/unused functionality/owners. ## Test plan To run these tests: ``` export OPSGENIE_API_KEY="..." go test -timeout 30s github.com/sourcegraph/sourcegraph/monitoring/monitoring -update -online ```
2023-12-29 22:07:35 +00:00
<sub>*Managed by the [Sourcegraph Infrastructure Org team](https://handbook.sourcegraph.com/departments/engineering/infrastructure).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
monitoring: relax mean_blocked_seconds_per_conn_request alerts (#59507) https://github.com/sourcegraph/sourcegraph/pull/59284 dramatically reduced the `mean_blocked_seconds_per_conn_request` issues we've been seeing, but overall delays are still higher, even with generally healthy Cloud SQL resource utilization. <img width="1630" alt="image" src="https://github.com/sourcegraph/sourcegraph/assets/23356519/91615471-5187-4d15-83e7-5cc94595303c"> Spot-checking the spikes in load in Cloud SQL, it seems that there is a variety of causes for each spike (analytics workloads, Cody Gateway syncs, code intel workloads, gitserver things, `ListSourcegraphDotComIndexableRepos` etc) so I'm chalking this up to "expected". Since this alert is seen firing on a Cloud instance, let's just relax it for now so that it only fires a critical alert on very significant delays.
2024-01-11 01:14:28 +00:00
Generated query for warning alert: `max((sum by (app_name, db_name) (increase(src_pgsql_conns_blocked_seconds{app_name="frontend"}[5m])) / sum by (app_name, db_name) (increase(src_pgsql_conns_waited_for{app_name="frontend"}[5m]))) >= 0.1)`
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
monitoring: relax mean_blocked_seconds_per_conn_request alerts (#59507) https://github.com/sourcegraph/sourcegraph/pull/59284 dramatically reduced the `mean_blocked_seconds_per_conn_request` issues we've been seeing, but overall delays are still higher, even with generally healthy Cloud SQL resource utilization. <img width="1630" alt="image" src="https://github.com/sourcegraph/sourcegraph/assets/23356519/91615471-5187-4d15-83e7-5cc94595303c"> Spot-checking the spikes in load in Cloud SQL, it seems that there is a variety of causes for each spike (analytics workloads, Cody Gateway syncs, code intel workloads, gitserver things, `ListSourcegraphDotComIndexableRepos` etc) so I'm chalking this up to "expected". Since this alert is seen firing on a Cloud instance, let's just relax it for now so that it only fires a critical alert on very significant delays.
2024-01-11 01:14:28 +00:00
Generated query for critical alert: `max((sum by (app_name, db_name) (increase(src_pgsql_conns_blocked_seconds{app_name="frontend"}[5m])) / sum by (app_name, db_name) (increase(src_pgsql_conns_waited_for{app_name="frontend"}[5m]))) >= 0.5)`
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
</details>
codeintel: Update metrics dashboards (#16806)
2020-12-18 17:57:28 +00:00
<br />
monitoring: Update dashboards (#23091)
2021-07-25 16:25:33 +00:00
## frontend: container_cpu_usage
codeintel: Update monitoring (#14695)
2020-10-14 14:19:15 +00:00
monitoring: Update dashboards (#23091)
2021-07-25 16:25:33 +00:00
<p class="subtitle">container cpu usage total (1m average) across all cores by instance</p>
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
codeintel: Update monitoring (#14695)
2020-10-14 14:19:15 +00:00
monitoring: Update dashboards (#23091)
2021-07-25 16:25:33 +00:00
- <span class="badge badge-warning">warning</span> frontend: 99%+ container cpu usage total (1m average) across all cores by instance
codeintel: Update monitoring (#14695)
2020-10-14 14:19:15 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
codeintel: Update monitoring (#14695)
2020-10-14 14:19:15 +00:00
monitoring: Update dashboards (#23091)
2021-07-25 16:25:33 +00:00
- **Kubernetes:** Consider increasing CPU limits in the the relevant `Deployment.yaml`.
- **Docker Compose:** Consider increasing `cpus:` of the (frontend|sourcegraph-frontend) container in `docker-compose.yml`.
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#frontend-container-cpu-usage).
codeintel: Update monitoring (#14695)
2020-10-14 14:19:15 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
monitoring: Update dashboards (#23091)
2021-07-25 16:25:33 +00:00
"warning_frontend_container_cpu_usage"
codeintel: Update monitoring (#14695)
2020-10-14 14:19:15 +00:00
]
```
monitoring: test owners for valid Opsgenie teams and handbook pages (#59251) In INC-264 it seems that certain alerts - such as [zoekt: less than 90% percentage pods available for 10m0s](https://opsg.in/a/i/sourcegraph/178a626f-0f28-4295-bee9-84da988bb473-1703759057681) - don't seem to end up going anywhere because the ObservableOwner is defunct. This change adds _opt-in_ testing to report: 1. How many owners have valid Opsgenie teams 2. How many owners have valid handbook pages In addition, we collect ObservableOwners that pass the test and use it to generate configuration for `site.json` in Sourcegraph.com: https://github.com/sourcegraph/deploy-sourcegraph-cloud/pull/18338 - this helps ensure the list is valid and not deceptively high-coverage. The results are not great, but **enforcing** that owners are valid isn't currently in scope: ``` 6/10 ObservableOwners do not have valid Opsgenie teams 3/10 ObservableOwners do not point to valid handbook pages ``` I also removed some defunct/unused functionality/owners. ## Test plan To run these tests: ``` export OPSGENIE_API_KEY="..." go test -timeout 30s github.com/sourcegraph/sourcegraph/monitoring/monitoring -update -online ```
2023-12-29 22:07:35 +00:00
<sub>*Managed by the [Sourcegraph Infrastructure Org team](https://handbook.sourcegraph.com/departments/engineering/infrastructure).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
Generated query for warning alert: `max((cadvisor_container_cpu_usage_percentage_total{name=~"^(frontend|sourcegraph-frontend).*"}) >= 99)`
</details>
monitoring: improve solutions formatting, add team information (#14878)
2020-10-20 05:11:20 +00:00
<br />
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: Update dashboards (#23091)
2021-07-25 16:25:33 +00:00
## frontend: container_memory_usage
codeintel: Update monitoring (#14695)
2020-10-14 14:19:15 +00:00
monitoring: Update dashboards (#23091)
2021-07-25 16:25:33 +00:00
<p class="subtitle">container memory usage by instance</p>
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
codeintel: Update monitoring (#14695)
2020-10-14 14:19:15 +00:00
monitoring: Update dashboards (#23091)
2021-07-25 16:25:33 +00:00
- <span class="badge badge-warning">warning</span> frontend: 99%+ container memory usage by instance
codeintel: Update monitoring (#14695)
2020-10-14 14:19:15 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
codeintel: Update monitoring (#14695)
2020-10-14 14:19:15 +00:00
monitoring: Update dashboards (#23091)
2021-07-25 16:25:33 +00:00
- **Kubernetes:** Consider increasing memory limit in relevant `Deployment.yaml`.
- **Docker Compose:** Consider increasing `memory:` of (frontend|sourcegraph-frontend) container in `docker-compose.yml`.
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#frontend-container-memory-usage).
codeintel: Update monitoring (#14695)
2020-10-14 14:19:15 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
monitoring: Update dashboards (#23091)
2021-07-25 16:25:33 +00:00
"warning_frontend_container_memory_usage"
codeintel: Update monitoring (#14695)
2020-10-14 14:19:15 +00:00
]
```
monitoring: test owners for valid Opsgenie teams and handbook pages (#59251) In INC-264 it seems that certain alerts - such as [zoekt: less than 90% percentage pods available for 10m0s](https://opsg.in/a/i/sourcegraph/178a626f-0f28-4295-bee9-84da988bb473-1703759057681) - don't seem to end up going anywhere because the ObservableOwner is defunct. This change adds _opt-in_ testing to report: 1. How many owners have valid Opsgenie teams 2. How many owners have valid handbook pages In addition, we collect ObservableOwners that pass the test and use it to generate configuration for `site.json` in Sourcegraph.com: https://github.com/sourcegraph/deploy-sourcegraph-cloud/pull/18338 - this helps ensure the list is valid and not deceptively high-coverage. The results are not great, but **enforcing** that owners are valid isn't currently in scope: ``` 6/10 ObservableOwners do not have valid Opsgenie teams 3/10 ObservableOwners do not point to valid handbook pages ``` I also removed some defunct/unused functionality/owners. ## Test plan To run these tests: ``` export OPSGENIE_API_KEY="..." go test -timeout 30s github.com/sourcegraph/sourcegraph/monitoring/monitoring -update -online ```
2023-12-29 22:07:35 +00:00
<sub>*Managed by the [Sourcegraph Infrastructure Org team](https://handbook.sourcegraph.com/departments/engineering/infrastructure).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
Generated query for warning alert: `max((cadvisor_container_memory_usage_percentage_total{name=~"^(frontend|sourcegraph-frontend).*"}) >= 99)`
</details>
monitoring: improve solutions formatting, add team information (#14878)
2020-10-20 05:11:20 +00:00
<br />
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: Update dashboards (#23091)
2021-07-25 16:25:33 +00:00
## frontend: provisioning_container_cpu_usage_long_term
codeintel: Update monitoring (#14695)
2020-10-14 14:19:15 +00:00
monitoring: Update dashboards (#23091)
2021-07-25 16:25:33 +00:00
<p class="subtitle">container cpu usage total (90th percentile over 1d) across all cores by instance</p>
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
codeintel: Update monitoring (#14695)
2020-10-14 14:19:15 +00:00
monitoring: Update dashboards (#23091)
2021-07-25 16:25:33 +00:00
- <span class="badge badge-warning">warning</span> frontend: 80%+ container cpu usage total (90th percentile over 1d) across all cores by instance for 336h0m0s
codeintel: Update monitoring (#14695)
2020-10-14 14:19:15 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
codeintel: Update monitoring (#14695)
2020-10-14 14:19:15 +00:00
monitoring: Update dashboards (#23091)
2021-07-25 16:25:33 +00:00
- **Kubernetes:** Consider increasing CPU limits in the `Deployment.yaml` for the (frontend|sourcegraph-frontend) service.
- **Docker Compose:** Consider increasing `cpus:` of the (frontend|sourcegraph-frontend) container in `docker-compose.yml`.
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#frontend-provisioning-container-cpu-usage-long-term).
codeintel: Update monitoring (#14695)
2020-10-14 14:19:15 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
monitoring: Update dashboards (#23091)
2021-07-25 16:25:33 +00:00
"warning_frontend_provisioning_container_cpu_usage_long_term"
codeintel: Update monitoring (#14695)
2020-10-14 14:19:15 +00:00
]
```
monitoring: test owners for valid Opsgenie teams and handbook pages (#59251) In INC-264 it seems that certain alerts - such as [zoekt: less than 90% percentage pods available for 10m0s](https://opsg.in/a/i/sourcegraph/178a626f-0f28-4295-bee9-84da988bb473-1703759057681) - don't seem to end up going anywhere because the ObservableOwner is defunct. This change adds _opt-in_ testing to report: 1. How many owners have valid Opsgenie teams 2. How many owners have valid handbook pages In addition, we collect ObservableOwners that pass the test and use it to generate configuration for `site.json` in Sourcegraph.com: https://github.com/sourcegraph/deploy-sourcegraph-cloud/pull/18338 - this helps ensure the list is valid and not deceptively high-coverage. The results are not great, but **enforcing** that owners are valid isn't currently in scope: ``` 6/10 ObservableOwners do not have valid Opsgenie teams 3/10 ObservableOwners do not point to valid handbook pages ``` I also removed some defunct/unused functionality/owners. ## Test plan To run these tests: ``` export OPSGENIE_API_KEY="..." go test -timeout 30s github.com/sourcegraph/sourcegraph/monitoring/monitoring -update -online ```
2023-12-29 22:07:35 +00:00
<sub>*Managed by the [Sourcegraph Infrastructure Org team](https://handbook.sourcegraph.com/departments/engineering/infrastructure).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
Generated query for warning alert: `max((quantile_over_time(0.9, cadvisor_container_cpu_usage_percentage_total{name=~"^(frontend|sourcegraph-frontend).*"}[1d])) >= 80)`
</details>
monitoring: improve solutions formatting, add team information (#14878)
2020-10-20 05:11:20 +00:00
<br />
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: Update dashboards (#23091)
2021-07-25 16:25:33 +00:00
## frontend: provisioning_container_memory_usage_long_term
codeintel: Update monitoring (#14695)
2020-10-14 14:19:15 +00:00
monitoring: Update dashboards (#23091)
2021-07-25 16:25:33 +00:00
<p class="subtitle">container memory usage (1d maximum) by instance</p>
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
codeintel: Update monitoring (#14695)
2020-10-14 14:19:15 +00:00
monitoring: Update dashboards (#23091)
2021-07-25 16:25:33 +00:00
- <span class="badge badge-warning">warning</span> frontend: 80%+ container memory usage (1d maximum) by instance for 336h0m0s
codeintel: Update monitoring (#14695)
2020-10-14 14:19:15 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
codeintel: Update monitoring (#14695)
2020-10-14 14:19:15 +00:00
monitoring: Update dashboards (#23091)
2021-07-25 16:25:33 +00:00
- **Kubernetes:** Consider increasing memory limits in the `Deployment.yaml` for the (frontend|sourcegraph-frontend) service.
- **Docker Compose:** Consider increasing `memory:` of the (frontend|sourcegraph-frontend) container in `docker-compose.yml`.
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#frontend-provisioning-container-memory-usage-long-term).
codeintel: Update monitoring (#14695)
2020-10-14 14:19:15 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
monitoring: Update dashboards (#23091)
2021-07-25 16:25:33 +00:00
"warning_frontend_provisioning_container_memory_usage_long_term"
codeintel: Update metrics (#15541)
2020-11-09 19:58:15 +00:00
]
```
monitoring: test owners for valid Opsgenie teams and handbook pages (#59251) In INC-264 it seems that certain alerts - such as [zoekt: less than 90% percentage pods available for 10m0s](https://opsg.in/a/i/sourcegraph/178a626f-0f28-4295-bee9-84da988bb473-1703759057681) - don't seem to end up going anywhere because the ObservableOwner is defunct. This change adds _opt-in_ testing to report: 1. How many owners have valid Opsgenie teams 2. How many owners have valid handbook pages In addition, we collect ObservableOwners that pass the test and use it to generate configuration for `site.json` in Sourcegraph.com: https://github.com/sourcegraph/deploy-sourcegraph-cloud/pull/18338 - this helps ensure the list is valid and not deceptively high-coverage. The results are not great, but **enforcing** that owners are valid isn't currently in scope: ``` 6/10 ObservableOwners do not have valid Opsgenie teams 3/10 ObservableOwners do not point to valid handbook pages ``` I also removed some defunct/unused functionality/owners. ## Test plan To run these tests: ``` export OPSGENIE_API_KEY="..." go test -timeout 30s github.com/sourcegraph/sourcegraph/monitoring/monitoring -update -online ```
2023-12-29 22:07:35 +00:00
<sub>*Managed by the [Sourcegraph Infrastructure Org team](https://handbook.sourcegraph.com/departments/engineering/infrastructure).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
Generated query for warning alert: `max((max_over_time(cadvisor_container_memory_usage_percentage_total{name=~"^(frontend|sourcegraph-frontend).*"}[1d])) >= 80)`
</details>
codeintel: Update metrics (#15541)
2020-11-09 19:58:15 +00:00
<br />
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: Update dashboards (#23091)
2021-07-25 16:25:33 +00:00
## frontend: provisioning_container_cpu_usage_short_term
codeintel: Update metrics (#15541)
2020-11-09 19:58:15 +00:00
monitoring: Update dashboards (#23091)
2021-07-25 16:25:33 +00:00
<p class="subtitle">container cpu usage total (5m maximum) across all cores by instance</p>
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
codeintel: Update metrics (#15541)
2020-11-09 19:58:15 +00:00
monitoring: Update dashboards (#23091)
2021-07-25 16:25:33 +00:00
- <span class="badge badge-warning">warning</span> frontend: 90%+ container cpu usage total (5m maximum) across all cores by instance for 30m0s
codeintel: Update metrics (#15541)
2020-11-09 19:58:15 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
codeintel: Update metrics (#15541)
2020-11-09 19:58:15 +00:00
monitoring: Update dashboards (#23091)
2021-07-25 16:25:33 +00:00
- **Kubernetes:** Consider increasing CPU limits in the the relevant `Deployment.yaml`.
- **Docker Compose:** Consider increasing `cpus:` of the (frontend|sourcegraph-frontend) container in `docker-compose.yml`.
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#frontend-provisioning-container-cpu-usage-short-term).
codeintel: Update metrics (#15541)
2020-11-09 19:58:15 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
monitoring: Update dashboards (#23091)
2021-07-25 16:25:33 +00:00
"warning_frontend_provisioning_container_cpu_usage_short_term"
codeintel: Update metrics (#15541)
2020-11-09 19:58:15 +00:00
]
```
monitoring: test owners for valid Opsgenie teams and handbook pages (#59251) In INC-264 it seems that certain alerts - such as [zoekt: less than 90% percentage pods available for 10m0s](https://opsg.in/a/i/sourcegraph/178a626f-0f28-4295-bee9-84da988bb473-1703759057681) - don't seem to end up going anywhere because the ObservableOwner is defunct. This change adds _opt-in_ testing to report: 1. How many owners have valid Opsgenie teams 2. How many owners have valid handbook pages In addition, we collect ObservableOwners that pass the test and use it to generate configuration for `site.json` in Sourcegraph.com: https://github.com/sourcegraph/deploy-sourcegraph-cloud/pull/18338 - this helps ensure the list is valid and not deceptively high-coverage. The results are not great, but **enforcing** that owners are valid isn't currently in scope: ``` 6/10 ObservableOwners do not have valid Opsgenie teams 3/10 ObservableOwners do not point to valid handbook pages ``` I also removed some defunct/unused functionality/owners. ## Test plan To run these tests: ``` export OPSGENIE_API_KEY="..." go test -timeout 30s github.com/sourcegraph/sourcegraph/monitoring/monitoring -update -online ```
2023-12-29 22:07:35 +00:00
<sub>*Managed by the [Sourcegraph Infrastructure Org team](https://handbook.sourcegraph.com/departments/engineering/infrastructure).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
Generated query for warning alert: `max((max_over_time(cadvisor_container_cpu_usage_percentage_total{name=~"^(frontend|sourcegraph-frontend).*"}[5m])) >= 90)`
</details>
codeintel: Update metrics (#15541)
2020-11-09 19:58:15 +00:00
<br />
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: Update dashboards (#23091)
2021-07-25 16:25:33 +00:00
## frontend: provisioning_container_memory_usage_short_term
codeintel: Update metrics (#15541)
2020-11-09 19:58:15 +00:00
monitoring: Update dashboards (#23091)
2021-07-25 16:25:33 +00:00
<p class="subtitle">container memory usage (5m maximum) by instance</p>
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
codeintel: Update metrics (#15541)
2020-11-09 19:58:15 +00:00
monitoring: Update dashboards (#23091)
2021-07-25 16:25:33 +00:00
- <span class="badge badge-warning">warning</span> frontend: 90%+ container memory usage (5m maximum) by instance
codeintel: Update metrics (#15541)
2020-11-09 19:58:15 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
codeintel: Update metrics (#15541)
2020-11-09 19:58:15 +00:00
monitoring: Update dashboards (#23091)
2021-07-25 16:25:33 +00:00
- **Kubernetes:** Consider increasing memory limit in relevant `Deployment.yaml`.
- **Docker Compose:** Consider increasing `memory:` of (frontend|sourcegraph-frontend) container in `docker-compose.yml`.
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#frontend-provisioning-container-memory-usage-short-term).
codeintel: Update metrics (#15541)
2020-11-09 19:58:15 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
monitoring: Update dashboards (#23091)
2021-07-25 16:25:33 +00:00
"warning_frontend_provisioning_container_memory_usage_short_term"
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
]
```
monitoring: test owners for valid Opsgenie teams and handbook pages (#59251) In INC-264 it seems that certain alerts - such as [zoekt: less than 90% percentage pods available for 10m0s](https://opsg.in/a/i/sourcegraph/178a626f-0f28-4295-bee9-84da988bb473-1703759057681) - don't seem to end up going anywhere because the ObservableOwner is defunct. This change adds _opt-in_ testing to report: 1. How many owners have valid Opsgenie teams 2. How many owners have valid handbook pages In addition, we collect ObservableOwners that pass the test and use it to generate configuration for `site.json` in Sourcegraph.com: https://github.com/sourcegraph/deploy-sourcegraph-cloud/pull/18338 - this helps ensure the list is valid and not deceptively high-coverage. The results are not great, but **enforcing** that owners are valid isn't currently in scope: ``` 6/10 ObservableOwners do not have valid Opsgenie teams 3/10 ObservableOwners do not point to valid handbook pages ``` I also removed some defunct/unused functionality/owners. ## Test plan To run these tests: ``` export OPSGENIE_API_KEY="..." go test -timeout 30s github.com/sourcegraph/sourcegraph/monitoring/monitoring -update -online ```
2023-12-29 22:07:35 +00:00
<sub>*Managed by the [Sourcegraph Infrastructure Org team](https://handbook.sourcegraph.com/departments/engineering/infrastructure).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
Generated query for warning alert: `max((max_over_time(cadvisor_container_memory_usage_percentage_total{name=~"^(frontend|sourcegraph-frontend).*"}[5m])) >= 90)`
</details>
monitoring: improve solutions formatting, add team information (#14878)
2020-10-20 05:11:20 +00:00
<br />
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: improve containers metrics (#34808) part of - https://github.com/sourcegraph/sourcegraph/issues/33437 - https://github.com/sourcegraph/sourcegraph/issues/33438 This PR adds the following - A shared observable metric to indicate how many times a container (or child processes) has been oom kill - It's added to provisioning indicators since it's an indicator of underprovisioned resources - A new dashboard that shows resources usage of all containers at a glance - It contains two groups, one for all containers and one for containers that may have potential scaling issues - the first group is extremely noisy so it's hidden by default - the second group provides a more focused experience for site-admin to quickly find problematic resources <!-- > ~it looks like the alert table will be rendered regardless, which is pretty annoying (I guess the implementation of the monitoring stack is based on the assumption that one service/container will have its dashboard)~ __Questions & Notes__ - ~Should I make the new dashboard filterable by container name? like I only want to see the graph of `frontend`. Is that something that's valuable?~ - ~Any other metrics you all would like to see?~ - graph for total available resources on the VM compared to usage from https://github.com/sourcegraph/sourcegraph/issues/33437 is not really doable since we do not have `node-exporter` in anyone of our distro. We would have to tackle this from https://github.com/sourcegraph/customer/issues/806 --> ## Test plan connect to `demo`'s prom ```sh gcloud compute start-iap-tunnel default-$NEW_DEPLOYMENT-instance 9090 --local-host-port=localhost:4445 --zone us-central1-f --project $PROJECT_PREFIX-$CUSTOMER ``` or use dogfood ```sh kubectl port-forward svc/prometheus 4445:30090 ``` Update `dev/grafana/all/datasources.yaml` ```yml # Configuration for all non-Linux platforms. apiVersion: 1 datasources: - name: Prometheus type: prometheus access: proxy url: http://docker.for.mac.localhost:4445 # use the remote prometheus instance on demo instead of the local one isDefault: true editable: false - name: Jaeger type: jaeger access: proxy url: http://host.docker.internal:16686/-/debug/jaeger - name: Loki type: loki access: proxy url: http://host.docker.internal:3100 ``` Start monitoring stack ```sh sg run grafana ``` see screenshot > the first group is hidden by default ![CleanShot 2022-05-04 at 10 48 26](https://user-images.githubusercontent.com/8373004/166748470-22460552-6e51-4b29-a26b-4c5adbe1c404.png) ![CleanShot 2022-05-04 at 10 48 03](https://user-images.githubusercontent.com/8373004/166748396-1d410c35-e5fe-45e2-83bd-418d845413cb.png) ![CleanShot 2022-05-04 at 11 25 13](https://user-images.githubusercontent.com/8373004/166801266-21e51b68-65fd-4cb7-97de-83520e31e498.png)
2022-05-09 23:56:47 +00:00
## frontend: container_oomkill_events_total
<p class="subtitle">container OOMKILL events total by instance</p>
**Descriptions**
- <span class="badge badge-warning">warning</span> frontend: 1+ container OOMKILL events total by instance
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: improve containers metrics (#34808) part of - https://github.com/sourcegraph/sourcegraph/issues/33437 - https://github.com/sourcegraph/sourcegraph/issues/33438 This PR adds the following - A shared observable metric to indicate how many times a container (or child processes) has been oom kill - It's added to provisioning indicators since it's an indicator of underprovisioned resources - A new dashboard that shows resources usage of all containers at a glance - It contains two groups, one for all containers and one for containers that may have potential scaling issues - the first group is extremely noisy so it's hidden by default - the second group provides a more focused experience for site-admin to quickly find problematic resources <!-- > ~it looks like the alert table will be rendered regardless, which is pretty annoying (I guess the implementation of the monitoring stack is based on the assumption that one service/container will have its dashboard)~ __Questions & Notes__ - ~Should I make the new dashboard filterable by container name? like I only want to see the graph of `frontend`. Is that something that's valuable?~ - ~Any other metrics you all would like to see?~ - graph for total available resources on the VM compared to usage from https://github.com/sourcegraph/sourcegraph/issues/33437 is not really doable since we do not have `node-exporter` in anyone of our distro. We would have to tackle this from https://github.com/sourcegraph/customer/issues/806 --> ## Test plan connect to `demo`'s prom ```sh gcloud compute start-iap-tunnel default-$NEW_DEPLOYMENT-instance 9090 --local-host-port=localhost:4445 --zone us-central1-f --project $PROJECT_PREFIX-$CUSTOMER ``` or use dogfood ```sh kubectl port-forward svc/prometheus 4445:30090 ``` Update `dev/grafana/all/datasources.yaml` ```yml # Configuration for all non-Linux platforms. apiVersion: 1 datasources: - name: Prometheus type: prometheus access: proxy url: http://docker.for.mac.localhost:4445 # use the remote prometheus instance on demo instead of the local one isDefault: true editable: false - name: Jaeger type: jaeger access: proxy url: http://host.docker.internal:16686/-/debug/jaeger - name: Loki type: loki access: proxy url: http://host.docker.internal:3100 ``` Start monitoring stack ```sh sg run grafana ``` see screenshot > the first group is hidden by default ![CleanShot 2022-05-04 at 10 48 26](https://user-images.githubusercontent.com/8373004/166748470-22460552-6e51-4b29-a26b-4c5adbe1c404.png) ![CleanShot 2022-05-04 at 10 48 03](https://user-images.githubusercontent.com/8373004/166748396-1d410c35-e5fe-45e2-83bd-418d845413cb.png) ![CleanShot 2022-05-04 at 11 25 13](https://user-images.githubusercontent.com/8373004/166801266-21e51b68-65fd-4cb7-97de-83520e31e498.png)
2022-05-09 23:56:47 +00:00
- **Kubernetes:** Consider increasing memory limit in relevant `Deployment.yaml`.
- **Docker Compose:** Consider increasing `memory:` of (frontend|sourcegraph-frontend) container in `docker-compose.yml`.
- More help interpreting this metric is available in the [dashboards reference](./dashboards.md#frontend-container-oomkill-events-total).
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"warning_frontend_container_oomkill_events_total"
]
```
monitoring: test owners for valid Opsgenie teams and handbook pages (#59251) In INC-264 it seems that certain alerts - such as [zoekt: less than 90% percentage pods available for 10m0s](https://opsg.in/a/i/sourcegraph/178a626f-0f28-4295-bee9-84da988bb473-1703759057681) - don't seem to end up going anywhere because the ObservableOwner is defunct. This change adds _opt-in_ testing to report: 1. How many owners have valid Opsgenie teams 2. How many owners have valid handbook pages In addition, we collect ObservableOwners that pass the test and use it to generate configuration for `site.json` in Sourcegraph.com: https://github.com/sourcegraph/deploy-sourcegraph-cloud/pull/18338 - this helps ensure the list is valid and not deceptively high-coverage. The results are not great, but **enforcing** that owners are valid isn't currently in scope: ``` 6/10 ObservableOwners do not have valid Opsgenie teams 3/10 ObservableOwners do not point to valid handbook pages ``` I also removed some defunct/unused functionality/owners. ## Test plan To run these tests: ``` export OPSGENIE_API_KEY="..." go test -timeout 30s github.com/sourcegraph/sourcegraph/monitoring/monitoring -update -online ```
2023-12-29 22:07:35 +00:00
<sub>*Managed by the [Sourcegraph Infrastructure Org team](https://handbook.sourcegraph.com/departments/engineering/infrastructure).*</sub>
monitoring: improve containers metrics (#34808) part of - https://github.com/sourcegraph/sourcegraph/issues/33437 - https://github.com/sourcegraph/sourcegraph/issues/33438 This PR adds the following - A shared observable metric to indicate how many times a container (or child processes) has been oom kill - It's added to provisioning indicators since it's an indicator of underprovisioned resources - A new dashboard that shows resources usage of all containers at a glance - It contains two groups, one for all containers and one for containers that may have potential scaling issues - the first group is extremely noisy so it's hidden by default - the second group provides a more focused experience for site-admin to quickly find problematic resources <!-- > ~it looks like the alert table will be rendered regardless, which is pretty annoying (I guess the implementation of the monitoring stack is based on the assumption that one service/container will have its dashboard)~ __Questions & Notes__ - ~Should I make the new dashboard filterable by container name? like I only want to see the graph of `frontend`. Is that something that's valuable?~ - ~Any other metrics you all would like to see?~ - graph for total available resources on the VM compared to usage from https://github.com/sourcegraph/sourcegraph/issues/33437 is not really doable since we do not have `node-exporter` in anyone of our distro. We would have to tackle this from https://github.com/sourcegraph/customer/issues/806 --> ## Test plan connect to `demo`'s prom ```sh gcloud compute start-iap-tunnel default-$NEW_DEPLOYMENT-instance 9090 --local-host-port=localhost:4445 --zone us-central1-f --project $PROJECT_PREFIX-$CUSTOMER ``` or use dogfood ```sh kubectl port-forward svc/prometheus 4445:30090 ``` Update `dev/grafana/all/datasources.yaml` ```yml # Configuration for all non-Linux platforms. apiVersion: 1 datasources: - name: Prometheus type: prometheus access: proxy url: http://docker.for.mac.localhost:4445 # use the remote prometheus instance on demo instead of the local one isDefault: true editable: false - name: Jaeger type: jaeger access: proxy url: http://host.docker.internal:16686/-/debug/jaeger - name: Loki type: loki access: proxy url: http://host.docker.internal:3100 ``` Start monitoring stack ```sh sg run grafana ``` see screenshot > the first group is hidden by default ![CleanShot 2022-05-04 at 10 48 26](https://user-images.githubusercontent.com/8373004/166748470-22460552-6e51-4b29-a26b-4c5adbe1c404.png) ![CleanShot 2022-05-04 at 10 48 03](https://user-images.githubusercontent.com/8373004/166748396-1d410c35-e5fe-45e2-83bd-418d845413cb.png) ![CleanShot 2022-05-04 at 11 25 13](https://user-images.githubusercontent.com/8373004/166801266-21e51b68-65fd-4cb7-97de-83520e31e498.png)
2022-05-09 23:56:47 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
monitoring: extract into a submodule (#45786) This change extracts `monitoring` into a submodule for import in `sourcegraph/controller` (https://github.com/sourcegraph/controller/pull/195) so that we can generate dashboards for Cloud instances. These steps were required: 1. Initialize a `go.mod` in `monitoring` 2. Extract `dev/sg/internal/cliutil` into `lib` to avoid illegal imports from `monitoring` 3. Add local replaces to both `sourcegraph/sourcegraph` and `monitoring` 4. `go mod tidy` on all submodules 5. Update `go generate ./monitoring` commands to use `sg`, since the `go generate` command no longer works 6. Update `grafana/build.sh`, `prometheus/build.sh` to build the submodule 7. Amend linters to check for multiple `go.mod` files and ban imports of `github.com/sourcegraph/sourcegraph` 8. Update `sg generate go` to run in directories rather than from root The only caveat is that if you use VS code, you will now need to open `monitoring` in a separate workspace or similar, like with `lib`. Co-authored-by: Joe Chen <joe@sourcegraph.com>
2022-12-19 17:49:25 +00:00
Generated query for warning alert: `max((max by (name) (container_oom_events_total{name=~"^(frontend|sourcegraph-frontend).*"})) >= 1)`
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
</details>
monitoring: improve containers metrics (#34808) part of - https://github.com/sourcegraph/sourcegraph/issues/33437 - https://github.com/sourcegraph/sourcegraph/issues/33438 This PR adds the following - A shared observable metric to indicate how many times a container (or child processes) has been oom kill - It's added to provisioning indicators since it's an indicator of underprovisioned resources - A new dashboard that shows resources usage of all containers at a glance - It contains two groups, one for all containers and one for containers that may have potential scaling issues - the first group is extremely noisy so it's hidden by default - the second group provides a more focused experience for site-admin to quickly find problematic resources <!-- > ~it looks like the alert table will be rendered regardless, which is pretty annoying (I guess the implementation of the monitoring stack is based on the assumption that one service/container will have its dashboard)~ __Questions & Notes__ - ~Should I make the new dashboard filterable by container name? like I only want to see the graph of `frontend`. Is that something that's valuable?~ - ~Any other metrics you all would like to see?~ - graph for total available resources on the VM compared to usage from https://github.com/sourcegraph/sourcegraph/issues/33437 is not really doable since we do not have `node-exporter` in anyone of our distro. We would have to tackle this from https://github.com/sourcegraph/customer/issues/806 --> ## Test plan connect to `demo`'s prom ```sh gcloud compute start-iap-tunnel default-$NEW_DEPLOYMENT-instance 9090 --local-host-port=localhost:4445 --zone us-central1-f --project $PROJECT_PREFIX-$CUSTOMER ``` or use dogfood ```sh kubectl port-forward svc/prometheus 4445:30090 ``` Update `dev/grafana/all/datasources.yaml` ```yml # Configuration for all non-Linux platforms. apiVersion: 1 datasources: - name: Prometheus type: prometheus access: proxy url: http://docker.for.mac.localhost:4445 # use the remote prometheus instance on demo instead of the local one isDefault: true editable: false - name: Jaeger type: jaeger access: proxy url: http://host.docker.internal:16686/-/debug/jaeger - name: Loki type: loki access: proxy url: http://host.docker.internal:3100 ``` Start monitoring stack ```sh sg run grafana ``` see screenshot > the first group is hidden by default ![CleanShot 2022-05-04 at 10 48 26](https://user-images.githubusercontent.com/8373004/166748470-22460552-6e51-4b29-a26b-4c5adbe1c404.png) ![CleanShot 2022-05-04 at 10 48 03](https://user-images.githubusercontent.com/8373004/166748396-1d410c35-e5fe-45e2-83bd-418d845413cb.png) ![CleanShot 2022-05-04 at 11 25 13](https://user-images.githubusercontent.com/8373004/166801266-21e51b68-65fd-4cb7-97de-83520e31e498.png)
2022-05-09 23:56:47 +00:00
<br />
monitoring: Update dashboards (#23091)
2021-07-25 16:25:33 +00:00
## frontend: go_goroutines
oobmigrations: Add metrics (#20304)
2021-04-22 16:00:44 +00:00
monitoring: Update dashboards (#23091)
2021-07-25 16:25:33 +00:00
<p class="subtitle">maximum active goroutines</p>
oobmigrations: Add metrics (#20304)
2021-04-22 16:00:44 +00:00
**Descriptions**
monitoring: Update dashboards (#23091)
2021-07-25 16:25:33 +00:00
- <span class="badge badge-warning">warning</span> frontend: 10000+ maximum active goroutines for 10m0s
oobmigrations: Add metrics (#20304)
2021-04-22 16:00:44 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
oobmigrations: Add metrics (#20304)
2021-04-22 16:00:44 +00:00
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- More help interpreting this metric is available in the [dashboards reference](./dashboards.md#frontend-go-goroutines).
oobmigrations: Add metrics (#20304)
2021-04-22 16:00:44 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
monitoring: Update dashboards (#23091)
2021-07-25 16:25:33 +00:00
"warning_frontend_go_goroutines"
oobmigrations: Add metrics (#20304)
2021-04-22 16:00:44 +00:00
]
```
monitoring: test owners for valid Opsgenie teams and handbook pages (#59251) In INC-264 it seems that certain alerts - such as [zoekt: less than 90% percentage pods available for 10m0s](https://opsg.in/a/i/sourcegraph/178a626f-0f28-4295-bee9-84da988bb473-1703759057681) - don't seem to end up going anywhere because the ObservableOwner is defunct. This change adds _opt-in_ testing to report: 1. How many owners have valid Opsgenie teams 2. How many owners have valid handbook pages In addition, we collect ObservableOwners that pass the test and use it to generate configuration for `site.json` in Sourcegraph.com: https://github.com/sourcegraph/deploy-sourcegraph-cloud/pull/18338 - this helps ensure the list is valid and not deceptively high-coverage. The results are not great, but **enforcing** that owners are valid isn't currently in scope: ``` 6/10 ObservableOwners do not have valid Opsgenie teams 3/10 ObservableOwners do not point to valid handbook pages ``` I also removed some defunct/unused functionality/owners. ## Test plan To run these tests: ``` export OPSGENIE_API_KEY="..." go test -timeout 30s github.com/sourcegraph/sourcegraph/monitoring/monitoring -update -online ```
2023-12-29 22:07:35 +00:00
<sub>*Managed by the [Sourcegraph Infrastructure Org team](https://handbook.sourcegraph.com/departments/engineering/infrastructure).*</sub>
oobmigrations: Add metrics (#20304)
2021-04-22 16:00:44 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
monitoring: extract into a submodule (#45786) This change extracts `monitoring` into a submodule for import in `sourcegraph/controller` (https://github.com/sourcegraph/controller/pull/195) so that we can generate dashboards for Cloud instances. These steps were required: 1. Initialize a `go.mod` in `monitoring` 2. Extract `dev/sg/internal/cliutil` into `lib` to avoid illegal imports from `monitoring` 3. Add local replaces to both `sourcegraph/sourcegraph` and `monitoring` 4. `go mod tidy` on all submodules 5. Update `go generate ./monitoring` commands to use `sg`, since the `go generate` command no longer works 6. Update `grafana/build.sh`, `prometheus/build.sh` to build the submodule 7. Amend linters to check for multiple `go.mod` files and ban imports of `github.com/sourcegraph/sourcegraph` 8. Update `sg generate go` to run in directories rather than from root The only caveat is that if you use VS code, you will now need to open `monitoring` in a separate workspace or similar, like with `lib`. Co-authored-by: Joe Chen <joe@sourcegraph.com>
2022-12-19 17:49:25 +00:00
Generated query for warning alert: `max((max by (instance) (go_goroutines{job=~".*(frontend|sourcegraph-frontend)"})) >= 10000)`
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
</details>
oobmigrations: Add metrics (#20304)
2021-04-22 16:00:44 +00:00
<br />
monitoring: Update dashboards (#23091)
2021-07-25 16:25:33 +00:00
## frontend: go_gc_duration_seconds
oobmigrations: Add metrics (#20304)
2021-04-22 16:00:44 +00:00
monitoring: Update dashboards (#23091)
2021-07-25 16:25:33 +00:00
<p class="subtitle">maximum go garbage collection duration</p>
oobmigrations: Add metrics (#20304)
2021-04-22 16:00:44 +00:00
**Descriptions**
monitoring: Update dashboards (#23091)
2021-07-25 16:25:33 +00:00
- <span class="badge badge-warning">warning</span> frontend: 2s+ maximum go garbage collection duration
oobmigrations: Add metrics (#20304)
2021-04-22 16:00:44 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
oobmigrations: Add metrics (#20304)
2021-04-22 16:00:44 +00:00
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#frontend-go-gc-duration-seconds).
oobmigrations: Add metrics (#20304)
2021-04-22 16:00:44 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
monitoring: Update dashboards (#23091)
2021-07-25 16:25:33 +00:00
"warning_frontend_go_gc_duration_seconds"
oobmigrations: Add metrics (#20304)
2021-04-22 16:00:44 +00:00
]
```
monitoring: test owners for valid Opsgenie teams and handbook pages (#59251) In INC-264 it seems that certain alerts - such as [zoekt: less than 90% percentage pods available for 10m0s](https://opsg.in/a/i/sourcegraph/178a626f-0f28-4295-bee9-84da988bb473-1703759057681) - don't seem to end up going anywhere because the ObservableOwner is defunct. This change adds _opt-in_ testing to report: 1. How many owners have valid Opsgenie teams 2. How many owners have valid handbook pages In addition, we collect ObservableOwners that pass the test and use it to generate configuration for `site.json` in Sourcegraph.com: https://github.com/sourcegraph/deploy-sourcegraph-cloud/pull/18338 - this helps ensure the list is valid and not deceptively high-coverage. The results are not great, but **enforcing** that owners are valid isn't currently in scope: ``` 6/10 ObservableOwners do not have valid Opsgenie teams 3/10 ObservableOwners do not point to valid handbook pages ``` I also removed some defunct/unused functionality/owners. ## Test plan To run these tests: ``` export OPSGENIE_API_KEY="..." go test -timeout 30s github.com/sourcegraph/sourcegraph/monitoring/monitoring -update -online ```
2023-12-29 22:07:35 +00:00
<sub>*Managed by the [Sourcegraph Infrastructure Org team](https://handbook.sourcegraph.com/departments/engineering/infrastructure).*</sub>
oobmigrations: Add metrics (#20304)
2021-04-22 16:00:44 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
monitoring: extract into a submodule (#45786) This change extracts `monitoring` into a submodule for import in `sourcegraph/controller` (https://github.com/sourcegraph/controller/pull/195) so that we can generate dashboards for Cloud instances. These steps were required: 1. Initialize a `go.mod` in `monitoring` 2. Extract `dev/sg/internal/cliutil` into `lib` to avoid illegal imports from `monitoring` 3. Add local replaces to both `sourcegraph/sourcegraph` and `monitoring` 4. `go mod tidy` on all submodules 5. Update `go generate ./monitoring` commands to use `sg`, since the `go generate` command no longer works 6. Update `grafana/build.sh`, `prometheus/build.sh` to build the submodule 7. Amend linters to check for multiple `go.mod` files and ban imports of `github.com/sourcegraph/sourcegraph` 8. Update `sg generate go` to run in directories rather than from root The only caveat is that if you use VS code, you will now need to open `monitoring` in a separate workspace or similar, like with `lib`. Co-authored-by: Joe Chen <joe@sourcegraph.com>
2022-12-19 17:49:25 +00:00
Generated query for warning alert: `max((max by (instance) (go_gc_duration_seconds{job=~".*(frontend|sourcegraph-frontend)"})) >= 2)`
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
</details>
oobmigrations: Add metrics (#20304)
2021-04-22 16:00:44 +00:00
<br />
monitoring: Update dashboards (#23091)
2021-07-25 16:25:33 +00:00
## frontend: pods_available_percentage
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
monitoring: Update dashboards (#23091)
2021-07-25 16:25:33 +00:00
<p class="subtitle">percentage pods available</p>
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
monitoring: Update dashboards (#23091)
2021-07-25 16:25:33 +00:00
- <span class="badge badge-critical">critical</span> frontend: less than 90% percentage pods available for 10m0s
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
monitoring: forbid critical alerts with no solutions (#36423) Adds the restriction that critical alerts must have solutions defined as a hard requirement within the generator, and fixes all the outstanding issues: - Add PossibleSolutions for the pods_available_percentage and pg_up critical alerts (we have a similar solution doc in the container_missing alert) - Removes all the other critical alerts without possible solutions Co-authored-by: Michael Lin <mlzc@hey.com>
2022-06-02 15:51:43 +00:00
- Determine if the pod was OOM killed using `kubectl describe pod (frontend|sourcegraph-frontend)` (look for `OOMKilled: true`) and, if so, consider increasing the memory limit in the relevant `Deployment.yaml`.
- Check the logs before the container restarted to see if there are `panic:` messages or similar using `kubectl logs -p (frontend|sourcegraph-frontend)`.
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#frontend-pods-available-percentage).
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
```json
"observability.silenceAlerts": [
monitoring: Update dashboards (#23091)
2021-07-25 16:25:33 +00:00
"critical_frontend_pods_available_percentage"
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
]
```
monitoring: test owners for valid Opsgenie teams and handbook pages (#59251) In INC-264 it seems that certain alerts - such as [zoekt: less than 90% percentage pods available for 10m0s](https://opsg.in/a/i/sourcegraph/178a626f-0f28-4295-bee9-84da988bb473-1703759057681) - don't seem to end up going anywhere because the ObservableOwner is defunct. This change adds _opt-in_ testing to report: 1. How many owners have valid Opsgenie teams 2. How many owners have valid handbook pages In addition, we collect ObservableOwners that pass the test and use it to generate configuration for `site.json` in Sourcegraph.com: https://github.com/sourcegraph/deploy-sourcegraph-cloud/pull/18338 - this helps ensure the list is valid and not deceptively high-coverage. The results are not great, but **enforcing** that owners are valid isn't currently in scope: ``` 6/10 ObservableOwners do not have valid Opsgenie teams 3/10 ObservableOwners do not point to valid handbook pages ``` I also removed some defunct/unused functionality/owners. ## Test plan To run these tests: ``` export OPSGENIE_API_KEY="..." go test -timeout 30s github.com/sourcegraph/sourcegraph/monitoring/monitoring -update -online ```
2023-12-29 22:07:35 +00:00
<sub>*Managed by the [Sourcegraph Infrastructure Org team](https://handbook.sourcegraph.com/departments/engineering/infrastructure).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
monitoring: extract into a submodule (#45786) This change extracts `monitoring` into a submodule for import in `sourcegraph/controller` (https://github.com/sourcegraph/controller/pull/195) so that we can generate dashboards for Cloud instances. These steps were required: 1. Initialize a `go.mod` in `monitoring` 2. Extract `dev/sg/internal/cliutil` into `lib` to avoid illegal imports from `monitoring` 3. Add local replaces to both `sourcegraph/sourcegraph` and `monitoring` 4. `go mod tidy` on all submodules 5. Update `go generate ./monitoring` commands to use `sg`, since the `go generate` command no longer works 6. Update `grafana/build.sh`, `prometheus/build.sh` to build the submodule 7. Amend linters to check for multiple `go.mod` files and ban imports of `github.com/sourcegraph/sourcegraph` 8. Update `sg generate go` to run in directories rather than from root The only caveat is that if you use VS code, you will now need to open `monitoring` in a separate workspace or similar, like with `lib`. Co-authored-by: Joe Chen <joe@sourcegraph.com>
2022-12-19 17:49:25 +00:00
Generated query for critical alert: `min((sum by (app) (up{app=~".*(frontend|sourcegraph-frontend)"}) / count by (app) (up{app=~".*(frontend|sourcegraph-frontend)"}) * 100) <= 90)`
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
</details>
monitoring: improve solutions formatting, add team information (#14878)
2020-10-20 05:11:20 +00:00
<br />
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: add email delivery monitoring (#43281)
2022-10-31 16:10:03 +00:00
## frontend: email_delivery_failures
monitoring: make email_delivery_failures on percentage, not count (#57045) This changes the threshold to critical-alert on 10% failure rate, and warn on any non-zero failure rate (as all email delivery failures impact user experience).
2023-09-26 16:04:29 +00:00
<p class="subtitle">email delivery failure rate over 30 minutes</p>
monitoring: add email delivery monitoring (#43281)
2022-10-31 16:10:03 +00:00
**Descriptions**
monitoring: make email_delivery_failures on percentage, not count (#57045) This changes the threshold to critical-alert on 10% failure rate, and warn on any non-zero failure rate (as all email delivery failures impact user experience).
2023-09-26 16:04:29 +00:00
- <span class="badge badge-warning">warning</span> frontend: 0%+ email delivery failure rate over 30 minutes
- <span class="badge badge-critical">critical</span> frontend: 10%+ email delivery failure rate over 30 minutes
monitoring: add email delivery monitoring (#43281)
2022-10-31 16:10:03 +00:00
**Next steps**
- Check your SMTP configuration in site configuration.
monitoring: make email_delivery_failures on percentage, not count (#57045) This changes the threshold to critical-alert on 10% failure rate, and warn on any non-zero failure rate (as all email delivery failures impact user experience).
2023-09-26 16:04:29 +00:00
- Check `sourcegraph-frontend` logs for more detailed error messages.
monitoring: add email delivery monitoring (#43281)
2022-10-31 16:10:03 +00:00
- Check your SMTP provider for more detailed error messages.
monitoring: make email_delivery_failures on percentage, not count (#57045) This changes the threshold to critical-alert on 10% failure rate, and warn on any non-zero failure rate (as all email delivery failures impact user experience).
2023-09-26 16:04:29 +00:00
- Use `sum(increase(src_email_send{success="false"}[30m]))` to check the raw count of delivery failures.
monitoring: add email delivery monitoring (#43281)
2022-10-31 16:10:03 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#frontend-email-delivery-failures).
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"warning_frontend_email_delivery_failures",
"critical_frontend_email_delivery_failures"
]
```
monitoring: test owners for valid Opsgenie teams and handbook pages (#59251) In INC-264 it seems that certain alerts - such as [zoekt: less than 90% percentage pods available for 10m0s](https://opsg.in/a/i/sourcegraph/178a626f-0f28-4295-bee9-84da988bb473-1703759057681) - don't seem to end up going anywhere because the ObservableOwner is defunct. This change adds _opt-in_ testing to report: 1. How many owners have valid Opsgenie teams 2. How many owners have valid handbook pages In addition, we collect ObservableOwners that pass the test and use it to generate configuration for `site.json` in Sourcegraph.com: https://github.com/sourcegraph/deploy-sourcegraph-cloud/pull/18338 - this helps ensure the list is valid and not deceptively high-coverage. The results are not great, but **enforcing** that owners are valid isn't currently in scope: ``` 6/10 ObservableOwners do not have valid Opsgenie teams 3/10 ObservableOwners do not point to valid handbook pages ``` I also removed some defunct/unused functionality/owners. ## Test plan To run these tests: ``` export OPSGENIE_API_KEY="..." go test -timeout 30s github.com/sourcegraph/sourcegraph/monitoring/monitoring -update -online ```
2023-12-29 22:07:35 +00:00
<sub>*Managed by the [Sourcegraph Infrastructure Org team](https://handbook.sourcegraph.com/departments/engineering/infrastructure).*</sub>
monitoring: add email delivery monitoring (#43281)
2022-10-31 16:10:03 +00:00
<details>
<summary>Technical details</summary>
monitoring: make email_delivery_failures on percentage, not count (#57045) This changes the threshold to critical-alert on 10% failure rate, and warn on any non-zero failure rate (as all email delivery failures impact user experience).
2023-09-26 16:04:29 +00:00
Generated query for warning alert: `max((sum(increase(src_email_send{success="false"}[30m])) / sum(increase(src_email_send[30m])) * 100) > 0)`
monitoring: add email delivery monitoring (#43281)
2022-10-31 16:10:03 +00:00
monitoring: make email_delivery_failures on percentage, not count (#57045) This changes the threshold to critical-alert on 10% failure rate, and warn on any non-zero failure rate (as all email delivery failures impact user experience).
2023-09-26 16:04:29 +00:00
Generated query for critical alert: `max((sum(increase(src_email_send{success="false"}[30m])) / sum(increase(src_email_send[30m])) * 100) >= 10)`
monitoring: add email delivery monitoring (#43281)
2022-10-31 16:10:03 +00:00
</details>
<br />
monitoring: relax alerting thresholds for sentinel queries (#34106)
2022-04-19 17:01:51 +00:00
## frontend: mean_successful_sentinel_duration_over_2h
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
monitoring: relax alerting thresholds for sentinel queries (#34106)
2022-04-19 17:01:51 +00:00
<p class="subtitle">mean successful sentinel search duration over 2h</p>
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
monitoring: relax alerting thresholds for sentinel queries (#34106)
2022-04-19 17:01:51 +00:00
- <span class="badge badge-warning">warning</span> frontend: 5s+ mean successful sentinel search duration over 2h for 15m0s
- <span class="badge badge-critical">critical</span> frontend: 8s+ mean successful sentinel search duration over 2h for 30m0s
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
monitoring: Update dashboards (#23091)
2021-07-25 16:25:33 +00:00
- Look at the breakdown by query to determine if a specific query type is being affected
- Check for high CPU usage on zoekt-webserver
- Check Honeycomb for unusual activity
monitoring: relax alerting thresholds for sentinel queries (#34106)
2022-04-19 17:01:51 +00:00
- More help interpreting this metric is available in the [dashboards reference](./dashboards.md#frontend-mean-successful-sentinel-duration-over-2h).
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
monitoring: relax alerting thresholds for sentinel queries (#34106)
2022-04-19 17:01:51 +00:00
"warning_frontend_mean_successful_sentinel_duration_over_2h",
"critical_frontend_mean_successful_sentinel_duration_over_2h"
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
]
```
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
Monitoring: update owners for code insights and batches (#59313) This updates ownership for search, code insights, and batch changes. --------- Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2024-01-04 20:18:23 +00:00
<sub>*Managed by the [Sourcegraph Code Search team](https://handbook.sourcegraph.com/departments/engineering/teams/code-search).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
Generated query for warning alert: `max((sum(rate(src_search_response_latency_seconds_sum{source=~"searchblitz.*",status="success"}[2h])) / sum(rate(src_search_response_latency_seconds_count{source=~"searchblitz.*",status="success"}[2h]))) >= 5)`
Generated query for critical alert: `max((sum(rate(src_search_response_latency_seconds_sum{source=~"searchblitz.*",status="success"}[2h])) / sum(rate(src_search_response_latency_seconds_count{source=~"searchblitz.*",status="success"}[2h]))) >= 8)`
</details>
monitoring: improve solutions formatting, add team information (#14878)
2020-10-20 05:11:20 +00:00
<br />
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: relax alerting thresholds for sentinel queries (#34106)
2022-04-19 17:01:51 +00:00
## frontend: mean_sentinel_stream_latency_over_2h
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
monitoring: relax alerting thresholds for sentinel queries (#34106)
2022-04-19 17:01:51 +00:00
<p class="subtitle">mean successful sentinel stream latency over 2h</p>
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
monitoring: relax alerting thresholds for sentinel queries (#34106)
2022-04-19 17:01:51 +00:00
- <span class="badge badge-warning">warning</span> frontend: 2s+ mean successful sentinel stream latency over 2h for 15m0s
- <span class="badge badge-critical">critical</span> frontend: 3s+ mean successful sentinel stream latency over 2h for 30m0s
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
monitoring: Update dashboards (#23091)
2021-07-25 16:25:33 +00:00
- Look at the breakdown by query to determine if a specific query type is being affected
- Check for high CPU usage on zoekt-webserver
- Check Honeycomb for unusual activity
monitoring: relax alerting thresholds for sentinel queries (#34106)
2022-04-19 17:01:51 +00:00
- More help interpreting this metric is available in the [dashboards reference](./dashboards.md#frontend-mean-sentinel-stream-latency-over-2h).
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
monitoring: relax alerting thresholds for sentinel queries (#34106)
2022-04-19 17:01:51 +00:00
"warning_frontend_mean_sentinel_stream_latency_over_2h",
"critical_frontend_mean_sentinel_stream_latency_over_2h"
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
]
```
Monitoring: update owners for code insights and batches (#59313) This updates ownership for search, code insights, and batch changes. --------- Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2024-01-04 20:18:23 +00:00
<sub>*Managed by the [Sourcegraph Code Search team](https://handbook.sourcegraph.com/departments/engineering/teams/code-search).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
Generated query for warning alert: `max((sum(rate(src_search_streaming_latency_seconds_sum{source=~"searchblitz.*"}[2h])) / sum(rate(src_search_streaming_latency_seconds_count{source=~"searchblitz.*"}[2h]))) >= 2)`
Generated query for critical alert: `max((sum(rate(src_search_streaming_latency_seconds_sum{source=~"searchblitz.*"}[2h])) / sum(rate(src_search_streaming_latency_seconds_count{source=~"searchblitz.*"}[2h]))) >= 3)`
</details>
monitoring: improve solutions formatting, add team information (#14878)
2020-10-20 05:11:20 +00:00
<br />
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: relax alerting thresholds for sentinel queries (#34106)
2022-04-19 17:01:51 +00:00
## frontend: 90th_percentile_successful_sentinel_duration_over_2h
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
monitoring: relax alerting thresholds for sentinel queries (#34106)
2022-04-19 17:01:51 +00:00
<p class="subtitle">90th percentile successful sentinel search duration over 2h</p>
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
monitoring: relax alerting thresholds for sentinel queries (#34106)
2022-04-19 17:01:51 +00:00
- <span class="badge badge-warning">warning</span> frontend: 5s+ 90th percentile successful sentinel search duration over 2h for 15m0s
- <span class="badge badge-critical">critical</span> frontend: 10s+ 90th percentile successful sentinel search duration over 2h for 3h30m0s
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
monitoring: Update dashboards (#23091)
2021-07-25 16:25:33 +00:00
- Look at the breakdown by query to determine if a specific query type is being affected
- Check for high CPU usage on zoekt-webserver
- Check Honeycomb for unusual activity
monitoring: relax alerting thresholds for sentinel queries (#34106)
2022-04-19 17:01:51 +00:00
- More help interpreting this metric is available in the [dashboards reference](./dashboards.md#frontend-90th-percentile-successful-sentinel-duration-over-2h).
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
monitoring: relax alerting thresholds for sentinel queries (#34106)
2022-04-19 17:01:51 +00:00
"warning_frontend_90th_percentile_successful_sentinel_duration_over_2h",
"critical_frontend_90th_percentile_successful_sentinel_duration_over_2h"
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
]
```
Monitoring: update owners for code insights and batches (#59313) This updates ownership for search, code insights, and batch changes. --------- Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2024-01-04 20:18:23 +00:00
<sub>*Managed by the [Sourcegraph Code Search team](https://handbook.sourcegraph.com/departments/engineering/teams/code-search).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
monitoring: extract into a submodule (#45786) This change extracts `monitoring` into a submodule for import in `sourcegraph/controller` (https://github.com/sourcegraph/controller/pull/195) so that we can generate dashboards for Cloud instances. These steps were required: 1. Initialize a `go.mod` in `monitoring` 2. Extract `dev/sg/internal/cliutil` into `lib` to avoid illegal imports from `monitoring` 3. Add local replaces to both `sourcegraph/sourcegraph` and `monitoring` 4. `go mod tidy` on all submodules 5. Update `go generate ./monitoring` commands to use `sg`, since the `go generate` command no longer works 6. Update `grafana/build.sh`, `prometheus/build.sh` to build the submodule 7. Amend linters to check for multiple `go.mod` files and ban imports of `github.com/sourcegraph/sourcegraph` 8. Update `sg generate go` to run in directories rather than from root The only caveat is that if you use VS code, you will now need to open `monitoring` in a separate workspace or similar, like with `lib`. Co-authored-by: Joe Chen <joe@sourcegraph.com>
2022-12-19 17:49:25 +00:00
Generated query for warning alert: `max((histogram_quantile(0.9, sum by (le) (label_replace(rate(src_search_response_latency_seconds_bucket{source=~"searchblitz.*",status="success"}[2h]), "source", "$1", "source", "searchblitz_(.*)")))) >= 5)`
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
monitoring: extract into a submodule (#45786) This change extracts `monitoring` into a submodule for import in `sourcegraph/controller` (https://github.com/sourcegraph/controller/pull/195) so that we can generate dashboards for Cloud instances. These steps were required: 1. Initialize a `go.mod` in `monitoring` 2. Extract `dev/sg/internal/cliutil` into `lib` to avoid illegal imports from `monitoring` 3. Add local replaces to both `sourcegraph/sourcegraph` and `monitoring` 4. `go mod tidy` on all submodules 5. Update `go generate ./monitoring` commands to use `sg`, since the `go generate` command no longer works 6. Update `grafana/build.sh`, `prometheus/build.sh` to build the submodule 7. Amend linters to check for multiple `go.mod` files and ban imports of `github.com/sourcegraph/sourcegraph` 8. Update `sg generate go` to run in directories rather than from root The only caveat is that if you use VS code, you will now need to open `monitoring` in a separate workspace or similar, like with `lib`. Co-authored-by: Joe Chen <joe@sourcegraph.com>
2022-12-19 17:49:25 +00:00
Generated query for critical alert: `max((histogram_quantile(0.9, sum by (le) (label_replace(rate(src_search_response_latency_seconds_bucket{source=~"searchblitz.*",status="success"}[2h]), "source", "$1", "source", "searchblitz_(.*)")))) >= 10)`
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
</details>
monitoring: improve solutions formatting, add team information (#14878)
2020-10-20 05:11:20 +00:00
<br />
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: relax alerting thresholds for sentinel queries (#34106)
2022-04-19 17:01:51 +00:00
## frontend: 90th_percentile_sentinel_stream_latency_over_2h
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
monitoring: relax alerting thresholds for sentinel queries (#34106)
2022-04-19 17:01:51 +00:00
<p class="subtitle">90th percentile successful sentinel stream latency over 2h</p>
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
monitoring: relax alerting thresholds for sentinel queries (#34106)
2022-04-19 17:01:51 +00:00
- <span class="badge badge-warning">warning</span> frontend: 4s+ 90th percentile successful sentinel stream latency over 2h for 15m0s
- <span class="badge badge-critical">critical</span> frontend: 6s+ 90th percentile successful sentinel stream latency over 2h for 3h30m0s
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
monitoring: Update dashboards (#23091)
2021-07-25 16:25:33 +00:00
- Look at the breakdown by query to determine if a specific query type is being affected
- Check for high CPU usage on zoekt-webserver
- Check Honeycomb for unusual activity
monitoring: relax alerting thresholds for sentinel queries (#34106)
2022-04-19 17:01:51 +00:00
- More help interpreting this metric is available in the [dashboards reference](./dashboards.md#frontend-90th-percentile-sentinel-stream-latency-over-2h).
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
monitoring: relax alerting thresholds for sentinel queries (#34106)
2022-04-19 17:01:51 +00:00
"warning_frontend_90th_percentile_sentinel_stream_latency_over_2h",
"critical_frontend_90th_percentile_sentinel_stream_latency_over_2h"
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
]
```
Monitoring: update owners for code insights and batches (#59313) This updates ownership for search, code insights, and batch changes. --------- Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2024-01-04 20:18:23 +00:00
<sub>*Managed by the [Sourcegraph Code Search team](https://handbook.sourcegraph.com/departments/engineering/teams/code-search).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: extract into a submodule (#45786) This change extracts `monitoring` into a submodule for import in `sourcegraph/controller` (https://github.com/sourcegraph/controller/pull/195) so that we can generate dashboards for Cloud instances. These steps were required: 1. Initialize a `go.mod` in `monitoring` 2. Extract `dev/sg/internal/cliutil` into `lib` to avoid illegal imports from `monitoring` 3. Add local replaces to both `sourcegraph/sourcegraph` and `monitoring` 4. `go mod tidy` on all submodules 5. Update `go generate ./monitoring` commands to use `sg`, since the `go generate` command no longer works 6. Update `grafana/build.sh`, `prometheus/build.sh` to build the submodule 7. Amend linters to check for multiple `go.mod` files and ban imports of `github.com/sourcegraph/sourcegraph` 8. Update `sg generate go` to run in directories rather than from root The only caveat is that if you use VS code, you will now need to open `monitoring` in a separate workspace or similar, like with `lib`. Co-authored-by: Joe Chen <joe@sourcegraph.com>
2022-12-19 17:49:25 +00:00
Generated query for warning alert: `max((histogram_quantile(0.9, sum by (le) (label_replace(rate(src_search_streaming_latency_seconds_bucket{source=~"searchblitz.*"}[2h]), "source", "$1", "source", "searchblitz_(.*)")))) >= 4)`
monitoring: add graphql endpoint to trigger alert, show owners (#12532) Adds a GraphQL endpoint that can be used to trigger a warning or critical alert, to help with validating `observability.alerts` configurations. Also updates the bug report alerts page to show alert owners.
2020-08-01 03:52:31 +00:00
monitoring: extract into a submodule (#45786) This change extracts `monitoring` into a submodule for import in `sourcegraph/controller` (https://github.com/sourcegraph/controller/pull/195) so that we can generate dashboards for Cloud instances. These steps were required: 1. Initialize a `go.mod` in `monitoring` 2. Extract `dev/sg/internal/cliutil` into `lib` to avoid illegal imports from `monitoring` 3. Add local replaces to both `sourcegraph/sourcegraph` and `monitoring` 4. `go mod tidy` on all submodules 5. Update `go generate ./monitoring` commands to use `sg`, since the `go generate` command no longer works 6. Update `grafana/build.sh`, `prometheus/build.sh` to build the submodule 7. Amend linters to check for multiple `go.mod` files and ban imports of `github.com/sourcegraph/sourcegraph` 8. Update `sg generate go` to run in directories rather than from root The only caveat is that if you use VS code, you will now need to open `monitoring` in a separate workspace or similar, like with `lib`. Co-authored-by: Joe Chen <joe@sourcegraph.com>
2022-12-19 17:49:25 +00:00
Generated query for critical alert: `max((histogram_quantile(0.9, sum by (le) (label_replace(rate(src_search_streaming_latency_seconds_bucket{source=~"searchblitz.*"}[2h]), "source", "$1", "source", "searchblitz_(.*)")))) >= 6)`
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
</details>
<br />
gitserver: Cleanup grafana dashboard (#60870) - Removes duplicative disk IO panels - Adds some warnings and next steps descriptions to the most critical things - Hides the backend panel by default - Adds a metric and alert for repo corruption events
2024-03-12 21:11:13 +00:00
## gitserver: cpu_throttling_time
<p class="subtitle">container CPU throttling time %</p>
**Descriptions**
- <span class="badge badge-warning">warning</span> gitserver: 75%+ container CPU throttling time % for 2m0s
- <span class="badge badge-critical">critical</span> gitserver: 90%+ container CPU throttling time % for 5m0s
**Next steps**
- - Consider increasing the CPU limit for the container.
- More help interpreting this metric is available in the [dashboards reference](./dashboards.md#gitserver-cpu-throttling-time).
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"warning_gitserver_cpu_throttling_time",
"critical_gitserver_cpu_throttling_time"
]
```
<sub>*Managed by the [Sourcegraph Source team](https://handbook.sourcegraph.com/departments/engineering/teams/source).*</sub>
<details>
<summary>Technical details</summary>
Generated query for warning alert: `max((sum by (container_label_io_kubernetes_pod_name) ((rate(container_cpu_cfs_throttled_periods_total{container_label_io_kubernetes_container_name="gitserver"}[5m]) / rate(container_cpu_cfs_periods_total{container_label_io_kubernetes_container_name="gitserver"}[5m])) * 100)) >= 75)`
Generated query for critical alert: `max((sum by (container_label_io_kubernetes_pod_name) ((rate(container_cpu_cfs_throttled_periods_total{container_label_io_kubernetes_container_name="gitserver"}[5m]) / rate(container_cpu_cfs_periods_total{container_label_io_kubernetes_container_name="gitserver"}[5m])) * 100)) >= 90)`
</details>
<br />
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
## gitserver: disk_space_remaining
gitserver: Cleanup grafana dashboard (#60870) - Removes duplicative disk IO panels - Adds some warnings and next steps descriptions to the most critical things - Hides the backend panel by default - Adds a metric and alert for repo corruption events
2024-03-12 21:11:13 +00:00
<p class="subtitle">disk space remaining</p>
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
monitoring: add graphql endpoint to trigger alert, show owners (#12532) Adds a GraphQL endpoint that can be used to trigger a warning or critical alert, to help with validating `observability.alerts` configurations. Also updates the bug report alerts page to show alert owners.
2020-08-01 03:52:31 +00:00
gitserver: Cleanup grafana dashboard (#60870) - Removes duplicative disk IO panels - Adds some warnings and next steps descriptions to the most critical things - Hides the backend panel by default - Adds a metric and alert for repo corruption events
2024-03-12 21:11:13 +00:00
- <span class="badge badge-warning">warning</span> gitserver: less than 15% disk space remaining
- <span class="badge badge-critical">critical</span> gitserver: less than 10% disk space remaining for 10m0s
monitoring: add graphql endpoint to trigger alert, show owners (#12532) Adds a GraphQL endpoint that can be used to trigger a warning or critical alert, to help with validating `observability.alerts` configurations. Also updates the bug report alerts page to show alert owners.
2020-08-01 03:52:31 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: add graphql endpoint to trigger alert, show owners (#12532) Adds a GraphQL endpoint that can be used to trigger a warning or critical alert, to help with validating `observability.alerts` configurations. Also updates the bug report alerts page to show alert owners.
2020-08-01 03:52:31 +00:00
gitserver: Cleanup grafana dashboard (#60870) - Removes duplicative disk IO panels - Adds some warnings and next steps descriptions to the most critical things - Hides the backend panel by default - Adds a metric and alert for repo corruption events
2024-03-12 21:11:13 +00:00
- On a warning alert, you may want to provision more disk space: Disk pressure may result in decreased performance, users having to wait for repositories to clone, etc.
- On a critical alert, you need to provision more disk space. Running out of disk space will result in decreased performance, or complete service outage.
monitoring/gitserver: do not alert before janitor threshold (#44768) Right now, we get a critical alert _before_ the janitor kicks in to enforce the default `SRC_REPOS_DESIRED_PERCENT_FREE`. A critical alert should only fire when the instance is in a critical state, but here the system may recover still by evicting deleted repositories, so we update the thresholds on `disk_space_remaining` such that: 1. warning fires when _approaching_ the default `SRC_REPOS_DESIRED_PERCENT_FREE` 2. critical fires if we surpass the default `SRC_REPOS_DESIRED_PERCENT_FREE` and gitserver is unable to recover in a short time span
2022-11-23 19:35:11 +00:00
- More help interpreting this metric is available in the [dashboards reference](./dashboards.md#gitserver-disk-space-remaining).
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
monitoring: Update dashboards (#23091)
2021-07-25 16:25:33 +00:00
"warning_gitserver_disk_space_remaining",
"critical_gitserver_disk_space_remaining"
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
]
```
monitoring: add graphql endpoint to trigger alert, show owners (#12532) Adds a GraphQL endpoint that can be used to trigger a warning or critical alert, to help with validating `observability.alerts` configurations. Also updates the bug report alerts page to show alert owners.
2020-08-01 03:52:31 +00:00
Update o11y ownership from IAM/repo-mgmt to Source (#54368) Does what it says in the title. ## Test plan - Ran `sg generate`
2023-06-28 13:49:16 +00:00
<sub>*Managed by the [Sourcegraph Source team](https://handbook.sourcegraph.com/departments/engineering/teams/source).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
monitoring/gitserver: do not alert before janitor threshold (#44768) Right now, we get a critical alert _before_ the janitor kicks in to enforce the default `SRC_REPOS_DESIRED_PERCENT_FREE`. A critical alert should only fire when the instance is in a critical state, but here the system may recover still by evicting deleted repositories, so we update the thresholds on `disk_space_remaining` such that: 1. warning fires when _approaching_ the default `SRC_REPOS_DESIRED_PERCENT_FREE` 2. critical fires if we surpass the default `SRC_REPOS_DESIRED_PERCENT_FREE` and gitserver is unable to recover in a short time span
2022-11-23 19:35:11 +00:00
Generated query for warning alert: `min(((src_gitserver_disk_space_available / src_gitserver_disk_space_total) * 100) < 15)`
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
monitoring/gitserver: do not alert before janitor threshold (#44768) Right now, we get a critical alert _before_ the janitor kicks in to enforce the default `SRC_REPOS_DESIRED_PERCENT_FREE`. A critical alert should only fire when the instance is in a critical state, but here the system may recover still by evicting deleted repositories, so we update the thresholds on `disk_space_remaining` such that: 1. warning fires when _approaching_ the default `SRC_REPOS_DESIRED_PERCENT_FREE` 2. critical fires if we surpass the default `SRC_REPOS_DESIRED_PERCENT_FREE` and gitserver is unable to recover in a short time span
2022-11-23 19:35:11 +00:00
Generated query for critical alert: `min(((src_gitserver_disk_space_available / src_gitserver_disk_space_total) * 100) < 10)`
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
</details>
monitoring: improve solutions formatting, add team information (#14878)
2020-10-20 05:11:20 +00:00
<br />
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: Update dashboards (#23091)
2021-07-25 16:25:33 +00:00
## gitserver: running_git_commands
monitoring: add graphql endpoint to trigger alert, show owners (#12532) Adds a GraphQL endpoint that can be used to trigger a warning or critical alert, to help with validating `observability.alerts` configurations. Also updates the bug report alerts page to show alert owners.
2020-08-01 03:52:31 +00:00
monitoring: Update dashboards (#23091)
2021-07-25 16:25:33 +00:00
<p class="subtitle">git commands running on each gitserver instance</p>
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
monitoring: add graphql endpoint to trigger alert, show owners (#12532) Adds a GraphQL endpoint that can be used to trigger a warning or critical alert, to help with validating `observability.alerts` configurations. Also updates the bug report alerts page to show alert owners.
2020-08-01 03:52:31 +00:00
monitoring: Update dashboards (#23091)
2021-07-25 16:25:33 +00:00
- <span class="badge badge-warning">warning</span> gitserver: 50+ git commands running on each gitserver instance for 2m0s
- <span class="badge badge-critical">critical</span> gitserver: 100+ git commands running on each gitserver instance for 5m0s
monitoring: add graphql endpoint to trigger alert, show owners (#12532) Adds a GraphQL endpoint that can be used to trigger a warning or critical alert, to help with validating `observability.alerts` configurations. Also updates the bug report alerts page to show alert owners.
2020-08-01 03:52:31 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: add graphql endpoint to trigger alert, show owners (#12532) Adds a GraphQL endpoint that can be used to trigger a warning or critical alert, to help with validating `observability.alerts` configurations. Also updates the bug report alerts page to show alert owners.
2020-08-01 03:52:31 +00:00
monitoring: Update dashboards (#23091)
2021-07-25 16:25:33 +00:00
- **Check if the problem may be an intermittent and temporary peak** using the "Container monitoring" section at the bottom of the Git Server dashboard.
Product Education: Rewrite for Overview, Docker Compose, and Docker Single Container to align to new IA and deployment focus (#34715) * Overall update to admin\install These changes impact the install section with an update to include a deployment overview and a reorganization of the Docker Compose and Single-container sections. Because of a directory name change, I also have updated all other links directly. * Remove old install files. Removed indexes for Docker Compose and Single Container, as well as the separate operations guides as they are now part of the main docs. * Updated admin index. Added some links to the admin main page. * Update redirects Fix to redirect links. * Update site_config.md Updated link. * Link fixes. Various links associated with directory/folder change for deployment fixed. * Various fixes based on review and link failures. Fixed Buildkite link failures and implemented issues found during review. * Update doc/admin/deploy/docker-compose/index.md Suggested edit for clarification. Co-authored-by: Michael Lin <mlzc@hey.com> * Edits to correct link issues. * Link fix. Removed sentence given redundant. * Update git_server.go Updated links to new location. * update gitserver solutions or sth * Deployment type table update. Removed checkmarks (which have varied meanings globally), and the colored circles for easy/medium/hard (which may not be accessible for some readers). * Update links for Kubernetes. Added distinct links for k8s w/ and w/out Helm. * Update doc/admin/how-to/redis_configmap.md Fix link typo. Co-authored-by: Crystal Augustus <91073224+caugustus-sourcegraph@users.noreply.github.com> * Update doc/admin/deploy/index.md Co-authored-by: Crystal Augustus <91073224+caugustus-sourcegraph@users.noreply.github.com> * Update doc/admin/deploy/index.md Co-authored-by: Crystal Augustus <91073224+caugustus-sourcegraph@users.noreply.github.com> * Update doc/admin/deploy/docker-single-container/index.md Co-authored-by: Crystal Augustus <91073224+caugustus-sourcegraph@users.noreply.github.com> * Various changes based on review. * Fix unintended update due to VS Code Extension. * Fixed markdown numbering issue. * Minor edits to correct links and deployment language. * Moved location of Cloud installation docs within Docker Compose IA * Updates based on review. Improved language related to k8s w/helm. Removed language around CPU core recommendation. * added more redirects to address link changes. Co-authored-by: Michael Lin <mlzc@hey.com> Co-authored-by: Robert Lin <robert@bobheadxi.dev> Co-authored-by: Crystal Augustus <91073224+caugustus-sourcegraph@users.noreply.github.com>
2022-05-06 16:54:45 +00:00
- **Single container deployments:** Consider upgrading to a [Docker Compose deployment](../deploy/docker-compose/migrate.md) which offers better scalability and resource isolation.
- **Kubernetes and Docker Compose:** Check that you are running a similar number of git server replicas and that their CPU/memory limits are allocated according to what is shown in the [Sourcegraph resource estimator](../deploy/resource_estimator.md).
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- More help interpreting this metric is available in the [dashboards reference](./dashboards.md#gitserver-running-git-commands).
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
monitoring: add graphql endpoint to trigger alert, show owners (#12532) Adds a GraphQL endpoint that can be used to trigger a warning or critical alert, to help with validating `observability.alerts` configurations. Also updates the bug report alerts page to show alert owners.
2020-08-01 03:52:31 +00:00
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
```json
"observability.silenceAlerts": [
monitoring: Update dashboards (#23091)
2021-07-25 16:25:33 +00:00
"warning_gitserver_running_git_commands",
"critical_gitserver_running_git_commands"
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
]
```
Update o11y ownership from IAM/repo-mgmt to Source (#54368) Does what it says in the title. ## Test plan - Ran `sg generate`
2023-06-28 13:49:16 +00:00
<sub>*Managed by the [Sourcegraph Source team](https://handbook.sourcegraph.com/departments/engineering/teams/source).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
monitoring: extract into a submodule (#45786) This change extracts `monitoring` into a submodule for import in `sourcegraph/controller` (https://github.com/sourcegraph/controller/pull/195) so that we can generate dashboards for Cloud instances. These steps were required: 1. Initialize a `go.mod` in `monitoring` 2. Extract `dev/sg/internal/cliutil` into `lib` to avoid illegal imports from `monitoring` 3. Add local replaces to both `sourcegraph/sourcegraph` and `monitoring` 4. `go mod tidy` on all submodules 5. Update `go generate ./monitoring` commands to use `sg`, since the `go generate` command no longer works 6. Update `grafana/build.sh`, `prometheus/build.sh` to build the submodule 7. Amend linters to check for multiple `go.mod` files and ban imports of `github.com/sourcegraph/sourcegraph` 8. Update `sg generate go` to run in directories rather than from root The only caveat is that if you use VS code, you will now need to open `monitoring` in a separate workspace or similar, like with `lib`. Co-authored-by: Joe Chen <joe@sourcegraph.com>
2022-12-19 17:49:25 +00:00
Generated query for warning alert: `max((sum by (instance, cmd) (src_gitserver_exec_running)) >= 50)`
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
monitoring: extract into a submodule (#45786) This change extracts `monitoring` into a submodule for import in `sourcegraph/controller` (https://github.com/sourcegraph/controller/pull/195) so that we can generate dashboards for Cloud instances. These steps were required: 1. Initialize a `go.mod` in `monitoring` 2. Extract `dev/sg/internal/cliutil` into `lib` to avoid illegal imports from `monitoring` 3. Add local replaces to both `sourcegraph/sourcegraph` and `monitoring` 4. `go mod tidy` on all submodules 5. Update `go generate ./monitoring` commands to use `sg`, since the `go generate` command no longer works 6. Update `grafana/build.sh`, `prometheus/build.sh` to build the submodule 7. Amend linters to check for multiple `go.mod` files and ban imports of `github.com/sourcegraph/sourcegraph` 8. Update `sg generate go` to run in directories rather than from root The only caveat is that if you use VS code, you will now need to open `monitoring` in a separate workspace or similar, like with `lib`. Co-authored-by: Joe Chen <joe@sourcegraph.com>
2022-12-19 17:49:25 +00:00
Generated query for critical alert: `max((sum by (instance, cmd) (src_gitserver_exec_running)) >= 100)`
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
</details>
monitoring: improve solutions formatting, add team information (#14878)
2020-10-20 05:11:20 +00:00
<br />
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
gitserver: Cleanup grafana dashboard (#60870) - Removes duplicative disk IO panels - Adds some warnings and next steps descriptions to the most critical things - Hides the backend panel by default - Adds a metric and alert for repo corruption events
2024-03-12 21:11:13 +00:00
## gitserver: echo_command_duration_test
<p class="subtitle">echo test command duration</p>
**Descriptions**
- <span class="badge badge-warning">warning</span> gitserver: 0.02s+ echo test command duration for 30s
**Next steps**
- **Single container deployments:** Upgrade to a [Docker Compose deployment](../deploy/docker-compose/migrate.md) which offers better scalability and resource isolation.
- **Kubernetes and Docker Compose:** Check that you are running a similar number of git server replicas and that their CPU/memory limits are allocated according to what is shown in the [Sourcegraph resource estimator](../deploy/resource_estimator.md).
- If your persistent volume is slow, you may want to provision more IOPS, usually by increasing the volume size.
- More help interpreting this metric is available in the [dashboards reference](./dashboards.md#gitserver-echo-command-duration-test).
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
Remove echo test critical alert (#63004) The 1s echo test alert for gitserver triggers on dotcom and doesn't have any actionable consequences, so we are removing it. The warning will remain.
2024-06-03 12:11:28 +00:00
"warning_gitserver_echo_command_duration_test"
gitserver: Cleanup grafana dashboard (#60870) - Removes duplicative disk IO panels - Adds some warnings and next steps descriptions to the most critical things - Hides the backend panel by default - Adds a metric and alert for repo corruption events
2024-03-12 21:11:13 +00:00
]
```
<sub>*Managed by the [Sourcegraph Source team](https://handbook.sourcegraph.com/departments/engineering/teams/source).*</sub>
<details>
<summary>Technical details</summary>
Generated query for warning alert: `max((max(src_gitserver_echo_duration_seconds)) >= 0.02)`
</details>
<br />
## gitserver: repo_corrupted
<p class="subtitle">number of times a repo corruption has been identified</p>
**Descriptions**
- <span class="badge badge-critical">critical</span> gitserver: 0+ number of times a repo corruption has been identified
**Next steps**
- Check the corruption logs for details. gitserver_repos.corruption_logs contains more information.
- More help interpreting this metric is available in the [dashboards reference](./dashboards.md#gitserver-repo-corrupted).
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"critical_gitserver_repo_corrupted"
]
```
<sub>*Managed by the [Sourcegraph Source team](https://handbook.sourcegraph.com/departments/engineering/teams/source).*</sub>
<details>
<summary>Technical details</summary>
Generated query for critical alert: `max((sum(rate(src_gitserver_repo_corrupted[5m]))) > 0)`
</details>
<br />
monitoring: Update dashboards (#23091)
2021-07-25 16:25:33 +00:00
## gitserver: repository_clone_queue_size
monitoring: Refactor provisioning dashboards (#23006)
2021-07-20 13:59:13 +00:00
monitoring: Update dashboards (#23091)
2021-07-25 16:25:33 +00:00
<p class="subtitle">repository clone queue size</p>
monitoring: Refactor provisioning dashboards (#23006)
2021-07-20 13:59:13 +00:00
**Descriptions**
monitoring: Update dashboards (#23091)
2021-07-25 16:25:33 +00:00
- <span class="badge badge-warning">warning</span> gitserver: 25+ repository clone queue size
monitoring: Refactor provisioning dashboards (#23006)
2021-07-20 13:59:13 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: Refactor provisioning dashboards (#23006)
2021-07-20 13:59:13 +00:00
monitoring: Update dashboards (#23091)
2021-07-25 16:25:33 +00:00
- **If you just added several repositories**, the warning may be expected.
- **Check which repositories need cloning**, by visiting e.g. https://sourcegraph.example.com/site-admin/repositories?filter=not-cloned
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#gitserver-repository-clone-queue-size).
monitoring: Refactor provisioning dashboards (#23006)
2021-07-20 13:59:13 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
monitoring: Update dashboards (#23091)
2021-07-25 16:25:33 +00:00
"warning_gitserver_repository_clone_queue_size"
monitoring: Refactor provisioning dashboards (#23006)
2021-07-20 13:59:13 +00:00
]
```
Update o11y ownership from IAM/repo-mgmt to Source (#54368) Does what it says in the title. ## Test plan - Ran `sg generate`
2023-06-28 13:49:16 +00:00
<sub>*Managed by the [Sourcegraph Source team](https://handbook.sourcegraph.com/departments/engineering/teams/source).*</sub>
monitoring: Refactor provisioning dashboards (#23006)
2021-07-20 13:59:13 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
Generated query for warning alert: `max((sum(src_gitserver_clone_queue)) >= 25)`
</details>
monitoring: Refactor provisioning dashboards (#23006)
2021-07-20 13:59:13 +00:00
<br />
conf: add metric and associated alert if clients fail to update site configuration within 5 minutes (#57682)
2023-10-18 23:53:55 +00:00
## gitserver: gitserver_site_configuration_duration_since_last_successful_update_by_instance
<p class="subtitle">maximum duration since last successful site configuration update (all "gitserver" instances)</p>
**Descriptions**
- <span class="badge badge-critical">critical</span> gitserver: 300s+ maximum duration since last successful site configuration update (all "gitserver" instances)
**Next steps**
- This indicates that one or more "gitserver" instances have not successfully updated the site configuration in over 5 minutes. This could be due to networking issues between services or problems with the site configuration service itself.
- Check for relevant errors in the "gitserver" logs, as well as frontend`s logs.
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#gitserver-gitserver-site-configuration-duration-since-last-successful-update-by-instance).
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"critical_gitserver_gitserver_site_configuration_duration_since_last_successful_update_by_instance"
]
```
monitoring: test owners for valid Opsgenie teams and handbook pages (#59251) In INC-264 it seems that certain alerts - such as [zoekt: less than 90% percentage pods available for 10m0s](https://opsg.in/a/i/sourcegraph/178a626f-0f28-4295-bee9-84da988bb473-1703759057681) - don't seem to end up going anywhere because the ObservableOwner is defunct. This change adds _opt-in_ testing to report: 1. How many owners have valid Opsgenie teams 2. How many owners have valid handbook pages In addition, we collect ObservableOwners that pass the test and use it to generate configuration for `site.json` in Sourcegraph.com: https://github.com/sourcegraph/deploy-sourcegraph-cloud/pull/18338 - this helps ensure the list is valid and not deceptively high-coverage. The results are not great, but **enforcing** that owners are valid isn't currently in scope: ``` 6/10 ObservableOwners do not have valid Opsgenie teams 3/10 ObservableOwners do not point to valid handbook pages ``` I also removed some defunct/unused functionality/owners. ## Test plan To run these tests: ``` export OPSGENIE_API_KEY="..." go test -timeout 30s github.com/sourcegraph/sourcegraph/monitoring/monitoring -update -online ```
2023-12-29 22:07:35 +00:00
<sub>*Managed by the [Sourcegraph Infrastructure Org team](https://handbook.sourcegraph.com/departments/engineering/infrastructure).*</sub>
conf: add metric and associated alert if clients fail to update site configuration within 5 minutes (#57682)
2023-10-18 23:53:55 +00:00
<details>
<summary>Technical details</summary>
monitoring: fix alert definition for site configuration by adding scrape job label (#59687) We discovered recently that the definition for the alert that fires if the site configuration hasn't been fetched within 5 minutes strips out the regex that targets individual services (since it uses a grafana variable). This means that every instance of this alert will fire if any individual service trips over this threshold. This PR fixes the issue by adding a new `job` filter for this alert that targets only the services that have that Prometheus scrape target name. This works around the previous issue by using a fixed value for the `job` value instead of a dynamic grafana value. The value of the job filter generally looks like `job=~.*$container_name` (following the strategy from https://sourcegraph.com/github.com/sourcegraph/sourcegraph@9a780f2e694238b5326e3e121d6a1828463001b9/-/blob/monitoring/monitoring/monitoring.go?L161 ) unless I noticed that there was different logic in the existing dashboard for the services. Ex: - `frontend`: already used `job=~"(sourcegraph-)?frontend"` for some metrics, so I used it again here - `worker`: `already used `job=~"^worker.*"` in some metrics, so I used it again and standarized the other existing panels to use the same shared variable ## Test plan I eyeballed the generated alert.md and dashboards.md to verify that my changes looked correct (that is, my refactors resulted in either no diff, or that the diff I generated still looked like valid regex).
2024-01-17 23:19:54 +00:00
Generated query for critical alert: `max((max(max_over_time(src_conf_client_time_since_last_successful_update_seconds{job=~".*gitserver"}[1m]))) >= 300)`
conf: add metric and associated alert if clients fail to update site configuration within 5 minutes (#57682)
2023-10-18 23:53:55 +00:00
</details>
<br />
Add alerts for mean db blocked seconds (#22822) Warn if the average database block time is over 5ms, alert if it exceeds 10ms
2021-07-16 01:21:24 +00:00
## gitserver: mean_blocked_seconds_per_conn_request
<p class="subtitle">mean blocked seconds per conn request</p>
**Descriptions**
monitoring: relax mean_blocked_seconds_per_conn_request alerts (#59507) https://github.com/sourcegraph/sourcegraph/pull/59284 dramatically reduced the `mean_blocked_seconds_per_conn_request` issues we've been seeing, but overall delays are still higher, even with generally healthy Cloud SQL resource utilization. <img width="1630" alt="image" src="https://github.com/sourcegraph/sourcegraph/assets/23356519/91615471-5187-4d15-83e7-5cc94595303c"> Spot-checking the spikes in load in Cloud SQL, it seems that there is a variety of causes for each spike (analytics workloads, Cody Gateway syncs, code intel workloads, gitserver things, `ListSourcegraphDotComIndexableRepos` etc) so I'm chalking this up to "expected". Since this alert is seen firing on a Cloud instance, let's just relax it for now so that it only fires a critical alert on very significant delays.
2024-01-11 01:14:28 +00:00
- <span class="badge badge-warning">warning</span> gitserver: 0.1s+ mean blocked seconds per conn request for 10m0s
- <span class="badge badge-critical">critical</span> gitserver: 0.5s+ mean blocked seconds per conn request for 10m0s
Add alerts for mean db blocked seconds (#22822) Warn if the average database block time is over 5ms, alert if it exceeds 10ms
2021-07-16 01:21:24 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
Add alerts for mean db blocked seconds (#22822) Warn if the average database block time is over 5ms, alert if it exceeds 10ms
2021-07-16 01:21:24 +00:00
Add possible solutions and make docs discoverable (#22945) Add possible solutions and make docs discoverable
2021-07-30 00:30:31 +00:00
- Increase SRC_PGSQL_MAX_OPEN together with giving more memory to the database if needed
Docs: update links to point to new site (#60381) We have a number of docs links in the product that point to the old doc site. Method: - Search the repo for `docs.sourcegraph.com` - Exclude the `doc/` dir, all test fixtures, and `CHANGELOG.md` - For each, replace `docs.sourcegraph.com` with `sourcegraph.com/docs` - Navigate to the resulting URL ensuring it's not a dead link, updating the URL if necessary Many of the URLs updated are just comments, but since I'm doing a manual audit of each URL anyways, I felt it was worth it to update these while I was at it.
2024-02-13 00:23:47 +00:00
- Scale up Postgres memory/cpus - [see our scaling guide](https://sourcegraph.com/docs/admin/config/postgres-conf)
monitoring: relax mean_blocked_seconds_per_conn_request alerts (#59507) https://github.com/sourcegraph/sourcegraph/pull/59284 dramatically reduced the `mean_blocked_seconds_per_conn_request` issues we've been seeing, but overall delays are still higher, even with generally healthy Cloud SQL resource utilization. <img width="1630" alt="image" src="https://github.com/sourcegraph/sourcegraph/assets/23356519/91615471-5187-4d15-83e7-5cc94595303c"> Spot-checking the spikes in load in Cloud SQL, it seems that there is a variety of causes for each spike (analytics workloads, Cody Gateway syncs, code intel workloads, gitserver things, `ListSourcegraphDotComIndexableRepos` etc) so I'm chalking this up to "expected". Since this alert is seen firing on a Cloud instance, let's just relax it for now so that it only fires a critical alert on very significant delays.
2024-01-11 01:14:28 +00:00
- If using GCP Cloud SQL, check for high lock waits or CPU usage in query insights
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#gitserver-mean-blocked-seconds-per-conn-request).
Add alerts for mean db blocked seconds (#22822) Warn if the average database block time is over 5ms, alert if it exceeds 10ms
2021-07-16 01:21:24 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"warning_gitserver_mean_blocked_seconds_per_conn_request",
"critical_gitserver_mean_blocked_seconds_per_conn_request"
]
```
monitoring: assign obvious owners to DatabaseConnectionsMonitoringGroup (#58474) updates `DatabaseConnectionsMonitoringGroup` to accept an owner - before it was just hardcoded to `ObservableOwnerDevOps`, which is not very helpful. This assigns some of the obvious service owners: 1. Source: gitserver, repo-updater 2. Cody: embeddings (but should eventually be @sourcegraph/search-platform, along with all embeddings alerts: https://github.com/sourcegraph/sourcegraph/pull/58474#issuecomment-1821505062) Source is an active owner based on [thread](https://sourcegraph.slack.com/archives/C0652SSUA20/p1700592165408089?thread_ts=1700549423.860019&cid=C0652SSUA20), and Cody is a fairly recent addition so hopefully it's valid. I'm not sure the Search one is still up-to-date, so I didn't change some of the obvious search services - for now, these still point to DevOps as they did before. If it becomes problematic we can revisit later.
2023-11-22 01:06:53 +00:00
<sub>*Managed by the [Sourcegraph Source team](https://handbook.sourcegraph.com/departments/engineering/teams/source).*</sub>
Add alerts for mean db blocked seconds (#22822) Warn if the average database block time is over 5ms, alert if it exceeds 10ms
2021-07-16 01:21:24 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
monitoring: relax mean_blocked_seconds_per_conn_request alerts (#59507) https://github.com/sourcegraph/sourcegraph/pull/59284 dramatically reduced the `mean_blocked_seconds_per_conn_request` issues we've been seeing, but overall delays are still higher, even with generally healthy Cloud SQL resource utilization. <img width="1630" alt="image" src="https://github.com/sourcegraph/sourcegraph/assets/23356519/91615471-5187-4d15-83e7-5cc94595303c"> Spot-checking the spikes in load in Cloud SQL, it seems that there is a variety of causes for each spike (analytics workloads, Cody Gateway syncs, code intel workloads, gitserver things, `ListSourcegraphDotComIndexableRepos` etc) so I'm chalking this up to "expected". Since this alert is seen firing on a Cloud instance, let's just relax it for now so that it only fires a critical alert on very significant delays.
2024-01-11 01:14:28 +00:00
Generated query for warning alert: `max((sum by (app_name, db_name) (increase(src_pgsql_conns_blocked_seconds{app_name="gitserver"}[5m])) / sum by (app_name, db_name) (increase(src_pgsql_conns_waited_for{app_name="gitserver"}[5m]))) >= 0.1)`
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
monitoring: relax mean_blocked_seconds_per_conn_request alerts (#59507) https://github.com/sourcegraph/sourcegraph/pull/59284 dramatically reduced the `mean_blocked_seconds_per_conn_request` issues we've been seeing, but overall delays are still higher, even with generally healthy Cloud SQL resource utilization. <img width="1630" alt="image" src="https://github.com/sourcegraph/sourcegraph/assets/23356519/91615471-5187-4d15-83e7-5cc94595303c"> Spot-checking the spikes in load in Cloud SQL, it seems that there is a variety of causes for each spike (analytics workloads, Cody Gateway syncs, code intel workloads, gitserver things, `ListSourcegraphDotComIndexableRepos` etc) so I'm chalking this up to "expected". Since this alert is seen firing on a Cloud instance, let's just relax it for now so that it only fires a critical alert on very significant delays.
2024-01-11 01:14:28 +00:00
Generated query for critical alert: `max((sum by (app_name, db_name) (increase(src_pgsql_conns_blocked_seconds{app_name="gitserver"}[5m])) / sum by (app_name, db_name) (increase(src_pgsql_conns_waited_for{app_name="gitserver"}[5m]))) >= 0.5)`
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
</details>
Add alerts for mean db blocked seconds (#22822) Warn if the average database block time is over 5ms, alert if it exceeds 10ms
2021-07-16 01:21:24 +00:00
<br />
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
## gitserver: container_cpu_usage
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
<p class="subtitle">container cpu usage total (1m average) across all cores by instance</p>
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
monitoring: dashboards reference, refresh observability docs (#16939) - New monitoring definition field, 'Interpretation' - New generated dashboards reference - Grafana panels now link to alert solutions and dashboards reference - Refreshed observability docs Co-authored-by: uwedeportivo <534011+uwedeportivo@users.noreply.github.com>
2020-12-23 02:36:41 +00:00
- <span class="badge badge-warning">warning</span> gitserver: 99%+ container cpu usage total (1m average) across all cores by instance
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
- **Kubernetes:** Consider increasing CPU limits in the the relevant `Deployment.yaml`.
- **Docker Compose:** Consider increasing `cpus:` of the gitserver container in `docker-compose.yml`.
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#gitserver-container-cpu-usage).
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"warning_gitserver_container_cpu_usage"
]
```
Update o11y ownership from IAM/repo-mgmt to Source (#54368) Does what it says in the title. ## Test plan - Ran `sg generate`
2023-06-28 13:49:16 +00:00
<sub>*Managed by the [Sourcegraph Source team](https://handbook.sourcegraph.com/departments/engineering/teams/source).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
Generated query for warning alert: `max((cadvisor_container_cpu_usage_percentage_total{name=~"^gitserver.*"}) >= 99)`
</details>
monitoring: improve solutions formatting, add team information (#14878)
2020-10-20 05:11:20 +00:00
<br />
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
## gitserver: container_memory_usage
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
<p class="subtitle">container memory usage by instance</p>
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
monitoring: dashboards reference, refresh observability docs (#16939) - New monitoring definition field, 'Interpretation' - New generated dashboards reference - Grafana panels now link to alert solutions and dashboards reference - Refreshed observability docs Co-authored-by: uwedeportivo <534011+uwedeportivo@users.noreply.github.com>
2020-12-23 02:36:41 +00:00
- <span class="badge badge-warning">warning</span> gitserver: 99%+ container memory usage by instance
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
- **Kubernetes:** Consider increasing memory limit in relevant `Deployment.yaml`.
- **Docker Compose:** Consider increasing `memory:` of gitserver container in `docker-compose.yml`.
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#gitserver-container-memory-usage).
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"warning_gitserver_container_memory_usage"
]
```
Update o11y ownership from IAM/repo-mgmt to Source (#54368) Does what it says in the title. ## Test plan - Ran `sg generate`
2023-06-28 13:49:16 +00:00
<sub>*Managed by the [Sourcegraph Source team](https://handbook.sourcegraph.com/departments/engineering/teams/source).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
Generated query for warning alert: `max((cadvisor_container_memory_usage_percentage_total{name=~"^gitserver.*"}) >= 99)`
</details>
monitoring: improve solutions formatting, add team information (#14878)
2020-10-20 05:11:20 +00:00
<br />
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: improve long-term provisioning alerts aggregation (#12778)
2020-08-08 06:44:33 +00:00
## gitserver: provisioning_container_cpu_usage_long_term
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
<p class="subtitle">container cpu usage total (90th percentile over 1d) across all cores by instance</p>
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
monitoring: dashboards reference, refresh observability docs (#16939) - New monitoring definition field, 'Interpretation' - New generated dashboards reference - Grafana panels now link to alert solutions and dashboards reference - Refreshed observability docs Co-authored-by: uwedeportivo <534011+uwedeportivo@users.noreply.github.com>
2020-12-23 02:36:41 +00:00
- <span class="badge badge-warning">warning</span> gitserver: 80%+ container cpu usage total (90th percentile over 1d) across all cores by instance for 336h0m0s
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
Remove container_*_usage_7d_low alerts (#16164)
2020-11-25 18:19:33 +00:00
- **Kubernetes:** Consider increasing CPU limits in the `Deployment.yaml` for the gitserver service.
- **Docker Compose:** Consider increasing `cpus:` of the gitserver container in `docker-compose.yml`.
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#gitserver-provisioning-container-cpu-usage-long-term).
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
monitoring: improve long-term provisioning alerts aggregation (#12778)
2020-08-08 06:44:33 +00:00
"warning_gitserver_provisioning_container_cpu_usage_long_term"
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
]
```
Update o11y ownership from IAM/repo-mgmt to Source (#54368) Does what it says in the title. ## Test plan - Ran `sg generate`
2023-06-28 13:49:16 +00:00
<sub>*Managed by the [Sourcegraph Source team](https://handbook.sourcegraph.com/departments/engineering/teams/source).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
Generated query for warning alert: `max((quantile_over_time(0.9, cadvisor_container_cpu_usage_percentage_total{name=~"^gitserver.*"}[1d])) >= 80)`
</details>
monitoring: improve solutions formatting, add team information (#14878)
2020-10-20 05:11:20 +00:00
<br />
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: improve long-term provisioning alerts aggregation (#12778)
2020-08-08 06:44:33 +00:00
## gitserver: provisioning_container_cpu_usage_short_term
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
<p class="subtitle">container cpu usage total (5m maximum) across all cores by instance</p>
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
monitoring: dashboards reference, refresh observability docs (#16939) - New monitoring definition field, 'Interpretation' - New generated dashboards reference - Grafana panels now link to alert solutions and dashboards reference - Refreshed observability docs Co-authored-by: uwedeportivo <534011+uwedeportivo@users.noreply.github.com>
2020-12-23 02:36:41 +00:00
- <span class="badge badge-warning">warning</span> gitserver: 90%+ container cpu usage total (5m maximum) across all cores by instance for 30m0s
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
- **Kubernetes:** Consider increasing CPU limits in the the relevant `Deployment.yaml`.
- **Docker Compose:** Consider increasing `cpus:` of the gitserver container in `docker-compose.yml`.
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#gitserver-provisioning-container-cpu-usage-short-term).
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
monitoring: improve long-term provisioning alerts aggregation (#12778)
2020-08-08 06:44:33 +00:00
"warning_gitserver_provisioning_container_cpu_usage_short_term"
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
]
```
Update o11y ownership from IAM/repo-mgmt to Source (#54368) Does what it says in the title. ## Test plan - Ran `sg generate`
2023-06-28 13:49:16 +00:00
<sub>*Managed by the [Sourcegraph Source team](https://handbook.sourcegraph.com/departments/engineering/teams/source).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
Generated query for warning alert: `max((max_over_time(cadvisor_container_cpu_usage_percentage_total{name=~"^gitserver.*"}[5m])) >= 90)`
</details>
monitoring: improve solutions formatting, add team information (#14878)
2020-10-20 05:11:20 +00:00
<br />
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: improve containers metrics (#34808) part of - https://github.com/sourcegraph/sourcegraph/issues/33437 - https://github.com/sourcegraph/sourcegraph/issues/33438 This PR adds the following - A shared observable metric to indicate how many times a container (or child processes) has been oom kill - It's added to provisioning indicators since it's an indicator of underprovisioned resources - A new dashboard that shows resources usage of all containers at a glance - It contains two groups, one for all containers and one for containers that may have potential scaling issues - the first group is extremely noisy so it's hidden by default - the second group provides a more focused experience for site-admin to quickly find problematic resources <!-- > ~it looks like the alert table will be rendered regardless, which is pretty annoying (I guess the implementation of the monitoring stack is based on the assumption that one service/container will have its dashboard)~ __Questions & Notes__ - ~Should I make the new dashboard filterable by container name? like I only want to see the graph of `frontend`. Is that something that's valuable?~ - ~Any other metrics you all would like to see?~ - graph for total available resources on the VM compared to usage from https://github.com/sourcegraph/sourcegraph/issues/33437 is not really doable since we do not have `node-exporter` in anyone of our distro. We would have to tackle this from https://github.com/sourcegraph/customer/issues/806 --> ## Test plan connect to `demo`'s prom ```sh gcloud compute start-iap-tunnel default-$NEW_DEPLOYMENT-instance 9090 --local-host-port=localhost:4445 --zone us-central1-f --project $PROJECT_PREFIX-$CUSTOMER ``` or use dogfood ```sh kubectl port-forward svc/prometheus 4445:30090 ``` Update `dev/grafana/all/datasources.yaml` ```yml # Configuration for all non-Linux platforms. apiVersion: 1 datasources: - name: Prometheus type: prometheus access: proxy url: http://docker.for.mac.localhost:4445 # use the remote prometheus instance on demo instead of the local one isDefault: true editable: false - name: Jaeger type: jaeger access: proxy url: http://host.docker.internal:16686/-/debug/jaeger - name: Loki type: loki access: proxy url: http://host.docker.internal:3100 ``` Start monitoring stack ```sh sg run grafana ``` see screenshot > the first group is hidden by default ![CleanShot 2022-05-04 at 10 48 26](https://user-images.githubusercontent.com/8373004/166748470-22460552-6e51-4b29-a26b-4c5adbe1c404.png) ![CleanShot 2022-05-04 at 10 48 03](https://user-images.githubusercontent.com/8373004/166748396-1d410c35-e5fe-45e2-83bd-418d845413cb.png) ![CleanShot 2022-05-04 at 11 25 13](https://user-images.githubusercontent.com/8373004/166801266-21e51b68-65fd-4cb7-97de-83520e31e498.png)
2022-05-09 23:56:47 +00:00
## gitserver: container_oomkill_events_total
<p class="subtitle">container OOMKILL events total by instance</p>
**Descriptions**
- <span class="badge badge-warning">warning</span> gitserver: 1+ container OOMKILL events total by instance
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: improve containers metrics (#34808) part of - https://github.com/sourcegraph/sourcegraph/issues/33437 - https://github.com/sourcegraph/sourcegraph/issues/33438 This PR adds the following - A shared observable metric to indicate how many times a container (or child processes) has been oom kill - It's added to provisioning indicators since it's an indicator of underprovisioned resources - A new dashboard that shows resources usage of all containers at a glance - It contains two groups, one for all containers and one for containers that may have potential scaling issues - the first group is extremely noisy so it's hidden by default - the second group provides a more focused experience for site-admin to quickly find problematic resources <!-- > ~it looks like the alert table will be rendered regardless, which is pretty annoying (I guess the implementation of the monitoring stack is based on the assumption that one service/container will have its dashboard)~ __Questions & Notes__ - ~Should I make the new dashboard filterable by container name? like I only want to see the graph of `frontend`. Is that something that's valuable?~ - ~Any other metrics you all would like to see?~ - graph for total available resources on the VM compared to usage from https://github.com/sourcegraph/sourcegraph/issues/33437 is not really doable since we do not have `node-exporter` in anyone of our distro. We would have to tackle this from https://github.com/sourcegraph/customer/issues/806 --> ## Test plan connect to `demo`'s prom ```sh gcloud compute start-iap-tunnel default-$NEW_DEPLOYMENT-instance 9090 --local-host-port=localhost:4445 --zone us-central1-f --project $PROJECT_PREFIX-$CUSTOMER ``` or use dogfood ```sh kubectl port-forward svc/prometheus 4445:30090 ``` Update `dev/grafana/all/datasources.yaml` ```yml # Configuration for all non-Linux platforms. apiVersion: 1 datasources: - name: Prometheus type: prometheus access: proxy url: http://docker.for.mac.localhost:4445 # use the remote prometheus instance on demo instead of the local one isDefault: true editable: false - name: Jaeger type: jaeger access: proxy url: http://host.docker.internal:16686/-/debug/jaeger - name: Loki type: loki access: proxy url: http://host.docker.internal:3100 ``` Start monitoring stack ```sh sg run grafana ``` see screenshot > the first group is hidden by default ![CleanShot 2022-05-04 at 10 48 26](https://user-images.githubusercontent.com/8373004/166748470-22460552-6e51-4b29-a26b-4c5adbe1c404.png) ![CleanShot 2022-05-04 at 10 48 03](https://user-images.githubusercontent.com/8373004/166748396-1d410c35-e5fe-45e2-83bd-418d845413cb.png) ![CleanShot 2022-05-04 at 11 25 13](https://user-images.githubusercontent.com/8373004/166801266-21e51b68-65fd-4cb7-97de-83520e31e498.png)
2022-05-09 23:56:47 +00:00
- **Kubernetes:** Consider increasing memory limit in relevant `Deployment.yaml`.
- **Docker Compose:** Consider increasing `memory:` of gitserver container in `docker-compose.yml`.
- More help interpreting this metric is available in the [dashboards reference](./dashboards.md#gitserver-container-oomkill-events-total).
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"warning_gitserver_container_oomkill_events_total"
]
```
Update o11y ownership from IAM/repo-mgmt to Source (#54368) Does what it says in the title. ## Test plan - Ran `sg generate`
2023-06-28 13:49:16 +00:00
<sub>*Managed by the [Sourcegraph Source team](https://handbook.sourcegraph.com/departments/engineering/teams/source).*</sub>
monitoring: improve containers metrics (#34808) part of - https://github.com/sourcegraph/sourcegraph/issues/33437 - https://github.com/sourcegraph/sourcegraph/issues/33438 This PR adds the following - A shared observable metric to indicate how many times a container (or child processes) has been oom kill - It's added to provisioning indicators since it's an indicator of underprovisioned resources - A new dashboard that shows resources usage of all containers at a glance - It contains two groups, one for all containers and one for containers that may have potential scaling issues - the first group is extremely noisy so it's hidden by default - the second group provides a more focused experience for site-admin to quickly find problematic resources <!-- > ~it looks like the alert table will be rendered regardless, which is pretty annoying (I guess the implementation of the monitoring stack is based on the assumption that one service/container will have its dashboard)~ __Questions & Notes__ - ~Should I make the new dashboard filterable by container name? like I only want to see the graph of `frontend`. Is that something that's valuable?~ - ~Any other metrics you all would like to see?~ - graph for total available resources on the VM compared to usage from https://github.com/sourcegraph/sourcegraph/issues/33437 is not really doable since we do not have `node-exporter` in anyone of our distro. We would have to tackle this from https://github.com/sourcegraph/customer/issues/806 --> ## Test plan connect to `demo`'s prom ```sh gcloud compute start-iap-tunnel default-$NEW_DEPLOYMENT-instance 9090 --local-host-port=localhost:4445 --zone us-central1-f --project $PROJECT_PREFIX-$CUSTOMER ``` or use dogfood ```sh kubectl port-forward svc/prometheus 4445:30090 ``` Update `dev/grafana/all/datasources.yaml` ```yml # Configuration for all non-Linux platforms. apiVersion: 1 datasources: - name: Prometheus type: prometheus access: proxy url: http://docker.for.mac.localhost:4445 # use the remote prometheus instance on demo instead of the local one isDefault: true editable: false - name: Jaeger type: jaeger access: proxy url: http://host.docker.internal:16686/-/debug/jaeger - name: Loki type: loki access: proxy url: http://host.docker.internal:3100 ``` Start monitoring stack ```sh sg run grafana ``` see screenshot > the first group is hidden by default ![CleanShot 2022-05-04 at 10 48 26](https://user-images.githubusercontent.com/8373004/166748470-22460552-6e51-4b29-a26b-4c5adbe1c404.png) ![CleanShot 2022-05-04 at 10 48 03](https://user-images.githubusercontent.com/8373004/166748396-1d410c35-e5fe-45e2-83bd-418d845413cb.png) ![CleanShot 2022-05-04 at 11 25 13](https://user-images.githubusercontent.com/8373004/166801266-21e51b68-65fd-4cb7-97de-83520e31e498.png)
2022-05-09 23:56:47 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
monitoring: extract into a submodule (#45786) This change extracts `monitoring` into a submodule for import in `sourcegraph/controller` (https://github.com/sourcegraph/controller/pull/195) so that we can generate dashboards for Cloud instances. These steps were required: 1. Initialize a `go.mod` in `monitoring` 2. Extract `dev/sg/internal/cliutil` into `lib` to avoid illegal imports from `monitoring` 3. Add local replaces to both `sourcegraph/sourcegraph` and `monitoring` 4. `go mod tidy` on all submodules 5. Update `go generate ./monitoring` commands to use `sg`, since the `go generate` command no longer works 6. Update `grafana/build.sh`, `prometheus/build.sh` to build the submodule 7. Amend linters to check for multiple `go.mod` files and ban imports of `github.com/sourcegraph/sourcegraph` 8. Update `sg generate go` to run in directories rather than from root The only caveat is that if you use VS code, you will now need to open `monitoring` in a separate workspace or similar, like with `lib`. Co-authored-by: Joe Chen <joe@sourcegraph.com>
2022-12-19 17:49:25 +00:00
Generated query for warning alert: `max((max by (name) (container_oom_events_total{name=~"^gitserver.*"})) >= 1)`
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
</details>
monitoring: improve containers metrics (#34808) part of - https://github.com/sourcegraph/sourcegraph/issues/33437 - https://github.com/sourcegraph/sourcegraph/issues/33438 This PR adds the following - A shared observable metric to indicate how many times a container (or child processes) has been oom kill - It's added to provisioning indicators since it's an indicator of underprovisioned resources - A new dashboard that shows resources usage of all containers at a glance - It contains two groups, one for all containers and one for containers that may have potential scaling issues - the first group is extremely noisy so it's hidden by default - the second group provides a more focused experience for site-admin to quickly find problematic resources <!-- > ~it looks like the alert table will be rendered regardless, which is pretty annoying (I guess the implementation of the monitoring stack is based on the assumption that one service/container will have its dashboard)~ __Questions & Notes__ - ~Should I make the new dashboard filterable by container name? like I only want to see the graph of `frontend`. Is that something that's valuable?~ - ~Any other metrics you all would like to see?~ - graph for total available resources on the VM compared to usage from https://github.com/sourcegraph/sourcegraph/issues/33437 is not really doable since we do not have `node-exporter` in anyone of our distro. We would have to tackle this from https://github.com/sourcegraph/customer/issues/806 --> ## Test plan connect to `demo`'s prom ```sh gcloud compute start-iap-tunnel default-$NEW_DEPLOYMENT-instance 9090 --local-host-port=localhost:4445 --zone us-central1-f --project $PROJECT_PREFIX-$CUSTOMER ``` or use dogfood ```sh kubectl port-forward svc/prometheus 4445:30090 ``` Update `dev/grafana/all/datasources.yaml` ```yml # Configuration for all non-Linux platforms. apiVersion: 1 datasources: - name: Prometheus type: prometheus access: proxy url: http://docker.for.mac.localhost:4445 # use the remote prometheus instance on demo instead of the local one isDefault: true editable: false - name: Jaeger type: jaeger access: proxy url: http://host.docker.internal:16686/-/debug/jaeger - name: Loki type: loki access: proxy url: http://host.docker.internal:3100 ``` Start monitoring stack ```sh sg run grafana ``` see screenshot > the first group is hidden by default ![CleanShot 2022-05-04 at 10 48 26](https://user-images.githubusercontent.com/8373004/166748470-22460552-6e51-4b29-a26b-4c5adbe1c404.png) ![CleanShot 2022-05-04 at 10 48 03](https://user-images.githubusercontent.com/8373004/166748396-1d410c35-e5fe-45e2-83bd-418d845413cb.png) ![CleanShot 2022-05-04 at 11 25 13](https://user-images.githubusercontent.com/8373004/166801266-21e51b68-65fd-4cb7-97de-83520e31e498.png)
2022-05-09 23:56:47 +00:00
<br />
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
## gitserver: go_goroutines
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
<p class="subtitle">maximum active goroutines</p>
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
monitoring: dashboards reference, refresh observability docs (#16939) - New monitoring definition field, 'Interpretation' - New generated dashboards reference - Grafana panels now link to alert solutions and dashboards reference - Refreshed observability docs Co-authored-by: uwedeportivo <534011+uwedeportivo@users.noreply.github.com>
2020-12-23 02:36:41 +00:00
- <span class="badge badge-warning">warning</span> gitserver: 10000+ maximum active goroutines for 10m0s
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- More help interpreting this metric is available in the [dashboards reference](./dashboards.md#gitserver-go-goroutines).
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"warning_gitserver_go_goroutines"
]
```
Update o11y ownership from IAM/repo-mgmt to Source (#54368) Does what it says in the title. ## Test plan - Ran `sg generate`
2023-06-28 13:49:16 +00:00
<sub>*Managed by the [Sourcegraph Source team](https://handbook.sourcegraph.com/departments/engineering/teams/source).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
monitoring: extract into a submodule (#45786) This change extracts `monitoring` into a submodule for import in `sourcegraph/controller` (https://github.com/sourcegraph/controller/pull/195) so that we can generate dashboards for Cloud instances. These steps were required: 1. Initialize a `go.mod` in `monitoring` 2. Extract `dev/sg/internal/cliutil` into `lib` to avoid illegal imports from `monitoring` 3. Add local replaces to both `sourcegraph/sourcegraph` and `monitoring` 4. `go mod tidy` on all submodules 5. Update `go generate ./monitoring` commands to use `sg`, since the `go generate` command no longer works 6. Update `grafana/build.sh`, `prometheus/build.sh` to build the submodule 7. Amend linters to check for multiple `go.mod` files and ban imports of `github.com/sourcegraph/sourcegraph` 8. Update `sg generate go` to run in directories rather than from root The only caveat is that if you use VS code, you will now need to open `monitoring` in a separate workspace or similar, like with `lib`. Co-authored-by: Joe Chen <joe@sourcegraph.com>
2022-12-19 17:49:25 +00:00
Generated query for warning alert: `max((max by (instance) (go_goroutines{job=~".*gitserver"})) >= 10000)`
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
</details>
monitoring: improve solutions formatting, add team information (#14878)
2020-10-20 05:11:20 +00:00
<br />
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
## gitserver: go_gc_duration_seconds
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
<p class="subtitle">maximum go garbage collection duration</p>
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
monitoring: dashboards reference, refresh observability docs (#16939) - New monitoring definition field, 'Interpretation' - New generated dashboards reference - Grafana panels now link to alert solutions and dashboards reference - Refreshed observability docs Co-authored-by: uwedeportivo <534011+uwedeportivo@users.noreply.github.com>
2020-12-23 02:36:41 +00:00
- <span class="badge badge-warning">warning</span> gitserver: 2s+ maximum go garbage collection duration
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#gitserver-go-gc-duration-seconds).
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"warning_gitserver_go_gc_duration_seconds"
]
```
Update o11y ownership from IAM/repo-mgmt to Source (#54368) Does what it says in the title. ## Test plan - Ran `sg generate`
2023-06-28 13:49:16 +00:00
<sub>*Managed by the [Sourcegraph Source team](https://handbook.sourcegraph.com/departments/engineering/teams/source).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
monitoring: extract into a submodule (#45786) This change extracts `monitoring` into a submodule for import in `sourcegraph/controller` (https://github.com/sourcegraph/controller/pull/195) so that we can generate dashboards for Cloud instances. These steps were required: 1. Initialize a `go.mod` in `monitoring` 2. Extract `dev/sg/internal/cliutil` into `lib` to avoid illegal imports from `monitoring` 3. Add local replaces to both `sourcegraph/sourcegraph` and `monitoring` 4. `go mod tidy` on all submodules 5. Update `go generate ./monitoring` commands to use `sg`, since the `go generate` command no longer works 6. Update `grafana/build.sh`, `prometheus/build.sh` to build the submodule 7. Amend linters to check for multiple `go.mod` files and ban imports of `github.com/sourcegraph/sourcegraph` 8. Update `sg generate go` to run in directories rather than from root The only caveat is that if you use VS code, you will now need to open `monitoring` in a separate workspace or similar, like with `lib`. Co-authored-by: Joe Chen <joe@sourcegraph.com>
2022-12-19 17:49:25 +00:00
Generated query for warning alert: `max((max by (instance) (go_gc_duration_seconds{job=~".*gitserver"})) >= 2)`
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
</details>
monitoring: improve solutions formatting, add team information (#14878)
2020-10-20 05:11:20 +00:00
<br />
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
## gitserver: pods_available_percentage
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
<p class="subtitle">percentage pods available</p>
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
monitoring: dashboards reference, refresh observability docs (#16939) - New monitoring definition field, 'Interpretation' - New generated dashboards reference - Grafana panels now link to alert solutions and dashboards reference - Refreshed observability docs Co-authored-by: uwedeportivo <534011+uwedeportivo@users.noreply.github.com>
2020-12-23 02:36:41 +00:00
- <span class="badge badge-critical">critical</span> gitserver: less than 90% percentage pods available for 10m0s
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
monitoring: forbid critical alerts with no solutions (#36423) Adds the restriction that critical alerts must have solutions defined as a hard requirement within the generator, and fixes all the outstanding issues: - Add PossibleSolutions for the pods_available_percentage and pg_up critical alerts (we have a similar solution doc in the container_missing alert) - Removes all the other critical alerts without possible solutions Co-authored-by: Michael Lin <mlzc@hey.com>
2022-06-02 15:51:43 +00:00
- Determine if the pod was OOM killed using `kubectl describe pod gitserver` (look for `OOMKilled: true`) and, if so, consider increasing the memory limit in the relevant `Deployment.yaml`.
- Check the logs before the container restarted to see if there are `panic:` messages or similar using `kubectl logs -p gitserver`.
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#gitserver-pods-available-percentage).
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"critical_gitserver_pods_available_percentage"
]
```
Update o11y ownership from IAM/repo-mgmt to Source (#54368) Does what it says in the title. ## Test plan - Ran `sg generate`
2023-06-28 13:49:16 +00:00
<sub>*Managed by the [Sourcegraph Source team](https://handbook.sourcegraph.com/departments/engineering/teams/source).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
monitoring: extract into a submodule (#45786) This change extracts `monitoring` into a submodule for import in `sourcegraph/controller` (https://github.com/sourcegraph/controller/pull/195) so that we can generate dashboards for Cloud instances. These steps were required: 1. Initialize a `go.mod` in `monitoring` 2. Extract `dev/sg/internal/cliutil` into `lib` to avoid illegal imports from `monitoring` 3. Add local replaces to both `sourcegraph/sourcegraph` and `monitoring` 4. `go mod tidy` on all submodules 5. Update `go generate ./monitoring` commands to use `sg`, since the `go generate` command no longer works 6. Update `grafana/build.sh`, `prometheus/build.sh` to build the submodule 7. Amend linters to check for multiple `go.mod` files and ban imports of `github.com/sourcegraph/sourcegraph` 8. Update `sg generate go` to run in directories rather than from root The only caveat is that if you use VS code, you will now need to open `monitoring` in a separate workspace or similar, like with `lib`. Co-authored-by: Joe Chen <joe@sourcegraph.com>
2022-12-19 17:49:25 +00:00
Generated query for critical alert: `min((sum by (app) (up{app=~".*gitserver"}) / count by (app) (up{app=~".*gitserver"}) * 100) <= 90)`
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
</details>
monitoring: improve solutions formatting, add team information (#14878)
2020-10-20 05:11:20 +00:00
<br />
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
Postgres grafana dashboards (#15641) Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2020-12-11 15:53:01 +00:00
## postgres: connections
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
<p class="subtitle">active connections</p>
Postgres grafana dashboards (#15641) Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2020-12-11 15:53:01 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
Postgres grafana dashboards (#15641) Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2020-12-11 15:53:01 +00:00
monitoring(postgres): fix panels, adjust transaction alerts (#17131)
2021-01-11 15:31:45 +00:00
- <span class="badge badge-warning">warning</span> postgres: less than 5 active connections for 5m0s
Postgres grafana dashboards (#15641) Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2020-12-11 15:53:01 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
Postgres grafana dashboards (#15641) Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2020-12-11 15:53:01 +00:00
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#postgres-connections).
Postgres grafana dashboards (#15641) Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2020-12-11 15:53:01 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"warning_postgres_connections"
]
```
monitoring: test owners for valid Opsgenie teams and handbook pages (#59251) In INC-264 it seems that certain alerts - such as [zoekt: less than 90% percentage pods available for 10m0s](https://opsg.in/a/i/sourcegraph/178a626f-0f28-4295-bee9-84da988bb473-1703759057681) - don't seem to end up going anywhere because the ObservableOwner is defunct. This change adds _opt-in_ testing to report: 1. How many owners have valid Opsgenie teams 2. How many owners have valid handbook pages In addition, we collect ObservableOwners that pass the test and use it to generate configuration for `site.json` in Sourcegraph.com: https://github.com/sourcegraph/deploy-sourcegraph-cloud/pull/18338 - this helps ensure the list is valid and not deceptively high-coverage. The results are not great, but **enforcing** that owners are valid isn't currently in scope: ``` 6/10 ObservableOwners do not have valid Opsgenie teams 3/10 ObservableOwners do not point to valid handbook pages ``` I also removed some defunct/unused functionality/owners. ## Test plan To run these tests: ``` export OPSGENIE_API_KEY="..." go test -timeout 30s github.com/sourcegraph/sourcegraph/monitoring/monitoring -update -online ```
2023-12-29 22:07:35 +00:00
<sub>*Managed by the [Sourcegraph Infrastructure Org team](https://handbook.sourcegraph.com/departments/engineering/infrastructure).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
monitoring: extract into a submodule (#45786) This change extracts `monitoring` into a submodule for import in `sourcegraph/controller` (https://github.com/sourcegraph/controller/pull/195) so that we can generate dashboards for Cloud instances. These steps were required: 1. Initialize a `go.mod` in `monitoring` 2. Extract `dev/sg/internal/cliutil` into `lib` to avoid illegal imports from `monitoring` 3. Add local replaces to both `sourcegraph/sourcegraph` and `monitoring` 4. `go mod tidy` on all submodules 5. Update `go generate ./monitoring` commands to use `sg`, since the `go generate` command no longer works 6. Update `grafana/build.sh`, `prometheus/build.sh` to build the submodule 7. Amend linters to check for multiple `go.mod` files and ban imports of `github.com/sourcegraph/sourcegraph` 8. Update `sg generate go` to run in directories rather than from root The only caveat is that if you use VS code, you will now need to open `monitoring` in a separate workspace or similar, like with `lib`. Co-authored-by: Joe Chen <joe@sourcegraph.com>
2022-12-19 17:49:25 +00:00
Generated query for warning alert: `min((sum by (job) (pg_stat_activity_count{datname!~"template.*|postgres|cloudsqladmin"}) or sum by (job) (pg_stat_activity_count{datname!~"template.*|cloudsqladmin",job="codeinsights-db"})) <= 5)`
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
</details>
Postgres grafana dashboards (#15641) Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2020-12-11 15:53:01 +00:00
<br />
monitoring: add pg connection usage (#34310)
2022-04-22 17:46:24 +00:00
## postgres: usage_connections_percentage
<p class="subtitle">connection in use</p>
**Descriptions**
- <span class="badge badge-warning">warning</span> postgres: 80%+ connection in use for 5m0s
- <span class="badge badge-critical">critical</span> postgres: 100%+ connection in use for 5m0s
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: add pg connection usage (#34310)
2022-04-22 17:46:24 +00:00
Docs: update links to point to new site (#60381) We have a number of docs links in the product that point to the old doc site. Method: - Search the repo for `docs.sourcegraph.com` - Exclude the `doc/` dir, all test fixtures, and `CHANGELOG.md` - For each, replace `docs.sourcegraph.com` with `sourcegraph.com/docs` - Navigate to the resulting URL ensuring it's not a dead link, updating the URL if necessary Many of the URLs updated are just comments, but since I'm doing a manual audit of each URL anyways, I felt it was worth it to update these while I was at it.
2024-02-13 00:23:47 +00:00
- Consider increasing [max_connections](https://www.postgresql.org/docs/current/runtime-config-connection.html#GUC-MAX-CONNECTIONS) of the database instance, [learn more](https://sourcegraph.com/docs/admin/config/postgres-conf)
monitoring: add pg connection usage (#34310)
2022-04-22 17:46:24 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#postgres-usage-connections-percentage).
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"warning_postgres_usage_connections_percentage",
"critical_postgres_usage_connections_percentage"
]
```
monitoring: test owners for valid Opsgenie teams and handbook pages (#59251) In INC-264 it seems that certain alerts - such as [zoekt: less than 90% percentage pods available for 10m0s](https://opsg.in/a/i/sourcegraph/178a626f-0f28-4295-bee9-84da988bb473-1703759057681) - don't seem to end up going anywhere because the ObservableOwner is defunct. This change adds _opt-in_ testing to report: 1. How many owners have valid Opsgenie teams 2. How many owners have valid handbook pages In addition, we collect ObservableOwners that pass the test and use it to generate configuration for `site.json` in Sourcegraph.com: https://github.com/sourcegraph/deploy-sourcegraph-cloud/pull/18338 - this helps ensure the list is valid and not deceptively high-coverage. The results are not great, but **enforcing** that owners are valid isn't currently in scope: ``` 6/10 ObservableOwners do not have valid Opsgenie teams 3/10 ObservableOwners do not point to valid handbook pages ``` I also removed some defunct/unused functionality/owners. ## Test plan To run these tests: ``` export OPSGENIE_API_KEY="..." go test -timeout 30s github.com/sourcegraph/sourcegraph/monitoring/monitoring -update -online ```
2023-12-29 22:07:35 +00:00
<sub>*Managed by the [Sourcegraph Infrastructure Org team](https://handbook.sourcegraph.com/departments/engineering/infrastructure).*</sub>
monitoring: add pg connection usage (#34310)
2022-04-22 17:46:24 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
monitoring: extract into a submodule (#45786) This change extracts `monitoring` into a submodule for import in `sourcegraph/controller` (https://github.com/sourcegraph/controller/pull/195) so that we can generate dashboards for Cloud instances. These steps were required: 1. Initialize a `go.mod` in `monitoring` 2. Extract `dev/sg/internal/cliutil` into `lib` to avoid illegal imports from `monitoring` 3. Add local replaces to both `sourcegraph/sourcegraph` and `monitoring` 4. `go mod tidy` on all submodules 5. Update `go generate ./monitoring` commands to use `sg`, since the `go generate` command no longer works 6. Update `grafana/build.sh`, `prometheus/build.sh` to build the submodule 7. Amend linters to check for multiple `go.mod` files and ban imports of `github.com/sourcegraph/sourcegraph` 8. Update `sg generate go` to run in directories rather than from root The only caveat is that if you use VS code, you will now need to open `monitoring` in a separate workspace or similar, like with `lib`. Co-authored-by: Joe Chen <joe@sourcegraph.com>
2022-12-19 17:49:25 +00:00
Generated query for warning alert: `max((sum by (job) (pg_stat_activity_count) / (sum by (job) (pg_settings_max_connections) - sum by (job) (pg_settings_superuser_reserved_connections)) * 100) >= 80)`
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
monitoring: extract into a submodule (#45786) This change extracts `monitoring` into a submodule for import in `sourcegraph/controller` (https://github.com/sourcegraph/controller/pull/195) so that we can generate dashboards for Cloud instances. These steps were required: 1. Initialize a `go.mod` in `monitoring` 2. Extract `dev/sg/internal/cliutil` into `lib` to avoid illegal imports from `monitoring` 3. Add local replaces to both `sourcegraph/sourcegraph` and `monitoring` 4. `go mod tidy` on all submodules 5. Update `go generate ./monitoring` commands to use `sg`, since the `go generate` command no longer works 6. Update `grafana/build.sh`, `prometheus/build.sh` to build the submodule 7. Amend linters to check for multiple `go.mod` files and ban imports of `github.com/sourcegraph/sourcegraph` 8. Update `sg generate go` to run in directories rather than from root The only caveat is that if you use VS code, you will now need to open `monitoring` in a separate workspace or similar, like with `lib`. Co-authored-by: Joe Chen <joe@sourcegraph.com>
2022-12-19 17:49:25 +00:00
Generated query for critical alert: `max((sum by (job) (pg_stat_activity_count) / (sum by (job) (pg_settings_max_connections) - sum by (job) (pg_settings_superuser_reserved_connections)) * 100) >= 100)`
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
</details>
monitoring: add pg connection usage (#34310)
2022-04-22 17:46:24 +00:00
<br />
monitoring(postgres): fix panels, adjust transaction alerts (#17131)
2021-01-11 15:31:45 +00:00
## postgres: transaction_durations
Postgres grafana dashboards (#15641) Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2020-12-11 15:53:01 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
<p class="subtitle">maximum transaction durations</p>
Postgres grafana dashboards (#15641) Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2020-12-11 15:53:01 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
Postgres grafana dashboards (#15641) Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2020-12-11 15:53:01 +00:00
update units of pg_stat_activity_max_tx_duration (#27294) The units were listed as milliseconds, but should actually be interpreted as seconds according to the exporter code here: https://sourcegraph.com/github.com/prometheus-community/postgres_exporter/-/blob/cmd/postgres_exporter/postgres_exporter.go?L305:5
2021-11-09 16:56:21 +00:00
- <span class="badge badge-warning">warning</span> postgres: 0.3s+ maximum transaction durations for 5m0s
Postgres grafana dashboards (#15641) Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2020-12-11 15:53:01 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
Postgres grafana dashboards (#15641) Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2020-12-11 15:53:01 +00:00
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#postgres-transaction-durations).
Postgres grafana dashboards (#15641) Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2020-12-11 15:53:01 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
monitoring: forbid critical alerts with no solutions (#36423) Adds the restriction that critical alerts must have solutions defined as a hard requirement within the generator, and fixes all the outstanding issues: - Add PossibleSolutions for the pods_available_percentage and pg_up critical alerts (we have a similar solution doc in the container_missing alert) - Removes all the other critical alerts without possible solutions Co-authored-by: Michael Lin <mlzc@hey.com>
2022-06-02 15:51:43 +00:00
"warning_postgres_transaction_durations"
Postgres grafana dashboards (#15641) Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2020-12-11 15:53:01 +00:00
]
```
monitoring: test owners for valid Opsgenie teams and handbook pages (#59251) In INC-264 it seems that certain alerts - such as [zoekt: less than 90% percentage pods available for 10m0s](https://opsg.in/a/i/sourcegraph/178a626f-0f28-4295-bee9-84da988bb473-1703759057681) - don't seem to end up going anywhere because the ObservableOwner is defunct. This change adds _opt-in_ testing to report: 1. How many owners have valid Opsgenie teams 2. How many owners have valid handbook pages In addition, we collect ObservableOwners that pass the test and use it to generate configuration for `site.json` in Sourcegraph.com: https://github.com/sourcegraph/deploy-sourcegraph-cloud/pull/18338 - this helps ensure the list is valid and not deceptively high-coverage. The results are not great, but **enforcing** that owners are valid isn't currently in scope: ``` 6/10 ObservableOwners do not have valid Opsgenie teams 3/10 ObservableOwners do not point to valid handbook pages ``` I also removed some defunct/unused functionality/owners. ## Test plan To run these tests: ``` export OPSGENIE_API_KEY="..." go test -timeout 30s github.com/sourcegraph/sourcegraph/monitoring/monitoring -update -online ```
2023-12-29 22:07:35 +00:00
<sub>*Managed by the [Sourcegraph Infrastructure Org team](https://handbook.sourcegraph.com/departments/engineering/infrastructure).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
monitoring: extract into a submodule (#45786) This change extracts `monitoring` into a submodule for import in `sourcegraph/controller` (https://github.com/sourcegraph/controller/pull/195) so that we can generate dashboards for Cloud instances. These steps were required: 1. Initialize a `go.mod` in `monitoring` 2. Extract `dev/sg/internal/cliutil` into `lib` to avoid illegal imports from `monitoring` 3. Add local replaces to both `sourcegraph/sourcegraph` and `monitoring` 4. `go mod tidy` on all submodules 5. Update `go generate ./monitoring` commands to use `sg`, since the `go generate` command no longer works 6. Update `grafana/build.sh`, `prometheus/build.sh` to build the submodule 7. Amend linters to check for multiple `go.mod` files and ban imports of `github.com/sourcegraph/sourcegraph` 8. Update `sg generate go` to run in directories rather than from root The only caveat is that if you use VS code, you will now need to open `monitoring` in a separate workspace or similar, like with `lib`. Co-authored-by: Joe Chen <joe@sourcegraph.com>
2022-12-19 17:49:25 +00:00
Generated query for warning alert: `max((sum by (job) (pg_stat_activity_max_tx_duration{datname!~"template.*|postgres|cloudsqladmin",job!="codeintel-db"}) or sum by (job) (pg_stat_activity_max_tx_duration{datname!~"template.*|cloudsqladmin",job="codeinsights-db"})) >= 0.3)`
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
</details>
Postgres grafana dashboards (#15641) Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2020-12-11 15:53:01 +00:00
<br />
## postgres: postgres_up
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
<p class="subtitle">database availability</p>
Postgres grafana dashboards (#15641) Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2020-12-11 15:53:01 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
Postgres grafana dashboards (#15641) Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2020-12-11 15:53:01 +00:00
monitoring(postgres): fix panels, adjust transaction alerts (#17131)
2021-01-11 15:31:45 +00:00
- <span class="badge badge-critical">critical</span> postgres: less than 0 database availability for 5m0s
Postgres grafana dashboards (#15641) Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2020-12-11 15:53:01 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
Postgres grafana dashboards (#15641) Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2020-12-11 15:53:01 +00:00
monitoring: forbid critical alerts with no solutions (#36423) Adds the restriction that critical alerts must have solutions defined as a hard requirement within the generator, and fixes all the outstanding issues: - Add PossibleSolutions for the pods_available_percentage and pg_up critical alerts (we have a similar solution doc in the container_missing alert) - Removes all the other critical alerts without possible solutions Co-authored-by: Michael Lin <mlzc@hey.com>
2022-06-02 15:51:43 +00:00
- **Kubernetes:**
- Determine if the pod was OOM killed using `kubectl describe pod (pgsql|codeintel-db|codeinsights)` (look for `OOMKilled: true`) and, if so, consider increasing the memory limit in the relevant `Deployment.yaml`.
- Check the logs before the container restarted to see if there are `panic:` messages or similar using `kubectl logs -p (pgsql|codeintel-db|codeinsights)`.
- Check if there is any OOMKILL event using the provisioning panels
- Check kernel logs using `dmesg` for OOMKILL events on worker nodes
- **Docker Compose:**
- Determine if the pod was OOM killed using `docker inspect -f '{{json .State}}' (pgsql|codeintel-db|codeinsights)` (look for `"OOMKilled":true`) and, if so, consider increasing the memory limit of the (pgsql|codeintel-db|codeinsights) container in `docker-compose.yml`.
- Check the logs before the container restarted to see if there are `panic:` messages or similar using `docker logs (pgsql|codeintel-db|codeinsights)` (note this will include logs from the previous and currently running container).
- Check if there is any OOMKILL event using the provisioning panels
- Check kernel logs using `dmesg` for OOMKILL events
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- More help interpreting this metric is available in the [dashboards reference](./dashboards.md#postgres-postgres-up).
Postgres grafana dashboards (#15641) Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2020-12-11 15:53:01 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"critical_postgres_postgres_up"
]
```
monitoring: test owners for valid Opsgenie teams and handbook pages (#59251) In INC-264 it seems that certain alerts - such as [zoekt: less than 90% percentage pods available for 10m0s](https://opsg.in/a/i/sourcegraph/178a626f-0f28-4295-bee9-84da988bb473-1703759057681) - don't seem to end up going anywhere because the ObservableOwner is defunct. This change adds _opt-in_ testing to report: 1. How many owners have valid Opsgenie teams 2. How many owners have valid handbook pages In addition, we collect ObservableOwners that pass the test and use it to generate configuration for `site.json` in Sourcegraph.com: https://github.com/sourcegraph/deploy-sourcegraph-cloud/pull/18338 - this helps ensure the list is valid and not deceptively high-coverage. The results are not great, but **enforcing** that owners are valid isn't currently in scope: ``` 6/10 ObservableOwners do not have valid Opsgenie teams 3/10 ObservableOwners do not point to valid handbook pages ``` I also removed some defunct/unused functionality/owners. ## Test plan To run these tests: ``` export OPSGENIE_API_KEY="..." go test -timeout 30s github.com/sourcegraph/sourcegraph/monitoring/monitoring -update -online ```
2023-12-29 22:07:35 +00:00
<sub>*Managed by the [Sourcegraph Infrastructure Org team](https://handbook.sourcegraph.com/departments/engineering/infrastructure).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
Generated query for critical alert: `min((pg_up) <= 0)`
</details>
Postgres grafana dashboards (#15641) Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2020-12-11 15:53:01 +00:00
<br />
monitoring: Alert on invalid indexes (#19694)
2021-04-06 14:14:03 +00:00
## postgres: invalid_indexes
<p class="subtitle">invalid indexes (unusable by the query planner)</p>
**Descriptions**
- <span class="badge badge-critical">critical</span> postgres: 1+ invalid indexes (unusable by the query planner)
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: Alert on invalid indexes (#19694)
2021-04-06 14:14:03 +00:00
Regenerate two md files (#47183)
2023-01-31 13:25:45 +00:00
- Drop and re-create the invalid trigger - please contact Sourcegraph to supply the trigger definition.
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- More help interpreting this metric is available in the [dashboards reference](./dashboards.md#postgres-invalid-indexes).
monitoring: Alert on invalid indexes (#19694)
2021-04-06 14:14:03 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"critical_postgres_invalid_indexes"
]
```
monitoring: test owners for valid Opsgenie teams and handbook pages (#59251) In INC-264 it seems that certain alerts - such as [zoekt: less than 90% percentage pods available for 10m0s](https://opsg.in/a/i/sourcegraph/178a626f-0f28-4295-bee9-84da988bb473-1703759057681) - don't seem to end up going anywhere because the ObservableOwner is defunct. This change adds _opt-in_ testing to report: 1. How many owners have valid Opsgenie teams 2. How many owners have valid handbook pages In addition, we collect ObservableOwners that pass the test and use it to generate configuration for `site.json` in Sourcegraph.com: https://github.com/sourcegraph/deploy-sourcegraph-cloud/pull/18338 - this helps ensure the list is valid and not deceptively high-coverage. The results are not great, but **enforcing** that owners are valid isn't currently in scope: ``` 6/10 ObservableOwners do not have valid Opsgenie teams 3/10 ObservableOwners do not point to valid handbook pages ``` I also removed some defunct/unused functionality/owners. ## Test plan To run these tests: ``` export OPSGENIE_API_KEY="..." go test -timeout 30s github.com/sourcegraph/sourcegraph/monitoring/monitoring -update -online ```
2023-12-29 22:07:35 +00:00
<sub>*Managed by the [Sourcegraph Infrastructure Org team](https://handbook.sourcegraph.com/departments/engineering/infrastructure).*</sub>
monitoring: Alert on invalid indexes (#19694)
2021-04-06 14:14:03 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
monitoring: extract into a submodule (#45786) This change extracts `monitoring` into a submodule for import in `sourcegraph/controller` (https://github.com/sourcegraph/controller/pull/195) so that we can generate dashboards for Cloud instances. These steps were required: 1. Initialize a `go.mod` in `monitoring` 2. Extract `dev/sg/internal/cliutil` into `lib` to avoid illegal imports from `monitoring` 3. Add local replaces to both `sourcegraph/sourcegraph` and `monitoring` 4. `go mod tidy` on all submodules 5. Update `go generate ./monitoring` commands to use `sg`, since the `go generate` command no longer works 6. Update `grafana/build.sh`, `prometheus/build.sh` to build the submodule 7. Amend linters to check for multiple `go.mod` files and ban imports of `github.com/sourcegraph/sourcegraph` 8. Update `sg generate go` to run in directories rather than from root The only caveat is that if you use VS code, you will now need to open `monitoring` in a separate workspace or similar, like with `lib`. Co-authored-by: Joe Chen <joe@sourcegraph.com>
2022-12-19 17:49:25 +00:00
Generated query for critical alert: `sum((max by (relname) (pg_invalid_index_count)) >= 1)`
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
</details>
monitoring: Alert on invalid indexes (#19694)
2021-04-06 14:14:03 +00:00
<br />
Postgres grafana dashboards (#15641) Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2020-12-11 15:53:01 +00:00
## postgres: pg_exporter_err
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
<p class="subtitle">errors scraping postgres exporter</p>
Postgres grafana dashboards (#15641) Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2020-12-11 15:53:01 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
Postgres grafana dashboards (#15641) Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2020-12-11 15:53:01 +00:00
monitoring: dashboards reference, refresh observability docs (#16939) - New monitoring definition field, 'Interpretation' - New generated dashboards reference - Grafana panels now link to alert solutions and dashboards reference - Refreshed observability docs Co-authored-by: uwedeportivo <534011+uwedeportivo@users.noreply.github.com>
2020-12-23 02:36:41 +00:00
- <span class="badge badge-warning">warning</span> postgres: 1+ errors scraping postgres exporter for 5m0s
Postgres grafana dashboards (#15641) Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2020-12-11 15:53:01 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
Postgres grafana dashboards (#15641) Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2020-12-11 15:53:01 +00:00
Update postgres monitoring with solutions (#17189) Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2021-01-15 01:50:05 +00:00
- Ensure the Postgres exporter can access the Postgres database. Also, check the Postgres exporter logs for errors.
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- More help interpreting this metric is available in the [dashboards reference](./dashboards.md#postgres-pg-exporter-err).
Postgres grafana dashboards (#15641) Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2020-12-11 15:53:01 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"warning_postgres_pg_exporter_err"
]
```
monitoring: test owners for valid Opsgenie teams and handbook pages (#59251) In INC-264 it seems that certain alerts - such as [zoekt: less than 90% percentage pods available for 10m0s](https://opsg.in/a/i/sourcegraph/178a626f-0f28-4295-bee9-84da988bb473-1703759057681) - don't seem to end up going anywhere because the ObservableOwner is defunct. This change adds _opt-in_ testing to report: 1. How many owners have valid Opsgenie teams 2. How many owners have valid handbook pages In addition, we collect ObservableOwners that pass the test and use it to generate configuration for `site.json` in Sourcegraph.com: https://github.com/sourcegraph/deploy-sourcegraph-cloud/pull/18338 - this helps ensure the list is valid and not deceptively high-coverage. The results are not great, but **enforcing** that owners are valid isn't currently in scope: ``` 6/10 ObservableOwners do not have valid Opsgenie teams 3/10 ObservableOwners do not point to valid handbook pages ``` I also removed some defunct/unused functionality/owners. ## Test plan To run these tests: ``` export OPSGENIE_API_KEY="..." go test -timeout 30s github.com/sourcegraph/sourcegraph/monitoring/monitoring -update -online ```
2023-12-29 22:07:35 +00:00
<sub>*Managed by the [Sourcegraph Infrastructure Org team](https://handbook.sourcegraph.com/departments/engineering/infrastructure).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
Generated query for warning alert: `max((pg_exporter_last_scrape_error) >= 1)`
</details>
Postgres grafana dashboards (#15641) Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2020-12-11 15:53:01 +00:00
<br />
## postgres: migration_in_progress
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
<p class="subtitle">active schema migration</p>
Postgres grafana dashboards (#15641) Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2020-12-11 15:53:01 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
Postgres grafana dashboards (#15641) Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2020-12-11 15:53:01 +00:00
monitoring(postgres): fix panels, adjust transaction alerts (#17131)
2021-01-11 15:31:45 +00:00
- <span class="badge badge-critical">critical</span> postgres: 1+ active schema migration for 5m0s
Postgres grafana dashboards (#15641) Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2020-12-11 15:53:01 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
Postgres grafana dashboards (#15641) Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2020-12-11 15:53:01 +00:00
Regenerate two md files (#47183)
2023-01-31 13:25:45 +00:00
- The database migration has been in progress for 5 or more minutes - please contact Sourcegraph if this persists.
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- More help interpreting this metric is available in the [dashboards reference](./dashboards.md#postgres-migration-in-progress).
Postgres grafana dashboards (#15641) Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2020-12-11 15:53:01 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"critical_postgres_migration_in_progress"
]
```
monitoring: test owners for valid Opsgenie teams and handbook pages (#59251) In INC-264 it seems that certain alerts - such as [zoekt: less than 90% percentage pods available for 10m0s](https://opsg.in/a/i/sourcegraph/178a626f-0f28-4295-bee9-84da988bb473-1703759057681) - don't seem to end up going anywhere because the ObservableOwner is defunct. This change adds _opt-in_ testing to report: 1. How many owners have valid Opsgenie teams 2. How many owners have valid handbook pages In addition, we collect ObservableOwners that pass the test and use it to generate configuration for `site.json` in Sourcegraph.com: https://github.com/sourcegraph/deploy-sourcegraph-cloud/pull/18338 - this helps ensure the list is valid and not deceptively high-coverage. The results are not great, but **enforcing** that owners are valid isn't currently in scope: ``` 6/10 ObservableOwners do not have valid Opsgenie teams 3/10 ObservableOwners do not point to valid handbook pages ``` I also removed some defunct/unused functionality/owners. ## Test plan To run these tests: ``` export OPSGENIE_API_KEY="..." go test -timeout 30s github.com/sourcegraph/sourcegraph/monitoring/monitoring -update -online ```
2023-12-29 22:07:35 +00:00
<sub>*Managed by the [Sourcegraph Infrastructure Org team](https://handbook.sourcegraph.com/departments/engineering/infrastructure).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
Generated query for critical alert: `max((pg_sg_migration_status) >= 1)`
</details>
Postgres grafana dashboards (#15641) Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2020-12-11 15:53:01 +00:00
<br />
## postgres: provisioning_container_cpu_usage_long_term
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
<p class="subtitle">container cpu usage total (90th percentile over 1d) across all cores by instance</p>
Postgres grafana dashboards (#15641) Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2020-12-11 15:53:01 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
Postgres grafana dashboards (#15641) Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2020-12-11 15:53:01 +00:00
monitoring: dashboards reference, refresh observability docs (#16939) - New monitoring definition field, 'Interpretation' - New generated dashboards reference - Grafana panels now link to alert solutions and dashboards reference - Refreshed observability docs Co-authored-by: uwedeportivo <534011+uwedeportivo@users.noreply.github.com>
2020-12-23 02:36:41 +00:00
- <span class="badge badge-warning">warning</span> postgres: 80%+ container cpu usage total (90th percentile over 1d) across all cores by instance for 336h0m0s
Postgres grafana dashboards (#15641) Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2020-12-11 15:53:01 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
Postgres grafana dashboards (#15641) Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2020-12-11 15:53:01 +00:00
monitoring: improve containers metrics (#34808) part of - https://github.com/sourcegraph/sourcegraph/issues/33437 - https://github.com/sourcegraph/sourcegraph/issues/33438 This PR adds the following - A shared observable metric to indicate how many times a container (or child processes) has been oom kill - It's added to provisioning indicators since it's an indicator of underprovisioned resources - A new dashboard that shows resources usage of all containers at a glance - It contains two groups, one for all containers and one for containers that may have potential scaling issues - the first group is extremely noisy so it's hidden by default - the second group provides a more focused experience for site-admin to quickly find problematic resources <!-- > ~it looks like the alert table will be rendered regardless, which is pretty annoying (I guess the implementation of the monitoring stack is based on the assumption that one service/container will have its dashboard)~ __Questions & Notes__ - ~Should I make the new dashboard filterable by container name? like I only want to see the graph of `frontend`. Is that something that's valuable?~ - ~Any other metrics you all would like to see?~ - graph for total available resources on the VM compared to usage from https://github.com/sourcegraph/sourcegraph/issues/33437 is not really doable since we do not have `node-exporter` in anyone of our distro. We would have to tackle this from https://github.com/sourcegraph/customer/issues/806 --> ## Test plan connect to `demo`'s prom ```sh gcloud compute start-iap-tunnel default-$NEW_DEPLOYMENT-instance 9090 --local-host-port=localhost:4445 --zone us-central1-f --project $PROJECT_PREFIX-$CUSTOMER ``` or use dogfood ```sh kubectl port-forward svc/prometheus 4445:30090 ``` Update `dev/grafana/all/datasources.yaml` ```yml # Configuration for all non-Linux platforms. apiVersion: 1 datasources: - name: Prometheus type: prometheus access: proxy url: http://docker.for.mac.localhost:4445 # use the remote prometheus instance on demo instead of the local one isDefault: true editable: false - name: Jaeger type: jaeger access: proxy url: http://host.docker.internal:16686/-/debug/jaeger - name: Loki type: loki access: proxy url: http://host.docker.internal:3100 ``` Start monitoring stack ```sh sg run grafana ``` see screenshot > the first group is hidden by default ![CleanShot 2022-05-04 at 10 48 26](https://user-images.githubusercontent.com/8373004/166748470-22460552-6e51-4b29-a26b-4c5adbe1c404.png) ![CleanShot 2022-05-04 at 10 48 03](https://user-images.githubusercontent.com/8373004/166748396-1d410c35-e5fe-45e2-83bd-418d845413cb.png) ![CleanShot 2022-05-04 at 11 25 13](https://user-images.githubusercontent.com/8373004/166801266-21e51b68-65fd-4cb7-97de-83520e31e498.png)
2022-05-09 23:56:47 +00:00
- **Kubernetes:** Consider increasing CPU limits in the `Deployment.yaml` for the (pgsql|codeintel-db|codeinsights) service.
- **Docker Compose:** Consider increasing `cpus:` of the (pgsql|codeintel-db|codeinsights) container in `docker-compose.yml`.
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#postgres-provisioning-container-cpu-usage-long-term).
Postgres grafana dashboards (#15641) Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2020-12-11 15:53:01 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"warning_postgres_provisioning_container_cpu_usage_long_term"
]
```
monitoring: test owners for valid Opsgenie teams and handbook pages (#59251) In INC-264 it seems that certain alerts - such as [zoekt: less than 90% percentage pods available for 10m0s](https://opsg.in/a/i/sourcegraph/178a626f-0f28-4295-bee9-84da988bb473-1703759057681) - don't seem to end up going anywhere because the ObservableOwner is defunct. This change adds _opt-in_ testing to report: 1. How many owners have valid Opsgenie teams 2. How many owners have valid handbook pages In addition, we collect ObservableOwners that pass the test and use it to generate configuration for `site.json` in Sourcegraph.com: https://github.com/sourcegraph/deploy-sourcegraph-cloud/pull/18338 - this helps ensure the list is valid and not deceptively high-coverage. The results are not great, but **enforcing** that owners are valid isn't currently in scope: ``` 6/10 ObservableOwners do not have valid Opsgenie teams 3/10 ObservableOwners do not point to valid handbook pages ``` I also removed some defunct/unused functionality/owners. ## Test plan To run these tests: ``` export OPSGENIE_API_KEY="..." go test -timeout 30s github.com/sourcegraph/sourcegraph/monitoring/monitoring -update -online ```
2023-12-29 22:07:35 +00:00
<sub>*Managed by the [Sourcegraph Infrastructure Org team](https://handbook.sourcegraph.com/departments/engineering/infrastructure).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
Generated query for warning alert: `max((quantile_over_time(0.9, cadvisor_container_cpu_usage_percentage_total{name=~"^(pgsql|codeintel-db|codeinsights).*"}[1d])) >= 80)`
</details>
Postgres grafana dashboards (#15641) Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2020-12-11 15:53:01 +00:00
<br />
## postgres: provisioning_container_memory_usage_long_term
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
<p class="subtitle">container memory usage (1d maximum) by instance</p>
Postgres grafana dashboards (#15641) Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2020-12-11 15:53:01 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
Postgres grafana dashboards (#15641) Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2020-12-11 15:53:01 +00:00
monitoring: dashboards reference, refresh observability docs (#16939) - New monitoring definition field, 'Interpretation' - New generated dashboards reference - Grafana panels now link to alert solutions and dashboards reference - Refreshed observability docs Co-authored-by: uwedeportivo <534011+uwedeportivo@users.noreply.github.com>
2020-12-23 02:36:41 +00:00
- <span class="badge badge-warning">warning</span> postgres: 80%+ container memory usage (1d maximum) by instance for 336h0m0s
Postgres grafana dashboards (#15641) Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2020-12-11 15:53:01 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
Postgres grafana dashboards (#15641) Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2020-12-11 15:53:01 +00:00
monitoring: improve containers metrics (#34808) part of - https://github.com/sourcegraph/sourcegraph/issues/33437 - https://github.com/sourcegraph/sourcegraph/issues/33438 This PR adds the following - A shared observable metric to indicate how many times a container (or child processes) has been oom kill - It's added to provisioning indicators since it's an indicator of underprovisioned resources - A new dashboard that shows resources usage of all containers at a glance - It contains two groups, one for all containers and one for containers that may have potential scaling issues - the first group is extremely noisy so it's hidden by default - the second group provides a more focused experience for site-admin to quickly find problematic resources <!-- > ~it looks like the alert table will be rendered regardless, which is pretty annoying (I guess the implementation of the monitoring stack is based on the assumption that one service/container will have its dashboard)~ __Questions & Notes__ - ~Should I make the new dashboard filterable by container name? like I only want to see the graph of `frontend`. Is that something that's valuable?~ - ~Any other metrics you all would like to see?~ - graph for total available resources on the VM compared to usage from https://github.com/sourcegraph/sourcegraph/issues/33437 is not really doable since we do not have `node-exporter` in anyone of our distro. We would have to tackle this from https://github.com/sourcegraph/customer/issues/806 --> ## Test plan connect to `demo`'s prom ```sh gcloud compute start-iap-tunnel default-$NEW_DEPLOYMENT-instance 9090 --local-host-port=localhost:4445 --zone us-central1-f --project $PROJECT_PREFIX-$CUSTOMER ``` or use dogfood ```sh kubectl port-forward svc/prometheus 4445:30090 ``` Update `dev/grafana/all/datasources.yaml` ```yml # Configuration for all non-Linux platforms. apiVersion: 1 datasources: - name: Prometheus type: prometheus access: proxy url: http://docker.for.mac.localhost:4445 # use the remote prometheus instance on demo instead of the local one isDefault: true editable: false - name: Jaeger type: jaeger access: proxy url: http://host.docker.internal:16686/-/debug/jaeger - name: Loki type: loki access: proxy url: http://host.docker.internal:3100 ``` Start monitoring stack ```sh sg run grafana ``` see screenshot > the first group is hidden by default ![CleanShot 2022-05-04 at 10 48 26](https://user-images.githubusercontent.com/8373004/166748470-22460552-6e51-4b29-a26b-4c5adbe1c404.png) ![CleanShot 2022-05-04 at 10 48 03](https://user-images.githubusercontent.com/8373004/166748396-1d410c35-e5fe-45e2-83bd-418d845413cb.png) ![CleanShot 2022-05-04 at 11 25 13](https://user-images.githubusercontent.com/8373004/166801266-21e51b68-65fd-4cb7-97de-83520e31e498.png)
2022-05-09 23:56:47 +00:00
- **Kubernetes:** Consider increasing memory limits in the `Deployment.yaml` for the (pgsql|codeintel-db|codeinsights) service.
- **Docker Compose:** Consider increasing `memory:` of the (pgsql|codeintel-db|codeinsights) container in `docker-compose.yml`.
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#postgres-provisioning-container-memory-usage-long-term).
Postgres grafana dashboards (#15641) Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2020-12-11 15:53:01 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"warning_postgres_provisioning_container_memory_usage_long_term"
]
```
monitoring: test owners for valid Opsgenie teams and handbook pages (#59251) In INC-264 it seems that certain alerts - such as [zoekt: less than 90% percentage pods available for 10m0s](https://opsg.in/a/i/sourcegraph/178a626f-0f28-4295-bee9-84da988bb473-1703759057681) - don't seem to end up going anywhere because the ObservableOwner is defunct. This change adds _opt-in_ testing to report: 1. How many owners have valid Opsgenie teams 2. How many owners have valid handbook pages In addition, we collect ObservableOwners that pass the test and use it to generate configuration for `site.json` in Sourcegraph.com: https://github.com/sourcegraph/deploy-sourcegraph-cloud/pull/18338 - this helps ensure the list is valid and not deceptively high-coverage. The results are not great, but **enforcing** that owners are valid isn't currently in scope: ``` 6/10 ObservableOwners do not have valid Opsgenie teams 3/10 ObservableOwners do not point to valid handbook pages ``` I also removed some defunct/unused functionality/owners. ## Test plan To run these tests: ``` export OPSGENIE_API_KEY="..." go test -timeout 30s github.com/sourcegraph/sourcegraph/monitoring/monitoring -update -online ```
2023-12-29 22:07:35 +00:00
<sub>*Managed by the [Sourcegraph Infrastructure Org team](https://handbook.sourcegraph.com/departments/engineering/infrastructure).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
Generated query for warning alert: `max((max_over_time(cadvisor_container_memory_usage_percentage_total{name=~"^(pgsql|codeintel-db|codeinsights).*"}[1d])) >= 80)`
</details>
Postgres grafana dashboards (#15641) Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2020-12-11 15:53:01 +00:00
<br />
## postgres: provisioning_container_cpu_usage_short_term
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
<p class="subtitle">container cpu usage total (5m maximum) across all cores by instance</p>
Postgres grafana dashboards (#15641) Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2020-12-11 15:53:01 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
Postgres grafana dashboards (#15641) Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2020-12-11 15:53:01 +00:00
monitoring: dashboards reference, refresh observability docs (#16939) - New monitoring definition field, 'Interpretation' - New generated dashboards reference - Grafana panels now link to alert solutions and dashboards reference - Refreshed observability docs Co-authored-by: uwedeportivo <534011+uwedeportivo@users.noreply.github.com>
2020-12-23 02:36:41 +00:00
- <span class="badge badge-warning">warning</span> postgres: 90%+ container cpu usage total (5m maximum) across all cores by instance for 30m0s
Postgres grafana dashboards (#15641) Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2020-12-11 15:53:01 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
Postgres grafana dashboards (#15641) Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2020-12-11 15:53:01 +00:00
- **Kubernetes:** Consider increasing CPU limits in the the relevant `Deployment.yaml`.
monitoring: improve containers metrics (#34808) part of - https://github.com/sourcegraph/sourcegraph/issues/33437 - https://github.com/sourcegraph/sourcegraph/issues/33438 This PR adds the following - A shared observable metric to indicate how many times a container (or child processes) has been oom kill - It's added to provisioning indicators since it's an indicator of underprovisioned resources - A new dashboard that shows resources usage of all containers at a glance - It contains two groups, one for all containers and one for containers that may have potential scaling issues - the first group is extremely noisy so it's hidden by default - the second group provides a more focused experience for site-admin to quickly find problematic resources <!-- > ~it looks like the alert table will be rendered regardless, which is pretty annoying (I guess the implementation of the monitoring stack is based on the assumption that one service/container will have its dashboard)~ __Questions & Notes__ - ~Should I make the new dashboard filterable by container name? like I only want to see the graph of `frontend`. Is that something that's valuable?~ - ~Any other metrics you all would like to see?~ - graph for total available resources on the VM compared to usage from https://github.com/sourcegraph/sourcegraph/issues/33437 is not really doable since we do not have `node-exporter` in anyone of our distro. We would have to tackle this from https://github.com/sourcegraph/customer/issues/806 --> ## Test plan connect to `demo`'s prom ```sh gcloud compute start-iap-tunnel default-$NEW_DEPLOYMENT-instance 9090 --local-host-port=localhost:4445 --zone us-central1-f --project $PROJECT_PREFIX-$CUSTOMER ``` or use dogfood ```sh kubectl port-forward svc/prometheus 4445:30090 ``` Update `dev/grafana/all/datasources.yaml` ```yml # Configuration for all non-Linux platforms. apiVersion: 1 datasources: - name: Prometheus type: prometheus access: proxy url: http://docker.for.mac.localhost:4445 # use the remote prometheus instance on demo instead of the local one isDefault: true editable: false - name: Jaeger type: jaeger access: proxy url: http://host.docker.internal:16686/-/debug/jaeger - name: Loki type: loki access: proxy url: http://host.docker.internal:3100 ``` Start monitoring stack ```sh sg run grafana ``` see screenshot > the first group is hidden by default ![CleanShot 2022-05-04 at 10 48 26](https://user-images.githubusercontent.com/8373004/166748470-22460552-6e51-4b29-a26b-4c5adbe1c404.png) ![CleanShot 2022-05-04 at 10 48 03](https://user-images.githubusercontent.com/8373004/166748396-1d410c35-e5fe-45e2-83bd-418d845413cb.png) ![CleanShot 2022-05-04 at 11 25 13](https://user-images.githubusercontent.com/8373004/166801266-21e51b68-65fd-4cb7-97de-83520e31e498.png)
2022-05-09 23:56:47 +00:00
- **Docker Compose:** Consider increasing `cpus:` of the (pgsql|codeintel-db|codeinsights) container in `docker-compose.yml`.
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#postgres-provisioning-container-cpu-usage-short-term).
Postgres grafana dashboards (#15641) Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2020-12-11 15:53:01 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"warning_postgres_provisioning_container_cpu_usage_short_term"
]
```
monitoring: test owners for valid Opsgenie teams and handbook pages (#59251) In INC-264 it seems that certain alerts - such as [zoekt: less than 90% percentage pods available for 10m0s](https://opsg.in/a/i/sourcegraph/178a626f-0f28-4295-bee9-84da988bb473-1703759057681) - don't seem to end up going anywhere because the ObservableOwner is defunct. This change adds _opt-in_ testing to report: 1. How many owners have valid Opsgenie teams 2. How many owners have valid handbook pages In addition, we collect ObservableOwners that pass the test and use it to generate configuration for `site.json` in Sourcegraph.com: https://github.com/sourcegraph/deploy-sourcegraph-cloud/pull/18338 - this helps ensure the list is valid and not deceptively high-coverage. The results are not great, but **enforcing** that owners are valid isn't currently in scope: ``` 6/10 ObservableOwners do not have valid Opsgenie teams 3/10 ObservableOwners do not point to valid handbook pages ``` I also removed some defunct/unused functionality/owners. ## Test plan To run these tests: ``` export OPSGENIE_API_KEY="..." go test -timeout 30s github.com/sourcegraph/sourcegraph/monitoring/monitoring -update -online ```
2023-12-29 22:07:35 +00:00
<sub>*Managed by the [Sourcegraph Infrastructure Org team](https://handbook.sourcegraph.com/departments/engineering/infrastructure).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
Generated query for warning alert: `max((max_over_time(cadvisor_container_cpu_usage_percentage_total{name=~"^(pgsql|codeintel-db|codeinsights).*"}[5m])) >= 90)`
</details>
Postgres grafana dashboards (#15641) Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2020-12-11 15:53:01 +00:00
<br />
## postgres: provisioning_container_memory_usage_short_term
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
<p class="subtitle">container memory usage (5m maximum) by instance</p>
Postgres grafana dashboards (#15641) Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2020-12-11 15:53:01 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
Postgres grafana dashboards (#15641) Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2020-12-11 15:53:01 +00:00
monitoring: dashboards reference, refresh observability docs (#16939) - New monitoring definition field, 'Interpretation' - New generated dashboards reference - Grafana panels now link to alert solutions and dashboards reference - Refreshed observability docs Co-authored-by: uwedeportivo <534011+uwedeportivo@users.noreply.github.com>
2020-12-23 02:36:41 +00:00
- <span class="badge badge-warning">warning</span> postgres: 90%+ container memory usage (5m maximum) by instance
Postgres grafana dashboards (#15641) Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2020-12-11 15:53:01 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
Postgres grafana dashboards (#15641) Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2020-12-11 15:53:01 +00:00
- **Kubernetes:** Consider increasing memory limit in relevant `Deployment.yaml`.
monitoring: improve containers metrics (#34808) part of - https://github.com/sourcegraph/sourcegraph/issues/33437 - https://github.com/sourcegraph/sourcegraph/issues/33438 This PR adds the following - A shared observable metric to indicate how many times a container (or child processes) has been oom kill - It's added to provisioning indicators since it's an indicator of underprovisioned resources - A new dashboard that shows resources usage of all containers at a glance - It contains two groups, one for all containers and one for containers that may have potential scaling issues - the first group is extremely noisy so it's hidden by default - the second group provides a more focused experience for site-admin to quickly find problematic resources <!-- > ~it looks like the alert table will be rendered regardless, which is pretty annoying (I guess the implementation of the monitoring stack is based on the assumption that one service/container will have its dashboard)~ __Questions & Notes__ - ~Should I make the new dashboard filterable by container name? like I only want to see the graph of `frontend`. Is that something that's valuable?~ - ~Any other metrics you all would like to see?~ - graph for total available resources on the VM compared to usage from https://github.com/sourcegraph/sourcegraph/issues/33437 is not really doable since we do not have `node-exporter` in anyone of our distro. We would have to tackle this from https://github.com/sourcegraph/customer/issues/806 --> ## Test plan connect to `demo`'s prom ```sh gcloud compute start-iap-tunnel default-$NEW_DEPLOYMENT-instance 9090 --local-host-port=localhost:4445 --zone us-central1-f --project $PROJECT_PREFIX-$CUSTOMER ``` or use dogfood ```sh kubectl port-forward svc/prometheus 4445:30090 ``` Update `dev/grafana/all/datasources.yaml` ```yml # Configuration for all non-Linux platforms. apiVersion: 1 datasources: - name: Prometheus type: prometheus access: proxy url: http://docker.for.mac.localhost:4445 # use the remote prometheus instance on demo instead of the local one isDefault: true editable: false - name: Jaeger type: jaeger access: proxy url: http://host.docker.internal:16686/-/debug/jaeger - name: Loki type: loki access: proxy url: http://host.docker.internal:3100 ``` Start monitoring stack ```sh sg run grafana ``` see screenshot > the first group is hidden by default ![CleanShot 2022-05-04 at 10 48 26](https://user-images.githubusercontent.com/8373004/166748470-22460552-6e51-4b29-a26b-4c5adbe1c404.png) ![CleanShot 2022-05-04 at 10 48 03](https://user-images.githubusercontent.com/8373004/166748396-1d410c35-e5fe-45e2-83bd-418d845413cb.png) ![CleanShot 2022-05-04 at 11 25 13](https://user-images.githubusercontent.com/8373004/166801266-21e51b68-65fd-4cb7-97de-83520e31e498.png)
2022-05-09 23:56:47 +00:00
- **Docker Compose:** Consider increasing `memory:` of (pgsql|codeintel-db|codeinsights) container in `docker-compose.yml`.
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#postgres-provisioning-container-memory-usage-short-term).
Postgres grafana dashboards (#15641) Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2020-12-11 15:53:01 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"warning_postgres_provisioning_container_memory_usage_short_term"
]
```
monitoring: test owners for valid Opsgenie teams and handbook pages (#59251) In INC-264 it seems that certain alerts - such as [zoekt: less than 90% percentage pods available for 10m0s](https://opsg.in/a/i/sourcegraph/178a626f-0f28-4295-bee9-84da988bb473-1703759057681) - don't seem to end up going anywhere because the ObservableOwner is defunct. This change adds _opt-in_ testing to report: 1. How many owners have valid Opsgenie teams 2. How many owners have valid handbook pages In addition, we collect ObservableOwners that pass the test and use it to generate configuration for `site.json` in Sourcegraph.com: https://github.com/sourcegraph/deploy-sourcegraph-cloud/pull/18338 - this helps ensure the list is valid and not deceptively high-coverage. The results are not great, but **enforcing** that owners are valid isn't currently in scope: ``` 6/10 ObservableOwners do not have valid Opsgenie teams 3/10 ObservableOwners do not point to valid handbook pages ``` I also removed some defunct/unused functionality/owners. ## Test plan To run these tests: ``` export OPSGENIE_API_KEY="..." go test -timeout 30s github.com/sourcegraph/sourcegraph/monitoring/monitoring -update -online ```
2023-12-29 22:07:35 +00:00
<sub>*Managed by the [Sourcegraph Infrastructure Org team](https://handbook.sourcegraph.com/departments/engineering/infrastructure).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
Generated query for warning alert: `max((max_over_time(cadvisor_container_memory_usage_percentage_total{name=~"^(pgsql|codeintel-db|codeinsights).*"}[5m])) >= 90)`
</details>
monitoring: improve solutions formatting, add team information (#14878)
2020-10-20 05:11:20 +00:00
<br />
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: improve containers metrics (#34808) part of - https://github.com/sourcegraph/sourcegraph/issues/33437 - https://github.com/sourcegraph/sourcegraph/issues/33438 This PR adds the following - A shared observable metric to indicate how many times a container (or child processes) has been oom kill - It's added to provisioning indicators since it's an indicator of underprovisioned resources - A new dashboard that shows resources usage of all containers at a glance - It contains two groups, one for all containers and one for containers that may have potential scaling issues - the first group is extremely noisy so it's hidden by default - the second group provides a more focused experience for site-admin to quickly find problematic resources <!-- > ~it looks like the alert table will be rendered regardless, which is pretty annoying (I guess the implementation of the monitoring stack is based on the assumption that one service/container will have its dashboard)~ __Questions & Notes__ - ~Should I make the new dashboard filterable by container name? like I only want to see the graph of `frontend`. Is that something that's valuable?~ - ~Any other metrics you all would like to see?~ - graph for total available resources on the VM compared to usage from https://github.com/sourcegraph/sourcegraph/issues/33437 is not really doable since we do not have `node-exporter` in anyone of our distro. We would have to tackle this from https://github.com/sourcegraph/customer/issues/806 --> ## Test plan connect to `demo`'s prom ```sh gcloud compute start-iap-tunnel default-$NEW_DEPLOYMENT-instance 9090 --local-host-port=localhost:4445 --zone us-central1-f --project $PROJECT_PREFIX-$CUSTOMER ``` or use dogfood ```sh kubectl port-forward svc/prometheus 4445:30090 ``` Update `dev/grafana/all/datasources.yaml` ```yml # Configuration for all non-Linux platforms. apiVersion: 1 datasources: - name: Prometheus type: prometheus access: proxy url: http://docker.for.mac.localhost:4445 # use the remote prometheus instance on demo instead of the local one isDefault: true editable: false - name: Jaeger type: jaeger access: proxy url: http://host.docker.internal:16686/-/debug/jaeger - name: Loki type: loki access: proxy url: http://host.docker.internal:3100 ``` Start monitoring stack ```sh sg run grafana ``` see screenshot > the first group is hidden by default ![CleanShot 2022-05-04 at 10 48 26](https://user-images.githubusercontent.com/8373004/166748470-22460552-6e51-4b29-a26b-4c5adbe1c404.png) ![CleanShot 2022-05-04 at 10 48 03](https://user-images.githubusercontent.com/8373004/166748396-1d410c35-e5fe-45e2-83bd-418d845413cb.png) ![CleanShot 2022-05-04 at 11 25 13](https://user-images.githubusercontent.com/8373004/166801266-21e51b68-65fd-4cb7-97de-83520e31e498.png)
2022-05-09 23:56:47 +00:00
## postgres: container_oomkill_events_total
<p class="subtitle">container OOMKILL events total by instance</p>
**Descriptions**
- <span class="badge badge-warning">warning</span> postgres: 1+ container OOMKILL events total by instance
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: improve containers metrics (#34808) part of - https://github.com/sourcegraph/sourcegraph/issues/33437 - https://github.com/sourcegraph/sourcegraph/issues/33438 This PR adds the following - A shared observable metric to indicate how many times a container (or child processes) has been oom kill - It's added to provisioning indicators since it's an indicator of underprovisioned resources - A new dashboard that shows resources usage of all containers at a glance - It contains two groups, one for all containers and one for containers that may have potential scaling issues - the first group is extremely noisy so it's hidden by default - the second group provides a more focused experience for site-admin to quickly find problematic resources <!-- > ~it looks like the alert table will be rendered regardless, which is pretty annoying (I guess the implementation of the monitoring stack is based on the assumption that one service/container will have its dashboard)~ __Questions & Notes__ - ~Should I make the new dashboard filterable by container name? like I only want to see the graph of `frontend`. Is that something that's valuable?~ - ~Any other metrics you all would like to see?~ - graph for total available resources on the VM compared to usage from https://github.com/sourcegraph/sourcegraph/issues/33437 is not really doable since we do not have `node-exporter` in anyone of our distro. We would have to tackle this from https://github.com/sourcegraph/customer/issues/806 --> ## Test plan connect to `demo`'s prom ```sh gcloud compute start-iap-tunnel default-$NEW_DEPLOYMENT-instance 9090 --local-host-port=localhost:4445 --zone us-central1-f --project $PROJECT_PREFIX-$CUSTOMER ``` or use dogfood ```sh kubectl port-forward svc/prometheus 4445:30090 ``` Update `dev/grafana/all/datasources.yaml` ```yml # Configuration for all non-Linux platforms. apiVersion: 1 datasources: - name: Prometheus type: prometheus access: proxy url: http://docker.for.mac.localhost:4445 # use the remote prometheus instance on demo instead of the local one isDefault: true editable: false - name: Jaeger type: jaeger access: proxy url: http://host.docker.internal:16686/-/debug/jaeger - name: Loki type: loki access: proxy url: http://host.docker.internal:3100 ``` Start monitoring stack ```sh sg run grafana ``` see screenshot > the first group is hidden by default ![CleanShot 2022-05-04 at 10 48 26](https://user-images.githubusercontent.com/8373004/166748470-22460552-6e51-4b29-a26b-4c5adbe1c404.png) ![CleanShot 2022-05-04 at 10 48 03](https://user-images.githubusercontent.com/8373004/166748396-1d410c35-e5fe-45e2-83bd-418d845413cb.png) ![CleanShot 2022-05-04 at 11 25 13](https://user-images.githubusercontent.com/8373004/166801266-21e51b68-65fd-4cb7-97de-83520e31e498.png)
2022-05-09 23:56:47 +00:00
- **Kubernetes:** Consider increasing memory limit in relevant `Deployment.yaml`.
- **Docker Compose:** Consider increasing `memory:` of (pgsql|codeintel-db|codeinsights) container in `docker-compose.yml`.
- More help interpreting this metric is available in the [dashboards reference](./dashboards.md#postgres-container-oomkill-events-total).
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"warning_postgres_container_oomkill_events_total"
]
```
monitoring: test owners for valid Opsgenie teams and handbook pages (#59251) In INC-264 it seems that certain alerts - such as [zoekt: less than 90% percentage pods available for 10m0s](https://opsg.in/a/i/sourcegraph/178a626f-0f28-4295-bee9-84da988bb473-1703759057681) - don't seem to end up going anywhere because the ObservableOwner is defunct. This change adds _opt-in_ testing to report: 1. How many owners have valid Opsgenie teams 2. How many owners have valid handbook pages In addition, we collect ObservableOwners that pass the test and use it to generate configuration for `site.json` in Sourcegraph.com: https://github.com/sourcegraph/deploy-sourcegraph-cloud/pull/18338 - this helps ensure the list is valid and not deceptively high-coverage. The results are not great, but **enforcing** that owners are valid isn't currently in scope: ``` 6/10 ObservableOwners do not have valid Opsgenie teams 3/10 ObservableOwners do not point to valid handbook pages ``` I also removed some defunct/unused functionality/owners. ## Test plan To run these tests: ``` export OPSGENIE_API_KEY="..." go test -timeout 30s github.com/sourcegraph/sourcegraph/monitoring/monitoring -update -online ```
2023-12-29 22:07:35 +00:00
<sub>*Managed by the [Sourcegraph Infrastructure Org team](https://handbook.sourcegraph.com/departments/engineering/infrastructure).*</sub>
monitoring: improve containers metrics (#34808) part of - https://github.com/sourcegraph/sourcegraph/issues/33437 - https://github.com/sourcegraph/sourcegraph/issues/33438 This PR adds the following - A shared observable metric to indicate how many times a container (or child processes) has been oom kill - It's added to provisioning indicators since it's an indicator of underprovisioned resources - A new dashboard that shows resources usage of all containers at a glance - It contains two groups, one for all containers and one for containers that may have potential scaling issues - the first group is extremely noisy so it's hidden by default - the second group provides a more focused experience for site-admin to quickly find problematic resources <!-- > ~it looks like the alert table will be rendered regardless, which is pretty annoying (I guess the implementation of the monitoring stack is based on the assumption that one service/container will have its dashboard)~ __Questions & Notes__ - ~Should I make the new dashboard filterable by container name? like I only want to see the graph of `frontend`. Is that something that's valuable?~ - ~Any other metrics you all would like to see?~ - graph for total available resources on the VM compared to usage from https://github.com/sourcegraph/sourcegraph/issues/33437 is not really doable since we do not have `node-exporter` in anyone of our distro. We would have to tackle this from https://github.com/sourcegraph/customer/issues/806 --> ## Test plan connect to `demo`'s prom ```sh gcloud compute start-iap-tunnel default-$NEW_DEPLOYMENT-instance 9090 --local-host-port=localhost:4445 --zone us-central1-f --project $PROJECT_PREFIX-$CUSTOMER ``` or use dogfood ```sh kubectl port-forward svc/prometheus 4445:30090 ``` Update `dev/grafana/all/datasources.yaml` ```yml # Configuration for all non-Linux platforms. apiVersion: 1 datasources: - name: Prometheus type: prometheus access: proxy url: http://docker.for.mac.localhost:4445 # use the remote prometheus instance on demo instead of the local one isDefault: true editable: false - name: Jaeger type: jaeger access: proxy url: http://host.docker.internal:16686/-/debug/jaeger - name: Loki type: loki access: proxy url: http://host.docker.internal:3100 ``` Start monitoring stack ```sh sg run grafana ``` see screenshot > the first group is hidden by default ![CleanShot 2022-05-04 at 10 48 26](https://user-images.githubusercontent.com/8373004/166748470-22460552-6e51-4b29-a26b-4c5adbe1c404.png) ![CleanShot 2022-05-04 at 10 48 03](https://user-images.githubusercontent.com/8373004/166748396-1d410c35-e5fe-45e2-83bd-418d845413cb.png) ![CleanShot 2022-05-04 at 11 25 13](https://user-images.githubusercontent.com/8373004/166801266-21e51b68-65fd-4cb7-97de-83520e31e498.png)
2022-05-09 23:56:47 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
monitoring: extract into a submodule (#45786) This change extracts `monitoring` into a submodule for import in `sourcegraph/controller` (https://github.com/sourcegraph/controller/pull/195) so that we can generate dashboards for Cloud instances. These steps were required: 1. Initialize a `go.mod` in `monitoring` 2. Extract `dev/sg/internal/cliutil` into `lib` to avoid illegal imports from `monitoring` 3. Add local replaces to both `sourcegraph/sourcegraph` and `monitoring` 4. `go mod tidy` on all submodules 5. Update `go generate ./monitoring` commands to use `sg`, since the `go generate` command no longer works 6. Update `grafana/build.sh`, `prometheus/build.sh` to build the submodule 7. Amend linters to check for multiple `go.mod` files and ban imports of `github.com/sourcegraph/sourcegraph` 8. Update `sg generate go` to run in directories rather than from root The only caveat is that if you use VS code, you will now need to open `monitoring` in a separate workspace or similar, like with `lib`. Co-authored-by: Joe Chen <joe@sourcegraph.com>
2022-12-19 17:49:25 +00:00
Generated query for warning alert: `max((max by (name) (container_oom_events_total{name=~"^(pgsql|codeintel-db|codeinsights).*"})) >= 1)`
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
</details>
monitoring: improve containers metrics (#34808) part of - https://github.com/sourcegraph/sourcegraph/issues/33437 - https://github.com/sourcegraph/sourcegraph/issues/33438 This PR adds the following - A shared observable metric to indicate how many times a container (or child processes) has been oom kill - It's added to provisioning indicators since it's an indicator of underprovisioned resources - A new dashboard that shows resources usage of all containers at a glance - It contains two groups, one for all containers and one for containers that may have potential scaling issues - the first group is extremely noisy so it's hidden by default - the second group provides a more focused experience for site-admin to quickly find problematic resources <!-- > ~it looks like the alert table will be rendered regardless, which is pretty annoying (I guess the implementation of the monitoring stack is based on the assumption that one service/container will have its dashboard)~ __Questions & Notes__ - ~Should I make the new dashboard filterable by container name? like I only want to see the graph of `frontend`. Is that something that's valuable?~ - ~Any other metrics you all would like to see?~ - graph for total available resources on the VM compared to usage from https://github.com/sourcegraph/sourcegraph/issues/33437 is not really doable since we do not have `node-exporter` in anyone of our distro. We would have to tackle this from https://github.com/sourcegraph/customer/issues/806 --> ## Test plan connect to `demo`'s prom ```sh gcloud compute start-iap-tunnel default-$NEW_DEPLOYMENT-instance 9090 --local-host-port=localhost:4445 --zone us-central1-f --project $PROJECT_PREFIX-$CUSTOMER ``` or use dogfood ```sh kubectl port-forward svc/prometheus 4445:30090 ``` Update `dev/grafana/all/datasources.yaml` ```yml # Configuration for all non-Linux platforms. apiVersion: 1 datasources: - name: Prometheus type: prometheus access: proxy url: http://docker.for.mac.localhost:4445 # use the remote prometheus instance on demo instead of the local one isDefault: true editable: false - name: Jaeger type: jaeger access: proxy url: http://host.docker.internal:16686/-/debug/jaeger - name: Loki type: loki access: proxy url: http://host.docker.internal:3100 ``` Start monitoring stack ```sh sg run grafana ``` see screenshot > the first group is hidden by default ![CleanShot 2022-05-04 at 10 48 26](https://user-images.githubusercontent.com/8373004/166748470-22460552-6e51-4b29-a26b-4c5adbe1c404.png) ![CleanShot 2022-05-04 at 10 48 03](https://user-images.githubusercontent.com/8373004/166748396-1d410c35-e5fe-45e2-83bd-418d845413cb.png) ![CleanShot 2022-05-04 at 11 25 13](https://user-images.githubusercontent.com/8373004/166801266-21e51b68-65fd-4cb7-97de-83520e31e498.png)
2022-05-09 23:56:47 +00:00
<br />
monitoring: Update dashboards (#23091)
2021-07-25 16:25:33 +00:00
## postgres: pods_available_percentage
monitoring: sync dot-com alerts to generator (#12581) Co-authored-by: Stephen Gutekanst <stephen.gutekanst@gmail.com>
2020-08-04 00:40:58 +00:00
monitoring: Update dashboards (#23091)
2021-07-25 16:25:33 +00:00
<p class="subtitle">percentage pods available</p>
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
monitoring: sync dot-com alerts to generator (#12581) Co-authored-by: Stephen Gutekanst <stephen.gutekanst@gmail.com>
2020-08-04 00:40:58 +00:00
monitoring: Update dashboards (#23091)
2021-07-25 16:25:33 +00:00
- <span class="badge badge-critical">critical</span> postgres: less than 90% percentage pods available for 10m0s
monitoring: sync dot-com alerts to generator (#12581) Co-authored-by: Stephen Gutekanst <stephen.gutekanst@gmail.com>
2020-08-04 00:40:58 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: sync dot-com alerts to generator (#12581) Co-authored-by: Stephen Gutekanst <stephen.gutekanst@gmail.com>
2020-08-04 00:40:58 +00:00
monitoring: forbid critical alerts with no solutions (#36423) Adds the restriction that critical alerts must have solutions defined as a hard requirement within the generator, and fixes all the outstanding issues: - Add PossibleSolutions for the pods_available_percentage and pg_up critical alerts (we have a similar solution doc in the container_missing alert) - Removes all the other critical alerts without possible solutions Co-authored-by: Michael Lin <mlzc@hey.com>
2022-06-02 15:51:43 +00:00
- Determine if the pod was OOM killed using `kubectl describe pod (pgsql|codeintel-db|codeinsights)` (look for `OOMKilled: true`) and, if so, consider increasing the memory limit in the relevant `Deployment.yaml`.
- Check the logs before the container restarted to see if there are `panic:` messages or similar using `kubectl logs -p (pgsql|codeintel-db|codeinsights)`.
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#postgres-pods-available-percentage).
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
monitoring: Update dashboards (#23091)
2021-07-25 16:25:33 +00:00
"critical_postgres_pods_available_percentage"
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
]
```
monitoring: sync dot-com alerts to generator (#12581) Co-authored-by: Stephen Gutekanst <stephen.gutekanst@gmail.com>
2020-08-04 00:40:58 +00:00
monitoring: test owners for valid Opsgenie teams and handbook pages (#59251) In INC-264 it seems that certain alerts - such as [zoekt: less than 90% percentage pods available for 10m0s](https://opsg.in/a/i/sourcegraph/178a626f-0f28-4295-bee9-84da988bb473-1703759057681) - don't seem to end up going anywhere because the ObservableOwner is defunct. This change adds _opt-in_ testing to report: 1. How many owners have valid Opsgenie teams 2. How many owners have valid handbook pages In addition, we collect ObservableOwners that pass the test and use it to generate configuration for `site.json` in Sourcegraph.com: https://github.com/sourcegraph/deploy-sourcegraph-cloud/pull/18338 - this helps ensure the list is valid and not deceptively high-coverage. The results are not great, but **enforcing** that owners are valid isn't currently in scope: ``` 6/10 ObservableOwners do not have valid Opsgenie teams 3/10 ObservableOwners do not point to valid handbook pages ``` I also removed some defunct/unused functionality/owners. ## Test plan To run these tests: ``` export OPSGENIE_API_KEY="..." go test -timeout 30s github.com/sourcegraph/sourcegraph/monitoring/monitoring -update -online ```
2023-12-29 22:07:35 +00:00
<sub>*Managed by the [Sourcegraph Infrastructure Org team](https://handbook.sourcegraph.com/departments/engineering/infrastructure).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
monitoring: extract into a submodule (#45786) This change extracts `monitoring` into a submodule for import in `sourcegraph/controller` (https://github.com/sourcegraph/controller/pull/195) so that we can generate dashboards for Cloud instances. These steps were required: 1. Initialize a `go.mod` in `monitoring` 2. Extract `dev/sg/internal/cliutil` into `lib` to avoid illegal imports from `monitoring` 3. Add local replaces to both `sourcegraph/sourcegraph` and `monitoring` 4. `go mod tidy` on all submodules 5. Update `go generate ./monitoring` commands to use `sg`, since the `go generate` command no longer works 6. Update `grafana/build.sh`, `prometheus/build.sh` to build the submodule 7. Amend linters to check for multiple `go.mod` files and ban imports of `github.com/sourcegraph/sourcegraph` 8. Update `sg generate go` to run in directories rather than from root The only caveat is that if you use VS code, you will now need to open `monitoring` in a separate workspace or similar, like with `lib`. Co-authored-by: Joe Chen <joe@sourcegraph.com>
2022-12-19 17:49:25 +00:00
Generated query for critical alert: `min((sum by (app) (up{app=~".*(pgsql|codeintel-db|codeinsights)"}) / count by (app) (up{app=~".*(pgsql|codeintel-db|codeinsights)"}) * 100) <= 90)`
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
</details>
monitoring: improve solutions formatting, add team information (#14878)
2020-10-20 05:11:20 +00:00
<br />
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
codeintel: Add additional metrics for duration in queues (#32147) Co-authored-by: Noah Santschi-Cooney <noah@santschi-cooney.ch>
2022-03-16 16:05:49 +00:00
## precise-code-intel-worker: codeintel_upload_queued_max_age
<p class="subtitle">unprocessed upload record queue longest time in queue</p>
**Descriptions**
codeintel: Downgrade queue size critical alert to warning (#60165) We've been running into spurious alerts on-and-off. We should add observability here to better narrow down why we're getting backlogs that are getting cleared later. In the meantime, it doesn't make sense for this to be a critical alert.
2024-02-05 14:58:55 +00:00
- <span class="badge badge-warning">warning</span> precise-code-intel-worker: 18000s+ unprocessed upload record queue longest time in queue
codeintel: Add additional metrics for duration in queues (#32147) Co-authored-by: Noah Santschi-Cooney <noah@santschi-cooney.ch>
2022-03-16 16:05:49 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
codeintel: Add additional metrics for duration in queues (#32147) Co-authored-by: Noah Santschi-Cooney <noah@santschi-cooney.ch>
2022-03-16 16:05:49 +00:00
- An alert here could be indicative of a few things: an upload surfacing a pathological performance characteristic,
precise-code-intel-worker being underprovisioned for the required upload processing throughput, or a higher replica
count being required for the volume of uploads.
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#precise-code-intel-worker-codeintel-upload-queued-max-age).
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
codeintel: Downgrade queue size critical alert to warning (#60165) We've been running into spurious alerts on-and-off. We should add observability here to better narrow down why we're getting backlogs that are getting cleared later. In the meantime, it doesn't make sense for this to be a critical alert.
2024-02-05 14:58:55 +00:00
"warning_precise-code-intel-worker_codeintel_upload_queued_max_age"
codeintel: Add additional metrics for duration in queues (#32147) Co-authored-by: Noah Santschi-Cooney <noah@santschi-cooney.ch>
2022-03-16 16:05:49 +00:00
]
```
monitoring: rename Cloud SaaS to IAM (#38268)
2022-07-07 02:13:27 +00:00
<sub>*Managed by the [Sourcegraph Code intelligence team](https://handbook.sourcegraph.com/departments/engineering/teams/code-intelligence).*</sub>
codeintel: Add additional metrics for duration in queues (#32147) Co-authored-by: Noah Santschi-Cooney <noah@santschi-cooney.ch>
2022-03-16 16:05:49 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
codeintel: Downgrade queue size critical alert to warning (#60165) We've been running into spurious alerts on-and-off. We should add observability here to better narrow down why we're getting backlogs that are getting cleared later. In the meantime, it doesn't make sense for this to be a critical alert.
2024-02-05 14:58:55 +00:00
Generated query for warning alert: `max((max(src_codeintel_upload_queued_duration_seconds_total{job=~"^precise-code-intel-worker.*"})) >= 18000)`
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
</details>
codeintel: Add additional metrics for duration in queues (#32147) Co-authored-by: Noah Santschi-Cooney <noah@santschi-cooney.ch>
2022-03-16 16:05:49 +00:00
<br />
monitoring: Refactor provisioning dashboards (#23006)
2021-07-20 13:59:13 +00:00
## precise-code-intel-worker: mean_blocked_seconds_per_conn_request
monitoring: sync dot-com alerts to generator (#12581) Co-authored-by: Stephen Gutekanst <stephen.gutekanst@gmail.com>
2020-08-04 00:40:58 +00:00
monitoring: Refactor provisioning dashboards (#23006)
2021-07-20 13:59:13 +00:00
<p class="subtitle">mean blocked seconds per conn request</p>
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
monitoring: sync dot-com alerts to generator (#12581) Co-authored-by: Stephen Gutekanst <stephen.gutekanst@gmail.com>
2020-08-04 00:40:58 +00:00
monitoring: relax mean_blocked_seconds_per_conn_request alerts (#59507) https://github.com/sourcegraph/sourcegraph/pull/59284 dramatically reduced the `mean_blocked_seconds_per_conn_request` issues we've been seeing, but overall delays are still higher, even with generally healthy Cloud SQL resource utilization. <img width="1630" alt="image" src="https://github.com/sourcegraph/sourcegraph/assets/23356519/91615471-5187-4d15-83e7-5cc94595303c"> Spot-checking the spikes in load in Cloud SQL, it seems that there is a variety of causes for each spike (analytics workloads, Cody Gateway syncs, code intel workloads, gitserver things, `ListSourcegraphDotComIndexableRepos` etc) so I'm chalking this up to "expected". Since this alert is seen firing on a Cloud instance, let's just relax it for now so that it only fires a critical alert on very significant delays.
2024-01-11 01:14:28 +00:00
- <span class="badge badge-warning">warning</span> precise-code-intel-worker: 0.1s+ mean blocked seconds per conn request for 10m0s
- <span class="badge badge-critical">critical</span> precise-code-intel-worker: 0.5s+ mean blocked seconds per conn request for 10m0s
monitoring: sync dot-com alerts to generator (#12581) Co-authored-by: Stephen Gutekanst <stephen.gutekanst@gmail.com>
2020-08-04 00:40:58 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: sync dot-com alerts to generator (#12581) Co-authored-by: Stephen Gutekanst <stephen.gutekanst@gmail.com>
2020-08-04 00:40:58 +00:00
Add possible solutions and make docs discoverable (#22945) Add possible solutions and make docs discoverable
2021-07-30 00:30:31 +00:00
- Increase SRC_PGSQL_MAX_OPEN together with giving more memory to the database if needed
Docs: update links to point to new site (#60381) We have a number of docs links in the product that point to the old doc site. Method: - Search the repo for `docs.sourcegraph.com` - Exclude the `doc/` dir, all test fixtures, and `CHANGELOG.md` - For each, replace `docs.sourcegraph.com` with `sourcegraph.com/docs` - Navigate to the resulting URL ensuring it's not a dead link, updating the URL if necessary Many of the URLs updated are just comments, but since I'm doing a manual audit of each URL anyways, I felt it was worth it to update these while I was at it.
2024-02-13 00:23:47 +00:00
- Scale up Postgres memory/cpus - [see our scaling guide](https://sourcegraph.com/docs/admin/config/postgres-conf)
monitoring: relax mean_blocked_seconds_per_conn_request alerts (#59507) https://github.com/sourcegraph/sourcegraph/pull/59284 dramatically reduced the `mean_blocked_seconds_per_conn_request` issues we've been seeing, but overall delays are still higher, even with generally healthy Cloud SQL resource utilization. <img width="1630" alt="image" src="https://github.com/sourcegraph/sourcegraph/assets/23356519/91615471-5187-4d15-83e7-5cc94595303c"> Spot-checking the spikes in load in Cloud SQL, it seems that there is a variety of causes for each spike (analytics workloads, Cody Gateway syncs, code intel workloads, gitserver things, `ListSourcegraphDotComIndexableRepos` etc) so I'm chalking this up to "expected". Since this alert is seen firing on a Cloud instance, let's just relax it for now so that it only fires a critical alert on very significant delays.
2024-01-11 01:14:28 +00:00
- If using GCP Cloud SQL, check for high lock waits or CPU usage in query insights
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#precise-code-intel-worker-mean-blocked-seconds-per-conn-request).
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
monitoring: Refactor provisioning dashboards (#23006)
2021-07-20 13:59:13 +00:00
"warning_precise-code-intel-worker_mean_blocked_seconds_per_conn_request",
"critical_precise-code-intel-worker_mean_blocked_seconds_per_conn_request"
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
]
```
monitoring: sync dot-com alerts to generator (#12581) Co-authored-by: Stephen Gutekanst <stephen.gutekanst@gmail.com>
2020-08-04 00:40:58 +00:00
monitoring: test owners for valid Opsgenie teams and handbook pages (#59251) In INC-264 it seems that certain alerts - such as [zoekt: less than 90% percentage pods available for 10m0s](https://opsg.in/a/i/sourcegraph/178a626f-0f28-4295-bee9-84da988bb473-1703759057681) - don't seem to end up going anywhere because the ObservableOwner is defunct. This change adds _opt-in_ testing to report: 1. How many owners have valid Opsgenie teams 2. How many owners have valid handbook pages In addition, we collect ObservableOwners that pass the test and use it to generate configuration for `site.json` in Sourcegraph.com: https://github.com/sourcegraph/deploy-sourcegraph-cloud/pull/18338 - this helps ensure the list is valid and not deceptively high-coverage. The results are not great, but **enforcing** that owners are valid isn't currently in scope: ``` 6/10 ObservableOwners do not have valid Opsgenie teams 3/10 ObservableOwners do not point to valid handbook pages ``` I also removed some defunct/unused functionality/owners. ## Test plan To run these tests: ``` export OPSGENIE_API_KEY="..." go test -timeout 30s github.com/sourcegraph/sourcegraph/monitoring/monitoring -update -online ```
2023-12-29 22:07:35 +00:00
<sub>*Managed by the [Sourcegraph Infrastructure Org team](https://handbook.sourcegraph.com/departments/engineering/infrastructure).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: relax mean_blocked_seconds_per_conn_request alerts (#59507) https://github.com/sourcegraph/sourcegraph/pull/59284 dramatically reduced the `mean_blocked_seconds_per_conn_request` issues we've been seeing, but overall delays are still higher, even with generally healthy Cloud SQL resource utilization. <img width="1630" alt="image" src="https://github.com/sourcegraph/sourcegraph/assets/23356519/91615471-5187-4d15-83e7-5cc94595303c"> Spot-checking the spikes in load in Cloud SQL, it seems that there is a variety of causes for each spike (analytics workloads, Cody Gateway syncs, code intel workloads, gitserver things, `ListSourcegraphDotComIndexableRepos` etc) so I'm chalking this up to "expected". Since this alert is seen firing on a Cloud instance, let's just relax it for now so that it only fires a critical alert on very significant delays.
2024-01-11 01:14:28 +00:00
Generated query for warning alert: `max((sum by (app_name, db_name) (increase(src_pgsql_conns_blocked_seconds{app_name="precise-code-intel-worker"}[5m])) / sum by (app_name, db_name) (increase(src_pgsql_conns_waited_for{app_name="precise-code-intel-worker"}[5m]))) >= 0.1)`
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
monitoring: relax mean_blocked_seconds_per_conn_request alerts (#59507) https://github.com/sourcegraph/sourcegraph/pull/59284 dramatically reduced the `mean_blocked_seconds_per_conn_request` issues we've been seeing, but overall delays are still higher, even with generally healthy Cloud SQL resource utilization. <img width="1630" alt="image" src="https://github.com/sourcegraph/sourcegraph/assets/23356519/91615471-5187-4d15-83e7-5cc94595303c"> Spot-checking the spikes in load in Cloud SQL, it seems that there is a variety of causes for each spike (analytics workloads, Cody Gateway syncs, code intel workloads, gitserver things, `ListSourcegraphDotComIndexableRepos` etc) so I'm chalking this up to "expected". Since this alert is seen firing on a Cloud instance, let's just relax it for now so that it only fires a critical alert on very significant delays.
2024-01-11 01:14:28 +00:00
Generated query for critical alert: `max((sum by (app_name, db_name) (increase(src_pgsql_conns_blocked_seconds{app_name="precise-code-intel-worker"}[5m])) / sum by (app_name, db_name) (increase(src_pgsql_conns_waited_for{app_name="precise-code-intel-worker"}[5m]))) >= 0.5)`
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
</details>
<br />
## precise-code-intel-worker: container_cpu_usage
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
<p class="subtitle">container cpu usage total (1m average) across all cores by instance</p>
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
monitoring: dashboards reference, refresh observability docs (#16939) - New monitoring definition field, 'Interpretation' - New generated dashboards reference - Grafana panels now link to alert solutions and dashboards reference - Refreshed observability docs Co-authored-by: uwedeportivo <534011+uwedeportivo@users.noreply.github.com>
2020-12-23 02:36:41 +00:00
- <span class="badge badge-warning">warning</span> precise-code-intel-worker: 99%+ container cpu usage total (1m average) across all cores by instance
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
monitoring: migrate out-of-band alerts to the generator (#12391) Migrates all out-of-band alert rules defined by deploy-sourcegraph to the generator, such that each alert now has a panel as part of a service dashboard like all our other alerts, with a few changes and additions.
2020-07-28 02:17:56 +00:00
- **Kubernetes:** Consider increasing CPU limits in the the relevant `Deployment.yaml`.
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
- **Docker Compose:** Consider increasing `cpus:` of the precise-code-intel-worker container in `docker-compose.yml`.
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#precise-code-intel-worker-container-cpu-usage).
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
```json
"observability.silenceAlerts": [
"warning_precise-code-intel-worker_container_cpu_usage"
]
```
monitoring: rename Cloud SaaS to IAM (#38268)
2022-07-07 02:13:27 +00:00
<sub>*Managed by the [Sourcegraph Code intelligence team](https://handbook.sourcegraph.com/departments/engineering/teams/code-intelligence).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
Generated query for warning alert: `max((cadvisor_container_cpu_usage_percentage_total{name=~"^precise-code-intel-worker.*"}) >= 99)`
</details>
monitoring: improve solutions formatting, add team information (#14878)
2020-10-20 05:11:20 +00:00
<br />
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
## precise-code-intel-worker: container_memory_usage
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
<p class="subtitle">container memory usage by instance</p>
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
monitoring: dashboards reference, refresh observability docs (#16939) - New monitoring definition field, 'Interpretation' - New generated dashboards reference - Grafana panels now link to alert solutions and dashboards reference - Refreshed observability docs Co-authored-by: uwedeportivo <534011+uwedeportivo@users.noreply.github.com>
2020-12-23 02:36:41 +00:00
- <span class="badge badge-warning">warning</span> precise-code-intel-worker: 99%+ container memory usage by instance
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
- **Kubernetes:** Consider increasing memory limit in relevant `Deployment.yaml`.
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
- **Docker Compose:** Consider increasing `memory:` of precise-code-intel-worker container in `docker-compose.yml`.
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#precise-code-intel-worker-container-memory-usage).
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"warning_precise-code-intel-worker_container_memory_usage"
]
```
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
monitoring: rename Cloud SaaS to IAM (#38268)
2022-07-07 02:13:27 +00:00
<sub>*Managed by the [Sourcegraph Code intelligence team](https://handbook.sourcegraph.com/departments/engineering/teams/code-intelligence).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
Generated query for warning alert: `max((cadvisor_container_memory_usage_percentage_total{name=~"^precise-code-intel-worker.*"}) >= 99)`
</details>
monitoring: improve solutions formatting, add team information (#14878)
2020-10-20 05:11:20 +00:00
<br />
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: improve long-term provisioning alerts aggregation (#12778)
2020-08-08 06:44:33 +00:00
## precise-code-intel-worker: provisioning_container_cpu_usage_long_term
monitoring: support multi-bound alerts, add lower-bound warnings for containers (#11082)
2020-06-16 07:09:25 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
<p class="subtitle">container cpu usage total (90th percentile over 1d) across all cores by instance</p>
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
monitoring: support multi-bound alerts, add lower-bound warnings for containers (#11082)
2020-06-16 07:09:25 +00:00
monitoring: dashboards reference, refresh observability docs (#16939) - New monitoring definition field, 'Interpretation' - New generated dashboards reference - Grafana panels now link to alert solutions and dashboards reference - Refreshed observability docs Co-authored-by: uwedeportivo <534011+uwedeportivo@users.noreply.github.com>
2020-12-23 02:36:41 +00:00
- <span class="badge badge-warning">warning</span> precise-code-intel-worker: 80%+ container cpu usage total (90th percentile over 1d) across all cores by instance for 336h0m0s
monitoring: support multi-bound alerts, add lower-bound warnings for containers (#11082)
2020-06-16 07:09:25 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: support multi-bound alerts, add lower-bound warnings for containers (#11082)
2020-06-16 07:09:25 +00:00
Remove container_*_usage_7d_low alerts (#16164)
2020-11-25 18:19:33 +00:00
- **Kubernetes:** Consider increasing CPU limits in the `Deployment.yaml` for the precise-code-intel-worker service.
- **Docker Compose:** Consider increasing `cpus:` of the precise-code-intel-worker container in `docker-compose.yml`.
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#precise-code-intel-worker-provisioning-container-cpu-usage-long-term).
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
monitoring: support multi-bound alerts, add lower-bound warnings for containers (#11082)
2020-06-16 07:09:25 +00:00
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
```json
"observability.silenceAlerts": [
monitoring: improve long-term provisioning alerts aggregation (#12778)
2020-08-08 06:44:33 +00:00
"warning_precise-code-intel-worker_provisioning_container_cpu_usage_long_term"
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
]
```
monitoring: rename Cloud SaaS to IAM (#38268)
2022-07-07 02:13:27 +00:00
<sub>*Managed by the [Sourcegraph Code intelligence team](https://handbook.sourcegraph.com/departments/engineering/teams/code-intelligence).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
Generated query for warning alert: `max((quantile_over_time(0.9, cadvisor_container_cpu_usage_percentage_total{name=~"^precise-code-intel-worker.*"}[1d])) >= 80)`
</details>
monitoring: improve solutions formatting, add team information (#14878)
2020-10-20 05:11:20 +00:00
<br />
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: improve long-term provisioning alerts aggregation (#12778)
2020-08-08 06:44:33 +00:00
## precise-code-intel-worker: provisioning_container_memory_usage_long_term
monitoring: support multi-bound alerts, add lower-bound warnings for containers (#11082)
2020-06-16 07:09:25 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
<p class="subtitle">container memory usage (1d maximum) by instance</p>
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
monitoring: support multi-bound alerts, add lower-bound warnings for containers (#11082)
2020-06-16 07:09:25 +00:00
monitoring: dashboards reference, refresh observability docs (#16939) - New monitoring definition field, 'Interpretation' - New generated dashboards reference - Grafana panels now link to alert solutions and dashboards reference - Refreshed observability docs Co-authored-by: uwedeportivo <534011+uwedeportivo@users.noreply.github.com>
2020-12-23 02:36:41 +00:00
- <span class="badge badge-warning">warning</span> precise-code-intel-worker: 80%+ container memory usage (1d maximum) by instance for 336h0m0s
monitoring: support multi-bound alerts, add lower-bound warnings for containers (#11082)
2020-06-16 07:09:25 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: support multi-bound alerts, add lower-bound warnings for containers (#11082)
2020-06-16 07:09:25 +00:00
Remove container_*_usage_7d_low alerts (#16164)
2020-11-25 18:19:33 +00:00
- **Kubernetes:** Consider increasing memory limits in the `Deployment.yaml` for the precise-code-intel-worker service.
- **Docker Compose:** Consider increasing `memory:` of the precise-code-intel-worker container in `docker-compose.yml`.
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#precise-code-intel-worker-provisioning-container-memory-usage-long-term).
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
monitoring: improve long-term provisioning alerts aggregation (#12778)
2020-08-08 06:44:33 +00:00
"warning_precise-code-intel-worker_provisioning_container_memory_usage_long_term"
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
]
```
monitoring: support multi-bound alerts, add lower-bound warnings for containers (#11082)
2020-06-16 07:09:25 +00:00
monitoring: rename Cloud SaaS to IAM (#38268)
2022-07-07 02:13:27 +00:00
<sub>*Managed by the [Sourcegraph Code intelligence team](https://handbook.sourcegraph.com/departments/engineering/teams/code-intelligence).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
Generated query for warning alert: `max((max_over_time(cadvisor_container_memory_usage_percentage_total{name=~"^precise-code-intel-worker.*"}[1d])) >= 80)`
</details>
monitoring: improve solutions formatting, add team information (#14878)
2020-10-20 05:11:20 +00:00
<br />
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: improve long-term provisioning alerts aggregation (#12778)
2020-08-08 06:44:33 +00:00
## precise-code-intel-worker: provisioning_container_cpu_usage_short_term
monitoring: support multi-bound alerts, add lower-bound warnings for containers (#11082)
2020-06-16 07:09:25 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
<p class="subtitle">container cpu usage total (5m maximum) across all cores by instance</p>
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
monitoring: support multi-bound alerts, add lower-bound warnings for containers (#11082)
2020-06-16 07:09:25 +00:00
monitoring: dashboards reference, refresh observability docs (#16939) - New monitoring definition field, 'Interpretation' - New generated dashboards reference - Grafana panels now link to alert solutions and dashboards reference - Refreshed observability docs Co-authored-by: uwedeportivo <534011+uwedeportivo@users.noreply.github.com>
2020-12-23 02:36:41 +00:00
- <span class="badge badge-warning">warning</span> precise-code-intel-worker: 90%+ container cpu usage total (5m maximum) across all cores by instance for 30m0s
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
- **Kubernetes:** Consider increasing CPU limits in the the relevant `Deployment.yaml`.
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
- **Docker Compose:** Consider increasing `cpus:` of the precise-code-intel-worker container in `docker-compose.yml`.
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#precise-code-intel-worker-provisioning-container-cpu-usage-short-term).
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
```json
"observability.silenceAlerts": [
monitoring: improve long-term provisioning alerts aggregation (#12778)
2020-08-08 06:44:33 +00:00
"warning_precise-code-intel-worker_provisioning_container_cpu_usage_short_term"
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
]
```
monitoring: rename Cloud SaaS to IAM (#38268)
2022-07-07 02:13:27 +00:00
<sub>*Managed by the [Sourcegraph Code intelligence team](https://handbook.sourcegraph.com/departments/engineering/teams/code-intelligence).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
Generated query for warning alert: `max((max_over_time(cadvisor_container_cpu_usage_percentage_total{name=~"^precise-code-intel-worker.*"}[5m])) >= 90)`
</details>
monitoring: improve solutions formatting, add team information (#14878)
2020-10-20 05:11:20 +00:00
<br />
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: improve long-term provisioning alerts aggregation (#12778)
2020-08-08 06:44:33 +00:00
## precise-code-intel-worker: provisioning_container_memory_usage_short_term
monitoring: support multi-bound alerts, add lower-bound warnings for containers (#11082)
2020-06-16 07:09:25 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
<p class="subtitle">container memory usage (5m maximum) by instance</p>
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
monitoring: support multi-bound alerts, add lower-bound warnings for containers (#11082)
2020-06-16 07:09:25 +00:00
monitoring: dashboards reference, refresh observability docs (#16939) - New monitoring definition field, 'Interpretation' - New generated dashboards reference - Grafana panels now link to alert solutions and dashboards reference - Refreshed observability docs Co-authored-by: uwedeportivo <534011+uwedeportivo@users.noreply.github.com>
2020-12-23 02:36:41 +00:00
- <span class="badge badge-warning">warning</span> precise-code-intel-worker: 90%+ container memory usage (5m maximum) by instance
monitoring: support multi-bound alerts, add lower-bound warnings for containers (#11082)
2020-06-16 07:09:25 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: support multi-bound alerts, add lower-bound warnings for containers (#11082)
2020-06-16 07:09:25 +00:00
- **Kubernetes:** Consider increasing memory limit in relevant `Deployment.yaml`.
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
- **Docker Compose:** Consider increasing `memory:` of precise-code-intel-worker container in `docker-compose.yml`.
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#precise-code-intel-worker-provisioning-container-memory-usage-short-term).
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
Fill out dashboards for precise-code-intel services (#10496)
2020-05-12 15:12:45 +00:00
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
```json
"observability.silenceAlerts": [
monitoring: improve long-term provisioning alerts aggregation (#12778)
2020-08-08 06:44:33 +00:00
"warning_precise-code-intel-worker_provisioning_container_memory_usage_short_term"
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
]
```
Fill out dashboards for precise-code-intel services (#10496)
2020-05-12 15:12:45 +00:00
monitoring: rename Cloud SaaS to IAM (#38268)
2022-07-07 02:13:27 +00:00
<sub>*Managed by the [Sourcegraph Code intelligence team](https://handbook.sourcegraph.com/departments/engineering/teams/code-intelligence).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
Generated query for warning alert: `max((max_over_time(cadvisor_container_memory_usage_percentage_total{name=~"^precise-code-intel-worker.*"}[5m])) >= 90)`
</details>
monitoring: improve solutions formatting, add team information (#14878)
2020-10-20 05:11:20 +00:00
<br />
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: improve containers metrics (#34808) part of - https://github.com/sourcegraph/sourcegraph/issues/33437 - https://github.com/sourcegraph/sourcegraph/issues/33438 This PR adds the following - A shared observable metric to indicate how many times a container (or child processes) has been oom kill - It's added to provisioning indicators since it's an indicator of underprovisioned resources - A new dashboard that shows resources usage of all containers at a glance - It contains two groups, one for all containers and one for containers that may have potential scaling issues - the first group is extremely noisy so it's hidden by default - the second group provides a more focused experience for site-admin to quickly find problematic resources <!-- > ~it looks like the alert table will be rendered regardless, which is pretty annoying (I guess the implementation of the monitoring stack is based on the assumption that one service/container will have its dashboard)~ __Questions & Notes__ - ~Should I make the new dashboard filterable by container name? like I only want to see the graph of `frontend`. Is that something that's valuable?~ - ~Any other metrics you all would like to see?~ - graph for total available resources on the VM compared to usage from https://github.com/sourcegraph/sourcegraph/issues/33437 is not really doable since we do not have `node-exporter` in anyone of our distro. We would have to tackle this from https://github.com/sourcegraph/customer/issues/806 --> ## Test plan connect to `demo`'s prom ```sh gcloud compute start-iap-tunnel default-$NEW_DEPLOYMENT-instance 9090 --local-host-port=localhost:4445 --zone us-central1-f --project $PROJECT_PREFIX-$CUSTOMER ``` or use dogfood ```sh kubectl port-forward svc/prometheus 4445:30090 ``` Update `dev/grafana/all/datasources.yaml` ```yml # Configuration for all non-Linux platforms. apiVersion: 1 datasources: - name: Prometheus type: prometheus access: proxy url: http://docker.for.mac.localhost:4445 # use the remote prometheus instance on demo instead of the local one isDefault: true editable: false - name: Jaeger type: jaeger access: proxy url: http://host.docker.internal:16686/-/debug/jaeger - name: Loki type: loki access: proxy url: http://host.docker.internal:3100 ``` Start monitoring stack ```sh sg run grafana ``` see screenshot > the first group is hidden by default ![CleanShot 2022-05-04 at 10 48 26](https://user-images.githubusercontent.com/8373004/166748470-22460552-6e51-4b29-a26b-4c5adbe1c404.png) ![CleanShot 2022-05-04 at 10 48 03](https://user-images.githubusercontent.com/8373004/166748396-1d410c35-e5fe-45e2-83bd-418d845413cb.png) ![CleanShot 2022-05-04 at 11 25 13](https://user-images.githubusercontent.com/8373004/166801266-21e51b68-65fd-4cb7-97de-83520e31e498.png)
2022-05-09 23:56:47 +00:00
## precise-code-intel-worker: container_oomkill_events_total
<p class="subtitle">container OOMKILL events total by instance</p>
**Descriptions**
- <span class="badge badge-warning">warning</span> precise-code-intel-worker: 1+ container OOMKILL events total by instance
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: improve containers metrics (#34808) part of - https://github.com/sourcegraph/sourcegraph/issues/33437 - https://github.com/sourcegraph/sourcegraph/issues/33438 This PR adds the following - A shared observable metric to indicate how many times a container (or child processes) has been oom kill - It's added to provisioning indicators since it's an indicator of underprovisioned resources - A new dashboard that shows resources usage of all containers at a glance - It contains two groups, one for all containers and one for containers that may have potential scaling issues - the first group is extremely noisy so it's hidden by default - the second group provides a more focused experience for site-admin to quickly find problematic resources <!-- > ~it looks like the alert table will be rendered regardless, which is pretty annoying (I guess the implementation of the monitoring stack is based on the assumption that one service/container will have its dashboard)~ __Questions & Notes__ - ~Should I make the new dashboard filterable by container name? like I only want to see the graph of `frontend`. Is that something that's valuable?~ - ~Any other metrics you all would like to see?~ - graph for total available resources on the VM compared to usage from https://github.com/sourcegraph/sourcegraph/issues/33437 is not really doable since we do not have `node-exporter` in anyone of our distro. We would have to tackle this from https://github.com/sourcegraph/customer/issues/806 --> ## Test plan connect to `demo`'s prom ```sh gcloud compute start-iap-tunnel default-$NEW_DEPLOYMENT-instance 9090 --local-host-port=localhost:4445 --zone us-central1-f --project $PROJECT_PREFIX-$CUSTOMER ``` or use dogfood ```sh kubectl port-forward svc/prometheus 4445:30090 ``` Update `dev/grafana/all/datasources.yaml` ```yml # Configuration for all non-Linux platforms. apiVersion: 1 datasources: - name: Prometheus type: prometheus access: proxy url: http://docker.for.mac.localhost:4445 # use the remote prometheus instance on demo instead of the local one isDefault: true editable: false - name: Jaeger type: jaeger access: proxy url: http://host.docker.internal:16686/-/debug/jaeger - name: Loki type: loki access: proxy url: http://host.docker.internal:3100 ``` Start monitoring stack ```sh sg run grafana ``` see screenshot > the first group is hidden by default ![CleanShot 2022-05-04 at 10 48 26](https://user-images.githubusercontent.com/8373004/166748470-22460552-6e51-4b29-a26b-4c5adbe1c404.png) ![CleanShot 2022-05-04 at 10 48 03](https://user-images.githubusercontent.com/8373004/166748396-1d410c35-e5fe-45e2-83bd-418d845413cb.png) ![CleanShot 2022-05-04 at 11 25 13](https://user-images.githubusercontent.com/8373004/166801266-21e51b68-65fd-4cb7-97de-83520e31e498.png)
2022-05-09 23:56:47 +00:00
- **Kubernetes:** Consider increasing memory limit in relevant `Deployment.yaml`.
- **Docker Compose:** Consider increasing `memory:` of precise-code-intel-worker container in `docker-compose.yml`.
- More help interpreting this metric is available in the [dashboards reference](./dashboards.md#precise-code-intel-worker-container-oomkill-events-total).
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"warning_precise-code-intel-worker_container_oomkill_events_total"
]
```
monitoring: rename Cloud SaaS to IAM (#38268)
2022-07-07 02:13:27 +00:00
<sub>*Managed by the [Sourcegraph Code intelligence team](https://handbook.sourcegraph.com/departments/engineering/teams/code-intelligence).*</sub>
monitoring: improve containers metrics (#34808) part of - https://github.com/sourcegraph/sourcegraph/issues/33437 - https://github.com/sourcegraph/sourcegraph/issues/33438 This PR adds the following - A shared observable metric to indicate how many times a container (or child processes) has been oom kill - It's added to provisioning indicators since it's an indicator of underprovisioned resources - A new dashboard that shows resources usage of all containers at a glance - It contains two groups, one for all containers and one for containers that may have potential scaling issues - the first group is extremely noisy so it's hidden by default - the second group provides a more focused experience for site-admin to quickly find problematic resources <!-- > ~it looks like the alert table will be rendered regardless, which is pretty annoying (I guess the implementation of the monitoring stack is based on the assumption that one service/container will have its dashboard)~ __Questions & Notes__ - ~Should I make the new dashboard filterable by container name? like I only want to see the graph of `frontend`. Is that something that's valuable?~ - ~Any other metrics you all would like to see?~ - graph for total available resources on the VM compared to usage from https://github.com/sourcegraph/sourcegraph/issues/33437 is not really doable since we do not have `node-exporter` in anyone of our distro. We would have to tackle this from https://github.com/sourcegraph/customer/issues/806 --> ## Test plan connect to `demo`'s prom ```sh gcloud compute start-iap-tunnel default-$NEW_DEPLOYMENT-instance 9090 --local-host-port=localhost:4445 --zone us-central1-f --project $PROJECT_PREFIX-$CUSTOMER ``` or use dogfood ```sh kubectl port-forward svc/prometheus 4445:30090 ``` Update `dev/grafana/all/datasources.yaml` ```yml # Configuration for all non-Linux platforms. apiVersion: 1 datasources: - name: Prometheus type: prometheus access: proxy url: http://docker.for.mac.localhost:4445 # use the remote prometheus instance on demo instead of the local one isDefault: true editable: false - name: Jaeger type: jaeger access: proxy url: http://host.docker.internal:16686/-/debug/jaeger - name: Loki type: loki access: proxy url: http://host.docker.internal:3100 ``` Start monitoring stack ```sh sg run grafana ``` see screenshot > the first group is hidden by default ![CleanShot 2022-05-04 at 10 48 26](https://user-images.githubusercontent.com/8373004/166748470-22460552-6e51-4b29-a26b-4c5adbe1c404.png) ![CleanShot 2022-05-04 at 10 48 03](https://user-images.githubusercontent.com/8373004/166748396-1d410c35-e5fe-45e2-83bd-418d845413cb.png) ![CleanShot 2022-05-04 at 11 25 13](https://user-images.githubusercontent.com/8373004/166801266-21e51b68-65fd-4cb7-97de-83520e31e498.png)
2022-05-09 23:56:47 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
monitoring: extract into a submodule (#45786) This change extracts `monitoring` into a submodule for import in `sourcegraph/controller` (https://github.com/sourcegraph/controller/pull/195) so that we can generate dashboards for Cloud instances. These steps were required: 1. Initialize a `go.mod` in `monitoring` 2. Extract `dev/sg/internal/cliutil` into `lib` to avoid illegal imports from `monitoring` 3. Add local replaces to both `sourcegraph/sourcegraph` and `monitoring` 4. `go mod tidy` on all submodules 5. Update `go generate ./monitoring` commands to use `sg`, since the `go generate` command no longer works 6. Update `grafana/build.sh`, `prometheus/build.sh` to build the submodule 7. Amend linters to check for multiple `go.mod` files and ban imports of `github.com/sourcegraph/sourcegraph` 8. Update `sg generate go` to run in directories rather than from root The only caveat is that if you use VS code, you will now need to open `monitoring` in a separate workspace or similar, like with `lib`. Co-authored-by: Joe Chen <joe@sourcegraph.com>
2022-12-19 17:49:25 +00:00
Generated query for warning alert: `max((max by (name) (container_oom_events_total{name=~"^precise-code-intel-worker.*"})) >= 1)`
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
</details>
monitoring: improve containers metrics (#34808) part of - https://github.com/sourcegraph/sourcegraph/issues/33437 - https://github.com/sourcegraph/sourcegraph/issues/33438 This PR adds the following - A shared observable metric to indicate how many times a container (or child processes) has been oom kill - It's added to provisioning indicators since it's an indicator of underprovisioned resources - A new dashboard that shows resources usage of all containers at a glance - It contains two groups, one for all containers and one for containers that may have potential scaling issues - the first group is extremely noisy so it's hidden by default - the second group provides a more focused experience for site-admin to quickly find problematic resources <!-- > ~it looks like the alert table will be rendered regardless, which is pretty annoying (I guess the implementation of the monitoring stack is based on the assumption that one service/container will have its dashboard)~ __Questions & Notes__ - ~Should I make the new dashboard filterable by container name? like I only want to see the graph of `frontend`. Is that something that's valuable?~ - ~Any other metrics you all would like to see?~ - graph for total available resources on the VM compared to usage from https://github.com/sourcegraph/sourcegraph/issues/33437 is not really doable since we do not have `node-exporter` in anyone of our distro. We would have to tackle this from https://github.com/sourcegraph/customer/issues/806 --> ## Test plan connect to `demo`'s prom ```sh gcloud compute start-iap-tunnel default-$NEW_DEPLOYMENT-instance 9090 --local-host-port=localhost:4445 --zone us-central1-f --project $PROJECT_PREFIX-$CUSTOMER ``` or use dogfood ```sh kubectl port-forward svc/prometheus 4445:30090 ``` Update `dev/grafana/all/datasources.yaml` ```yml # Configuration for all non-Linux platforms. apiVersion: 1 datasources: - name: Prometheus type: prometheus access: proxy url: http://docker.for.mac.localhost:4445 # use the remote prometheus instance on demo instead of the local one isDefault: true editable: false - name: Jaeger type: jaeger access: proxy url: http://host.docker.internal:16686/-/debug/jaeger - name: Loki type: loki access: proxy url: http://host.docker.internal:3100 ``` Start monitoring stack ```sh sg run grafana ``` see screenshot > the first group is hidden by default ![CleanShot 2022-05-04 at 10 48 26](https://user-images.githubusercontent.com/8373004/166748470-22460552-6e51-4b29-a26b-4c5adbe1c404.png) ![CleanShot 2022-05-04 at 10 48 03](https://user-images.githubusercontent.com/8373004/166748396-1d410c35-e5fe-45e2-83bd-418d845413cb.png) ![CleanShot 2022-05-04 at 11 25 13](https://user-images.githubusercontent.com/8373004/166801266-21e51b68-65fd-4cb7-97de-83520e31e498.png)
2022-05-09 23:56:47 +00:00
<br />
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
## precise-code-intel-worker: go_goroutines
Fill out dashboards for precise-code-intel services (#10496)
2020-05-12 15:12:45 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
<p class="subtitle">maximum active goroutines</p>
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
Fill out dashboards for precise-code-intel services (#10496)
2020-05-12 15:12:45 +00:00
monitoring: dashboards reference, refresh observability docs (#16939) - New monitoring definition field, 'Interpretation' - New generated dashboards reference - Grafana panels now link to alert solutions and dashboards reference - Refreshed observability docs Co-authored-by: uwedeportivo <534011+uwedeportivo@users.noreply.github.com>
2020-12-23 02:36:41 +00:00
- <span class="badge badge-warning">warning</span> precise-code-intel-worker: 10000+ maximum active goroutines for 10m0s
Fill out dashboards for precise-code-intel services (#10496)
2020-05-12 15:12:45 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
Fill out dashboards for precise-code-intel services (#10496)
2020-05-12 15:12:45 +00:00
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- More help interpreting this metric is available in the [dashboards reference](./dashboards.md#precise-code-intel-worker-go-goroutines).
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
```json
"observability.silenceAlerts": [
"warning_precise-code-intel-worker_go_goroutines"
]
```
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
monitoring: rename Cloud SaaS to IAM (#38268)
2022-07-07 02:13:27 +00:00
<sub>*Managed by the [Sourcegraph Code intelligence team](https://handbook.sourcegraph.com/departments/engineering/teams/code-intelligence).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
monitoring: extract into a submodule (#45786) This change extracts `monitoring` into a submodule for import in `sourcegraph/controller` (https://github.com/sourcegraph/controller/pull/195) so that we can generate dashboards for Cloud instances. These steps were required: 1. Initialize a `go.mod` in `monitoring` 2. Extract `dev/sg/internal/cliutil` into `lib` to avoid illegal imports from `monitoring` 3. Add local replaces to both `sourcegraph/sourcegraph` and `monitoring` 4. `go mod tidy` on all submodules 5. Update `go generate ./monitoring` commands to use `sg`, since the `go generate` command no longer works 6. Update `grafana/build.sh`, `prometheus/build.sh` to build the submodule 7. Amend linters to check for multiple `go.mod` files and ban imports of `github.com/sourcegraph/sourcegraph` 8. Update `sg generate go` to run in directories rather than from root The only caveat is that if you use VS code, you will now need to open `monitoring` in a separate workspace or similar, like with `lib`. Co-authored-by: Joe Chen <joe@sourcegraph.com>
2022-12-19 17:49:25 +00:00
Generated query for warning alert: `max((max by (instance) (go_goroutines{job=~".*precise-code-intel-worker"})) >= 10000)`
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
</details>
monitoring: improve solutions formatting, add team information (#14878)
2020-10-20 05:11:20 +00:00
<br />
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
## precise-code-intel-worker: go_gc_duration_seconds
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
<p class="subtitle">maximum go garbage collection duration</p>
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
monitoring: dashboards reference, refresh observability docs (#16939) - New monitoring definition field, 'Interpretation' - New generated dashboards reference - Grafana panels now link to alert solutions and dashboards reference - Refreshed observability docs Co-authored-by: uwedeportivo <534011+uwedeportivo@users.noreply.github.com>
2020-12-23 02:36:41 +00:00
- <span class="badge badge-warning">warning</span> precise-code-intel-worker: 2s+ maximum go garbage collection duration
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#precise-code-intel-worker-go-gc-duration-seconds).
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"warning_precise-code-intel-worker_go_gc_duration_seconds"
]
```
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
monitoring: rename Cloud SaaS to IAM (#38268)
2022-07-07 02:13:27 +00:00
<sub>*Managed by the [Sourcegraph Code intelligence team](https://handbook.sourcegraph.com/departments/engineering/teams/code-intelligence).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
monitoring: extract into a submodule (#45786) This change extracts `monitoring` into a submodule for import in `sourcegraph/controller` (https://github.com/sourcegraph/controller/pull/195) so that we can generate dashboards for Cloud instances. These steps were required: 1. Initialize a `go.mod` in `monitoring` 2. Extract `dev/sg/internal/cliutil` into `lib` to avoid illegal imports from `monitoring` 3. Add local replaces to both `sourcegraph/sourcegraph` and `monitoring` 4. `go mod tidy` on all submodules 5. Update `go generate ./monitoring` commands to use `sg`, since the `go generate` command no longer works 6. Update `grafana/build.sh`, `prometheus/build.sh` to build the submodule 7. Amend linters to check for multiple `go.mod` files and ban imports of `github.com/sourcegraph/sourcegraph` 8. Update `sg generate go` to run in directories rather than from root The only caveat is that if you use VS code, you will now need to open `monitoring` in a separate workspace or similar, like with `lib`. Co-authored-by: Joe Chen <joe@sourcegraph.com>
2022-12-19 17:49:25 +00:00
Generated query for warning alert: `max((max by (instance) (go_gc_duration_seconds{job=~".*precise-code-intel-worker"})) >= 2)`
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
</details>
monitoring: improve solutions formatting, add team information (#14878)
2020-10-20 05:11:20 +00:00
<br />
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
## precise-code-intel-worker: pods_available_percentage
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
<p class="subtitle">percentage pods available</p>
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
monitoring: dashboards reference, refresh observability docs (#16939) - New monitoring definition field, 'Interpretation' - New generated dashboards reference - Grafana panels now link to alert solutions and dashboards reference - Refreshed observability docs Co-authored-by: uwedeportivo <534011+uwedeportivo@users.noreply.github.com>
2020-12-23 02:36:41 +00:00
- <span class="badge badge-critical">critical</span> precise-code-intel-worker: less than 90% percentage pods available for 10m0s
monitoring: support multi-bound alerts, add lower-bound warnings for containers (#11082)
2020-06-16 07:09:25 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: support multi-bound alerts, add lower-bound warnings for containers (#11082)
2020-06-16 07:09:25 +00:00
monitoring: forbid critical alerts with no solutions (#36423) Adds the restriction that critical alerts must have solutions defined as a hard requirement within the generator, and fixes all the outstanding issues: - Add PossibleSolutions for the pods_available_percentage and pg_up critical alerts (we have a similar solution doc in the container_missing alert) - Removes all the other critical alerts without possible solutions Co-authored-by: Michael Lin <mlzc@hey.com>
2022-06-02 15:51:43 +00:00
- Determine if the pod was OOM killed using `kubectl describe pod precise-code-intel-worker` (look for `OOMKilled: true`) and, if so, consider increasing the memory limit in the relevant `Deployment.yaml`.
- Check the logs before the container restarted to see if there are `panic:` messages or similar using `kubectl logs -p precise-code-intel-worker`.
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#precise-code-intel-worker-pods-available-percentage).
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
monitoring: migrate out-of-band alerts to the generator (#12391) Migrates all out-of-band alert rules defined by deploy-sourcegraph to the generator, such that each alert now has a panel as part of a service dashboard like all our other alerts, with a few changes and additions.
2020-07-28 02:17:56 +00:00
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
```json
"observability.silenceAlerts": [
"critical_precise-code-intel-worker_pods_available_percentage"
]
```
monitoring: rename Cloud SaaS to IAM (#38268)
2022-07-07 02:13:27 +00:00
<sub>*Managed by the [Sourcegraph Code intelligence team](https://handbook.sourcegraph.com/departments/engineering/teams/code-intelligence).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
monitoring: extract into a submodule (#45786) This change extracts `monitoring` into a submodule for import in `sourcegraph/controller` (https://github.com/sourcegraph/controller/pull/195) so that we can generate dashboards for Cloud instances. These steps were required: 1. Initialize a `go.mod` in `monitoring` 2. Extract `dev/sg/internal/cliutil` into `lib` to avoid illegal imports from `monitoring` 3. Add local replaces to both `sourcegraph/sourcegraph` and `monitoring` 4. `go mod tidy` on all submodules 5. Update `go generate ./monitoring` commands to use `sg`, since the `go generate` command no longer works 6. Update `grafana/build.sh`, `prometheus/build.sh` to build the submodule 7. Amend linters to check for multiple `go.mod` files and ban imports of `github.com/sourcegraph/sourcegraph` 8. Update `sg generate go` to run in directories rather than from root The only caveat is that if you use VS code, you will now need to open `monitoring` in a separate workspace or similar, like with `lib`. Co-authored-by: Joe Chen <joe@sourcegraph.com>
2022-12-19 17:49:25 +00:00
Generated query for critical alert: `min((sum by (app) (up{app=~".*precise-code-intel-worker"}) / count by (app) (up{app=~".*precise-code-intel-worker"}) * 100) <= 90)`
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
</details>
monitoring: improve solutions formatting, add team information (#14878)
2020-10-20 05:11:20 +00:00
<br />
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
Add Redis monitoring dashboard (#26969) Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2021-11-08 17:09:12 +00:00
## redis: redis-store_up
<p class="subtitle">redis-store availability</p>
**Descriptions**
- <span class="badge badge-critical">critical</span> redis: less than 1 redis-store availability for 10s
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
Add Redis monitoring dashboard (#26969) Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2021-11-08 17:09:12 +00:00
- Ensure redis-store is running
- More help interpreting this metric is available in the [dashboards reference](./dashboards.md#redis-redis-store-up).
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"critical_redis_redis-store_up"
]
```
monitoring: test owners for valid Opsgenie teams and handbook pages (#59251) In INC-264 it seems that certain alerts - such as [zoekt: less than 90% percentage pods available for 10m0s](https://opsg.in/a/i/sourcegraph/178a626f-0f28-4295-bee9-84da988bb473-1703759057681) - don't seem to end up going anywhere because the ObservableOwner is defunct. This change adds _opt-in_ testing to report: 1. How many owners have valid Opsgenie teams 2. How many owners have valid handbook pages In addition, we collect ObservableOwners that pass the test and use it to generate configuration for `site.json` in Sourcegraph.com: https://github.com/sourcegraph/deploy-sourcegraph-cloud/pull/18338 - this helps ensure the list is valid and not deceptively high-coverage. The results are not great, but **enforcing** that owners are valid isn't currently in scope: ``` 6/10 ObservableOwners do not have valid Opsgenie teams 3/10 ObservableOwners do not point to valid handbook pages ``` I also removed some defunct/unused functionality/owners. ## Test plan To run these tests: ``` export OPSGENIE_API_KEY="..." go test -timeout 30s github.com/sourcegraph/sourcegraph/monitoring/monitoring -update -online ```
2023-12-29 22:07:35 +00:00
<sub>*Managed by the [Sourcegraph Infrastructure Org team](https://handbook.sourcegraph.com/departments/engineering/infrastructure).*</sub>
Add Redis monitoring dashboard (#26969) Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2021-11-08 17:09:12 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
Generated query for critical alert: `min((redis_up{app="redis-store"}) < 1)`
</details>
Add Redis monitoring dashboard (#26969) Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2021-11-08 17:09:12 +00:00
<br />
## redis: redis-cache_up
<p class="subtitle">redis-cache availability</p>
**Descriptions**
- <span class="badge badge-critical">critical</span> redis: less than 1 redis-cache availability for 10s
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
Add Redis monitoring dashboard (#26969) Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2021-11-08 17:09:12 +00:00
- Ensure redis-cache is running
- More help interpreting this metric is available in the [dashboards reference](./dashboards.md#redis-redis-cache-up).
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"critical_redis_redis-cache_up"
]
```
monitoring: test owners for valid Opsgenie teams and handbook pages (#59251) In INC-264 it seems that certain alerts - such as [zoekt: less than 90% percentage pods available for 10m0s](https://opsg.in/a/i/sourcegraph/178a626f-0f28-4295-bee9-84da988bb473-1703759057681) - don't seem to end up going anywhere because the ObservableOwner is defunct. This change adds _opt-in_ testing to report: 1. How many owners have valid Opsgenie teams 2. How many owners have valid handbook pages In addition, we collect ObservableOwners that pass the test and use it to generate configuration for `site.json` in Sourcegraph.com: https://github.com/sourcegraph/deploy-sourcegraph-cloud/pull/18338 - this helps ensure the list is valid and not deceptively high-coverage. The results are not great, but **enforcing** that owners are valid isn't currently in scope: ``` 6/10 ObservableOwners do not have valid Opsgenie teams 3/10 ObservableOwners do not point to valid handbook pages ``` I also removed some defunct/unused functionality/owners. ## Test plan To run these tests: ``` export OPSGENIE_API_KEY="..." go test -timeout 30s github.com/sourcegraph/sourcegraph/monitoring/monitoring -update -online ```
2023-12-29 22:07:35 +00:00
<sub>*Managed by the [Sourcegraph Infrastructure Org team](https://handbook.sourcegraph.com/departments/engineering/infrastructure).*</sub>
Add Redis monitoring dashboard (#26969) Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2021-11-08 17:09:12 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
Generated query for critical alert: `min((redis_up{app="redis-cache"}) < 1)`
</details>
Add Redis monitoring dashboard (#26969) Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2021-11-08 17:09:12 +00:00
<br />
## redis: provisioning_container_cpu_usage_long_term
<p class="subtitle">container cpu usage total (90th percentile over 1d) across all cores by instance</p>
**Descriptions**
- <span class="badge badge-warning">warning</span> redis: 80%+ container cpu usage total (90th percentile over 1d) across all cores by instance for 336h0m0s
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
Add Redis monitoring dashboard (#26969) Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2021-11-08 17:09:12 +00:00
- **Kubernetes:** Consider increasing CPU limits in the `Deployment.yaml` for the redis-cache service.
- **Docker Compose:** Consider increasing `cpus:` of the redis-cache container in `docker-compose.yml`.
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#redis-provisioning-container-cpu-usage-long-term).
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"warning_redis_provisioning_container_cpu_usage_long_term"
]
```
monitoring: test owners for valid Opsgenie teams and handbook pages (#59251) In INC-264 it seems that certain alerts - such as [zoekt: less than 90% percentage pods available for 10m0s](https://opsg.in/a/i/sourcegraph/178a626f-0f28-4295-bee9-84da988bb473-1703759057681) - don't seem to end up going anywhere because the ObservableOwner is defunct. This change adds _opt-in_ testing to report: 1. How many owners have valid Opsgenie teams 2. How many owners have valid handbook pages In addition, we collect ObservableOwners that pass the test and use it to generate configuration for `site.json` in Sourcegraph.com: https://github.com/sourcegraph/deploy-sourcegraph-cloud/pull/18338 - this helps ensure the list is valid and not deceptively high-coverage. The results are not great, but **enforcing** that owners are valid isn't currently in scope: ``` 6/10 ObservableOwners do not have valid Opsgenie teams 3/10 ObservableOwners do not point to valid handbook pages ``` I also removed some defunct/unused functionality/owners. ## Test plan To run these tests: ``` export OPSGENIE_API_KEY="..." go test -timeout 30s github.com/sourcegraph/sourcegraph/monitoring/monitoring -update -online ```
2023-12-29 22:07:35 +00:00
<sub>*Managed by the [Sourcegraph Infrastructure Org team](https://handbook.sourcegraph.com/departments/engineering/infrastructure).*</sub>
Add Redis monitoring dashboard (#26969) Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2021-11-08 17:09:12 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
Generated query for warning alert: `max((quantile_over_time(0.9, cadvisor_container_cpu_usage_percentage_total{name=~"^redis-cache.*"}[1d])) >= 80)`
</details>
Add Redis monitoring dashboard (#26969) Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2021-11-08 17:09:12 +00:00
<br />
## redis: provisioning_container_memory_usage_long_term
<p class="subtitle">container memory usage (1d maximum) by instance</p>
**Descriptions**
- <span class="badge badge-warning">warning</span> redis: 80%+ container memory usage (1d maximum) by instance for 336h0m0s
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
Add Redis monitoring dashboard (#26969) Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2021-11-08 17:09:12 +00:00
- **Kubernetes:** Consider increasing memory limits in the `Deployment.yaml` for the redis-cache service.
- **Docker Compose:** Consider increasing `memory:` of the redis-cache container in `docker-compose.yml`.
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#redis-provisioning-container-memory-usage-long-term).
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"warning_redis_provisioning_container_memory_usage_long_term"
]
```
monitoring: test owners for valid Opsgenie teams and handbook pages (#59251) In INC-264 it seems that certain alerts - such as [zoekt: less than 90% percentage pods available for 10m0s](https://opsg.in/a/i/sourcegraph/178a626f-0f28-4295-bee9-84da988bb473-1703759057681) - don't seem to end up going anywhere because the ObservableOwner is defunct. This change adds _opt-in_ testing to report: 1. How many owners have valid Opsgenie teams 2. How many owners have valid handbook pages In addition, we collect ObservableOwners that pass the test and use it to generate configuration for `site.json` in Sourcegraph.com: https://github.com/sourcegraph/deploy-sourcegraph-cloud/pull/18338 - this helps ensure the list is valid and not deceptively high-coverage. The results are not great, but **enforcing** that owners are valid isn't currently in scope: ``` 6/10 ObservableOwners do not have valid Opsgenie teams 3/10 ObservableOwners do not point to valid handbook pages ``` I also removed some defunct/unused functionality/owners. ## Test plan To run these tests: ``` export OPSGENIE_API_KEY="..." go test -timeout 30s github.com/sourcegraph/sourcegraph/monitoring/monitoring -update -online ```
2023-12-29 22:07:35 +00:00
<sub>*Managed by the [Sourcegraph Infrastructure Org team](https://handbook.sourcegraph.com/departments/engineering/infrastructure).*</sub>
Add Redis monitoring dashboard (#26969) Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2021-11-08 17:09:12 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
Generated query for warning alert: `max((max_over_time(cadvisor_container_memory_usage_percentage_total{name=~"^redis-cache.*"}[1d])) >= 80)`
</details>
Add Redis monitoring dashboard (#26969) Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2021-11-08 17:09:12 +00:00
<br />
## redis: provisioning_container_cpu_usage_short_term
<p class="subtitle">container cpu usage total (5m maximum) across all cores by instance</p>
**Descriptions**
- <span class="badge badge-warning">warning</span> redis: 90%+ container cpu usage total (5m maximum) across all cores by instance for 30m0s
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
Add Redis monitoring dashboard (#26969) Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2021-11-08 17:09:12 +00:00
- **Kubernetes:** Consider increasing CPU limits in the the relevant `Deployment.yaml`.
- **Docker Compose:** Consider increasing `cpus:` of the redis-cache container in `docker-compose.yml`.
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#redis-provisioning-container-cpu-usage-short-term).
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"warning_redis_provisioning_container_cpu_usage_short_term"
]
```
monitoring: test owners for valid Opsgenie teams and handbook pages (#59251) In INC-264 it seems that certain alerts - such as [zoekt: less than 90% percentage pods available for 10m0s](https://opsg.in/a/i/sourcegraph/178a626f-0f28-4295-bee9-84da988bb473-1703759057681) - don't seem to end up going anywhere because the ObservableOwner is defunct. This change adds _opt-in_ testing to report: 1. How many owners have valid Opsgenie teams 2. How many owners have valid handbook pages In addition, we collect ObservableOwners that pass the test and use it to generate configuration for `site.json` in Sourcegraph.com: https://github.com/sourcegraph/deploy-sourcegraph-cloud/pull/18338 - this helps ensure the list is valid and not deceptively high-coverage. The results are not great, but **enforcing** that owners are valid isn't currently in scope: ``` 6/10 ObservableOwners do not have valid Opsgenie teams 3/10 ObservableOwners do not point to valid handbook pages ``` I also removed some defunct/unused functionality/owners. ## Test plan To run these tests: ``` export OPSGENIE_API_KEY="..." go test -timeout 30s github.com/sourcegraph/sourcegraph/monitoring/monitoring -update -online ```
2023-12-29 22:07:35 +00:00
<sub>*Managed by the [Sourcegraph Infrastructure Org team](https://handbook.sourcegraph.com/departments/engineering/infrastructure).*</sub>
Add Redis monitoring dashboard (#26969) Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2021-11-08 17:09:12 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
Generated query for warning alert: `max((max_over_time(cadvisor_container_cpu_usage_percentage_total{name=~"^redis-cache.*"}[5m])) >= 90)`
</details>
Add Redis monitoring dashboard (#26969) Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2021-11-08 17:09:12 +00:00
<br />
## redis: provisioning_container_memory_usage_short_term
<p class="subtitle">container memory usage (5m maximum) by instance</p>
**Descriptions**
- <span class="badge badge-warning">warning</span> redis: 90%+ container memory usage (5m maximum) by instance
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
Add Redis monitoring dashboard (#26969) Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2021-11-08 17:09:12 +00:00
- **Kubernetes:** Consider increasing memory limit in relevant `Deployment.yaml`.
- **Docker Compose:** Consider increasing `memory:` of redis-cache container in `docker-compose.yml`.
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#redis-provisioning-container-memory-usage-short-term).
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"warning_redis_provisioning_container_memory_usage_short_term"
]
```
monitoring: test owners for valid Opsgenie teams and handbook pages (#59251) In INC-264 it seems that certain alerts - such as [zoekt: less than 90% percentage pods available for 10m0s](https://opsg.in/a/i/sourcegraph/178a626f-0f28-4295-bee9-84da988bb473-1703759057681) - don't seem to end up going anywhere because the ObservableOwner is defunct. This change adds _opt-in_ testing to report: 1. How many owners have valid Opsgenie teams 2. How many owners have valid handbook pages In addition, we collect ObservableOwners that pass the test and use it to generate configuration for `site.json` in Sourcegraph.com: https://github.com/sourcegraph/deploy-sourcegraph-cloud/pull/18338 - this helps ensure the list is valid and not deceptively high-coverage. The results are not great, but **enforcing** that owners are valid isn't currently in scope: ``` 6/10 ObservableOwners do not have valid Opsgenie teams 3/10 ObservableOwners do not point to valid handbook pages ``` I also removed some defunct/unused functionality/owners. ## Test plan To run these tests: ``` export OPSGENIE_API_KEY="..." go test -timeout 30s github.com/sourcegraph/sourcegraph/monitoring/monitoring -update -online ```
2023-12-29 22:07:35 +00:00
<sub>*Managed by the [Sourcegraph Infrastructure Org team](https://handbook.sourcegraph.com/departments/engineering/infrastructure).*</sub>
Add Redis monitoring dashboard (#26969) Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2021-11-08 17:09:12 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
Generated query for warning alert: `max((max_over_time(cadvisor_container_memory_usage_percentage_total{name=~"^redis-cache.*"}[5m])) >= 90)`
</details>
Add Redis monitoring dashboard (#26969) Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2021-11-08 17:09:12 +00:00
<br />
monitoring: improve containers metrics (#34808) part of - https://github.com/sourcegraph/sourcegraph/issues/33437 - https://github.com/sourcegraph/sourcegraph/issues/33438 This PR adds the following - A shared observable metric to indicate how many times a container (or child processes) has been oom kill - It's added to provisioning indicators since it's an indicator of underprovisioned resources - A new dashboard that shows resources usage of all containers at a glance - It contains two groups, one for all containers and one for containers that may have potential scaling issues - the first group is extremely noisy so it's hidden by default - the second group provides a more focused experience for site-admin to quickly find problematic resources <!-- > ~it looks like the alert table will be rendered regardless, which is pretty annoying (I guess the implementation of the monitoring stack is based on the assumption that one service/container will have its dashboard)~ __Questions & Notes__ - ~Should I make the new dashboard filterable by container name? like I only want to see the graph of `frontend`. Is that something that's valuable?~ - ~Any other metrics you all would like to see?~ - graph for total available resources on the VM compared to usage from https://github.com/sourcegraph/sourcegraph/issues/33437 is not really doable since we do not have `node-exporter` in anyone of our distro. We would have to tackle this from https://github.com/sourcegraph/customer/issues/806 --> ## Test plan connect to `demo`'s prom ```sh gcloud compute start-iap-tunnel default-$NEW_DEPLOYMENT-instance 9090 --local-host-port=localhost:4445 --zone us-central1-f --project $PROJECT_PREFIX-$CUSTOMER ``` or use dogfood ```sh kubectl port-forward svc/prometheus 4445:30090 ``` Update `dev/grafana/all/datasources.yaml` ```yml # Configuration for all non-Linux platforms. apiVersion: 1 datasources: - name: Prometheus type: prometheus access: proxy url: http://docker.for.mac.localhost:4445 # use the remote prometheus instance on demo instead of the local one isDefault: true editable: false - name: Jaeger type: jaeger access: proxy url: http://host.docker.internal:16686/-/debug/jaeger - name: Loki type: loki access: proxy url: http://host.docker.internal:3100 ``` Start monitoring stack ```sh sg run grafana ``` see screenshot > the first group is hidden by default ![CleanShot 2022-05-04 at 10 48 26](https://user-images.githubusercontent.com/8373004/166748470-22460552-6e51-4b29-a26b-4c5adbe1c404.png) ![CleanShot 2022-05-04 at 10 48 03](https://user-images.githubusercontent.com/8373004/166748396-1d410c35-e5fe-45e2-83bd-418d845413cb.png) ![CleanShot 2022-05-04 at 11 25 13](https://user-images.githubusercontent.com/8373004/166801266-21e51b68-65fd-4cb7-97de-83520e31e498.png)
2022-05-09 23:56:47 +00:00
## redis: container_oomkill_events_total
<p class="subtitle">container OOMKILL events total by instance</p>
**Descriptions**
- <span class="badge badge-warning">warning</span> redis: 1+ container OOMKILL events total by instance
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: improve containers metrics (#34808) part of - https://github.com/sourcegraph/sourcegraph/issues/33437 - https://github.com/sourcegraph/sourcegraph/issues/33438 This PR adds the following - A shared observable metric to indicate how many times a container (or child processes) has been oom kill - It's added to provisioning indicators since it's an indicator of underprovisioned resources - A new dashboard that shows resources usage of all containers at a glance - It contains two groups, one for all containers and one for containers that may have potential scaling issues - the first group is extremely noisy so it's hidden by default - the second group provides a more focused experience for site-admin to quickly find problematic resources <!-- > ~it looks like the alert table will be rendered regardless, which is pretty annoying (I guess the implementation of the monitoring stack is based on the assumption that one service/container will have its dashboard)~ __Questions & Notes__ - ~Should I make the new dashboard filterable by container name? like I only want to see the graph of `frontend`. Is that something that's valuable?~ - ~Any other metrics you all would like to see?~ - graph for total available resources on the VM compared to usage from https://github.com/sourcegraph/sourcegraph/issues/33437 is not really doable since we do not have `node-exporter` in anyone of our distro. We would have to tackle this from https://github.com/sourcegraph/customer/issues/806 --> ## Test plan connect to `demo`'s prom ```sh gcloud compute start-iap-tunnel default-$NEW_DEPLOYMENT-instance 9090 --local-host-port=localhost:4445 --zone us-central1-f --project $PROJECT_PREFIX-$CUSTOMER ``` or use dogfood ```sh kubectl port-forward svc/prometheus 4445:30090 ``` Update `dev/grafana/all/datasources.yaml` ```yml # Configuration for all non-Linux platforms. apiVersion: 1 datasources: - name: Prometheus type: prometheus access: proxy url: http://docker.for.mac.localhost:4445 # use the remote prometheus instance on demo instead of the local one isDefault: true editable: false - name: Jaeger type: jaeger access: proxy url: http://host.docker.internal:16686/-/debug/jaeger - name: Loki type: loki access: proxy url: http://host.docker.internal:3100 ``` Start monitoring stack ```sh sg run grafana ``` see screenshot > the first group is hidden by default ![CleanShot 2022-05-04 at 10 48 26](https://user-images.githubusercontent.com/8373004/166748470-22460552-6e51-4b29-a26b-4c5adbe1c404.png) ![CleanShot 2022-05-04 at 10 48 03](https://user-images.githubusercontent.com/8373004/166748396-1d410c35-e5fe-45e2-83bd-418d845413cb.png) ![CleanShot 2022-05-04 at 11 25 13](https://user-images.githubusercontent.com/8373004/166801266-21e51b68-65fd-4cb7-97de-83520e31e498.png)
2022-05-09 23:56:47 +00:00
- **Kubernetes:** Consider increasing memory limit in relevant `Deployment.yaml`.
- **Docker Compose:** Consider increasing `memory:` of redis-cache container in `docker-compose.yml`.
- More help interpreting this metric is available in the [dashboards reference](./dashboards.md#redis-container-oomkill-events-total).
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"warning_redis_container_oomkill_events_total"
]
```
monitoring: test owners for valid Opsgenie teams and handbook pages (#59251) In INC-264 it seems that certain alerts - such as [zoekt: less than 90% percentage pods available for 10m0s](https://opsg.in/a/i/sourcegraph/178a626f-0f28-4295-bee9-84da988bb473-1703759057681) - don't seem to end up going anywhere because the ObservableOwner is defunct. This change adds _opt-in_ testing to report: 1. How many owners have valid Opsgenie teams 2. How many owners have valid handbook pages In addition, we collect ObservableOwners that pass the test and use it to generate configuration for `site.json` in Sourcegraph.com: https://github.com/sourcegraph/deploy-sourcegraph-cloud/pull/18338 - this helps ensure the list is valid and not deceptively high-coverage. The results are not great, but **enforcing** that owners are valid isn't currently in scope: ``` 6/10 ObservableOwners do not have valid Opsgenie teams 3/10 ObservableOwners do not point to valid handbook pages ``` I also removed some defunct/unused functionality/owners. ## Test plan To run these tests: ``` export OPSGENIE_API_KEY="..." go test -timeout 30s github.com/sourcegraph/sourcegraph/monitoring/monitoring -update -online ```
2023-12-29 22:07:35 +00:00
<sub>*Managed by the [Sourcegraph Infrastructure Org team](https://handbook.sourcegraph.com/departments/engineering/infrastructure).*</sub>
monitoring: improve containers metrics (#34808) part of - https://github.com/sourcegraph/sourcegraph/issues/33437 - https://github.com/sourcegraph/sourcegraph/issues/33438 This PR adds the following - A shared observable metric to indicate how many times a container (or child processes) has been oom kill - It's added to provisioning indicators since it's an indicator of underprovisioned resources - A new dashboard that shows resources usage of all containers at a glance - It contains two groups, one for all containers and one for containers that may have potential scaling issues - the first group is extremely noisy so it's hidden by default - the second group provides a more focused experience for site-admin to quickly find problematic resources <!-- > ~it looks like the alert table will be rendered regardless, which is pretty annoying (I guess the implementation of the monitoring stack is based on the assumption that one service/container will have its dashboard)~ __Questions & Notes__ - ~Should I make the new dashboard filterable by container name? like I only want to see the graph of `frontend`. Is that something that's valuable?~ - ~Any other metrics you all would like to see?~ - graph for total available resources on the VM compared to usage from https://github.com/sourcegraph/sourcegraph/issues/33437 is not really doable since we do not have `node-exporter` in anyone of our distro. We would have to tackle this from https://github.com/sourcegraph/customer/issues/806 --> ## Test plan connect to `demo`'s prom ```sh gcloud compute start-iap-tunnel default-$NEW_DEPLOYMENT-instance 9090 --local-host-port=localhost:4445 --zone us-central1-f --project $PROJECT_PREFIX-$CUSTOMER ``` or use dogfood ```sh kubectl port-forward svc/prometheus 4445:30090 ``` Update `dev/grafana/all/datasources.yaml` ```yml # Configuration for all non-Linux platforms. apiVersion: 1 datasources: - name: Prometheus type: prometheus access: proxy url: http://docker.for.mac.localhost:4445 # use the remote prometheus instance on demo instead of the local one isDefault: true editable: false - name: Jaeger type: jaeger access: proxy url: http://host.docker.internal:16686/-/debug/jaeger - name: Loki type: loki access: proxy url: http://host.docker.internal:3100 ``` Start monitoring stack ```sh sg run grafana ``` see screenshot > the first group is hidden by default ![CleanShot 2022-05-04 at 10 48 26](https://user-images.githubusercontent.com/8373004/166748470-22460552-6e51-4b29-a26b-4c5adbe1c404.png) ![CleanShot 2022-05-04 at 10 48 03](https://user-images.githubusercontent.com/8373004/166748396-1d410c35-e5fe-45e2-83bd-418d845413cb.png) ![CleanShot 2022-05-04 at 11 25 13](https://user-images.githubusercontent.com/8373004/166801266-21e51b68-65fd-4cb7-97de-83520e31e498.png)
2022-05-09 23:56:47 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
monitoring: extract into a submodule (#45786) This change extracts `monitoring` into a submodule for import in `sourcegraph/controller` (https://github.com/sourcegraph/controller/pull/195) so that we can generate dashboards for Cloud instances. These steps were required: 1. Initialize a `go.mod` in `monitoring` 2. Extract `dev/sg/internal/cliutil` into `lib` to avoid illegal imports from `monitoring` 3. Add local replaces to both `sourcegraph/sourcegraph` and `monitoring` 4. `go mod tidy` on all submodules 5. Update `go generate ./monitoring` commands to use `sg`, since the `go generate` command no longer works 6. Update `grafana/build.sh`, `prometheus/build.sh` to build the submodule 7. Amend linters to check for multiple `go.mod` files and ban imports of `github.com/sourcegraph/sourcegraph` 8. Update `sg generate go` to run in directories rather than from root The only caveat is that if you use VS code, you will now need to open `monitoring` in a separate workspace or similar, like with `lib`. Co-authored-by: Joe Chen <joe@sourcegraph.com>
2022-12-19 17:49:25 +00:00
Generated query for warning alert: `max((max by (name) (container_oom_events_total{name=~"^redis-cache.*"})) >= 1)`
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
</details>
monitoring: improve containers metrics (#34808) part of - https://github.com/sourcegraph/sourcegraph/issues/33437 - https://github.com/sourcegraph/sourcegraph/issues/33438 This PR adds the following - A shared observable metric to indicate how many times a container (or child processes) has been oom kill - It's added to provisioning indicators since it's an indicator of underprovisioned resources - A new dashboard that shows resources usage of all containers at a glance - It contains two groups, one for all containers and one for containers that may have potential scaling issues - the first group is extremely noisy so it's hidden by default - the second group provides a more focused experience for site-admin to quickly find problematic resources <!-- > ~it looks like the alert table will be rendered regardless, which is pretty annoying (I guess the implementation of the monitoring stack is based on the assumption that one service/container will have its dashboard)~ __Questions & Notes__ - ~Should I make the new dashboard filterable by container name? like I only want to see the graph of `frontend`. Is that something that's valuable?~ - ~Any other metrics you all would like to see?~ - graph for total available resources on the VM compared to usage from https://github.com/sourcegraph/sourcegraph/issues/33437 is not really doable since we do not have `node-exporter` in anyone of our distro. We would have to tackle this from https://github.com/sourcegraph/customer/issues/806 --> ## Test plan connect to `demo`'s prom ```sh gcloud compute start-iap-tunnel default-$NEW_DEPLOYMENT-instance 9090 --local-host-port=localhost:4445 --zone us-central1-f --project $PROJECT_PREFIX-$CUSTOMER ``` or use dogfood ```sh kubectl port-forward svc/prometheus 4445:30090 ``` Update `dev/grafana/all/datasources.yaml` ```yml # Configuration for all non-Linux platforms. apiVersion: 1 datasources: - name: Prometheus type: prometheus access: proxy url: http://docker.for.mac.localhost:4445 # use the remote prometheus instance on demo instead of the local one isDefault: true editable: false - name: Jaeger type: jaeger access: proxy url: http://host.docker.internal:16686/-/debug/jaeger - name: Loki type: loki access: proxy url: http://host.docker.internal:3100 ``` Start monitoring stack ```sh sg run grafana ``` see screenshot > the first group is hidden by default ![CleanShot 2022-05-04 at 10 48 26](https://user-images.githubusercontent.com/8373004/166748470-22460552-6e51-4b29-a26b-4c5adbe1c404.png) ![CleanShot 2022-05-04 at 10 48 03](https://user-images.githubusercontent.com/8373004/166748396-1d410c35-e5fe-45e2-83bd-418d845413cb.png) ![CleanShot 2022-05-04 at 11 25 13](https://user-images.githubusercontent.com/8373004/166801266-21e51b68-65fd-4cb7-97de-83520e31e498.png)
2022-05-09 23:56:47 +00:00
<br />
Add Redis monitoring dashboard (#26969) Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2021-11-08 17:09:12 +00:00
## redis: provisioning_container_cpu_usage_long_term
<p class="subtitle">container cpu usage total (90th percentile over 1d) across all cores by instance</p>
**Descriptions**
- <span class="badge badge-warning">warning</span> redis: 80%+ container cpu usage total (90th percentile over 1d) across all cores by instance for 336h0m0s
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
Add Redis monitoring dashboard (#26969) Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2021-11-08 17:09:12 +00:00
- **Kubernetes:** Consider increasing CPU limits in the `Deployment.yaml` for the redis-store service.
- **Docker Compose:** Consider increasing `cpus:` of the redis-store container in `docker-compose.yml`.
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#redis-provisioning-container-cpu-usage-long-term).
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"warning_redis_provisioning_container_cpu_usage_long_term"
]
```
monitoring: test owners for valid Opsgenie teams and handbook pages (#59251) In INC-264 it seems that certain alerts - such as [zoekt: less than 90% percentage pods available for 10m0s](https://opsg.in/a/i/sourcegraph/178a626f-0f28-4295-bee9-84da988bb473-1703759057681) - don't seem to end up going anywhere because the ObservableOwner is defunct. This change adds _opt-in_ testing to report: 1. How many owners have valid Opsgenie teams 2. How many owners have valid handbook pages In addition, we collect ObservableOwners that pass the test and use it to generate configuration for `site.json` in Sourcegraph.com: https://github.com/sourcegraph/deploy-sourcegraph-cloud/pull/18338 - this helps ensure the list is valid and not deceptively high-coverage. The results are not great, but **enforcing** that owners are valid isn't currently in scope: ``` 6/10 ObservableOwners do not have valid Opsgenie teams 3/10 ObservableOwners do not point to valid handbook pages ``` I also removed some defunct/unused functionality/owners. ## Test plan To run these tests: ``` export OPSGENIE_API_KEY="..." go test -timeout 30s github.com/sourcegraph/sourcegraph/monitoring/monitoring -update -online ```
2023-12-29 22:07:35 +00:00
<sub>*Managed by the [Sourcegraph Infrastructure Org team](https://handbook.sourcegraph.com/departments/engineering/infrastructure).*</sub>
Add Redis monitoring dashboard (#26969) Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2021-11-08 17:09:12 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
Generated query for warning alert: `max((quantile_over_time(0.9, cadvisor_container_cpu_usage_percentage_total{name=~"^redis-store.*"}[1d])) >= 80)`
</details>
Add Redis monitoring dashboard (#26969) Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2021-11-08 17:09:12 +00:00
<br />
## redis: provisioning_container_memory_usage_long_term
<p class="subtitle">container memory usage (1d maximum) by instance</p>
**Descriptions**
- <span class="badge badge-warning">warning</span> redis: 80%+ container memory usage (1d maximum) by instance for 336h0m0s
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
Add Redis monitoring dashboard (#26969) Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2021-11-08 17:09:12 +00:00
- **Kubernetes:** Consider increasing memory limits in the `Deployment.yaml` for the redis-store service.
- **Docker Compose:** Consider increasing `memory:` of the redis-store container in `docker-compose.yml`.
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#redis-provisioning-container-memory-usage-long-term).
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"warning_redis_provisioning_container_memory_usage_long_term"
]
```
monitoring: test owners for valid Opsgenie teams and handbook pages (#59251) In INC-264 it seems that certain alerts - such as [zoekt: less than 90% percentage pods available for 10m0s](https://opsg.in/a/i/sourcegraph/178a626f-0f28-4295-bee9-84da988bb473-1703759057681) - don't seem to end up going anywhere because the ObservableOwner is defunct. This change adds _opt-in_ testing to report: 1. How many owners have valid Opsgenie teams 2. How many owners have valid handbook pages In addition, we collect ObservableOwners that pass the test and use it to generate configuration for `site.json` in Sourcegraph.com: https://github.com/sourcegraph/deploy-sourcegraph-cloud/pull/18338 - this helps ensure the list is valid and not deceptively high-coverage. The results are not great, but **enforcing** that owners are valid isn't currently in scope: ``` 6/10 ObservableOwners do not have valid Opsgenie teams 3/10 ObservableOwners do not point to valid handbook pages ``` I also removed some defunct/unused functionality/owners. ## Test plan To run these tests: ``` export OPSGENIE_API_KEY="..." go test -timeout 30s github.com/sourcegraph/sourcegraph/monitoring/monitoring -update -online ```
2023-12-29 22:07:35 +00:00
<sub>*Managed by the [Sourcegraph Infrastructure Org team](https://handbook.sourcegraph.com/departments/engineering/infrastructure).*</sub>
Add Redis monitoring dashboard (#26969) Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2021-11-08 17:09:12 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
Generated query for warning alert: `max((max_over_time(cadvisor_container_memory_usage_percentage_total{name=~"^redis-store.*"}[1d])) >= 80)`
</details>
Add Redis monitoring dashboard (#26969) Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2021-11-08 17:09:12 +00:00
<br />
## redis: provisioning_container_cpu_usage_short_term
<p class="subtitle">container cpu usage total (5m maximum) across all cores by instance</p>
**Descriptions**
- <span class="badge badge-warning">warning</span> redis: 90%+ container cpu usage total (5m maximum) across all cores by instance for 30m0s
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
Add Redis monitoring dashboard (#26969) Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2021-11-08 17:09:12 +00:00
- **Kubernetes:** Consider increasing CPU limits in the the relevant `Deployment.yaml`.
- **Docker Compose:** Consider increasing `cpus:` of the redis-store container in `docker-compose.yml`.
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#redis-provisioning-container-cpu-usage-short-term).
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"warning_redis_provisioning_container_cpu_usage_short_term"
]
```
monitoring: test owners for valid Opsgenie teams and handbook pages (#59251) In INC-264 it seems that certain alerts - such as [zoekt: less than 90% percentage pods available for 10m0s](https://opsg.in/a/i/sourcegraph/178a626f-0f28-4295-bee9-84da988bb473-1703759057681) - don't seem to end up going anywhere because the ObservableOwner is defunct. This change adds _opt-in_ testing to report: 1. How many owners have valid Opsgenie teams 2. How many owners have valid handbook pages In addition, we collect ObservableOwners that pass the test and use it to generate configuration for `site.json` in Sourcegraph.com: https://github.com/sourcegraph/deploy-sourcegraph-cloud/pull/18338 - this helps ensure the list is valid and not deceptively high-coverage. The results are not great, but **enforcing** that owners are valid isn't currently in scope: ``` 6/10 ObservableOwners do not have valid Opsgenie teams 3/10 ObservableOwners do not point to valid handbook pages ``` I also removed some defunct/unused functionality/owners. ## Test plan To run these tests: ``` export OPSGENIE_API_KEY="..." go test -timeout 30s github.com/sourcegraph/sourcegraph/monitoring/monitoring -update -online ```
2023-12-29 22:07:35 +00:00
<sub>*Managed by the [Sourcegraph Infrastructure Org team](https://handbook.sourcegraph.com/departments/engineering/infrastructure).*</sub>
Add Redis monitoring dashboard (#26969) Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2021-11-08 17:09:12 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
Generated query for warning alert: `max((max_over_time(cadvisor_container_cpu_usage_percentage_total{name=~"^redis-store.*"}[5m])) >= 90)`
</details>
Add Redis monitoring dashboard (#26969) Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2021-11-08 17:09:12 +00:00
<br />
## redis: provisioning_container_memory_usage_short_term
<p class="subtitle">container memory usage (5m maximum) by instance</p>
**Descriptions**
- <span class="badge badge-warning">warning</span> redis: 90%+ container memory usage (5m maximum) by instance
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
Add Redis monitoring dashboard (#26969) Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2021-11-08 17:09:12 +00:00
- **Kubernetes:** Consider increasing memory limit in relevant `Deployment.yaml`.
- **Docker Compose:** Consider increasing `memory:` of redis-store container in `docker-compose.yml`.
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#redis-provisioning-container-memory-usage-short-term).
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"warning_redis_provisioning_container_memory_usage_short_term"
]
```
monitoring: test owners for valid Opsgenie teams and handbook pages (#59251) In INC-264 it seems that certain alerts - such as [zoekt: less than 90% percentage pods available for 10m0s](https://opsg.in/a/i/sourcegraph/178a626f-0f28-4295-bee9-84da988bb473-1703759057681) - don't seem to end up going anywhere because the ObservableOwner is defunct. This change adds _opt-in_ testing to report: 1. How many owners have valid Opsgenie teams 2. How many owners have valid handbook pages In addition, we collect ObservableOwners that pass the test and use it to generate configuration for `site.json` in Sourcegraph.com: https://github.com/sourcegraph/deploy-sourcegraph-cloud/pull/18338 - this helps ensure the list is valid and not deceptively high-coverage. The results are not great, but **enforcing** that owners are valid isn't currently in scope: ``` 6/10 ObservableOwners do not have valid Opsgenie teams 3/10 ObservableOwners do not point to valid handbook pages ``` I also removed some defunct/unused functionality/owners. ## Test plan To run these tests: ``` export OPSGENIE_API_KEY="..." go test -timeout 30s github.com/sourcegraph/sourcegraph/monitoring/monitoring -update -online ```
2023-12-29 22:07:35 +00:00
<sub>*Managed by the [Sourcegraph Infrastructure Org team](https://handbook.sourcegraph.com/departments/engineering/infrastructure).*</sub>
Add Redis monitoring dashboard (#26969) Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2021-11-08 17:09:12 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
Generated query for warning alert: `max((max_over_time(cadvisor_container_memory_usage_percentage_total{name=~"^redis-store.*"}[5m])) >= 90)`
</details>
Add Redis monitoring dashboard (#26969) Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2021-11-08 17:09:12 +00:00
<br />
monitoring: improve containers metrics (#34808) part of - https://github.com/sourcegraph/sourcegraph/issues/33437 - https://github.com/sourcegraph/sourcegraph/issues/33438 This PR adds the following - A shared observable metric to indicate how many times a container (or child processes) has been oom kill - It's added to provisioning indicators since it's an indicator of underprovisioned resources - A new dashboard that shows resources usage of all containers at a glance - It contains two groups, one for all containers and one for containers that may have potential scaling issues - the first group is extremely noisy so it's hidden by default - the second group provides a more focused experience for site-admin to quickly find problematic resources <!-- > ~it looks like the alert table will be rendered regardless, which is pretty annoying (I guess the implementation of the monitoring stack is based on the assumption that one service/container will have its dashboard)~ __Questions & Notes__ - ~Should I make the new dashboard filterable by container name? like I only want to see the graph of `frontend`. Is that something that's valuable?~ - ~Any other metrics you all would like to see?~ - graph for total available resources on the VM compared to usage from https://github.com/sourcegraph/sourcegraph/issues/33437 is not really doable since we do not have `node-exporter` in anyone of our distro. We would have to tackle this from https://github.com/sourcegraph/customer/issues/806 --> ## Test plan connect to `demo`'s prom ```sh gcloud compute start-iap-tunnel default-$NEW_DEPLOYMENT-instance 9090 --local-host-port=localhost:4445 --zone us-central1-f --project $PROJECT_PREFIX-$CUSTOMER ``` or use dogfood ```sh kubectl port-forward svc/prometheus 4445:30090 ``` Update `dev/grafana/all/datasources.yaml` ```yml # Configuration for all non-Linux platforms. apiVersion: 1 datasources: - name: Prometheus type: prometheus access: proxy url: http://docker.for.mac.localhost:4445 # use the remote prometheus instance on demo instead of the local one isDefault: true editable: false - name: Jaeger type: jaeger access: proxy url: http://host.docker.internal:16686/-/debug/jaeger - name: Loki type: loki access: proxy url: http://host.docker.internal:3100 ``` Start monitoring stack ```sh sg run grafana ``` see screenshot > the first group is hidden by default ![CleanShot 2022-05-04 at 10 48 26](https://user-images.githubusercontent.com/8373004/166748470-22460552-6e51-4b29-a26b-4c5adbe1c404.png) ![CleanShot 2022-05-04 at 10 48 03](https://user-images.githubusercontent.com/8373004/166748396-1d410c35-e5fe-45e2-83bd-418d845413cb.png) ![CleanShot 2022-05-04 at 11 25 13](https://user-images.githubusercontent.com/8373004/166801266-21e51b68-65fd-4cb7-97de-83520e31e498.png)
2022-05-09 23:56:47 +00:00
## redis: container_oomkill_events_total
<p class="subtitle">container OOMKILL events total by instance</p>
**Descriptions**
- <span class="badge badge-warning">warning</span> redis: 1+ container OOMKILL events total by instance
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: improve containers metrics (#34808) part of - https://github.com/sourcegraph/sourcegraph/issues/33437 - https://github.com/sourcegraph/sourcegraph/issues/33438 This PR adds the following - A shared observable metric to indicate how many times a container (or child processes) has been oom kill - It's added to provisioning indicators since it's an indicator of underprovisioned resources - A new dashboard that shows resources usage of all containers at a glance - It contains two groups, one for all containers and one for containers that may have potential scaling issues - the first group is extremely noisy so it's hidden by default - the second group provides a more focused experience for site-admin to quickly find problematic resources <!-- > ~it looks like the alert table will be rendered regardless, which is pretty annoying (I guess the implementation of the monitoring stack is based on the assumption that one service/container will have its dashboard)~ __Questions & Notes__ - ~Should I make the new dashboard filterable by container name? like I only want to see the graph of `frontend`. Is that something that's valuable?~ - ~Any other metrics you all would like to see?~ - graph for total available resources on the VM compared to usage from https://github.com/sourcegraph/sourcegraph/issues/33437 is not really doable since we do not have `node-exporter` in anyone of our distro. We would have to tackle this from https://github.com/sourcegraph/customer/issues/806 --> ## Test plan connect to `demo`'s prom ```sh gcloud compute start-iap-tunnel default-$NEW_DEPLOYMENT-instance 9090 --local-host-port=localhost:4445 --zone us-central1-f --project $PROJECT_PREFIX-$CUSTOMER ``` or use dogfood ```sh kubectl port-forward svc/prometheus 4445:30090 ``` Update `dev/grafana/all/datasources.yaml` ```yml # Configuration for all non-Linux platforms. apiVersion: 1 datasources: - name: Prometheus type: prometheus access: proxy url: http://docker.for.mac.localhost:4445 # use the remote prometheus instance on demo instead of the local one isDefault: true editable: false - name: Jaeger type: jaeger access: proxy url: http://host.docker.internal:16686/-/debug/jaeger - name: Loki type: loki access: proxy url: http://host.docker.internal:3100 ``` Start monitoring stack ```sh sg run grafana ``` see screenshot > the first group is hidden by default ![CleanShot 2022-05-04 at 10 48 26](https://user-images.githubusercontent.com/8373004/166748470-22460552-6e51-4b29-a26b-4c5adbe1c404.png) ![CleanShot 2022-05-04 at 10 48 03](https://user-images.githubusercontent.com/8373004/166748396-1d410c35-e5fe-45e2-83bd-418d845413cb.png) ![CleanShot 2022-05-04 at 11 25 13](https://user-images.githubusercontent.com/8373004/166801266-21e51b68-65fd-4cb7-97de-83520e31e498.png)
2022-05-09 23:56:47 +00:00
- **Kubernetes:** Consider increasing memory limit in relevant `Deployment.yaml`.
- **Docker Compose:** Consider increasing `memory:` of redis-store container in `docker-compose.yml`.
- More help interpreting this metric is available in the [dashboards reference](./dashboards.md#redis-container-oomkill-events-total).
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"warning_redis_container_oomkill_events_total"
]
```
monitoring: test owners for valid Opsgenie teams and handbook pages (#59251) In INC-264 it seems that certain alerts - such as [zoekt: less than 90% percentage pods available for 10m0s](https://opsg.in/a/i/sourcegraph/178a626f-0f28-4295-bee9-84da988bb473-1703759057681) - don't seem to end up going anywhere because the ObservableOwner is defunct. This change adds _opt-in_ testing to report: 1. How many owners have valid Opsgenie teams 2. How many owners have valid handbook pages In addition, we collect ObservableOwners that pass the test and use it to generate configuration for `site.json` in Sourcegraph.com: https://github.com/sourcegraph/deploy-sourcegraph-cloud/pull/18338 - this helps ensure the list is valid and not deceptively high-coverage. The results are not great, but **enforcing** that owners are valid isn't currently in scope: ``` 6/10 ObservableOwners do not have valid Opsgenie teams 3/10 ObservableOwners do not point to valid handbook pages ``` I also removed some defunct/unused functionality/owners. ## Test plan To run these tests: ``` export OPSGENIE_API_KEY="..." go test -timeout 30s github.com/sourcegraph/sourcegraph/monitoring/monitoring -update -online ```
2023-12-29 22:07:35 +00:00
<sub>*Managed by the [Sourcegraph Infrastructure Org team](https://handbook.sourcegraph.com/departments/engineering/infrastructure).*</sub>
monitoring: improve containers metrics (#34808) part of - https://github.com/sourcegraph/sourcegraph/issues/33437 - https://github.com/sourcegraph/sourcegraph/issues/33438 This PR adds the following - A shared observable metric to indicate how many times a container (or child processes) has been oom kill - It's added to provisioning indicators since it's an indicator of underprovisioned resources - A new dashboard that shows resources usage of all containers at a glance - It contains two groups, one for all containers and one for containers that may have potential scaling issues - the first group is extremely noisy so it's hidden by default - the second group provides a more focused experience for site-admin to quickly find problematic resources <!-- > ~it looks like the alert table will be rendered regardless, which is pretty annoying (I guess the implementation of the monitoring stack is based on the assumption that one service/container will have its dashboard)~ __Questions & Notes__ - ~Should I make the new dashboard filterable by container name? like I only want to see the graph of `frontend`. Is that something that's valuable?~ - ~Any other metrics you all would like to see?~ - graph for total available resources on the VM compared to usage from https://github.com/sourcegraph/sourcegraph/issues/33437 is not really doable since we do not have `node-exporter` in anyone of our distro. We would have to tackle this from https://github.com/sourcegraph/customer/issues/806 --> ## Test plan connect to `demo`'s prom ```sh gcloud compute start-iap-tunnel default-$NEW_DEPLOYMENT-instance 9090 --local-host-port=localhost:4445 --zone us-central1-f --project $PROJECT_PREFIX-$CUSTOMER ``` or use dogfood ```sh kubectl port-forward svc/prometheus 4445:30090 ``` Update `dev/grafana/all/datasources.yaml` ```yml # Configuration for all non-Linux platforms. apiVersion: 1 datasources: - name: Prometheus type: prometheus access: proxy url: http://docker.for.mac.localhost:4445 # use the remote prometheus instance on demo instead of the local one isDefault: true editable: false - name: Jaeger type: jaeger access: proxy url: http://host.docker.internal:16686/-/debug/jaeger - name: Loki type: loki access: proxy url: http://host.docker.internal:3100 ``` Start monitoring stack ```sh sg run grafana ``` see screenshot > the first group is hidden by default ![CleanShot 2022-05-04 at 10 48 26](https://user-images.githubusercontent.com/8373004/166748470-22460552-6e51-4b29-a26b-4c5adbe1c404.png) ![CleanShot 2022-05-04 at 10 48 03](https://user-images.githubusercontent.com/8373004/166748396-1d410c35-e5fe-45e2-83bd-418d845413cb.png) ![CleanShot 2022-05-04 at 11 25 13](https://user-images.githubusercontent.com/8373004/166801266-21e51b68-65fd-4cb7-97de-83520e31e498.png)
2022-05-09 23:56:47 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
monitoring: extract into a submodule (#45786) This change extracts `monitoring` into a submodule for import in `sourcegraph/controller` (https://github.com/sourcegraph/controller/pull/195) so that we can generate dashboards for Cloud instances. These steps were required: 1. Initialize a `go.mod` in `monitoring` 2. Extract `dev/sg/internal/cliutil` into `lib` to avoid illegal imports from `monitoring` 3. Add local replaces to both `sourcegraph/sourcegraph` and `monitoring` 4. `go mod tidy` on all submodules 5. Update `go generate ./monitoring` commands to use `sg`, since the `go generate` command no longer works 6. Update `grafana/build.sh`, `prometheus/build.sh` to build the submodule 7. Amend linters to check for multiple `go.mod` files and ban imports of `github.com/sourcegraph/sourcegraph` 8. Update `sg generate go` to run in directories rather than from root The only caveat is that if you use VS code, you will now need to open `monitoring` in a separate workspace or similar, like with `lib`. Co-authored-by: Joe Chen <joe@sourcegraph.com>
2022-12-19 17:49:25 +00:00
Generated query for warning alert: `max((max by (name) (container_oom_events_total{name=~"^redis-store.*"})) >= 1)`
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
</details>
monitoring: improve containers metrics (#34808) part of - https://github.com/sourcegraph/sourcegraph/issues/33437 - https://github.com/sourcegraph/sourcegraph/issues/33438 This PR adds the following - A shared observable metric to indicate how many times a container (or child processes) has been oom kill - It's added to provisioning indicators since it's an indicator of underprovisioned resources - A new dashboard that shows resources usage of all containers at a glance - It contains two groups, one for all containers and one for containers that may have potential scaling issues - the first group is extremely noisy so it's hidden by default - the second group provides a more focused experience for site-admin to quickly find problematic resources <!-- > ~it looks like the alert table will be rendered regardless, which is pretty annoying (I guess the implementation of the monitoring stack is based on the assumption that one service/container will have its dashboard)~ __Questions & Notes__ - ~Should I make the new dashboard filterable by container name? like I only want to see the graph of `frontend`. Is that something that's valuable?~ - ~Any other metrics you all would like to see?~ - graph for total available resources on the VM compared to usage from https://github.com/sourcegraph/sourcegraph/issues/33437 is not really doable since we do not have `node-exporter` in anyone of our distro. We would have to tackle this from https://github.com/sourcegraph/customer/issues/806 --> ## Test plan connect to `demo`'s prom ```sh gcloud compute start-iap-tunnel default-$NEW_DEPLOYMENT-instance 9090 --local-host-port=localhost:4445 --zone us-central1-f --project $PROJECT_PREFIX-$CUSTOMER ``` or use dogfood ```sh kubectl port-forward svc/prometheus 4445:30090 ``` Update `dev/grafana/all/datasources.yaml` ```yml # Configuration for all non-Linux platforms. apiVersion: 1 datasources: - name: Prometheus type: prometheus access: proxy url: http://docker.for.mac.localhost:4445 # use the remote prometheus instance on demo instead of the local one isDefault: true editable: false - name: Jaeger type: jaeger access: proxy url: http://host.docker.internal:16686/-/debug/jaeger - name: Loki type: loki access: proxy url: http://host.docker.internal:3100 ``` Start monitoring stack ```sh sg run grafana ``` see screenshot > the first group is hidden by default ![CleanShot 2022-05-04 at 10 48 26](https://user-images.githubusercontent.com/8373004/166748470-22460552-6e51-4b29-a26b-4c5adbe1c404.png) ![CleanShot 2022-05-04 at 10 48 03](https://user-images.githubusercontent.com/8373004/166748396-1d410c35-e5fe-45e2-83bd-418d845413cb.png) ![CleanShot 2022-05-04 at 11 25 13](https://user-images.githubusercontent.com/8373004/166801266-21e51b68-65fd-4cb7-97de-83520e31e498.png)
2022-05-09 23:56:47 +00:00
<br />
Add Redis monitoring dashboard (#26969) Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2021-11-08 17:09:12 +00:00
## redis: pods_available_percentage
<p class="subtitle">percentage pods available</p>
**Descriptions**
- <span class="badge badge-critical">critical</span> redis: less than 90% percentage pods available for 10m0s
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
Add Redis monitoring dashboard (#26969) Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2021-11-08 17:09:12 +00:00
monitoring: forbid critical alerts with no solutions (#36423) Adds the restriction that critical alerts must have solutions defined as a hard requirement within the generator, and fixes all the outstanding issues: - Add PossibleSolutions for the pods_available_percentage and pg_up critical alerts (we have a similar solution doc in the container_missing alert) - Removes all the other critical alerts without possible solutions Co-authored-by: Michael Lin <mlzc@hey.com>
2022-06-02 15:51:43 +00:00
- Determine if the pod was OOM killed using `kubectl describe pod redis-cache` (look for `OOMKilled: true`) and, if so, consider increasing the memory limit in the relevant `Deployment.yaml`.
- Check the logs before the container restarted to see if there are `panic:` messages or similar using `kubectl logs -p redis-cache`.
Add Redis monitoring dashboard (#26969) Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2021-11-08 17:09:12 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#redis-pods-available-percentage).
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"critical_redis_pods_available_percentage"
]
```
monitoring: test owners for valid Opsgenie teams and handbook pages (#59251) In INC-264 it seems that certain alerts - such as [zoekt: less than 90% percentage pods available for 10m0s](https://opsg.in/a/i/sourcegraph/178a626f-0f28-4295-bee9-84da988bb473-1703759057681) - don't seem to end up going anywhere because the ObservableOwner is defunct. This change adds _opt-in_ testing to report: 1. How many owners have valid Opsgenie teams 2. How many owners have valid handbook pages In addition, we collect ObservableOwners that pass the test and use it to generate configuration for `site.json` in Sourcegraph.com: https://github.com/sourcegraph/deploy-sourcegraph-cloud/pull/18338 - this helps ensure the list is valid and not deceptively high-coverage. The results are not great, but **enforcing** that owners are valid isn't currently in scope: ``` 6/10 ObservableOwners do not have valid Opsgenie teams 3/10 ObservableOwners do not point to valid handbook pages ``` I also removed some defunct/unused functionality/owners. ## Test plan To run these tests: ``` export OPSGENIE_API_KEY="..." go test -timeout 30s github.com/sourcegraph/sourcegraph/monitoring/monitoring -update -online ```
2023-12-29 22:07:35 +00:00
<sub>*Managed by the [Sourcegraph Infrastructure Org team](https://handbook.sourcegraph.com/departments/engineering/infrastructure).*</sub>
Add Redis monitoring dashboard (#26969) Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2021-11-08 17:09:12 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
monitoring: extract into a submodule (#45786) This change extracts `monitoring` into a submodule for import in `sourcegraph/controller` (https://github.com/sourcegraph/controller/pull/195) so that we can generate dashboards for Cloud instances. These steps were required: 1. Initialize a `go.mod` in `monitoring` 2. Extract `dev/sg/internal/cliutil` into `lib` to avoid illegal imports from `monitoring` 3. Add local replaces to both `sourcegraph/sourcegraph` and `monitoring` 4. `go mod tidy` on all submodules 5. Update `go generate ./monitoring` commands to use `sg`, since the `go generate` command no longer works 6. Update `grafana/build.sh`, `prometheus/build.sh` to build the submodule 7. Amend linters to check for multiple `go.mod` files and ban imports of `github.com/sourcegraph/sourcegraph` 8. Update `sg generate go` to run in directories rather than from root The only caveat is that if you use VS code, you will now need to open `monitoring` in a separate workspace or similar, like with `lib`. Co-authored-by: Joe Chen <joe@sourcegraph.com>
2022-12-19 17:49:25 +00:00
Generated query for critical alert: `min((sum by (app) (up{app=~".*redis-cache"}) / count by (app) (up{app=~".*redis-cache"}) * 100) <= 90)`
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
</details>
Add Redis monitoring dashboard (#26969) Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2021-11-08 17:09:12 +00:00
<br />
## redis: pods_available_percentage
<p class="subtitle">percentage pods available</p>
**Descriptions**
- <span class="badge badge-critical">critical</span> redis: less than 90% percentage pods available for 10m0s
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
Add Redis monitoring dashboard (#26969) Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2021-11-08 17:09:12 +00:00
monitoring: forbid critical alerts with no solutions (#36423) Adds the restriction that critical alerts must have solutions defined as a hard requirement within the generator, and fixes all the outstanding issues: - Add PossibleSolutions for the pods_available_percentage and pg_up critical alerts (we have a similar solution doc in the container_missing alert) - Removes all the other critical alerts without possible solutions Co-authored-by: Michael Lin <mlzc@hey.com>
2022-06-02 15:51:43 +00:00
- Determine if the pod was OOM killed using `kubectl describe pod redis-store` (look for `OOMKilled: true`) and, if so, consider increasing the memory limit in the relevant `Deployment.yaml`.
- Check the logs before the container restarted to see if there are `panic:` messages or similar using `kubectl logs -p redis-store`.
Add Redis monitoring dashboard (#26969) Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2021-11-08 17:09:12 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#redis-pods-available-percentage).
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"critical_redis_pods_available_percentage"
]
```
monitoring: test owners for valid Opsgenie teams and handbook pages (#59251) In INC-264 it seems that certain alerts - such as [zoekt: less than 90% percentage pods available for 10m0s](https://opsg.in/a/i/sourcegraph/178a626f-0f28-4295-bee9-84da988bb473-1703759057681) - don't seem to end up going anywhere because the ObservableOwner is defunct. This change adds _opt-in_ testing to report: 1. How many owners have valid Opsgenie teams 2. How many owners have valid handbook pages In addition, we collect ObservableOwners that pass the test and use it to generate configuration for `site.json` in Sourcegraph.com: https://github.com/sourcegraph/deploy-sourcegraph-cloud/pull/18338 - this helps ensure the list is valid and not deceptively high-coverage. The results are not great, but **enforcing** that owners are valid isn't currently in scope: ``` 6/10 ObservableOwners do not have valid Opsgenie teams 3/10 ObservableOwners do not point to valid handbook pages ``` I also removed some defunct/unused functionality/owners. ## Test plan To run these tests: ``` export OPSGENIE_API_KEY="..." go test -timeout 30s github.com/sourcegraph/sourcegraph/monitoring/monitoring -update -online ```
2023-12-29 22:07:35 +00:00
<sub>*Managed by the [Sourcegraph Infrastructure Org team](https://handbook.sourcegraph.com/departments/engineering/infrastructure).*</sub>
Add Redis monitoring dashboard (#26969) Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2021-11-08 17:09:12 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
monitoring: extract into a submodule (#45786) This change extracts `monitoring` into a submodule for import in `sourcegraph/controller` (https://github.com/sourcegraph/controller/pull/195) so that we can generate dashboards for Cloud instances. These steps were required: 1. Initialize a `go.mod` in `monitoring` 2. Extract `dev/sg/internal/cliutil` into `lib` to avoid illegal imports from `monitoring` 3. Add local replaces to both `sourcegraph/sourcegraph` and `monitoring` 4. `go mod tidy` on all submodules 5. Update `go generate ./monitoring` commands to use `sg`, since the `go generate` command no longer works 6. Update `grafana/build.sh`, `prometheus/build.sh` to build the submodule 7. Amend linters to check for multiple `go.mod` files and ban imports of `github.com/sourcegraph/sourcegraph` 8. Update `sg generate go` to run in directories rather than from root The only caveat is that if you use VS code, you will now need to open `monitoring` in a separate workspace or similar, like with `lib`. Co-authored-by: Joe Chen <joe@sourcegraph.com>
2022-12-19 17:49:25 +00:00
Generated query for critical alert: `min((sum by (app) (up{app=~".*redis-store"}) / count by (app) (up{app=~".*redis-store"}) * 100) <= 90)`
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
</details>
Add Redis monitoring dashboard (#26969) Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2021-11-08 17:09:12 +00:00
<br />
codeintel: Fix worker task alerts (#36405)
2022-06-01 19:08:01 +00:00
## worker: worker_job_codeintel-upload-janitor_count
worker: Migrate codeintel janitor (#21842)
2021-06-10 01:13:39 +00:00
codeintel: Fix worker task alerts (#36405)
2022-06-01 19:08:01 +00:00
<p class="subtitle">number of worker instances running the codeintel-upload-janitor job</p>
worker: Migrate codeintel janitor (#21842)
2021-06-10 01:13:39 +00:00
**Descriptions**
codeintel: Fix worker task alerts (#36405)
2022-06-01 19:08:01 +00:00
- <span class="badge badge-warning">warning</span> worker: less than 1 number of worker instances running the codeintel-upload-janitor job for 1m0s
- <span class="badge badge-critical">critical</span> worker: less than 1 number of worker instances running the codeintel-upload-janitor job for 5m0s
worker: Migrate codeintel janitor (#21842)
2021-06-10 01:13:39 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
worker: Migrate codeintel janitor (#21842)
2021-06-10 01:13:39 +00:00
monitoring: Update dashboards (#23091)
2021-07-25 16:25:33 +00:00
- Ensure your instance defines a worker container such that:
codeintel: Fix worker task alerts (#36405)
2022-06-01 19:08:01 +00:00
- `WORKER_JOB_ALLOWLIST` contains "codeintel-upload-janitor" (or "all"), and
- `WORKER_JOB_BLOCKLIST` does not contain "codeintel-upload-janitor"
monitoring: Update dashboards (#23091)
2021-07-25 16:25:33 +00:00
- Ensure that such a container is not failing to start or stay active
codeintel: Fix worker task alerts (#36405)
2022-06-01 19:08:01 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#worker-worker-job-codeintel-upload-janitor-count).
worker: Migrate codeintel janitor (#21842)
2021-06-10 01:13:39 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
codeintel: Fix worker task alerts (#36405)
2022-06-01 19:08:01 +00:00
"warning_worker_worker_job_codeintel-upload-janitor_count",
"critical_worker_worker_job_codeintel-upload-janitor_count"
worker: Migrate codeintel janitor (#21842)
2021-06-10 01:13:39 +00:00
]
```
monitoring: rename Cloud SaaS to IAM (#38268)
2022-07-07 02:13:27 +00:00
<sub>*Managed by the [Sourcegraph Code intelligence team](https://handbook.sourcegraph.com/departments/engineering/teams/code-intelligence).*</sub>
worker: Migrate codeintel janitor (#21842)
2021-06-10 01:13:39 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
monitoring: fix alert definition for site configuration by adding scrape job label (#59687) We discovered recently that the definition for the alert that fires if the site configuration hasn't been fetched within 5 minutes strips out the regex that targets individual services (since it uses a grafana variable). This means that every instance of this alert will fire if any individual service trips over this threshold. This PR fixes the issue by adding a new `job` filter for this alert that targets only the services that have that Prometheus scrape target name. This works around the previous issue by using a fixed value for the `job` value instead of a dynamic grafana value. The value of the job filter generally looks like `job=~.*$container_name` (following the strategy from https://sourcegraph.com/github.com/sourcegraph/sourcegraph@9a780f2e694238b5326e3e121d6a1828463001b9/-/blob/monitoring/monitoring/monitoring.go?L161 ) unless I noticed that there was different logic in the existing dashboard for the services. Ex: - `frontend`: already used `job=~"(sourcegraph-)?frontend"` for some metrics, so I used it again here - `worker`: `already used `job=~"^worker.*"` in some metrics, so I used it again and standarized the other existing panels to use the same shared variable ## Test plan I eyeballed the generated alert.md and dashboards.md to verify that my changes looked correct (that is, my refactors resulted in either no diff, or that the diff I generated still looked like valid regex).
2024-01-17 23:19:54 +00:00
Generated query for warning alert: `(min((sum(src_worker_jobs{job=~"^worker.*",job_name="codeintel-upload-janitor"})) < 1)) or (absent(sum(src_worker_jobs{job=~"^worker.*",job_name="codeintel-upload-janitor"})) == 1)`
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
monitoring: fix alert definition for site configuration by adding scrape job label (#59687) We discovered recently that the definition for the alert that fires if the site configuration hasn't been fetched within 5 minutes strips out the regex that targets individual services (since it uses a grafana variable). This means that every instance of this alert will fire if any individual service trips over this threshold. This PR fixes the issue by adding a new `job` filter for this alert that targets only the services that have that Prometheus scrape target name. This works around the previous issue by using a fixed value for the `job` value instead of a dynamic grafana value. The value of the job filter generally looks like `job=~.*$container_name` (following the strategy from https://sourcegraph.com/github.com/sourcegraph/sourcegraph@9a780f2e694238b5326e3e121d6a1828463001b9/-/blob/monitoring/monitoring/monitoring.go?L161 ) unless I noticed that there was different logic in the existing dashboard for the services. Ex: - `frontend`: already used `job=~"(sourcegraph-)?frontend"` for some metrics, so I used it again here - `worker`: `already used `job=~"^worker.*"` in some metrics, so I used it again and standarized the other existing panels to use the same shared variable ## Test plan I eyeballed the generated alert.md and dashboards.md to verify that my changes looked correct (that is, my refactors resulted in either no diff, or that the diff I generated still looked like valid regex).
2024-01-17 23:19:54 +00:00
Generated query for critical alert: `(min((sum(src_worker_jobs{job=~"^worker.*",job_name="codeintel-upload-janitor"})) < 1)) or (absent(sum(src_worker_jobs{job=~"^worker.*",job_name="codeintel-upload-janitor"})) == 1)`
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
</details>
worker: Migrate codeintel janitor (#21842)
2021-06-10 01:13:39 +00:00
<br />
codeintel: Fix worker task alerts (#36405)
2022-06-01 19:08:01 +00:00
## worker: worker_job_codeintel-commitgraph-updater_count
worker: Migrate codeintel auto indexing jobs (#21977)
2021-06-10 18:18:26 +00:00
codeintel: Fix worker task alerts (#36405)
2022-06-01 19:08:01 +00:00
<p class="subtitle">number of worker instances running the codeintel-commitgraph-updater job</p>
worker: Migrate codeintel auto indexing jobs (#21977)
2021-06-10 18:18:26 +00:00
**Descriptions**
codeintel: Fix worker task alerts (#36405)
2022-06-01 19:08:01 +00:00
- <span class="badge badge-warning">warning</span> worker: less than 1 number of worker instances running the codeintel-commitgraph-updater job for 1m0s
- <span class="badge badge-critical">critical</span> worker: less than 1 number of worker instances running the codeintel-commitgraph-updater job for 5m0s
worker: Migrate codeintel auto indexing jobs (#21977)
2021-06-10 18:18:26 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
worker: Migrate codeintel auto indexing jobs (#21977)
2021-06-10 18:18:26 +00:00
monitoring: Update dashboards (#23091)
2021-07-25 16:25:33 +00:00
- Ensure your instance defines a worker container such that:
codeintel: Fix worker task alerts (#36405)
2022-06-01 19:08:01 +00:00
- `WORKER_JOB_ALLOWLIST` contains "codeintel-commitgraph-updater" (or "all"), and
- `WORKER_JOB_BLOCKLIST` does not contain "codeintel-commitgraph-updater"
monitoring: Update dashboards (#23091)
2021-07-25 16:25:33 +00:00
- Ensure that such a container is not failing to start or stay active
codeintel: Fix worker task alerts (#36405)
2022-06-01 19:08:01 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#worker-worker-job-codeintel-commitgraph-updater-count).
worker: Migrate codeintel auto indexing jobs (#21977)
2021-06-10 18:18:26 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
codeintel: Fix worker task alerts (#36405)
2022-06-01 19:08:01 +00:00
"warning_worker_worker_job_codeintel-commitgraph-updater_count",
"critical_worker_worker_job_codeintel-commitgraph-updater_count"
worker: Migrate codeintel auto indexing jobs (#21977)
2021-06-10 18:18:26 +00:00
]
```
monitoring: rename Cloud SaaS to IAM (#38268)
2022-07-07 02:13:27 +00:00
<sub>*Managed by the [Sourcegraph Code intelligence team](https://handbook.sourcegraph.com/departments/engineering/teams/code-intelligence).*</sub>
worker: Migrate codeintel auto indexing jobs (#21977)
2021-06-10 18:18:26 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
monitoring: fix alert definition for site configuration by adding scrape job label (#59687) We discovered recently that the definition for the alert that fires if the site configuration hasn't been fetched within 5 minutes strips out the regex that targets individual services (since it uses a grafana variable). This means that every instance of this alert will fire if any individual service trips over this threshold. This PR fixes the issue by adding a new `job` filter for this alert that targets only the services that have that Prometheus scrape target name. This works around the previous issue by using a fixed value for the `job` value instead of a dynamic grafana value. The value of the job filter generally looks like `job=~.*$container_name` (following the strategy from https://sourcegraph.com/github.com/sourcegraph/sourcegraph@9a780f2e694238b5326e3e121d6a1828463001b9/-/blob/monitoring/monitoring/monitoring.go?L161 ) unless I noticed that there was different logic in the existing dashboard for the services. Ex: - `frontend`: already used `job=~"(sourcegraph-)?frontend"` for some metrics, so I used it again here - `worker`: `already used `job=~"^worker.*"` in some metrics, so I used it again and standarized the other existing panels to use the same shared variable ## Test plan I eyeballed the generated alert.md and dashboards.md to verify that my changes looked correct (that is, my refactors resulted in either no diff, or that the diff I generated still looked like valid regex).
2024-01-17 23:19:54 +00:00
Generated query for warning alert: `(min((sum(src_worker_jobs{job=~"^worker.*",job_name="codeintel-commitgraph-updater"})) < 1)) or (absent(sum(src_worker_jobs{job=~"^worker.*",job_name="codeintel-commitgraph-updater"})) == 1)`
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
monitoring: fix alert definition for site configuration by adding scrape job label (#59687) We discovered recently that the definition for the alert that fires if the site configuration hasn't been fetched within 5 minutes strips out the regex that targets individual services (since it uses a grafana variable). This means that every instance of this alert will fire if any individual service trips over this threshold. This PR fixes the issue by adding a new `job` filter for this alert that targets only the services that have that Prometheus scrape target name. This works around the previous issue by using a fixed value for the `job` value instead of a dynamic grafana value. The value of the job filter generally looks like `job=~.*$container_name` (following the strategy from https://sourcegraph.com/github.com/sourcegraph/sourcegraph@9a780f2e694238b5326e3e121d6a1828463001b9/-/blob/monitoring/monitoring/monitoring.go?L161 ) unless I noticed that there was different logic in the existing dashboard for the services. Ex: - `frontend`: already used `job=~"(sourcegraph-)?frontend"` for some metrics, so I used it again here - `worker`: `already used `job=~"^worker.*"` in some metrics, so I used it again and standarized the other existing panels to use the same shared variable ## Test plan I eyeballed the generated alert.md and dashboards.md to verify that my changes looked correct (that is, my refactors resulted in either no diff, or that the diff I generated still looked like valid regex).
2024-01-17 23:19:54 +00:00
Generated query for critical alert: `(min((sum(src_worker_jobs{job=~"^worker.*",job_name="codeintel-commitgraph-updater"})) < 1)) or (absent(sum(src_worker_jobs{job=~"^worker.*",job_name="codeintel-commitgraph-updater"})) == 1)`
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
</details>
worker: Migrate codeintel auto indexing jobs (#21977)
2021-06-10 18:18:26 +00:00
<br />
codeintel: Fix worker task alerts (#36405)
2022-06-01 19:08:01 +00:00
## worker: worker_job_codeintel-autoindexing-scheduler_count
worker: Migrate codeintel auto indexing jobs (#21977)
2021-06-10 18:18:26 +00:00
codeintel: Fix worker task alerts (#36405)
2022-06-01 19:08:01 +00:00
<p class="subtitle">number of worker instances running the codeintel-autoindexing-scheduler job</p>
worker: Migrate codeintel auto indexing jobs (#21977)
2021-06-10 18:18:26 +00:00
**Descriptions**
codeintel: Fix worker task alerts (#36405)
2022-06-01 19:08:01 +00:00
- <span class="badge badge-warning">warning</span> worker: less than 1 number of worker instances running the codeintel-autoindexing-scheduler job for 1m0s
- <span class="badge badge-critical">critical</span> worker: less than 1 number of worker instances running the codeintel-autoindexing-scheduler job for 5m0s
worker: Migrate codeintel auto indexing jobs (#21977)
2021-06-10 18:18:26 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
worker: Migrate codeintel auto indexing jobs (#21977)
2021-06-10 18:18:26 +00:00
monitoring: Update dashboards (#23091)
2021-07-25 16:25:33 +00:00
- Ensure your instance defines a worker container such that:
codeintel: Fix worker task alerts (#36405)
2022-06-01 19:08:01 +00:00
- `WORKER_JOB_ALLOWLIST` contains "codeintel-autoindexing-scheduler" (or "all"), and
- `WORKER_JOB_BLOCKLIST` does not contain "codeintel-autoindexing-scheduler"
monitoring: Update dashboards (#23091)
2021-07-25 16:25:33 +00:00
- Ensure that such a container is not failing to start or stay active
codeintel: Fix worker task alerts (#36405)
2022-06-01 19:08:01 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#worker-worker-job-codeintel-autoindexing-scheduler-count).
worker: Migrate codeintel auto indexing jobs (#21977)
2021-06-10 18:18:26 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
codeintel: Fix worker task alerts (#36405)
2022-06-01 19:08:01 +00:00
"warning_worker_worker_job_codeintel-autoindexing-scheduler_count",
"critical_worker_worker_job_codeintel-autoindexing-scheduler_count"
worker: Migrate codeintel auto indexing jobs (#21977)
2021-06-10 18:18:26 +00:00
]
```
monitoring: rename Cloud SaaS to IAM (#38268)
2022-07-07 02:13:27 +00:00
<sub>*Managed by the [Sourcegraph Code intelligence team](https://handbook.sourcegraph.com/departments/engineering/teams/code-intelligence).*</sub>
worker: Migrate codeintel auto indexing jobs (#21977)
2021-06-10 18:18:26 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
monitoring: fix alert definition for site configuration by adding scrape job label (#59687) We discovered recently that the definition for the alert that fires if the site configuration hasn't been fetched within 5 minutes strips out the regex that targets individual services (since it uses a grafana variable). This means that every instance of this alert will fire if any individual service trips over this threshold. This PR fixes the issue by adding a new `job` filter for this alert that targets only the services that have that Prometheus scrape target name. This works around the previous issue by using a fixed value for the `job` value instead of a dynamic grafana value. The value of the job filter generally looks like `job=~.*$container_name` (following the strategy from https://sourcegraph.com/github.com/sourcegraph/sourcegraph@9a780f2e694238b5326e3e121d6a1828463001b9/-/blob/monitoring/monitoring/monitoring.go?L161 ) unless I noticed that there was different logic in the existing dashboard for the services. Ex: - `frontend`: already used `job=~"(sourcegraph-)?frontend"` for some metrics, so I used it again here - `worker`: `already used `job=~"^worker.*"` in some metrics, so I used it again and standarized the other existing panels to use the same shared variable ## Test plan I eyeballed the generated alert.md and dashboards.md to verify that my changes looked correct (that is, my refactors resulted in either no diff, or that the diff I generated still looked like valid regex).
2024-01-17 23:19:54 +00:00
Generated query for warning alert: `(min((sum(src_worker_jobs{job=~"^worker.*",job_name="codeintel-autoindexing-scheduler"})) < 1)) or (absent(sum(src_worker_jobs{job=~"^worker.*",job_name="codeintel-autoindexing-scheduler"})) == 1)`
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
monitoring: fix alert definition for site configuration by adding scrape job label (#59687) We discovered recently that the definition for the alert that fires if the site configuration hasn't been fetched within 5 minutes strips out the regex that targets individual services (since it uses a grafana variable). This means that every instance of this alert will fire if any individual service trips over this threshold. This PR fixes the issue by adding a new `job` filter for this alert that targets only the services that have that Prometheus scrape target name. This works around the previous issue by using a fixed value for the `job` value instead of a dynamic grafana value. The value of the job filter generally looks like `job=~.*$container_name` (following the strategy from https://sourcegraph.com/github.com/sourcegraph/sourcegraph@9a780f2e694238b5326e3e121d6a1828463001b9/-/blob/monitoring/monitoring/monitoring.go?L161 ) unless I noticed that there was different logic in the existing dashboard for the services. Ex: - `frontend`: already used `job=~"(sourcegraph-)?frontend"` for some metrics, so I used it again here - `worker`: `already used `job=~"^worker.*"` in some metrics, so I used it again and standarized the other existing panels to use the same shared variable ## Test plan I eyeballed the generated alert.md and dashboards.md to verify that my changes looked correct (that is, my refactors resulted in either no diff, or that the diff I generated still looked like valid regex).
2024-01-17 23:19:54 +00:00
Generated query for critical alert: `(min((sum(src_worker_jobs{job=~"^worker.*",job_name="codeintel-autoindexing-scheduler"})) < 1)) or (absent(sum(src_worker_jobs{job=~"^worker.*",job_name="codeintel-autoindexing-scheduler"})) == 1)`
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
</details>
worker: Migrate codeintel auto indexing jobs (#21977)
2021-06-10 18:18:26 +00:00
<br />
codeintel: Add additional metrics for duration in queues (#32147) Co-authored-by: Noah Santschi-Cooney <noah@santschi-cooney.ch>
2022-03-16 16:05:49 +00:00
## worker: codeintel_commit_graph_queued_max_age
<p class="subtitle">repository queue longest time in queue</p>
**Descriptions**
monitoring: downgrade commit_graph staleness critical alert to warning (#49059) This alert has paged the Cloud team on several occasions, and doesn't seem immediately actionable - see https://github.com/sourcegraph/customer/issues/1988 and related discussions. tl;dr it seems that this most often happens when a few repos are too large or take too long to clone or work on, which isn't actionable enough to warrant a critical alert (all the recent occurrences indicate none of the relevant services are starved for resources) - this PR proposes that we downgrade it to a warning. ## Test plan CI
2023-03-10 01:43:15 +00:00
- <span class="badge badge-warning">warning</span> worker: 3600s+ repository queue longest time in queue
codeintel: Add additional metrics for duration in queues (#32147) Co-authored-by: Noah Santschi-Cooney <noah@santschi-cooney.ch>
2022-03-16 16:05:49 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
codeintel: Add additional metrics for duration in queues (#32147) Co-authored-by: Noah Santschi-Cooney <noah@santschi-cooney.ch>
2022-03-16 16:05:49 +00:00
- An alert here is generally indicative of either underprovisioned worker instance(s) and/or
an underprovisioned main postgres instance.
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#worker-codeintel-commit-graph-queued-max-age).
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
monitoring: downgrade commit_graph staleness critical alert to warning (#49059) This alert has paged the Cloud team on several occasions, and doesn't seem immediately actionable - see https://github.com/sourcegraph/customer/issues/1988 and related discussions. tl;dr it seems that this most often happens when a few repos are too large or take too long to clone or work on, which isn't actionable enough to warrant a critical alert (all the recent occurrences indicate none of the relevant services are starved for resources) - this PR proposes that we downgrade it to a warning. ## Test plan CI
2023-03-10 01:43:15 +00:00
"warning_worker_codeintel_commit_graph_queued_max_age"
codeintel: Add additional metrics for duration in queues (#32147) Co-authored-by: Noah Santschi-Cooney <noah@santschi-cooney.ch>
2022-03-16 16:05:49 +00:00
]
```
monitoring: rename Cloud SaaS to IAM (#38268)
2022-07-07 02:13:27 +00:00
<sub>*Managed by the [Sourcegraph Code intelligence team](https://handbook.sourcegraph.com/departments/engineering/teams/code-intelligence).*</sub>
codeintel: Add additional metrics for duration in queues (#32147) Co-authored-by: Noah Santschi-Cooney <noah@santschi-cooney.ch>
2022-03-16 16:05:49 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
monitoring: downgrade commit_graph staleness critical alert to warning (#49059) This alert has paged the Cloud team on several occasions, and doesn't seem immediately actionable - see https://github.com/sourcegraph/customer/issues/1988 and related discussions. tl;dr it seems that this most often happens when a few repos are too large or take too long to clone or work on, which isn't actionable enough to warrant a critical alert (all the recent occurrences indicate none of the relevant services are starved for resources) - this PR proposes that we downgrade it to a warning. ## Test plan CI
2023-03-10 01:43:15 +00:00
Generated query for warning alert: `max((max(src_codeintel_commit_graph_queued_duration_seconds_total{job=~"^worker.*"})) >= 3600)`
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
</details>
codeintel: Add additional metrics for duration in queues (#32147) Co-authored-by: Noah Santschi-Cooney <noah@santschi-cooney.ch>
2022-03-16 16:05:49 +00:00
<br />
Move repo perms syncer to worker (#59510) Since we have distributed rate limits now, the last dependency is broken and we can move this subsystem around freely. To make repo-updater more lightweight, worker will be the new home of this system. ## Test plan Ran stack locally, CI still passes including integration tests.
2024-01-11 20:09:46 +00:00
## worker: perms_syncer_outdated_perms
<p class="subtitle">number of entities with outdated permissions</p>
**Descriptions**
- <span class="badge badge-warning">warning</span> worker: 100+ number of entities with outdated permissions for 5m0s
**Next steps**
- **Enabled permissions for the first time:** Wait for few minutes and see if the number goes down.
fix(Source): Fix documentation URLs for code hosts help pages (#63274) It seems many of our doc links for code hosts are broken in production due to a url changed from external_services to code_hosts. I did a find an replace to update all the ones I could find.
2024-06-17 18:32:46 +00:00
- **Otherwise:** Increase the API rate limit to [GitHub](https://sourcegraph.com/docs/admin/code_hosts/github#github-com-rate-limits), [GitLab](https://sourcegraph.com/docs/admin/code_hosts/gitlab#internal-rate-limits) or [Bitbucket Server](https://sourcegraph.com/docs/admin/code_hosts/bitbucket_server#internal-rate-limits).
Move repo perms syncer to worker (#59510) Since we have distributed rate limits now, the last dependency is broken and we can move this subsystem around freely. To make repo-updater more lightweight, worker will be the new home of this system. ## Test plan Ran stack locally, CI still passes including integration tests.
2024-01-11 20:09:46 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#worker-perms-syncer-outdated-perms).
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"warning_worker_perms_syncer_outdated_perms"
]
```
<sub>*Managed by the [Sourcegraph Source team](https://handbook.sourcegraph.com/departments/engineering/teams/source).*</sub>
<details>
<summary>Technical details</summary>
Generated query for warning alert: `max((max by (type) (src_repo_perms_syncer_outdated_perms)) >= 100)`
</details>
<br />
## worker: perms_syncer_sync_duration
<p class="subtitle">95th permissions sync duration</p>
**Descriptions**
- <span class="badge badge-warning">warning</span> worker: 30s+ 95th permissions sync duration for 5m0s
**Next steps**
- Check the network latency is reasonable (<50ms) between the Sourcegraph and the code host.
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#worker-perms-syncer-sync-duration).
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"warning_worker_perms_syncer_sync_duration"
]
```
<sub>*Managed by the [Sourcegraph Source team](https://handbook.sourcegraph.com/departments/engineering/teams/source).*</sub>
<details>
<summary>Technical details</summary>
Generated query for warning alert: `max((histogram_quantile(0.95, max by (le, type) (rate(src_repo_perms_syncer_sync_duration_seconds_bucket[1m])))) >= 30)`
</details>
<br />
## worker: perms_syncer_sync_errors
<p class="subtitle">permissions sync error rate</p>
**Descriptions**
- <span class="badge badge-critical">critical</span> worker: 1+ permissions sync error rate for 1m0s
**Next steps**
- Check the network connectivity the Sourcegraph and the code host.
- Check if API rate limit quota is exhausted on the code host.
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#worker-perms-syncer-sync-errors).
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"critical_worker_perms_syncer_sync_errors"
]
```
<sub>*Managed by the [Sourcegraph Source team](https://handbook.sourcegraph.com/departments/engineering/teams/source).*</sub>
<details>
<summary>Technical details</summary>
Generated query for critical alert: `max((max by (type) (ceil(rate(src_repo_perms_syncer_sync_errors_total[1m])))) >= 1)`
</details>
<br />
insights: add panel and warning alert for unutilized queue (#25190)
2021-09-21 22:21:20 +00:00
## worker: insights_queue_unutilized_size
<p class="subtitle">insights queue size that is not utilized (not processing)</p>
**Descriptions**
- <span class="badge badge-warning">warning</span> worker: 0+ insights queue size that is not utilized (not processing) for 30m0s
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
insights: add panel and warning alert for unutilized queue (#25190)
2021-09-21 22:21:20 +00:00
- Verify code insights worker job has successfully started. Restart worker service and monitoring startup logs, looking for worker panics.
- More help interpreting this metric is available in the [dashboards reference](./dashboards.md#worker-insights-queue-unutilized-size).
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"warning_worker_insights_queue_unutilized_size"
]
```
Monitoring: update owners for code insights and batches (#59313) This updates ownership for search, code insights, and batch changes. --------- Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2024-01-04 20:18:23 +00:00
<sub>*Managed by the [Sourcegraph Code Search team](https://handbook.sourcegraph.com/departments/engineering/teams/code-search).*</sub>
insights: add panel and warning alert for unutilized queue (#25190)
2021-09-21 22:21:20 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
monitoring: extract into a submodule (#45786) This change extracts `monitoring` into a submodule for import in `sourcegraph/controller` (https://github.com/sourcegraph/controller/pull/195) so that we can generate dashboards for Cloud instances. These steps were required: 1. Initialize a `go.mod` in `monitoring` 2. Extract `dev/sg/internal/cliutil` into `lib` to avoid illegal imports from `monitoring` 3. Add local replaces to both `sourcegraph/sourcegraph` and `monitoring` 4. `go mod tidy` on all submodules 5. Update `go generate ./monitoring` commands to use `sg`, since the `go generate` command no longer works 6. Update `grafana/build.sh`, `prometheus/build.sh` to build the submodule 7. Amend linters to check for multiple `go.mod` files and ban imports of `github.com/sourcegraph/sourcegraph` 8. Update `sg generate go` to run in directories rather than from root The only caveat is that if you use VS code, you will now need to open `monitoring` in a separate workspace or similar, like with `lib`. Co-authored-by: Joe Chen <joe@sourcegraph.com>
2022-12-19 17:49:25 +00:00
Generated query for warning alert: `max((max(src_query_runner_worker_total{job=~"^worker.*"}) > 0 and on (job) sum by (op) (increase(src_workerutil_dbworker_store_insights_query_runner_jobs_store_total{job=~"^worker.*",op="Dequeue"}[5m])) < 1) > 0)`
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
</details>
insights: add panel and warning alert for unutilized queue (#25190)
2021-09-21 22:21:20 +00:00
<br />
monitoring: Refactor provisioning dashboards (#23006)
2021-07-20 13:59:13 +00:00
## worker: mean_blocked_seconds_per_conn_request
worker: Add skeleton service (#21768)
2021-06-04 19:48:13 +00:00
monitoring: Refactor provisioning dashboards (#23006)
2021-07-20 13:59:13 +00:00
<p class="subtitle">mean blocked seconds per conn request</p>
worker: Add skeleton service (#21768)
2021-06-04 19:48:13 +00:00
**Descriptions**
monitoring: relax mean_blocked_seconds_per_conn_request alerts (#59507) https://github.com/sourcegraph/sourcegraph/pull/59284 dramatically reduced the `mean_blocked_seconds_per_conn_request` issues we've been seeing, but overall delays are still higher, even with generally healthy Cloud SQL resource utilization. <img width="1630" alt="image" src="https://github.com/sourcegraph/sourcegraph/assets/23356519/91615471-5187-4d15-83e7-5cc94595303c"> Spot-checking the spikes in load in Cloud SQL, it seems that there is a variety of causes for each spike (analytics workloads, Cody Gateway syncs, code intel workloads, gitserver things, `ListSourcegraphDotComIndexableRepos` etc) so I'm chalking this up to "expected". Since this alert is seen firing on a Cloud instance, let's just relax it for now so that it only fires a critical alert on very significant delays.
2024-01-11 01:14:28 +00:00
- <span class="badge badge-warning">warning</span> worker: 0.1s+ mean blocked seconds per conn request for 10m0s
- <span class="badge badge-critical">critical</span> worker: 0.5s+ mean blocked seconds per conn request for 10m0s
worker: Add skeleton service (#21768)
2021-06-04 19:48:13 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
worker: Add skeleton service (#21768)
2021-06-04 19:48:13 +00:00
Add possible solutions and make docs discoverable (#22945) Add possible solutions and make docs discoverable
2021-07-30 00:30:31 +00:00
- Increase SRC_PGSQL_MAX_OPEN together with giving more memory to the database if needed
Docs: update links to point to new site (#60381) We have a number of docs links in the product that point to the old doc site. Method: - Search the repo for `docs.sourcegraph.com` - Exclude the `doc/` dir, all test fixtures, and `CHANGELOG.md` - For each, replace `docs.sourcegraph.com` with `sourcegraph.com/docs` - Navigate to the resulting URL ensuring it's not a dead link, updating the URL if necessary Many of the URLs updated are just comments, but since I'm doing a manual audit of each URL anyways, I felt it was worth it to update these while I was at it.
2024-02-13 00:23:47 +00:00
- Scale up Postgres memory/cpus - [see our scaling guide](https://sourcegraph.com/docs/admin/config/postgres-conf)
monitoring: relax mean_blocked_seconds_per_conn_request alerts (#59507) https://github.com/sourcegraph/sourcegraph/pull/59284 dramatically reduced the `mean_blocked_seconds_per_conn_request` issues we've been seeing, but overall delays are still higher, even with generally healthy Cloud SQL resource utilization. <img width="1630" alt="image" src="https://github.com/sourcegraph/sourcegraph/assets/23356519/91615471-5187-4d15-83e7-5cc94595303c"> Spot-checking the spikes in load in Cloud SQL, it seems that there is a variety of causes for each spike (analytics workloads, Cody Gateway syncs, code intel workloads, gitserver things, `ListSourcegraphDotComIndexableRepos` etc) so I'm chalking this up to "expected". Since this alert is seen firing on a Cloud instance, let's just relax it for now so that it only fires a critical alert on very significant delays.
2024-01-11 01:14:28 +00:00
- If using GCP Cloud SQL, check for high lock waits or CPU usage in query insights
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#worker-mean-blocked-seconds-per-conn-request).
worker: Add skeleton service (#21768)
2021-06-04 19:48:13 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
monitoring: Refactor provisioning dashboards (#23006)
2021-07-20 13:59:13 +00:00
"warning_worker_mean_blocked_seconds_per_conn_request",
"critical_worker_mean_blocked_seconds_per_conn_request"
worker: Add skeleton service (#21768)
2021-06-04 19:48:13 +00:00
]
```
monitoring: test owners for valid Opsgenie teams and handbook pages (#59251) In INC-264 it seems that certain alerts - such as [zoekt: less than 90% percentage pods available for 10m0s](https://opsg.in/a/i/sourcegraph/178a626f-0f28-4295-bee9-84da988bb473-1703759057681) - don't seem to end up going anywhere because the ObservableOwner is defunct. This change adds _opt-in_ testing to report: 1. How many owners have valid Opsgenie teams 2. How many owners have valid handbook pages In addition, we collect ObservableOwners that pass the test and use it to generate configuration for `site.json` in Sourcegraph.com: https://github.com/sourcegraph/deploy-sourcegraph-cloud/pull/18338 - this helps ensure the list is valid and not deceptively high-coverage. The results are not great, but **enforcing** that owners are valid isn't currently in scope: ``` 6/10 ObservableOwners do not have valid Opsgenie teams 3/10 ObservableOwners do not point to valid handbook pages ``` I also removed some defunct/unused functionality/owners. ## Test plan To run these tests: ``` export OPSGENIE_API_KEY="..." go test -timeout 30s github.com/sourcegraph/sourcegraph/monitoring/monitoring -update -online ```
2023-12-29 22:07:35 +00:00
<sub>*Managed by the [Sourcegraph Infrastructure Org team](https://handbook.sourcegraph.com/departments/engineering/infrastructure).*</sub>
worker: Add skeleton service (#21768)
2021-06-04 19:48:13 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
monitoring: relax mean_blocked_seconds_per_conn_request alerts (#59507) https://github.com/sourcegraph/sourcegraph/pull/59284 dramatically reduced the `mean_blocked_seconds_per_conn_request` issues we've been seeing, but overall delays are still higher, even with generally healthy Cloud SQL resource utilization. <img width="1630" alt="image" src="https://github.com/sourcegraph/sourcegraph/assets/23356519/91615471-5187-4d15-83e7-5cc94595303c"> Spot-checking the spikes in load in Cloud SQL, it seems that there is a variety of causes for each spike (analytics workloads, Cody Gateway syncs, code intel workloads, gitserver things, `ListSourcegraphDotComIndexableRepos` etc) so I'm chalking this up to "expected". Since this alert is seen firing on a Cloud instance, let's just relax it for now so that it only fires a critical alert on very significant delays.
2024-01-11 01:14:28 +00:00
Generated query for warning alert: `max((sum by (app_name, db_name) (increase(src_pgsql_conns_blocked_seconds{app_name="worker"}[5m])) / sum by (app_name, db_name) (increase(src_pgsql_conns_waited_for{app_name="worker"}[5m]))) >= 0.1)`
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
monitoring: relax mean_blocked_seconds_per_conn_request alerts (#59507) https://github.com/sourcegraph/sourcegraph/pull/59284 dramatically reduced the `mean_blocked_seconds_per_conn_request` issues we've been seeing, but overall delays are still higher, even with generally healthy Cloud SQL resource utilization. <img width="1630" alt="image" src="https://github.com/sourcegraph/sourcegraph/assets/23356519/91615471-5187-4d15-83e7-5cc94595303c"> Spot-checking the spikes in load in Cloud SQL, it seems that there is a variety of causes for each spike (analytics workloads, Cody Gateway syncs, code intel workloads, gitserver things, `ListSourcegraphDotComIndexableRepos` etc) so I'm chalking this up to "expected". Since this alert is seen firing on a Cloud instance, let's just relax it for now so that it only fires a critical alert on very significant delays.
2024-01-11 01:14:28 +00:00
Generated query for critical alert: `max((sum by (app_name, db_name) (increase(src_pgsql_conns_blocked_seconds{app_name="worker"}[5m])) / sum by (app_name, db_name) (increase(src_pgsql_conns_waited_for{app_name="worker"}[5m]))) >= 0.5)`
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
</details>
worker: Add skeleton service (#21768)
2021-06-04 19:48:13 +00:00
<br />
## worker: container_cpu_usage
<p class="subtitle">container cpu usage total (1m average) across all cores by instance</p>
**Descriptions**
- <span class="badge badge-warning">warning</span> worker: 99%+ container cpu usage total (1m average) across all cores by instance
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
worker: Add skeleton service (#21768)
2021-06-04 19:48:13 +00:00
- **Kubernetes:** Consider increasing CPU limits in the the relevant `Deployment.yaml`.
- **Docker Compose:** Consider increasing `cpus:` of the worker container in `docker-compose.yml`.
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#worker-container-cpu-usage).
worker: Add skeleton service (#21768)
2021-06-04 19:48:13 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"warning_worker_container_cpu_usage"
]
```
monitoring: rename Cloud SaaS to IAM (#38268)
2022-07-07 02:13:27 +00:00
<sub>*Managed by the [Sourcegraph Code intelligence team](https://handbook.sourcegraph.com/departments/engineering/teams/code-intelligence).*</sub>
worker: Add skeleton service (#21768)
2021-06-04 19:48:13 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
Generated query for warning alert: `max((cadvisor_container_cpu_usage_percentage_total{name=~"^worker.*"}) >= 99)`
</details>
worker: Add skeleton service (#21768)
2021-06-04 19:48:13 +00:00
<br />
## worker: container_memory_usage
<p class="subtitle">container memory usage by instance</p>
**Descriptions**
- <span class="badge badge-warning">warning</span> worker: 99%+ container memory usage by instance
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
worker: Add skeleton service (#21768)
2021-06-04 19:48:13 +00:00
- **Kubernetes:** Consider increasing memory limit in relevant `Deployment.yaml`.
- **Docker Compose:** Consider increasing `memory:` of worker container in `docker-compose.yml`.
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#worker-container-memory-usage).
worker: Add skeleton service (#21768)
2021-06-04 19:48:13 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"warning_worker_container_memory_usage"
]
```
monitoring: rename Cloud SaaS to IAM (#38268)
2022-07-07 02:13:27 +00:00
<sub>*Managed by the [Sourcegraph Code intelligence team](https://handbook.sourcegraph.com/departments/engineering/teams/code-intelligence).*</sub>
worker: Add skeleton service (#21768)
2021-06-04 19:48:13 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
Generated query for warning alert: `max((cadvisor_container_memory_usage_percentage_total{name=~"^worker.*"}) >= 99)`
</details>
worker: Add skeleton service (#21768)
2021-06-04 19:48:13 +00:00
<br />
## worker: provisioning_container_cpu_usage_long_term
<p class="subtitle">container cpu usage total (90th percentile over 1d) across all cores by instance</p>
**Descriptions**
- <span class="badge badge-warning">warning</span> worker: 80%+ container cpu usage total (90th percentile over 1d) across all cores by instance for 336h0m0s
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
worker: Add skeleton service (#21768)
2021-06-04 19:48:13 +00:00
- **Kubernetes:** Consider increasing CPU limits in the `Deployment.yaml` for the worker service.
- **Docker Compose:** Consider increasing `cpus:` of the worker container in `docker-compose.yml`.
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#worker-provisioning-container-cpu-usage-long-term).
worker: Add skeleton service (#21768)
2021-06-04 19:48:13 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"warning_worker_provisioning_container_cpu_usage_long_term"
]
```
monitoring: rename Cloud SaaS to IAM (#38268)
2022-07-07 02:13:27 +00:00
<sub>*Managed by the [Sourcegraph Code intelligence team](https://handbook.sourcegraph.com/departments/engineering/teams/code-intelligence).*</sub>
worker: Add skeleton service (#21768)
2021-06-04 19:48:13 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
Generated query for warning alert: `max((quantile_over_time(0.9, cadvisor_container_cpu_usage_percentage_total{name=~"^worker.*"}[1d])) >= 80)`
</details>
worker: Add skeleton service (#21768)
2021-06-04 19:48:13 +00:00
<br />
## worker: provisioning_container_memory_usage_long_term
<p class="subtitle">container memory usage (1d maximum) by instance</p>
**Descriptions**
- <span class="badge badge-warning">warning</span> worker: 80%+ container memory usage (1d maximum) by instance for 336h0m0s
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
worker: Add skeleton service (#21768)
2021-06-04 19:48:13 +00:00
- **Kubernetes:** Consider increasing memory limits in the `Deployment.yaml` for the worker service.
- **Docker Compose:** Consider increasing `memory:` of the worker container in `docker-compose.yml`.
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#worker-provisioning-container-memory-usage-long-term).
worker: Add skeleton service (#21768)
2021-06-04 19:48:13 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"warning_worker_provisioning_container_memory_usage_long_term"
]
```
monitoring: rename Cloud SaaS to IAM (#38268)
2022-07-07 02:13:27 +00:00
<sub>*Managed by the [Sourcegraph Code intelligence team](https://handbook.sourcegraph.com/departments/engineering/teams/code-intelligence).*</sub>
worker: Add skeleton service (#21768)
2021-06-04 19:48:13 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
Generated query for warning alert: `max((max_over_time(cadvisor_container_memory_usage_percentage_total{name=~"^worker.*"}[1d])) >= 80)`
</details>
worker: Add skeleton service (#21768)
2021-06-04 19:48:13 +00:00
<br />
## worker: provisioning_container_cpu_usage_short_term
<p class="subtitle">container cpu usage total (5m maximum) across all cores by instance</p>
**Descriptions**
- <span class="badge badge-warning">warning</span> worker: 90%+ container cpu usage total (5m maximum) across all cores by instance for 30m0s
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
worker: Add skeleton service (#21768)
2021-06-04 19:48:13 +00:00
- **Kubernetes:** Consider increasing CPU limits in the the relevant `Deployment.yaml`.
- **Docker Compose:** Consider increasing `cpus:` of the worker container in `docker-compose.yml`.
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#worker-provisioning-container-cpu-usage-short-term).
worker: Add skeleton service (#21768)
2021-06-04 19:48:13 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"warning_worker_provisioning_container_cpu_usage_short_term"
]
```
monitoring: rename Cloud SaaS to IAM (#38268)
2022-07-07 02:13:27 +00:00
<sub>*Managed by the [Sourcegraph Code intelligence team](https://handbook.sourcegraph.com/departments/engineering/teams/code-intelligence).*</sub>
worker: Add skeleton service (#21768)
2021-06-04 19:48:13 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
Generated query for warning alert: `max((max_over_time(cadvisor_container_cpu_usage_percentage_total{name=~"^worker.*"}[5m])) >= 90)`
</details>
worker: Add skeleton service (#21768)
2021-06-04 19:48:13 +00:00
<br />
## worker: provisioning_container_memory_usage_short_term
<p class="subtitle">container memory usage (5m maximum) by instance</p>
**Descriptions**
- <span class="badge badge-warning">warning</span> worker: 90%+ container memory usage (5m maximum) by instance
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
worker: Add skeleton service (#21768)
2021-06-04 19:48:13 +00:00
- **Kubernetes:** Consider increasing memory limit in relevant `Deployment.yaml`.
- **Docker Compose:** Consider increasing `memory:` of worker container in `docker-compose.yml`.
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#worker-provisioning-container-memory-usage-short-term).
worker: Add skeleton service (#21768)
2021-06-04 19:48:13 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"warning_worker_provisioning_container_memory_usage_short_term"
]
```
monitoring: rename Cloud SaaS to IAM (#38268)
2022-07-07 02:13:27 +00:00
<sub>*Managed by the [Sourcegraph Code intelligence team](https://handbook.sourcegraph.com/departments/engineering/teams/code-intelligence).*</sub>
worker: Add skeleton service (#21768)
2021-06-04 19:48:13 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
Generated query for warning alert: `max((max_over_time(cadvisor_container_memory_usage_percentage_total{name=~"^worker.*"}[5m])) >= 90)`
</details>
worker: Add skeleton service (#21768)
2021-06-04 19:48:13 +00:00
<br />
monitoring: improve containers metrics (#34808) part of - https://github.com/sourcegraph/sourcegraph/issues/33437 - https://github.com/sourcegraph/sourcegraph/issues/33438 This PR adds the following - A shared observable metric to indicate how many times a container (or child processes) has been oom kill - It's added to provisioning indicators since it's an indicator of underprovisioned resources - A new dashboard that shows resources usage of all containers at a glance - It contains two groups, one for all containers and one for containers that may have potential scaling issues - the first group is extremely noisy so it's hidden by default - the second group provides a more focused experience for site-admin to quickly find problematic resources <!-- > ~it looks like the alert table will be rendered regardless, which is pretty annoying (I guess the implementation of the monitoring stack is based on the assumption that one service/container will have its dashboard)~ __Questions & Notes__ - ~Should I make the new dashboard filterable by container name? like I only want to see the graph of `frontend`. Is that something that's valuable?~ - ~Any other metrics you all would like to see?~ - graph for total available resources on the VM compared to usage from https://github.com/sourcegraph/sourcegraph/issues/33437 is not really doable since we do not have `node-exporter` in anyone of our distro. We would have to tackle this from https://github.com/sourcegraph/customer/issues/806 --> ## Test plan connect to `demo`'s prom ```sh gcloud compute start-iap-tunnel default-$NEW_DEPLOYMENT-instance 9090 --local-host-port=localhost:4445 --zone us-central1-f --project $PROJECT_PREFIX-$CUSTOMER ``` or use dogfood ```sh kubectl port-forward svc/prometheus 4445:30090 ``` Update `dev/grafana/all/datasources.yaml` ```yml # Configuration for all non-Linux platforms. apiVersion: 1 datasources: - name: Prometheus type: prometheus access: proxy url: http://docker.for.mac.localhost:4445 # use the remote prometheus instance on demo instead of the local one isDefault: true editable: false - name: Jaeger type: jaeger access: proxy url: http://host.docker.internal:16686/-/debug/jaeger - name: Loki type: loki access: proxy url: http://host.docker.internal:3100 ``` Start monitoring stack ```sh sg run grafana ``` see screenshot > the first group is hidden by default ![CleanShot 2022-05-04 at 10 48 26](https://user-images.githubusercontent.com/8373004/166748470-22460552-6e51-4b29-a26b-4c5adbe1c404.png) ![CleanShot 2022-05-04 at 10 48 03](https://user-images.githubusercontent.com/8373004/166748396-1d410c35-e5fe-45e2-83bd-418d845413cb.png) ![CleanShot 2022-05-04 at 11 25 13](https://user-images.githubusercontent.com/8373004/166801266-21e51b68-65fd-4cb7-97de-83520e31e498.png)
2022-05-09 23:56:47 +00:00
## worker: container_oomkill_events_total
<p class="subtitle">container OOMKILL events total by instance</p>
**Descriptions**
- <span class="badge badge-warning">warning</span> worker: 1+ container OOMKILL events total by instance
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: improve containers metrics (#34808) part of - https://github.com/sourcegraph/sourcegraph/issues/33437 - https://github.com/sourcegraph/sourcegraph/issues/33438 This PR adds the following - A shared observable metric to indicate how many times a container (or child processes) has been oom kill - It's added to provisioning indicators since it's an indicator of underprovisioned resources - A new dashboard that shows resources usage of all containers at a glance - It contains two groups, one for all containers and one for containers that may have potential scaling issues - the first group is extremely noisy so it's hidden by default - the second group provides a more focused experience for site-admin to quickly find problematic resources <!-- > ~it looks like the alert table will be rendered regardless, which is pretty annoying (I guess the implementation of the monitoring stack is based on the assumption that one service/container will have its dashboard)~ __Questions & Notes__ - ~Should I make the new dashboard filterable by container name? like I only want to see the graph of `frontend`. Is that something that's valuable?~ - ~Any other metrics you all would like to see?~ - graph for total available resources on the VM compared to usage from https://github.com/sourcegraph/sourcegraph/issues/33437 is not really doable since we do not have `node-exporter` in anyone of our distro. We would have to tackle this from https://github.com/sourcegraph/customer/issues/806 --> ## Test plan connect to `demo`'s prom ```sh gcloud compute start-iap-tunnel default-$NEW_DEPLOYMENT-instance 9090 --local-host-port=localhost:4445 --zone us-central1-f --project $PROJECT_PREFIX-$CUSTOMER ``` or use dogfood ```sh kubectl port-forward svc/prometheus 4445:30090 ``` Update `dev/grafana/all/datasources.yaml` ```yml # Configuration for all non-Linux platforms. apiVersion: 1 datasources: - name: Prometheus type: prometheus access: proxy url: http://docker.for.mac.localhost:4445 # use the remote prometheus instance on demo instead of the local one isDefault: true editable: false - name: Jaeger type: jaeger access: proxy url: http://host.docker.internal:16686/-/debug/jaeger - name: Loki type: loki access: proxy url: http://host.docker.internal:3100 ``` Start monitoring stack ```sh sg run grafana ``` see screenshot > the first group is hidden by default ![CleanShot 2022-05-04 at 10 48 26](https://user-images.githubusercontent.com/8373004/166748470-22460552-6e51-4b29-a26b-4c5adbe1c404.png) ![CleanShot 2022-05-04 at 10 48 03](https://user-images.githubusercontent.com/8373004/166748396-1d410c35-e5fe-45e2-83bd-418d845413cb.png) ![CleanShot 2022-05-04 at 11 25 13](https://user-images.githubusercontent.com/8373004/166801266-21e51b68-65fd-4cb7-97de-83520e31e498.png)
2022-05-09 23:56:47 +00:00
- **Kubernetes:** Consider increasing memory limit in relevant `Deployment.yaml`.
- **Docker Compose:** Consider increasing `memory:` of worker container in `docker-compose.yml`.
- More help interpreting this metric is available in the [dashboards reference](./dashboards.md#worker-container-oomkill-events-total).
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"warning_worker_container_oomkill_events_total"
]
```
monitoring: rename Cloud SaaS to IAM (#38268)
2022-07-07 02:13:27 +00:00
<sub>*Managed by the [Sourcegraph Code intelligence team](https://handbook.sourcegraph.com/departments/engineering/teams/code-intelligence).*</sub>
monitoring: improve containers metrics (#34808) part of - https://github.com/sourcegraph/sourcegraph/issues/33437 - https://github.com/sourcegraph/sourcegraph/issues/33438 This PR adds the following - A shared observable metric to indicate how many times a container (or child processes) has been oom kill - It's added to provisioning indicators since it's an indicator of underprovisioned resources - A new dashboard that shows resources usage of all containers at a glance - It contains two groups, one for all containers and one for containers that may have potential scaling issues - the first group is extremely noisy so it's hidden by default - the second group provides a more focused experience for site-admin to quickly find problematic resources <!-- > ~it looks like the alert table will be rendered regardless, which is pretty annoying (I guess the implementation of the monitoring stack is based on the assumption that one service/container will have its dashboard)~ __Questions & Notes__ - ~Should I make the new dashboard filterable by container name? like I only want to see the graph of `frontend`. Is that something that's valuable?~ - ~Any other metrics you all would like to see?~ - graph for total available resources on the VM compared to usage from https://github.com/sourcegraph/sourcegraph/issues/33437 is not really doable since we do not have `node-exporter` in anyone of our distro. We would have to tackle this from https://github.com/sourcegraph/customer/issues/806 --> ## Test plan connect to `demo`'s prom ```sh gcloud compute start-iap-tunnel default-$NEW_DEPLOYMENT-instance 9090 --local-host-port=localhost:4445 --zone us-central1-f --project $PROJECT_PREFIX-$CUSTOMER ``` or use dogfood ```sh kubectl port-forward svc/prometheus 4445:30090 ``` Update `dev/grafana/all/datasources.yaml` ```yml # Configuration for all non-Linux platforms. apiVersion: 1 datasources: - name: Prometheus type: prometheus access: proxy url: http://docker.for.mac.localhost:4445 # use the remote prometheus instance on demo instead of the local one isDefault: true editable: false - name: Jaeger type: jaeger access: proxy url: http://host.docker.internal:16686/-/debug/jaeger - name: Loki type: loki access: proxy url: http://host.docker.internal:3100 ``` Start monitoring stack ```sh sg run grafana ``` see screenshot > the first group is hidden by default ![CleanShot 2022-05-04 at 10 48 26](https://user-images.githubusercontent.com/8373004/166748470-22460552-6e51-4b29-a26b-4c5adbe1c404.png) ![CleanShot 2022-05-04 at 10 48 03](https://user-images.githubusercontent.com/8373004/166748396-1d410c35-e5fe-45e2-83bd-418d845413cb.png) ![CleanShot 2022-05-04 at 11 25 13](https://user-images.githubusercontent.com/8373004/166801266-21e51b68-65fd-4cb7-97de-83520e31e498.png)
2022-05-09 23:56:47 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
monitoring: extract into a submodule (#45786) This change extracts `monitoring` into a submodule for import in `sourcegraph/controller` (https://github.com/sourcegraph/controller/pull/195) so that we can generate dashboards for Cloud instances. These steps were required: 1. Initialize a `go.mod` in `monitoring` 2. Extract `dev/sg/internal/cliutil` into `lib` to avoid illegal imports from `monitoring` 3. Add local replaces to both `sourcegraph/sourcegraph` and `monitoring` 4. `go mod tidy` on all submodules 5. Update `go generate ./monitoring` commands to use `sg`, since the `go generate` command no longer works 6. Update `grafana/build.sh`, `prometheus/build.sh` to build the submodule 7. Amend linters to check for multiple `go.mod` files and ban imports of `github.com/sourcegraph/sourcegraph` 8. Update `sg generate go` to run in directories rather than from root The only caveat is that if you use VS code, you will now need to open `monitoring` in a separate workspace or similar, like with `lib`. Co-authored-by: Joe Chen <joe@sourcegraph.com>
2022-12-19 17:49:25 +00:00
Generated query for warning alert: `max((max by (name) (container_oom_events_total{name=~"^worker.*"})) >= 1)`
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
</details>
monitoring: improve containers metrics (#34808) part of - https://github.com/sourcegraph/sourcegraph/issues/33437 - https://github.com/sourcegraph/sourcegraph/issues/33438 This PR adds the following - A shared observable metric to indicate how many times a container (or child processes) has been oom kill - It's added to provisioning indicators since it's an indicator of underprovisioned resources - A new dashboard that shows resources usage of all containers at a glance - It contains two groups, one for all containers and one for containers that may have potential scaling issues - the first group is extremely noisy so it's hidden by default - the second group provides a more focused experience for site-admin to quickly find problematic resources <!-- > ~it looks like the alert table will be rendered regardless, which is pretty annoying (I guess the implementation of the monitoring stack is based on the assumption that one service/container will have its dashboard)~ __Questions & Notes__ - ~Should I make the new dashboard filterable by container name? like I only want to see the graph of `frontend`. Is that something that's valuable?~ - ~Any other metrics you all would like to see?~ - graph for total available resources on the VM compared to usage from https://github.com/sourcegraph/sourcegraph/issues/33437 is not really doable since we do not have `node-exporter` in anyone of our distro. We would have to tackle this from https://github.com/sourcegraph/customer/issues/806 --> ## Test plan connect to `demo`'s prom ```sh gcloud compute start-iap-tunnel default-$NEW_DEPLOYMENT-instance 9090 --local-host-port=localhost:4445 --zone us-central1-f --project $PROJECT_PREFIX-$CUSTOMER ``` or use dogfood ```sh kubectl port-forward svc/prometheus 4445:30090 ``` Update `dev/grafana/all/datasources.yaml` ```yml # Configuration for all non-Linux platforms. apiVersion: 1 datasources: - name: Prometheus type: prometheus access: proxy url: http://docker.for.mac.localhost:4445 # use the remote prometheus instance on demo instead of the local one isDefault: true editable: false - name: Jaeger type: jaeger access: proxy url: http://host.docker.internal:16686/-/debug/jaeger - name: Loki type: loki access: proxy url: http://host.docker.internal:3100 ``` Start monitoring stack ```sh sg run grafana ``` see screenshot > the first group is hidden by default ![CleanShot 2022-05-04 at 10 48 26](https://user-images.githubusercontent.com/8373004/166748470-22460552-6e51-4b29-a26b-4c5adbe1c404.png) ![CleanShot 2022-05-04 at 10 48 03](https://user-images.githubusercontent.com/8373004/166748396-1d410c35-e5fe-45e2-83bd-418d845413cb.png) ![CleanShot 2022-05-04 at 11 25 13](https://user-images.githubusercontent.com/8373004/166801266-21e51b68-65fd-4cb7-97de-83520e31e498.png)
2022-05-09 23:56:47 +00:00
<br />
worker: Add skeleton service (#21768)
2021-06-04 19:48:13 +00:00
## worker: go_goroutines
<p class="subtitle">maximum active goroutines</p>
**Descriptions**
- <span class="badge badge-warning">warning</span> worker: 10000+ maximum active goroutines for 10m0s
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
worker: Add skeleton service (#21768)
2021-06-04 19:48:13 +00:00
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- More help interpreting this metric is available in the [dashboards reference](./dashboards.md#worker-go-goroutines).
worker: Add skeleton service (#21768)
2021-06-04 19:48:13 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"warning_worker_go_goroutines"
]
```
monitoring: rename Cloud SaaS to IAM (#38268)
2022-07-07 02:13:27 +00:00
<sub>*Managed by the [Sourcegraph Code intelligence team](https://handbook.sourcegraph.com/departments/engineering/teams/code-intelligence).*</sub>
worker: Add skeleton service (#21768)
2021-06-04 19:48:13 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
monitoring: extract into a submodule (#45786) This change extracts `monitoring` into a submodule for import in `sourcegraph/controller` (https://github.com/sourcegraph/controller/pull/195) so that we can generate dashboards for Cloud instances. These steps were required: 1. Initialize a `go.mod` in `monitoring` 2. Extract `dev/sg/internal/cliutil` into `lib` to avoid illegal imports from `monitoring` 3. Add local replaces to both `sourcegraph/sourcegraph` and `monitoring` 4. `go mod tidy` on all submodules 5. Update `go generate ./monitoring` commands to use `sg`, since the `go generate` command no longer works 6. Update `grafana/build.sh`, `prometheus/build.sh` to build the submodule 7. Amend linters to check for multiple `go.mod` files and ban imports of `github.com/sourcegraph/sourcegraph` 8. Update `sg generate go` to run in directories rather than from root The only caveat is that if you use VS code, you will now need to open `monitoring` in a separate workspace or similar, like with `lib`. Co-authored-by: Joe Chen <joe@sourcegraph.com>
2022-12-19 17:49:25 +00:00
Generated query for warning alert: `max((max by (instance) (go_goroutines{job=~".*worker"})) >= 10000)`
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
</details>
worker: Add skeleton service (#21768)
2021-06-04 19:48:13 +00:00
<br />
## worker: go_gc_duration_seconds
<p class="subtitle">maximum go garbage collection duration</p>
**Descriptions**
- <span class="badge badge-warning">warning</span> worker: 2s+ maximum go garbage collection duration
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
worker: Add skeleton service (#21768)
2021-06-04 19:48:13 +00:00
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#worker-go-gc-duration-seconds).
worker: Add skeleton service (#21768)
2021-06-04 19:48:13 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"warning_worker_go_gc_duration_seconds"
]
```
monitoring: rename Cloud SaaS to IAM (#38268)
2022-07-07 02:13:27 +00:00
<sub>*Managed by the [Sourcegraph Code intelligence team](https://handbook.sourcegraph.com/departments/engineering/teams/code-intelligence).*</sub>
worker: Add skeleton service (#21768)
2021-06-04 19:48:13 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
monitoring: extract into a submodule (#45786) This change extracts `monitoring` into a submodule for import in `sourcegraph/controller` (https://github.com/sourcegraph/controller/pull/195) so that we can generate dashboards for Cloud instances. These steps were required: 1. Initialize a `go.mod` in `monitoring` 2. Extract `dev/sg/internal/cliutil` into `lib` to avoid illegal imports from `monitoring` 3. Add local replaces to both `sourcegraph/sourcegraph` and `monitoring` 4. `go mod tidy` on all submodules 5. Update `go generate ./monitoring` commands to use `sg`, since the `go generate` command no longer works 6. Update `grafana/build.sh`, `prometheus/build.sh` to build the submodule 7. Amend linters to check for multiple `go.mod` files and ban imports of `github.com/sourcegraph/sourcegraph` 8. Update `sg generate go` to run in directories rather than from root The only caveat is that if you use VS code, you will now need to open `monitoring` in a separate workspace or similar, like with `lib`. Co-authored-by: Joe Chen <joe@sourcegraph.com>
2022-12-19 17:49:25 +00:00
Generated query for warning alert: `max((max by (instance) (go_gc_duration_seconds{job=~".*worker"})) >= 2)`
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
</details>
worker: Add skeleton service (#21768)
2021-06-04 19:48:13 +00:00
<br />
## worker: pods_available_percentage
<p class="subtitle">percentage pods available</p>
**Descriptions**
- <span class="badge badge-critical">critical</span> worker: less than 90% percentage pods available for 10m0s
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
worker: Add skeleton service (#21768)
2021-06-04 19:48:13 +00:00
monitoring: forbid critical alerts with no solutions (#36423) Adds the restriction that critical alerts must have solutions defined as a hard requirement within the generator, and fixes all the outstanding issues: - Add PossibleSolutions for the pods_available_percentage and pg_up critical alerts (we have a similar solution doc in the container_missing alert) - Removes all the other critical alerts without possible solutions Co-authored-by: Michael Lin <mlzc@hey.com>
2022-06-02 15:51:43 +00:00
- Determine if the pod was OOM killed using `kubectl describe pod worker` (look for `OOMKilled: true`) and, if so, consider increasing the memory limit in the relevant `Deployment.yaml`.
- Check the logs before the container restarted to see if there are `panic:` messages or similar using `kubectl logs -p worker`.
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#worker-pods-available-percentage).
worker: Add skeleton service (#21768)
2021-06-04 19:48:13 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"critical_worker_pods_available_percentage"
]
```
monitoring: rename Cloud SaaS to IAM (#38268)
2022-07-07 02:13:27 +00:00
<sub>*Managed by the [Sourcegraph Code intelligence team](https://handbook.sourcegraph.com/departments/engineering/teams/code-intelligence).*</sub>
worker: Add skeleton service (#21768)
2021-06-04 19:48:13 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
worker: Add skeleton service (#21768)
2021-06-04 19:48:13 +00:00
monitoring: extract into a submodule (#45786) This change extracts `monitoring` into a submodule for import in `sourcegraph/controller` (https://github.com/sourcegraph/controller/pull/195) so that we can generate dashboards for Cloud instances. These steps were required: 1. Initialize a `go.mod` in `monitoring` 2. Extract `dev/sg/internal/cliutil` into `lib` to avoid illegal imports from `monitoring` 3. Add local replaces to both `sourcegraph/sourcegraph` and `monitoring` 4. `go mod tidy` on all submodules 5. Update `go generate ./monitoring` commands to use `sg`, since the `go generate` command no longer works 6. Update `grafana/build.sh`, `prometheus/build.sh` to build the submodule 7. Amend linters to check for multiple `go.mod` files and ban imports of `github.com/sourcegraph/sourcegraph` 8. Update `sg generate go` to run in directories rather than from root The only caveat is that if you use VS code, you will now need to open `monitoring` in a separate workspace or similar, like with `lib`. Co-authored-by: Joe Chen <joe@sourcegraph.com>
2022-12-19 17:49:25 +00:00
Generated query for critical alert: `min((sum by (app) (up{app=~".*worker"}) / count by (app) (up{app=~".*worker"}) * 100) <= 90)`
monitoring: add more panels to Grafana repo-updater dashboard (#14269) Improves the repo-updater Grafana dashboard with many new panels as well as observing more of repo-updater's internal state. * monitoring: add more panels to Grafana repo-updater dashboard * Remove comment and fix typo * Add dashboard panels for external_service_sync_jobs state * Increase alert threshold * Remove duplicate metrics schedKnownRepos and schedHeapCount were measuring the same thing in different ways and the old version of schedKnownRepos was assuming a single syncer. Remove the old version of schedKnowRepos and use schedHeapCount instead. * Observe the number of repos queued for update * Fix observable name * Observe when a sync was started Grouped by sync "family" and external service id * Improve metric name * Don't tag metrics with external service id This could increase the cardinality of our metrics too much * Observe the time since the oldest sync * Simplify function Co-authored-by: Ryan Slade <ryanslade@gmail.com>
2020-10-08 15:27:12 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
</details>
<br />
conf: add metric and associated alert if clients fail to update site configuration within 5 minutes (#57682)
2023-10-18 23:53:55 +00:00
## worker: worker_site_configuration_duration_since_last_successful_update_by_instance
<p class="subtitle">maximum duration since last successful site configuration update (all "worker" instances)</p>
**Descriptions**
- <span class="badge badge-critical">critical</span> worker: 300s+ maximum duration since last successful site configuration update (all "worker" instances)
**Next steps**
- This indicates that one or more "worker" instances have not successfully updated the site configuration in over 5 minutes. This could be due to networking issues between services or problems with the site configuration service itself.
- Check for relevant errors in the "worker" logs, as well as frontend`s logs.
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#worker-worker-site-configuration-duration-since-last-successful-update-by-instance).
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"critical_worker_worker_site_configuration_duration_since_last_successful_update_by_instance"
]
```
monitoring: test owners for valid Opsgenie teams and handbook pages (#59251) In INC-264 it seems that certain alerts - such as [zoekt: less than 90% percentage pods available for 10m0s](https://opsg.in/a/i/sourcegraph/178a626f-0f28-4295-bee9-84da988bb473-1703759057681) - don't seem to end up going anywhere because the ObservableOwner is defunct. This change adds _opt-in_ testing to report: 1. How many owners have valid Opsgenie teams 2. How many owners have valid handbook pages In addition, we collect ObservableOwners that pass the test and use it to generate configuration for `site.json` in Sourcegraph.com: https://github.com/sourcegraph/deploy-sourcegraph-cloud/pull/18338 - this helps ensure the list is valid and not deceptively high-coverage. The results are not great, but **enforcing** that owners are valid isn't currently in scope: ``` 6/10 ObservableOwners do not have valid Opsgenie teams 3/10 ObservableOwners do not point to valid handbook pages ``` I also removed some defunct/unused functionality/owners. ## Test plan To run these tests: ``` export OPSGENIE_API_KEY="..." go test -timeout 30s github.com/sourcegraph/sourcegraph/monitoring/monitoring -update -online ```
2023-12-29 22:07:35 +00:00
<sub>*Managed by the [Sourcegraph Infrastructure Org team](https://handbook.sourcegraph.com/departments/engineering/infrastructure).*</sub>
conf: add metric and associated alert if clients fail to update site configuration within 5 minutes (#57682)
2023-10-18 23:53:55 +00:00
<details>
<summary>Technical details</summary>
monitoring: fix alert definition for site configuration by adding scrape job label (#59687) We discovered recently that the definition for the alert that fires if the site configuration hasn't been fetched within 5 minutes strips out the regex that targets individual services (since it uses a grafana variable). This means that every instance of this alert will fire if any individual service trips over this threshold. This PR fixes the issue by adding a new `job` filter for this alert that targets only the services that have that Prometheus scrape target name. This works around the previous issue by using a fixed value for the `job` value instead of a dynamic grafana value. The value of the job filter generally looks like `job=~.*$container_name` (following the strategy from https://sourcegraph.com/github.com/sourcegraph/sourcegraph@9a780f2e694238b5326e3e121d6a1828463001b9/-/blob/monitoring/monitoring/monitoring.go?L161 ) unless I noticed that there was different logic in the existing dashboard for the services. Ex: - `frontend`: already used `job=~"(sourcegraph-)?frontend"` for some metrics, so I used it again here - `worker`: `already used `job=~"^worker.*"` in some metrics, so I used it again and standarized the other existing panels to use the same shared variable ## Test plan I eyeballed the generated alert.md and dashboards.md to verify that my changes looked correct (that is, my refactors resulted in either no diff, or that the diff I generated still looked like valid regex).
2024-01-17 23:19:54 +00:00
Generated query for critical alert: `max((max(max_over_time(src_conf_client_time_since_last_successful_update_seconds{job=~"^worker.*"}[1m]))) >= 300)`
conf: add metric and associated alert if clients fail to update site configuration within 5 minutes (#57682)
2023-10-18 23:53:55 +00:00
</details>
<br />
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
## repo-updater: src_repoupdater_max_sync_backoff
<p class="subtitle">time since oldest sync</p>
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
monitoring: add more panels to Grafana repo-updater dashboard (#14269) Improves the repo-updater Grafana dashboard with many new panels as well as observing more of repo-updater's internal state. * monitoring: add more panels to Grafana repo-updater dashboard * Remove comment and fix typo * Add dashboard panels for external_service_sync_jobs state * Increase alert threshold * Remove duplicate metrics schedKnownRepos and schedHeapCount were measuring the same thing in different ways and the old version of schedKnownRepos was assuming a single syncer. Remove the old version of schedKnowRepos and use schedHeapCount instead. * Observe the number of repos queued for update * Fix observable name * Observe when a sync was started Grouped by sync "family" and external service id * Improve metric name * Don't tag metrics with external service id This could increase the cardinality of our metrics too much * Observe the time since the oldest sync * Simplify function Co-authored-by: Ryan Slade <ryanslade@gmail.com>
2020-10-08 15:27:12 +00:00
monitoring: dashboards reference, refresh observability docs (#16939) - New monitoring definition field, 'Interpretation' - New generated dashboards reference - Grafana panels now link to alert solutions and dashboards reference - Refreshed observability docs Co-authored-by: uwedeportivo <534011+uwedeportivo@users.noreply.github.com>
2020-12-23 02:36:41 +00:00
- <span class="badge badge-critical">critical</span> repo-updater: 32400s+ time since oldest sync for 10m0s
monitoring: add more panels to Grafana repo-updater dashboard (#14269) Improves the repo-updater Grafana dashboard with many new panels as well as observing more of repo-updater's internal state. * monitoring: add more panels to Grafana repo-updater dashboard * Remove comment and fix typo * Add dashboard panels for external_service_sync_jobs state * Increase alert threshold * Remove duplicate metrics schedKnownRepos and schedHeapCount were measuring the same thing in different ways and the old version of schedKnownRepos was assuming a single syncer. Remove the old version of schedKnowRepos and use schedHeapCount instead. * Observe the number of repos queued for update * Fix observable name * Observe when a sync was started Grouped by sync "family" and external service id * Improve metric name * Don't tag metrics with external service id This could increase the cardinality of our metrics too much * Observe the time since the oldest sync * Simplify function Co-authored-by: Ryan Slade <ryanslade@gmail.com>
2020-10-08 15:27:12 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: add more panels to Grafana repo-updater dashboard (#14269) Improves the repo-updater Grafana dashboard with many new panels as well as observing more of repo-updater's internal state. * monitoring: add more panels to Grafana repo-updater dashboard * Remove comment and fix typo * Add dashboard panels for external_service_sync_jobs state * Increase alert threshold * Remove duplicate metrics schedKnownRepos and schedHeapCount were measuring the same thing in different ways and the old version of schedKnownRepos was assuming a single syncer. Remove the old version of schedKnowRepos and use schedHeapCount instead. * Observe the number of repos queued for update * Fix observable name * Observe when a sync was started Grouped by sync "family" and external service id * Improve metric name * Don't tag metrics with external service id This could increase the cardinality of our metrics too much * Observe the time since the oldest sync * Simplify function Co-authored-by: Ryan Slade <ryanslade@gmail.com>
2020-10-08 15:27:12 +00:00
monitoring: Improve possible solutions for repo-updater alert (#15614) monitoring: Improve possible solutions for repo-updater alert Co-authored-by: ᴜɴᴋɴᴡᴏɴ <joe@sourcegraph.com>
2020-11-11 19:51:22 +00:00
- An alert here indicates that no code host connections have synced in at least 9h0m0s. This indicates that there could be a configuration issue
with your code hosts connections or networking issues affecting communication with your code hosts.
- Check the code host status indicator (cloud icon in top right of Sourcegraph homepage) for errors.
- Make sure external services do not have invalid tokens by navigating to them in the web UI and clicking save. If there are no errors, they are valid.
- Check the repo-updater logs for errors about syncing.
- Confirm that outbound network connections are allowed where repo-updater is deployed.
- Check back in an hour to see if the issue has resolved itself.
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#repo-updater-src-repoupdater-max-sync-backoff).
monitoring: add more panels to Grafana repo-updater dashboard (#14269) Improves the repo-updater Grafana dashboard with many new panels as well as observing more of repo-updater's internal state. * monitoring: add more panels to Grafana repo-updater dashboard * Remove comment and fix typo * Add dashboard panels for external_service_sync_jobs state * Increase alert threshold * Remove duplicate metrics schedKnownRepos and schedHeapCount were measuring the same thing in different ways and the old version of schedKnownRepos was assuming a single syncer. Remove the old version of schedKnowRepos and use schedHeapCount instead. * Observe the number of repos queued for update * Fix observable name * Observe when a sync was started Grouped by sync "family" and external service id * Improve metric name * Don't tag metrics with external service id This could increase the cardinality of our metrics too much * Observe the time since the oldest sync * Simplify function Co-authored-by: Ryan Slade <ryanslade@gmail.com>
2020-10-08 15:27:12 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
repo-updater: Add critical alerts (#14530) * repo-updater: Add critical alerts The number of user added repos is greater than 90% of our hard limit of 200k. Number of external services added is great than 20k No external services have synced in more than 8 hours. By default we backoff for up to 8 hours, so anything higher than this indicates a problem. * syncer_sync_start should be a rate Alert when we've performed <= 1 syncs for 8 hours. Ideally we want this to be <= 0 but our monitoring package doesn't allow this. Therefore we need to make this a warning as it'll fire for instances that only have 1 external service defined. * Fill in possible solutions and remove unnecessary panel * Adjust rate * Update monitoring/repo_updater.go Co-authored-by: Tomás Senart <tomas@sourcegraph.com> * Update monitoring/repo_updater.go Co-authored-by: Robert Lin <robert@bobheadxi.dev> * Update possible error descriptions * Fix alert * Switch alert to critical Co-authored-by: Tomás Senart <tomas@sourcegraph.com> Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2020-10-09 13:58:35 +00:00
"critical_repo-updater_src_repoupdater_max_sync_backoff"
monitoring: add more panels to Grafana repo-updater dashboard (#14269) Improves the repo-updater Grafana dashboard with many new panels as well as observing more of repo-updater's internal state. * monitoring: add more panels to Grafana repo-updater dashboard * Remove comment and fix typo * Add dashboard panels for external_service_sync_jobs state * Increase alert threshold * Remove duplicate metrics schedKnownRepos and schedHeapCount were measuring the same thing in different ways and the old version of schedKnownRepos was assuming a single syncer. Remove the old version of schedKnowRepos and use schedHeapCount instead. * Observe the number of repos queued for update * Fix observable name * Observe when a sync was started Grouped by sync "family" and external service id * Improve metric name * Don't tag metrics with external service id This could increase the cardinality of our metrics too much * Observe the time since the oldest sync * Simplify function Co-authored-by: Ryan Slade <ryanslade@gmail.com>
2020-10-08 15:27:12 +00:00
]
```
Update o11y ownership from IAM/repo-mgmt to Source (#54368) Does what it says in the title. ## Test plan - Ran `sg generate`
2023-06-28 13:49:16 +00:00
<sub>*Managed by the [Sourcegraph Source team](https://handbook.sourcegraph.com/departments/engineering/teams/source).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
Generated query for critical alert: `max((max(src_repoupdater_max_sync_backoff)) >= 32400)`
</details>
monitoring: improve solutions formatting, add team information (#14878)
2020-10-20 05:11:20 +00:00
<br />
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
repo-updater: Instrument code host sync errors. (#16505)
2020-12-07 13:50:33 +00:00
## repo-updater: src_repoupdater_syncer_sync_errors_total
monitoring: Only monitor site level external services error rates (#22292) Tweak monitoring to only consider site level external services since we have no control over how user added external services are configured. Also tweak our alert that looks for zero sync for 9 hours so that the rate is also calculated over 9 hours. If not, then external services that only sync once every eight hours would also trigger our alert since our old rate was only calculated over 5 minutes.
2021-06-23 12:15:22 +00:00
<p class="subtitle">site level external service sync error rate</p>
repo-updater: Instrument code host sync errors. (#16505)
2020-12-07 13:50:33 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
repo-updater: Instrument code host sync errors. (#16505)
2020-12-07 13:50:33 +00:00
monitoring: Update syncer sync errors alerting (#24071) In this commit we: 1. Add a Warning alert if the threshold is greater than 0.5 over 10 minutes. 2. Modify the Critical alert to fire if the threshold is greater than 1 over 10 minutes. Historically, over the last six months approximately, the value of this metric has rarely gone above 0.5 and has recovered almost immediately. Only in one instance has this gone over 1. At the moment we get paged for this alert for what might be intermittent errors on the code host's end which get resolved immediately, reducing the signal to noise ration in our alerting. Link to production dashboard for reference: https://sourcegraph.com/-/debug/grafana/explore?orgId=1&left=%5B%221614537000000%22,%221629484199000%22,%22Prometheus%22,%7B%22expr%22:%22max%20by%20(family)%20(rate(src_repoupdater_syncer_sync_errors_total%7Bowner!%3D%5C%22user%5C%22%7D%5B5m%5D))%20OR%20on()%20vector(0)%22,%22datasource%22:%22Prometheus%22,%22exemplar%22:true,%22requestId%22:%22Q-6ba88982-7cc8-4c07-9135-56645d70c8ca-0A%22%7D%5D
2021-08-18 09:58:03 +00:00
- <span class="badge badge-warning">warning</span> repo-updater: 0.5+ site level external service sync error rate for 10m0s
- <span class="badge badge-critical">critical</span> repo-updater: 1+ site level external service sync error rate for 10m0s
repo-updater: Instrument code host sync errors. (#16505)
2020-12-07 13:50:33 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
repo-updater: Instrument code host sync errors. (#16505)
2020-12-07 13:50:33 +00:00
monitoring: Only monitor site level external services error rates (#22292) Tweak monitoring to only consider site level external services since we have no control over how user added external services are configured. Also tweak our alert that looks for zero sync for 9 hours so that the rate is also calculated over 9 hours. If not, then external services that only sync once every eight hours would also trigger our alert since our old rate was only calculated over 5 minutes.
2021-06-23 12:15:22 +00:00
- An alert here indicates errors syncing site level repo metadata with code hosts. This indicates that there could be a configuration issue
repo-updater: Instrument code host sync errors. (#16505)
2020-12-07 13:50:33 +00:00
with your code hosts connections or networking issues affecting communication with your code hosts.
- Check the code host status indicator (cloud icon in top right of Sourcegraph homepage) for errors.
- Make sure external services do not have invalid tokens by navigating to them in the web UI and clicking save. If there are no errors, they are valid.
- Check the repo-updater logs for errors about syncing.
- Confirm that outbound network connections are allowed where repo-updater is deployed.
- Check back in an hour to see if the issue has resolved itself.
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#repo-updater-src-repoupdater-syncer-sync-errors-total).
repo-updater: Instrument code host sync errors. (#16505)
2020-12-07 13:50:33 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
monitoring: Update syncer sync errors alerting (#24071) In this commit we: 1. Add a Warning alert if the threshold is greater than 0.5 over 10 minutes. 2. Modify the Critical alert to fire if the threshold is greater than 1 over 10 minutes. Historically, over the last six months approximately, the value of this metric has rarely gone above 0.5 and has recovered almost immediately. Only in one instance has this gone over 1. At the moment we get paged for this alert for what might be intermittent errors on the code host's end which get resolved immediately, reducing the signal to noise ration in our alerting. Link to production dashboard for reference: https://sourcegraph.com/-/debug/grafana/explore?orgId=1&left=%5B%221614537000000%22,%221629484199000%22,%22Prometheus%22,%7B%22expr%22:%22max%20by%20(family)%20(rate(src_repoupdater_syncer_sync_errors_total%7Bowner!%3D%5C%22user%5C%22%7D%5B5m%5D))%20OR%20on()%20vector(0)%22,%22datasource%22:%22Prometheus%22,%22exemplar%22:true,%22requestId%22:%22Q-6ba88982-7cc8-4c07-9135-56645d70c8ca-0A%22%7D%5D
2021-08-18 09:58:03 +00:00
"warning_repo-updater_src_repoupdater_syncer_sync_errors_total",
repo-updater: Instrument code host sync errors. (#16505)
2020-12-07 13:50:33 +00:00
"critical_repo-updater_src_repoupdater_syncer_sync_errors_total"
]
```
Update o11y ownership from IAM/repo-mgmt to Source (#54368) Does what it says in the title. ## Test plan - Ran `sg generate`
2023-06-28 13:49:16 +00:00
<sub>*Managed by the [Sourcegraph Source team](https://handbook.sourcegraph.com/departments/engineering/teams/source).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
monitoring: extract into a submodule (#45786) This change extracts `monitoring` into a submodule for import in `sourcegraph/controller` (https://github.com/sourcegraph/controller/pull/195) so that we can generate dashboards for Cloud instances. These steps were required: 1. Initialize a `go.mod` in `monitoring` 2. Extract `dev/sg/internal/cliutil` into `lib` to avoid illegal imports from `monitoring` 3. Add local replaces to both `sourcegraph/sourcegraph` and `monitoring` 4. `go mod tidy` on all submodules 5. Update `go generate ./monitoring` commands to use `sg`, since the `go generate` command no longer works 6. Update `grafana/build.sh`, `prometheus/build.sh` to build the submodule 7. Amend linters to check for multiple `go.mod` files and ban imports of `github.com/sourcegraph/sourcegraph` 8. Update `sg generate go` to run in directories rather than from root The only caveat is that if you use VS code, you will now need to open `monitoring` in a separate workspace or similar, like with `lib`. Co-authored-by: Joe Chen <joe@sourcegraph.com>
2022-12-19 17:49:25 +00:00
Generated query for warning alert: `max((max by (family) (rate(src_repoupdater_syncer_sync_errors_total{owner!="user",reason!="internal_rate_limit",reason!="invalid_npm_path"}[5m]))) > 0.5)`
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
monitoring: extract into a submodule (#45786) This change extracts `monitoring` into a submodule for import in `sourcegraph/controller` (https://github.com/sourcegraph/controller/pull/195) so that we can generate dashboards for Cloud instances. These steps were required: 1. Initialize a `go.mod` in `monitoring` 2. Extract `dev/sg/internal/cliutil` into `lib` to avoid illegal imports from `monitoring` 3. Add local replaces to both `sourcegraph/sourcegraph` and `monitoring` 4. `go mod tidy` on all submodules 5. Update `go generate ./monitoring` commands to use `sg`, since the `go generate` command no longer works 6. Update `grafana/build.sh`, `prometheus/build.sh` to build the submodule 7. Amend linters to check for multiple `go.mod` files and ban imports of `github.com/sourcegraph/sourcegraph` 8. Update `sg generate go` to run in directories rather than from root The only caveat is that if you use VS code, you will now need to open `monitoring` in a separate workspace or similar, like with `lib`. Co-authored-by: Joe Chen <joe@sourcegraph.com>
2022-12-19 17:49:25 +00:00
Generated query for critical alert: `max((max by (family) (rate(src_repoupdater_syncer_sync_errors_total{owner!="user",reason!="internal_rate_limit",reason!="invalid_npm_path"}[5m]))) > 1)`
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
</details>
repo-updater: Instrument code host sync errors. (#16505)
2020-12-07 13:50:33 +00:00
<br />
monitoring: add more panels to Grafana repo-updater dashboard (#14269) Improves the repo-updater Grafana dashboard with many new panels as well as observing more of repo-updater's internal state. * monitoring: add more panels to Grafana repo-updater dashboard * Remove comment and fix typo * Add dashboard panels for external_service_sync_jobs state * Increase alert threshold * Remove duplicate metrics schedKnownRepos and schedHeapCount were measuring the same thing in different ways and the old version of schedKnownRepos was assuming a single syncer. Remove the old version of schedKnowRepos and use schedHeapCount instead. * Observe the number of repos queued for update * Fix observable name * Observe when a sync was started Grouped by sync "family" and external service id * Improve metric name * Don't tag metrics with external service id This could increase the cardinality of our metrics too much * Observe the time since the oldest sync * Simplify function Co-authored-by: Ryan Slade <ryanslade@gmail.com>
2020-10-08 15:27:12 +00:00
## repo-updater: syncer_sync_start
monitoring: Only monitor site level external services error rates (#22292) Tweak monitoring to only consider site level external services since we have no control over how user added external services are configured. Also tweak our alert that looks for zero sync for 9 hours so that the rate is also calculated over 9 hours. If not, then external services that only sync once every eight hours would also trigger our alert since our old rate was only calculated over 5 minutes.
2021-06-23 12:15:22 +00:00
<p class="subtitle">repo metadata sync was started</p>
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
monitoring: add more panels to Grafana repo-updater dashboard (#14269) Improves the repo-updater Grafana dashboard with many new panels as well as observing more of repo-updater's internal state. * monitoring: add more panels to Grafana repo-updater dashboard * Remove comment and fix typo * Add dashboard panels for external_service_sync_jobs state * Increase alert threshold * Remove duplicate metrics schedKnownRepos and schedHeapCount were measuring the same thing in different ways and the old version of schedKnownRepos was assuming a single syncer. Remove the old version of schedKnowRepos and use schedHeapCount instead. * Observe the number of repos queued for update * Fix observable name * Observe when a sync was started Grouped by sync "family" and external service id * Improve metric name * Don't tag metrics with external service id This could increase the cardinality of our metrics too much * Observe the time since the oldest sync * Simplify function Co-authored-by: Ryan Slade <ryanslade@gmail.com>
2020-10-08 15:27:12 +00:00
monitoring: Only monitor site level external services error rates (#22292) Tweak monitoring to only consider site level external services since we have no control over how user added external services are configured. Also tweak our alert that looks for zero sync for 9 hours so that the rate is also calculated over 9 hours. If not, then external services that only sync once every eight hours would also trigger our alert since our old rate was only calculated over 5 minutes.
2021-06-23 12:15:22 +00:00
- <span class="badge badge-warning">warning</span> repo-updater: less than 0 repo metadata sync was started for 9h0m0s
monitoring: add more panels to Grafana repo-updater dashboard (#14269) Improves the repo-updater Grafana dashboard with many new panels as well as observing more of repo-updater's internal state. * monitoring: add more panels to Grafana repo-updater dashboard * Remove comment and fix typo * Add dashboard panels for external_service_sync_jobs state * Increase alert threshold * Remove duplicate metrics schedKnownRepos and schedHeapCount were measuring the same thing in different ways and the old version of schedKnownRepos was assuming a single syncer. Remove the old version of schedKnowRepos and use schedHeapCount instead. * Observe the number of repos queued for update * Fix observable name * Observe when a sync was started Grouped by sync "family" and external service id * Improve metric name * Don't tag metrics with external service id This could increase the cardinality of our metrics too much * Observe the time since the oldest sync * Simplify function Co-authored-by: Ryan Slade <ryanslade@gmail.com>
2020-10-08 15:27:12 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: add more panels to Grafana repo-updater dashboard (#14269) Improves the repo-updater Grafana dashboard with many new panels as well as observing more of repo-updater's internal state. * monitoring: add more panels to Grafana repo-updater dashboard * Remove comment and fix typo * Add dashboard panels for external_service_sync_jobs state * Increase alert threshold * Remove duplicate metrics schedKnownRepos and schedHeapCount were measuring the same thing in different ways and the old version of schedKnownRepos was assuming a single syncer. Remove the old version of schedKnowRepos and use schedHeapCount instead. * Observe the number of repos queued for update * Fix observable name * Observe when a sync was started Grouped by sync "family" and external service id * Improve metric name * Don't tag metrics with external service id This could increase the cardinality of our metrics too much * Observe the time since the oldest sync * Simplify function Co-authored-by: Ryan Slade <ryanslade@gmail.com>
2020-10-08 15:27:12 +00:00
monitoring: Properly aggregate repo-updater metrics (#14704) * monitoring: Properly aggregate repo-updater metrics Since we only have one instance we should sum the results. * Alert on zero values where appropriate
2020-10-14 13:54:45 +00:00
- Check repo-updater logs for errors.
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#repo-updater-syncer-sync-start).
monitoring: add more panels to Grafana repo-updater dashboard (#14269) Improves the repo-updater Grafana dashboard with many new panels as well as observing more of repo-updater's internal state. * monitoring: add more panels to Grafana repo-updater dashboard * Remove comment and fix typo * Add dashboard panels for external_service_sync_jobs state * Increase alert threshold * Remove duplicate metrics schedKnownRepos and schedHeapCount were measuring the same thing in different ways and the old version of schedKnownRepos was assuming a single syncer. Remove the old version of schedKnowRepos and use schedHeapCount instead. * Observe the number of repos queued for update * Fix observable name * Observe when a sync was started Grouped by sync "family" and external service id * Improve metric name * Don't tag metrics with external service id This could increase the cardinality of our metrics too much * Observe the time since the oldest sync * Simplify function Co-authored-by: Ryan Slade <ryanslade@gmail.com>
2020-10-08 15:27:12 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"warning_repo-updater_syncer_sync_start"
]
```
Update o11y ownership from IAM/repo-mgmt to Source (#54368) Does what it says in the title. ## Test plan - Ran `sg generate`
2023-06-28 13:49:16 +00:00
<sub>*Managed by the [Sourcegraph Source team](https://handbook.sourcegraph.com/departments/engineering/teams/source).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
monitoring: extract into a submodule (#45786) This change extracts `monitoring` into a submodule for import in `sourcegraph/controller` (https://github.com/sourcegraph/controller/pull/195) so that we can generate dashboards for Cloud instances. These steps were required: 1. Initialize a `go.mod` in `monitoring` 2. Extract `dev/sg/internal/cliutil` into `lib` to avoid illegal imports from `monitoring` 3. Add local replaces to both `sourcegraph/sourcegraph` and `monitoring` 4. `go mod tidy` on all submodules 5. Update `go generate ./monitoring` commands to use `sg`, since the `go generate` command no longer works 6. Update `grafana/build.sh`, `prometheus/build.sh` to build the submodule 7. Amend linters to check for multiple `go.mod` files and ban imports of `github.com/sourcegraph/sourcegraph` 8. Update `sg generate go` to run in directories rather than from root The only caveat is that if you use VS code, you will now need to open `monitoring` in a separate workspace or similar, like with `lib`. Co-authored-by: Joe Chen <joe@sourcegraph.com>
2022-12-19 17:49:25 +00:00
Generated query for warning alert: `min((max by (family) (rate(src_repoupdater_syncer_start_sync{family="Syncer.SyncExternalService"}[9h]))) <= 0)`
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
</details>
monitoring: improve solutions formatting, add team information (#14878)
2020-10-20 05:11:20 +00:00
<br />
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: add more panels to Grafana repo-updater dashboard (#14269) Improves the repo-updater Grafana dashboard with many new panels as well as observing more of repo-updater's internal state. * monitoring: add more panels to Grafana repo-updater dashboard * Remove comment and fix typo * Add dashboard panels for external_service_sync_jobs state * Increase alert threshold * Remove duplicate metrics schedKnownRepos and schedHeapCount were measuring the same thing in different ways and the old version of schedKnownRepos was assuming a single syncer. Remove the old version of schedKnowRepos and use schedHeapCount instead. * Observe the number of repos queued for update * Fix observable name * Observe when a sync was started Grouped by sync "family" and external service id * Improve metric name * Don't tag metrics with external service id This could increase the cardinality of our metrics too much * Observe the time since the oldest sync * Simplify function Co-authored-by: Ryan Slade <ryanslade@gmail.com>
2020-10-08 15:27:12 +00:00
## repo-updater: syncer_sync_duration
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
<p class="subtitle">95th repositories sync duration</p>
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
monitoring: add more panels to Grafana repo-updater dashboard (#14269) Improves the repo-updater Grafana dashboard with many new panels as well as observing more of repo-updater's internal state. * monitoring: add more panels to Grafana repo-updater dashboard * Remove comment and fix typo * Add dashboard panels for external_service_sync_jobs state * Increase alert threshold * Remove duplicate metrics schedKnownRepos and schedHeapCount were measuring the same thing in different ways and the old version of schedKnownRepos was assuming a single syncer. Remove the old version of schedKnowRepos and use schedHeapCount instead. * Observe the number of repos queued for update * Fix observable name * Observe when a sync was started Grouped by sync "family" and external service id * Improve metric name * Don't tag metrics with external service id This could increase the cardinality of our metrics too much * Observe the time since the oldest sync * Simplify function Co-authored-by: Ryan Slade <ryanslade@gmail.com>
2020-10-08 15:27:12 +00:00
monitoring: dashboards reference, refresh observability docs (#16939) - New monitoring definition field, 'Interpretation' - New generated dashboards reference - Grafana panels now link to alert solutions and dashboards reference - Refreshed observability docs Co-authored-by: uwedeportivo <534011+uwedeportivo@users.noreply.github.com>
2020-12-23 02:36:41 +00:00
- <span class="badge badge-warning">warning</span> repo-updater: 30s+ 95th repositories sync duration for 5m0s
monitoring: add more panels to Grafana repo-updater dashboard (#14269) Improves the repo-updater Grafana dashboard with many new panels as well as observing more of repo-updater's internal state. * monitoring: add more panels to Grafana repo-updater dashboard * Remove comment and fix typo * Add dashboard panels for external_service_sync_jobs state * Increase alert threshold * Remove duplicate metrics schedKnownRepos and schedHeapCount were measuring the same thing in different ways and the old version of schedKnownRepos was assuming a single syncer. Remove the old version of schedKnowRepos and use schedHeapCount instead. * Observe the number of repos queued for update * Fix observable name * Observe when a sync was started Grouped by sync "family" and external service id * Improve metric name * Don't tag metrics with external service id This could increase the cardinality of our metrics too much * Observe the time since the oldest sync * Simplify function Co-authored-by: Ryan Slade <ryanslade@gmail.com>
2020-10-08 15:27:12 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: add more panels to Grafana repo-updater dashboard (#14269) Improves the repo-updater Grafana dashboard with many new panels as well as observing more of repo-updater's internal state. * monitoring: add more panels to Grafana repo-updater dashboard * Remove comment and fix typo * Add dashboard panels for external_service_sync_jobs state * Increase alert threshold * Remove duplicate metrics schedKnownRepos and schedHeapCount were measuring the same thing in different ways and the old version of schedKnownRepos was assuming a single syncer. Remove the old version of schedKnowRepos and use schedHeapCount instead. * Observe the number of repos queued for update * Fix observable name * Observe when a sync was started Grouped by sync "family" and external service id * Improve metric name * Don't tag metrics with external service id This could increase the cardinality of our metrics too much * Observe the time since the oldest sync * Simplify function Co-authored-by: Ryan Slade <ryanslade@gmail.com>
2020-10-08 15:27:12 +00:00
repo-updater: Add critical alerts (#14530) * repo-updater: Add critical alerts The number of user added repos is greater than 90% of our hard limit of 200k. Number of external services added is great than 20k No external services have synced in more than 8 hours. By default we backoff for up to 8 hours, so anything higher than this indicates a problem. * syncer_sync_start should be a rate Alert when we've performed <= 1 syncs for 8 hours. Ideally we want this to be <= 0 but our monitoring package doesn't allow this. Therefore we need to make this a warning as it'll fire for instances that only have 1 external service defined. * Fill in possible solutions and remove unnecessary panel * Adjust rate * Update monitoring/repo_updater.go Co-authored-by: Tomás Senart <tomas@sourcegraph.com> * Update monitoring/repo_updater.go Co-authored-by: Robert Lin <robert@bobheadxi.dev> * Update possible error descriptions * Fix alert * Switch alert to critical Co-authored-by: Tomás Senart <tomas@sourcegraph.com> Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2020-10-09 13:58:35 +00:00
- Check the network latency is reasonable (<50ms) between the Sourcegraph and the code host
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#repo-updater-syncer-sync-duration).
monitoring: add more panels to Grafana repo-updater dashboard (#14269) Improves the repo-updater Grafana dashboard with many new panels as well as observing more of repo-updater's internal state. * monitoring: add more panels to Grafana repo-updater dashboard * Remove comment and fix typo * Add dashboard panels for external_service_sync_jobs state * Increase alert threshold * Remove duplicate metrics schedKnownRepos and schedHeapCount were measuring the same thing in different ways and the old version of schedKnownRepos was assuming a single syncer. Remove the old version of schedKnowRepos and use schedHeapCount instead. * Observe the number of repos queued for update * Fix observable name * Observe when a sync was started Grouped by sync "family" and external service id * Improve metric name * Don't tag metrics with external service id This could increase the cardinality of our metrics too much * Observe the time since the oldest sync * Simplify function Co-authored-by: Ryan Slade <ryanslade@gmail.com>
2020-10-08 15:27:12 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"warning_repo-updater_syncer_sync_duration"
]
```
Update o11y ownership from IAM/repo-mgmt to Source (#54368) Does what it says in the title. ## Test plan - Ran `sg generate`
2023-06-28 13:49:16 +00:00
<sub>*Managed by the [Sourcegraph Source team](https://handbook.sourcegraph.com/departments/engineering/teams/source).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
monitoring: extract into a submodule (#45786) This change extracts `monitoring` into a submodule for import in `sourcegraph/controller` (https://github.com/sourcegraph/controller/pull/195) so that we can generate dashboards for Cloud instances. These steps were required: 1. Initialize a `go.mod` in `monitoring` 2. Extract `dev/sg/internal/cliutil` into `lib` to avoid illegal imports from `monitoring` 3. Add local replaces to both `sourcegraph/sourcegraph` and `monitoring` 4. `go mod tidy` on all submodules 5. Update `go generate ./monitoring` commands to use `sg`, since the `go generate` command no longer works 6. Update `grafana/build.sh`, `prometheus/build.sh` to build the submodule 7. Amend linters to check for multiple `go.mod` files and ban imports of `github.com/sourcegraph/sourcegraph` 8. Update `sg generate go` to run in directories rather than from root The only caveat is that if you use VS code, you will now need to open `monitoring` in a separate workspace or similar, like with `lib`. Co-authored-by: Joe Chen <joe@sourcegraph.com>
2022-12-19 17:49:25 +00:00
Generated query for warning alert: `max((histogram_quantile(0.95, max by (le, family, success) (rate(src_repoupdater_syncer_sync_duration_seconds_bucket[1m])))) >= 30)`
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
</details>
monitoring: improve solutions formatting, add team information (#14878)
2020-10-20 05:11:20 +00:00
<br />
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: add more panels to Grafana repo-updater dashboard (#14269) Improves the repo-updater Grafana dashboard with many new panels as well as observing more of repo-updater's internal state. * monitoring: add more panels to Grafana repo-updater dashboard * Remove comment and fix typo * Add dashboard panels for external_service_sync_jobs state * Increase alert threshold * Remove duplicate metrics schedKnownRepos and schedHeapCount were measuring the same thing in different ways and the old version of schedKnownRepos was assuming a single syncer. Remove the old version of schedKnowRepos and use schedHeapCount instead. * Observe the number of repos queued for update * Fix observable name * Observe when a sync was started Grouped by sync "family" and external service id * Improve metric name * Don't tag metrics with external service id This could increase the cardinality of our metrics too much * Observe the time since the oldest sync * Simplify function Co-authored-by: Ryan Slade <ryanslade@gmail.com>
2020-10-08 15:27:12 +00:00
## repo-updater: source_duration
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
<p class="subtitle">95th repositories source duration</p>
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
monitoring: add more panels to Grafana repo-updater dashboard (#14269) Improves the repo-updater Grafana dashboard with many new panels as well as observing more of repo-updater's internal state. * monitoring: add more panels to Grafana repo-updater dashboard * Remove comment and fix typo * Add dashboard panels for external_service_sync_jobs state * Increase alert threshold * Remove duplicate metrics schedKnownRepos and schedHeapCount were measuring the same thing in different ways and the old version of schedKnownRepos was assuming a single syncer. Remove the old version of schedKnowRepos and use schedHeapCount instead. * Observe the number of repos queued for update * Fix observable name * Observe when a sync was started Grouped by sync "family" and external service id * Improve metric name * Don't tag metrics with external service id This could increase the cardinality of our metrics too much * Observe the time since the oldest sync * Simplify function Co-authored-by: Ryan Slade <ryanslade@gmail.com>
2020-10-08 15:27:12 +00:00
monitoring: dashboards reference, refresh observability docs (#16939) - New monitoring definition field, 'Interpretation' - New generated dashboards reference - Grafana panels now link to alert solutions and dashboards reference - Refreshed observability docs Co-authored-by: uwedeportivo <534011+uwedeportivo@users.noreply.github.com>
2020-12-23 02:36:41 +00:00
- <span class="badge badge-warning">warning</span> repo-updater: 30s+ 95th repositories source duration for 5m0s
monitoring: add more panels to Grafana repo-updater dashboard (#14269) Improves the repo-updater Grafana dashboard with many new panels as well as observing more of repo-updater's internal state. * monitoring: add more panels to Grafana repo-updater dashboard * Remove comment and fix typo * Add dashboard panels for external_service_sync_jobs state * Increase alert threshold * Remove duplicate metrics schedKnownRepos and schedHeapCount were measuring the same thing in different ways and the old version of schedKnownRepos was assuming a single syncer. Remove the old version of schedKnowRepos and use schedHeapCount instead. * Observe the number of repos queued for update * Fix observable name * Observe when a sync was started Grouped by sync "family" and external service id * Improve metric name * Don't tag metrics with external service id This could increase the cardinality of our metrics too much * Observe the time since the oldest sync * Simplify function Co-authored-by: Ryan Slade <ryanslade@gmail.com>
2020-10-08 15:27:12 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: add more panels to Grafana repo-updater dashboard (#14269) Improves the repo-updater Grafana dashboard with many new panels as well as observing more of repo-updater's internal state. * monitoring: add more panels to Grafana repo-updater dashboard * Remove comment and fix typo * Add dashboard panels for external_service_sync_jobs state * Increase alert threshold * Remove duplicate metrics schedKnownRepos and schedHeapCount were measuring the same thing in different ways and the old version of schedKnownRepos was assuming a single syncer. Remove the old version of schedKnowRepos and use schedHeapCount instead. * Observe the number of repos queued for update * Fix observable name * Observe when a sync was started Grouped by sync "family" and external service id * Improve metric name * Don't tag metrics with external service id This could increase the cardinality of our metrics too much * Observe the time since the oldest sync * Simplify function Co-authored-by: Ryan Slade <ryanslade@gmail.com>
2020-10-08 15:27:12 +00:00
repo-updater: Add critical alerts (#14530) * repo-updater: Add critical alerts The number of user added repos is greater than 90% of our hard limit of 200k. Number of external services added is great than 20k No external services have synced in more than 8 hours. By default we backoff for up to 8 hours, so anything higher than this indicates a problem. * syncer_sync_start should be a rate Alert when we've performed <= 1 syncs for 8 hours. Ideally we want this to be <= 0 but our monitoring package doesn't allow this. Therefore we need to make this a warning as it'll fire for instances that only have 1 external service defined. * Fill in possible solutions and remove unnecessary panel * Adjust rate * Update monitoring/repo_updater.go Co-authored-by: Tomás Senart <tomas@sourcegraph.com> * Update monitoring/repo_updater.go Co-authored-by: Robert Lin <robert@bobheadxi.dev> * Update possible error descriptions * Fix alert * Switch alert to critical Co-authored-by: Tomás Senart <tomas@sourcegraph.com> Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2020-10-09 13:58:35 +00:00
- Check the network latency is reasonable (<50ms) between the Sourcegraph and the code host
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#repo-updater-source-duration).
monitoring: add more panels to Grafana repo-updater dashboard (#14269) Improves the repo-updater Grafana dashboard with many new panels as well as observing more of repo-updater's internal state. * monitoring: add more panels to Grafana repo-updater dashboard * Remove comment and fix typo * Add dashboard panels for external_service_sync_jobs state * Increase alert threshold * Remove duplicate metrics schedKnownRepos and schedHeapCount were measuring the same thing in different ways and the old version of schedKnownRepos was assuming a single syncer. Remove the old version of schedKnowRepos and use schedHeapCount instead. * Observe the number of repos queued for update * Fix observable name * Observe when a sync was started Grouped by sync "family" and external service id * Improve metric name * Don't tag metrics with external service id This could increase the cardinality of our metrics too much * Observe the time since the oldest sync * Simplify function Co-authored-by: Ryan Slade <ryanslade@gmail.com>
2020-10-08 15:27:12 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"warning_repo-updater_source_duration"
]
```
Update o11y ownership from IAM/repo-mgmt to Source (#54368) Does what it says in the title. ## Test plan - Ran `sg generate`
2023-06-28 13:49:16 +00:00
<sub>*Managed by the [Sourcegraph Source team](https://handbook.sourcegraph.com/departments/engineering/teams/source).*</sub>
Revert "monitoring: remove repo-updater: syncer_synced_repos (#24678)" (#25208) This reverts commit c19529bbd64bd48e6e7b2f5bea535b692bcf59f4.
2021-09-22 08:42:50 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
monitoring: extract into a submodule (#45786) This change extracts `monitoring` into a submodule for import in `sourcegraph/controller` (https://github.com/sourcegraph/controller/pull/195) so that we can generate dashboards for Cloud instances. These steps were required: 1. Initialize a `go.mod` in `monitoring` 2. Extract `dev/sg/internal/cliutil` into `lib` to avoid illegal imports from `monitoring` 3. Add local replaces to both `sourcegraph/sourcegraph` and `monitoring` 4. `go mod tidy` on all submodules 5. Update `go generate ./monitoring` commands to use `sg`, since the `go generate` command no longer works 6. Update `grafana/build.sh`, `prometheus/build.sh` to build the submodule 7. Amend linters to check for multiple `go.mod` files and ban imports of `github.com/sourcegraph/sourcegraph` 8. Update `sg generate go` to run in directories rather than from root The only caveat is that if you use VS code, you will now need to open `monitoring` in a separate workspace or similar, like with `lib`. Co-authored-by: Joe Chen <joe@sourcegraph.com>
2022-12-19 17:49:25 +00:00
Generated query for warning alert: `max((histogram_quantile(0.95, max by (le) (rate(src_repoupdater_source_duration_seconds_bucket[1m])))) >= 30)`
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
</details>
Revert "monitoring: remove repo-updater: syncer_synced_repos (#24678)" (#25208) This reverts commit c19529bbd64bd48e6e7b2f5bea535b692bcf59f4.
2021-09-22 08:42:50 +00:00
<br />
## repo-updater: syncer_synced_repos
<p class="subtitle">repositories synced</p>
**Descriptions**
- <span class="badge badge-warning">warning</span> repo-updater: less than 0 repositories synced for 9h0m0s
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
Revert "monitoring: remove repo-updater: syncer_synced_repos (#24678)" (#25208) This reverts commit c19529bbd64bd48e6e7b2f5bea535b692bcf59f4.
2021-09-22 08:42:50 +00:00
- Check network connectivity to code hosts
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#repo-updater-syncer-synced-repos).
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"warning_repo-updater_syncer_synced_repos"
]
```
Update o11y ownership from IAM/repo-mgmt to Source (#54368) Does what it says in the title. ## Test plan - Ran `sg generate`
2023-06-28 13:49:16 +00:00
<sub>*Managed by the [Sourcegraph Source team](https://handbook.sourcegraph.com/departments/engineering/teams/source).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
Generated query for warning alert: `max((max(rate(src_repoupdater_syncer_synced_repos_total[1m]))) <= 0)`
</details>
monitoring: improve solutions formatting, add team information (#14878)
2020-10-20 05:11:20 +00:00
<br />
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: add more panels to Grafana repo-updater dashboard (#14269) Improves the repo-updater Grafana dashboard with many new panels as well as observing more of repo-updater's internal state. * monitoring: add more panels to Grafana repo-updater dashboard * Remove comment and fix typo * Add dashboard panels for external_service_sync_jobs state * Increase alert threshold * Remove duplicate metrics schedKnownRepos and schedHeapCount were measuring the same thing in different ways and the old version of schedKnownRepos was assuming a single syncer. Remove the old version of schedKnowRepos and use schedHeapCount instead. * Observe the number of repos queued for update * Fix observable name * Observe when a sync was started Grouped by sync "family" and external service id * Improve metric name * Don't tag metrics with external service id This could increase the cardinality of our metrics too much * Observe the time since the oldest sync * Simplify function Co-authored-by: Ryan Slade <ryanslade@gmail.com>
2020-10-08 15:27:12 +00:00
## repo-updater: sourced_repos
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
<p class="subtitle">repositories sourced</p>
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
monitoring: add more panels to Grafana repo-updater dashboard (#14269) Improves the repo-updater Grafana dashboard with many new panels as well as observing more of repo-updater's internal state. * monitoring: add more panels to Grafana repo-updater dashboard * Remove comment and fix typo * Add dashboard panels for external_service_sync_jobs state * Increase alert threshold * Remove duplicate metrics schedKnownRepos and schedHeapCount were measuring the same thing in different ways and the old version of schedKnownRepos was assuming a single syncer. Remove the old version of schedKnowRepos and use schedHeapCount instead. * Observe the number of repos queued for update * Fix observable name * Observe when a sync was started Grouped by sync "family" and external service id * Improve metric name * Don't tag metrics with external service id This could increase the cardinality of our metrics too much * Observe the time since the oldest sync * Simplify function Co-authored-by: Ryan Slade <ryanslade@gmail.com>
2020-10-08 15:27:12 +00:00
monitoring: dashboards reference, refresh observability docs (#16939) - New monitoring definition field, 'Interpretation' - New generated dashboards reference - Grafana panels now link to alert solutions and dashboards reference - Refreshed observability docs Co-authored-by: uwedeportivo <534011+uwedeportivo@users.noreply.github.com>
2020-12-23 02:36:41 +00:00
- <span class="badge badge-warning">warning</span> repo-updater: less than 0 repositories sourced for 9h0m0s
monitoring: add more panels to Grafana repo-updater dashboard (#14269) Improves the repo-updater Grafana dashboard with many new panels as well as observing more of repo-updater's internal state. * monitoring: add more panels to Grafana repo-updater dashboard * Remove comment and fix typo * Add dashboard panels for external_service_sync_jobs state * Increase alert threshold * Remove duplicate metrics schedKnownRepos and schedHeapCount were measuring the same thing in different ways and the old version of schedKnownRepos was assuming a single syncer. Remove the old version of schedKnowRepos and use schedHeapCount instead. * Observe the number of repos queued for update * Fix observable name * Observe when a sync was started Grouped by sync "family" and external service id * Improve metric name * Don't tag metrics with external service id This could increase the cardinality of our metrics too much * Observe the time since the oldest sync * Simplify function Co-authored-by: Ryan Slade <ryanslade@gmail.com>
2020-10-08 15:27:12 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: add more panels to Grafana repo-updater dashboard (#14269) Improves the repo-updater Grafana dashboard with many new panels as well as observing more of repo-updater's internal state. * monitoring: add more panels to Grafana repo-updater dashboard * Remove comment and fix typo * Add dashboard panels for external_service_sync_jobs state * Increase alert threshold * Remove duplicate metrics schedKnownRepos and schedHeapCount were measuring the same thing in different ways and the old version of schedKnownRepos was assuming a single syncer. Remove the old version of schedKnowRepos and use schedHeapCount instead. * Observe the number of repos queued for update * Fix observable name * Observe when a sync was started Grouped by sync "family" and external service id * Improve metric name * Don't tag metrics with external service id This could increase the cardinality of our metrics too much * Observe the time since the oldest sync * Simplify function Co-authored-by: Ryan Slade <ryanslade@gmail.com>
2020-10-08 15:27:12 +00:00
repo-updater: Add critical alerts (#14530) * repo-updater: Add critical alerts The number of user added repos is greater than 90% of our hard limit of 200k. Number of external services added is great than 20k No external services have synced in more than 8 hours. By default we backoff for up to 8 hours, so anything higher than this indicates a problem. * syncer_sync_start should be a rate Alert when we've performed <= 1 syncs for 8 hours. Ideally we want this to be <= 0 but our monitoring package doesn't allow this. Therefore we need to make this a warning as it'll fire for instances that only have 1 external service defined. * Fill in possible solutions and remove unnecessary panel * Adjust rate * Update monitoring/repo_updater.go Co-authored-by: Tomás Senart <tomas@sourcegraph.com> * Update monitoring/repo_updater.go Co-authored-by: Robert Lin <robert@bobheadxi.dev> * Update possible error descriptions * Fix alert * Switch alert to critical Co-authored-by: Tomás Senart <tomas@sourcegraph.com> Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2020-10-09 13:58:35 +00:00
- Check network connectivity to code hosts
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#repo-updater-sourced-repos).
monitoring: add more panels to Grafana repo-updater dashboard (#14269) Improves the repo-updater Grafana dashboard with many new panels as well as observing more of repo-updater's internal state. * monitoring: add more panels to Grafana repo-updater dashboard * Remove comment and fix typo * Add dashboard panels for external_service_sync_jobs state * Increase alert threshold * Remove duplicate metrics schedKnownRepos and schedHeapCount were measuring the same thing in different ways and the old version of schedKnownRepos was assuming a single syncer. Remove the old version of schedKnowRepos and use schedHeapCount instead. * Observe the number of repos queued for update * Fix observable name * Observe when a sync was started Grouped by sync "family" and external service id * Improve metric name * Don't tag metrics with external service id This could increase the cardinality of our metrics too much * Observe the time since the oldest sync * Simplify function Co-authored-by: Ryan Slade <ryanslade@gmail.com>
2020-10-08 15:27:12 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"warning_repo-updater_sourced_repos"
]
```
Update o11y ownership from IAM/repo-mgmt to Source (#54368) Does what it says in the title. ## Test plan - Ran `sg generate`
2023-06-28 13:49:16 +00:00
<sub>*Managed by the [Sourcegraph Source team](https://handbook.sourcegraph.com/departments/engineering/teams/source).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
Generated query for warning alert: `min((max(rate(src_repoupdater_source_repos_total[1m]))) <= 0)`
</details>
monitoring: improve solutions formatting, add team information (#14878)
2020-10-20 05:11:20 +00:00
<br />
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
repo-updater: Add critical alerts (#14530) * repo-updater: Add critical alerts The number of user added repos is greater than 90% of our hard limit of 200k. Number of external services added is great than 20k No external services have synced in more than 8 hours. By default we backoff for up to 8 hours, so anything higher than this indicates a problem. * syncer_sync_start should be a rate Alert when we've performed <= 1 syncs for 8 hours. Ideally we want this to be <= 0 but our monitoring package doesn't allow this. Therefore we need to make this a warning as it'll fire for instances that only have 1 external service defined. * Fill in possible solutions and remove unnecessary panel * Adjust rate * Update monitoring/repo_updater.go Co-authored-by: Tomás Senart <tomas@sourcegraph.com> * Update monitoring/repo_updater.go Co-authored-by: Robert Lin <robert@bobheadxi.dev> * Update possible error descriptions * Fix alert * Switch alert to critical Co-authored-by: Tomás Senart <tomas@sourcegraph.com> Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2020-10-09 13:58:35 +00:00
## repo-updater: purge_failed
monitoring: add more panels to Grafana repo-updater dashboard (#14269) Improves the repo-updater Grafana dashboard with many new panels as well as observing more of repo-updater's internal state. * monitoring: add more panels to Grafana repo-updater dashboard * Remove comment and fix typo * Add dashboard panels for external_service_sync_jobs state * Increase alert threshold * Remove duplicate metrics schedKnownRepos and schedHeapCount were measuring the same thing in different ways and the old version of schedKnownRepos was assuming a single syncer. Remove the old version of schedKnowRepos and use schedHeapCount instead. * Observe the number of repos queued for update * Fix observable name * Observe when a sync was started Grouped by sync "family" and external service id * Improve metric name * Don't tag metrics with external service id This could increase the cardinality of our metrics too much * Observe the time since the oldest sync * Simplify function Co-authored-by: Ryan Slade <ryanslade@gmail.com>
2020-10-08 15:27:12 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
<p class="subtitle">repositories purge failed</p>
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
monitoring: add more panels to Grafana repo-updater dashboard (#14269) Improves the repo-updater Grafana dashboard with many new panels as well as observing more of repo-updater's internal state. * monitoring: add more panels to Grafana repo-updater dashboard * Remove comment and fix typo * Add dashboard panels for external_service_sync_jobs state * Increase alert threshold * Remove duplicate metrics schedKnownRepos and schedHeapCount were measuring the same thing in different ways and the old version of schedKnownRepos was assuming a single syncer. Remove the old version of schedKnowRepos and use schedHeapCount instead. * Observe the number of repos queued for update * Fix observable name * Observe when a sync was started Grouped by sync "family" and external service id * Improve metric name * Don't tag metrics with external service id This could increase the cardinality of our metrics too much * Observe the time since the oldest sync * Simplify function Co-authored-by: Ryan Slade <ryanslade@gmail.com>
2020-10-08 15:27:12 +00:00
monitoring: dashboards reference, refresh observability docs (#16939) - New monitoring definition field, 'Interpretation' - New generated dashboards reference - Grafana panels now link to alert solutions and dashboards reference - Refreshed observability docs Co-authored-by: uwedeportivo <534011+uwedeportivo@users.noreply.github.com>
2020-12-23 02:36:41 +00:00
- <span class="badge badge-warning">warning</span> repo-updater: 0+ repositories purge failed for 5m0s
monitoring: add more panels to Grafana repo-updater dashboard (#14269) Improves the repo-updater Grafana dashboard with many new panels as well as observing more of repo-updater's internal state. * monitoring: add more panels to Grafana repo-updater dashboard * Remove comment and fix typo * Add dashboard panels for external_service_sync_jobs state * Increase alert threshold * Remove duplicate metrics schedKnownRepos and schedHeapCount were measuring the same thing in different ways and the old version of schedKnownRepos was assuming a single syncer. Remove the old version of schedKnowRepos and use schedHeapCount instead. * Observe the number of repos queued for update * Fix observable name * Observe when a sync was started Grouped by sync "family" and external service id * Improve metric name * Don't tag metrics with external service id This could increase the cardinality of our metrics too much * Observe the time since the oldest sync * Simplify function Co-authored-by: Ryan Slade <ryanslade@gmail.com>
2020-10-08 15:27:12 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: add more panels to Grafana repo-updater dashboard (#14269) Improves the repo-updater Grafana dashboard with many new panels as well as observing more of repo-updater's internal state. * monitoring: add more panels to Grafana repo-updater dashboard * Remove comment and fix typo * Add dashboard panels for external_service_sync_jobs state * Increase alert threshold * Remove duplicate metrics schedKnownRepos and schedHeapCount were measuring the same thing in different ways and the old version of schedKnownRepos was assuming a single syncer. Remove the old version of schedKnowRepos and use schedHeapCount instead. * Observe the number of repos queued for update * Fix observable name * Observe when a sync was started Grouped by sync "family" and external service id * Improve metric name * Don't tag metrics with external service id This could increase the cardinality of our metrics too much * Observe the time since the oldest sync * Simplify function Co-authored-by: Ryan Slade <ryanslade@gmail.com>
2020-10-08 15:27:12 +00:00
repo-updater: Add critical alerts (#14530) * repo-updater: Add critical alerts The number of user added repos is greater than 90% of our hard limit of 200k. Number of external services added is great than 20k No external services have synced in more than 8 hours. By default we backoff for up to 8 hours, so anything higher than this indicates a problem. * syncer_sync_start should be a rate Alert when we've performed <= 1 syncs for 8 hours. Ideally we want this to be <= 0 but our monitoring package doesn't allow this. Therefore we need to make this a warning as it'll fire for instances that only have 1 external service defined. * Fill in possible solutions and remove unnecessary panel * Adjust rate * Update monitoring/repo_updater.go Co-authored-by: Tomás Senart <tomas@sourcegraph.com> * Update monitoring/repo_updater.go Co-authored-by: Robert Lin <robert@bobheadxi.dev> * Update possible error descriptions * Fix alert * Switch alert to critical Co-authored-by: Tomás Senart <tomas@sourcegraph.com> Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2020-10-09 13:58:35 +00:00
- Check repo-updater`s connectivity with gitserver and gitserver logs
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#repo-updater-purge-failed).
monitoring: add more panels to Grafana repo-updater dashboard (#14269) Improves the repo-updater Grafana dashboard with many new panels as well as observing more of repo-updater's internal state. * monitoring: add more panels to Grafana repo-updater dashboard * Remove comment and fix typo * Add dashboard panels for external_service_sync_jobs state * Increase alert threshold * Remove duplicate metrics schedKnownRepos and schedHeapCount were measuring the same thing in different ways and the old version of schedKnownRepos was assuming a single syncer. Remove the old version of schedKnowRepos and use schedHeapCount instead. * Observe the number of repos queued for update * Fix observable name * Observe when a sync was started Grouped by sync "family" and external service id * Improve metric name * Don't tag metrics with external service id This could increase the cardinality of our metrics too much * Observe the time since the oldest sync * Simplify function Co-authored-by: Ryan Slade <ryanslade@gmail.com>
2020-10-08 15:27:12 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
repo-updater: Add critical alerts (#14530) * repo-updater: Add critical alerts The number of user added repos is greater than 90% of our hard limit of 200k. Number of external services added is great than 20k No external services have synced in more than 8 hours. By default we backoff for up to 8 hours, so anything higher than this indicates a problem. * syncer_sync_start should be a rate Alert when we've performed <= 1 syncs for 8 hours. Ideally we want this to be <= 0 but our monitoring package doesn't allow this. Therefore we need to make this a warning as it'll fire for instances that only have 1 external service defined. * Fill in possible solutions and remove unnecessary panel * Adjust rate * Update monitoring/repo_updater.go Co-authored-by: Tomás Senart <tomas@sourcegraph.com> * Update monitoring/repo_updater.go Co-authored-by: Robert Lin <robert@bobheadxi.dev> * Update possible error descriptions * Fix alert * Switch alert to critical Co-authored-by: Tomás Senart <tomas@sourcegraph.com> Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2020-10-09 13:58:35 +00:00
"warning_repo-updater_purge_failed"
monitoring: add more panels to Grafana repo-updater dashboard (#14269) Improves the repo-updater Grafana dashboard with many new panels as well as observing more of repo-updater's internal state. * monitoring: add more panels to Grafana repo-updater dashboard * Remove comment and fix typo * Add dashboard panels for external_service_sync_jobs state * Increase alert threshold * Remove duplicate metrics schedKnownRepos and schedHeapCount were measuring the same thing in different ways and the old version of schedKnownRepos was assuming a single syncer. Remove the old version of schedKnowRepos and use schedHeapCount instead. * Observe the number of repos queued for update * Fix observable name * Observe when a sync was started Grouped by sync "family" and external service id * Improve metric name * Don't tag metrics with external service id This could increase the cardinality of our metrics too much * Observe the time since the oldest sync * Simplify function Co-authored-by: Ryan Slade <ryanslade@gmail.com>
2020-10-08 15:27:12 +00:00
]
```
Update o11y ownership from IAM/repo-mgmt to Source (#54368) Does what it says in the title. ## Test plan - Ran `sg generate`
2023-06-28 13:49:16 +00:00
<sub>*Managed by the [Sourcegraph Source team](https://handbook.sourcegraph.com/departments/engineering/teams/source).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
Generated query for warning alert: `max((max(rate(src_repoupdater_purge_failed[1m]))) > 0)`
</details>
monitoring: improve solutions formatting, add team information (#14878)
2020-10-20 05:11:20 +00:00
<br />
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: add more panels to Grafana repo-updater dashboard (#14269) Improves the repo-updater Grafana dashboard with many new panels as well as observing more of repo-updater's internal state. * monitoring: add more panels to Grafana repo-updater dashboard * Remove comment and fix typo * Add dashboard panels for external_service_sync_jobs state * Increase alert threshold * Remove duplicate metrics schedKnownRepos and schedHeapCount were measuring the same thing in different ways and the old version of schedKnownRepos was assuming a single syncer. Remove the old version of schedKnowRepos and use schedHeapCount instead. * Observe the number of repos queued for update * Fix observable name * Observe when a sync was started Grouped by sync "family" and external service id * Improve metric name * Don't tag metrics with external service id This could increase the cardinality of our metrics too much * Observe the time since the oldest sync * Simplify function Co-authored-by: Ryan Slade <ryanslade@gmail.com>
2020-10-08 15:27:12 +00:00
## repo-updater: sched_auto_fetch
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
<p class="subtitle">repositories scheduled due to hitting a deadline</p>
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
monitoring: add more panels to Grafana repo-updater dashboard (#14269) Improves the repo-updater Grafana dashboard with many new panels as well as observing more of repo-updater's internal state. * monitoring: add more panels to Grafana repo-updater dashboard * Remove comment and fix typo * Add dashboard panels for external_service_sync_jobs state * Increase alert threshold * Remove duplicate metrics schedKnownRepos and schedHeapCount were measuring the same thing in different ways and the old version of schedKnownRepos was assuming a single syncer. Remove the old version of schedKnowRepos and use schedHeapCount instead. * Observe the number of repos queued for update * Fix observable name * Observe when a sync was started Grouped by sync "family" and external service id * Improve metric name * Don't tag metrics with external service id This could increase the cardinality of our metrics too much * Observe the time since the oldest sync * Simplify function Co-authored-by: Ryan Slade <ryanslade@gmail.com>
2020-10-08 15:27:12 +00:00
monitoring: dashboards reference, refresh observability docs (#16939) - New monitoring definition field, 'Interpretation' - New generated dashboards reference - Grafana panels now link to alert solutions and dashboards reference - Refreshed observability docs Co-authored-by: uwedeportivo <534011+uwedeportivo@users.noreply.github.com>
2020-12-23 02:36:41 +00:00
- <span class="badge badge-warning">warning</span> repo-updater: less than 0 repositories scheduled due to hitting a deadline for 9h0m0s
monitoring: add more panels to Grafana repo-updater dashboard (#14269) Improves the repo-updater Grafana dashboard with many new panels as well as observing more of repo-updater's internal state. * monitoring: add more panels to Grafana repo-updater dashboard * Remove comment and fix typo * Add dashboard panels for external_service_sync_jobs state * Increase alert threshold * Remove duplicate metrics schedKnownRepos and schedHeapCount were measuring the same thing in different ways and the old version of schedKnownRepos was assuming a single syncer. Remove the old version of schedKnowRepos and use schedHeapCount instead. * Observe the number of repos queued for update * Fix observable name * Observe when a sync was started Grouped by sync "family" and external service id * Improve metric name * Don't tag metrics with external service id This could increase the cardinality of our metrics too much * Observe the time since the oldest sync * Simplify function Co-authored-by: Ryan Slade <ryanslade@gmail.com>
2020-10-08 15:27:12 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: add more panels to Grafana repo-updater dashboard (#14269) Improves the repo-updater Grafana dashboard with many new panels as well as observing more of repo-updater's internal state. * monitoring: add more panels to Grafana repo-updater dashboard * Remove comment and fix typo * Add dashboard panels for external_service_sync_jobs state * Increase alert threshold * Remove duplicate metrics schedKnownRepos and schedHeapCount were measuring the same thing in different ways and the old version of schedKnownRepos was assuming a single syncer. Remove the old version of schedKnowRepos and use schedHeapCount instead. * Observe the number of repos queued for update * Fix observable name * Observe when a sync was started Grouped by sync "family" and external service id * Improve metric name * Don't tag metrics with external service id This could increase the cardinality of our metrics too much * Observe the time since the oldest sync * Simplify function Co-authored-by: Ryan Slade <ryanslade@gmail.com>
2020-10-08 15:27:12 +00:00
repos: Remove more code associated with user / org added code hosts (#43187) Remove more code associated with user or org added code hosts. This includes a bunch of test cases and metrics too.
2022-10-19 15:42:14 +00:00
- Check repo-updater logs.
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#repo-updater-sched-auto-fetch).
monitoring: add more panels to Grafana repo-updater dashboard (#14269) Improves the repo-updater Grafana dashboard with many new panels as well as observing more of repo-updater's internal state. * monitoring: add more panels to Grafana repo-updater dashboard * Remove comment and fix typo * Add dashboard panels for external_service_sync_jobs state * Increase alert threshold * Remove duplicate metrics schedKnownRepos and schedHeapCount were measuring the same thing in different ways and the old version of schedKnownRepos was assuming a single syncer. Remove the old version of schedKnowRepos and use schedHeapCount instead. * Observe the number of repos queued for update * Fix observable name * Observe when a sync was started Grouped by sync "family" and external service id * Improve metric name * Don't tag metrics with external service id This could increase the cardinality of our metrics too much * Observe the time since the oldest sync * Simplify function Co-authored-by: Ryan Slade <ryanslade@gmail.com>
2020-10-08 15:27:12 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"warning_repo-updater_sched_auto_fetch"
]
```
Update o11y ownership from IAM/repo-mgmt to Source (#54368) Does what it says in the title. ## Test plan - Ran `sg generate`
2023-06-28 13:49:16 +00:00
<sub>*Managed by the [Sourcegraph Source team](https://handbook.sourcegraph.com/departments/engineering/teams/source).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
Generated query for warning alert: `min((max(rate(src_repoupdater_sched_auto_fetch[1m]))) <= 0)`
</details>
monitoring: improve solutions formatting, add team information (#14878)
2020-10-20 05:11:20 +00:00
<br />
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: add more panels to Grafana repo-updater dashboard (#14269) Improves the repo-updater Grafana dashboard with many new panels as well as observing more of repo-updater's internal state. * monitoring: add more panels to Grafana repo-updater dashboard * Remove comment and fix typo * Add dashboard panels for external_service_sync_jobs state * Increase alert threshold * Remove duplicate metrics schedKnownRepos and schedHeapCount were measuring the same thing in different ways and the old version of schedKnownRepos was assuming a single syncer. Remove the old version of schedKnowRepos and use schedHeapCount instead. * Observe the number of repos queued for update * Fix observable name * Observe when a sync was started Grouped by sync "family" and external service id * Improve metric name * Don't tag metrics with external service id This could increase the cardinality of our metrics too much * Observe the time since the oldest sync * Simplify function Co-authored-by: Ryan Slade <ryanslade@gmail.com>
2020-10-08 15:27:12 +00:00
## repo-updater: sched_known_repos
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
<p class="subtitle">repositories managed by the scheduler</p>
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
monitoring: add more panels to Grafana repo-updater dashboard (#14269) Improves the repo-updater Grafana dashboard with many new panels as well as observing more of repo-updater's internal state. * monitoring: add more panels to Grafana repo-updater dashboard * Remove comment and fix typo * Add dashboard panels for external_service_sync_jobs state * Increase alert threshold * Remove duplicate metrics schedKnownRepos and schedHeapCount were measuring the same thing in different ways and the old version of schedKnownRepos was assuming a single syncer. Remove the old version of schedKnowRepos and use schedHeapCount instead. * Observe the number of repos queued for update * Fix observable name * Observe when a sync was started Grouped by sync "family" and external service id * Improve metric name * Don't tag metrics with external service id This could increase the cardinality of our metrics too much * Observe the time since the oldest sync * Simplify function Co-authored-by: Ryan Slade <ryanslade@gmail.com>
2020-10-08 15:27:12 +00:00
monitoring: dashboards reference, refresh observability docs (#16939) - New monitoring definition field, 'Interpretation' - New generated dashboards reference - Grafana panels now link to alert solutions and dashboards reference - Refreshed observability docs Co-authored-by: uwedeportivo <534011+uwedeportivo@users.noreply.github.com>
2020-12-23 02:36:41 +00:00
- <span class="badge badge-warning">warning</span> repo-updater: less than 0 repositories managed by the scheduler for 10m0s
monitoring: add more panels to Grafana repo-updater dashboard (#14269) Improves the repo-updater Grafana dashboard with many new panels as well as observing more of repo-updater's internal state. * monitoring: add more panels to Grafana repo-updater dashboard * Remove comment and fix typo * Add dashboard panels for external_service_sync_jobs state * Increase alert threshold * Remove duplicate metrics schedKnownRepos and schedHeapCount were measuring the same thing in different ways and the old version of schedKnownRepos was assuming a single syncer. Remove the old version of schedKnowRepos and use schedHeapCount instead. * Observe the number of repos queued for update * Fix observable name * Observe when a sync was started Grouped by sync "family" and external service id * Improve metric name * Don't tag metrics with external service id This could increase the cardinality of our metrics too much * Observe the time since the oldest sync * Simplify function Co-authored-by: Ryan Slade <ryanslade@gmail.com>
2020-10-08 15:27:12 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: add more panels to Grafana repo-updater dashboard (#14269) Improves the repo-updater Grafana dashboard with many new panels as well as observing more of repo-updater's internal state. * monitoring: add more panels to Grafana repo-updater dashboard * Remove comment and fix typo * Add dashboard panels for external_service_sync_jobs state * Increase alert threshold * Remove duplicate metrics schedKnownRepos and schedHeapCount were measuring the same thing in different ways and the old version of schedKnownRepos was assuming a single syncer. Remove the old version of schedKnowRepos and use schedHeapCount instead. * Observe the number of repos queued for update * Fix observable name * Observe when a sync was started Grouped by sync "family" and external service id * Improve metric name * Don't tag metrics with external service id This could increase the cardinality of our metrics too much * Observe the time since the oldest sync * Simplify function Co-authored-by: Ryan Slade <ryanslade@gmail.com>
2020-10-08 15:27:12 +00:00
repo-updater: Add critical alerts (#14530) * repo-updater: Add critical alerts The number of user added repos is greater than 90% of our hard limit of 200k. Number of external services added is great than 20k No external services have synced in more than 8 hours. By default we backoff for up to 8 hours, so anything higher than this indicates a problem. * syncer_sync_start should be a rate Alert when we've performed <= 1 syncs for 8 hours. Ideally we want this to be <= 0 but our monitoring package doesn't allow this. Therefore we need to make this a warning as it'll fire for instances that only have 1 external service defined. * Fill in possible solutions and remove unnecessary panel * Adjust rate * Update monitoring/repo_updater.go Co-authored-by: Tomás Senart <tomas@sourcegraph.com> * Update monitoring/repo_updater.go Co-authored-by: Robert Lin <robert@bobheadxi.dev> * Update possible error descriptions * Fix alert * Switch alert to critical Co-authored-by: Tomás Senart <tomas@sourcegraph.com> Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2020-10-09 13:58:35 +00:00
- Check repo-updater logs. This is expected to fire if there are no user added code hosts
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#repo-updater-sched-known-repos).
monitoring: add more panels to Grafana repo-updater dashboard (#14269) Improves the repo-updater Grafana dashboard with many new panels as well as observing more of repo-updater's internal state. * monitoring: add more panels to Grafana repo-updater dashboard * Remove comment and fix typo * Add dashboard panels for external_service_sync_jobs state * Increase alert threshold * Remove duplicate metrics schedKnownRepos and schedHeapCount were measuring the same thing in different ways and the old version of schedKnownRepos was assuming a single syncer. Remove the old version of schedKnowRepos and use schedHeapCount instead. * Observe the number of repos queued for update * Fix observable name * Observe when a sync was started Grouped by sync "family" and external service id * Improve metric name * Don't tag metrics with external service id This could increase the cardinality of our metrics too much * Observe the time since the oldest sync * Simplify function Co-authored-by: Ryan Slade <ryanslade@gmail.com>
2020-10-08 15:27:12 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"warning_repo-updater_sched_known_repos"
]
```
Update o11y ownership from IAM/repo-mgmt to Source (#54368) Does what it says in the title. ## Test plan - Ran `sg generate`
2023-06-28 13:49:16 +00:00
<sub>*Managed by the [Sourcegraph Source team](https://handbook.sourcegraph.com/departments/engineering/teams/source).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
Generated query for warning alert: `min((max(src_repoupdater_sched_known_repos)) <= 0)`
</details>
monitoring: improve solutions formatting, add team information (#14878)
2020-10-20 05:11:20 +00:00
<br />
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: add more panels to Grafana repo-updater dashboard (#14269) Improves the repo-updater Grafana dashboard with many new panels as well as observing more of repo-updater's internal state. * monitoring: add more panels to Grafana repo-updater dashboard * Remove comment and fix typo * Add dashboard panels for external_service_sync_jobs state * Increase alert threshold * Remove duplicate metrics schedKnownRepos and schedHeapCount were measuring the same thing in different ways and the old version of schedKnownRepos was assuming a single syncer. Remove the old version of schedKnowRepos and use schedHeapCount instead. * Observe the number of repos queued for update * Fix observable name * Observe when a sync was started Grouped by sync "family" and external service id * Improve metric name * Don't tag metrics with external service id This could increase the cardinality of our metrics too much * Observe the time since the oldest sync * Simplify function Co-authored-by: Ryan Slade <ryanslade@gmail.com>
2020-10-08 15:27:12 +00:00
## repo-updater: sched_update_queue_length
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
<p class="subtitle">rate of growth of update queue length over 5 minutes</p>
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
monitoring: add more panels to Grafana repo-updater dashboard (#14269) Improves the repo-updater Grafana dashboard with many new panels as well as observing more of repo-updater's internal state. * monitoring: add more panels to Grafana repo-updater dashboard * Remove comment and fix typo * Add dashboard panels for external_service_sync_jobs state * Increase alert threshold * Remove duplicate metrics schedKnownRepos and schedHeapCount were measuring the same thing in different ways and the old version of schedKnownRepos was assuming a single syncer. Remove the old version of schedKnowRepos and use schedHeapCount instead. * Observe the number of repos queued for update * Fix observable name * Observe when a sync was started Grouped by sync "family" and external service id * Improve metric name * Don't tag metrics with external service id This could increase the cardinality of our metrics too much * Observe the time since the oldest sync * Simplify function Co-authored-by: Ryan Slade <ryanslade@gmail.com>
2020-10-08 15:27:12 +00:00
monitoring: Increase alert time range (#17269) Increase the time range for the src_repoupdater_sched_update_queue_length metric from 30 minutes to 2 hours. This should stop false positives due to expected spikes in the update queue length when updates are queued due to age.
2021-01-14 13:26:08 +00:00
- <span class="badge badge-critical">critical</span> repo-updater: 0+ rate of growth of update queue length over 5 minutes for 2h0m0s
monitoring: add more panels to Grafana repo-updater dashboard (#14269) Improves the repo-updater Grafana dashboard with many new panels as well as observing more of repo-updater's internal state. * monitoring: add more panels to Grafana repo-updater dashboard * Remove comment and fix typo * Add dashboard panels for external_service_sync_jobs state * Increase alert threshold * Remove duplicate metrics schedKnownRepos and schedHeapCount were measuring the same thing in different ways and the old version of schedKnownRepos was assuming a single syncer. Remove the old version of schedKnowRepos and use schedHeapCount instead. * Observe the number of repos queued for update * Fix observable name * Observe when a sync was started Grouped by sync "family" and external service id * Improve metric name * Don't tag metrics with external service id This could increase the cardinality of our metrics too much * Observe the time since the oldest sync * Simplify function Co-authored-by: Ryan Slade <ryanslade@gmail.com>
2020-10-08 15:27:12 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: add more panels to Grafana repo-updater dashboard (#14269) Improves the repo-updater Grafana dashboard with many new panels as well as observing more of repo-updater's internal state. * monitoring: add more panels to Grafana repo-updater dashboard * Remove comment and fix typo * Add dashboard panels for external_service_sync_jobs state * Increase alert threshold * Remove duplicate metrics schedKnownRepos and schedHeapCount were measuring the same thing in different ways and the old version of schedKnownRepos was assuming a single syncer. Remove the old version of schedKnowRepos and use schedHeapCount instead. * Observe the number of repos queued for update * Fix observable name * Observe when a sync was started Grouped by sync "family" and external service id * Improve metric name * Don't tag metrics with external service id This could increase the cardinality of our metrics too much * Observe the time since the oldest sync * Simplify function Co-authored-by: Ryan Slade <ryanslade@gmail.com>
2020-10-08 15:27:12 +00:00
monitoring: Amend repo-updater critical alerts (#14757) Increase alert threshold to 9 hours instead of 8 to give the syncer some breathing room. Alert on the rate of change of `sched_update_queue_length` Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2020-10-15 12:02:06 +00:00
- Check repo-updater logs for indications that the queue is not being processed. The queue length should trend downwards over time as items are sent to GitServer
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#repo-updater-sched-update-queue-length).
monitoring: add more panels to Grafana repo-updater dashboard (#14269) Improves the repo-updater Grafana dashboard with many new panels as well as observing more of repo-updater's internal state. * monitoring: add more panels to Grafana repo-updater dashboard * Remove comment and fix typo * Add dashboard panels for external_service_sync_jobs state * Increase alert threshold * Remove duplicate metrics schedKnownRepos and schedHeapCount were measuring the same thing in different ways and the old version of schedKnownRepos was assuming a single syncer. Remove the old version of schedKnowRepos and use schedHeapCount instead. * Observe the number of repos queued for update * Fix observable name * Observe when a sync was started Grouped by sync "family" and external service id * Improve metric name * Don't tag metrics with external service id This could increase the cardinality of our metrics too much * Observe the time since the oldest sync * Simplify function Co-authored-by: Ryan Slade <ryanslade@gmail.com>
2020-10-08 15:27:12 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
repo-updater: Add critical alerts (#14530) * repo-updater: Add critical alerts The number of user added repos is greater than 90% of our hard limit of 200k. Number of external services added is great than 20k No external services have synced in more than 8 hours. By default we backoff for up to 8 hours, so anything higher than this indicates a problem. * syncer_sync_start should be a rate Alert when we've performed <= 1 syncs for 8 hours. Ideally we want this to be <= 0 but our monitoring package doesn't allow this. Therefore we need to make this a warning as it'll fire for instances that only have 1 external service defined. * Fill in possible solutions and remove unnecessary panel * Adjust rate * Update monitoring/repo_updater.go Co-authored-by: Tomás Senart <tomas@sourcegraph.com> * Update monitoring/repo_updater.go Co-authored-by: Robert Lin <robert@bobheadxi.dev> * Update possible error descriptions * Fix alert * Switch alert to critical Co-authored-by: Tomás Senart <tomas@sourcegraph.com> Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2020-10-09 13:58:35 +00:00
"critical_repo-updater_sched_update_queue_length"
monitoring: add more panels to Grafana repo-updater dashboard (#14269) Improves the repo-updater Grafana dashboard with many new panels as well as observing more of repo-updater's internal state. * monitoring: add more panels to Grafana repo-updater dashboard * Remove comment and fix typo * Add dashboard panels for external_service_sync_jobs state * Increase alert threshold * Remove duplicate metrics schedKnownRepos and schedHeapCount were measuring the same thing in different ways and the old version of schedKnownRepos was assuming a single syncer. Remove the old version of schedKnowRepos and use schedHeapCount instead. * Observe the number of repos queued for update * Fix observable name * Observe when a sync was started Grouped by sync "family" and external service id * Improve metric name * Don't tag metrics with external service id This could increase the cardinality of our metrics too much * Observe the time since the oldest sync * Simplify function Co-authored-by: Ryan Slade <ryanslade@gmail.com>
2020-10-08 15:27:12 +00:00
]
```
Update o11y ownership from IAM/repo-mgmt to Source (#54368) Does what it says in the title. ## Test plan - Ran `sg generate`
2023-06-28 13:49:16 +00:00
<sub>*Managed by the [Sourcegraph Source team](https://handbook.sourcegraph.com/departments/engineering/teams/source).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
Generated query for critical alert: `max((max(deriv(src_repoupdater_sched_update_queue_length[5m]))) > 0)`
</details>
monitoring: improve solutions formatting, add team information (#14878)
2020-10-20 05:11:20 +00:00
<br />
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: add more panels to Grafana repo-updater dashboard (#14269) Improves the repo-updater Grafana dashboard with many new panels as well as observing more of repo-updater's internal state. * monitoring: add more panels to Grafana repo-updater dashboard * Remove comment and fix typo * Add dashboard panels for external_service_sync_jobs state * Increase alert threshold * Remove duplicate metrics schedKnownRepos and schedHeapCount were measuring the same thing in different ways and the old version of schedKnownRepos was assuming a single syncer. Remove the old version of schedKnowRepos and use schedHeapCount instead. * Observe the number of repos queued for update * Fix observable name * Observe when a sync was started Grouped by sync "family" and external service id * Improve metric name * Don't tag metrics with external service id This could increase the cardinality of our metrics too much * Observe the time since the oldest sync * Simplify function Co-authored-by: Ryan Slade <ryanslade@gmail.com>
2020-10-08 15:27:12 +00:00
## repo-updater: sched_loops
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
<p class="subtitle">scheduler loops</p>
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
monitoring: add more panels to Grafana repo-updater dashboard (#14269) Improves the repo-updater Grafana dashboard with many new panels as well as observing more of repo-updater's internal state. * monitoring: add more panels to Grafana repo-updater dashboard * Remove comment and fix typo * Add dashboard panels for external_service_sync_jobs state * Increase alert threshold * Remove duplicate metrics schedKnownRepos and schedHeapCount were measuring the same thing in different ways and the old version of schedKnownRepos was assuming a single syncer. Remove the old version of schedKnowRepos and use schedHeapCount instead. * Observe the number of repos queued for update * Fix observable name * Observe when a sync was started Grouped by sync "family" and external service id * Improve metric name * Don't tag metrics with external service id This could increase the cardinality of our metrics too much * Observe the time since the oldest sync * Simplify function Co-authored-by: Ryan Slade <ryanslade@gmail.com>
2020-10-08 15:27:12 +00:00
monitoring: dashboards reference, refresh observability docs (#16939) - New monitoring definition field, 'Interpretation' - New generated dashboards reference - Grafana panels now link to alert solutions and dashboards reference - Refreshed observability docs Co-authored-by: uwedeportivo <534011+uwedeportivo@users.noreply.github.com>
2020-12-23 02:36:41 +00:00
- <span class="badge badge-warning">warning</span> repo-updater: less than 0 scheduler loops for 9h0m0s
monitoring: add more panels to Grafana repo-updater dashboard (#14269) Improves the repo-updater Grafana dashboard with many new panels as well as observing more of repo-updater's internal state. * monitoring: add more panels to Grafana repo-updater dashboard * Remove comment and fix typo * Add dashboard panels for external_service_sync_jobs state * Increase alert threshold * Remove duplicate metrics schedKnownRepos and schedHeapCount were measuring the same thing in different ways and the old version of schedKnownRepos was assuming a single syncer. Remove the old version of schedKnowRepos and use schedHeapCount instead. * Observe the number of repos queued for update * Fix observable name * Observe when a sync was started Grouped by sync "family" and external service id * Improve metric name * Don't tag metrics with external service id This could increase the cardinality of our metrics too much * Observe the time since the oldest sync * Simplify function Co-authored-by: Ryan Slade <ryanslade@gmail.com>
2020-10-08 15:27:12 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: add more panels to Grafana repo-updater dashboard (#14269) Improves the repo-updater Grafana dashboard with many new panels as well as observing more of repo-updater's internal state. * monitoring: add more panels to Grafana repo-updater dashboard * Remove comment and fix typo * Add dashboard panels for external_service_sync_jobs state * Increase alert threshold * Remove duplicate metrics schedKnownRepos and schedHeapCount were measuring the same thing in different ways and the old version of schedKnownRepos was assuming a single syncer. Remove the old version of schedKnowRepos and use schedHeapCount instead. * Observe the number of repos queued for update * Fix observable name * Observe when a sync was started Grouped by sync "family" and external service id * Improve metric name * Don't tag metrics with external service id This could increase the cardinality of our metrics too much * Observe the time since the oldest sync * Simplify function Co-authored-by: Ryan Slade <ryanslade@gmail.com>
2020-10-08 15:27:12 +00:00
repo-updater: Add critical alerts (#14530) * repo-updater: Add critical alerts The number of user added repos is greater than 90% of our hard limit of 200k. Number of external services added is great than 20k No external services have synced in more than 8 hours. By default we backoff for up to 8 hours, so anything higher than this indicates a problem. * syncer_sync_start should be a rate Alert when we've performed <= 1 syncs for 8 hours. Ideally we want this to be <= 0 but our monitoring package doesn't allow this. Therefore we need to make this a warning as it'll fire for instances that only have 1 external service defined. * Fill in possible solutions and remove unnecessary panel * Adjust rate * Update monitoring/repo_updater.go Co-authored-by: Tomás Senart <tomas@sourcegraph.com> * Update monitoring/repo_updater.go Co-authored-by: Robert Lin <robert@bobheadxi.dev> * Update possible error descriptions * Fix alert * Switch alert to critical Co-authored-by: Tomás Senart <tomas@sourcegraph.com> Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2020-10-09 13:58:35 +00:00
- Check repo-updater logs for errors. This is expected to fire if there are no user added code hosts
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#repo-updater-sched-loops).
monitoring: add more panels to Grafana repo-updater dashboard (#14269) Improves the repo-updater Grafana dashboard with many new panels as well as observing more of repo-updater's internal state. * monitoring: add more panels to Grafana repo-updater dashboard * Remove comment and fix typo * Add dashboard panels for external_service_sync_jobs state * Increase alert threshold * Remove duplicate metrics schedKnownRepos and schedHeapCount were measuring the same thing in different ways and the old version of schedKnownRepos was assuming a single syncer. Remove the old version of schedKnowRepos and use schedHeapCount instead. * Observe the number of repos queued for update * Fix observable name * Observe when a sync was started Grouped by sync "family" and external service id * Improve metric name * Don't tag metrics with external service id This could increase the cardinality of our metrics too much * Observe the time since the oldest sync * Simplify function Co-authored-by: Ryan Slade <ryanslade@gmail.com>
2020-10-08 15:27:12 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"warning_repo-updater_sched_loops"
]
```
Update o11y ownership from IAM/repo-mgmt to Source (#54368) Does what it says in the title. ## Test plan - Ran `sg generate`
2023-06-28 13:49:16 +00:00
<sub>*Managed by the [Sourcegraph Source team](https://handbook.sourcegraph.com/departments/engineering/teams/source).*</sub>
monitoring: Add an warning alert when repos haven't synced for 8 hours (#32838) Add a warning alert when we see repos that have not updated successfully in at least 8 hours. We always return zero for now on Sourcegraph.com because we have a lot of repos in this state for the Starburst service.
2022-03-22 10:32:04 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
Generated query for warning alert: `min((max(rate(src_repoupdater_sched_loops[1m]))) <= 0)`
</details>
monitoring: Add an warning alert when repos haven't synced for 8 hours (#32838) Add a warning alert when we see repos that have not updated successfully in at least 8 hours. We always return zero for now on Sourcegraph.com because we have a lot of repos in this state for the Starburst service.
2022-03-22 10:32:04 +00:00
<br />
## repo-updater: src_repoupdater_stale_repos
<p class="subtitle">repos that haven't been fetched in more than 8 hours</p>
**Descriptions**
- <span class="badge badge-warning">warning</span> repo-updater: 1+ repos that haven't been fetched in more than 8 hours for 25m0s
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: Add an warning alert when repos haven't synced for 8 hours (#32838) Add a warning alert when we see repos that have not updated successfully in at least 8 hours. We always return zero for now on Sourcegraph.com because we have a lot of repos in this state for the Starburst service.
2022-03-22 10:32:04 +00:00
- Check repo-updater logs for errors.
Check for rows in gitserver_repos where LastError is not an empty string.
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#repo-updater-src-repoupdater-stale-repos).
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"warning_repo-updater_src_repoupdater_stale_repos"
]
```
Update o11y ownership from IAM/repo-mgmt to Source (#54368) Does what it says in the title. ## Test plan - Ran `sg generate`
2023-06-28 13:49:16 +00:00
<sub>*Managed by the [Sourcegraph Source team](https://handbook.sourcegraph.com/departments/engineering/teams/source).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
Generated query for warning alert: `max((max(src_repoupdater_stale_repos)) >= 1)`
</details>
monitoring: improve solutions formatting, add team information (#14878)
2020-10-20 05:11:20 +00:00
<br />
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: add more panels to Grafana repo-updater dashboard (#14269) Improves the repo-updater Grafana dashboard with many new panels as well as observing more of repo-updater's internal state. * monitoring: add more panels to Grafana repo-updater dashboard * Remove comment and fix typo * Add dashboard panels for external_service_sync_jobs state * Increase alert threshold * Remove duplicate metrics schedKnownRepos and schedHeapCount were measuring the same thing in different ways and the old version of schedKnownRepos was assuming a single syncer. Remove the old version of schedKnowRepos and use schedHeapCount instead. * Observe the number of repos queued for update * Fix observable name * Observe when a sync was started Grouped by sync "family" and external service id * Improve metric name * Don't tag metrics with external service id This could increase the cardinality of our metrics too much * Observe the time since the oldest sync * Simplify function Co-authored-by: Ryan Slade <ryanslade@gmail.com>
2020-10-08 15:27:12 +00:00
## repo-updater: sched_error
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
<p class="subtitle">repositories schedule error rate</p>
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
monitoring: add more panels to Grafana repo-updater dashboard (#14269) Improves the repo-updater Grafana dashboard with many new panels as well as observing more of repo-updater's internal state. * monitoring: add more panels to Grafana repo-updater dashboard * Remove comment and fix typo * Add dashboard panels for external_service_sync_jobs state * Increase alert threshold * Remove duplicate metrics schedKnownRepos and schedHeapCount were measuring the same thing in different ways and the old version of schedKnownRepos was assuming a single syncer. Remove the old version of schedKnowRepos and use schedHeapCount instead. * Observe the number of repos queued for update * Fix observable name * Observe when a sync was started Grouped by sync "family" and external service id * Improve metric name * Don't tag metrics with external service id This could increase the cardinality of our metrics too much * Observe the time since the oldest sync * Simplify function Co-authored-by: Ryan Slade <ryanslade@gmail.com>
2020-10-08 15:27:12 +00:00
monitoring: Increase alert timing threshold (#30682) This metric spikes during gitserver deployments which can take longer than 15 minutes so we want to avoid this alert going off during every deployment.
2022-02-04 14:21:19 +00:00
- <span class="badge badge-critical">critical</span> repo-updater: 1+ repositories schedule error rate for 25m0s
monitoring: add more panels to Grafana repo-updater dashboard (#14269) Improves the repo-updater Grafana dashboard with many new panels as well as observing more of repo-updater's internal state. * monitoring: add more panels to Grafana repo-updater dashboard * Remove comment and fix typo * Add dashboard panels for external_service_sync_jobs state * Increase alert threshold * Remove duplicate metrics schedKnownRepos and schedHeapCount were measuring the same thing in different ways and the old version of schedKnownRepos was assuming a single syncer. Remove the old version of schedKnowRepos and use schedHeapCount instead. * Observe the number of repos queued for update * Fix observable name * Observe when a sync was started Grouped by sync "family" and external service id * Improve metric name * Don't tag metrics with external service id This could increase the cardinality of our metrics too much * Observe the time since the oldest sync * Simplify function Co-authored-by: Ryan Slade <ryanslade@gmail.com>
2020-10-08 15:27:12 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: add more panels to Grafana repo-updater dashboard (#14269) Improves the repo-updater Grafana dashboard with many new panels as well as observing more of repo-updater's internal state. * monitoring: add more panels to Grafana repo-updater dashboard * Remove comment and fix typo * Add dashboard panels for external_service_sync_jobs state * Increase alert threshold * Remove duplicate metrics schedKnownRepos and schedHeapCount were measuring the same thing in different ways and the old version of schedKnownRepos was assuming a single syncer. Remove the old version of schedKnowRepos and use schedHeapCount instead. * Observe the number of repos queued for update * Fix observable name * Observe when a sync was started Grouped by sync "family" and external service id * Improve metric name * Don't tag metrics with external service id This could increase the cardinality of our metrics too much * Observe the time since the oldest sync * Simplify function Co-authored-by: Ryan Slade <ryanslade@gmail.com>
2020-10-08 15:27:12 +00:00
repo-updater: Add critical alerts (#14530) * repo-updater: Add critical alerts The number of user added repos is greater than 90% of our hard limit of 200k. Number of external services added is great than 20k No external services have synced in more than 8 hours. By default we backoff for up to 8 hours, so anything higher than this indicates a problem. * syncer_sync_start should be a rate Alert when we've performed <= 1 syncs for 8 hours. Ideally we want this to be <= 0 but our monitoring package doesn't allow this. Therefore we need to make this a warning as it'll fire for instances that only have 1 external service defined. * Fill in possible solutions and remove unnecessary panel * Adjust rate * Update monitoring/repo_updater.go Co-authored-by: Tomás Senart <tomas@sourcegraph.com> * Update monitoring/repo_updater.go Co-authored-by: Robert Lin <robert@bobheadxi.dev> * Update possible error descriptions * Fix alert * Switch alert to critical Co-authored-by: Tomás Senart <tomas@sourcegraph.com> Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2020-10-09 13:58:35 +00:00
- Check repo-updater logs for errors
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#repo-updater-sched-error).
monitoring: add more panels to Grafana repo-updater dashboard (#14269) Improves the repo-updater Grafana dashboard with many new panels as well as observing more of repo-updater's internal state. * monitoring: add more panels to Grafana repo-updater dashboard * Remove comment and fix typo * Add dashboard panels for external_service_sync_jobs state * Increase alert threshold * Remove duplicate metrics schedKnownRepos and schedHeapCount were measuring the same thing in different ways and the old version of schedKnownRepos was assuming a single syncer. Remove the old version of schedKnowRepos and use schedHeapCount instead. * Observe the number of repos queued for update * Fix observable name * Observe when a sync was started Grouped by sync "family" and external service id * Improve metric name * Don't tag metrics with external service id This could increase the cardinality of our metrics too much * Observe the time since the oldest sync * Simplify function Co-authored-by: Ryan Slade <ryanslade@gmail.com>
2020-10-08 15:27:12 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"critical_repo-updater_sched_error"
]
```
Update o11y ownership from IAM/repo-mgmt to Source (#54368) Does what it says in the title. ## Test plan - Ran `sg generate`
2023-06-28 13:49:16 +00:00
<sub>*Managed by the [Sourcegraph Source team](https://handbook.sourcegraph.com/departments/engineering/teams/source).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
Generated query for critical alert: `max((max(rate(src_repoupdater_sched_error[1m]))) >= 1)`
</details>
monitoring: improve solutions formatting, add team information (#14878)
2020-10-20 05:11:20 +00:00
<br />
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: add more panels to Grafana repo-updater dashboard (#14269) Improves the repo-updater Grafana dashboard with many new panels as well as observing more of repo-updater's internal state. * monitoring: add more panels to Grafana repo-updater dashboard * Remove comment and fix typo * Add dashboard panels for external_service_sync_jobs state * Increase alert threshold * Remove duplicate metrics schedKnownRepos and schedHeapCount were measuring the same thing in different ways and the old version of schedKnownRepos was assuming a single syncer. Remove the old version of schedKnowRepos and use schedHeapCount instead. * Observe the number of repos queued for update * Fix observable name * Observe when a sync was started Grouped by sync "family" and external service id * Improve metric name * Don't tag metrics with external service id This could increase the cardinality of our metrics too much * Observe the time since the oldest sync * Simplify function Co-authored-by: Ryan Slade <ryanslade@gmail.com>
2020-10-08 15:27:12 +00:00
## repo-updater: src_repoupdater_external_services_total
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
<p class="subtitle">the total number of external services</p>
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
monitoring: add more panels to Grafana repo-updater dashboard (#14269) Improves the repo-updater Grafana dashboard with many new panels as well as observing more of repo-updater's internal state. * monitoring: add more panels to Grafana repo-updater dashboard * Remove comment and fix typo * Add dashboard panels for external_service_sync_jobs state * Increase alert threshold * Remove duplicate metrics schedKnownRepos and schedHeapCount were measuring the same thing in different ways and the old version of schedKnownRepos was assuming a single syncer. Remove the old version of schedKnowRepos and use schedHeapCount instead. * Observe the number of repos queued for update * Fix observable name * Observe when a sync was started Grouped by sync "family" and external service id * Improve metric name * Don't tag metrics with external service id This could increase the cardinality of our metrics too much * Observe the time since the oldest sync * Simplify function Co-authored-by: Ryan Slade <ryanslade@gmail.com>
2020-10-08 15:27:12 +00:00
monitoring: dashboards reference, refresh observability docs (#16939) - New monitoring definition field, 'Interpretation' - New generated dashboards reference - Grafana panels now link to alert solutions and dashboards reference - Refreshed observability docs Co-authored-by: uwedeportivo <534011+uwedeportivo@users.noreply.github.com>
2020-12-23 02:36:41 +00:00
- <span class="badge badge-critical">critical</span> repo-updater: 20000+ the total number of external services for 1h0m0s
monitoring: add more panels to Grafana repo-updater dashboard (#14269) Improves the repo-updater Grafana dashboard with many new panels as well as observing more of repo-updater's internal state. * monitoring: add more panels to Grafana repo-updater dashboard * Remove comment and fix typo * Add dashboard panels for external_service_sync_jobs state * Increase alert threshold * Remove duplicate metrics schedKnownRepos and schedHeapCount were measuring the same thing in different ways and the old version of schedKnownRepos was assuming a single syncer. Remove the old version of schedKnowRepos and use schedHeapCount instead. * Observe the number of repos queued for update * Fix observable name * Observe when a sync was started Grouped by sync "family" and external service id * Improve metric name * Don't tag metrics with external service id This could increase the cardinality of our metrics too much * Observe the time since the oldest sync * Simplify function Co-authored-by: Ryan Slade <ryanslade@gmail.com>
2020-10-08 15:27:12 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: add more panels to Grafana repo-updater dashboard (#14269) Improves the repo-updater Grafana dashboard with many new panels as well as observing more of repo-updater's internal state. * monitoring: add more panels to Grafana repo-updater dashboard * Remove comment and fix typo * Add dashboard panels for external_service_sync_jobs state * Increase alert threshold * Remove duplicate metrics schedKnownRepos and schedHeapCount were measuring the same thing in different ways and the old version of schedKnownRepos was assuming a single syncer. Remove the old version of schedKnowRepos and use schedHeapCount instead. * Observe the number of repos queued for update * Fix observable name * Observe when a sync was started Grouped by sync "family" and external service id * Improve metric name * Don't tag metrics with external service id This could increase the cardinality of our metrics too much * Observe the time since the oldest sync * Simplify function Co-authored-by: Ryan Slade <ryanslade@gmail.com>
2020-10-08 15:27:12 +00:00
repo-updater: Add critical alerts (#14530) * repo-updater: Add critical alerts The number of user added repos is greater than 90% of our hard limit of 200k. Number of external services added is great than 20k No external services have synced in more than 8 hours. By default we backoff for up to 8 hours, so anything higher than this indicates a problem. * syncer_sync_start should be a rate Alert when we've performed <= 1 syncs for 8 hours. Ideally we want this to be <= 0 but our monitoring package doesn't allow this. Therefore we need to make this a warning as it'll fire for instances that only have 1 external service defined. * Fill in possible solutions and remove unnecessary panel * Adjust rate * Update monitoring/repo_updater.go Co-authored-by: Tomás Senart <tomas@sourcegraph.com> * Update monitoring/repo_updater.go Co-authored-by: Robert Lin <robert@bobheadxi.dev> * Update possible error descriptions * Fix alert * Switch alert to critical Co-authored-by: Tomás Senart <tomas@sourcegraph.com> Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2020-10-09 13:58:35 +00:00
- Check for spikes in external services, could be abuse
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#repo-updater-src-repoupdater-external-services-total).
monitoring: add more panels to Grafana repo-updater dashboard (#14269) Improves the repo-updater Grafana dashboard with many new panels as well as observing more of repo-updater's internal state. * monitoring: add more panels to Grafana repo-updater dashboard * Remove comment and fix typo * Add dashboard panels for external_service_sync_jobs state * Increase alert threshold * Remove duplicate metrics schedKnownRepos and schedHeapCount were measuring the same thing in different ways and the old version of schedKnownRepos was assuming a single syncer. Remove the old version of schedKnowRepos and use schedHeapCount instead. * Observe the number of repos queued for update * Fix observable name * Observe when a sync was started Grouped by sync "family" and external service id * Improve metric name * Don't tag metrics with external service id This could increase the cardinality of our metrics too much * Observe the time since the oldest sync * Simplify function Co-authored-by: Ryan Slade <ryanslade@gmail.com>
2020-10-08 15:27:12 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
repo-updater: Add critical alerts (#14530) * repo-updater: Add critical alerts The number of user added repos is greater than 90% of our hard limit of 200k. Number of external services added is great than 20k No external services have synced in more than 8 hours. By default we backoff for up to 8 hours, so anything higher than this indicates a problem. * syncer_sync_start should be a rate Alert when we've performed <= 1 syncs for 8 hours. Ideally we want this to be <= 0 but our monitoring package doesn't allow this. Therefore we need to make this a warning as it'll fire for instances that only have 1 external service defined. * Fill in possible solutions and remove unnecessary panel * Adjust rate * Update monitoring/repo_updater.go Co-authored-by: Tomás Senart <tomas@sourcegraph.com> * Update monitoring/repo_updater.go Co-authored-by: Robert Lin <robert@bobheadxi.dev> * Update possible error descriptions * Fix alert * Switch alert to critical Co-authored-by: Tomás Senart <tomas@sourcegraph.com> Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2020-10-09 13:58:35 +00:00
"critical_repo-updater_src_repoupdater_external_services_total"
monitoring: add more panels to Grafana repo-updater dashboard (#14269) Improves the repo-updater Grafana dashboard with many new panels as well as observing more of repo-updater's internal state. * monitoring: add more panels to Grafana repo-updater dashboard * Remove comment and fix typo * Add dashboard panels for external_service_sync_jobs state * Increase alert threshold * Remove duplicate metrics schedKnownRepos and schedHeapCount were measuring the same thing in different ways and the old version of schedKnownRepos was assuming a single syncer. Remove the old version of schedKnowRepos and use schedHeapCount instead. * Observe the number of repos queued for update * Fix observable name * Observe when a sync was started Grouped by sync "family" and external service id * Improve metric name * Don't tag metrics with external service id This could increase the cardinality of our metrics too much * Observe the time since the oldest sync * Simplify function Co-authored-by: Ryan Slade <ryanslade@gmail.com>
2020-10-08 15:27:12 +00:00
]
```
Update o11y ownership from IAM/repo-mgmt to Source (#54368) Does what it says in the title. ## Test plan - Ran `sg generate`
2023-06-28 13:49:16 +00:00
<sub>*Managed by the [Sourcegraph Source team](https://handbook.sourcegraph.com/departments/engineering/teams/source).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
Generated query for critical alert: `max((max(src_repoupdater_external_services_total)) >= 20000)`
</details>
monitoring: improve solutions formatting, add team information (#14878)
2020-10-20 05:11:20 +00:00
<br />
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
Add alert and metric for total queued jobs (#14435) This adds a metric counting the number of jobs that are queued and an alert when their number is above 100 for a whole hour.
2020-10-06 13:50:55 +00:00
## repo-updater: repoupdater_queued_sync_jobs_total
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
<p class="subtitle">the total number of queued sync jobs</p>
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
Add alert and metric for total queued jobs (#14435) This adds a metric counting the number of jobs that are queued and an alert when their number is above 100 for a whole hour.
2020-10-06 13:50:55 +00:00
monitoring: dashboards reference, refresh observability docs (#16939) - New monitoring definition field, 'Interpretation' - New generated dashboards reference - Grafana panels now link to alert solutions and dashboards reference - Refreshed observability docs Co-authored-by: uwedeportivo <534011+uwedeportivo@users.noreply.github.com>
2020-12-23 02:36:41 +00:00
- <span class="badge badge-warning">warning</span> repo-updater: 100+ the total number of queued sync jobs for 1h0m0s
Add alert and metric for total queued jobs (#14435) This adds a metric counting the number of jobs that are queued and an alert when their number is above 100 for a whole hour.
2020-10-06 13:50:55 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
Add alert and metric for total queued jobs (#14435) This adds a metric counting the number of jobs that are queued and an alert when their number is above 100 for a whole hour.
2020-10-06 13:50:55 +00:00
- **Check if jobs are failing to sync:** "SELECT * FROM external_service_sync_jobs WHERE state = `errored`";
- **Increase the number of workers** using the `repoConcurrentExternalServiceSyncers` site config.
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#repo-updater-repoupdater-queued-sync-jobs-total).
Add alert and metric for total queued jobs (#14435) This adds a metric counting the number of jobs that are queued and an alert when their number is above 100 for a whole hour.
2020-10-06 13:50:55 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"warning_repo-updater_repoupdater_queued_sync_jobs_total"
]
```
Update o11y ownership from IAM/repo-mgmt to Source (#54368) Does what it says in the title. ## Test plan - Ran `sg generate`
2023-06-28 13:49:16 +00:00
<sub>*Managed by the [Sourcegraph Source team](https://handbook.sourcegraph.com/departments/engineering/teams/source).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
Generated query for warning alert: `max((max(src_repoupdater_queued_sync_jobs_total)) >= 100)`
</details>
monitoring: improve solutions formatting, add team information (#14878)
2020-10-20 05:11:20 +00:00
<br />
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: add more panels to Grafana repo-updater dashboard (#14269) Improves the repo-updater Grafana dashboard with many new panels as well as observing more of repo-updater's internal state. * monitoring: add more panels to Grafana repo-updater dashboard * Remove comment and fix typo * Add dashboard panels for external_service_sync_jobs state * Increase alert threshold * Remove duplicate metrics schedKnownRepos and schedHeapCount were measuring the same thing in different ways and the old version of schedKnownRepos was assuming a single syncer. Remove the old version of schedKnowRepos and use schedHeapCount instead. * Observe the number of repos queued for update * Fix observable name * Observe when a sync was started Grouped by sync "family" and external service id * Improve metric name * Don't tag metrics with external service id This could increase the cardinality of our metrics too much * Observe the time since the oldest sync * Simplify function Co-authored-by: Ryan Slade <ryanslade@gmail.com>
2020-10-08 15:27:12 +00:00
## repo-updater: repoupdater_completed_sync_jobs_total
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
<p class="subtitle">the total number of completed sync jobs</p>
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
monitoring: add more panels to Grafana repo-updater dashboard (#14269) Improves the repo-updater Grafana dashboard with many new panels as well as observing more of repo-updater's internal state. * monitoring: add more panels to Grafana repo-updater dashboard * Remove comment and fix typo * Add dashboard panels for external_service_sync_jobs state * Increase alert threshold * Remove duplicate metrics schedKnownRepos and schedHeapCount were measuring the same thing in different ways and the old version of schedKnownRepos was assuming a single syncer. Remove the old version of schedKnowRepos and use schedHeapCount instead. * Observe the number of repos queued for update * Fix observable name * Observe when a sync was started Grouped by sync "family" and external service id * Improve metric name * Don't tag metrics with external service id This could increase the cardinality of our metrics too much * Observe the time since the oldest sync * Simplify function Co-authored-by: Ryan Slade <ryanslade@gmail.com>
2020-10-08 15:27:12 +00:00
monitoring: dashboards reference, refresh observability docs (#16939) - New monitoring definition field, 'Interpretation' - New generated dashboards reference - Grafana panels now link to alert solutions and dashboards reference - Refreshed observability docs Co-authored-by: uwedeportivo <534011+uwedeportivo@users.noreply.github.com>
2020-12-23 02:36:41 +00:00
- <span class="badge badge-warning">warning</span> repo-updater: 100000+ the total number of completed sync jobs for 1h0m0s
monitoring: add more panels to Grafana repo-updater dashboard (#14269) Improves the repo-updater Grafana dashboard with many new panels as well as observing more of repo-updater's internal state. * monitoring: add more panels to Grafana repo-updater dashboard * Remove comment and fix typo * Add dashboard panels for external_service_sync_jobs state * Increase alert threshold * Remove duplicate metrics schedKnownRepos and schedHeapCount were measuring the same thing in different ways and the old version of schedKnownRepos was assuming a single syncer. Remove the old version of schedKnowRepos and use schedHeapCount instead. * Observe the number of repos queued for update * Fix observable name * Observe when a sync was started Grouped by sync "family" and external service id * Improve metric name * Don't tag metrics with external service id This could increase the cardinality of our metrics too much * Observe the time since the oldest sync * Simplify function Co-authored-by: Ryan Slade <ryanslade@gmail.com>
2020-10-08 15:27:12 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: add more panels to Grafana repo-updater dashboard (#14269) Improves the repo-updater Grafana dashboard with many new panels as well as observing more of repo-updater's internal state. * monitoring: add more panels to Grafana repo-updater dashboard * Remove comment and fix typo * Add dashboard panels for external_service_sync_jobs state * Increase alert threshold * Remove duplicate metrics schedKnownRepos and schedHeapCount were measuring the same thing in different ways and the old version of schedKnownRepos was assuming a single syncer. Remove the old version of schedKnowRepos and use schedHeapCount instead. * Observe the number of repos queued for update * Fix observable name * Observe when a sync was started Grouped by sync "family" and external service id * Improve metric name * Don't tag metrics with external service id This could increase the cardinality of our metrics too much * Observe the time since the oldest sync * Simplify function Co-authored-by: Ryan Slade <ryanslade@gmail.com>
2020-10-08 15:27:12 +00:00
repo-updater: Add critical alerts (#14530) * repo-updater: Add critical alerts The number of user added repos is greater than 90% of our hard limit of 200k. Number of external services added is great than 20k No external services have synced in more than 8 hours. By default we backoff for up to 8 hours, so anything higher than this indicates a problem. * syncer_sync_start should be a rate Alert when we've performed <= 1 syncs for 8 hours. Ideally we want this to be <= 0 but our monitoring package doesn't allow this. Therefore we need to make this a warning as it'll fire for instances that only have 1 external service defined. * Fill in possible solutions and remove unnecessary panel * Adjust rate * Update monitoring/repo_updater.go Co-authored-by: Tomás Senart <tomas@sourcegraph.com> * Update monitoring/repo_updater.go Co-authored-by: Robert Lin <robert@bobheadxi.dev> * Update possible error descriptions * Fix alert * Switch alert to critical Co-authored-by: Tomás Senart <tomas@sourcegraph.com> Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2020-10-09 13:58:35 +00:00
- Check repo-updater logs. Jobs older than 1 day should have been removed.
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#repo-updater-repoupdater-completed-sync-jobs-total).
monitoring: add more panels to Grafana repo-updater dashboard (#14269) Improves the repo-updater Grafana dashboard with many new panels as well as observing more of repo-updater's internal state. * monitoring: add more panels to Grafana repo-updater dashboard * Remove comment and fix typo * Add dashboard panels for external_service_sync_jobs state * Increase alert threshold * Remove duplicate metrics schedKnownRepos and schedHeapCount were measuring the same thing in different ways and the old version of schedKnownRepos was assuming a single syncer. Remove the old version of schedKnowRepos and use schedHeapCount instead. * Observe the number of repos queued for update * Fix observable name * Observe when a sync was started Grouped by sync "family" and external service id * Improve metric name * Don't tag metrics with external service id This could increase the cardinality of our metrics too much * Observe the time since the oldest sync * Simplify function Co-authored-by: Ryan Slade <ryanslade@gmail.com>
2020-10-08 15:27:12 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"warning_repo-updater_repoupdater_completed_sync_jobs_total"
]
```
Update o11y ownership from IAM/repo-mgmt to Source (#54368) Does what it says in the title. ## Test plan - Ran `sg generate`
2023-06-28 13:49:16 +00:00
<sub>*Managed by the [Sourcegraph Source team](https://handbook.sourcegraph.com/departments/engineering/teams/source).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
Generated query for warning alert: `max((max(src_repoupdater_completed_sync_jobs_total)) >= 100000)`
</details>
monitoring: improve solutions formatting, add team information (#14878)
2020-10-20 05:11:20 +00:00
<br />
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: Switch to monitoring percentage of sync failures (#22350) Alerting on a fixed number of 100 doesn't scale. Instead, we'll alert when more than 10% of external services are currently failing to sync.
2021-06-24 18:48:56 +00:00
## repo-updater: repoupdater_errored_sync_jobs_percentage
monitoring: add more panels to Grafana repo-updater dashboard (#14269) Improves the repo-updater Grafana dashboard with many new panels as well as observing more of repo-updater's internal state. * monitoring: add more panels to Grafana repo-updater dashboard * Remove comment and fix typo * Add dashboard panels for external_service_sync_jobs state * Increase alert threshold * Remove duplicate metrics schedKnownRepos and schedHeapCount were measuring the same thing in different ways and the old version of schedKnownRepos was assuming a single syncer. Remove the old version of schedKnowRepos and use schedHeapCount instead. * Observe the number of repos queued for update * Fix observable name * Observe when a sync was started Grouped by sync "family" and external service id * Improve metric name * Don't tag metrics with external service id This could increase the cardinality of our metrics too much * Observe the time since the oldest sync * Simplify function Co-authored-by: Ryan Slade <ryanslade@gmail.com>
2020-10-08 15:27:12 +00:00
monitoring: Switch to monitoring percentage of sync failures (#22350) Alerting on a fixed number of 100 doesn't scale. Instead, we'll alert when more than 10% of external services are currently failing to sync.
2021-06-24 18:48:56 +00:00
<p class="subtitle">the percentage of external services that have failed their most recent sync</p>
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
monitoring: add more panels to Grafana repo-updater dashboard (#14269) Improves the repo-updater Grafana dashboard with many new panels as well as observing more of repo-updater's internal state. * monitoring: add more panels to Grafana repo-updater dashboard * Remove comment and fix typo * Add dashboard panels for external_service_sync_jobs state * Increase alert threshold * Remove duplicate metrics schedKnownRepos and schedHeapCount were measuring the same thing in different ways and the old version of schedKnownRepos was assuming a single syncer. Remove the old version of schedKnowRepos and use schedHeapCount instead. * Observe the number of repos queued for update * Fix observable name * Observe when a sync was started Grouped by sync "family" and external service id * Improve metric name * Don't tag metrics with external service id This could increase the cardinality of our metrics too much * Observe the time since the oldest sync * Simplify function Co-authored-by: Ryan Slade <ryanslade@gmail.com>
2020-10-08 15:27:12 +00:00
monitoring: Switch to monitoring percentage of sync failures (#22350) Alerting on a fixed number of 100 doesn't scale. Instead, we'll alert when more than 10% of external services are currently failing to sync.
2021-06-24 18:48:56 +00:00
- <span class="badge badge-warning">warning</span> repo-updater: 10%+ the percentage of external services that have failed their most recent sync for 1h0m0s
monitoring: add more panels to Grafana repo-updater dashboard (#14269) Improves the repo-updater Grafana dashboard with many new panels as well as observing more of repo-updater's internal state. * monitoring: add more panels to Grafana repo-updater dashboard * Remove comment and fix typo * Add dashboard panels for external_service_sync_jobs state * Increase alert threshold * Remove duplicate metrics schedKnownRepos and schedHeapCount were measuring the same thing in different ways and the old version of schedKnownRepos was assuming a single syncer. Remove the old version of schedKnowRepos and use schedHeapCount instead. * Observe the number of repos queued for update * Fix observable name * Observe when a sync was started Grouped by sync "family" and external service id * Improve metric name * Don't tag metrics with external service id This could increase the cardinality of our metrics too much * Observe the time since the oldest sync * Simplify function Co-authored-by: Ryan Slade <ryanslade@gmail.com>
2020-10-08 15:27:12 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: add more panels to Grafana repo-updater dashboard (#14269) Improves the repo-updater Grafana dashboard with many new panels as well as observing more of repo-updater's internal state. * monitoring: add more panels to Grafana repo-updater dashboard * Remove comment and fix typo * Add dashboard panels for external_service_sync_jobs state * Increase alert threshold * Remove duplicate metrics schedKnownRepos and schedHeapCount were measuring the same thing in different ways and the old version of schedKnownRepos was assuming a single syncer. Remove the old version of schedKnowRepos and use schedHeapCount instead. * Observe the number of repos queued for update * Fix observable name * Observe when a sync was started Grouped by sync "family" and external service id * Improve metric name * Don't tag metrics with external service id This could increase the cardinality of our metrics too much * Observe the time since the oldest sync * Simplify function Co-authored-by: Ryan Slade <ryanslade@gmail.com>
2020-10-08 15:27:12 +00:00
repo-updater: Add critical alerts (#14530) * repo-updater: Add critical alerts The number of user added repos is greater than 90% of our hard limit of 200k. Number of external services added is great than 20k No external services have synced in more than 8 hours. By default we backoff for up to 8 hours, so anything higher than this indicates a problem. * syncer_sync_start should be a rate Alert when we've performed <= 1 syncs for 8 hours. Ideally we want this to be <= 0 but our monitoring package doesn't allow this. Therefore we need to make this a warning as it'll fire for instances that only have 1 external service defined. * Fill in possible solutions and remove unnecessary panel * Adjust rate * Update monitoring/repo_updater.go Co-authored-by: Tomás Senart <tomas@sourcegraph.com> * Update monitoring/repo_updater.go Co-authored-by: Robert Lin <robert@bobheadxi.dev> * Update possible error descriptions * Fix alert * Switch alert to critical Co-authored-by: Tomás Senart <tomas@sourcegraph.com> Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2020-10-09 13:58:35 +00:00
- Check repo-updater logs. Check code host connectivity
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#repo-updater-repoupdater-errored-sync-jobs-percentage).
monitoring: add more panels to Grafana repo-updater dashboard (#14269) Improves the repo-updater Grafana dashboard with many new panels as well as observing more of repo-updater's internal state. * monitoring: add more panels to Grafana repo-updater dashboard * Remove comment and fix typo * Add dashboard panels for external_service_sync_jobs state * Increase alert threshold * Remove duplicate metrics schedKnownRepos and schedHeapCount were measuring the same thing in different ways and the old version of schedKnownRepos was assuming a single syncer. Remove the old version of schedKnowRepos and use schedHeapCount instead. * Observe the number of repos queued for update * Fix observable name * Observe when a sync was started Grouped by sync "family" and external service id * Improve metric name * Don't tag metrics with external service id This could increase the cardinality of our metrics too much * Observe the time since the oldest sync * Simplify function Co-authored-by: Ryan Slade <ryanslade@gmail.com>
2020-10-08 15:27:12 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
monitoring: Switch to monitoring percentage of sync failures (#22350) Alerting on a fixed number of 100 doesn't scale. Instead, we'll alert when more than 10% of external services are currently failing to sync.
2021-06-24 18:48:56 +00:00
"warning_repo-updater_repoupdater_errored_sync_jobs_percentage"
monitoring: add more panels to Grafana repo-updater dashboard (#14269) Improves the repo-updater Grafana dashboard with many new panels as well as observing more of repo-updater's internal state. * monitoring: add more panels to Grafana repo-updater dashboard * Remove comment and fix typo * Add dashboard panels for external_service_sync_jobs state * Increase alert threshold * Remove duplicate metrics schedKnownRepos and schedHeapCount were measuring the same thing in different ways and the old version of schedKnownRepos was assuming a single syncer. Remove the old version of schedKnowRepos and use schedHeapCount instead. * Observe the number of repos queued for update * Fix observable name * Observe when a sync was started Grouped by sync "family" and external service id * Improve metric name * Don't tag metrics with external service id This could increase the cardinality of our metrics too much * Observe the time since the oldest sync * Simplify function Co-authored-by: Ryan Slade <ryanslade@gmail.com>
2020-10-08 15:27:12 +00:00
]
```
Update o11y ownership from IAM/repo-mgmt to Source (#54368) Does what it says in the title. ## Test plan - Ran `sg generate`
2023-06-28 13:49:16 +00:00
<sub>*Managed by the [Sourcegraph Source team](https://handbook.sourcegraph.com/departments/engineering/teams/source).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
Generated query for warning alert: `max((max(src_repoupdater_errored_sync_jobs_percentage)) > 10)`
</details>
monitoring: improve solutions formatting, add team information (#14878)
2020-10-20 05:11:20 +00:00
<br />
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
Instrument GitHub and GitLab rate limit remaining per code host connection (#16750) Instrument rate limit remaining per code host connection for GitHub. We measure the remaining limit for their rest, graphql and search APIs separately. Instrument rate limit remaining per code host connection for GitLab. We only measure the remaining limit for their rest API as they only have one limit. Panels were added to repo-updater with alerts when the rate limit is nearly depleted.
2020-12-15 14:45:44 +00:00
## repo-updater: github_graphql_rate_limit_remaining
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
<p class="subtitle">remaining calls to GitHub graphql API before hitting the rate limit</p>
Instrument GitHub and GitLab rate limit remaining per code host connection (#16750) Instrument rate limit remaining per code host connection for GitHub. We measure the remaining limit for their rest, graphql and search APIs separately. Instrument rate limit remaining per code host connection for GitLab. We only measure the remaining limit for their rest API as they only have one limit. Panels were added to repo-updater with alerts when the rate limit is nearly depleted.
2020-12-15 14:45:44 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
Instrument GitHub and GitLab rate limit remaining per code host connection (#16750) Instrument rate limit remaining per code host connection for GitHub. We measure the remaining limit for their rest, graphql and search APIs separately. Instrument rate limit remaining per code host connection for GitLab. We only measure the remaining limit for their rest API as they only have one limit. Panels were added to repo-updater with alerts when the rate limit is nearly depleted.
2020-12-15 14:45:44 +00:00
monitoring(repo-updater): add duration on critical rate limit alerts, update next steps (#36683) The existing advice is not great, because restarting the pod doesn't guarantee a new public IP, and it's possible customers are using a public gateway (e.g. Cloud NAT) with VM without a public IP address. It's also noted that this alert comes up quite frequently - I think on larger instances running up against the limit is quite common and even expected, and Sourcegraph should for the most part continue working even if it exhausts its rate limits. It becomes a critical issue only if the rate limit is exhausted immediately after a rate limit reset. We can detect this by checking to see if the rate limit is below the threshold for ~most of a window. This change addresses both issues with updates to the alerts and NextSteps.
2022-06-07 15:18:48 +00:00
- <span class="badge badge-warning">warning</span> repo-updater: less than 250 remaining calls to GitHub graphql API before hitting the rate limit
Instrument GitHub and GitLab rate limit remaining per code host connection (#16750) Instrument rate limit remaining per code host connection for GitHub. We measure the remaining limit for their rest, graphql and search APIs separately. Instrument rate limit remaining per code host connection for GitLab. We only measure the remaining limit for their rest API as they only have one limit. Panels were added to repo-updater with alerts when the rate limit is nearly depleted.
2020-12-15 14:45:44 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
Instrument GitHub and GitLab rate limit remaining per code host connection (#16750) Instrument rate limit remaining per code host connection for GitHub. We measure the remaining limit for their rest, graphql and search APIs separately. Instrument rate limit remaining per code host connection for GitLab. We only measure the remaining limit for their rest API as they only have one limit. Panels were added to repo-updater with alerts when the rate limit is nearly depleted.
2020-12-15 14:45:44 +00:00
monitoring(repo-updater): add duration on critical rate limit alerts, update next steps (#36683) The existing advice is not great, because restarting the pod doesn't guarantee a new public IP, and it's possible customers are using a public gateway (e.g. Cloud NAT) with VM without a public IP address. It's also noted that this alert comes up quite frequently - I think on larger instances running up against the limit is quite common and even expected, and Sourcegraph should for the most part continue working even if it exhausts its rate limits. It becomes a critical issue only if the rate limit is exhausted immediately after a rate limit reset. We can detect this by checking to see if the rate limit is below the threshold for ~most of a window. This change addresses both issues with updates to the alerts and NextSteps.
2022-06-07 15:18:48 +00:00
- Consider creating a new token for the indicated resource (the `name` label for series below the threshold in the dashboard) under a dedicated machine user to reduce rate limit pressure.
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#repo-updater-github-graphql-rate-limit-remaining).
Instrument GitHub and GitLab rate limit remaining per code host connection (#16750) Instrument rate limit remaining per code host connection for GitHub. We measure the remaining limit for their rest, graphql and search APIs separately. Instrument rate limit remaining per code host connection for GitLab. We only measure the remaining limit for their rest API as they only have one limit. Panels were added to repo-updater with alerts when the rate limit is nearly depleted.
2020-12-15 14:45:44 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
Remove code host rate limit critical alerts (#39818)
2022-08-02 14:15:53 +00:00
"warning_repo-updater_github_graphql_rate_limit_remaining"
Instrument GitHub and GitLab rate limit remaining per code host connection (#16750) Instrument rate limit remaining per code host connection for GitHub. We measure the remaining limit for their rest, graphql and search APIs separately. Instrument rate limit remaining per code host connection for GitLab. We only measure the remaining limit for their rest API as they only have one limit. Panels were added to repo-updater with alerts when the rate limit is nearly depleted.
2020-12-15 14:45:44 +00:00
]
```
Update o11y ownership from IAM/repo-mgmt to Source (#54368) Does what it says in the title. ## Test plan - Ran `sg generate`
2023-06-28 13:49:16 +00:00
<sub>*Managed by the [Sourcegraph Source team](https://handbook.sourcegraph.com/departments/engineering/teams/source).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
monitoring: extract into a submodule (#45786) This change extracts `monitoring` into a submodule for import in `sourcegraph/controller` (https://github.com/sourcegraph/controller/pull/195) so that we can generate dashboards for Cloud instances. These steps were required: 1. Initialize a `go.mod` in `monitoring` 2. Extract `dev/sg/internal/cliutil` into `lib` to avoid illegal imports from `monitoring` 3. Add local replaces to both `sourcegraph/sourcegraph` and `monitoring` 4. `go mod tidy` on all submodules 5. Update `go generate ./monitoring` commands to use `sg`, since the `go generate` command no longer works 6. Update `grafana/build.sh`, `prometheus/build.sh` to build the submodule 7. Amend linters to check for multiple `go.mod` files and ban imports of `github.com/sourcegraph/sourcegraph` 8. Update `sg generate go` to run in directories rather than from root The only caveat is that if you use VS code, you will now need to open `monitoring` in a separate workspace or similar, like with `lib`. Co-authored-by: Joe Chen <joe@sourcegraph.com>
2022-12-19 17:49:25 +00:00
Generated query for warning alert: `min((max by (name) (src_github_rate_limit_remaining_v2{resource="graphql"})) <= 250)`
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
</details>
Instrument GitHub and GitLab rate limit remaining per code host connection (#16750) Instrument rate limit remaining per code host connection for GitHub. We measure the remaining limit for their rest, graphql and search APIs separately. Instrument rate limit remaining per code host connection for GitLab. We only measure the remaining limit for their rest API as they only have one limit. Panels were added to repo-updater with alerts when the rate limit is nearly depleted.
2020-12-15 14:45:44 +00:00
<br />
## repo-updater: github_rest_rate_limit_remaining
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
<p class="subtitle">remaining calls to GitHub rest API before hitting the rate limit</p>
Instrument GitHub and GitLab rate limit remaining per code host connection (#16750) Instrument rate limit remaining per code host connection for GitHub. We measure the remaining limit for their rest, graphql and search APIs separately. Instrument rate limit remaining per code host connection for GitLab. We only measure the remaining limit for their rest API as they only have one limit. Panels were added to repo-updater with alerts when the rate limit is nearly depleted.
2020-12-15 14:45:44 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
Instrument GitHub and GitLab rate limit remaining per code host connection (#16750) Instrument rate limit remaining per code host connection for GitHub. We measure the remaining limit for their rest, graphql and search APIs separately. Instrument rate limit remaining per code host connection for GitLab. We only measure the remaining limit for their rest API as they only have one limit. Panels were added to repo-updater with alerts when the rate limit is nearly depleted.
2020-12-15 14:45:44 +00:00
monitoring(repo-updater): add duration on critical rate limit alerts, update next steps (#36683) The existing advice is not great, because restarting the pod doesn't guarantee a new public IP, and it's possible customers are using a public gateway (e.g. Cloud NAT) with VM without a public IP address. It's also noted that this alert comes up quite frequently - I think on larger instances running up against the limit is quite common and even expected, and Sourcegraph should for the most part continue working even if it exhausts its rate limits. It becomes a critical issue only if the rate limit is exhausted immediately after a rate limit reset. We can detect this by checking to see if the rate limit is below the threshold for ~most of a window. This change addresses both issues with updates to the alerts and NextSteps.
2022-06-07 15:18:48 +00:00
- <span class="badge badge-warning">warning</span> repo-updater: less than 250 remaining calls to GitHub rest API before hitting the rate limit
Instrument GitHub and GitLab rate limit remaining per code host connection (#16750) Instrument rate limit remaining per code host connection for GitHub. We measure the remaining limit for their rest, graphql and search APIs separately. Instrument rate limit remaining per code host connection for GitLab. We only measure the remaining limit for their rest API as they only have one limit. Panels were added to repo-updater with alerts when the rate limit is nearly depleted.
2020-12-15 14:45:44 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
Instrument GitHub and GitLab rate limit remaining per code host connection (#16750) Instrument rate limit remaining per code host connection for GitHub. We measure the remaining limit for their rest, graphql and search APIs separately. Instrument rate limit remaining per code host connection for GitLab. We only measure the remaining limit for their rest API as they only have one limit. Panels were added to repo-updater with alerts when the rate limit is nearly depleted.
2020-12-15 14:45:44 +00:00
monitoring(repo-updater): add duration on critical rate limit alerts, update next steps (#36683) The existing advice is not great, because restarting the pod doesn't guarantee a new public IP, and it's possible customers are using a public gateway (e.g. Cloud NAT) with VM without a public IP address. It's also noted that this alert comes up quite frequently - I think on larger instances running up against the limit is quite common and even expected, and Sourcegraph should for the most part continue working even if it exhausts its rate limits. It becomes a critical issue only if the rate limit is exhausted immediately after a rate limit reset. We can detect this by checking to see if the rate limit is below the threshold for ~most of a window. This change addresses both issues with updates to the alerts and NextSteps.
2022-06-07 15:18:48 +00:00
- Consider creating a new token for the indicated resource (the `name` label for series below the threshold in the dashboard) under a dedicated machine user to reduce rate limit pressure.
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#repo-updater-github-rest-rate-limit-remaining).
Instrument GitHub and GitLab rate limit remaining per code host connection (#16750) Instrument rate limit remaining per code host connection for GitHub. We measure the remaining limit for their rest, graphql and search APIs separately. Instrument rate limit remaining per code host connection for GitLab. We only measure the remaining limit for their rest API as they only have one limit. Panels were added to repo-updater with alerts when the rate limit is nearly depleted.
2020-12-15 14:45:44 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
Remove code host rate limit critical alerts (#39818)
2022-08-02 14:15:53 +00:00
"warning_repo-updater_github_rest_rate_limit_remaining"
Instrument GitHub and GitLab rate limit remaining per code host connection (#16750) Instrument rate limit remaining per code host connection for GitHub. We measure the remaining limit for their rest, graphql and search APIs separately. Instrument rate limit remaining per code host connection for GitLab. We only measure the remaining limit for their rest API as they only have one limit. Panels were added to repo-updater with alerts when the rate limit is nearly depleted.
2020-12-15 14:45:44 +00:00
]
```
Update o11y ownership from IAM/repo-mgmt to Source (#54368) Does what it says in the title. ## Test plan - Ran `sg generate`
2023-06-28 13:49:16 +00:00
<sub>*Managed by the [Sourcegraph Source team](https://handbook.sourcegraph.com/departments/engineering/teams/source).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
monitoring: extract into a submodule (#45786) This change extracts `monitoring` into a submodule for import in `sourcegraph/controller` (https://github.com/sourcegraph/controller/pull/195) so that we can generate dashboards for Cloud instances. These steps were required: 1. Initialize a `go.mod` in `monitoring` 2. Extract `dev/sg/internal/cliutil` into `lib` to avoid illegal imports from `monitoring` 3. Add local replaces to both `sourcegraph/sourcegraph` and `monitoring` 4. `go mod tidy` on all submodules 5. Update `go generate ./monitoring` commands to use `sg`, since the `go generate` command no longer works 6. Update `grafana/build.sh`, `prometheus/build.sh` to build the submodule 7. Amend linters to check for multiple `go.mod` files and ban imports of `github.com/sourcegraph/sourcegraph` 8. Update `sg generate go` to run in directories rather than from root The only caveat is that if you use VS code, you will now need to open `monitoring` in a separate workspace or similar, like with `lib`. Co-authored-by: Joe Chen <joe@sourcegraph.com>
2022-12-19 17:49:25 +00:00
Generated query for warning alert: `min((max by (name) (src_github_rate_limit_remaining_v2{resource="rest"})) <= 250)`
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
</details>
Instrument GitHub and GitLab rate limit remaining per code host connection (#16750) Instrument rate limit remaining per code host connection for GitHub. We measure the remaining limit for their rest, graphql and search APIs separately. Instrument rate limit remaining per code host connection for GitLab. We only measure the remaining limit for their rest API as they only have one limit. Panels were added to repo-updater with alerts when the rate limit is nearly depleted.
2020-12-15 14:45:44 +00:00
<br />
## repo-updater: github_search_rate_limit_remaining
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
<p class="subtitle">remaining calls to GitHub search API before hitting the rate limit</p>
Instrument GitHub and GitLab rate limit remaining per code host connection (#16750) Instrument rate limit remaining per code host connection for GitHub. We measure the remaining limit for their rest, graphql and search APIs separately. Instrument rate limit remaining per code host connection for GitLab. We only measure the remaining limit for their rest API as they only have one limit. Panels were added to repo-updater with alerts when the rate limit is nearly depleted.
2020-12-15 14:45:44 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
Instrument GitHub and GitLab rate limit remaining per code host connection (#16750) Instrument rate limit remaining per code host connection for GitHub. We measure the remaining limit for their rest, graphql and search APIs separately. Instrument rate limit remaining per code host connection for GitLab. We only measure the remaining limit for their rest API as they only have one limit. Panels were added to repo-updater with alerts when the rate limit is nearly depleted.
2020-12-15 14:45:44 +00:00
monitoring(repo-updater): add duration on critical rate limit alerts, update next steps (#36683) The existing advice is not great, because restarting the pod doesn't guarantee a new public IP, and it's possible customers are using a public gateway (e.g. Cloud NAT) with VM without a public IP address. It's also noted that this alert comes up quite frequently - I think on larger instances running up against the limit is quite common and even expected, and Sourcegraph should for the most part continue working even if it exhausts its rate limits. It becomes a critical issue only if the rate limit is exhausted immediately after a rate limit reset. We can detect this by checking to see if the rate limit is below the threshold for ~most of a window. This change addresses both issues with updates to the alerts and NextSteps.
2022-06-07 15:18:48 +00:00
- <span class="badge badge-warning">warning</span> repo-updater: less than 5 remaining calls to GitHub search API before hitting the rate limit
Instrument GitHub and GitLab rate limit remaining per code host connection (#16750) Instrument rate limit remaining per code host connection for GitHub. We measure the remaining limit for their rest, graphql and search APIs separately. Instrument rate limit remaining per code host connection for GitLab. We only measure the remaining limit for their rest API as they only have one limit. Panels were added to repo-updater with alerts when the rate limit is nearly depleted.
2020-12-15 14:45:44 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
Instrument GitHub and GitLab rate limit remaining per code host connection (#16750) Instrument rate limit remaining per code host connection for GitHub. We measure the remaining limit for their rest, graphql and search APIs separately. Instrument rate limit remaining per code host connection for GitLab. We only measure the remaining limit for their rest API as they only have one limit. Panels were added to repo-updater with alerts when the rate limit is nearly depleted.
2020-12-15 14:45:44 +00:00
monitoring(repo-updater): add duration on critical rate limit alerts, update next steps (#36683) The existing advice is not great, because restarting the pod doesn't guarantee a new public IP, and it's possible customers are using a public gateway (e.g. Cloud NAT) with VM without a public IP address. It's also noted that this alert comes up quite frequently - I think on larger instances running up against the limit is quite common and even expected, and Sourcegraph should for the most part continue working even if it exhausts its rate limits. It becomes a critical issue only if the rate limit is exhausted immediately after a rate limit reset. We can detect this by checking to see if the rate limit is below the threshold for ~most of a window. This change addresses both issues with updates to the alerts and NextSteps.
2022-06-07 15:18:48 +00:00
- Consider creating a new token for the indicated resource (the `name` label for series below the threshold in the dashboard) under a dedicated machine user to reduce rate limit pressure.
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#repo-updater-github-search-rate-limit-remaining).
Instrument GitHub and GitLab rate limit remaining per code host connection (#16750) Instrument rate limit remaining per code host connection for GitHub. We measure the remaining limit for their rest, graphql and search APIs separately. Instrument rate limit remaining per code host connection for GitLab. We only measure the remaining limit for their rest API as they only have one limit. Panels were added to repo-updater with alerts when the rate limit is nearly depleted.
2020-12-15 14:45:44 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
Remove code host rate limit critical alerts (#39818)
2022-08-02 14:15:53 +00:00
"warning_repo-updater_github_search_rate_limit_remaining"
Instrument GitHub and GitLab rate limit remaining per code host connection (#16750) Instrument rate limit remaining per code host connection for GitHub. We measure the remaining limit for their rest, graphql and search APIs separately. Instrument rate limit remaining per code host connection for GitLab. We only measure the remaining limit for their rest API as they only have one limit. Panels were added to repo-updater with alerts when the rate limit is nearly depleted.
2020-12-15 14:45:44 +00:00
]
```
Update o11y ownership from IAM/repo-mgmt to Source (#54368) Does what it says in the title. ## Test plan - Ran `sg generate`
2023-06-28 13:49:16 +00:00
<sub>*Managed by the [Sourcegraph Source team](https://handbook.sourcegraph.com/departments/engineering/teams/source).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
monitoring: extract into a submodule (#45786) This change extracts `monitoring` into a submodule for import in `sourcegraph/controller` (https://github.com/sourcegraph/controller/pull/195) so that we can generate dashboards for Cloud instances. These steps were required: 1. Initialize a `go.mod` in `monitoring` 2. Extract `dev/sg/internal/cliutil` into `lib` to avoid illegal imports from `monitoring` 3. Add local replaces to both `sourcegraph/sourcegraph` and `monitoring` 4. `go mod tidy` on all submodules 5. Update `go generate ./monitoring` commands to use `sg`, since the `go generate` command no longer works 6. Update `grafana/build.sh`, `prometheus/build.sh` to build the submodule 7. Amend linters to check for multiple `go.mod` files and ban imports of `github.com/sourcegraph/sourcegraph` 8. Update `sg generate go` to run in directories rather than from root The only caveat is that if you use VS code, you will now need to open `monitoring` in a separate workspace or similar, like with `lib`. Co-authored-by: Joe Chen <joe@sourcegraph.com>
2022-12-19 17:49:25 +00:00
Generated query for warning alert: `min((max by (name) (src_github_rate_limit_remaining_v2{resource="search"})) <= 5)`
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
</details>
Instrument GitHub and GitLab rate limit remaining per code host connection (#16750) Instrument rate limit remaining per code host connection for GitHub. We measure the remaining limit for their rest, graphql and search APIs separately. Instrument rate limit remaining per code host connection for GitLab. We only measure the remaining limit for their rest API as they only have one limit. Panels were added to repo-updater with alerts when the rate limit is nearly depleted.
2020-12-15 14:45:44 +00:00
<br />
## repo-updater: gitlab_rest_rate_limit_remaining
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
<p class="subtitle">remaining calls to GitLab rest API before hitting the rate limit</p>
Instrument GitHub and GitLab rate limit remaining per code host connection (#16750) Instrument rate limit remaining per code host connection for GitHub. We measure the remaining limit for their rest, graphql and search APIs separately. Instrument rate limit remaining per code host connection for GitLab. We only measure the remaining limit for their rest API as they only have one limit. Panels were added to repo-updater with alerts when the rate limit is nearly depleted.
2020-12-15 14:45:44 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
Instrument GitHub and GitLab rate limit remaining per code host connection (#16750) Instrument rate limit remaining per code host connection for GitHub. We measure the remaining limit for their rest, graphql and search APIs separately. Instrument rate limit remaining per code host connection for GitLab. We only measure the remaining limit for their rest API as they only have one limit. Panels were added to repo-updater with alerts when the rate limit is nearly depleted.
2020-12-15 14:45:44 +00:00
monitoring: dashboards reference, refresh observability docs (#16939) - New monitoring definition field, 'Interpretation' - New generated dashboards reference - Grafana panels now link to alert solutions and dashboards reference - Refreshed observability docs Co-authored-by: uwedeportivo <534011+uwedeportivo@users.noreply.github.com>
2020-12-23 02:36:41 +00:00
- <span class="badge badge-critical">critical</span> repo-updater: less than 30 remaining calls to GitLab rest API before hitting the rate limit
Instrument GitHub and GitLab rate limit remaining per code host connection (#16750) Instrument rate limit remaining per code host connection for GitHub. We measure the remaining limit for their rest, graphql and search APIs separately. Instrument rate limit remaining per code host connection for GitLab. We only measure the remaining limit for their rest API as they only have one limit. Panels were added to repo-updater with alerts when the rate limit is nearly depleted.
2020-12-15 14:45:44 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
Instrument GitHub and GitLab rate limit remaining per code host connection (#16750) Instrument rate limit remaining per code host connection for GitHub. We measure the remaining limit for their rest, graphql and search APIs separately. Instrument rate limit remaining per code host connection for GitLab. We only measure the remaining limit for their rest API as they only have one limit. Panels were added to repo-updater with alerts when the rate limit is nearly depleted.
2020-12-15 14:45:44 +00:00
- Try restarting the pod to get a different public IP.
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#repo-updater-gitlab-rest-rate-limit-remaining).
Instrument GitHub and GitLab rate limit remaining per code host connection (#16750) Instrument rate limit remaining per code host connection for GitHub. We measure the remaining limit for their rest, graphql and search APIs separately. Instrument rate limit remaining per code host connection for GitLab. We only measure the remaining limit for their rest API as they only have one limit. Panels were added to repo-updater with alerts when the rate limit is nearly depleted.
2020-12-15 14:45:44 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"critical_repo-updater_gitlab_rest_rate_limit_remaining"
]
```
Update o11y ownership from IAM/repo-mgmt to Source (#54368) Does what it says in the title. ## Test plan - Ran `sg generate`
2023-06-28 13:49:16 +00:00
<sub>*Managed by the [Sourcegraph Source team](https://handbook.sourcegraph.com/departments/engineering/teams/source).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
monitoring: extract into a submodule (#45786) This change extracts `monitoring` into a submodule for import in `sourcegraph/controller` (https://github.com/sourcegraph/controller/pull/195) so that we can generate dashboards for Cloud instances. These steps were required: 1. Initialize a `go.mod` in `monitoring` 2. Extract `dev/sg/internal/cliutil` into `lib` to avoid illegal imports from `monitoring` 3. Add local replaces to both `sourcegraph/sourcegraph` and `monitoring` 4. `go mod tidy` on all submodules 5. Update `go generate ./monitoring` commands to use `sg`, since the `go generate` command no longer works 6. Update `grafana/build.sh`, `prometheus/build.sh` to build the submodule 7. Amend linters to check for multiple `go.mod` files and ban imports of `github.com/sourcegraph/sourcegraph` 8. Update `sg generate go` to run in directories rather than from root The only caveat is that if you use VS code, you will now need to open `monitoring` in a separate workspace or similar, like with `lib`. Co-authored-by: Joe Chen <joe@sourcegraph.com>
2022-12-19 17:49:25 +00:00
Generated query for critical alert: `min((max by (name) (src_gitlab_rate_limit_remaining{resource="rest"})) <= 30)`
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
</details>
Instrument GitHub and GitLab rate limit remaining per code host connection (#16750) Instrument rate limit remaining per code host connection for GitHub. We measure the remaining limit for their rest, graphql and search APIs separately. Instrument rate limit remaining per code host connection for GitLab. We only measure the remaining limit for their rest API as they only have one limit. Panels were added to repo-updater with alerts when the rate limit is nearly depleted.
2020-12-15 14:45:44 +00:00
<br />
conf: add metric and associated alert if clients fail to update site configuration within 5 minutes (#57682)
2023-10-18 23:53:55 +00:00
## repo-updater: repo_updater_site_configuration_duration_since_last_successful_update_by_instance
<p class="subtitle">maximum duration since last successful site configuration update (all "repo_updater" instances)</p>
**Descriptions**
- <span class="badge badge-critical">critical</span> repo-updater: 300s+ maximum duration since last successful site configuration update (all "repo_updater" instances)
**Next steps**
- This indicates that one or more "repo_updater" instances have not successfully updated the site configuration in over 5 minutes. This could be due to networking issues between services or problems with the site configuration service itself.
- Check for relevant errors in the "repo_updater" logs, as well as frontend`s logs.
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#repo-updater-repo-updater-site-configuration-duration-since-last-successful-update-by-instance).
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"critical_repo-updater_repo_updater_site_configuration_duration_since_last_successful_update_by_instance"
]
```
monitoring: test owners for valid Opsgenie teams and handbook pages (#59251) In INC-264 it seems that certain alerts - such as [zoekt: less than 90% percentage pods available for 10m0s](https://opsg.in/a/i/sourcegraph/178a626f-0f28-4295-bee9-84da988bb473-1703759057681) - don't seem to end up going anywhere because the ObservableOwner is defunct. This change adds _opt-in_ testing to report: 1. How many owners have valid Opsgenie teams 2. How many owners have valid handbook pages In addition, we collect ObservableOwners that pass the test and use it to generate configuration for `site.json` in Sourcegraph.com: https://github.com/sourcegraph/deploy-sourcegraph-cloud/pull/18338 - this helps ensure the list is valid and not deceptively high-coverage. The results are not great, but **enforcing** that owners are valid isn't currently in scope: ``` 6/10 ObservableOwners do not have valid Opsgenie teams 3/10 ObservableOwners do not point to valid handbook pages ``` I also removed some defunct/unused functionality/owners. ## Test plan To run these tests: ``` export OPSGENIE_API_KEY="..." go test -timeout 30s github.com/sourcegraph/sourcegraph/monitoring/monitoring -update -online ```
2023-12-29 22:07:35 +00:00
<sub>*Managed by the [Sourcegraph Infrastructure Org team](https://handbook.sourcegraph.com/departments/engineering/infrastructure).*</sub>
conf: add metric and associated alert if clients fail to update site configuration within 5 minutes (#57682)
2023-10-18 23:53:55 +00:00
<details>
<summary>Technical details</summary>
monitoring: fix alert definition for site configuration by adding scrape job label (#59687) We discovered recently that the definition for the alert that fires if the site configuration hasn't been fetched within 5 minutes strips out the regex that targets individual services (since it uses a grafana variable). This means that every instance of this alert will fire if any individual service trips over this threshold. This PR fixes the issue by adding a new `job` filter for this alert that targets only the services that have that Prometheus scrape target name. This works around the previous issue by using a fixed value for the `job` value instead of a dynamic grafana value. The value of the job filter generally looks like `job=~.*$container_name` (following the strategy from https://sourcegraph.com/github.com/sourcegraph/sourcegraph@9a780f2e694238b5326e3e121d6a1828463001b9/-/blob/monitoring/monitoring/monitoring.go?L161 ) unless I noticed that there was different logic in the existing dashboard for the services. Ex: - `frontend`: already used `job=~"(sourcegraph-)?frontend"` for some metrics, so I used it again here - `worker`: `already used `job=~"^worker.*"` in some metrics, so I used it again and standarized the other existing panels to use the same shared variable ## Test plan I eyeballed the generated alert.md and dashboards.md to verify that my changes looked correct (that is, my refactors resulted in either no diff, or that the diff I generated still looked like valid regex).
2024-01-17 23:19:54 +00:00
Generated query for critical alert: `max((max(max_over_time(src_conf_client_time_since_last_successful_update_seconds{job=~".*repo-updater"}[1m]))) >= 300)`
conf: add metric and associated alert if clients fail to update site configuration within 5 minutes (#57682)
2023-10-18 23:53:55 +00:00
</details>
<br />
Add alerts for mean db blocked seconds (#22822) Warn if the average database block time is over 5ms, alert if it exceeds 10ms
2021-07-16 01:21:24 +00:00
## repo-updater: mean_blocked_seconds_per_conn_request
<p class="subtitle">mean blocked seconds per conn request</p>
**Descriptions**
monitoring: relax mean_blocked_seconds_per_conn_request alerts (#59507) https://github.com/sourcegraph/sourcegraph/pull/59284 dramatically reduced the `mean_blocked_seconds_per_conn_request` issues we've been seeing, but overall delays are still higher, even with generally healthy Cloud SQL resource utilization. <img width="1630" alt="image" src="https://github.com/sourcegraph/sourcegraph/assets/23356519/91615471-5187-4d15-83e7-5cc94595303c"> Spot-checking the spikes in load in Cloud SQL, it seems that there is a variety of causes for each spike (analytics workloads, Cody Gateway syncs, code intel workloads, gitserver things, `ListSourcegraphDotComIndexableRepos` etc) so I'm chalking this up to "expected". Since this alert is seen firing on a Cloud instance, let's just relax it for now so that it only fires a critical alert on very significant delays.
2024-01-11 01:14:28 +00:00
- <span class="badge badge-warning">warning</span> repo-updater: 0.1s+ mean blocked seconds per conn request for 10m0s
- <span class="badge badge-critical">critical</span> repo-updater: 0.5s+ mean blocked seconds per conn request for 10m0s
Add alerts for mean db blocked seconds (#22822) Warn if the average database block time is over 5ms, alert if it exceeds 10ms
2021-07-16 01:21:24 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
Add alerts for mean db blocked seconds (#22822) Warn if the average database block time is over 5ms, alert if it exceeds 10ms
2021-07-16 01:21:24 +00:00
Add possible solutions and make docs discoverable (#22945) Add possible solutions and make docs discoverable
2021-07-30 00:30:31 +00:00
- Increase SRC_PGSQL_MAX_OPEN together with giving more memory to the database if needed
Docs: update links to point to new site (#60381) We have a number of docs links in the product that point to the old doc site. Method: - Search the repo for `docs.sourcegraph.com` - Exclude the `doc/` dir, all test fixtures, and `CHANGELOG.md` - For each, replace `docs.sourcegraph.com` with `sourcegraph.com/docs` - Navigate to the resulting URL ensuring it's not a dead link, updating the URL if necessary Many of the URLs updated are just comments, but since I'm doing a manual audit of each URL anyways, I felt it was worth it to update these while I was at it.
2024-02-13 00:23:47 +00:00
- Scale up Postgres memory/cpus - [see our scaling guide](https://sourcegraph.com/docs/admin/config/postgres-conf)
monitoring: relax mean_blocked_seconds_per_conn_request alerts (#59507) https://github.com/sourcegraph/sourcegraph/pull/59284 dramatically reduced the `mean_blocked_seconds_per_conn_request` issues we've been seeing, but overall delays are still higher, even with generally healthy Cloud SQL resource utilization. <img width="1630" alt="image" src="https://github.com/sourcegraph/sourcegraph/assets/23356519/91615471-5187-4d15-83e7-5cc94595303c"> Spot-checking the spikes in load in Cloud SQL, it seems that there is a variety of causes for each spike (analytics workloads, Cody Gateway syncs, code intel workloads, gitserver things, `ListSourcegraphDotComIndexableRepos` etc) so I'm chalking this up to "expected". Since this alert is seen firing on a Cloud instance, let's just relax it for now so that it only fires a critical alert on very significant delays.
2024-01-11 01:14:28 +00:00
- If using GCP Cloud SQL, check for high lock waits or CPU usage in query insights
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#repo-updater-mean-blocked-seconds-per-conn-request).
Add alerts for mean db blocked seconds (#22822) Warn if the average database block time is over 5ms, alert if it exceeds 10ms
2021-07-16 01:21:24 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"warning_repo-updater_mean_blocked_seconds_per_conn_request",
"critical_repo-updater_mean_blocked_seconds_per_conn_request"
]
```
monitoring: assign obvious owners to DatabaseConnectionsMonitoringGroup (#58474) updates `DatabaseConnectionsMonitoringGroup` to accept an owner - before it was just hardcoded to `ObservableOwnerDevOps`, which is not very helpful. This assigns some of the obvious service owners: 1. Source: gitserver, repo-updater 2. Cody: embeddings (but should eventually be @sourcegraph/search-platform, along with all embeddings alerts: https://github.com/sourcegraph/sourcegraph/pull/58474#issuecomment-1821505062) Source is an active owner based on [thread](https://sourcegraph.slack.com/archives/C0652SSUA20/p1700592165408089?thread_ts=1700549423.860019&cid=C0652SSUA20), and Cody is a fairly recent addition so hopefully it's valid. I'm not sure the Search one is still up-to-date, so I didn't change some of the obvious search services - for now, these still point to DevOps as they did before. If it becomes problematic we can revisit later.
2023-11-22 01:06:53 +00:00
<sub>*Managed by the [Sourcegraph Source team](https://handbook.sourcegraph.com/departments/engineering/teams/source).*</sub>
Add alerts for mean db blocked seconds (#22822) Warn if the average database block time is over 5ms, alert if it exceeds 10ms
2021-07-16 01:21:24 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
monitoring: relax mean_blocked_seconds_per_conn_request alerts (#59507) https://github.com/sourcegraph/sourcegraph/pull/59284 dramatically reduced the `mean_blocked_seconds_per_conn_request` issues we've been seeing, but overall delays are still higher, even with generally healthy Cloud SQL resource utilization. <img width="1630" alt="image" src="https://github.com/sourcegraph/sourcegraph/assets/23356519/91615471-5187-4d15-83e7-5cc94595303c"> Spot-checking the spikes in load in Cloud SQL, it seems that there is a variety of causes for each spike (analytics workloads, Cody Gateway syncs, code intel workloads, gitserver things, `ListSourcegraphDotComIndexableRepos` etc) so I'm chalking this up to "expected". Since this alert is seen firing on a Cloud instance, let's just relax it for now so that it only fires a critical alert on very significant delays.
2024-01-11 01:14:28 +00:00
Generated query for warning alert: `max((sum by (app_name, db_name) (increase(src_pgsql_conns_blocked_seconds{app_name="repo-updater"}[5m])) / sum by (app_name, db_name) (increase(src_pgsql_conns_waited_for{app_name="repo-updater"}[5m]))) >= 0.1)`
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
monitoring: relax mean_blocked_seconds_per_conn_request alerts (#59507) https://github.com/sourcegraph/sourcegraph/pull/59284 dramatically reduced the `mean_blocked_seconds_per_conn_request` issues we've been seeing, but overall delays are still higher, even with generally healthy Cloud SQL resource utilization. <img width="1630" alt="image" src="https://github.com/sourcegraph/sourcegraph/assets/23356519/91615471-5187-4d15-83e7-5cc94595303c"> Spot-checking the spikes in load in Cloud SQL, it seems that there is a variety of causes for each spike (analytics workloads, Cody Gateway syncs, code intel workloads, gitserver things, `ListSourcegraphDotComIndexableRepos` etc) so I'm chalking this up to "expected". Since this alert is seen firing on a Cloud instance, let's just relax it for now so that it only fires a critical alert on very significant delays.
2024-01-11 01:14:28 +00:00
Generated query for critical alert: `max((sum by (app_name, db_name) (increase(src_pgsql_conns_blocked_seconds{app_name="repo-updater"}[5m])) / sum by (app_name, db_name) (increase(src_pgsql_conns_waited_for{app_name="repo-updater"}[5m]))) >= 0.5)`
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
</details>
Add alerts for mean db blocked seconds (#22822) Warn if the average database block time is over 5ms, alert if it exceeds 10ms
2021-07-16 01:21:24 +00:00
<br />
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
## repo-updater: container_cpu_usage
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
<p class="subtitle">container cpu usage total (1m average) across all cores by instance</p>
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
monitoring: dashboards reference, refresh observability docs (#16939) - New monitoring definition field, 'Interpretation' - New generated dashboards reference - Grafana panels now link to alert solutions and dashboards reference - Refreshed observability docs Co-authored-by: uwedeportivo <534011+uwedeportivo@users.noreply.github.com>
2020-12-23 02:36:41 +00:00
- <span class="badge badge-warning">warning</span> repo-updater: 99%+ container cpu usage total (1m average) across all cores by instance
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
monitoring: migrate out-of-band alerts to the generator (#12391) Migrates all out-of-band alert rules defined by deploy-sourcegraph to the generator, such that each alert now has a panel as part of a service dashboard like all our other alerts, with a few changes and additions.
2020-07-28 02:17:56 +00:00
- **Kubernetes:** Consider increasing CPU limits in the the relevant `Deployment.yaml`.
- **Docker Compose:** Consider increasing `cpus:` of the repo-updater container in `docker-compose.yml`.
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#repo-updater-container-cpu-usage).
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
```json
"observability.silenceAlerts": [
"warning_repo-updater_container_cpu_usage"
]
```
Update o11y ownership from IAM/repo-mgmt to Source (#54368) Does what it says in the title. ## Test plan - Ran `sg generate`
2023-06-28 13:49:16 +00:00
<sub>*Managed by the [Sourcegraph Source team](https://handbook.sourcegraph.com/departments/engineering/teams/source).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
Generated query for warning alert: `max((cadvisor_container_cpu_usage_percentage_total{name=~"^repo-updater.*"}) >= 99)`
</details>
monitoring: improve solutions formatting, add team information (#14878)
2020-10-20 05:11:20 +00:00
<br />
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
## repo-updater: container_memory_usage
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
<p class="subtitle">container memory usage by instance</p>
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
monitoring: At 10 minute duration check for repo-updater mem (#22461)
2021-07-01 08:28:06 +00:00
- <span class="badge badge-critical">critical</span> repo-updater: 90%+ container memory usage by instance for 10m0s
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
- **Kubernetes:** Consider increasing memory limit in relevant `Deployment.yaml`.
- **Docker Compose:** Consider increasing `memory:` of repo-updater container in `docker-compose.yml`.
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#repo-updater-container-memory-usage).
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
monitoring: Tune some repo-updater alerts (#16664) * monitoring: repo-updater critical alert at 90% memory utilisation This commit makes it so we alert on 90% memory utilisation of repo-updater. * fixup! Alert on pod availability * fixup! Generate * Update monitoring/definitions/repo_updater.go Co-authored-by: Robert Lin <robert@bobheadxi.dev> * fixup! Generate again Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2020-12-12 11:52:46 +00:00
"critical_repo-updater_container_memory_usage"
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
]
```
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
Update o11y ownership from IAM/repo-mgmt to Source (#54368) Does what it says in the title. ## Test plan - Ran `sg generate`
2023-06-28 13:49:16 +00:00
<sub>*Managed by the [Sourcegraph Source team](https://handbook.sourcegraph.com/departments/engineering/teams/source).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
Generated query for critical alert: `max((cadvisor_container_memory_usage_percentage_total{name=~"^repo-updater.*"}) >= 90)`
</details>
monitoring: improve solutions formatting, add team information (#14878)
2020-10-20 05:11:20 +00:00
<br />
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: improve long-term provisioning alerts aggregation (#12778)
2020-08-08 06:44:33 +00:00
## repo-updater: provisioning_container_cpu_usage_long_term
monitoring: support multi-bound alerts, add lower-bound warnings for containers (#11082)
2020-06-16 07:09:25 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
<p class="subtitle">container cpu usage total (90th percentile over 1d) across all cores by instance</p>
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
monitoring: support multi-bound alerts, add lower-bound warnings for containers (#11082)
2020-06-16 07:09:25 +00:00
monitoring: dashboards reference, refresh observability docs (#16939) - New monitoring definition field, 'Interpretation' - New generated dashboards reference - Grafana panels now link to alert solutions and dashboards reference - Refreshed observability docs Co-authored-by: uwedeportivo <534011+uwedeportivo@users.noreply.github.com>
2020-12-23 02:36:41 +00:00
- <span class="badge badge-warning">warning</span> repo-updater: 80%+ container cpu usage total (90th percentile over 1d) across all cores by instance for 336h0m0s
monitoring: support multi-bound alerts, add lower-bound warnings for containers (#11082)
2020-06-16 07:09:25 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: support multi-bound alerts, add lower-bound warnings for containers (#11082)
2020-06-16 07:09:25 +00:00
Remove container_*_usage_7d_low alerts (#16164)
2020-11-25 18:19:33 +00:00
- **Kubernetes:** Consider increasing CPU limits in the `Deployment.yaml` for the repo-updater service.
- **Docker Compose:** Consider increasing `cpus:` of the repo-updater container in `docker-compose.yml`.
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#repo-updater-provisioning-container-cpu-usage-long-term).
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
monitoring: support multi-bound alerts, add lower-bound warnings for containers (#11082)
2020-06-16 07:09:25 +00:00
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
```json
"observability.silenceAlerts": [
monitoring: improve long-term provisioning alerts aggregation (#12778)
2020-08-08 06:44:33 +00:00
"warning_repo-updater_provisioning_container_cpu_usage_long_term"
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
]
```
Update o11y ownership from IAM/repo-mgmt to Source (#54368) Does what it says in the title. ## Test plan - Ran `sg generate`
2023-06-28 13:49:16 +00:00
<sub>*Managed by the [Sourcegraph Source team](https://handbook.sourcegraph.com/departments/engineering/teams/source).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
Generated query for warning alert: `max((quantile_over_time(0.9, cadvisor_container_cpu_usage_percentage_total{name=~"^repo-updater.*"}[1d])) >= 80)`
</details>
monitoring: improve solutions formatting, add team information (#14878)
2020-10-20 05:11:20 +00:00
<br />
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: improve long-term provisioning alerts aggregation (#12778)
2020-08-08 06:44:33 +00:00
## repo-updater: provisioning_container_memory_usage_long_term
monitoring: support multi-bound alerts, add lower-bound warnings for containers (#11082)
2020-06-16 07:09:25 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
<p class="subtitle">container memory usage (1d maximum) by instance</p>
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
monitoring: support multi-bound alerts, add lower-bound warnings for containers (#11082)
2020-06-16 07:09:25 +00:00
monitoring: dashboards reference, refresh observability docs (#16939) - New monitoring definition field, 'Interpretation' - New generated dashboards reference - Grafana panels now link to alert solutions and dashboards reference - Refreshed observability docs Co-authored-by: uwedeportivo <534011+uwedeportivo@users.noreply.github.com>
2020-12-23 02:36:41 +00:00
- <span class="badge badge-warning">warning</span> repo-updater: 80%+ container memory usage (1d maximum) by instance for 336h0m0s
monitoring: support multi-bound alerts, add lower-bound warnings for containers (#11082)
2020-06-16 07:09:25 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: support multi-bound alerts, add lower-bound warnings for containers (#11082)
2020-06-16 07:09:25 +00:00
Remove container_*_usage_7d_low alerts (#16164)
2020-11-25 18:19:33 +00:00
- **Kubernetes:** Consider increasing memory limits in the `Deployment.yaml` for the repo-updater service.
- **Docker Compose:** Consider increasing `memory:` of the repo-updater container in `docker-compose.yml`.
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#repo-updater-provisioning-container-memory-usage-long-term).
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
monitoring: improve long-term provisioning alerts aggregation (#12778)
2020-08-08 06:44:33 +00:00
"warning_repo-updater_provisioning_container_memory_usage_long_term"
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
]
```
monitoring: support multi-bound alerts, add lower-bound warnings for containers (#11082)
2020-06-16 07:09:25 +00:00
Update o11y ownership from IAM/repo-mgmt to Source (#54368) Does what it says in the title. ## Test plan - Ran `sg generate`
2023-06-28 13:49:16 +00:00
<sub>*Managed by the [Sourcegraph Source team](https://handbook.sourcegraph.com/departments/engineering/teams/source).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
Generated query for warning alert: `max((max_over_time(cadvisor_container_memory_usage_percentage_total{name=~"^repo-updater.*"}[1d])) >= 80)`
</details>
monitoring: improve solutions formatting, add team information (#14878)
2020-10-20 05:11:20 +00:00
<br />
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: improve long-term provisioning alerts aggregation (#12778)
2020-08-08 06:44:33 +00:00
## repo-updater: provisioning_container_cpu_usage_short_term
monitoring: support multi-bound alerts, add lower-bound warnings for containers (#11082)
2020-06-16 07:09:25 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
<p class="subtitle">container cpu usage total (5m maximum) across all cores by instance</p>
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
monitoring: support multi-bound alerts, add lower-bound warnings for containers (#11082)
2020-06-16 07:09:25 +00:00
monitoring: dashboards reference, refresh observability docs (#16939) - New monitoring definition field, 'Interpretation' - New generated dashboards reference - Grafana panels now link to alert solutions and dashboards reference - Refreshed observability docs Co-authored-by: uwedeportivo <534011+uwedeportivo@users.noreply.github.com>
2020-12-23 02:36:41 +00:00
- <span class="badge badge-warning">warning</span> repo-updater: 90%+ container cpu usage total (5m maximum) across all cores by instance for 30m0s
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
- **Kubernetes:** Consider increasing CPU limits in the the relevant `Deployment.yaml`.
- **Docker Compose:** Consider increasing `cpus:` of the repo-updater container in `docker-compose.yml`.
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#repo-updater-provisioning-container-cpu-usage-short-term).
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
```json
"observability.silenceAlerts": [
monitoring: improve long-term provisioning alerts aggregation (#12778)
2020-08-08 06:44:33 +00:00
"warning_repo-updater_provisioning_container_cpu_usage_short_term"
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
]
```
Update o11y ownership from IAM/repo-mgmt to Source (#54368) Does what it says in the title. ## Test plan - Ran `sg generate`
2023-06-28 13:49:16 +00:00
<sub>*Managed by the [Sourcegraph Source team](https://handbook.sourcegraph.com/departments/engineering/teams/source).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
Generated query for warning alert: `max((max_over_time(cadvisor_container_cpu_usage_percentage_total{name=~"^repo-updater.*"}[5m])) >= 90)`
</details>
monitoring: improve solutions formatting, add team information (#14878)
2020-10-20 05:11:20 +00:00
<br />
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: improve long-term provisioning alerts aggregation (#12778)
2020-08-08 06:44:33 +00:00
## repo-updater: provisioning_container_memory_usage_short_term
monitoring: support multi-bound alerts, add lower-bound warnings for containers (#11082)
2020-06-16 07:09:25 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
<p class="subtitle">container memory usage (5m maximum) by instance</p>
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
monitoring: support multi-bound alerts, add lower-bound warnings for containers (#11082)
2020-06-16 07:09:25 +00:00
monitoring: dashboards reference, refresh observability docs (#16939) - New monitoring definition field, 'Interpretation' - New generated dashboards reference - Grafana panels now link to alert solutions and dashboards reference - Refreshed observability docs Co-authored-by: uwedeportivo <534011+uwedeportivo@users.noreply.github.com>
2020-12-23 02:36:41 +00:00
- <span class="badge badge-warning">warning</span> repo-updater: 90%+ container memory usage (5m maximum) by instance
monitoring: support multi-bound alerts, add lower-bound warnings for containers (#11082)
2020-06-16 07:09:25 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: support multi-bound alerts, add lower-bound warnings for containers (#11082)
2020-06-16 07:09:25 +00:00
- **Kubernetes:** Consider increasing memory limit in relevant `Deployment.yaml`.
- **Docker Compose:** Consider increasing `memory:` of repo-updater container in `docker-compose.yml`.
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#repo-updater-provisioning-container-memory-usage-short-term).
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
monitoring: improve long-term provisioning alerts aggregation (#12778)
2020-08-08 06:44:33 +00:00
"warning_repo-updater_provisioning_container_memory_usage_short_term"
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
]
```
Update o11y ownership from IAM/repo-mgmt to Source (#54368) Does what it says in the title. ## Test plan - Ran `sg generate`
2023-06-28 13:49:16 +00:00
<sub>*Managed by the [Sourcegraph Source team](https://handbook.sourcegraph.com/departments/engineering/teams/source).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
Generated query for warning alert: `max((max_over_time(cadvisor_container_memory_usage_percentage_total{name=~"^repo-updater.*"}[5m])) >= 90)`
</details>
monitoring: improve solutions formatting, add team information (#14878)
2020-10-20 05:11:20 +00:00
<br />
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: improve containers metrics (#34808) part of - https://github.com/sourcegraph/sourcegraph/issues/33437 - https://github.com/sourcegraph/sourcegraph/issues/33438 This PR adds the following - A shared observable metric to indicate how many times a container (or child processes) has been oom kill - It's added to provisioning indicators since it's an indicator of underprovisioned resources - A new dashboard that shows resources usage of all containers at a glance - It contains two groups, one for all containers and one for containers that may have potential scaling issues - the first group is extremely noisy so it's hidden by default - the second group provides a more focused experience for site-admin to quickly find problematic resources <!-- > ~it looks like the alert table will be rendered regardless, which is pretty annoying (I guess the implementation of the monitoring stack is based on the assumption that one service/container will have its dashboard)~ __Questions & Notes__ - ~Should I make the new dashboard filterable by container name? like I only want to see the graph of `frontend`. Is that something that's valuable?~ - ~Any other metrics you all would like to see?~ - graph for total available resources on the VM compared to usage from https://github.com/sourcegraph/sourcegraph/issues/33437 is not really doable since we do not have `node-exporter` in anyone of our distro. We would have to tackle this from https://github.com/sourcegraph/customer/issues/806 --> ## Test plan connect to `demo`'s prom ```sh gcloud compute start-iap-tunnel default-$NEW_DEPLOYMENT-instance 9090 --local-host-port=localhost:4445 --zone us-central1-f --project $PROJECT_PREFIX-$CUSTOMER ``` or use dogfood ```sh kubectl port-forward svc/prometheus 4445:30090 ``` Update `dev/grafana/all/datasources.yaml` ```yml # Configuration for all non-Linux platforms. apiVersion: 1 datasources: - name: Prometheus type: prometheus access: proxy url: http://docker.for.mac.localhost:4445 # use the remote prometheus instance on demo instead of the local one isDefault: true editable: false - name: Jaeger type: jaeger access: proxy url: http://host.docker.internal:16686/-/debug/jaeger - name: Loki type: loki access: proxy url: http://host.docker.internal:3100 ``` Start monitoring stack ```sh sg run grafana ``` see screenshot > the first group is hidden by default ![CleanShot 2022-05-04 at 10 48 26](https://user-images.githubusercontent.com/8373004/166748470-22460552-6e51-4b29-a26b-4c5adbe1c404.png) ![CleanShot 2022-05-04 at 10 48 03](https://user-images.githubusercontent.com/8373004/166748396-1d410c35-e5fe-45e2-83bd-418d845413cb.png) ![CleanShot 2022-05-04 at 11 25 13](https://user-images.githubusercontent.com/8373004/166801266-21e51b68-65fd-4cb7-97de-83520e31e498.png)
2022-05-09 23:56:47 +00:00
## repo-updater: container_oomkill_events_total
<p class="subtitle">container OOMKILL events total by instance</p>
**Descriptions**
- <span class="badge badge-warning">warning</span> repo-updater: 1+ container OOMKILL events total by instance
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: improve containers metrics (#34808) part of - https://github.com/sourcegraph/sourcegraph/issues/33437 - https://github.com/sourcegraph/sourcegraph/issues/33438 This PR adds the following - A shared observable metric to indicate how many times a container (or child processes) has been oom kill - It's added to provisioning indicators since it's an indicator of underprovisioned resources - A new dashboard that shows resources usage of all containers at a glance - It contains two groups, one for all containers and one for containers that may have potential scaling issues - the first group is extremely noisy so it's hidden by default - the second group provides a more focused experience for site-admin to quickly find problematic resources <!-- > ~it looks like the alert table will be rendered regardless, which is pretty annoying (I guess the implementation of the monitoring stack is based on the assumption that one service/container will have its dashboard)~ __Questions & Notes__ - ~Should I make the new dashboard filterable by container name? like I only want to see the graph of `frontend`. Is that something that's valuable?~ - ~Any other metrics you all would like to see?~ - graph for total available resources on the VM compared to usage from https://github.com/sourcegraph/sourcegraph/issues/33437 is not really doable since we do not have `node-exporter` in anyone of our distro. We would have to tackle this from https://github.com/sourcegraph/customer/issues/806 --> ## Test plan connect to `demo`'s prom ```sh gcloud compute start-iap-tunnel default-$NEW_DEPLOYMENT-instance 9090 --local-host-port=localhost:4445 --zone us-central1-f --project $PROJECT_PREFIX-$CUSTOMER ``` or use dogfood ```sh kubectl port-forward svc/prometheus 4445:30090 ``` Update `dev/grafana/all/datasources.yaml` ```yml # Configuration for all non-Linux platforms. apiVersion: 1 datasources: - name: Prometheus type: prometheus access: proxy url: http://docker.for.mac.localhost:4445 # use the remote prometheus instance on demo instead of the local one isDefault: true editable: false - name: Jaeger type: jaeger access: proxy url: http://host.docker.internal:16686/-/debug/jaeger - name: Loki type: loki access: proxy url: http://host.docker.internal:3100 ``` Start monitoring stack ```sh sg run grafana ``` see screenshot > the first group is hidden by default ![CleanShot 2022-05-04 at 10 48 26](https://user-images.githubusercontent.com/8373004/166748470-22460552-6e51-4b29-a26b-4c5adbe1c404.png) ![CleanShot 2022-05-04 at 10 48 03](https://user-images.githubusercontent.com/8373004/166748396-1d410c35-e5fe-45e2-83bd-418d845413cb.png) ![CleanShot 2022-05-04 at 11 25 13](https://user-images.githubusercontent.com/8373004/166801266-21e51b68-65fd-4cb7-97de-83520e31e498.png)
2022-05-09 23:56:47 +00:00
- **Kubernetes:** Consider increasing memory limit in relevant `Deployment.yaml`.
- **Docker Compose:** Consider increasing `memory:` of repo-updater container in `docker-compose.yml`.
- More help interpreting this metric is available in the [dashboards reference](./dashboards.md#repo-updater-container-oomkill-events-total).
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"warning_repo-updater_container_oomkill_events_total"
]
```
Update o11y ownership from IAM/repo-mgmt to Source (#54368) Does what it says in the title. ## Test plan - Ran `sg generate`
2023-06-28 13:49:16 +00:00
<sub>*Managed by the [Sourcegraph Source team](https://handbook.sourcegraph.com/departments/engineering/teams/source).*</sub>
monitoring: improve containers metrics (#34808) part of - https://github.com/sourcegraph/sourcegraph/issues/33437 - https://github.com/sourcegraph/sourcegraph/issues/33438 This PR adds the following - A shared observable metric to indicate how many times a container (or child processes) has been oom kill - It's added to provisioning indicators since it's an indicator of underprovisioned resources - A new dashboard that shows resources usage of all containers at a glance - It contains two groups, one for all containers and one for containers that may have potential scaling issues - the first group is extremely noisy so it's hidden by default - the second group provides a more focused experience for site-admin to quickly find problematic resources <!-- > ~it looks like the alert table will be rendered regardless, which is pretty annoying (I guess the implementation of the monitoring stack is based on the assumption that one service/container will have its dashboard)~ __Questions & Notes__ - ~Should I make the new dashboard filterable by container name? like I only want to see the graph of `frontend`. Is that something that's valuable?~ - ~Any other metrics you all would like to see?~ - graph for total available resources on the VM compared to usage from https://github.com/sourcegraph/sourcegraph/issues/33437 is not really doable since we do not have `node-exporter` in anyone of our distro. We would have to tackle this from https://github.com/sourcegraph/customer/issues/806 --> ## Test plan connect to `demo`'s prom ```sh gcloud compute start-iap-tunnel default-$NEW_DEPLOYMENT-instance 9090 --local-host-port=localhost:4445 --zone us-central1-f --project $PROJECT_PREFIX-$CUSTOMER ``` or use dogfood ```sh kubectl port-forward svc/prometheus 4445:30090 ``` Update `dev/grafana/all/datasources.yaml` ```yml # Configuration for all non-Linux platforms. apiVersion: 1 datasources: - name: Prometheus type: prometheus access: proxy url: http://docker.for.mac.localhost:4445 # use the remote prometheus instance on demo instead of the local one isDefault: true editable: false - name: Jaeger type: jaeger access: proxy url: http://host.docker.internal:16686/-/debug/jaeger - name: Loki type: loki access: proxy url: http://host.docker.internal:3100 ``` Start monitoring stack ```sh sg run grafana ``` see screenshot > the first group is hidden by default ![CleanShot 2022-05-04 at 10 48 26](https://user-images.githubusercontent.com/8373004/166748470-22460552-6e51-4b29-a26b-4c5adbe1c404.png) ![CleanShot 2022-05-04 at 10 48 03](https://user-images.githubusercontent.com/8373004/166748396-1d410c35-e5fe-45e2-83bd-418d845413cb.png) ![CleanShot 2022-05-04 at 11 25 13](https://user-images.githubusercontent.com/8373004/166801266-21e51b68-65fd-4cb7-97de-83520e31e498.png)
2022-05-09 23:56:47 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
monitoring: extract into a submodule (#45786) This change extracts `monitoring` into a submodule for import in `sourcegraph/controller` (https://github.com/sourcegraph/controller/pull/195) so that we can generate dashboards for Cloud instances. These steps were required: 1. Initialize a `go.mod` in `monitoring` 2. Extract `dev/sg/internal/cliutil` into `lib` to avoid illegal imports from `monitoring` 3. Add local replaces to both `sourcegraph/sourcegraph` and `monitoring` 4. `go mod tidy` on all submodules 5. Update `go generate ./monitoring` commands to use `sg`, since the `go generate` command no longer works 6. Update `grafana/build.sh`, `prometheus/build.sh` to build the submodule 7. Amend linters to check for multiple `go.mod` files and ban imports of `github.com/sourcegraph/sourcegraph` 8. Update `sg generate go` to run in directories rather than from root The only caveat is that if you use VS code, you will now need to open `monitoring` in a separate workspace or similar, like with `lib`. Co-authored-by: Joe Chen <joe@sourcegraph.com>
2022-12-19 17:49:25 +00:00
Generated query for warning alert: `max((max by (name) (container_oom_events_total{name=~"^repo-updater.*"})) >= 1)`
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
</details>
monitoring: improve containers metrics (#34808) part of - https://github.com/sourcegraph/sourcegraph/issues/33437 - https://github.com/sourcegraph/sourcegraph/issues/33438 This PR adds the following - A shared observable metric to indicate how many times a container (or child processes) has been oom kill - It's added to provisioning indicators since it's an indicator of underprovisioned resources - A new dashboard that shows resources usage of all containers at a glance - It contains two groups, one for all containers and one for containers that may have potential scaling issues - the first group is extremely noisy so it's hidden by default - the second group provides a more focused experience for site-admin to quickly find problematic resources <!-- > ~it looks like the alert table will be rendered regardless, which is pretty annoying (I guess the implementation of the monitoring stack is based on the assumption that one service/container will have its dashboard)~ __Questions & Notes__ - ~Should I make the new dashboard filterable by container name? like I only want to see the graph of `frontend`. Is that something that's valuable?~ - ~Any other metrics you all would like to see?~ - graph for total available resources on the VM compared to usage from https://github.com/sourcegraph/sourcegraph/issues/33437 is not really doable since we do not have `node-exporter` in anyone of our distro. We would have to tackle this from https://github.com/sourcegraph/customer/issues/806 --> ## Test plan connect to `demo`'s prom ```sh gcloud compute start-iap-tunnel default-$NEW_DEPLOYMENT-instance 9090 --local-host-port=localhost:4445 --zone us-central1-f --project $PROJECT_PREFIX-$CUSTOMER ``` or use dogfood ```sh kubectl port-forward svc/prometheus 4445:30090 ``` Update `dev/grafana/all/datasources.yaml` ```yml # Configuration for all non-Linux platforms. apiVersion: 1 datasources: - name: Prometheus type: prometheus access: proxy url: http://docker.for.mac.localhost:4445 # use the remote prometheus instance on demo instead of the local one isDefault: true editable: false - name: Jaeger type: jaeger access: proxy url: http://host.docker.internal:16686/-/debug/jaeger - name: Loki type: loki access: proxy url: http://host.docker.internal:3100 ``` Start monitoring stack ```sh sg run grafana ``` see screenshot > the first group is hidden by default ![CleanShot 2022-05-04 at 10 48 26](https://user-images.githubusercontent.com/8373004/166748470-22460552-6e51-4b29-a26b-4c5adbe1c404.png) ![CleanShot 2022-05-04 at 10 48 03](https://user-images.githubusercontent.com/8373004/166748396-1d410c35-e5fe-45e2-83bd-418d845413cb.png) ![CleanShot 2022-05-04 at 11 25 13](https://user-images.githubusercontent.com/8373004/166801266-21e51b68-65fd-4cb7-97de-83520e31e498.png)
2022-05-09 23:56:47 +00:00
<br />
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
## repo-updater: go_goroutines
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
<p class="subtitle">maximum active goroutines</p>
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
monitoring: dashboards reference, refresh observability docs (#16939) - New monitoring definition field, 'Interpretation' - New generated dashboards reference - Grafana panels now link to alert solutions and dashboards reference - Refreshed observability docs Co-authored-by: uwedeportivo <534011+uwedeportivo@users.noreply.github.com>
2020-12-23 02:36:41 +00:00
- <span class="badge badge-warning">warning</span> repo-updater: 10000+ maximum active goroutines for 10m0s
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- More help interpreting this metric is available in the [dashboards reference](./dashboards.md#repo-updater-go-goroutines).
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"warning_repo-updater_go_goroutines"
]
```
Update o11y ownership from IAM/repo-mgmt to Source (#54368) Does what it says in the title. ## Test plan - Ran `sg generate`
2023-06-28 13:49:16 +00:00
<sub>*Managed by the [Sourcegraph Source team](https://handbook.sourcegraph.com/departments/engineering/teams/source).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
monitoring: extract into a submodule (#45786) This change extracts `monitoring` into a submodule for import in `sourcegraph/controller` (https://github.com/sourcegraph/controller/pull/195) so that we can generate dashboards for Cloud instances. These steps were required: 1. Initialize a `go.mod` in `monitoring` 2. Extract `dev/sg/internal/cliutil` into `lib` to avoid illegal imports from `monitoring` 3. Add local replaces to both `sourcegraph/sourcegraph` and `monitoring` 4. `go mod tidy` on all submodules 5. Update `go generate ./monitoring` commands to use `sg`, since the `go generate` command no longer works 6. Update `grafana/build.sh`, `prometheus/build.sh` to build the submodule 7. Amend linters to check for multiple `go.mod` files and ban imports of `github.com/sourcegraph/sourcegraph` 8. Update `sg generate go` to run in directories rather than from root The only caveat is that if you use VS code, you will now need to open `monitoring` in a separate workspace or similar, like with `lib`. Co-authored-by: Joe Chen <joe@sourcegraph.com>
2022-12-19 17:49:25 +00:00
Generated query for warning alert: `max((max by (instance) (go_goroutines{job=~".*repo-updater"})) >= 10000)`
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
</details>
monitoring: improve solutions formatting, add team information (#14878)
2020-10-20 05:11:20 +00:00
<br />
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
## repo-updater: go_gc_duration_seconds
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
<p class="subtitle">maximum go garbage collection duration</p>
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
monitoring: dashboards reference, refresh observability docs (#16939) - New monitoring definition field, 'Interpretation' - New generated dashboards reference - Grafana panels now link to alert solutions and dashboards reference - Refreshed observability docs Co-authored-by: uwedeportivo <534011+uwedeportivo@users.noreply.github.com>
2020-12-23 02:36:41 +00:00
- <span class="badge badge-warning">warning</span> repo-updater: 2s+ maximum go garbage collection duration
monitoring: support multi-bound alerts, add lower-bound warnings for containers (#11082)
2020-06-16 07:09:25 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#repo-updater-go-gc-duration-seconds).
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"warning_repo-updater_go_gc_duration_seconds"
]
```
Update o11y ownership from IAM/repo-mgmt to Source (#54368) Does what it says in the title. ## Test plan - Ran `sg generate`
2023-06-28 13:49:16 +00:00
<sub>*Managed by the [Sourcegraph Source team](https://handbook.sourcegraph.com/departments/engineering/teams/source).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
monitoring: extract into a submodule (#45786) This change extracts `monitoring` into a submodule for import in `sourcegraph/controller` (https://github.com/sourcegraph/controller/pull/195) so that we can generate dashboards for Cloud instances. These steps were required: 1. Initialize a `go.mod` in `monitoring` 2. Extract `dev/sg/internal/cliutil` into `lib` to avoid illegal imports from `monitoring` 3. Add local replaces to both `sourcegraph/sourcegraph` and `monitoring` 4. `go mod tidy` on all submodules 5. Update `go generate ./monitoring` commands to use `sg`, since the `go generate` command no longer works 6. Update `grafana/build.sh`, `prometheus/build.sh` to build the submodule 7. Amend linters to check for multiple `go.mod` files and ban imports of `github.com/sourcegraph/sourcegraph` 8. Update `sg generate go` to run in directories rather than from root The only caveat is that if you use VS code, you will now need to open `monitoring` in a separate workspace or similar, like with `lib`. Co-authored-by: Joe Chen <joe@sourcegraph.com>
2022-12-19 17:49:25 +00:00
Generated query for warning alert: `max((max by (instance) (go_gc_duration_seconds{job=~".*repo-updater"})) >= 2)`
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
</details>
<br />
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
## repo-updater: pods_available_percentage
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
<p class="subtitle">percentage pods available</p>
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
monitoring: dashboards reference, refresh observability docs (#16939) - New monitoring definition field, 'Interpretation' - New generated dashboards reference - Grafana panels now link to alert solutions and dashboards reference - Refreshed observability docs Co-authored-by: uwedeportivo <534011+uwedeportivo@users.noreply.github.com>
2020-12-23 02:36:41 +00:00
- <span class="badge badge-critical">critical</span> repo-updater: less than 90% percentage pods available for 10m0s
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
monitoring: forbid critical alerts with no solutions (#36423) Adds the restriction that critical alerts must have solutions defined as a hard requirement within the generator, and fixes all the outstanding issues: - Add PossibleSolutions for the pods_available_percentage and pg_up critical alerts (we have a similar solution doc in the container_missing alert) - Removes all the other critical alerts without possible solutions Co-authored-by: Michael Lin <mlzc@hey.com>
2022-06-02 15:51:43 +00:00
- Determine if the pod was OOM killed using `kubectl describe pod repo-updater` (look for `OOMKilled: true`) and, if so, consider increasing the memory limit in the relevant `Deployment.yaml`.
- Check the logs before the container restarted to see if there are `panic:` messages or similar using `kubectl logs -p repo-updater`.
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#repo-updater-pods-available-percentage).
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"critical_repo-updater_pods_available_percentage"
]
```
Update o11y ownership from IAM/repo-mgmt to Source (#54368) Does what it says in the title. ## Test plan - Ran `sg generate`
2023-06-28 13:49:16 +00:00
<sub>*Managed by the [Sourcegraph Source team](https://handbook.sourcegraph.com/departments/engineering/teams/source).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
monitoring: extract into a submodule (#45786) This change extracts `monitoring` into a submodule for import in `sourcegraph/controller` (https://github.com/sourcegraph/controller/pull/195) so that we can generate dashboards for Cloud instances. These steps were required: 1. Initialize a `go.mod` in `monitoring` 2. Extract `dev/sg/internal/cliutil` into `lib` to avoid illegal imports from `monitoring` 3. Add local replaces to both `sourcegraph/sourcegraph` and `monitoring` 4. `go mod tidy` on all submodules 5. Update `go generate ./monitoring` commands to use `sg`, since the `go generate` command no longer works 6. Update `grafana/build.sh`, `prometheus/build.sh` to build the submodule 7. Amend linters to check for multiple `go.mod` files and ban imports of `github.com/sourcegraph/sourcegraph` 8. Update `sg generate go` to run in directories rather than from root The only caveat is that if you use VS code, you will now need to open `monitoring` in a separate workspace or similar, like with `lib`. Co-authored-by: Joe Chen <joe@sourcegraph.com>
2022-12-19 17:49:25 +00:00
Generated query for critical alert: `min((sum by (app) (up{app=~".*repo-updater"}) / count by (app) (up{app=~".*repo-updater"}) * 100) <= 90)`
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
</details>
monitoring: improve solutions formatting, add team information (#14878)
2020-10-20 05:11:20 +00:00
<br />
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: vastly improved searcher dashboard (#47654) While investigating searcher metrics I noticed our dashboard for it pretty much has no information vs the wealth of metrics we record. As such I spiked an hour this morning adding dashboards to grafana. For context see these notes https://gist.github.com/keegancsmith/9e53a1df12b5b863249c59539c0410fd Test Plan: Ran monitoring stack against production prometheus. Note: I wasted a lot of time trying to get this work on linux, but it seems a bunch of our stuff is broken there for local dev + grafana. So this only worked on my macbook. kubectl --namespace=prod port-forward svc/prometheus 9090:30090 sg run grafana monitoring-generator
2023-02-15 16:05:14 +00:00
## searcher: replica_traffic
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
monitoring: vastly improved searcher dashboard (#47654) While investigating searcher metrics I noticed our dashboard for it pretty much has no information vs the wealth of metrics we record. As such I spiked an hour this morning adding dashboards to grafana. For context see these notes https://gist.github.com/keegancsmith/9e53a1df12b5b863249c59539c0410fd Test Plan: Ran monitoring stack against production prometheus. Note: I wasted a lot of time trying to get this work on linux, but it seems a bunch of our stuff is broken there for local dev + grafana. So this only worked on my macbook. kubectl --namespace=prod port-forward svc/prometheus 9090:30090 sg run grafana monitoring-generator
2023-02-15 16:05:14 +00:00
<p class="subtitle">requests per second per replica over 10m</p>
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
monitoring: vastly improved searcher dashboard (#47654) While investigating searcher metrics I noticed our dashboard for it pretty much has no information vs the wealth of metrics we record. As such I spiked an hour this morning adding dashboards to grafana. For context see these notes https://gist.github.com/keegancsmith/9e53a1df12b5b863249c59539c0410fd Test Plan: Ran monitoring stack against production prometheus. Note: I wasted a lot of time trying to get this work on linux, but it seems a bunch of our stuff is broken there for local dev + grafana. So this only worked on my macbook. kubectl --namespace=prod port-forward svc/prometheus 9090:30090 sg run grafana monitoring-generator
2023-02-15 16:05:14 +00:00
- <span class="badge badge-warning">warning</span> searcher: 5+ requests per second per replica over 10m
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
monitoring: vastly improved searcher dashboard (#47654) While investigating searcher metrics I noticed our dashboard for it pretty much has no information vs the wealth of metrics we record. As such I spiked an hour this morning adding dashboards to grafana. For context see these notes https://gist.github.com/keegancsmith/9e53a1df12b5b863249c59539c0410fd Test Plan: Ran monitoring stack against production prometheus. Note: I wasted a lot of time trying to get this work on linux, but it seems a bunch of our stuff is broken there for local dev + grafana. So this only worked on my macbook. kubectl --namespace=prod port-forward svc/prometheus 9090:30090 sg run grafana monitoring-generator
2023-02-15 16:05:14 +00:00
- More help interpreting this metric is available in the [dashboards reference](./dashboards.md#searcher-replica-traffic).
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
monitoring: vastly improved searcher dashboard (#47654) While investigating searcher metrics I noticed our dashboard for it pretty much has no information vs the wealth of metrics we record. As such I spiked an hour this morning adding dashboards to grafana. For context see these notes https://gist.github.com/keegancsmith/9e53a1df12b5b863249c59539c0410fd Test Plan: Ran monitoring stack against production prometheus. Note: I wasted a lot of time trying to get this work on linux, but it seems a bunch of our stuff is broken there for local dev + grafana. So this only worked on my macbook. kubectl --namespace=prod port-forward svc/prometheus 9090:30090 sg run grafana monitoring-generator
2023-02-15 16:05:14 +00:00
"warning_searcher_replica_traffic"
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
]
```
Monitoring: update owner for search platform (#59316) The 'search core' team is now called 'search platform'. --------- Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2024-01-04 20:55:41 +00:00
<sub>*Managed by the [Sourcegraph Search Platform team](https://handbook.sourcegraph.com/departments/engineering/teams/search/core).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
monitoring: vastly improved searcher dashboard (#47654) While investigating searcher metrics I noticed our dashboard for it pretty much has no information vs the wealth of metrics we record. As such I spiked an hour this morning adding dashboards to grafana. For context see these notes https://gist.github.com/keegancsmith/9e53a1df12b5b863249c59539c0410fd Test Plan: Ran monitoring stack against production prometheus. Note: I wasted a lot of time trying to get this work on linux, but it seems a bunch of our stuff is broken there for local dev + grafana. So this only worked on my macbook. kubectl --namespace=prod port-forward svc/prometheus 9090:30090 sg run grafana monitoring-generator
2023-02-15 16:05:14 +00:00
Generated query for warning alert: `max((sum by (instance) (rate(searcher_service_request_total[10m]))) >= 5)`
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
</details>
monitoring: improve solutions formatting, add team information (#14878)
2020-10-20 05:11:20 +00:00
<br />
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: vastly improved searcher dashboard (#47654) While investigating searcher metrics I noticed our dashboard for it pretty much has no information vs the wealth of metrics we record. As such I spiked an hour this morning adding dashboards to grafana. For context see these notes https://gist.github.com/keegancsmith/9e53a1df12b5b863249c59539c0410fd Test Plan: Ran monitoring stack against production prometheus. Note: I wasted a lot of time trying to get this work on linux, but it seems a bunch of our stuff is broken there for local dev + grafana. So this only worked on my macbook. kubectl --namespace=prod port-forward svc/prometheus 9090:30090 sg run grafana monitoring-generator
2023-02-15 16:05:14 +00:00
## searcher: unindexed_search_request_errors
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
monitoring: vastly improved searcher dashboard (#47654) While investigating searcher metrics I noticed our dashboard for it pretty much has no information vs the wealth of metrics we record. As such I spiked an hour this morning adding dashboards to grafana. For context see these notes https://gist.github.com/keegancsmith/9e53a1df12b5b863249c59539c0410fd Test Plan: Ran monitoring stack against production prometheus. Note: I wasted a lot of time trying to get this work on linux, but it seems a bunch of our stuff is broken there for local dev + grafana. So this only worked on my macbook. kubectl --namespace=prod port-forward svc/prometheus 9090:30090 sg run grafana monitoring-generator
2023-02-15 16:05:14 +00:00
<p class="subtitle">unindexed search request errors every 5m by code</p>
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
monitoring: vastly improved searcher dashboard (#47654) While investigating searcher metrics I noticed our dashboard for it pretty much has no information vs the wealth of metrics we record. As such I spiked an hour this morning adding dashboards to grafana. For context see these notes https://gist.github.com/keegancsmith/9e53a1df12b5b863249c59539c0410fd Test Plan: Ran monitoring stack against production prometheus. Note: I wasted a lot of time trying to get this work on linux, but it seems a bunch of our stuff is broken there for local dev + grafana. So this only worked on my macbook. kubectl --namespace=prod port-forward svc/prometheus 9090:30090 sg run grafana monitoring-generator
2023-02-15 16:05:14 +00:00
- <span class="badge badge-warning">warning</span> searcher: 5%+ unindexed search request errors every 5m by code for 5m0s
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
monitoring: vastly improved searcher dashboard (#47654) While investigating searcher metrics I noticed our dashboard for it pretty much has no information vs the wealth of metrics we record. As such I spiked an hour this morning adding dashboards to grafana. For context see these notes https://gist.github.com/keegancsmith/9e53a1df12b5b863249c59539c0410fd Test Plan: Ran monitoring stack against production prometheus. Note: I wasted a lot of time trying to get this work on linux, but it seems a bunch of our stuff is broken there for local dev + grafana. So this only worked on my macbook. kubectl --namespace=prod port-forward svc/prometheus 9090:30090 sg run grafana monitoring-generator
2023-02-15 16:05:14 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#searcher-unindexed-search-request-errors).
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
monitoring: vastly improved searcher dashboard (#47654) While investigating searcher metrics I noticed our dashboard for it pretty much has no information vs the wealth of metrics we record. As such I spiked an hour this morning adding dashboards to grafana. For context see these notes https://gist.github.com/keegancsmith/9e53a1df12b5b863249c59539c0410fd Test Plan: Ran monitoring stack against production prometheus. Note: I wasted a lot of time trying to get this work on linux, but it seems a bunch of our stuff is broken there for local dev + grafana. So this only worked on my macbook. kubectl --namespace=prod port-forward svc/prometheus 9090:30090 sg run grafana monitoring-generator
2023-02-15 16:05:14 +00:00
"warning_searcher_unindexed_search_request_errors"
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
]
```
Monitoring: update owner for search platform (#59316) The 'search core' team is now called 'search platform'. --------- Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2024-01-04 20:55:41 +00:00
<sub>*Managed by the [Sourcegraph Search Platform team](https://handbook.sourcegraph.com/departments/engineering/teams/search/core).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
monitoring: vastly improved searcher dashboard (#47654) While investigating searcher metrics I noticed our dashboard for it pretty much has no information vs the wealth of metrics we record. As such I spiked an hour this morning adding dashboards to grafana. For context see these notes https://gist.github.com/keegancsmith/9e53a1df12b5b863249c59539c0410fd Test Plan: Ran monitoring stack against production prometheus. Note: I wasted a lot of time trying to get this work on linux, but it seems a bunch of our stuff is broken there for local dev + grafana. So this only worked on my macbook. kubectl --namespace=prod port-forward svc/prometheus 9090:30090 sg run grafana monitoring-generator
2023-02-15 16:05:14 +00:00
Generated query for warning alert: `max((sum by (code) (increase(searcher_service_request_total{code!="200",code!="canceled"}[5m])) / ignoring (code) group_left () sum(increase(searcher_service_request_total[5m])) * 100) >= 5)`
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
</details>
monitoring: improve solutions formatting, add team information (#14878)
2020-10-20 05:11:20 +00:00
<br />
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
conf: add metric and associated alert if clients fail to update site configuration within 5 minutes (#57682)
2023-10-18 23:53:55 +00:00
## searcher: searcher_site_configuration_duration_since_last_successful_update_by_instance
<p class="subtitle">maximum duration since last successful site configuration update (all "searcher" instances)</p>
**Descriptions**
- <span class="badge badge-critical">critical</span> searcher: 300s+ maximum duration since last successful site configuration update (all "searcher" instances)
**Next steps**
- This indicates that one or more "searcher" instances have not successfully updated the site configuration in over 5 minutes. This could be due to networking issues between services or problems with the site configuration service itself.
- Check for relevant errors in the "searcher" logs, as well as frontend`s logs.
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#searcher-searcher-site-configuration-duration-since-last-successful-update-by-instance).
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"critical_searcher_searcher_site_configuration_duration_since_last_successful_update_by_instance"
]
```
monitoring: test owners for valid Opsgenie teams and handbook pages (#59251) In INC-264 it seems that certain alerts - such as [zoekt: less than 90% percentage pods available for 10m0s](https://opsg.in/a/i/sourcegraph/178a626f-0f28-4295-bee9-84da988bb473-1703759057681) - don't seem to end up going anywhere because the ObservableOwner is defunct. This change adds _opt-in_ testing to report: 1. How many owners have valid Opsgenie teams 2. How many owners have valid handbook pages In addition, we collect ObservableOwners that pass the test and use it to generate configuration for `site.json` in Sourcegraph.com: https://github.com/sourcegraph/deploy-sourcegraph-cloud/pull/18338 - this helps ensure the list is valid and not deceptively high-coverage. The results are not great, but **enforcing** that owners are valid isn't currently in scope: ``` 6/10 ObservableOwners do not have valid Opsgenie teams 3/10 ObservableOwners do not point to valid handbook pages ``` I also removed some defunct/unused functionality/owners. ## Test plan To run these tests: ``` export OPSGENIE_API_KEY="..." go test -timeout 30s github.com/sourcegraph/sourcegraph/monitoring/monitoring -update -online ```
2023-12-29 22:07:35 +00:00
<sub>*Managed by the [Sourcegraph Infrastructure Org team](https://handbook.sourcegraph.com/departments/engineering/infrastructure).*</sub>
conf: add metric and associated alert if clients fail to update site configuration within 5 minutes (#57682)
2023-10-18 23:53:55 +00:00
<details>
<summary>Technical details</summary>
monitoring: fix alert definition for site configuration by adding scrape job label (#59687) We discovered recently that the definition for the alert that fires if the site configuration hasn't been fetched within 5 minutes strips out the regex that targets individual services (since it uses a grafana variable). This means that every instance of this alert will fire if any individual service trips over this threshold. This PR fixes the issue by adding a new `job` filter for this alert that targets only the services that have that Prometheus scrape target name. This works around the previous issue by using a fixed value for the `job` value instead of a dynamic grafana value. The value of the job filter generally looks like `job=~.*$container_name` (following the strategy from https://sourcegraph.com/github.com/sourcegraph/sourcegraph@9a780f2e694238b5326e3e121d6a1828463001b9/-/blob/monitoring/monitoring/monitoring.go?L161 ) unless I noticed that there was different logic in the existing dashboard for the services. Ex: - `frontend`: already used `job=~"(sourcegraph-)?frontend"` for some metrics, so I used it again here - `worker`: `already used `job=~"^worker.*"` in some metrics, so I used it again and standarized the other existing panels to use the same shared variable ## Test plan I eyeballed the generated alert.md and dashboards.md to verify that my changes looked correct (that is, my refactors resulted in either no diff, or that the diff I generated still looked like valid regex).
2024-01-17 23:19:54 +00:00
Generated query for critical alert: `max((max(max_over_time(src_conf_client_time_since_last_successful_update_seconds{job=~".*searcher"}[1m]))) >= 300)`
conf: add metric and associated alert if clients fail to update site configuration within 5 minutes (#57682)
2023-10-18 23:53:55 +00:00
</details>
<br />
gitserver: add db connections which are needed for future instantiation of gitserver clients with db connections (#32769)
2022-03-22 08:07:54 +00:00
## searcher: mean_blocked_seconds_per_conn_request
<p class="subtitle">mean blocked seconds per conn request</p>
**Descriptions**
monitoring: relax mean_blocked_seconds_per_conn_request alerts (#59507) https://github.com/sourcegraph/sourcegraph/pull/59284 dramatically reduced the `mean_blocked_seconds_per_conn_request` issues we've been seeing, but overall delays are still higher, even with generally healthy Cloud SQL resource utilization. <img width="1630" alt="image" src="https://github.com/sourcegraph/sourcegraph/assets/23356519/91615471-5187-4d15-83e7-5cc94595303c"> Spot-checking the spikes in load in Cloud SQL, it seems that there is a variety of causes for each spike (analytics workloads, Cody Gateway syncs, code intel workloads, gitserver things, `ListSourcegraphDotComIndexableRepos` etc) so I'm chalking this up to "expected". Since this alert is seen firing on a Cloud instance, let's just relax it for now so that it only fires a critical alert on very significant delays.
2024-01-11 01:14:28 +00:00
- <span class="badge badge-warning">warning</span> searcher: 0.1s+ mean blocked seconds per conn request for 10m0s
- <span class="badge badge-critical">critical</span> searcher: 0.5s+ mean blocked seconds per conn request for 10m0s
gitserver: add db connections which are needed for future instantiation of gitserver clients with db connections (#32769)
2022-03-22 08:07:54 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
gitserver: add db connections which are needed for future instantiation of gitserver clients with db connections (#32769)
2022-03-22 08:07:54 +00:00
- Increase SRC_PGSQL_MAX_OPEN together with giving more memory to the database if needed
Docs: update links to point to new site (#60381) We have a number of docs links in the product that point to the old doc site. Method: - Search the repo for `docs.sourcegraph.com` - Exclude the `doc/` dir, all test fixtures, and `CHANGELOG.md` - For each, replace `docs.sourcegraph.com` with `sourcegraph.com/docs` - Navigate to the resulting URL ensuring it's not a dead link, updating the URL if necessary Many of the URLs updated are just comments, but since I'm doing a manual audit of each URL anyways, I felt it was worth it to update these while I was at it.
2024-02-13 00:23:47 +00:00
- Scale up Postgres memory/cpus - [see our scaling guide](https://sourcegraph.com/docs/admin/config/postgres-conf)
monitoring: relax mean_blocked_seconds_per_conn_request alerts (#59507) https://github.com/sourcegraph/sourcegraph/pull/59284 dramatically reduced the `mean_blocked_seconds_per_conn_request` issues we've been seeing, but overall delays are still higher, even with generally healthy Cloud SQL resource utilization. <img width="1630" alt="image" src="https://github.com/sourcegraph/sourcegraph/assets/23356519/91615471-5187-4d15-83e7-5cc94595303c"> Spot-checking the spikes in load in Cloud SQL, it seems that there is a variety of causes for each spike (analytics workloads, Cody Gateway syncs, code intel workloads, gitserver things, `ListSourcegraphDotComIndexableRepos` etc) so I'm chalking this up to "expected". Since this alert is seen firing on a Cloud instance, let's just relax it for now so that it only fires a critical alert on very significant delays.
2024-01-11 01:14:28 +00:00
- If using GCP Cloud SQL, check for high lock waits or CPU usage in query insights
gitserver: add db connections which are needed for future instantiation of gitserver clients with db connections (#32769)
2022-03-22 08:07:54 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#searcher-mean-blocked-seconds-per-conn-request).
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"warning_searcher_mean_blocked_seconds_per_conn_request",
"critical_searcher_mean_blocked_seconds_per_conn_request"
]
```
monitoring: test owners for valid Opsgenie teams and handbook pages (#59251) In INC-264 it seems that certain alerts - such as [zoekt: less than 90% percentage pods available for 10m0s](https://opsg.in/a/i/sourcegraph/178a626f-0f28-4295-bee9-84da988bb473-1703759057681) - don't seem to end up going anywhere because the ObservableOwner is defunct. This change adds _opt-in_ testing to report: 1. How many owners have valid Opsgenie teams 2. How many owners have valid handbook pages In addition, we collect ObservableOwners that pass the test and use it to generate configuration for `site.json` in Sourcegraph.com: https://github.com/sourcegraph/deploy-sourcegraph-cloud/pull/18338 - this helps ensure the list is valid and not deceptively high-coverage. The results are not great, but **enforcing** that owners are valid isn't currently in scope: ``` 6/10 ObservableOwners do not have valid Opsgenie teams 3/10 ObservableOwners do not point to valid handbook pages ``` I also removed some defunct/unused functionality/owners. ## Test plan To run these tests: ``` export OPSGENIE_API_KEY="..." go test -timeout 30s github.com/sourcegraph/sourcegraph/monitoring/monitoring -update -online ```
2023-12-29 22:07:35 +00:00
<sub>*Managed by the [Sourcegraph Infrastructure Org team](https://handbook.sourcegraph.com/departments/engineering/infrastructure).*</sub>
gitserver: add db connections which are needed for future instantiation of gitserver clients with db connections (#32769)
2022-03-22 08:07:54 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
monitoring: relax mean_blocked_seconds_per_conn_request alerts (#59507) https://github.com/sourcegraph/sourcegraph/pull/59284 dramatically reduced the `mean_blocked_seconds_per_conn_request` issues we've been seeing, but overall delays are still higher, even with generally healthy Cloud SQL resource utilization. <img width="1630" alt="image" src="https://github.com/sourcegraph/sourcegraph/assets/23356519/91615471-5187-4d15-83e7-5cc94595303c"> Spot-checking the spikes in load in Cloud SQL, it seems that there is a variety of causes for each spike (analytics workloads, Cody Gateway syncs, code intel workloads, gitserver things, `ListSourcegraphDotComIndexableRepos` etc) so I'm chalking this up to "expected". Since this alert is seen firing on a Cloud instance, let's just relax it for now so that it only fires a critical alert on very significant delays.
2024-01-11 01:14:28 +00:00
Generated query for warning alert: `max((sum by (app_name, db_name) (increase(src_pgsql_conns_blocked_seconds{app_name="searcher"}[5m])) / sum by (app_name, db_name) (increase(src_pgsql_conns_waited_for{app_name="searcher"}[5m]))) >= 0.1)`
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
monitoring: relax mean_blocked_seconds_per_conn_request alerts (#59507) https://github.com/sourcegraph/sourcegraph/pull/59284 dramatically reduced the `mean_blocked_seconds_per_conn_request` issues we've been seeing, but overall delays are still higher, even with generally healthy Cloud SQL resource utilization. <img width="1630" alt="image" src="https://github.com/sourcegraph/sourcegraph/assets/23356519/91615471-5187-4d15-83e7-5cc94595303c"> Spot-checking the spikes in load in Cloud SQL, it seems that there is a variety of causes for each spike (analytics workloads, Cody Gateway syncs, code intel workloads, gitserver things, `ListSourcegraphDotComIndexableRepos` etc) so I'm chalking this up to "expected". Since this alert is seen firing on a Cloud instance, let's just relax it for now so that it only fires a critical alert on very significant delays.
2024-01-11 01:14:28 +00:00
Generated query for critical alert: `max((sum by (app_name, db_name) (increase(src_pgsql_conns_blocked_seconds{app_name="searcher"}[5m])) / sum by (app_name, db_name) (increase(src_pgsql_conns_waited_for{app_name="searcher"}[5m]))) >= 0.5)`
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
</details>
gitserver: add db connections which are needed for future instantiation of gitserver clients with db connections (#32769)
2022-03-22 08:07:54 +00:00
<br />
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
## searcher: container_cpu_usage
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
<p class="subtitle">container cpu usage total (1m average) across all cores by instance</p>
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
monitoring: dashboards reference, refresh observability docs (#16939) - New monitoring definition field, 'Interpretation' - New generated dashboards reference - Grafana panels now link to alert solutions and dashboards reference - Refreshed observability docs Co-authored-by: uwedeportivo <534011+uwedeportivo@users.noreply.github.com>
2020-12-23 02:36:41 +00:00
- <span class="badge badge-warning">warning</span> searcher: 99%+ container cpu usage total (1m average) across all cores by instance
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
monitoring: migrate out-of-band alerts to the generator (#12391) Migrates all out-of-band alert rules defined by deploy-sourcegraph to the generator, such that each alert now has a panel as part of a service dashboard like all our other alerts, with a few changes and additions.
2020-07-28 02:17:56 +00:00
- **Kubernetes:** Consider increasing CPU limits in the the relevant `Deployment.yaml`.
- **Docker Compose:** Consider increasing `cpus:` of the searcher container in `docker-compose.yml`.
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#searcher-container-cpu-usage).
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"warning_searcher_container_cpu_usage"
]
```
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
Monitoring: update owner for search platform (#59316) The 'search core' team is now called 'search platform'. --------- Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2024-01-04 20:55:41 +00:00
<sub>*Managed by the [Sourcegraph Search Platform team](https://handbook.sourcegraph.com/departments/engineering/teams/search/core).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
Generated query for warning alert: `max((cadvisor_container_cpu_usage_percentage_total{name=~"^searcher.*"}) >= 99)`
</details>
monitoring: improve solutions formatting, add team information (#14878)
2020-10-20 05:11:20 +00:00
<br />
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
## searcher: container_memory_usage
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
<p class="subtitle">container memory usage by instance</p>
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
monitoring: dashboards reference, refresh observability docs (#16939) - New monitoring definition field, 'Interpretation' - New generated dashboards reference - Grafana panels now link to alert solutions and dashboards reference - Refreshed observability docs Co-authored-by: uwedeportivo <534011+uwedeportivo@users.noreply.github.com>
2020-12-23 02:36:41 +00:00
- <span class="badge badge-warning">warning</span> searcher: 99%+ container memory usage by instance
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
- **Kubernetes:** Consider increasing memory limit in relevant `Deployment.yaml`.
- **Docker Compose:** Consider increasing `memory:` of searcher container in `docker-compose.yml`.
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#searcher-container-memory-usage).
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
```json
"observability.silenceAlerts": [
"warning_searcher_container_memory_usage"
]
```
Monitoring: update owner for search platform (#59316) The 'search core' team is now called 'search platform'. --------- Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2024-01-04 20:55:41 +00:00
<sub>*Managed by the [Sourcegraph Search Platform team](https://handbook.sourcegraph.com/departments/engineering/teams/search/core).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
Generated query for warning alert: `max((cadvisor_container_memory_usage_percentage_total{name=~"^searcher.*"}) >= 99)`
</details>
monitoring: improve solutions formatting, add team information (#14878)
2020-10-20 05:11:20 +00:00
<br />
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: improve long-term provisioning alerts aggregation (#12778)
2020-08-08 06:44:33 +00:00
## searcher: provisioning_container_cpu_usage_long_term
monitoring: support multi-bound alerts, add lower-bound warnings for containers (#11082)
2020-06-16 07:09:25 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
<p class="subtitle">container cpu usage total (90th percentile over 1d) across all cores by instance</p>
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
monitoring: support multi-bound alerts, add lower-bound warnings for containers (#11082)
2020-06-16 07:09:25 +00:00
monitoring: dashboards reference, refresh observability docs (#16939) - New monitoring definition field, 'Interpretation' - New generated dashboards reference - Grafana panels now link to alert solutions and dashboards reference - Refreshed observability docs Co-authored-by: uwedeportivo <534011+uwedeportivo@users.noreply.github.com>
2020-12-23 02:36:41 +00:00
- <span class="badge badge-warning">warning</span> searcher: 80%+ container cpu usage total (90th percentile over 1d) across all cores by instance for 336h0m0s
monitoring: support multi-bound alerts, add lower-bound warnings for containers (#11082)
2020-06-16 07:09:25 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: support multi-bound alerts, add lower-bound warnings for containers (#11082)
2020-06-16 07:09:25 +00:00
Remove container_*_usage_7d_low alerts (#16164)
2020-11-25 18:19:33 +00:00
- **Kubernetes:** Consider increasing CPU limits in the `Deployment.yaml` for the searcher service.
- **Docker Compose:** Consider increasing `cpus:` of the searcher container in `docker-compose.yml`.
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#searcher-provisioning-container-cpu-usage-long-term).
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
monitoring: improve long-term provisioning alerts aggregation (#12778)
2020-08-08 06:44:33 +00:00
"warning_searcher_provisioning_container_cpu_usage_long_term"
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
]
```
monitoring: support multi-bound alerts, add lower-bound warnings for containers (#11082)
2020-06-16 07:09:25 +00:00
Monitoring: update owner for search platform (#59316) The 'search core' team is now called 'search platform'. --------- Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2024-01-04 20:55:41 +00:00
<sub>*Managed by the [Sourcegraph Search Platform team](https://handbook.sourcegraph.com/departments/engineering/teams/search/core).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
Generated query for warning alert: `max((quantile_over_time(0.9, cadvisor_container_cpu_usage_percentage_total{name=~"^searcher.*"}[1d])) >= 80)`
</details>
monitoring: improve solutions formatting, add team information (#14878)
2020-10-20 05:11:20 +00:00
<br />
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: improve long-term provisioning alerts aggregation (#12778)
2020-08-08 06:44:33 +00:00
## searcher: provisioning_container_memory_usage_long_term
monitoring: support multi-bound alerts, add lower-bound warnings for containers (#11082)
2020-06-16 07:09:25 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
<p class="subtitle">container memory usage (1d maximum) by instance</p>
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
monitoring: support multi-bound alerts, add lower-bound warnings for containers (#11082)
2020-06-16 07:09:25 +00:00
monitoring: dashboards reference, refresh observability docs (#16939) - New monitoring definition field, 'Interpretation' - New generated dashboards reference - Grafana panels now link to alert solutions and dashboards reference - Refreshed observability docs Co-authored-by: uwedeportivo <534011+uwedeportivo@users.noreply.github.com>
2020-12-23 02:36:41 +00:00
- <span class="badge badge-warning">warning</span> searcher: 80%+ container memory usage (1d maximum) by instance for 336h0m0s
monitoring: support multi-bound alerts, add lower-bound warnings for containers (#11082)
2020-06-16 07:09:25 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: support multi-bound alerts, add lower-bound warnings for containers (#11082)
2020-06-16 07:09:25 +00:00
Remove container_*_usage_7d_low alerts (#16164)
2020-11-25 18:19:33 +00:00
- **Kubernetes:** Consider increasing memory limits in the `Deployment.yaml` for the searcher service.
- **Docker Compose:** Consider increasing `memory:` of the searcher container in `docker-compose.yml`.
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#searcher-provisioning-container-memory-usage-long-term).
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
monitoring: support multi-bound alerts, add lower-bound warnings for containers (#11082)
2020-06-16 07:09:25 +00:00
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
```json
"observability.silenceAlerts": [
monitoring: improve long-term provisioning alerts aggregation (#12778)
2020-08-08 06:44:33 +00:00
"warning_searcher_provisioning_container_memory_usage_long_term"
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
]
```
Monitoring: update owner for search platform (#59316) The 'search core' team is now called 'search platform'. --------- Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2024-01-04 20:55:41 +00:00
<sub>*Managed by the [Sourcegraph Search Platform team](https://handbook.sourcegraph.com/departments/engineering/teams/search/core).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
Generated query for warning alert: `max((max_over_time(cadvisor_container_memory_usage_percentage_total{name=~"^searcher.*"}[1d])) >= 80)`
</details>
monitoring: improve solutions formatting, add team information (#14878)
2020-10-20 05:11:20 +00:00
<br />
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: improve long-term provisioning alerts aggregation (#12778)
2020-08-08 06:44:33 +00:00
## searcher: provisioning_container_cpu_usage_short_term
monitoring: support multi-bound alerts, add lower-bound warnings for containers (#11082)
2020-06-16 07:09:25 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
<p class="subtitle">container cpu usage total (5m maximum) across all cores by instance</p>
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
monitoring: support multi-bound alerts, add lower-bound warnings for containers (#11082)
2020-06-16 07:09:25 +00:00
monitoring: dashboards reference, refresh observability docs (#16939) - New monitoring definition field, 'Interpretation' - New generated dashboards reference - Grafana panels now link to alert solutions and dashboards reference - Refreshed observability docs Co-authored-by: uwedeportivo <534011+uwedeportivo@users.noreply.github.com>
2020-12-23 02:36:41 +00:00
- <span class="badge badge-warning">warning</span> searcher: 90%+ container cpu usage total (5m maximum) across all cores by instance for 30m0s
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
- **Kubernetes:** Consider increasing CPU limits in the the relevant `Deployment.yaml`.
- **Docker Compose:** Consider increasing `cpus:` of the searcher container in `docker-compose.yml`.
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#searcher-provisioning-container-cpu-usage-short-term).
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
monitoring: improve long-term provisioning alerts aggregation (#12778)
2020-08-08 06:44:33 +00:00
"warning_searcher_provisioning_container_cpu_usage_short_term"
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
]
```
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
Monitoring: update owner for search platform (#59316) The 'search core' team is now called 'search platform'. --------- Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2024-01-04 20:55:41 +00:00
<sub>*Managed by the [Sourcegraph Search Platform team](https://handbook.sourcegraph.com/departments/engineering/teams/search/core).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
Generated query for warning alert: `max((max_over_time(cadvisor_container_cpu_usage_percentage_total{name=~"^searcher.*"}[5m])) >= 90)`
</details>
monitoring: improve solutions formatting, add team information (#14878)
2020-10-20 05:11:20 +00:00
<br />
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: improve long-term provisioning alerts aggregation (#12778)
2020-08-08 06:44:33 +00:00
## searcher: provisioning_container_memory_usage_short_term
monitoring: support multi-bound alerts, add lower-bound warnings for containers (#11082)
2020-06-16 07:09:25 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
<p class="subtitle">container memory usage (5m maximum) by instance</p>
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
monitoring: support multi-bound alerts, add lower-bound warnings for containers (#11082)
2020-06-16 07:09:25 +00:00
monitoring: dashboards reference, refresh observability docs (#16939) - New monitoring definition field, 'Interpretation' - New generated dashboards reference - Grafana panels now link to alert solutions and dashboards reference - Refreshed observability docs Co-authored-by: uwedeportivo <534011+uwedeportivo@users.noreply.github.com>
2020-12-23 02:36:41 +00:00
- <span class="badge badge-warning">warning</span> searcher: 90%+ container memory usage (5m maximum) by instance
monitoring: support multi-bound alerts, add lower-bound warnings for containers (#11082)
2020-06-16 07:09:25 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: support multi-bound alerts, add lower-bound warnings for containers (#11082)
2020-06-16 07:09:25 +00:00
- **Kubernetes:** Consider increasing memory limit in relevant `Deployment.yaml`.
- **Docker Compose:** Consider increasing `memory:` of searcher container in `docker-compose.yml`.
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#searcher-provisioning-container-memory-usage-short-term).
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
monitoring: improve long-term provisioning alerts aggregation (#12778)
2020-08-08 06:44:33 +00:00
"warning_searcher_provisioning_container_memory_usage_short_term"
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
]
```
Monitoring: update owner for search platform (#59316) The 'search core' team is now called 'search platform'. --------- Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2024-01-04 20:55:41 +00:00
<sub>*Managed by the [Sourcegraph Search Platform team](https://handbook.sourcegraph.com/departments/engineering/teams/search/core).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
Generated query for warning alert: `max((max_over_time(cadvisor_container_memory_usage_percentage_total{name=~"^searcher.*"}[5m])) >= 90)`
</details>
monitoring: improve solutions formatting, add team information (#14878)
2020-10-20 05:11:20 +00:00
<br />
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: improve containers metrics (#34808) part of - https://github.com/sourcegraph/sourcegraph/issues/33437 - https://github.com/sourcegraph/sourcegraph/issues/33438 This PR adds the following - A shared observable metric to indicate how many times a container (or child processes) has been oom kill - It's added to provisioning indicators since it's an indicator of underprovisioned resources - A new dashboard that shows resources usage of all containers at a glance - It contains two groups, one for all containers and one for containers that may have potential scaling issues - the first group is extremely noisy so it's hidden by default - the second group provides a more focused experience for site-admin to quickly find problematic resources <!-- > ~it looks like the alert table will be rendered regardless, which is pretty annoying (I guess the implementation of the monitoring stack is based on the assumption that one service/container will have its dashboard)~ __Questions & Notes__ - ~Should I make the new dashboard filterable by container name? like I only want to see the graph of `frontend`. Is that something that's valuable?~ - ~Any other metrics you all would like to see?~ - graph for total available resources on the VM compared to usage from https://github.com/sourcegraph/sourcegraph/issues/33437 is not really doable since we do not have `node-exporter` in anyone of our distro. We would have to tackle this from https://github.com/sourcegraph/customer/issues/806 --> ## Test plan connect to `demo`'s prom ```sh gcloud compute start-iap-tunnel default-$NEW_DEPLOYMENT-instance 9090 --local-host-port=localhost:4445 --zone us-central1-f --project $PROJECT_PREFIX-$CUSTOMER ``` or use dogfood ```sh kubectl port-forward svc/prometheus 4445:30090 ``` Update `dev/grafana/all/datasources.yaml` ```yml # Configuration for all non-Linux platforms. apiVersion: 1 datasources: - name: Prometheus type: prometheus access: proxy url: http://docker.for.mac.localhost:4445 # use the remote prometheus instance on demo instead of the local one isDefault: true editable: false - name: Jaeger type: jaeger access: proxy url: http://host.docker.internal:16686/-/debug/jaeger - name: Loki type: loki access: proxy url: http://host.docker.internal:3100 ``` Start monitoring stack ```sh sg run grafana ``` see screenshot > the first group is hidden by default ![CleanShot 2022-05-04 at 10 48 26](https://user-images.githubusercontent.com/8373004/166748470-22460552-6e51-4b29-a26b-4c5adbe1c404.png) ![CleanShot 2022-05-04 at 10 48 03](https://user-images.githubusercontent.com/8373004/166748396-1d410c35-e5fe-45e2-83bd-418d845413cb.png) ![CleanShot 2022-05-04 at 11 25 13](https://user-images.githubusercontent.com/8373004/166801266-21e51b68-65fd-4cb7-97de-83520e31e498.png)
2022-05-09 23:56:47 +00:00
## searcher: container_oomkill_events_total
<p class="subtitle">container OOMKILL events total by instance</p>
**Descriptions**
- <span class="badge badge-warning">warning</span> searcher: 1+ container OOMKILL events total by instance
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: improve containers metrics (#34808) part of - https://github.com/sourcegraph/sourcegraph/issues/33437 - https://github.com/sourcegraph/sourcegraph/issues/33438 This PR adds the following - A shared observable metric to indicate how many times a container (or child processes) has been oom kill - It's added to provisioning indicators since it's an indicator of underprovisioned resources - A new dashboard that shows resources usage of all containers at a glance - It contains two groups, one for all containers and one for containers that may have potential scaling issues - the first group is extremely noisy so it's hidden by default - the second group provides a more focused experience for site-admin to quickly find problematic resources <!-- > ~it looks like the alert table will be rendered regardless, which is pretty annoying (I guess the implementation of the monitoring stack is based on the assumption that one service/container will have its dashboard)~ __Questions & Notes__ - ~Should I make the new dashboard filterable by container name? like I only want to see the graph of `frontend`. Is that something that's valuable?~ - ~Any other metrics you all would like to see?~ - graph for total available resources on the VM compared to usage from https://github.com/sourcegraph/sourcegraph/issues/33437 is not really doable since we do not have `node-exporter` in anyone of our distro. We would have to tackle this from https://github.com/sourcegraph/customer/issues/806 --> ## Test plan connect to `demo`'s prom ```sh gcloud compute start-iap-tunnel default-$NEW_DEPLOYMENT-instance 9090 --local-host-port=localhost:4445 --zone us-central1-f --project $PROJECT_PREFIX-$CUSTOMER ``` or use dogfood ```sh kubectl port-forward svc/prometheus 4445:30090 ``` Update `dev/grafana/all/datasources.yaml` ```yml # Configuration for all non-Linux platforms. apiVersion: 1 datasources: - name: Prometheus type: prometheus access: proxy url: http://docker.for.mac.localhost:4445 # use the remote prometheus instance on demo instead of the local one isDefault: true editable: false - name: Jaeger type: jaeger access: proxy url: http://host.docker.internal:16686/-/debug/jaeger - name: Loki type: loki access: proxy url: http://host.docker.internal:3100 ``` Start monitoring stack ```sh sg run grafana ``` see screenshot > the first group is hidden by default ![CleanShot 2022-05-04 at 10 48 26](https://user-images.githubusercontent.com/8373004/166748470-22460552-6e51-4b29-a26b-4c5adbe1c404.png) ![CleanShot 2022-05-04 at 10 48 03](https://user-images.githubusercontent.com/8373004/166748396-1d410c35-e5fe-45e2-83bd-418d845413cb.png) ![CleanShot 2022-05-04 at 11 25 13](https://user-images.githubusercontent.com/8373004/166801266-21e51b68-65fd-4cb7-97de-83520e31e498.png)
2022-05-09 23:56:47 +00:00
- **Kubernetes:** Consider increasing memory limit in relevant `Deployment.yaml`.
- **Docker Compose:** Consider increasing `memory:` of searcher container in `docker-compose.yml`.
- More help interpreting this metric is available in the [dashboards reference](./dashboards.md#searcher-container-oomkill-events-total).
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"warning_searcher_container_oomkill_events_total"
]
```
Monitoring: update owner for search platform (#59316) The 'search core' team is now called 'search platform'. --------- Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2024-01-04 20:55:41 +00:00
<sub>*Managed by the [Sourcegraph Search Platform team](https://handbook.sourcegraph.com/departments/engineering/teams/search/core).*</sub>
monitoring: improve containers metrics (#34808) part of - https://github.com/sourcegraph/sourcegraph/issues/33437 - https://github.com/sourcegraph/sourcegraph/issues/33438 This PR adds the following - A shared observable metric to indicate how many times a container (or child processes) has been oom kill - It's added to provisioning indicators since it's an indicator of underprovisioned resources - A new dashboard that shows resources usage of all containers at a glance - It contains two groups, one for all containers and one for containers that may have potential scaling issues - the first group is extremely noisy so it's hidden by default - the second group provides a more focused experience for site-admin to quickly find problematic resources <!-- > ~it looks like the alert table will be rendered regardless, which is pretty annoying (I guess the implementation of the monitoring stack is based on the assumption that one service/container will have its dashboard)~ __Questions & Notes__ - ~Should I make the new dashboard filterable by container name? like I only want to see the graph of `frontend`. Is that something that's valuable?~ - ~Any other metrics you all would like to see?~ - graph for total available resources on the VM compared to usage from https://github.com/sourcegraph/sourcegraph/issues/33437 is not really doable since we do not have `node-exporter` in anyone of our distro. We would have to tackle this from https://github.com/sourcegraph/customer/issues/806 --> ## Test plan connect to `demo`'s prom ```sh gcloud compute start-iap-tunnel default-$NEW_DEPLOYMENT-instance 9090 --local-host-port=localhost:4445 --zone us-central1-f --project $PROJECT_PREFIX-$CUSTOMER ``` or use dogfood ```sh kubectl port-forward svc/prometheus 4445:30090 ``` Update `dev/grafana/all/datasources.yaml` ```yml # Configuration for all non-Linux platforms. apiVersion: 1 datasources: - name: Prometheus type: prometheus access: proxy url: http://docker.for.mac.localhost:4445 # use the remote prometheus instance on demo instead of the local one isDefault: true editable: false - name: Jaeger type: jaeger access: proxy url: http://host.docker.internal:16686/-/debug/jaeger - name: Loki type: loki access: proxy url: http://host.docker.internal:3100 ``` Start monitoring stack ```sh sg run grafana ``` see screenshot > the first group is hidden by default ![CleanShot 2022-05-04 at 10 48 26](https://user-images.githubusercontent.com/8373004/166748470-22460552-6e51-4b29-a26b-4c5adbe1c404.png) ![CleanShot 2022-05-04 at 10 48 03](https://user-images.githubusercontent.com/8373004/166748396-1d410c35-e5fe-45e2-83bd-418d845413cb.png) ![CleanShot 2022-05-04 at 11 25 13](https://user-images.githubusercontent.com/8373004/166801266-21e51b68-65fd-4cb7-97de-83520e31e498.png)
2022-05-09 23:56:47 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
monitoring: extract into a submodule (#45786) This change extracts `monitoring` into a submodule for import in `sourcegraph/controller` (https://github.com/sourcegraph/controller/pull/195) so that we can generate dashboards for Cloud instances. These steps were required: 1. Initialize a `go.mod` in `monitoring` 2. Extract `dev/sg/internal/cliutil` into `lib` to avoid illegal imports from `monitoring` 3. Add local replaces to both `sourcegraph/sourcegraph` and `monitoring` 4. `go mod tidy` on all submodules 5. Update `go generate ./monitoring` commands to use `sg`, since the `go generate` command no longer works 6. Update `grafana/build.sh`, `prometheus/build.sh` to build the submodule 7. Amend linters to check for multiple `go.mod` files and ban imports of `github.com/sourcegraph/sourcegraph` 8. Update `sg generate go` to run in directories rather than from root The only caveat is that if you use VS code, you will now need to open `monitoring` in a separate workspace or similar, like with `lib`. Co-authored-by: Joe Chen <joe@sourcegraph.com>
2022-12-19 17:49:25 +00:00
Generated query for warning alert: `max((max by (name) (container_oom_events_total{name=~"^searcher.*"})) >= 1)`
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
</details>
monitoring: improve containers metrics (#34808) part of - https://github.com/sourcegraph/sourcegraph/issues/33437 - https://github.com/sourcegraph/sourcegraph/issues/33438 This PR adds the following - A shared observable metric to indicate how many times a container (or child processes) has been oom kill - It's added to provisioning indicators since it's an indicator of underprovisioned resources - A new dashboard that shows resources usage of all containers at a glance - It contains two groups, one for all containers and one for containers that may have potential scaling issues - the first group is extremely noisy so it's hidden by default - the second group provides a more focused experience for site-admin to quickly find problematic resources <!-- > ~it looks like the alert table will be rendered regardless, which is pretty annoying (I guess the implementation of the monitoring stack is based on the assumption that one service/container will have its dashboard)~ __Questions & Notes__ - ~Should I make the new dashboard filterable by container name? like I only want to see the graph of `frontend`. Is that something that's valuable?~ - ~Any other metrics you all would like to see?~ - graph for total available resources on the VM compared to usage from https://github.com/sourcegraph/sourcegraph/issues/33437 is not really doable since we do not have `node-exporter` in anyone of our distro. We would have to tackle this from https://github.com/sourcegraph/customer/issues/806 --> ## Test plan connect to `demo`'s prom ```sh gcloud compute start-iap-tunnel default-$NEW_DEPLOYMENT-instance 9090 --local-host-port=localhost:4445 --zone us-central1-f --project $PROJECT_PREFIX-$CUSTOMER ``` or use dogfood ```sh kubectl port-forward svc/prometheus 4445:30090 ``` Update `dev/grafana/all/datasources.yaml` ```yml # Configuration for all non-Linux platforms. apiVersion: 1 datasources: - name: Prometheus type: prometheus access: proxy url: http://docker.for.mac.localhost:4445 # use the remote prometheus instance on demo instead of the local one isDefault: true editable: false - name: Jaeger type: jaeger access: proxy url: http://host.docker.internal:16686/-/debug/jaeger - name: Loki type: loki access: proxy url: http://host.docker.internal:3100 ``` Start monitoring stack ```sh sg run grafana ``` see screenshot > the first group is hidden by default ![CleanShot 2022-05-04 at 10 48 26](https://user-images.githubusercontent.com/8373004/166748470-22460552-6e51-4b29-a26b-4c5adbe1c404.png) ![CleanShot 2022-05-04 at 10 48 03](https://user-images.githubusercontent.com/8373004/166748396-1d410c35-e5fe-45e2-83bd-418d845413cb.png) ![CleanShot 2022-05-04 at 11 25 13](https://user-images.githubusercontent.com/8373004/166801266-21e51b68-65fd-4cb7-97de-83520e31e498.png)
2022-05-09 23:56:47 +00:00
<br />
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
## searcher: go_goroutines
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
<p class="subtitle">maximum active goroutines</p>
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
monitoring: dashboards reference, refresh observability docs (#16939) - New monitoring definition field, 'Interpretation' - New generated dashboards reference - Grafana panels now link to alert solutions and dashboards reference - Refreshed observability docs Co-authored-by: uwedeportivo <534011+uwedeportivo@users.noreply.github.com>
2020-12-23 02:36:41 +00:00
- <span class="badge badge-warning">warning</span> searcher: 10000+ maximum active goroutines for 10m0s
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- More help interpreting this metric is available in the [dashboards reference](./dashboards.md#searcher-go-goroutines).
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"warning_searcher_go_goroutines"
]
```
Monitoring: update owner for search platform (#59316) The 'search core' team is now called 'search platform'. --------- Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2024-01-04 20:55:41 +00:00
<sub>*Managed by the [Sourcegraph Search Platform team](https://handbook.sourcegraph.com/departments/engineering/teams/search/core).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
monitoring: extract into a submodule (#45786) This change extracts `monitoring` into a submodule for import in `sourcegraph/controller` (https://github.com/sourcegraph/controller/pull/195) so that we can generate dashboards for Cloud instances. These steps were required: 1. Initialize a `go.mod` in `monitoring` 2. Extract `dev/sg/internal/cliutil` into `lib` to avoid illegal imports from `monitoring` 3. Add local replaces to both `sourcegraph/sourcegraph` and `monitoring` 4. `go mod tidy` on all submodules 5. Update `go generate ./monitoring` commands to use `sg`, since the `go generate` command no longer works 6. Update `grafana/build.sh`, `prometheus/build.sh` to build the submodule 7. Amend linters to check for multiple `go.mod` files and ban imports of `github.com/sourcegraph/sourcegraph` 8. Update `sg generate go` to run in directories rather than from root The only caveat is that if you use VS code, you will now need to open `monitoring` in a separate workspace or similar, like with `lib`. Co-authored-by: Joe Chen <joe@sourcegraph.com>
2022-12-19 17:49:25 +00:00
Generated query for warning alert: `max((max by (instance) (go_goroutines{job=~".*searcher"})) >= 10000)`
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
</details>
monitoring: improve solutions formatting, add team information (#14878)
2020-10-20 05:11:20 +00:00
<br />
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
## searcher: go_gc_duration_seconds
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
<p class="subtitle">maximum go garbage collection duration</p>
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
monitoring: dashboards reference, refresh observability docs (#16939) - New monitoring definition field, 'Interpretation' - New generated dashboards reference - Grafana panels now link to alert solutions and dashboards reference - Refreshed observability docs Co-authored-by: uwedeportivo <534011+uwedeportivo@users.noreply.github.com>
2020-12-23 02:36:41 +00:00
- <span class="badge badge-warning">warning</span> searcher: 2s+ maximum go garbage collection duration
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#searcher-go-gc-duration-seconds).
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"warning_searcher_go_gc_duration_seconds"
]
```
Monitoring: update owner for search platform (#59316) The 'search core' team is now called 'search platform'. --------- Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2024-01-04 20:55:41 +00:00
<sub>*Managed by the [Sourcegraph Search Platform team](https://handbook.sourcegraph.com/departments/engineering/teams/search/core).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
monitoring: extract into a submodule (#45786) This change extracts `monitoring` into a submodule for import in `sourcegraph/controller` (https://github.com/sourcegraph/controller/pull/195) so that we can generate dashboards for Cloud instances. These steps were required: 1. Initialize a `go.mod` in `monitoring` 2. Extract `dev/sg/internal/cliutil` into `lib` to avoid illegal imports from `monitoring` 3. Add local replaces to both `sourcegraph/sourcegraph` and `monitoring` 4. `go mod tidy` on all submodules 5. Update `go generate ./monitoring` commands to use `sg`, since the `go generate` command no longer works 6. Update `grafana/build.sh`, `prometheus/build.sh` to build the submodule 7. Amend linters to check for multiple `go.mod` files and ban imports of `github.com/sourcegraph/sourcegraph` 8. Update `sg generate go` to run in directories rather than from root The only caveat is that if you use VS code, you will now need to open `monitoring` in a separate workspace or similar, like with `lib`. Co-authored-by: Joe Chen <joe@sourcegraph.com>
2022-12-19 17:49:25 +00:00
Generated query for warning alert: `max((max by (instance) (go_gc_duration_seconds{job=~".*searcher"})) >= 2)`
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
</details>
monitoring: improve solutions formatting, add team information (#14878)
2020-10-20 05:11:20 +00:00
<br />
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
## searcher: pods_available_percentage
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
<p class="subtitle">percentage pods available</p>
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
monitoring: dashboards reference, refresh observability docs (#16939) - New monitoring definition field, 'Interpretation' - New generated dashboards reference - Grafana panels now link to alert solutions and dashboards reference - Refreshed observability docs Co-authored-by: uwedeportivo <534011+uwedeportivo@users.noreply.github.com>
2020-12-23 02:36:41 +00:00
- <span class="badge badge-critical">critical</span> searcher: less than 90% percentage pods available for 10m0s
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
monitoring: forbid critical alerts with no solutions (#36423) Adds the restriction that critical alerts must have solutions defined as a hard requirement within the generator, and fixes all the outstanding issues: - Add PossibleSolutions for the pods_available_percentage and pg_up critical alerts (we have a similar solution doc in the container_missing alert) - Removes all the other critical alerts without possible solutions Co-authored-by: Michael Lin <mlzc@hey.com>
2022-06-02 15:51:43 +00:00
- Determine if the pod was OOM killed using `kubectl describe pod searcher` (look for `OOMKilled: true`) and, if so, consider increasing the memory limit in the relevant `Deployment.yaml`.
- Check the logs before the container restarted to see if there are `panic:` messages or similar using `kubectl logs -p searcher`.
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#searcher-pods-available-percentage).
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"critical_searcher_pods_available_percentage"
]
```
Monitoring: update owner for search platform (#59316) The 'search core' team is now called 'search platform'. --------- Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2024-01-04 20:55:41 +00:00
<sub>*Managed by the [Sourcegraph Search Platform team](https://handbook.sourcegraph.com/departments/engineering/teams/search/core).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
monitoring: extract into a submodule (#45786) This change extracts `monitoring` into a submodule for import in `sourcegraph/controller` (https://github.com/sourcegraph/controller/pull/195) so that we can generate dashboards for Cloud instances. These steps were required: 1. Initialize a `go.mod` in `monitoring` 2. Extract `dev/sg/internal/cliutil` into `lib` to avoid illegal imports from `monitoring` 3. Add local replaces to both `sourcegraph/sourcegraph` and `monitoring` 4. `go mod tidy` on all submodules 5. Update `go generate ./monitoring` commands to use `sg`, since the `go generate` command no longer works 6. Update `grafana/build.sh`, `prometheus/build.sh` to build the submodule 7. Amend linters to check for multiple `go.mod` files and ban imports of `github.com/sourcegraph/sourcegraph` 8. Update `sg generate go` to run in directories rather than from root The only caveat is that if you use VS code, you will now need to open `monitoring` in a separate workspace or similar, like with `lib`. Co-authored-by: Joe Chen <joe@sourcegraph.com>
2022-12-19 17:49:25 +00:00
Generated query for critical alert: `min((sum by (app) (up{app=~".*searcher"}) / count by (app) (up{app=~".*searcher"}) * 100) <= 90)`
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
</details>
monitoring: improve solutions formatting, add team information (#14878)
2020-10-20 05:11:20 +00:00
<br />
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
conf: add metric and associated alert if clients fail to update site configuration within 5 minutes (#57682)
2023-10-18 23:53:55 +00:00
## symbols: symbols_site_configuration_duration_since_last_successful_update_by_instance
<p class="subtitle">maximum duration since last successful site configuration update (all "symbols" instances)</p>
**Descriptions**
- <span class="badge badge-critical">critical</span> symbols: 300s+ maximum duration since last successful site configuration update (all "symbols" instances)
**Next steps**
- This indicates that one or more "symbols" instances have not successfully updated the site configuration in over 5 minutes. This could be due to networking issues between services or problems with the site configuration service itself.
- Check for relevant errors in the "symbols" logs, as well as frontend`s logs.
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#symbols-symbols-site-configuration-duration-since-last-successful-update-by-instance).
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"critical_symbols_symbols_site_configuration_duration_since_last_successful_update_by_instance"
]
```
monitoring: test owners for valid Opsgenie teams and handbook pages (#59251) In INC-264 it seems that certain alerts - such as [zoekt: less than 90% percentage pods available for 10m0s](https://opsg.in/a/i/sourcegraph/178a626f-0f28-4295-bee9-84da988bb473-1703759057681) - don't seem to end up going anywhere because the ObservableOwner is defunct. This change adds _opt-in_ testing to report: 1. How many owners have valid Opsgenie teams 2. How many owners have valid handbook pages In addition, we collect ObservableOwners that pass the test and use it to generate configuration for `site.json` in Sourcegraph.com: https://github.com/sourcegraph/deploy-sourcegraph-cloud/pull/18338 - this helps ensure the list is valid and not deceptively high-coverage. The results are not great, but **enforcing** that owners are valid isn't currently in scope: ``` 6/10 ObservableOwners do not have valid Opsgenie teams 3/10 ObservableOwners do not point to valid handbook pages ``` I also removed some defunct/unused functionality/owners. ## Test plan To run these tests: ``` export OPSGENIE_API_KEY="..." go test -timeout 30s github.com/sourcegraph/sourcegraph/monitoring/monitoring -update -online ```
2023-12-29 22:07:35 +00:00
<sub>*Managed by the [Sourcegraph Infrastructure Org team](https://handbook.sourcegraph.com/departments/engineering/infrastructure).*</sub>
conf: add metric and associated alert if clients fail to update site configuration within 5 minutes (#57682)
2023-10-18 23:53:55 +00:00
<details>
<summary>Technical details</summary>
monitoring: fix alert definition for site configuration by adding scrape job label (#59687) We discovered recently that the definition for the alert that fires if the site configuration hasn't been fetched within 5 minutes strips out the regex that targets individual services (since it uses a grafana variable). This means that every instance of this alert will fire if any individual service trips over this threshold. This PR fixes the issue by adding a new `job` filter for this alert that targets only the services that have that Prometheus scrape target name. This works around the previous issue by using a fixed value for the `job` value instead of a dynamic grafana value. The value of the job filter generally looks like `job=~.*$container_name` (following the strategy from https://sourcegraph.com/github.com/sourcegraph/sourcegraph@9a780f2e694238b5326e3e121d6a1828463001b9/-/blob/monitoring/monitoring/monitoring.go?L161 ) unless I noticed that there was different logic in the existing dashboard for the services. Ex: - `frontend`: already used `job=~"(sourcegraph-)?frontend"` for some metrics, so I used it again here - `worker`: `already used `job=~"^worker.*"` in some metrics, so I used it again and standarized the other existing panels to use the same shared variable ## Test plan I eyeballed the generated alert.md and dashboards.md to verify that my changes looked correct (that is, my refactors resulted in either no diff, or that the diff I generated still looked like valid regex).
2024-01-17 23:19:54 +00:00
Generated query for critical alert: `max((max(max_over_time(src_conf_client_time_since_last_successful_update_seconds{job=~".*symbols"}[1m]))) >= 300)`
conf: add metric and associated alert if clients fail to update site configuration within 5 minutes (#57682)
2023-10-18 23:53:55 +00:00
</details>
<br />
symbols: Add DB connection for gitserver client (#39503) This adds a DB connection to be used for the gitserver client in the symbols service. A follow-up PR will make better use of it. We can also use the DB for some optimizations in the future.
2022-07-27 20:25:01 +00:00
## symbols: mean_blocked_seconds_per_conn_request
<p class="subtitle">mean blocked seconds per conn request</p>
**Descriptions**
monitoring: relax mean_blocked_seconds_per_conn_request alerts (#59507) https://github.com/sourcegraph/sourcegraph/pull/59284 dramatically reduced the `mean_blocked_seconds_per_conn_request` issues we've been seeing, but overall delays are still higher, even with generally healthy Cloud SQL resource utilization. <img width="1630" alt="image" src="https://github.com/sourcegraph/sourcegraph/assets/23356519/91615471-5187-4d15-83e7-5cc94595303c"> Spot-checking the spikes in load in Cloud SQL, it seems that there is a variety of causes for each spike (analytics workloads, Cody Gateway syncs, code intel workloads, gitserver things, `ListSourcegraphDotComIndexableRepos` etc) so I'm chalking this up to "expected". Since this alert is seen firing on a Cloud instance, let's just relax it for now so that it only fires a critical alert on very significant delays.
2024-01-11 01:14:28 +00:00
- <span class="badge badge-warning">warning</span> symbols: 0.1s+ mean blocked seconds per conn request for 10m0s
- <span class="badge badge-critical">critical</span> symbols: 0.5s+ mean blocked seconds per conn request for 10m0s
symbols: Add DB connection for gitserver client (#39503) This adds a DB connection to be used for the gitserver client in the symbols service. A follow-up PR will make better use of it. We can also use the DB for some optimizations in the future.
2022-07-27 20:25:01 +00:00
**Next steps**
- Increase SRC_PGSQL_MAX_OPEN together with giving more memory to the database if needed
Docs: update links to point to new site (#60381) We have a number of docs links in the product that point to the old doc site. Method: - Search the repo for `docs.sourcegraph.com` - Exclude the `doc/` dir, all test fixtures, and `CHANGELOG.md` - For each, replace `docs.sourcegraph.com` with `sourcegraph.com/docs` - Navigate to the resulting URL ensuring it's not a dead link, updating the URL if necessary Many of the URLs updated are just comments, but since I'm doing a manual audit of each URL anyways, I felt it was worth it to update these while I was at it.
2024-02-13 00:23:47 +00:00
- Scale up Postgres memory/cpus - [see our scaling guide](https://sourcegraph.com/docs/admin/config/postgres-conf)
monitoring: relax mean_blocked_seconds_per_conn_request alerts (#59507) https://github.com/sourcegraph/sourcegraph/pull/59284 dramatically reduced the `mean_blocked_seconds_per_conn_request` issues we've been seeing, but overall delays are still higher, even with generally healthy Cloud SQL resource utilization. <img width="1630" alt="image" src="https://github.com/sourcegraph/sourcegraph/assets/23356519/91615471-5187-4d15-83e7-5cc94595303c"> Spot-checking the spikes in load in Cloud SQL, it seems that there is a variety of causes for each spike (analytics workloads, Cody Gateway syncs, code intel workloads, gitserver things, `ListSourcegraphDotComIndexableRepos` etc) so I'm chalking this up to "expected". Since this alert is seen firing on a Cloud instance, let's just relax it for now so that it only fires a critical alert on very significant delays.
2024-01-11 01:14:28 +00:00
- If using GCP Cloud SQL, check for high lock waits or CPU usage in query insights
symbols: Add DB connection for gitserver client (#39503) This adds a DB connection to be used for the gitserver client in the symbols service. A follow-up PR will make better use of it. We can also use the DB for some optimizations in the future.
2022-07-27 20:25:01 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#symbols-mean-blocked-seconds-per-conn-request).
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"warning_symbols_mean_blocked_seconds_per_conn_request",
"critical_symbols_mean_blocked_seconds_per_conn_request"
]
```
monitoring: test owners for valid Opsgenie teams and handbook pages (#59251) In INC-264 it seems that certain alerts - such as [zoekt: less than 90% percentage pods available for 10m0s](https://opsg.in/a/i/sourcegraph/178a626f-0f28-4295-bee9-84da988bb473-1703759057681) - don't seem to end up going anywhere because the ObservableOwner is defunct. This change adds _opt-in_ testing to report: 1. How many owners have valid Opsgenie teams 2. How many owners have valid handbook pages In addition, we collect ObservableOwners that pass the test and use it to generate configuration for `site.json` in Sourcegraph.com: https://github.com/sourcegraph/deploy-sourcegraph-cloud/pull/18338 - this helps ensure the list is valid and not deceptively high-coverage. The results are not great, but **enforcing** that owners are valid isn't currently in scope: ``` 6/10 ObservableOwners do not have valid Opsgenie teams 3/10 ObservableOwners do not point to valid handbook pages ``` I also removed some defunct/unused functionality/owners. ## Test plan To run these tests: ``` export OPSGENIE_API_KEY="..." go test -timeout 30s github.com/sourcegraph/sourcegraph/monitoring/monitoring -update -online ```
2023-12-29 22:07:35 +00:00
<sub>*Managed by the [Sourcegraph Infrastructure Org team](https://handbook.sourcegraph.com/departments/engineering/infrastructure).*</sub>
symbols: Add DB connection for gitserver client (#39503) This adds a DB connection to be used for the gitserver client in the symbols service. A follow-up PR will make better use of it. We can also use the DB for some optimizations in the future.
2022-07-27 20:25:01 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
monitoring: relax mean_blocked_seconds_per_conn_request alerts (#59507) https://github.com/sourcegraph/sourcegraph/pull/59284 dramatically reduced the `mean_blocked_seconds_per_conn_request` issues we've been seeing, but overall delays are still higher, even with generally healthy Cloud SQL resource utilization. <img width="1630" alt="image" src="https://github.com/sourcegraph/sourcegraph/assets/23356519/91615471-5187-4d15-83e7-5cc94595303c"> Spot-checking the spikes in load in Cloud SQL, it seems that there is a variety of causes for each spike (analytics workloads, Cody Gateway syncs, code intel workloads, gitserver things, `ListSourcegraphDotComIndexableRepos` etc) so I'm chalking this up to "expected". Since this alert is seen firing on a Cloud instance, let's just relax it for now so that it only fires a critical alert on very significant delays.
2024-01-11 01:14:28 +00:00
Generated query for warning alert: `max((sum by (app_name, db_name) (increase(src_pgsql_conns_blocked_seconds{app_name="symbols"}[5m])) / sum by (app_name, db_name) (increase(src_pgsql_conns_waited_for{app_name="symbols"}[5m]))) >= 0.1)`
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
monitoring: relax mean_blocked_seconds_per_conn_request alerts (#59507) https://github.com/sourcegraph/sourcegraph/pull/59284 dramatically reduced the `mean_blocked_seconds_per_conn_request` issues we've been seeing, but overall delays are still higher, even with generally healthy Cloud SQL resource utilization. <img width="1630" alt="image" src="https://github.com/sourcegraph/sourcegraph/assets/23356519/91615471-5187-4d15-83e7-5cc94595303c"> Spot-checking the spikes in load in Cloud SQL, it seems that there is a variety of causes for each spike (analytics workloads, Cody Gateway syncs, code intel workloads, gitserver things, `ListSourcegraphDotComIndexableRepos` etc) so I'm chalking this up to "expected". Since this alert is seen firing on a Cloud instance, let's just relax it for now so that it only fires a critical alert on very significant delays.
2024-01-11 01:14:28 +00:00
Generated query for critical alert: `max((sum by (app_name, db_name) (increase(src_pgsql_conns_blocked_seconds{app_name="symbols"}[5m])) / sum by (app_name, db_name) (increase(src_pgsql_conns_waited_for{app_name="symbols"}[5m]))) >= 0.5)`
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
</details>
symbols: Add DB connection for gitserver client (#39503) This adds a DB connection to be used for the gitserver client in the symbols service. A follow-up PR will make better use of it. We can also use the DB for some optimizations in the future.
2022-07-27 20:25:01 +00:00
<br />
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
## symbols: container_cpu_usage
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
<p class="subtitle">container cpu usage total (1m average) across all cores by instance</p>
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
monitoring: dashboards reference, refresh observability docs (#16939) - New monitoring definition field, 'Interpretation' - New generated dashboards reference - Grafana panels now link to alert solutions and dashboards reference - Refreshed observability docs Co-authored-by: uwedeportivo <534011+uwedeportivo@users.noreply.github.com>
2020-12-23 02:36:41 +00:00
- <span class="badge badge-warning">warning</span> symbols: 99%+ container cpu usage total (1m average) across all cores by instance
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
monitoring: migrate out-of-band alerts to the generator (#12391) Migrates all out-of-band alert rules defined by deploy-sourcegraph to the generator, such that each alert now has a panel as part of a service dashboard like all our other alerts, with a few changes and additions.
2020-07-28 02:17:56 +00:00
- **Kubernetes:** Consider increasing CPU limits in the the relevant `Deployment.yaml`.
- **Docker Compose:** Consider increasing `cpus:` of the symbols container in `docker-compose.yml`.
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#symbols-container-cpu-usage).
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
```json
"observability.silenceAlerts": [
"warning_symbols_container_cpu_usage"
]
```
monitoring: rename Cloud SaaS to IAM (#38268)
2022-07-07 02:13:27 +00:00
<sub>*Managed by the [Sourcegraph Code intelligence team](https://handbook.sourcegraph.com/departments/engineering/teams/code-intelligence).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
Generated query for warning alert: `max((cadvisor_container_cpu_usage_percentage_total{name=~"^symbols.*"}) >= 99)`
</details>
monitoring: improve solutions formatting, add team information (#14878)
2020-10-20 05:11:20 +00:00
<br />
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
## symbols: container_memory_usage
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
<p class="subtitle">container memory usage by instance</p>
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
monitoring: dashboards reference, refresh observability docs (#16939) - New monitoring definition field, 'Interpretation' - New generated dashboards reference - Grafana panels now link to alert solutions and dashboards reference - Refreshed observability docs Co-authored-by: uwedeportivo <534011+uwedeportivo@users.noreply.github.com>
2020-12-23 02:36:41 +00:00
- <span class="badge badge-warning">warning</span> symbols: 99%+ container memory usage by instance
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
- **Kubernetes:** Consider increasing memory limit in relevant `Deployment.yaml`.
- **Docker Compose:** Consider increasing `memory:` of symbols container in `docker-compose.yml`.
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#symbols-container-memory-usage).
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"warning_symbols_container_memory_usage"
]
```
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
monitoring: rename Cloud SaaS to IAM (#38268)
2022-07-07 02:13:27 +00:00
<sub>*Managed by the [Sourcegraph Code intelligence team](https://handbook.sourcegraph.com/departments/engineering/teams/code-intelligence).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
Generated query for warning alert: `max((cadvisor_container_memory_usage_percentage_total{name=~"^symbols.*"}) >= 99)`
</details>
monitoring: improve solutions formatting, add team information (#14878)
2020-10-20 05:11:20 +00:00
<br />
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: improve long-term provisioning alerts aggregation (#12778)
2020-08-08 06:44:33 +00:00
## symbols: provisioning_container_cpu_usage_long_term
monitoring: support multi-bound alerts, add lower-bound warnings for containers (#11082)
2020-06-16 07:09:25 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
<p class="subtitle">container cpu usage total (90th percentile over 1d) across all cores by instance</p>
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
monitoring: support multi-bound alerts, add lower-bound warnings for containers (#11082)
2020-06-16 07:09:25 +00:00
monitoring: dashboards reference, refresh observability docs (#16939) - New monitoring definition field, 'Interpretation' - New generated dashboards reference - Grafana panels now link to alert solutions and dashboards reference - Refreshed observability docs Co-authored-by: uwedeportivo <534011+uwedeportivo@users.noreply.github.com>
2020-12-23 02:36:41 +00:00
- <span class="badge badge-warning">warning</span> symbols: 80%+ container cpu usage total (90th percentile over 1d) across all cores by instance for 336h0m0s
monitoring: support multi-bound alerts, add lower-bound warnings for containers (#11082)
2020-06-16 07:09:25 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: support multi-bound alerts, add lower-bound warnings for containers (#11082)
2020-06-16 07:09:25 +00:00
Remove container_*_usage_7d_low alerts (#16164)
2020-11-25 18:19:33 +00:00
- **Kubernetes:** Consider increasing CPU limits in the `Deployment.yaml` for the symbols service.
- **Docker Compose:** Consider increasing `cpus:` of the symbols container in `docker-compose.yml`.
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#symbols-provisioning-container-cpu-usage-long-term).
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
monitoring: support multi-bound alerts, add lower-bound warnings for containers (#11082)
2020-06-16 07:09:25 +00:00
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
```json
"observability.silenceAlerts": [
monitoring: improve long-term provisioning alerts aggregation (#12778)
2020-08-08 06:44:33 +00:00
"warning_symbols_provisioning_container_cpu_usage_long_term"
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
]
```
monitoring: rename Cloud SaaS to IAM (#38268)
2022-07-07 02:13:27 +00:00
<sub>*Managed by the [Sourcegraph Code intelligence team](https://handbook.sourcegraph.com/departments/engineering/teams/code-intelligence).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
Generated query for warning alert: `max((quantile_over_time(0.9, cadvisor_container_cpu_usage_percentage_total{name=~"^symbols.*"}[1d])) >= 80)`
</details>
monitoring: improve solutions formatting, add team information (#14878)
2020-10-20 05:11:20 +00:00
<br />
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: improve long-term provisioning alerts aggregation (#12778)
2020-08-08 06:44:33 +00:00
## symbols: provisioning_container_memory_usage_long_term
monitoring: support multi-bound alerts, add lower-bound warnings for containers (#11082)
2020-06-16 07:09:25 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
<p class="subtitle">container memory usage (1d maximum) by instance</p>
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
monitoring: support multi-bound alerts, add lower-bound warnings for containers (#11082)
2020-06-16 07:09:25 +00:00
monitoring: dashboards reference, refresh observability docs (#16939) - New monitoring definition field, 'Interpretation' - New generated dashboards reference - Grafana panels now link to alert solutions and dashboards reference - Refreshed observability docs Co-authored-by: uwedeportivo <534011+uwedeportivo@users.noreply.github.com>
2020-12-23 02:36:41 +00:00
- <span class="badge badge-warning">warning</span> symbols: 80%+ container memory usage (1d maximum) by instance for 336h0m0s
monitoring: support multi-bound alerts, add lower-bound warnings for containers (#11082)
2020-06-16 07:09:25 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: support multi-bound alerts, add lower-bound warnings for containers (#11082)
2020-06-16 07:09:25 +00:00
Remove container_*_usage_7d_low alerts (#16164)
2020-11-25 18:19:33 +00:00
- **Kubernetes:** Consider increasing memory limits in the `Deployment.yaml` for the symbols service.
- **Docker Compose:** Consider increasing `memory:` of the symbols container in `docker-compose.yml`.
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#symbols-provisioning-container-memory-usage-long-term).
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
monitoring: improve long-term provisioning alerts aggregation (#12778)
2020-08-08 06:44:33 +00:00
"warning_symbols_provisioning_container_memory_usage_long_term"
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
]
```
monitoring: support multi-bound alerts, add lower-bound warnings for containers (#11082)
2020-06-16 07:09:25 +00:00
monitoring: rename Cloud SaaS to IAM (#38268)
2022-07-07 02:13:27 +00:00
<sub>*Managed by the [Sourcegraph Code intelligence team](https://handbook.sourcegraph.com/departments/engineering/teams/code-intelligence).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
Generated query for warning alert: `max((max_over_time(cadvisor_container_memory_usage_percentage_total{name=~"^symbols.*"}[1d])) >= 80)`
</details>
monitoring: improve solutions formatting, add team information (#14878)
2020-10-20 05:11:20 +00:00
<br />
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: improve long-term provisioning alerts aggregation (#12778)
2020-08-08 06:44:33 +00:00
## symbols: provisioning_container_cpu_usage_short_term
monitoring: support multi-bound alerts, add lower-bound warnings for containers (#11082)
2020-06-16 07:09:25 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
<p class="subtitle">container cpu usage total (5m maximum) across all cores by instance</p>
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
monitoring: support multi-bound alerts, add lower-bound warnings for containers (#11082)
2020-06-16 07:09:25 +00:00
monitoring: dashboards reference, refresh observability docs (#16939) - New monitoring definition field, 'Interpretation' - New generated dashboards reference - Grafana panels now link to alert solutions and dashboards reference - Refreshed observability docs Co-authored-by: uwedeportivo <534011+uwedeportivo@users.noreply.github.com>
2020-12-23 02:36:41 +00:00
- <span class="badge badge-warning">warning</span> symbols: 90%+ container cpu usage total (5m maximum) across all cores by instance for 30m0s
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
- **Kubernetes:** Consider increasing CPU limits in the the relevant `Deployment.yaml`.
- **Docker Compose:** Consider increasing `cpus:` of the symbols container in `docker-compose.yml`.
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#symbols-provisioning-container-cpu-usage-short-term).
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
```json
"observability.silenceAlerts": [
monitoring: improve long-term provisioning alerts aggregation (#12778)
2020-08-08 06:44:33 +00:00
"warning_symbols_provisioning_container_cpu_usage_short_term"
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
]
```
monitoring: rename Cloud SaaS to IAM (#38268)
2022-07-07 02:13:27 +00:00
<sub>*Managed by the [Sourcegraph Code intelligence team](https://handbook.sourcegraph.com/departments/engineering/teams/code-intelligence).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
Generated query for warning alert: `max((max_over_time(cadvisor_container_cpu_usage_percentage_total{name=~"^symbols.*"}[5m])) >= 90)`
</details>
monitoring: improve solutions formatting, add team information (#14878)
2020-10-20 05:11:20 +00:00
<br />
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: improve long-term provisioning alerts aggregation (#12778)
2020-08-08 06:44:33 +00:00
## symbols: provisioning_container_memory_usage_short_term
monitoring: support multi-bound alerts, add lower-bound warnings for containers (#11082)
2020-06-16 07:09:25 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
<p class="subtitle">container memory usage (5m maximum) by instance</p>
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
monitoring: support multi-bound alerts, add lower-bound warnings for containers (#11082)
2020-06-16 07:09:25 +00:00
monitoring: dashboards reference, refresh observability docs (#16939) - New monitoring definition field, 'Interpretation' - New generated dashboards reference - Grafana panels now link to alert solutions and dashboards reference - Refreshed observability docs Co-authored-by: uwedeportivo <534011+uwedeportivo@users.noreply.github.com>
2020-12-23 02:36:41 +00:00
- <span class="badge badge-warning">warning</span> symbols: 90%+ container memory usage (5m maximum) by instance
monitoring: support multi-bound alerts, add lower-bound warnings for containers (#11082)
2020-06-16 07:09:25 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: support multi-bound alerts, add lower-bound warnings for containers (#11082)
2020-06-16 07:09:25 +00:00
- **Kubernetes:** Consider increasing memory limit in relevant `Deployment.yaml`.
- **Docker Compose:** Consider increasing `memory:` of symbols container in `docker-compose.yml`.
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#symbols-provisioning-container-memory-usage-short-term).
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
monitoring: improve long-term provisioning alerts aggregation (#12778)
2020-08-08 06:44:33 +00:00
"warning_symbols_provisioning_container_memory_usage_short_term"
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
]
```
monitoring: rename Cloud SaaS to IAM (#38268)
2022-07-07 02:13:27 +00:00
<sub>*Managed by the [Sourcegraph Code intelligence team](https://handbook.sourcegraph.com/departments/engineering/teams/code-intelligence).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
Generated query for warning alert: `max((max_over_time(cadvisor_container_memory_usage_percentage_total{name=~"^symbols.*"}[5m])) >= 90)`
</details>
monitoring: improve solutions formatting, add team information (#14878)
2020-10-20 05:11:20 +00:00
<br />
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: improve containers metrics (#34808) part of - https://github.com/sourcegraph/sourcegraph/issues/33437 - https://github.com/sourcegraph/sourcegraph/issues/33438 This PR adds the following - A shared observable metric to indicate how many times a container (or child processes) has been oom kill - It's added to provisioning indicators since it's an indicator of underprovisioned resources - A new dashboard that shows resources usage of all containers at a glance - It contains two groups, one for all containers and one for containers that may have potential scaling issues - the first group is extremely noisy so it's hidden by default - the second group provides a more focused experience for site-admin to quickly find problematic resources <!-- > ~it looks like the alert table will be rendered regardless, which is pretty annoying (I guess the implementation of the monitoring stack is based on the assumption that one service/container will have its dashboard)~ __Questions & Notes__ - ~Should I make the new dashboard filterable by container name? like I only want to see the graph of `frontend`. Is that something that's valuable?~ - ~Any other metrics you all would like to see?~ - graph for total available resources on the VM compared to usage from https://github.com/sourcegraph/sourcegraph/issues/33437 is not really doable since we do not have `node-exporter` in anyone of our distro. We would have to tackle this from https://github.com/sourcegraph/customer/issues/806 --> ## Test plan connect to `demo`'s prom ```sh gcloud compute start-iap-tunnel default-$NEW_DEPLOYMENT-instance 9090 --local-host-port=localhost:4445 --zone us-central1-f --project $PROJECT_PREFIX-$CUSTOMER ``` or use dogfood ```sh kubectl port-forward svc/prometheus 4445:30090 ``` Update `dev/grafana/all/datasources.yaml` ```yml # Configuration for all non-Linux platforms. apiVersion: 1 datasources: - name: Prometheus type: prometheus access: proxy url: http://docker.for.mac.localhost:4445 # use the remote prometheus instance on demo instead of the local one isDefault: true editable: false - name: Jaeger type: jaeger access: proxy url: http://host.docker.internal:16686/-/debug/jaeger - name: Loki type: loki access: proxy url: http://host.docker.internal:3100 ``` Start monitoring stack ```sh sg run grafana ``` see screenshot > the first group is hidden by default ![CleanShot 2022-05-04 at 10 48 26](https://user-images.githubusercontent.com/8373004/166748470-22460552-6e51-4b29-a26b-4c5adbe1c404.png) ![CleanShot 2022-05-04 at 10 48 03](https://user-images.githubusercontent.com/8373004/166748396-1d410c35-e5fe-45e2-83bd-418d845413cb.png) ![CleanShot 2022-05-04 at 11 25 13](https://user-images.githubusercontent.com/8373004/166801266-21e51b68-65fd-4cb7-97de-83520e31e498.png)
2022-05-09 23:56:47 +00:00
## symbols: container_oomkill_events_total
<p class="subtitle">container OOMKILL events total by instance</p>
**Descriptions**
- <span class="badge badge-warning">warning</span> symbols: 1+ container OOMKILL events total by instance
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: improve containers metrics (#34808) part of - https://github.com/sourcegraph/sourcegraph/issues/33437 - https://github.com/sourcegraph/sourcegraph/issues/33438 This PR adds the following - A shared observable metric to indicate how many times a container (or child processes) has been oom kill - It's added to provisioning indicators since it's an indicator of underprovisioned resources - A new dashboard that shows resources usage of all containers at a glance - It contains two groups, one for all containers and one for containers that may have potential scaling issues - the first group is extremely noisy so it's hidden by default - the second group provides a more focused experience for site-admin to quickly find problematic resources <!-- > ~it looks like the alert table will be rendered regardless, which is pretty annoying (I guess the implementation of the monitoring stack is based on the assumption that one service/container will have its dashboard)~ __Questions & Notes__ - ~Should I make the new dashboard filterable by container name? like I only want to see the graph of `frontend`. Is that something that's valuable?~ - ~Any other metrics you all would like to see?~ - graph for total available resources on the VM compared to usage from https://github.com/sourcegraph/sourcegraph/issues/33437 is not really doable since we do not have `node-exporter` in anyone of our distro. We would have to tackle this from https://github.com/sourcegraph/customer/issues/806 --> ## Test plan connect to `demo`'s prom ```sh gcloud compute start-iap-tunnel default-$NEW_DEPLOYMENT-instance 9090 --local-host-port=localhost:4445 --zone us-central1-f --project $PROJECT_PREFIX-$CUSTOMER ``` or use dogfood ```sh kubectl port-forward svc/prometheus 4445:30090 ``` Update `dev/grafana/all/datasources.yaml` ```yml # Configuration for all non-Linux platforms. apiVersion: 1 datasources: - name: Prometheus type: prometheus access: proxy url: http://docker.for.mac.localhost:4445 # use the remote prometheus instance on demo instead of the local one isDefault: true editable: false - name: Jaeger type: jaeger access: proxy url: http://host.docker.internal:16686/-/debug/jaeger - name: Loki type: loki access: proxy url: http://host.docker.internal:3100 ``` Start monitoring stack ```sh sg run grafana ``` see screenshot > the first group is hidden by default ![CleanShot 2022-05-04 at 10 48 26](https://user-images.githubusercontent.com/8373004/166748470-22460552-6e51-4b29-a26b-4c5adbe1c404.png) ![CleanShot 2022-05-04 at 10 48 03](https://user-images.githubusercontent.com/8373004/166748396-1d410c35-e5fe-45e2-83bd-418d845413cb.png) ![CleanShot 2022-05-04 at 11 25 13](https://user-images.githubusercontent.com/8373004/166801266-21e51b68-65fd-4cb7-97de-83520e31e498.png)
2022-05-09 23:56:47 +00:00
- **Kubernetes:** Consider increasing memory limit in relevant `Deployment.yaml`.
- **Docker Compose:** Consider increasing `memory:` of symbols container in `docker-compose.yml`.
- More help interpreting this metric is available in the [dashboards reference](./dashboards.md#symbols-container-oomkill-events-total).
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"warning_symbols_container_oomkill_events_total"
]
```
monitoring: rename Cloud SaaS to IAM (#38268)
2022-07-07 02:13:27 +00:00
<sub>*Managed by the [Sourcegraph Code intelligence team](https://handbook.sourcegraph.com/departments/engineering/teams/code-intelligence).*</sub>
monitoring: improve containers metrics (#34808) part of - https://github.com/sourcegraph/sourcegraph/issues/33437 - https://github.com/sourcegraph/sourcegraph/issues/33438 This PR adds the following - A shared observable metric to indicate how many times a container (or child processes) has been oom kill - It's added to provisioning indicators since it's an indicator of underprovisioned resources - A new dashboard that shows resources usage of all containers at a glance - It contains two groups, one for all containers and one for containers that may have potential scaling issues - the first group is extremely noisy so it's hidden by default - the second group provides a more focused experience for site-admin to quickly find problematic resources <!-- > ~it looks like the alert table will be rendered regardless, which is pretty annoying (I guess the implementation of the monitoring stack is based on the assumption that one service/container will have its dashboard)~ __Questions & Notes__ - ~Should I make the new dashboard filterable by container name? like I only want to see the graph of `frontend`. Is that something that's valuable?~ - ~Any other metrics you all would like to see?~ - graph for total available resources on the VM compared to usage from https://github.com/sourcegraph/sourcegraph/issues/33437 is not really doable since we do not have `node-exporter` in anyone of our distro. We would have to tackle this from https://github.com/sourcegraph/customer/issues/806 --> ## Test plan connect to `demo`'s prom ```sh gcloud compute start-iap-tunnel default-$NEW_DEPLOYMENT-instance 9090 --local-host-port=localhost:4445 --zone us-central1-f --project $PROJECT_PREFIX-$CUSTOMER ``` or use dogfood ```sh kubectl port-forward svc/prometheus 4445:30090 ``` Update `dev/grafana/all/datasources.yaml` ```yml # Configuration for all non-Linux platforms. apiVersion: 1 datasources: - name: Prometheus type: prometheus access: proxy url: http://docker.for.mac.localhost:4445 # use the remote prometheus instance on demo instead of the local one isDefault: true editable: false - name: Jaeger type: jaeger access: proxy url: http://host.docker.internal:16686/-/debug/jaeger - name: Loki type: loki access: proxy url: http://host.docker.internal:3100 ``` Start monitoring stack ```sh sg run grafana ``` see screenshot > the first group is hidden by default ![CleanShot 2022-05-04 at 10 48 26](https://user-images.githubusercontent.com/8373004/166748470-22460552-6e51-4b29-a26b-4c5adbe1c404.png) ![CleanShot 2022-05-04 at 10 48 03](https://user-images.githubusercontent.com/8373004/166748396-1d410c35-e5fe-45e2-83bd-418d845413cb.png) ![CleanShot 2022-05-04 at 11 25 13](https://user-images.githubusercontent.com/8373004/166801266-21e51b68-65fd-4cb7-97de-83520e31e498.png)
2022-05-09 23:56:47 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
monitoring: extract into a submodule (#45786) This change extracts `monitoring` into a submodule for import in `sourcegraph/controller` (https://github.com/sourcegraph/controller/pull/195) so that we can generate dashboards for Cloud instances. These steps were required: 1. Initialize a `go.mod` in `monitoring` 2. Extract `dev/sg/internal/cliutil` into `lib` to avoid illegal imports from `monitoring` 3. Add local replaces to both `sourcegraph/sourcegraph` and `monitoring` 4. `go mod tidy` on all submodules 5. Update `go generate ./monitoring` commands to use `sg`, since the `go generate` command no longer works 6. Update `grafana/build.sh`, `prometheus/build.sh` to build the submodule 7. Amend linters to check for multiple `go.mod` files and ban imports of `github.com/sourcegraph/sourcegraph` 8. Update `sg generate go` to run in directories rather than from root The only caveat is that if you use VS code, you will now need to open `monitoring` in a separate workspace or similar, like with `lib`. Co-authored-by: Joe Chen <joe@sourcegraph.com>
2022-12-19 17:49:25 +00:00
Generated query for warning alert: `max((max by (name) (container_oom_events_total{name=~"^symbols.*"})) >= 1)`
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
</details>
monitoring: improve containers metrics (#34808) part of - https://github.com/sourcegraph/sourcegraph/issues/33437 - https://github.com/sourcegraph/sourcegraph/issues/33438 This PR adds the following - A shared observable metric to indicate how many times a container (or child processes) has been oom kill - It's added to provisioning indicators since it's an indicator of underprovisioned resources - A new dashboard that shows resources usage of all containers at a glance - It contains two groups, one for all containers and one for containers that may have potential scaling issues - the first group is extremely noisy so it's hidden by default - the second group provides a more focused experience for site-admin to quickly find problematic resources <!-- > ~it looks like the alert table will be rendered regardless, which is pretty annoying (I guess the implementation of the monitoring stack is based on the assumption that one service/container will have its dashboard)~ __Questions & Notes__ - ~Should I make the new dashboard filterable by container name? like I only want to see the graph of `frontend`. Is that something that's valuable?~ - ~Any other metrics you all would like to see?~ - graph for total available resources on the VM compared to usage from https://github.com/sourcegraph/sourcegraph/issues/33437 is not really doable since we do not have `node-exporter` in anyone of our distro. We would have to tackle this from https://github.com/sourcegraph/customer/issues/806 --> ## Test plan connect to `demo`'s prom ```sh gcloud compute start-iap-tunnel default-$NEW_DEPLOYMENT-instance 9090 --local-host-port=localhost:4445 --zone us-central1-f --project $PROJECT_PREFIX-$CUSTOMER ``` or use dogfood ```sh kubectl port-forward svc/prometheus 4445:30090 ``` Update `dev/grafana/all/datasources.yaml` ```yml # Configuration for all non-Linux platforms. apiVersion: 1 datasources: - name: Prometheus type: prometheus access: proxy url: http://docker.for.mac.localhost:4445 # use the remote prometheus instance on demo instead of the local one isDefault: true editable: false - name: Jaeger type: jaeger access: proxy url: http://host.docker.internal:16686/-/debug/jaeger - name: Loki type: loki access: proxy url: http://host.docker.internal:3100 ``` Start monitoring stack ```sh sg run grafana ``` see screenshot > the first group is hidden by default ![CleanShot 2022-05-04 at 10 48 26](https://user-images.githubusercontent.com/8373004/166748470-22460552-6e51-4b29-a26b-4c5adbe1c404.png) ![CleanShot 2022-05-04 at 10 48 03](https://user-images.githubusercontent.com/8373004/166748396-1d410c35-e5fe-45e2-83bd-418d845413cb.png) ![CleanShot 2022-05-04 at 11 25 13](https://user-images.githubusercontent.com/8373004/166801266-21e51b68-65fd-4cb7-97de-83520e31e498.png)
2022-05-09 23:56:47 +00:00
<br />
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
## symbols: go_goroutines
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
<p class="subtitle">maximum active goroutines</p>
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
monitoring: dashboards reference, refresh observability docs (#16939) - New monitoring definition field, 'Interpretation' - New generated dashboards reference - Grafana panels now link to alert solutions and dashboards reference - Refreshed observability docs Co-authored-by: uwedeportivo <534011+uwedeportivo@users.noreply.github.com>
2020-12-23 02:36:41 +00:00
- <span class="badge badge-warning">warning</span> symbols: 10000+ maximum active goroutines for 10m0s
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- More help interpreting this metric is available in the [dashboards reference](./dashboards.md#symbols-go-goroutines).
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"warning_symbols_go_goroutines"
]
```
monitoring: rename Cloud SaaS to IAM (#38268)
2022-07-07 02:13:27 +00:00
<sub>*Managed by the [Sourcegraph Code intelligence team](https://handbook.sourcegraph.com/departments/engineering/teams/code-intelligence).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
monitoring: extract into a submodule (#45786) This change extracts `monitoring` into a submodule for import in `sourcegraph/controller` (https://github.com/sourcegraph/controller/pull/195) so that we can generate dashboards for Cloud instances. These steps were required: 1. Initialize a `go.mod` in `monitoring` 2. Extract `dev/sg/internal/cliutil` into `lib` to avoid illegal imports from `monitoring` 3. Add local replaces to both `sourcegraph/sourcegraph` and `monitoring` 4. `go mod tidy` on all submodules 5. Update `go generate ./monitoring` commands to use `sg`, since the `go generate` command no longer works 6. Update `grafana/build.sh`, `prometheus/build.sh` to build the submodule 7. Amend linters to check for multiple `go.mod` files and ban imports of `github.com/sourcegraph/sourcegraph` 8. Update `sg generate go` to run in directories rather than from root The only caveat is that if you use VS code, you will now need to open `monitoring` in a separate workspace or similar, like with `lib`. Co-authored-by: Joe Chen <joe@sourcegraph.com>
2022-12-19 17:49:25 +00:00
Generated query for warning alert: `max((max by (instance) (go_goroutines{job=~".*symbols"})) >= 10000)`
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
</details>
monitoring: improve solutions formatting, add team information (#14878)
2020-10-20 05:11:20 +00:00
<br />
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
## symbols: go_gc_duration_seconds
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
<p class="subtitle">maximum go garbage collection duration</p>
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
monitoring: dashboards reference, refresh observability docs (#16939) - New monitoring definition field, 'Interpretation' - New generated dashboards reference - Grafana panels now link to alert solutions and dashboards reference - Refreshed observability docs Co-authored-by: uwedeportivo <534011+uwedeportivo@users.noreply.github.com>
2020-12-23 02:36:41 +00:00
- <span class="badge badge-warning">warning</span> symbols: 2s+ maximum go garbage collection duration
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#symbols-go-gc-duration-seconds).
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"warning_symbols_go_gc_duration_seconds"
]
```
monitoring: rename Cloud SaaS to IAM (#38268)
2022-07-07 02:13:27 +00:00
<sub>*Managed by the [Sourcegraph Code intelligence team](https://handbook.sourcegraph.com/departments/engineering/teams/code-intelligence).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
monitoring: extract into a submodule (#45786) This change extracts `monitoring` into a submodule for import in `sourcegraph/controller` (https://github.com/sourcegraph/controller/pull/195) so that we can generate dashboards for Cloud instances. These steps were required: 1. Initialize a `go.mod` in `monitoring` 2. Extract `dev/sg/internal/cliutil` into `lib` to avoid illegal imports from `monitoring` 3. Add local replaces to both `sourcegraph/sourcegraph` and `monitoring` 4. `go mod tidy` on all submodules 5. Update `go generate ./monitoring` commands to use `sg`, since the `go generate` command no longer works 6. Update `grafana/build.sh`, `prometheus/build.sh` to build the submodule 7. Amend linters to check for multiple `go.mod` files and ban imports of `github.com/sourcegraph/sourcegraph` 8. Update `sg generate go` to run in directories rather than from root The only caveat is that if you use VS code, you will now need to open `monitoring` in a separate workspace or similar, like with `lib`. Co-authored-by: Joe Chen <joe@sourcegraph.com>
2022-12-19 17:49:25 +00:00
Generated query for warning alert: `max((max by (instance) (go_gc_duration_seconds{job=~".*symbols"})) >= 2)`
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
</details>
monitoring: improve solutions formatting, add team information (#14878)
2020-10-20 05:11:20 +00:00
<br />
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
## symbols: pods_available_percentage
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
<p class="subtitle">percentage pods available</p>
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
monitoring: dashboards reference, refresh observability docs (#16939) - New monitoring definition field, 'Interpretation' - New generated dashboards reference - Grafana panels now link to alert solutions and dashboards reference - Refreshed observability docs Co-authored-by: uwedeportivo <534011+uwedeportivo@users.noreply.github.com>
2020-12-23 02:36:41 +00:00
- <span class="badge badge-critical">critical</span> symbols: less than 90% percentage pods available for 10m0s
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
monitoring: forbid critical alerts with no solutions (#36423) Adds the restriction that critical alerts must have solutions defined as a hard requirement within the generator, and fixes all the outstanding issues: - Add PossibleSolutions for the pods_available_percentage and pg_up critical alerts (we have a similar solution doc in the container_missing alert) - Removes all the other critical alerts without possible solutions Co-authored-by: Michael Lin <mlzc@hey.com>
2022-06-02 15:51:43 +00:00
- Determine if the pod was OOM killed using `kubectl describe pod symbols` (look for `OOMKilled: true`) and, if so, consider increasing the memory limit in the relevant `Deployment.yaml`.
- Check the logs before the container restarted to see if there are `panic:` messages or similar using `kubectl logs -p symbols`.
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#symbols-pods-available-percentage).
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"critical_symbols_pods_available_percentage"
]
```
monitoring: rename Cloud SaaS to IAM (#38268)
2022-07-07 02:13:27 +00:00
<sub>*Managed by the [Sourcegraph Code intelligence team](https://handbook.sourcegraph.com/departments/engineering/teams/code-intelligence).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
monitoring: extract into a submodule (#45786) This change extracts `monitoring` into a submodule for import in `sourcegraph/controller` (https://github.com/sourcegraph/controller/pull/195) so that we can generate dashboards for Cloud instances. These steps were required: 1. Initialize a `go.mod` in `monitoring` 2. Extract `dev/sg/internal/cliutil` into `lib` to avoid illegal imports from `monitoring` 3. Add local replaces to both `sourcegraph/sourcegraph` and `monitoring` 4. `go mod tidy` on all submodules 5. Update `go generate ./monitoring` commands to use `sg`, since the `go generate` command no longer works 6. Update `grafana/build.sh`, `prometheus/build.sh` to build the submodule 7. Amend linters to check for multiple `go.mod` files and ban imports of `github.com/sourcegraph/sourcegraph` 8. Update `sg generate go` to run in directories rather than from root The only caveat is that if you use VS code, you will now need to open `monitoring` in a separate workspace or similar, like with `lib`. Co-authored-by: Joe Chen <joe@sourcegraph.com>
2022-12-19 17:49:25 +00:00
Generated query for critical alert: `min((sum by (app) (up{app=~".*symbols"}) / count by (app) (up{app=~".*symbols"}) * 100) <= 90)`
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
</details>
monitoring: improve solutions formatting, add team information (#14878)
2020-10-20 05:11:20 +00:00
<br />
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
## syntect-server: container_cpu_usage
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
<p class="subtitle">container cpu usage total (1m average) across all cores by instance</p>
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
monitoring: dashboards reference, refresh observability docs (#16939) - New monitoring definition field, 'Interpretation' - New generated dashboards reference - Grafana panels now link to alert solutions and dashboards reference - Refreshed observability docs Co-authored-by: uwedeportivo <534011+uwedeportivo@users.noreply.github.com>
2020-12-23 02:36:41 +00:00
- <span class="badge badge-warning">warning</span> syntect-server: 99%+ container cpu usage total (1m average) across all cores by instance
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
monitoring: migrate out-of-band alerts to the generator (#12391) Migrates all out-of-band alert rules defined by deploy-sourcegraph to the generator, such that each alert now has a panel as part of a service dashboard like all our other alerts, with a few changes and additions.
2020-07-28 02:17:56 +00:00
- **Kubernetes:** Consider increasing CPU limits in the the relevant `Deployment.yaml`.
- **Docker Compose:** Consider increasing `cpus:` of the syntect-server container in `docker-compose.yml`.
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#syntect-server-container-cpu-usage).
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
```json
"observability.silenceAlerts": [
"warning_syntect-server_container_cpu_usage"
]
```
monitoring: test owners for valid Opsgenie teams and handbook pages (#59251) In INC-264 it seems that certain alerts - such as [zoekt: less than 90% percentage pods available for 10m0s](https://opsg.in/a/i/sourcegraph/178a626f-0f28-4295-bee9-84da988bb473-1703759057681) - don't seem to end up going anywhere because the ObservableOwner is defunct. This change adds _opt-in_ testing to report: 1. How many owners have valid Opsgenie teams 2. How many owners have valid handbook pages In addition, we collect ObservableOwners that pass the test and use it to generate configuration for `site.json` in Sourcegraph.com: https://github.com/sourcegraph/deploy-sourcegraph-cloud/pull/18338 - this helps ensure the list is valid and not deceptively high-coverage. The results are not great, but **enforcing** that owners are valid isn't currently in scope: ``` 6/10 ObservableOwners do not have valid Opsgenie teams 3/10 ObservableOwners do not point to valid handbook pages ``` I also removed some defunct/unused functionality/owners. ## Test plan To run these tests: ``` export OPSGENIE_API_KEY="..." go test -timeout 30s github.com/sourcegraph/sourcegraph/monitoring/monitoring -update -online ```
2023-12-29 22:07:35 +00:00
<sub>*Managed by the [Sourcegraph Infrastructure Org team](https://handbook.sourcegraph.com/departments/engineering/infrastructure).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
Generated query for warning alert: `max((cadvisor_container_cpu_usage_percentage_total{name=~"^syntect-server.*"}) >= 99)`
</details>
monitoring: improve solutions formatting, add team information (#14878)
2020-10-20 05:11:20 +00:00
<br />
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
## syntect-server: container_memory_usage
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
<p class="subtitle">container memory usage by instance</p>
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
monitoring: dashboards reference, refresh observability docs (#16939) - New monitoring definition field, 'Interpretation' - New generated dashboards reference - Grafana panels now link to alert solutions and dashboards reference - Refreshed observability docs Co-authored-by: uwedeportivo <534011+uwedeportivo@users.noreply.github.com>
2020-12-23 02:36:41 +00:00
- <span class="badge badge-warning">warning</span> syntect-server: 99%+ container memory usage by instance
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
- **Kubernetes:** Consider increasing memory limit in relevant `Deployment.yaml`.
- **Docker Compose:** Consider increasing `memory:` of syntect-server container in `docker-compose.yml`.
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#syntect-server-container-memory-usage).
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"warning_syntect-server_container_memory_usage"
]
```
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
monitoring: test owners for valid Opsgenie teams and handbook pages (#59251) In INC-264 it seems that certain alerts - such as [zoekt: less than 90% percentage pods available for 10m0s](https://opsg.in/a/i/sourcegraph/178a626f-0f28-4295-bee9-84da988bb473-1703759057681) - don't seem to end up going anywhere because the ObservableOwner is defunct. This change adds _opt-in_ testing to report: 1. How many owners have valid Opsgenie teams 2. How many owners have valid handbook pages In addition, we collect ObservableOwners that pass the test and use it to generate configuration for `site.json` in Sourcegraph.com: https://github.com/sourcegraph/deploy-sourcegraph-cloud/pull/18338 - this helps ensure the list is valid and not deceptively high-coverage. The results are not great, but **enforcing** that owners are valid isn't currently in scope: ``` 6/10 ObservableOwners do not have valid Opsgenie teams 3/10 ObservableOwners do not point to valid handbook pages ``` I also removed some defunct/unused functionality/owners. ## Test plan To run these tests: ``` export OPSGENIE_API_KEY="..." go test -timeout 30s github.com/sourcegraph/sourcegraph/monitoring/monitoring -update -online ```
2023-12-29 22:07:35 +00:00
<sub>*Managed by the [Sourcegraph Infrastructure Org team](https://handbook.sourcegraph.com/departments/engineering/infrastructure).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
Generated query for warning alert: `max((cadvisor_container_memory_usage_percentage_total{name=~"^syntect-server.*"}) >= 99)`
</details>
monitoring: improve solutions formatting, add team information (#14878)
2020-10-20 05:11:20 +00:00
<br />
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: improve long-term provisioning alerts aggregation (#12778)
2020-08-08 06:44:33 +00:00
## syntect-server: provisioning_container_cpu_usage_long_term
monitoring: support multi-bound alerts, add lower-bound warnings for containers (#11082)
2020-06-16 07:09:25 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
<p class="subtitle">container cpu usage total (90th percentile over 1d) across all cores by instance</p>
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
monitoring: support multi-bound alerts, add lower-bound warnings for containers (#11082)
2020-06-16 07:09:25 +00:00
monitoring: dashboards reference, refresh observability docs (#16939) - New monitoring definition field, 'Interpretation' - New generated dashboards reference - Grafana panels now link to alert solutions and dashboards reference - Refreshed observability docs Co-authored-by: uwedeportivo <534011+uwedeportivo@users.noreply.github.com>
2020-12-23 02:36:41 +00:00
- <span class="badge badge-warning">warning</span> syntect-server: 80%+ container cpu usage total (90th percentile over 1d) across all cores by instance for 336h0m0s
monitoring: support multi-bound alerts, add lower-bound warnings for containers (#11082)
2020-06-16 07:09:25 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: support multi-bound alerts, add lower-bound warnings for containers (#11082)
2020-06-16 07:09:25 +00:00
Remove container_*_usage_7d_low alerts (#16164)
2020-11-25 18:19:33 +00:00
- **Kubernetes:** Consider increasing CPU limits in the `Deployment.yaml` for the syntect-server service.
- **Docker Compose:** Consider increasing `cpus:` of the syntect-server container in `docker-compose.yml`.
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#syntect-server-provisioning-container-cpu-usage-long-term).
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
monitoring: support multi-bound alerts, add lower-bound warnings for containers (#11082)
2020-06-16 07:09:25 +00:00
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
```json
"observability.silenceAlerts": [
monitoring: improve long-term provisioning alerts aggregation (#12778)
2020-08-08 06:44:33 +00:00
"warning_syntect-server_provisioning_container_cpu_usage_long_term"
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
]
```
monitoring: test owners for valid Opsgenie teams and handbook pages (#59251) In INC-264 it seems that certain alerts - such as [zoekt: less than 90% percentage pods available for 10m0s](https://opsg.in/a/i/sourcegraph/178a626f-0f28-4295-bee9-84da988bb473-1703759057681) - don't seem to end up going anywhere because the ObservableOwner is defunct. This change adds _opt-in_ testing to report: 1. How many owners have valid Opsgenie teams 2. How many owners have valid handbook pages In addition, we collect ObservableOwners that pass the test and use it to generate configuration for `site.json` in Sourcegraph.com: https://github.com/sourcegraph/deploy-sourcegraph-cloud/pull/18338 - this helps ensure the list is valid and not deceptively high-coverage. The results are not great, but **enforcing** that owners are valid isn't currently in scope: ``` 6/10 ObservableOwners do not have valid Opsgenie teams 3/10 ObservableOwners do not point to valid handbook pages ``` I also removed some defunct/unused functionality/owners. ## Test plan To run these tests: ``` export OPSGENIE_API_KEY="..." go test -timeout 30s github.com/sourcegraph/sourcegraph/monitoring/monitoring -update -online ```
2023-12-29 22:07:35 +00:00
<sub>*Managed by the [Sourcegraph Infrastructure Org team](https://handbook.sourcegraph.com/departments/engineering/infrastructure).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
Generated query for warning alert: `max((quantile_over_time(0.9, cadvisor_container_cpu_usage_percentage_total{name=~"^syntect-server.*"}[1d])) >= 80)`
</details>
monitoring: improve solutions formatting, add team information (#14878)
2020-10-20 05:11:20 +00:00
<br />
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: improve long-term provisioning alerts aggregation (#12778)
2020-08-08 06:44:33 +00:00
## syntect-server: provisioning_container_memory_usage_long_term
monitoring: support multi-bound alerts, add lower-bound warnings for containers (#11082)
2020-06-16 07:09:25 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
<p class="subtitle">container memory usage (1d maximum) by instance</p>
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
monitoring: support multi-bound alerts, add lower-bound warnings for containers (#11082)
2020-06-16 07:09:25 +00:00
monitoring: dashboards reference, refresh observability docs (#16939) - New monitoring definition field, 'Interpretation' - New generated dashboards reference - Grafana panels now link to alert solutions and dashboards reference - Refreshed observability docs Co-authored-by: uwedeportivo <534011+uwedeportivo@users.noreply.github.com>
2020-12-23 02:36:41 +00:00
- <span class="badge badge-warning">warning</span> syntect-server: 80%+ container memory usage (1d maximum) by instance for 336h0m0s
monitoring: support multi-bound alerts, add lower-bound warnings for containers (#11082)
2020-06-16 07:09:25 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: support multi-bound alerts, add lower-bound warnings for containers (#11082)
2020-06-16 07:09:25 +00:00
Remove container_*_usage_7d_low alerts (#16164)
2020-11-25 18:19:33 +00:00
- **Kubernetes:** Consider increasing memory limits in the `Deployment.yaml` for the syntect-server service.
- **Docker Compose:** Consider increasing `memory:` of the syntect-server container in `docker-compose.yml`.
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#syntect-server-provisioning-container-memory-usage-long-term).
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
monitoring: improve long-term provisioning alerts aggregation (#12778)
2020-08-08 06:44:33 +00:00
"warning_syntect-server_provisioning_container_memory_usage_long_term"
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
]
```
monitoring: support multi-bound alerts, add lower-bound warnings for containers (#11082)
2020-06-16 07:09:25 +00:00
monitoring: test owners for valid Opsgenie teams and handbook pages (#59251) In INC-264 it seems that certain alerts - such as [zoekt: less than 90% percentage pods available for 10m0s](https://opsg.in/a/i/sourcegraph/178a626f-0f28-4295-bee9-84da988bb473-1703759057681) - don't seem to end up going anywhere because the ObservableOwner is defunct. This change adds _opt-in_ testing to report: 1. How many owners have valid Opsgenie teams 2. How many owners have valid handbook pages In addition, we collect ObservableOwners that pass the test and use it to generate configuration for `site.json` in Sourcegraph.com: https://github.com/sourcegraph/deploy-sourcegraph-cloud/pull/18338 - this helps ensure the list is valid and not deceptively high-coverage. The results are not great, but **enforcing** that owners are valid isn't currently in scope: ``` 6/10 ObservableOwners do not have valid Opsgenie teams 3/10 ObservableOwners do not point to valid handbook pages ``` I also removed some defunct/unused functionality/owners. ## Test plan To run these tests: ``` export OPSGENIE_API_KEY="..." go test -timeout 30s github.com/sourcegraph/sourcegraph/monitoring/monitoring -update -online ```
2023-12-29 22:07:35 +00:00
<sub>*Managed by the [Sourcegraph Infrastructure Org team](https://handbook.sourcegraph.com/departments/engineering/infrastructure).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
Generated query for warning alert: `max((max_over_time(cadvisor_container_memory_usage_percentage_total{name=~"^syntect-server.*"}[1d])) >= 80)`
</details>
monitoring: improve solutions formatting, add team information (#14878)
2020-10-20 05:11:20 +00:00
<br />
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: improve long-term provisioning alerts aggregation (#12778)
2020-08-08 06:44:33 +00:00
## syntect-server: provisioning_container_cpu_usage_short_term
monitoring: support multi-bound alerts, add lower-bound warnings for containers (#11082)
2020-06-16 07:09:25 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
<p class="subtitle">container cpu usage total (5m maximum) across all cores by instance</p>
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
monitoring: support multi-bound alerts, add lower-bound warnings for containers (#11082)
2020-06-16 07:09:25 +00:00
monitoring: dashboards reference, refresh observability docs (#16939) - New monitoring definition field, 'Interpretation' - New generated dashboards reference - Grafana panels now link to alert solutions and dashboards reference - Refreshed observability docs Co-authored-by: uwedeportivo <534011+uwedeportivo@users.noreply.github.com>
2020-12-23 02:36:41 +00:00
- <span class="badge badge-warning">warning</span> syntect-server: 90%+ container cpu usage total (5m maximum) across all cores by instance for 30m0s
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
- **Kubernetes:** Consider increasing CPU limits in the the relevant `Deployment.yaml`.
- **Docker Compose:** Consider increasing `cpus:` of the syntect-server container in `docker-compose.yml`.
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#syntect-server-provisioning-container-cpu-usage-short-term).
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
```json
"observability.silenceAlerts": [
monitoring: improve long-term provisioning alerts aggregation (#12778)
2020-08-08 06:44:33 +00:00
"warning_syntect-server_provisioning_container_cpu_usage_short_term"
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
]
```
monitoring: test owners for valid Opsgenie teams and handbook pages (#59251) In INC-264 it seems that certain alerts - such as [zoekt: less than 90% percentage pods available for 10m0s](https://opsg.in/a/i/sourcegraph/178a626f-0f28-4295-bee9-84da988bb473-1703759057681) - don't seem to end up going anywhere because the ObservableOwner is defunct. This change adds _opt-in_ testing to report: 1. How many owners have valid Opsgenie teams 2. How many owners have valid handbook pages In addition, we collect ObservableOwners that pass the test and use it to generate configuration for `site.json` in Sourcegraph.com: https://github.com/sourcegraph/deploy-sourcegraph-cloud/pull/18338 - this helps ensure the list is valid and not deceptively high-coverage. The results are not great, but **enforcing** that owners are valid isn't currently in scope: ``` 6/10 ObservableOwners do not have valid Opsgenie teams 3/10 ObservableOwners do not point to valid handbook pages ``` I also removed some defunct/unused functionality/owners. ## Test plan To run these tests: ``` export OPSGENIE_API_KEY="..." go test -timeout 30s github.com/sourcegraph/sourcegraph/monitoring/monitoring -update -online ```
2023-12-29 22:07:35 +00:00
<sub>*Managed by the [Sourcegraph Infrastructure Org team](https://handbook.sourcegraph.com/departments/engineering/infrastructure).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
Generated query for warning alert: `max((max_over_time(cadvisor_container_cpu_usage_percentage_total{name=~"^syntect-server.*"}[5m])) >= 90)`
</details>
monitoring: improve solutions formatting, add team information (#14878)
2020-10-20 05:11:20 +00:00
<br />
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: improve long-term provisioning alerts aggregation (#12778)
2020-08-08 06:44:33 +00:00
## syntect-server: provisioning_container_memory_usage_short_term
monitoring: support multi-bound alerts, add lower-bound warnings for containers (#11082)
2020-06-16 07:09:25 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
<p class="subtitle">container memory usage (5m maximum) by instance</p>
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
monitoring: support multi-bound alerts, add lower-bound warnings for containers (#11082)
2020-06-16 07:09:25 +00:00
monitoring: dashboards reference, refresh observability docs (#16939) - New monitoring definition field, 'Interpretation' - New generated dashboards reference - Grafana panels now link to alert solutions and dashboards reference - Refreshed observability docs Co-authored-by: uwedeportivo <534011+uwedeportivo@users.noreply.github.com>
2020-12-23 02:36:41 +00:00
- <span class="badge badge-warning">warning</span> syntect-server: 90%+ container memory usage (5m maximum) by instance
monitoring: support multi-bound alerts, add lower-bound warnings for containers (#11082)
2020-06-16 07:09:25 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: support multi-bound alerts, add lower-bound warnings for containers (#11082)
2020-06-16 07:09:25 +00:00
- **Kubernetes:** Consider increasing memory limit in relevant `Deployment.yaml`.
- **Docker Compose:** Consider increasing `memory:` of syntect-server container in `docker-compose.yml`.
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#syntect-server-provisioning-container-memory-usage-short-term).
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
monitoring: improve long-term provisioning alerts aggregation (#12778)
2020-08-08 06:44:33 +00:00
"warning_syntect-server_provisioning_container_memory_usage_short_term"
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
]
```
monitoring: test owners for valid Opsgenie teams and handbook pages (#59251) In INC-264 it seems that certain alerts - such as [zoekt: less than 90% percentage pods available for 10m0s](https://opsg.in/a/i/sourcegraph/178a626f-0f28-4295-bee9-84da988bb473-1703759057681) - don't seem to end up going anywhere because the ObservableOwner is defunct. This change adds _opt-in_ testing to report: 1. How many owners have valid Opsgenie teams 2. How many owners have valid handbook pages In addition, we collect ObservableOwners that pass the test and use it to generate configuration for `site.json` in Sourcegraph.com: https://github.com/sourcegraph/deploy-sourcegraph-cloud/pull/18338 - this helps ensure the list is valid and not deceptively high-coverage. The results are not great, but **enforcing** that owners are valid isn't currently in scope: ``` 6/10 ObservableOwners do not have valid Opsgenie teams 3/10 ObservableOwners do not point to valid handbook pages ``` I also removed some defunct/unused functionality/owners. ## Test plan To run these tests: ``` export OPSGENIE_API_KEY="..." go test -timeout 30s github.com/sourcegraph/sourcegraph/monitoring/monitoring -update -online ```
2023-12-29 22:07:35 +00:00
<sub>*Managed by the [Sourcegraph Infrastructure Org team](https://handbook.sourcegraph.com/departments/engineering/infrastructure).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
Generated query for warning alert: `max((max_over_time(cadvisor_container_memory_usage_percentage_total{name=~"^syntect-server.*"}[5m])) >= 90)`
</details>
monitoring: improve solutions formatting, add team information (#14878)
2020-10-20 05:11:20 +00:00
<br />
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: improve containers metrics (#34808) part of - https://github.com/sourcegraph/sourcegraph/issues/33437 - https://github.com/sourcegraph/sourcegraph/issues/33438 This PR adds the following - A shared observable metric to indicate how many times a container (or child processes) has been oom kill - It's added to provisioning indicators since it's an indicator of underprovisioned resources - A new dashboard that shows resources usage of all containers at a glance - It contains two groups, one for all containers and one for containers that may have potential scaling issues - the first group is extremely noisy so it's hidden by default - the second group provides a more focused experience for site-admin to quickly find problematic resources <!-- > ~it looks like the alert table will be rendered regardless, which is pretty annoying (I guess the implementation of the monitoring stack is based on the assumption that one service/container will have its dashboard)~ __Questions & Notes__ - ~Should I make the new dashboard filterable by container name? like I only want to see the graph of `frontend`. Is that something that's valuable?~ - ~Any other metrics you all would like to see?~ - graph for total available resources on the VM compared to usage from https://github.com/sourcegraph/sourcegraph/issues/33437 is not really doable since we do not have `node-exporter` in anyone of our distro. We would have to tackle this from https://github.com/sourcegraph/customer/issues/806 --> ## Test plan connect to `demo`'s prom ```sh gcloud compute start-iap-tunnel default-$NEW_DEPLOYMENT-instance 9090 --local-host-port=localhost:4445 --zone us-central1-f --project $PROJECT_PREFIX-$CUSTOMER ``` or use dogfood ```sh kubectl port-forward svc/prometheus 4445:30090 ``` Update `dev/grafana/all/datasources.yaml` ```yml # Configuration for all non-Linux platforms. apiVersion: 1 datasources: - name: Prometheus type: prometheus access: proxy url: http://docker.for.mac.localhost:4445 # use the remote prometheus instance on demo instead of the local one isDefault: true editable: false - name: Jaeger type: jaeger access: proxy url: http://host.docker.internal:16686/-/debug/jaeger - name: Loki type: loki access: proxy url: http://host.docker.internal:3100 ``` Start monitoring stack ```sh sg run grafana ``` see screenshot > the first group is hidden by default ![CleanShot 2022-05-04 at 10 48 26](https://user-images.githubusercontent.com/8373004/166748470-22460552-6e51-4b29-a26b-4c5adbe1c404.png) ![CleanShot 2022-05-04 at 10 48 03](https://user-images.githubusercontent.com/8373004/166748396-1d410c35-e5fe-45e2-83bd-418d845413cb.png) ![CleanShot 2022-05-04 at 11 25 13](https://user-images.githubusercontent.com/8373004/166801266-21e51b68-65fd-4cb7-97de-83520e31e498.png)
2022-05-09 23:56:47 +00:00
## syntect-server: container_oomkill_events_total
<p class="subtitle">container OOMKILL events total by instance</p>
**Descriptions**
- <span class="badge badge-warning">warning</span> syntect-server: 1+ container OOMKILL events total by instance
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: improve containers metrics (#34808) part of - https://github.com/sourcegraph/sourcegraph/issues/33437 - https://github.com/sourcegraph/sourcegraph/issues/33438 This PR adds the following - A shared observable metric to indicate how many times a container (or child processes) has been oom kill - It's added to provisioning indicators since it's an indicator of underprovisioned resources - A new dashboard that shows resources usage of all containers at a glance - It contains two groups, one for all containers and one for containers that may have potential scaling issues - the first group is extremely noisy so it's hidden by default - the second group provides a more focused experience for site-admin to quickly find problematic resources <!-- > ~it looks like the alert table will be rendered regardless, which is pretty annoying (I guess the implementation of the monitoring stack is based on the assumption that one service/container will have its dashboard)~ __Questions & Notes__ - ~Should I make the new dashboard filterable by container name? like I only want to see the graph of `frontend`. Is that something that's valuable?~ - ~Any other metrics you all would like to see?~ - graph for total available resources on the VM compared to usage from https://github.com/sourcegraph/sourcegraph/issues/33437 is not really doable since we do not have `node-exporter` in anyone of our distro. We would have to tackle this from https://github.com/sourcegraph/customer/issues/806 --> ## Test plan connect to `demo`'s prom ```sh gcloud compute start-iap-tunnel default-$NEW_DEPLOYMENT-instance 9090 --local-host-port=localhost:4445 --zone us-central1-f --project $PROJECT_PREFIX-$CUSTOMER ``` or use dogfood ```sh kubectl port-forward svc/prometheus 4445:30090 ``` Update `dev/grafana/all/datasources.yaml` ```yml # Configuration for all non-Linux platforms. apiVersion: 1 datasources: - name: Prometheus type: prometheus access: proxy url: http://docker.for.mac.localhost:4445 # use the remote prometheus instance on demo instead of the local one isDefault: true editable: false - name: Jaeger type: jaeger access: proxy url: http://host.docker.internal:16686/-/debug/jaeger - name: Loki type: loki access: proxy url: http://host.docker.internal:3100 ``` Start monitoring stack ```sh sg run grafana ``` see screenshot > the first group is hidden by default ![CleanShot 2022-05-04 at 10 48 26](https://user-images.githubusercontent.com/8373004/166748470-22460552-6e51-4b29-a26b-4c5adbe1c404.png) ![CleanShot 2022-05-04 at 10 48 03](https://user-images.githubusercontent.com/8373004/166748396-1d410c35-e5fe-45e2-83bd-418d845413cb.png) ![CleanShot 2022-05-04 at 11 25 13](https://user-images.githubusercontent.com/8373004/166801266-21e51b68-65fd-4cb7-97de-83520e31e498.png)
2022-05-09 23:56:47 +00:00
- **Kubernetes:** Consider increasing memory limit in relevant `Deployment.yaml`.
- **Docker Compose:** Consider increasing `memory:` of syntect-server container in `docker-compose.yml`.
- More help interpreting this metric is available in the [dashboards reference](./dashboards.md#syntect-server-container-oomkill-events-total).
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"warning_syntect-server_container_oomkill_events_total"
]
```
monitoring: test owners for valid Opsgenie teams and handbook pages (#59251) In INC-264 it seems that certain alerts - such as [zoekt: less than 90% percentage pods available for 10m0s](https://opsg.in/a/i/sourcegraph/178a626f-0f28-4295-bee9-84da988bb473-1703759057681) - don't seem to end up going anywhere because the ObservableOwner is defunct. This change adds _opt-in_ testing to report: 1. How many owners have valid Opsgenie teams 2. How many owners have valid handbook pages In addition, we collect ObservableOwners that pass the test and use it to generate configuration for `site.json` in Sourcegraph.com: https://github.com/sourcegraph/deploy-sourcegraph-cloud/pull/18338 - this helps ensure the list is valid and not deceptively high-coverage. The results are not great, but **enforcing** that owners are valid isn't currently in scope: ``` 6/10 ObservableOwners do not have valid Opsgenie teams 3/10 ObservableOwners do not point to valid handbook pages ``` I also removed some defunct/unused functionality/owners. ## Test plan To run these tests: ``` export OPSGENIE_API_KEY="..." go test -timeout 30s github.com/sourcegraph/sourcegraph/monitoring/monitoring -update -online ```
2023-12-29 22:07:35 +00:00
<sub>*Managed by the [Sourcegraph Infrastructure Org team](https://handbook.sourcegraph.com/departments/engineering/infrastructure).*</sub>
monitoring: improve containers metrics (#34808) part of - https://github.com/sourcegraph/sourcegraph/issues/33437 - https://github.com/sourcegraph/sourcegraph/issues/33438 This PR adds the following - A shared observable metric to indicate how many times a container (or child processes) has been oom kill - It's added to provisioning indicators since it's an indicator of underprovisioned resources - A new dashboard that shows resources usage of all containers at a glance - It contains two groups, one for all containers and one for containers that may have potential scaling issues - the first group is extremely noisy so it's hidden by default - the second group provides a more focused experience for site-admin to quickly find problematic resources <!-- > ~it looks like the alert table will be rendered regardless, which is pretty annoying (I guess the implementation of the monitoring stack is based on the assumption that one service/container will have its dashboard)~ __Questions & Notes__ - ~Should I make the new dashboard filterable by container name? like I only want to see the graph of `frontend`. Is that something that's valuable?~ - ~Any other metrics you all would like to see?~ - graph for total available resources on the VM compared to usage from https://github.com/sourcegraph/sourcegraph/issues/33437 is not really doable since we do not have `node-exporter` in anyone of our distro. We would have to tackle this from https://github.com/sourcegraph/customer/issues/806 --> ## Test plan connect to `demo`'s prom ```sh gcloud compute start-iap-tunnel default-$NEW_DEPLOYMENT-instance 9090 --local-host-port=localhost:4445 --zone us-central1-f --project $PROJECT_PREFIX-$CUSTOMER ``` or use dogfood ```sh kubectl port-forward svc/prometheus 4445:30090 ``` Update `dev/grafana/all/datasources.yaml` ```yml # Configuration for all non-Linux platforms. apiVersion: 1 datasources: - name: Prometheus type: prometheus access: proxy url: http://docker.for.mac.localhost:4445 # use the remote prometheus instance on demo instead of the local one isDefault: true editable: false - name: Jaeger type: jaeger access: proxy url: http://host.docker.internal:16686/-/debug/jaeger - name: Loki type: loki access: proxy url: http://host.docker.internal:3100 ``` Start monitoring stack ```sh sg run grafana ``` see screenshot > the first group is hidden by default ![CleanShot 2022-05-04 at 10 48 26](https://user-images.githubusercontent.com/8373004/166748470-22460552-6e51-4b29-a26b-4c5adbe1c404.png) ![CleanShot 2022-05-04 at 10 48 03](https://user-images.githubusercontent.com/8373004/166748396-1d410c35-e5fe-45e2-83bd-418d845413cb.png) ![CleanShot 2022-05-04 at 11 25 13](https://user-images.githubusercontent.com/8373004/166801266-21e51b68-65fd-4cb7-97de-83520e31e498.png)
2022-05-09 23:56:47 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
monitoring: extract into a submodule (#45786) This change extracts `monitoring` into a submodule for import in `sourcegraph/controller` (https://github.com/sourcegraph/controller/pull/195) so that we can generate dashboards for Cloud instances. These steps were required: 1. Initialize a `go.mod` in `monitoring` 2. Extract `dev/sg/internal/cliutil` into `lib` to avoid illegal imports from `monitoring` 3. Add local replaces to both `sourcegraph/sourcegraph` and `monitoring` 4. `go mod tidy` on all submodules 5. Update `go generate ./monitoring` commands to use `sg`, since the `go generate` command no longer works 6. Update `grafana/build.sh`, `prometheus/build.sh` to build the submodule 7. Amend linters to check for multiple `go.mod` files and ban imports of `github.com/sourcegraph/sourcegraph` 8. Update `sg generate go` to run in directories rather than from root The only caveat is that if you use VS code, you will now need to open `monitoring` in a separate workspace or similar, like with `lib`. Co-authored-by: Joe Chen <joe@sourcegraph.com>
2022-12-19 17:49:25 +00:00
Generated query for warning alert: `max((max by (name) (container_oom_events_total{name=~"^syntect-server.*"})) >= 1)`
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
</details>
monitoring: improve containers metrics (#34808) part of - https://github.com/sourcegraph/sourcegraph/issues/33437 - https://github.com/sourcegraph/sourcegraph/issues/33438 This PR adds the following - A shared observable metric to indicate how many times a container (or child processes) has been oom kill - It's added to provisioning indicators since it's an indicator of underprovisioned resources - A new dashboard that shows resources usage of all containers at a glance - It contains two groups, one for all containers and one for containers that may have potential scaling issues - the first group is extremely noisy so it's hidden by default - the second group provides a more focused experience for site-admin to quickly find problematic resources <!-- > ~it looks like the alert table will be rendered regardless, which is pretty annoying (I guess the implementation of the monitoring stack is based on the assumption that one service/container will have its dashboard)~ __Questions & Notes__ - ~Should I make the new dashboard filterable by container name? like I only want to see the graph of `frontend`. Is that something that's valuable?~ - ~Any other metrics you all would like to see?~ - graph for total available resources on the VM compared to usage from https://github.com/sourcegraph/sourcegraph/issues/33437 is not really doable since we do not have `node-exporter` in anyone of our distro. We would have to tackle this from https://github.com/sourcegraph/customer/issues/806 --> ## Test plan connect to `demo`'s prom ```sh gcloud compute start-iap-tunnel default-$NEW_DEPLOYMENT-instance 9090 --local-host-port=localhost:4445 --zone us-central1-f --project $PROJECT_PREFIX-$CUSTOMER ``` or use dogfood ```sh kubectl port-forward svc/prometheus 4445:30090 ``` Update `dev/grafana/all/datasources.yaml` ```yml # Configuration for all non-Linux platforms. apiVersion: 1 datasources: - name: Prometheus type: prometheus access: proxy url: http://docker.for.mac.localhost:4445 # use the remote prometheus instance on demo instead of the local one isDefault: true editable: false - name: Jaeger type: jaeger access: proxy url: http://host.docker.internal:16686/-/debug/jaeger - name: Loki type: loki access: proxy url: http://host.docker.internal:3100 ``` Start monitoring stack ```sh sg run grafana ``` see screenshot > the first group is hidden by default ![CleanShot 2022-05-04 at 10 48 26](https://user-images.githubusercontent.com/8373004/166748470-22460552-6e51-4b29-a26b-4c5adbe1c404.png) ![CleanShot 2022-05-04 at 10 48 03](https://user-images.githubusercontent.com/8373004/166748396-1d410c35-e5fe-45e2-83bd-418d845413cb.png) ![CleanShot 2022-05-04 at 11 25 13](https://user-images.githubusercontent.com/8373004/166801266-21e51b68-65fd-4cb7-97de-83520e31e498.png)
2022-05-09 23:56:47 +00:00
<br />
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
## syntect-server: pods_available_percentage
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
<p class="subtitle">percentage pods available</p>
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
monitoring: dashboards reference, refresh observability docs (#16939) - New monitoring definition field, 'Interpretation' - New generated dashboards reference - Grafana panels now link to alert solutions and dashboards reference - Refreshed observability docs Co-authored-by: uwedeportivo <534011+uwedeportivo@users.noreply.github.com>
2020-12-23 02:36:41 +00:00
- <span class="badge badge-critical">critical</span> syntect-server: less than 90% percentage pods available for 10m0s
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
monitoring: forbid critical alerts with no solutions (#36423) Adds the restriction that critical alerts must have solutions defined as a hard requirement within the generator, and fixes all the outstanding issues: - Add PossibleSolutions for the pods_available_percentage and pg_up critical alerts (we have a similar solution doc in the container_missing alert) - Removes all the other critical alerts without possible solutions Co-authored-by: Michael Lin <mlzc@hey.com>
2022-06-02 15:51:43 +00:00
- Determine if the pod was OOM killed using `kubectl describe pod syntect-server` (look for `OOMKilled: true`) and, if so, consider increasing the memory limit in the relevant `Deployment.yaml`.
- Check the logs before the container restarted to see if there are `panic:` messages or similar using `kubectl logs -p syntect-server`.
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#syntect-server-pods-available-percentage).
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"critical_syntect-server_pods_available_percentage"
]
```
monitoring: test owners for valid Opsgenie teams and handbook pages (#59251) In INC-264 it seems that certain alerts - such as [zoekt: less than 90% percentage pods available for 10m0s](https://opsg.in/a/i/sourcegraph/178a626f-0f28-4295-bee9-84da988bb473-1703759057681) - don't seem to end up going anywhere because the ObservableOwner is defunct. This change adds _opt-in_ testing to report: 1. How many owners have valid Opsgenie teams 2. How many owners have valid handbook pages In addition, we collect ObservableOwners that pass the test and use it to generate configuration for `site.json` in Sourcegraph.com: https://github.com/sourcegraph/deploy-sourcegraph-cloud/pull/18338 - this helps ensure the list is valid and not deceptively high-coverage. The results are not great, but **enforcing** that owners are valid isn't currently in scope: ``` 6/10 ObservableOwners do not have valid Opsgenie teams 3/10 ObservableOwners do not point to valid handbook pages ``` I also removed some defunct/unused functionality/owners. ## Test plan To run these tests: ``` export OPSGENIE_API_KEY="..." go test -timeout 30s github.com/sourcegraph/sourcegraph/monitoring/monitoring -update -online ```
2023-12-29 22:07:35 +00:00
<sub>*Managed by the [Sourcegraph Infrastructure Org team](https://handbook.sourcegraph.com/departments/engineering/infrastructure).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
monitoring: extract into a submodule (#45786) This change extracts `monitoring` into a submodule for import in `sourcegraph/controller` (https://github.com/sourcegraph/controller/pull/195) so that we can generate dashboards for Cloud instances. These steps were required: 1. Initialize a `go.mod` in `monitoring` 2. Extract `dev/sg/internal/cliutil` into `lib` to avoid illegal imports from `monitoring` 3. Add local replaces to both `sourcegraph/sourcegraph` and `monitoring` 4. `go mod tidy` on all submodules 5. Update `go generate ./monitoring` commands to use `sg`, since the `go generate` command no longer works 6. Update `grafana/build.sh`, `prometheus/build.sh` to build the submodule 7. Amend linters to check for multiple `go.mod` files and ban imports of `github.com/sourcegraph/sourcegraph` 8. Update `sg generate go` to run in directories rather than from root The only caveat is that if you use VS code, you will now need to open `monitoring` in a separate workspace or similar, like with `lib`. Co-authored-by: Joe Chen <joe@sourcegraph.com>
2022-12-19 17:49:25 +00:00
Generated query for critical alert: `min((sum by (app) (up{app=~".*syntect-server"}) / count by (app) (up{app=~".*syntect-server"}) * 100) <= 90)`
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
</details>
monitoring: improve solutions formatting, add team information (#14878)
2020-10-20 05:11:20 +00:00
<br />
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
zoekt: combine zoekt-webserver and zoekt-indexserver Grafana dashboards (#29768)
2022-01-14 18:17:07 +00:00
## zoekt: average_resolve_revision_duration
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
<p class="subtitle">average resolve revision duration over 5m</p>
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
zoekt: combine zoekt-webserver and zoekt-indexserver Grafana dashboards (#29768)
2022-01-14 18:17:07 +00:00
- <span class="badge badge-warning">warning</span> zoekt: 15s+ average resolve revision duration over 5m
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
zoekt: combine zoekt-webserver and zoekt-indexserver Grafana dashboards (#29768)
2022-01-14 18:17:07 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#zoekt-average-resolve-revision-duration).
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
monitoring: support multi-bound alerts, add lower-bound warnings for containers (#11082)
2020-06-16 07:09:25 +00:00
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
```json
"observability.silenceAlerts": [
monitoring: forbid critical alerts with no solutions (#36423) Adds the restriction that critical alerts must have solutions defined as a hard requirement within the generator, and fixes all the outstanding issues: - Add PossibleSolutions for the pods_available_percentage and pg_up critical alerts (we have a similar solution doc in the container_missing alert) - Removes all the other critical alerts without possible solutions Co-authored-by: Michael Lin <mlzc@hey.com>
2022-06-02 15:51:43 +00:00
"warning_zoekt_average_resolve_revision_duration"
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
]
```
Monitoring: update owner for search platform (#59316) The 'search core' team is now called 'search platform'. --------- Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2024-01-04 20:55:41 +00:00
<sub>*Managed by the [Sourcegraph Search Platform team](https://handbook.sourcegraph.com/departments/engineering/teams/search/core).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
Generated query for warning alert: `max((sum(rate(resolve_revision_seconds_sum[5m])) / sum(rate(resolve_revision_seconds_count[5m]))) >= 15)`
</details>
monitoring: improve solutions formatting, add team information (#14878)
2020-10-20 05:11:20 +00:00
<br />
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
zoekt: combine zoekt-webserver and zoekt-indexserver Grafana dashboards (#29768)
2022-01-14 18:17:07 +00:00
## zoekt: get_index_options_error_increase
monitoring: add alert for zoekt indexserver get index options (#24984) This would of helped us diagnose a site issue on Friday when we did a bad deployment.
2021-09-15 15:18:45 +00:00
<p class="subtitle">the number of repositories we failed to get indexing options over 5m</p>
**Descriptions**
monitoring: inc get_index_options_error_increase alert threshold (#30877) This has been super spammy recently. Test Plan: CI
2022-02-09 13:02:57 +00:00
- <span class="badge badge-warning">warning</span> zoekt: 100+ the number of repositories we failed to get indexing options over 5m for 5m0s
- <span class="badge badge-critical">critical</span> zoekt: 100+ the number of repositories we failed to get indexing options over 5m for 35m0s
monitoring: add alert for zoekt indexserver get index options (#24984) This would of helped us diagnose a site issue on Friday when we did a bad deployment.
2021-09-15 15:18:45 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: add alert for zoekt indexserver get index options (#24984) This would of helped us diagnose a site issue on Friday when we did a bad deployment.
2021-09-15 15:18:45 +00:00
- View error rates on gitserver and frontend to identify root cause.
- Rollback frontend/gitserver deployment if due to a bad code change.
- View error logs for `getIndexOptions` via net/trace debug interface. For example click on a `indexed-search-indexer-` on https://sourcegraph.com/-/debug/. Then click on Traces. Replace sourcegraph.com with your instance address.
zoekt: combine zoekt-webserver and zoekt-indexserver Grafana dashboards (#29768)
2022-01-14 18:17:07 +00:00
- More help interpreting this metric is available in the [dashboards reference](./dashboards.md#zoekt-get-index-options-error-increase).
monitoring: add alert for zoekt indexserver get index options (#24984) This would of helped us diagnose a site issue on Friday when we did a bad deployment.
2021-09-15 15:18:45 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
zoekt: combine zoekt-webserver and zoekt-indexserver Grafana dashboards (#29768)
2022-01-14 18:17:07 +00:00
"warning_zoekt_get_index_options_error_increase",
"critical_zoekt_get_index_options_error_increase"
monitoring: add alert for zoekt indexserver get index options (#24984) This would of helped us diagnose a site issue on Friday when we did a bad deployment.
2021-09-15 15:18:45 +00:00
]
```
Monitoring: update owner for search platform (#59316) The 'search core' team is now called 'search platform'. --------- Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2024-01-04 20:55:41 +00:00
<sub>*Managed by the [Sourcegraph Search Platform team](https://handbook.sourcegraph.com/departments/engineering/teams/search/core).*</sub>
monitoring: add alert for zoekt indexserver get index options (#24984) This would of helped us diagnose a site issue on Friday when we did a bad deployment.
2021-09-15 15:18:45 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
Generated query for warning alert: `max((sum(increase(get_index_options_error_total[5m]))) >= 100)`
Generated query for critical alert: `max((sum(increase(get_index_options_error_total[5m]))) >= 100)`
</details>
monitoring: add alert for zoekt indexserver get index options (#24984) This would of helped us diagnose a site issue on Friday when we did a bad deployment.
2021-09-15 15:18:45 +00:00
<br />
zoekt: combine zoekt-webserver and zoekt-indexserver Grafana dashboards (#29768)
2022-01-14 18:17:07 +00:00
## zoekt: indexed_search_request_errors
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
zoekt: combine zoekt-webserver and zoekt-indexserver Grafana dashboards (#29768)
2022-01-14 18:17:07 +00:00
<p class="subtitle">indexed search request errors every 5m by code</p>
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
zoekt: combine zoekt-webserver and zoekt-indexserver Grafana dashboards (#29768)
2022-01-14 18:17:07 +00:00
- <span class="badge badge-warning">warning</span> zoekt: 5%+ indexed search request errors every 5m by code for 5m0s
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
zoekt: combine zoekt-webserver and zoekt-indexserver Grafana dashboards (#29768)
2022-01-14 18:17:07 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#zoekt-indexed-search-request-errors).
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
zoekt: combine zoekt-webserver and zoekt-indexserver Grafana dashboards (#29768)
2022-01-14 18:17:07 +00:00
"warning_zoekt_indexed_search_request_errors"
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
]
```
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
Monitoring: update owner for search platform (#59316) The 'search core' team is now called 'search platform'. --------- Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2024-01-04 20:55:41 +00:00
<sub>*Managed by the [Sourcegraph Search Platform team](https://handbook.sourcegraph.com/departments/engineering/teams/search/core).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
monitoring: extract into a submodule (#45786) This change extracts `monitoring` into a submodule for import in `sourcegraph/controller` (https://github.com/sourcegraph/controller/pull/195) so that we can generate dashboards for Cloud instances. These steps were required: 1. Initialize a `go.mod` in `monitoring` 2. Extract `dev/sg/internal/cliutil` into `lib` to avoid illegal imports from `monitoring` 3. Add local replaces to both `sourcegraph/sourcegraph` and `monitoring` 4. `go mod tidy` on all submodules 5. Update `go generate ./monitoring` commands to use `sg`, since the `go generate` command no longer works 6. Update `grafana/build.sh`, `prometheus/build.sh` to build the submodule 7. Amend linters to check for multiple `go.mod` files and ban imports of `github.com/sourcegraph/sourcegraph` 8. Update `sg generate go` to run in directories rather than from root The only caveat is that if you use VS code, you will now need to open `monitoring` in a separate workspace or similar, like with `lib`. Co-authored-by: Joe Chen <joe@sourcegraph.com>
2022-12-19 17:49:25 +00:00
Generated query for warning alert: `max((sum by (code) (increase(src_zoekt_request_duration_seconds_count{code!~"2.."}[5m])) / ignoring (code) group_left () sum(increase(src_zoekt_request_duration_seconds_count[5m])) * 100) >= 5)`
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
</details>
monitoring: improve solutions formatting, add team information (#14878)
2020-10-20 05:11:20 +00:00
<br />
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: zoekt - add dashboard to track percentage of used memory areas (#42190) Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2022-10-03 15:55:16 +00:00
## zoekt: memory_map_areas_percentage_used
<p class="subtitle">process memory map areas percentage used (per instance)</p>
**Descriptions**
monitoring: zoekt - lower alert thresholds for memory map areas by 10% (#44376)
2022-11-28 18:27:34 +00:00
- <span class="badge badge-warning">warning</span> zoekt: 60%+ process memory map areas percentage used (per instance)
- <span class="badge badge-critical">critical</span> zoekt: 80%+ process memory map areas percentage used (per instance)
monitoring: zoekt - add dashboard to track percentage of used memory areas (#42190) Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2022-10-03 15:55:16 +00:00
**Next steps**
- If you are running out of memory map areas, you could resolve this by:
alert: mention shard merging in alert's next steps (#47895) Shard merging reduces the number of mmapped files and is a good option to resolve this alert. ## Test plan - just a copy change <!-- All pull requests REQUIRE a test plan: https://docs.sourcegraph.com/dev/background-information/testing_principles -->
2023-02-24 09:07:23 +00:00
- Enabling shard merging for Zoekt: Set SRC_ENABLE_SHARD_MERGING="1" for zoekt-indexserver. Use this option
if your corpus of repositories has a high percentage of small, rarely updated repositories. See
Docs: update links to point to new site (#60381) We have a number of docs links in the product that point to the old doc site. Method: - Search the repo for `docs.sourcegraph.com` - Exclude the `doc/` dir, all test fixtures, and `CHANGELOG.md` - For each, replace `docs.sourcegraph.com` with `sourcegraph.com/docs` - Navigate to the resulting URL ensuring it's not a dead link, updating the URL if necessary Many of the URLs updated are just comments, but since I'm doing a manual audit of each URL anyways, I felt it was worth it to update these while I was at it.
2024-02-13 00:23:47 +00:00
[documentation](https://sourcegraph.com/docs/code-search/features#shard-merging).
monitoring: zoekt - add dashboard to track percentage of used memory areas (#42190) Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2022-10-03 15:55:16 +00:00
- Creating additional Zoekt replicas: This spreads all the shards out amongst more replicas, which
means that each _individual_ replica will have fewer shards. This, in turn, decreases the
amount of memory map areas that a _single_ replica can create (in order to load the shards into memory).
alert: mention shard merging in alert's next steps (#47895) Shard merging reduces the number of mmapped files and is a good option to resolve this alert. ## Test plan - just a copy change <!-- All pull requests REQUIRE a test plan: https://docs.sourcegraph.com/dev/background-information/testing_principles -->
2023-02-24 09:07:23 +00:00
- Increasing the virtual memory subsystem`s "max_map_count" parameter which defines the upper limit of memory areas
a process can use. The default value of max_map_count is usually 65536. We recommend to set this value to 2x the number
of repos to be indexed per Zoekt instance. This means, if you want to index 240k repositories with 3 Zoekt instances,
set max_map_count to (240000 / 3) * 2 = 160000. The exact instructions for tuning this parameter can differ depending
on your environment. See https://kernel.org/doc/Documentation/sysctl/vm.txt for more information.
monitoring: zoekt - add dashboard to track percentage of used memory areas (#42190) Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2022-10-03 15:55:16 +00:00
- More help interpreting this metric is available in the [dashboards reference](./dashboards.md#zoekt-memory-map-areas-percentage-used).
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"warning_zoekt_memory_map_areas_percentage_used",
"critical_zoekt_memory_map_areas_percentage_used"
]
```
Monitoring: update owner for search platform (#59316) The 'search core' team is now called 'search platform'. --------- Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2024-01-04 20:55:41 +00:00
<sub>*Managed by the [Sourcegraph Search Platform team](https://handbook.sourcegraph.com/departments/engineering/teams/search/core).*</sub>
monitoring: zoekt - add dashboard to track percentage of used memory areas (#42190) Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2022-10-03 15:55:16 +00:00
<details>
<summary>Technical details</summary>
monitoring: zoekt - lower alert thresholds for memory map areas by 10% (#44376)
2022-11-28 18:27:34 +00:00
Generated query for warning alert: `max(((proc_metrics_memory_map_current_count / proc_metrics_memory_map_max_limit) * 100) >= 60)`
monitoring: zoekt - add dashboard to track percentage of used memory areas (#42190) Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2022-10-03 15:55:16 +00:00
monitoring: zoekt - lower alert thresholds for memory map areas by 10% (#44376)
2022-11-28 18:27:34 +00:00
Generated query for critical alert: `max(((proc_metrics_memory_map_current_count / proc_metrics_memory_map_max_limit) * 100) >= 80)`
monitoring: zoekt - add dashboard to track percentage of used memory areas (#42190) Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2022-10-03 15:55:16 +00:00
</details>
<br />
zoekt: combine zoekt-webserver and zoekt-indexserver Grafana dashboards (#29768)
2022-01-14 18:17:07 +00:00
## zoekt: container_cpu_usage
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
zoekt: combine zoekt-webserver and zoekt-indexserver Grafana dashboards (#29768)
2022-01-14 18:17:07 +00:00
<p class="subtitle">container cpu usage total (1m average) across all cores by instance</p>
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
zoekt: combine zoekt-webserver and zoekt-indexserver Grafana dashboards (#29768)
2022-01-14 18:17:07 +00:00
- <span class="badge badge-warning">warning</span> zoekt: 99%+ container cpu usage total (1m average) across all cores by instance
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
zoekt: combine zoekt-webserver and zoekt-indexserver Grafana dashboards (#29768)
2022-01-14 18:17:07 +00:00
- **Kubernetes:** Consider increasing CPU limits in the the relevant `Deployment.yaml`.
- **Docker Compose:** Consider increasing `cpus:` of the zoekt-indexserver container in `docker-compose.yml`.
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#zoekt-container-cpu-usage).
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
```json
"observability.silenceAlerts": [
zoekt: combine zoekt-webserver and zoekt-indexserver Grafana dashboards (#29768)
2022-01-14 18:17:07 +00:00
"warning_zoekt_container_cpu_usage"
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
]
```
Monitoring: update owner for search platform (#59316) The 'search core' team is now called 'search platform'. --------- Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2024-01-04 20:55:41 +00:00
<sub>*Managed by the [Sourcegraph Search Platform team](https://handbook.sourcegraph.com/departments/engineering/teams/search/core).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
Generated query for warning alert: `max((cadvisor_container_cpu_usage_percentage_total{name=~"^zoekt-indexserver.*"}) >= 99)`
</details>
monitoring: improve solutions formatting, add team information (#14878)
2020-10-20 05:11:20 +00:00
<br />
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
zoekt: combine zoekt-webserver and zoekt-indexserver Grafana dashboards (#29768)
2022-01-14 18:17:07 +00:00
## zoekt: container_memory_usage
monitoring: support multi-bound alerts, add lower-bound warnings for containers (#11082)
2020-06-16 07:09:25 +00:00
zoekt: combine zoekt-webserver and zoekt-indexserver Grafana dashboards (#29768)
2022-01-14 18:17:07 +00:00
<p class="subtitle">container memory usage by instance</p>
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
monitoring: support multi-bound alerts, add lower-bound warnings for containers (#11082)
2020-06-16 07:09:25 +00:00
zoekt: combine zoekt-webserver and zoekt-indexserver Grafana dashboards (#29768)
2022-01-14 18:17:07 +00:00
- <span class="badge badge-warning">warning</span> zoekt: 99%+ container memory usage by instance
monitoring: support multi-bound alerts, add lower-bound warnings for containers (#11082)
2020-06-16 07:09:25 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: support multi-bound alerts, add lower-bound warnings for containers (#11082)
2020-06-16 07:09:25 +00:00
zoekt: combine zoekt-webserver and zoekt-indexserver Grafana dashboards (#29768)
2022-01-14 18:17:07 +00:00
- **Kubernetes:** Consider increasing memory limit in relevant `Deployment.yaml`.
- **Docker Compose:** Consider increasing `memory:` of zoekt-indexserver container in `docker-compose.yml`.
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#zoekt-container-memory-usage).
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
zoekt: combine zoekt-webserver and zoekt-indexserver Grafana dashboards (#29768)
2022-01-14 18:17:07 +00:00
"warning_zoekt_container_memory_usage"
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
]
```
monitoring: support multi-bound alerts, add lower-bound warnings for containers (#11082)
2020-06-16 07:09:25 +00:00
Monitoring: update owner for search platform (#59316) The 'search core' team is now called 'search platform'. --------- Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2024-01-04 20:55:41 +00:00
<sub>*Managed by the [Sourcegraph Search Platform team](https://handbook.sourcegraph.com/departments/engineering/teams/search/core).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
Generated query for warning alert: `max((cadvisor_container_memory_usage_percentage_total{name=~"^zoekt-indexserver.*"}) >= 99)`
</details>
monitoring: improve solutions formatting, add team information (#14878)
2020-10-20 05:11:20 +00:00
<br />
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
zoekt: combine zoekt-webserver and zoekt-indexserver Grafana dashboards (#29768)
2022-01-14 18:17:07 +00:00
## zoekt: container_cpu_usage
monitoring: support multi-bound alerts, add lower-bound warnings for containers (#11082)
2020-06-16 07:09:25 +00:00
zoekt: combine zoekt-webserver and zoekt-indexserver Grafana dashboards (#29768)
2022-01-14 18:17:07 +00:00
<p class="subtitle">container cpu usage total (1m average) across all cores by instance</p>
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
monitoring: support multi-bound alerts, add lower-bound warnings for containers (#11082)
2020-06-16 07:09:25 +00:00
zoekt: combine zoekt-webserver and zoekt-indexserver Grafana dashboards (#29768)
2022-01-14 18:17:07 +00:00
- <span class="badge badge-warning">warning</span> zoekt: 99%+ container cpu usage total (1m average) across all cores by instance
monitoring: support multi-bound alerts, add lower-bound warnings for containers (#11082)
2020-06-16 07:09:25 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: support multi-bound alerts, add lower-bound warnings for containers (#11082)
2020-06-16 07:09:25 +00:00
zoekt: combine zoekt-webserver and zoekt-indexserver Grafana dashboards (#29768)
2022-01-14 18:17:07 +00:00
- **Kubernetes:** Consider increasing CPU limits in the the relevant `Deployment.yaml`.
- **Docker Compose:** Consider increasing `cpus:` of the zoekt-webserver container in `docker-compose.yml`.
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#zoekt-container-cpu-usage).
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
monitoring: support multi-bound alerts, add lower-bound warnings for containers (#11082)
2020-06-16 07:09:25 +00:00
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
```json
"observability.silenceAlerts": [
zoekt: combine zoekt-webserver and zoekt-indexserver Grafana dashboards (#29768)
2022-01-14 18:17:07 +00:00
"warning_zoekt_container_cpu_usage"
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
]
```
Monitoring: update owner for search platform (#59316) The 'search core' team is now called 'search platform'. --------- Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2024-01-04 20:55:41 +00:00
<sub>*Managed by the [Sourcegraph Search Platform team](https://handbook.sourcegraph.com/departments/engineering/teams/search/core).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
Generated query for warning alert: `max((cadvisor_container_cpu_usage_percentage_total{name=~"^zoekt-webserver.*"}) >= 99)`
</details>
monitoring: improve solutions formatting, add team information (#14878)
2020-10-20 05:11:20 +00:00
<br />
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
zoekt: combine zoekt-webserver and zoekt-indexserver Grafana dashboards (#29768)
2022-01-14 18:17:07 +00:00
## zoekt: container_memory_usage
monitoring: support multi-bound alerts, add lower-bound warnings for containers (#11082)
2020-06-16 07:09:25 +00:00
zoekt: combine zoekt-webserver and zoekt-indexserver Grafana dashboards (#29768)
2022-01-14 18:17:07 +00:00
<p class="subtitle">container memory usage by instance</p>
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
monitoring: support multi-bound alerts, add lower-bound warnings for containers (#11082)
2020-06-16 07:09:25 +00:00
zoekt: combine zoekt-webserver and zoekt-indexserver Grafana dashboards (#29768)
2022-01-14 18:17:07 +00:00
- <span class="badge badge-warning">warning</span> zoekt: 99%+ container memory usage by instance
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
zoekt: combine zoekt-webserver and zoekt-indexserver Grafana dashboards (#29768)
2022-01-14 18:17:07 +00:00
- **Kubernetes:** Consider increasing memory limit in relevant `Deployment.yaml`.
- **Docker Compose:** Consider increasing `memory:` of zoekt-webserver container in `docker-compose.yml`.
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#zoekt-container-memory-usage).
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
zoekt: combine zoekt-webserver and zoekt-indexserver Grafana dashboards (#29768)
2022-01-14 18:17:07 +00:00
"warning_zoekt_container_memory_usage"
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
]
```
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
Monitoring: update owner for search platform (#59316) The 'search core' team is now called 'search platform'. --------- Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2024-01-04 20:55:41 +00:00
<sub>*Managed by the [Sourcegraph Search Platform team](https://handbook.sourcegraph.com/departments/engineering/teams/search/core).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
Generated query for warning alert: `max((cadvisor_container_memory_usage_percentage_total{name=~"^zoekt-webserver.*"}) >= 99)`
</details>
monitoring: improve solutions formatting, add team information (#14878)
2020-10-20 05:11:20 +00:00
<br />
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
zoekt: combine zoekt-webserver and zoekt-indexserver Grafana dashboards (#29768)
2022-01-14 18:17:07 +00:00
## zoekt: provisioning_container_cpu_usage_long_term
monitoring: support multi-bound alerts, add lower-bound warnings for containers (#11082)
2020-06-16 07:09:25 +00:00
zoekt: combine zoekt-webserver and zoekt-indexserver Grafana dashboards (#29768)
2022-01-14 18:17:07 +00:00
<p class="subtitle">container cpu usage total (90th percentile over 1d) across all cores by instance</p>
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
monitoring: support multi-bound alerts, add lower-bound warnings for containers (#11082)
2020-06-16 07:09:25 +00:00
zoekt: combine zoekt-webserver and zoekt-indexserver Grafana dashboards (#29768)
2022-01-14 18:17:07 +00:00
- <span class="badge badge-warning">warning</span> zoekt: 80%+ container cpu usage total (90th percentile over 1d) across all cores by instance for 336h0m0s
monitoring: support multi-bound alerts, add lower-bound warnings for containers (#11082)
2020-06-16 07:09:25 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: support multi-bound alerts, add lower-bound warnings for containers (#11082)
2020-06-16 07:09:25 +00:00
zoekt: combine zoekt-webserver and zoekt-indexserver Grafana dashboards (#29768)
2022-01-14 18:17:07 +00:00
- **Kubernetes:** Consider increasing CPU limits in the `Deployment.yaml` for the zoekt-indexserver service.
- **Docker Compose:** Consider increasing `cpus:` of the zoekt-indexserver container in `docker-compose.yml`.
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#zoekt-provisioning-container-cpu-usage-long-term).
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
zoekt: combine zoekt-webserver and zoekt-indexserver Grafana dashboards (#29768)
2022-01-14 18:17:07 +00:00
"warning_zoekt_provisioning_container_cpu_usage_long_term"
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
]
```
Monitoring: update owner for search platform (#59316) The 'search core' team is now called 'search platform'. --------- Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2024-01-04 20:55:41 +00:00
<sub>*Managed by the [Sourcegraph Search Platform team](https://handbook.sourcegraph.com/departments/engineering/teams/search/core).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
Generated query for warning alert: `max((quantile_over_time(0.9, cadvisor_container_cpu_usage_percentage_total{name=~"^zoekt-indexserver.*"}[1d])) >= 80)`
</details>
monitoring: improve solutions formatting, add team information (#14878)
2020-10-20 05:11:20 +00:00
<br />
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
zoekt: combine zoekt-webserver and zoekt-indexserver Grafana dashboards (#29768)
2022-01-14 18:17:07 +00:00
## zoekt: provisioning_container_memory_usage_long_term
monitoring: support multi-bound alerts, add lower-bound warnings for containers (#11082)
2020-06-16 07:09:25 +00:00
zoekt: combine zoekt-webserver and zoekt-indexserver Grafana dashboards (#29768)
2022-01-14 18:17:07 +00:00
<p class="subtitle">container memory usage (1d maximum) by instance</p>
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
zoekt: combine zoekt-webserver and zoekt-indexserver Grafana dashboards (#29768)
2022-01-14 18:17:07 +00:00
- <span class="badge badge-warning">warning</span> zoekt: 80%+ container memory usage (1d maximum) by instance for 336h0m0s
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
zoekt: combine zoekt-webserver and zoekt-indexserver Grafana dashboards (#29768)
2022-01-14 18:17:07 +00:00
- **Kubernetes:** Consider increasing memory limits in the `Deployment.yaml` for the zoekt-indexserver service.
- **Docker Compose:** Consider increasing `memory:` of the zoekt-indexserver container in `docker-compose.yml`.
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#zoekt-provisioning-container-memory-usage-long-term).
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
zoekt: combine zoekt-webserver and zoekt-indexserver Grafana dashboards (#29768)
2022-01-14 18:17:07 +00:00
"warning_zoekt_provisioning_container_memory_usage_long_term"
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
]
```
Monitoring: update owner for search platform (#59316) The 'search core' team is now called 'search platform'. --------- Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2024-01-04 20:55:41 +00:00
<sub>*Managed by the [Sourcegraph Search Platform team](https://handbook.sourcegraph.com/departments/engineering/teams/search/core).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
Generated query for warning alert: `max((max_over_time(cadvisor_container_memory_usage_percentage_total{name=~"^zoekt-indexserver.*"}[1d])) >= 80)`
</details>
monitoring: improve solutions formatting, add team information (#14878)
2020-10-20 05:11:20 +00:00
<br />
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
zoekt: combine zoekt-webserver and zoekt-indexserver Grafana dashboards (#29768)
2022-01-14 18:17:07 +00:00
## zoekt: provisioning_container_cpu_usage_short_term
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
zoekt: combine zoekt-webserver and zoekt-indexserver Grafana dashboards (#29768)
2022-01-14 18:17:07 +00:00
<p class="subtitle">container cpu usage total (5m maximum) across all cores by instance</p>
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
zoekt: combine zoekt-webserver and zoekt-indexserver Grafana dashboards (#29768)
2022-01-14 18:17:07 +00:00
- <span class="badge badge-warning">warning</span> zoekt: 90%+ container cpu usage total (5m maximum) across all cores by instance for 30m0s
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
zoekt: combine zoekt-webserver and zoekt-indexserver Grafana dashboards (#29768)
2022-01-14 18:17:07 +00:00
- **Kubernetes:** Consider increasing CPU limits in the the relevant `Deployment.yaml`.
- **Docker Compose:** Consider increasing `cpus:` of the zoekt-indexserver container in `docker-compose.yml`.
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#zoekt-provisioning-container-cpu-usage-short-term).
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
zoekt: combine zoekt-webserver and zoekt-indexserver Grafana dashboards (#29768)
2022-01-14 18:17:07 +00:00
"warning_zoekt_provisioning_container_cpu_usage_short_term"
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
]
```
Monitoring: update owner for search platform (#59316) The 'search core' team is now called 'search platform'. --------- Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2024-01-04 20:55:41 +00:00
<sub>*Managed by the [Sourcegraph Search Platform team](https://handbook.sourcegraph.com/departments/engineering/teams/search/core).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
Generated query for warning alert: `max((max_over_time(cadvisor_container_cpu_usage_percentage_total{name=~"^zoekt-indexserver.*"}[5m])) >= 90)`
</details>
monitoring: improve solutions formatting, add team information (#14878)
2020-10-20 05:11:20 +00:00
<br />
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
zoekt: combine zoekt-webserver and zoekt-indexserver Grafana dashboards (#29768)
2022-01-14 18:17:07 +00:00
## zoekt: provisioning_container_memory_usage_short_term
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
zoekt: combine zoekt-webserver and zoekt-indexserver Grafana dashboards (#29768)
2022-01-14 18:17:07 +00:00
<p class="subtitle">container memory usage (5m maximum) by instance</p>
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
zoekt: combine zoekt-webserver and zoekt-indexserver Grafana dashboards (#29768)
2022-01-14 18:17:07 +00:00
- <span class="badge badge-warning">warning</span> zoekt: 90%+ container memory usage (5m maximum) by instance
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
zoekt: combine zoekt-webserver and zoekt-indexserver Grafana dashboards (#29768)
2022-01-14 18:17:07 +00:00
- **Kubernetes:** Consider increasing memory limit in relevant `Deployment.yaml`.
- **Docker Compose:** Consider increasing `memory:` of zoekt-indexserver container in `docker-compose.yml`.
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#zoekt-provisioning-container-memory-usage-short-term).
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
zoekt: combine zoekt-webserver and zoekt-indexserver Grafana dashboards (#29768)
2022-01-14 18:17:07 +00:00
"warning_zoekt_provisioning_container_memory_usage_short_term"
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
]
```
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
Monitoring: update owner for search platform (#59316) The 'search core' team is now called 'search platform'. --------- Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2024-01-04 20:55:41 +00:00
<sub>*Managed by the [Sourcegraph Search Platform team](https://handbook.sourcegraph.com/departments/engineering/teams/search/core).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
Generated query for warning alert: `max((max_over_time(cadvisor_container_memory_usage_percentage_total{name=~"^zoekt-indexserver.*"}[5m])) >= 90)`
</details>
monitoring: improve solutions formatting, add team information (#14878)
2020-10-20 05:11:20 +00:00
<br />
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: improve containers metrics (#34808) part of - https://github.com/sourcegraph/sourcegraph/issues/33437 - https://github.com/sourcegraph/sourcegraph/issues/33438 This PR adds the following - A shared observable metric to indicate how many times a container (or child processes) has been oom kill - It's added to provisioning indicators since it's an indicator of underprovisioned resources - A new dashboard that shows resources usage of all containers at a glance - It contains two groups, one for all containers and one for containers that may have potential scaling issues - the first group is extremely noisy so it's hidden by default - the second group provides a more focused experience for site-admin to quickly find problematic resources <!-- > ~it looks like the alert table will be rendered regardless, which is pretty annoying (I guess the implementation of the monitoring stack is based on the assumption that one service/container will have its dashboard)~ __Questions & Notes__ - ~Should I make the new dashboard filterable by container name? like I only want to see the graph of `frontend`. Is that something that's valuable?~ - ~Any other metrics you all would like to see?~ - graph for total available resources on the VM compared to usage from https://github.com/sourcegraph/sourcegraph/issues/33437 is not really doable since we do not have `node-exporter` in anyone of our distro. We would have to tackle this from https://github.com/sourcegraph/customer/issues/806 --> ## Test plan connect to `demo`'s prom ```sh gcloud compute start-iap-tunnel default-$NEW_DEPLOYMENT-instance 9090 --local-host-port=localhost:4445 --zone us-central1-f --project $PROJECT_PREFIX-$CUSTOMER ``` or use dogfood ```sh kubectl port-forward svc/prometheus 4445:30090 ``` Update `dev/grafana/all/datasources.yaml` ```yml # Configuration for all non-Linux platforms. apiVersion: 1 datasources: - name: Prometheus type: prometheus access: proxy url: http://docker.for.mac.localhost:4445 # use the remote prometheus instance on demo instead of the local one isDefault: true editable: false - name: Jaeger type: jaeger access: proxy url: http://host.docker.internal:16686/-/debug/jaeger - name: Loki type: loki access: proxy url: http://host.docker.internal:3100 ``` Start monitoring stack ```sh sg run grafana ``` see screenshot > the first group is hidden by default ![CleanShot 2022-05-04 at 10 48 26](https://user-images.githubusercontent.com/8373004/166748470-22460552-6e51-4b29-a26b-4c5adbe1c404.png) ![CleanShot 2022-05-04 at 10 48 03](https://user-images.githubusercontent.com/8373004/166748396-1d410c35-e5fe-45e2-83bd-418d845413cb.png) ![CleanShot 2022-05-04 at 11 25 13](https://user-images.githubusercontent.com/8373004/166801266-21e51b68-65fd-4cb7-97de-83520e31e498.png)
2022-05-09 23:56:47 +00:00
## zoekt: container_oomkill_events_total
<p class="subtitle">container OOMKILL events total by instance</p>
**Descriptions**
- <span class="badge badge-warning">warning</span> zoekt: 1+ container OOMKILL events total by instance
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: improve containers metrics (#34808) part of - https://github.com/sourcegraph/sourcegraph/issues/33437 - https://github.com/sourcegraph/sourcegraph/issues/33438 This PR adds the following - A shared observable metric to indicate how many times a container (or child processes) has been oom kill - It's added to provisioning indicators since it's an indicator of underprovisioned resources - A new dashboard that shows resources usage of all containers at a glance - It contains two groups, one for all containers and one for containers that may have potential scaling issues - the first group is extremely noisy so it's hidden by default - the second group provides a more focused experience for site-admin to quickly find problematic resources <!-- > ~it looks like the alert table will be rendered regardless, which is pretty annoying (I guess the implementation of the monitoring stack is based on the assumption that one service/container will have its dashboard)~ __Questions & Notes__ - ~Should I make the new dashboard filterable by container name? like I only want to see the graph of `frontend`. Is that something that's valuable?~ - ~Any other metrics you all would like to see?~ - graph for total available resources on the VM compared to usage from https://github.com/sourcegraph/sourcegraph/issues/33437 is not really doable since we do not have `node-exporter` in anyone of our distro. We would have to tackle this from https://github.com/sourcegraph/customer/issues/806 --> ## Test plan connect to `demo`'s prom ```sh gcloud compute start-iap-tunnel default-$NEW_DEPLOYMENT-instance 9090 --local-host-port=localhost:4445 --zone us-central1-f --project $PROJECT_PREFIX-$CUSTOMER ``` or use dogfood ```sh kubectl port-forward svc/prometheus 4445:30090 ``` Update `dev/grafana/all/datasources.yaml` ```yml # Configuration for all non-Linux platforms. apiVersion: 1 datasources: - name: Prometheus type: prometheus access: proxy url: http://docker.for.mac.localhost:4445 # use the remote prometheus instance on demo instead of the local one isDefault: true editable: false - name: Jaeger type: jaeger access: proxy url: http://host.docker.internal:16686/-/debug/jaeger - name: Loki type: loki access: proxy url: http://host.docker.internal:3100 ``` Start monitoring stack ```sh sg run grafana ``` see screenshot > the first group is hidden by default ![CleanShot 2022-05-04 at 10 48 26](https://user-images.githubusercontent.com/8373004/166748470-22460552-6e51-4b29-a26b-4c5adbe1c404.png) ![CleanShot 2022-05-04 at 10 48 03](https://user-images.githubusercontent.com/8373004/166748396-1d410c35-e5fe-45e2-83bd-418d845413cb.png) ![CleanShot 2022-05-04 at 11 25 13](https://user-images.githubusercontent.com/8373004/166801266-21e51b68-65fd-4cb7-97de-83520e31e498.png)
2022-05-09 23:56:47 +00:00
- **Kubernetes:** Consider increasing memory limit in relevant `Deployment.yaml`.
- **Docker Compose:** Consider increasing `memory:` of zoekt-indexserver container in `docker-compose.yml`.
- More help interpreting this metric is available in the [dashboards reference](./dashboards.md#zoekt-container-oomkill-events-total).
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"warning_zoekt_container_oomkill_events_total"
]
```
Monitoring: update owner for search platform (#59316) The 'search core' team is now called 'search platform'. --------- Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2024-01-04 20:55:41 +00:00
<sub>*Managed by the [Sourcegraph Search Platform team](https://handbook.sourcegraph.com/departments/engineering/teams/search/core).*</sub>
monitoring: improve containers metrics (#34808) part of - https://github.com/sourcegraph/sourcegraph/issues/33437 - https://github.com/sourcegraph/sourcegraph/issues/33438 This PR adds the following - A shared observable metric to indicate how many times a container (or child processes) has been oom kill - It's added to provisioning indicators since it's an indicator of underprovisioned resources - A new dashboard that shows resources usage of all containers at a glance - It contains two groups, one for all containers and one for containers that may have potential scaling issues - the first group is extremely noisy so it's hidden by default - the second group provides a more focused experience for site-admin to quickly find problematic resources <!-- > ~it looks like the alert table will be rendered regardless, which is pretty annoying (I guess the implementation of the monitoring stack is based on the assumption that one service/container will have its dashboard)~ __Questions & Notes__ - ~Should I make the new dashboard filterable by container name? like I only want to see the graph of `frontend`. Is that something that's valuable?~ - ~Any other metrics you all would like to see?~ - graph for total available resources on the VM compared to usage from https://github.com/sourcegraph/sourcegraph/issues/33437 is not really doable since we do not have `node-exporter` in anyone of our distro. We would have to tackle this from https://github.com/sourcegraph/customer/issues/806 --> ## Test plan connect to `demo`'s prom ```sh gcloud compute start-iap-tunnel default-$NEW_DEPLOYMENT-instance 9090 --local-host-port=localhost:4445 --zone us-central1-f --project $PROJECT_PREFIX-$CUSTOMER ``` or use dogfood ```sh kubectl port-forward svc/prometheus 4445:30090 ``` Update `dev/grafana/all/datasources.yaml` ```yml # Configuration for all non-Linux platforms. apiVersion: 1 datasources: - name: Prometheus type: prometheus access: proxy url: http://docker.for.mac.localhost:4445 # use the remote prometheus instance on demo instead of the local one isDefault: true editable: false - name: Jaeger type: jaeger access: proxy url: http://host.docker.internal:16686/-/debug/jaeger - name: Loki type: loki access: proxy url: http://host.docker.internal:3100 ``` Start monitoring stack ```sh sg run grafana ``` see screenshot > the first group is hidden by default ![CleanShot 2022-05-04 at 10 48 26](https://user-images.githubusercontent.com/8373004/166748470-22460552-6e51-4b29-a26b-4c5adbe1c404.png) ![CleanShot 2022-05-04 at 10 48 03](https://user-images.githubusercontent.com/8373004/166748396-1d410c35-e5fe-45e2-83bd-418d845413cb.png) ![CleanShot 2022-05-04 at 11 25 13](https://user-images.githubusercontent.com/8373004/166801266-21e51b68-65fd-4cb7-97de-83520e31e498.png)
2022-05-09 23:56:47 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
monitoring: extract into a submodule (#45786) This change extracts `monitoring` into a submodule for import in `sourcegraph/controller` (https://github.com/sourcegraph/controller/pull/195) so that we can generate dashboards for Cloud instances. These steps were required: 1. Initialize a `go.mod` in `monitoring` 2. Extract `dev/sg/internal/cliutil` into `lib` to avoid illegal imports from `monitoring` 3. Add local replaces to both `sourcegraph/sourcegraph` and `monitoring` 4. `go mod tidy` on all submodules 5. Update `go generate ./monitoring` commands to use `sg`, since the `go generate` command no longer works 6. Update `grafana/build.sh`, `prometheus/build.sh` to build the submodule 7. Amend linters to check for multiple `go.mod` files and ban imports of `github.com/sourcegraph/sourcegraph` 8. Update `sg generate go` to run in directories rather than from root The only caveat is that if you use VS code, you will now need to open `monitoring` in a separate workspace or similar, like with `lib`. Co-authored-by: Joe Chen <joe@sourcegraph.com>
2022-12-19 17:49:25 +00:00
Generated query for warning alert: `max((max by (name) (container_oom_events_total{name=~"^zoekt-indexserver.*"})) >= 1)`
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
</details>
monitoring: improve containers metrics (#34808) part of - https://github.com/sourcegraph/sourcegraph/issues/33437 - https://github.com/sourcegraph/sourcegraph/issues/33438 This PR adds the following - A shared observable metric to indicate how many times a container (or child processes) has been oom kill - It's added to provisioning indicators since it's an indicator of underprovisioned resources - A new dashboard that shows resources usage of all containers at a glance - It contains two groups, one for all containers and one for containers that may have potential scaling issues - the first group is extremely noisy so it's hidden by default - the second group provides a more focused experience for site-admin to quickly find problematic resources <!-- > ~it looks like the alert table will be rendered regardless, which is pretty annoying (I guess the implementation of the monitoring stack is based on the assumption that one service/container will have its dashboard)~ __Questions & Notes__ - ~Should I make the new dashboard filterable by container name? like I only want to see the graph of `frontend`. Is that something that's valuable?~ - ~Any other metrics you all would like to see?~ - graph for total available resources on the VM compared to usage from https://github.com/sourcegraph/sourcegraph/issues/33437 is not really doable since we do not have `node-exporter` in anyone of our distro. We would have to tackle this from https://github.com/sourcegraph/customer/issues/806 --> ## Test plan connect to `demo`'s prom ```sh gcloud compute start-iap-tunnel default-$NEW_DEPLOYMENT-instance 9090 --local-host-port=localhost:4445 --zone us-central1-f --project $PROJECT_PREFIX-$CUSTOMER ``` or use dogfood ```sh kubectl port-forward svc/prometheus 4445:30090 ``` Update `dev/grafana/all/datasources.yaml` ```yml # Configuration for all non-Linux platforms. apiVersion: 1 datasources: - name: Prometheus type: prometheus access: proxy url: http://docker.for.mac.localhost:4445 # use the remote prometheus instance on demo instead of the local one isDefault: true editable: false - name: Jaeger type: jaeger access: proxy url: http://host.docker.internal:16686/-/debug/jaeger - name: Loki type: loki access: proxy url: http://host.docker.internal:3100 ``` Start monitoring stack ```sh sg run grafana ``` see screenshot > the first group is hidden by default ![CleanShot 2022-05-04 at 10 48 26](https://user-images.githubusercontent.com/8373004/166748470-22460552-6e51-4b29-a26b-4c5adbe1c404.png) ![CleanShot 2022-05-04 at 10 48 03](https://user-images.githubusercontent.com/8373004/166748396-1d410c35-e5fe-45e2-83bd-418d845413cb.png) ![CleanShot 2022-05-04 at 11 25 13](https://user-images.githubusercontent.com/8373004/166801266-21e51b68-65fd-4cb7-97de-83520e31e498.png)
2022-05-09 23:56:47 +00:00
<br />
zoekt: combine zoekt-webserver and zoekt-indexserver Grafana dashboards (#29768)
2022-01-14 18:17:07 +00:00
## zoekt: provisioning_container_cpu_usage_long_term
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
zoekt: combine zoekt-webserver and zoekt-indexserver Grafana dashboards (#29768)
2022-01-14 18:17:07 +00:00
<p class="subtitle">container cpu usage total (90th percentile over 1d) across all cores by instance</p>
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
zoekt: combine zoekt-webserver and zoekt-indexserver Grafana dashboards (#29768)
2022-01-14 18:17:07 +00:00
- <span class="badge badge-warning">warning</span> zoekt: 80%+ container cpu usage total (90th percentile over 1d) across all cores by instance for 336h0m0s
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
zoekt: combine zoekt-webserver and zoekt-indexserver Grafana dashboards (#29768)
2022-01-14 18:17:07 +00:00
- **Kubernetes:** Consider increasing CPU limits in the `Deployment.yaml` for the zoekt-webserver service.
- **Docker Compose:** Consider increasing `cpus:` of the zoekt-webserver container in `docker-compose.yml`.
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#zoekt-provisioning-container-cpu-usage-long-term).
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
```json
"observability.silenceAlerts": [
zoekt: combine zoekt-webserver and zoekt-indexserver Grafana dashboards (#29768)
2022-01-14 18:17:07 +00:00
"warning_zoekt_provisioning_container_cpu_usage_long_term"
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
]
```
Monitoring: update owner for search platform (#59316) The 'search core' team is now called 'search platform'. --------- Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2024-01-04 20:55:41 +00:00
<sub>*Managed by the [Sourcegraph Search Platform team](https://handbook.sourcegraph.com/departments/engineering/teams/search/core).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
Generated query for warning alert: `max((quantile_over_time(0.9, cadvisor_container_cpu_usage_percentage_total{name=~"^zoekt-webserver.*"}[1d])) >= 80)`
</details>
monitoring: improve solutions formatting, add team information (#14878)
2020-10-20 05:11:20 +00:00
<br />
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
zoekt: combine zoekt-webserver and zoekt-indexserver Grafana dashboards (#29768)
2022-01-14 18:17:07 +00:00
## zoekt: provisioning_container_memory_usage_long_term
monitoring: support multi-bound alerts, add lower-bound warnings for containers (#11082)
2020-06-16 07:09:25 +00:00
zoekt: combine zoekt-webserver and zoekt-indexserver Grafana dashboards (#29768)
2022-01-14 18:17:07 +00:00
<p class="subtitle">container memory usage (1d maximum) by instance</p>
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
monitoring: support multi-bound alerts, add lower-bound warnings for containers (#11082)
2020-06-16 07:09:25 +00:00
zoekt: combine zoekt-webserver and zoekt-indexserver Grafana dashboards (#29768)
2022-01-14 18:17:07 +00:00
- <span class="badge badge-warning">warning</span> zoekt: 80%+ container memory usage (1d maximum) by instance for 336h0m0s
monitoring: support multi-bound alerts, add lower-bound warnings for containers (#11082)
2020-06-16 07:09:25 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: support multi-bound alerts, add lower-bound warnings for containers (#11082)
2020-06-16 07:09:25 +00:00
zoekt: combine zoekt-webserver and zoekt-indexserver Grafana dashboards (#29768)
2022-01-14 18:17:07 +00:00
- **Kubernetes:** Consider increasing memory limits in the `Deployment.yaml` for the zoekt-webserver service.
- **Docker Compose:** Consider increasing `memory:` of the zoekt-webserver container in `docker-compose.yml`.
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#zoekt-provisioning-container-memory-usage-long-term).
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
zoekt: combine zoekt-webserver and zoekt-indexserver Grafana dashboards (#29768)
2022-01-14 18:17:07 +00:00
"warning_zoekt_provisioning_container_memory_usage_long_term"
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
]
```
monitoring: support multi-bound alerts, add lower-bound warnings for containers (#11082)
2020-06-16 07:09:25 +00:00
Monitoring: update owner for search platform (#59316) The 'search core' team is now called 'search platform'. --------- Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2024-01-04 20:55:41 +00:00
<sub>*Managed by the [Sourcegraph Search Platform team](https://handbook.sourcegraph.com/departments/engineering/teams/search/core).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
Generated query for warning alert: `max((max_over_time(cadvisor_container_memory_usage_percentage_total{name=~"^zoekt-webserver.*"}[1d])) >= 80)`
</details>
monitoring: improve solutions formatting, add team information (#14878)
2020-10-20 05:11:20 +00:00
<br />
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
zoekt: combine zoekt-webserver and zoekt-indexserver Grafana dashboards (#29768)
2022-01-14 18:17:07 +00:00
## zoekt: provisioning_container_cpu_usage_short_term
monitoring: support multi-bound alerts, add lower-bound warnings for containers (#11082)
2020-06-16 07:09:25 +00:00
zoekt: combine zoekt-webserver and zoekt-indexserver Grafana dashboards (#29768)
2022-01-14 18:17:07 +00:00
<p class="subtitle">container cpu usage total (5m maximum) across all cores by instance</p>
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
monitoring: support multi-bound alerts, add lower-bound warnings for containers (#11082)
2020-06-16 07:09:25 +00:00
zoekt: combine zoekt-webserver and zoekt-indexserver Grafana dashboards (#29768)
2022-01-14 18:17:07 +00:00
- <span class="badge badge-warning">warning</span> zoekt: 90%+ container cpu usage total (5m maximum) across all cores by instance for 30m0s
monitoring: support multi-bound alerts, add lower-bound warnings for containers (#11082)
2020-06-16 07:09:25 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: support multi-bound alerts, add lower-bound warnings for containers (#11082)
2020-06-16 07:09:25 +00:00
zoekt: combine zoekt-webserver and zoekt-indexserver Grafana dashboards (#29768)
2022-01-14 18:17:07 +00:00
- **Kubernetes:** Consider increasing CPU limits in the the relevant `Deployment.yaml`.
- **Docker Compose:** Consider increasing `cpus:` of the zoekt-webserver container in `docker-compose.yml`.
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#zoekt-provisioning-container-cpu-usage-short-term).
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
monitoring: support multi-bound alerts, add lower-bound warnings for containers (#11082)
2020-06-16 07:09:25 +00:00
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
```json
"observability.silenceAlerts": [
zoekt: combine zoekt-webserver and zoekt-indexserver Grafana dashboards (#29768)
2022-01-14 18:17:07 +00:00
"warning_zoekt_provisioning_container_cpu_usage_short_term"
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
]
```
Monitoring: update owner for search platform (#59316) The 'search core' team is now called 'search platform'. --------- Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2024-01-04 20:55:41 +00:00
<sub>*Managed by the [Sourcegraph Search Platform team](https://handbook.sourcegraph.com/departments/engineering/teams/search/core).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
Generated query for warning alert: `max((max_over_time(cadvisor_container_cpu_usage_percentage_total{name=~"^zoekt-webserver.*"}[5m])) >= 90)`
</details>
monitoring: improve solutions formatting, add team information (#14878)
2020-10-20 05:11:20 +00:00
<br />
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
zoekt: combine zoekt-webserver and zoekt-indexserver Grafana dashboards (#29768)
2022-01-14 18:17:07 +00:00
## zoekt: provisioning_container_memory_usage_short_term
monitoring: support multi-bound alerts, add lower-bound warnings for containers (#11082)
2020-06-16 07:09:25 +00:00
zoekt: combine zoekt-webserver and zoekt-indexserver Grafana dashboards (#29768)
2022-01-14 18:17:07 +00:00
<p class="subtitle">container memory usage (5m maximum) by instance</p>
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
monitoring: support multi-bound alerts, add lower-bound warnings for containers (#11082)
2020-06-16 07:09:25 +00:00
zoekt: combine zoekt-webserver and zoekt-indexserver Grafana dashboards (#29768)
2022-01-14 18:17:07 +00:00
- <span class="badge badge-warning">warning</span> zoekt: 90%+ container memory usage (5m maximum) by instance
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
zoekt: combine zoekt-webserver and zoekt-indexserver Grafana dashboards (#29768)
2022-01-14 18:17:07 +00:00
- **Kubernetes:** Consider increasing memory limit in relevant `Deployment.yaml`.
- **Docker Compose:** Consider increasing `memory:` of zoekt-webserver container in `docker-compose.yml`.
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#zoekt-provisioning-container-memory-usage-short-term).
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
zoekt: combine zoekt-webserver and zoekt-indexserver Grafana dashboards (#29768)
2022-01-14 18:17:07 +00:00
"warning_zoekt_provisioning_container_memory_usage_short_term"
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
]
```
monitoring: generate solutions documentation from Go monitoring definitions (#10142) * monitoring: fix incorrect usage of shared monitoring * monitoring: support generating solutions documentation from observable definitions
2020-04-23 20:09:35 +00:00
Monitoring: update owner for search platform (#59316) The 'search core' team is now called 'search platform'. --------- Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2024-01-04 20:55:41 +00:00
<sub>*Managed by the [Sourcegraph Search Platform team](https://handbook.sourcegraph.com/departments/engineering/teams/search/core).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
Generated query for warning alert: `max((max_over_time(cadvisor_container_memory_usage_percentage_total{name=~"^zoekt-webserver.*"}[5m])) >= 90)`
</details>
monitoring: improve solutions formatting, add team information (#14878)
2020-10-20 05:11:20 +00:00
<br />
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: improve containers metrics (#34808) part of - https://github.com/sourcegraph/sourcegraph/issues/33437 - https://github.com/sourcegraph/sourcegraph/issues/33438 This PR adds the following - A shared observable metric to indicate how many times a container (or child processes) has been oom kill - It's added to provisioning indicators since it's an indicator of underprovisioned resources - A new dashboard that shows resources usage of all containers at a glance - It contains two groups, one for all containers and one for containers that may have potential scaling issues - the first group is extremely noisy so it's hidden by default - the second group provides a more focused experience for site-admin to quickly find problematic resources <!-- > ~it looks like the alert table will be rendered regardless, which is pretty annoying (I guess the implementation of the monitoring stack is based on the assumption that one service/container will have its dashboard)~ __Questions & Notes__ - ~Should I make the new dashboard filterable by container name? like I only want to see the graph of `frontend`. Is that something that's valuable?~ - ~Any other metrics you all would like to see?~ - graph for total available resources on the VM compared to usage from https://github.com/sourcegraph/sourcegraph/issues/33437 is not really doable since we do not have `node-exporter` in anyone of our distro. We would have to tackle this from https://github.com/sourcegraph/customer/issues/806 --> ## Test plan connect to `demo`'s prom ```sh gcloud compute start-iap-tunnel default-$NEW_DEPLOYMENT-instance 9090 --local-host-port=localhost:4445 --zone us-central1-f --project $PROJECT_PREFIX-$CUSTOMER ``` or use dogfood ```sh kubectl port-forward svc/prometheus 4445:30090 ``` Update `dev/grafana/all/datasources.yaml` ```yml # Configuration for all non-Linux platforms. apiVersion: 1 datasources: - name: Prometheus type: prometheus access: proxy url: http://docker.for.mac.localhost:4445 # use the remote prometheus instance on demo instead of the local one isDefault: true editable: false - name: Jaeger type: jaeger access: proxy url: http://host.docker.internal:16686/-/debug/jaeger - name: Loki type: loki access: proxy url: http://host.docker.internal:3100 ``` Start monitoring stack ```sh sg run grafana ``` see screenshot > the first group is hidden by default ![CleanShot 2022-05-04 at 10 48 26](https://user-images.githubusercontent.com/8373004/166748470-22460552-6e51-4b29-a26b-4c5adbe1c404.png) ![CleanShot 2022-05-04 at 10 48 03](https://user-images.githubusercontent.com/8373004/166748396-1d410c35-e5fe-45e2-83bd-418d845413cb.png) ![CleanShot 2022-05-04 at 11 25 13](https://user-images.githubusercontent.com/8373004/166801266-21e51b68-65fd-4cb7-97de-83520e31e498.png)
2022-05-09 23:56:47 +00:00
## zoekt: container_oomkill_events_total
<p class="subtitle">container OOMKILL events total by instance</p>
**Descriptions**
- <span class="badge badge-warning">warning</span> zoekt: 1+ container OOMKILL events total by instance
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: improve containers metrics (#34808) part of - https://github.com/sourcegraph/sourcegraph/issues/33437 - https://github.com/sourcegraph/sourcegraph/issues/33438 This PR adds the following - A shared observable metric to indicate how many times a container (or child processes) has been oom kill - It's added to provisioning indicators since it's an indicator of underprovisioned resources - A new dashboard that shows resources usage of all containers at a glance - It contains two groups, one for all containers and one for containers that may have potential scaling issues - the first group is extremely noisy so it's hidden by default - the second group provides a more focused experience for site-admin to quickly find problematic resources <!-- > ~it looks like the alert table will be rendered regardless, which is pretty annoying (I guess the implementation of the monitoring stack is based on the assumption that one service/container will have its dashboard)~ __Questions & Notes__ - ~Should I make the new dashboard filterable by container name? like I only want to see the graph of `frontend`. Is that something that's valuable?~ - ~Any other metrics you all would like to see?~ - graph for total available resources on the VM compared to usage from https://github.com/sourcegraph/sourcegraph/issues/33437 is not really doable since we do not have `node-exporter` in anyone of our distro. We would have to tackle this from https://github.com/sourcegraph/customer/issues/806 --> ## Test plan connect to `demo`'s prom ```sh gcloud compute start-iap-tunnel default-$NEW_DEPLOYMENT-instance 9090 --local-host-port=localhost:4445 --zone us-central1-f --project $PROJECT_PREFIX-$CUSTOMER ``` or use dogfood ```sh kubectl port-forward svc/prometheus 4445:30090 ``` Update `dev/grafana/all/datasources.yaml` ```yml # Configuration for all non-Linux platforms. apiVersion: 1 datasources: - name: Prometheus type: prometheus access: proxy url: http://docker.for.mac.localhost:4445 # use the remote prometheus instance on demo instead of the local one isDefault: true editable: false - name: Jaeger type: jaeger access: proxy url: http://host.docker.internal:16686/-/debug/jaeger - name: Loki type: loki access: proxy url: http://host.docker.internal:3100 ``` Start monitoring stack ```sh sg run grafana ``` see screenshot > the first group is hidden by default ![CleanShot 2022-05-04 at 10 48 26](https://user-images.githubusercontent.com/8373004/166748470-22460552-6e51-4b29-a26b-4c5adbe1c404.png) ![CleanShot 2022-05-04 at 10 48 03](https://user-images.githubusercontent.com/8373004/166748396-1d410c35-e5fe-45e2-83bd-418d845413cb.png) ![CleanShot 2022-05-04 at 11 25 13](https://user-images.githubusercontent.com/8373004/166801266-21e51b68-65fd-4cb7-97de-83520e31e498.png)
2022-05-09 23:56:47 +00:00
- **Kubernetes:** Consider increasing memory limit in relevant `Deployment.yaml`.
- **Docker Compose:** Consider increasing `memory:` of zoekt-webserver container in `docker-compose.yml`.
- More help interpreting this metric is available in the [dashboards reference](./dashboards.md#zoekt-container-oomkill-events-total).
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"warning_zoekt_container_oomkill_events_total"
]
```
Monitoring: update owner for search platform (#59316) The 'search core' team is now called 'search platform'. --------- Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2024-01-04 20:55:41 +00:00
<sub>*Managed by the [Sourcegraph Search Platform team](https://handbook.sourcegraph.com/departments/engineering/teams/search/core).*</sub>
monitoring: improve containers metrics (#34808) part of - https://github.com/sourcegraph/sourcegraph/issues/33437 - https://github.com/sourcegraph/sourcegraph/issues/33438 This PR adds the following - A shared observable metric to indicate how many times a container (or child processes) has been oom kill - It's added to provisioning indicators since it's an indicator of underprovisioned resources - A new dashboard that shows resources usage of all containers at a glance - It contains two groups, one for all containers and one for containers that may have potential scaling issues - the first group is extremely noisy so it's hidden by default - the second group provides a more focused experience for site-admin to quickly find problematic resources <!-- > ~it looks like the alert table will be rendered regardless, which is pretty annoying (I guess the implementation of the monitoring stack is based on the assumption that one service/container will have its dashboard)~ __Questions & Notes__ - ~Should I make the new dashboard filterable by container name? like I only want to see the graph of `frontend`. Is that something that's valuable?~ - ~Any other metrics you all would like to see?~ - graph for total available resources on the VM compared to usage from https://github.com/sourcegraph/sourcegraph/issues/33437 is not really doable since we do not have `node-exporter` in anyone of our distro. We would have to tackle this from https://github.com/sourcegraph/customer/issues/806 --> ## Test plan connect to `demo`'s prom ```sh gcloud compute start-iap-tunnel default-$NEW_DEPLOYMENT-instance 9090 --local-host-port=localhost:4445 --zone us-central1-f --project $PROJECT_PREFIX-$CUSTOMER ``` or use dogfood ```sh kubectl port-forward svc/prometheus 4445:30090 ``` Update `dev/grafana/all/datasources.yaml` ```yml # Configuration for all non-Linux platforms. apiVersion: 1 datasources: - name: Prometheus type: prometheus access: proxy url: http://docker.for.mac.localhost:4445 # use the remote prometheus instance on demo instead of the local one isDefault: true editable: false - name: Jaeger type: jaeger access: proxy url: http://host.docker.internal:16686/-/debug/jaeger - name: Loki type: loki access: proxy url: http://host.docker.internal:3100 ``` Start monitoring stack ```sh sg run grafana ``` see screenshot > the first group is hidden by default ![CleanShot 2022-05-04 at 10 48 26](https://user-images.githubusercontent.com/8373004/166748470-22460552-6e51-4b29-a26b-4c5adbe1c404.png) ![CleanShot 2022-05-04 at 10 48 03](https://user-images.githubusercontent.com/8373004/166748396-1d410c35-e5fe-45e2-83bd-418d845413cb.png) ![CleanShot 2022-05-04 at 11 25 13](https://user-images.githubusercontent.com/8373004/166801266-21e51b68-65fd-4cb7-97de-83520e31e498.png)
2022-05-09 23:56:47 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
monitoring: extract into a submodule (#45786) This change extracts `monitoring` into a submodule for import in `sourcegraph/controller` (https://github.com/sourcegraph/controller/pull/195) so that we can generate dashboards for Cloud instances. These steps were required: 1. Initialize a `go.mod` in `monitoring` 2. Extract `dev/sg/internal/cliutil` into `lib` to avoid illegal imports from `monitoring` 3. Add local replaces to both `sourcegraph/sourcegraph` and `monitoring` 4. `go mod tidy` on all submodules 5. Update `go generate ./monitoring` commands to use `sg`, since the `go generate` command no longer works 6. Update `grafana/build.sh`, `prometheus/build.sh` to build the submodule 7. Amend linters to check for multiple `go.mod` files and ban imports of `github.com/sourcegraph/sourcegraph` 8. Update `sg generate go` to run in directories rather than from root The only caveat is that if you use VS code, you will now need to open `monitoring` in a separate workspace or similar, like with `lib`. Co-authored-by: Joe Chen <joe@sourcegraph.com>
2022-12-19 17:49:25 +00:00
Generated query for warning alert: `max((max by (name) (container_oom_events_total{name=~"^zoekt-webserver.*"})) >= 1)`
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
</details>
monitoring: improve containers metrics (#34808) part of - https://github.com/sourcegraph/sourcegraph/issues/33437 - https://github.com/sourcegraph/sourcegraph/issues/33438 This PR adds the following - A shared observable metric to indicate how many times a container (or child processes) has been oom kill - It's added to provisioning indicators since it's an indicator of underprovisioned resources - A new dashboard that shows resources usage of all containers at a glance - It contains two groups, one for all containers and one for containers that may have potential scaling issues - the first group is extremely noisy so it's hidden by default - the second group provides a more focused experience for site-admin to quickly find problematic resources <!-- > ~it looks like the alert table will be rendered regardless, which is pretty annoying (I guess the implementation of the monitoring stack is based on the assumption that one service/container will have its dashboard)~ __Questions & Notes__ - ~Should I make the new dashboard filterable by container name? like I only want to see the graph of `frontend`. Is that something that's valuable?~ - ~Any other metrics you all would like to see?~ - graph for total available resources on the VM compared to usage from https://github.com/sourcegraph/sourcegraph/issues/33437 is not really doable since we do not have `node-exporter` in anyone of our distro. We would have to tackle this from https://github.com/sourcegraph/customer/issues/806 --> ## Test plan connect to `demo`'s prom ```sh gcloud compute start-iap-tunnel default-$NEW_DEPLOYMENT-instance 9090 --local-host-port=localhost:4445 --zone us-central1-f --project $PROJECT_PREFIX-$CUSTOMER ``` or use dogfood ```sh kubectl port-forward svc/prometheus 4445:30090 ``` Update `dev/grafana/all/datasources.yaml` ```yml # Configuration for all non-Linux platforms. apiVersion: 1 datasources: - name: Prometheus type: prometheus access: proxy url: http://docker.for.mac.localhost:4445 # use the remote prometheus instance on demo instead of the local one isDefault: true editable: false - name: Jaeger type: jaeger access: proxy url: http://host.docker.internal:16686/-/debug/jaeger - name: Loki type: loki access: proxy url: http://host.docker.internal:3100 ``` Start monitoring stack ```sh sg run grafana ``` see screenshot > the first group is hidden by default ![CleanShot 2022-05-04 at 10 48 26](https://user-images.githubusercontent.com/8373004/166748470-22460552-6e51-4b29-a26b-4c5adbe1c404.png) ![CleanShot 2022-05-04 at 10 48 03](https://user-images.githubusercontent.com/8373004/166748396-1d410c35-e5fe-45e2-83bd-418d845413cb.png) ![CleanShot 2022-05-04 at 11 25 13](https://user-images.githubusercontent.com/8373004/166801266-21e51b68-65fd-4cb7-97de-83520e31e498.png)
2022-05-09 23:56:47 +00:00
<br />
monitoring: add golang monitoring for zoekt (#61731) Noticed this omission when I was wondering if we had goroutine leaks. Our other services define this. I added a simple way to indicate the container name in title since this is the first service we added which needs this. Test Plan: go test. Copy paste generated query into grafana explore on dotcom.
2024-04-12 13:46:11 +00:00
## zoekt: go_goroutines
<p class="subtitle">maximum active goroutines</p>
**Descriptions**
- <span class="badge badge-warning">warning</span> zoekt: 10000+ maximum active goroutines for 10m0s
**Next steps**
- More help interpreting this metric is available in the [dashboards reference](./dashboards.md#zoekt-go-goroutines).
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"warning_zoekt_go_goroutines"
]
```
<sub>*Managed by the [Sourcegraph Search Platform team](https://handbook.sourcegraph.com/departments/engineering/teams/search/core).*</sub>
<details>
<summary>Technical details</summary>
Generated query for warning alert: `max((max by (instance) (go_goroutines{job=~".*indexed-search-indexer"})) >= 10000)`
</details>
<br />
## zoekt: go_gc_duration_seconds
<p class="subtitle">maximum go garbage collection duration</p>
**Descriptions**
- <span class="badge badge-warning">warning</span> zoekt: 2s+ maximum go garbage collection duration
**Next steps**
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#zoekt-go-gc-duration-seconds).
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"warning_zoekt_go_gc_duration_seconds"
]
```
<sub>*Managed by the [Sourcegraph Search Platform team](https://handbook.sourcegraph.com/departments/engineering/teams/search/core).*</sub>
<details>
<summary>Technical details</summary>
Generated query for warning alert: `max((max by (instance) (go_gc_duration_seconds{job=~".*indexed-search-indexer"})) >= 2)`
</details>
<br />
## zoekt: go_goroutines
<p class="subtitle">maximum active goroutines</p>
**Descriptions**
- <span class="badge badge-warning">warning</span> zoekt: 10000+ maximum active goroutines for 10m0s
**Next steps**
- More help interpreting this metric is available in the [dashboards reference](./dashboards.md#zoekt-go-goroutines).
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"warning_zoekt_go_goroutines"
]
```
<sub>*Managed by the [Sourcegraph Search Platform team](https://handbook.sourcegraph.com/departments/engineering/teams/search/core).*</sub>
<details>
<summary>Technical details</summary>
Generated query for warning alert: `max((max by (instance) (go_goroutines{job=~".*indexed-search"})) >= 10000)`
</details>
<br />
## zoekt: go_gc_duration_seconds
<p class="subtitle">maximum go garbage collection duration</p>
**Descriptions**
- <span class="badge badge-warning">warning</span> zoekt: 2s+ maximum go garbage collection duration
**Next steps**
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#zoekt-go-gc-duration-seconds).
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"warning_zoekt_go_gc_duration_seconds"
]
```
<sub>*Managed by the [Sourcegraph Search Platform team](https://handbook.sourcegraph.com/departments/engineering/teams/search/core).*</sub>
<details>
<summary>Technical details</summary>
Generated query for warning alert: `max((max by (instance) (go_gc_duration_seconds{job=~".*indexed-search"})) >= 2)`
</details>
<br />
zoekt: combine zoekt-webserver and zoekt-indexserver Grafana dashboards (#29768)
2022-01-14 18:17:07 +00:00
## zoekt: pods_available_percentage
monitoring: support multi-bound alerts, add lower-bound warnings for containers (#11082)
2020-06-16 07:09:25 +00:00
zoekt: combine zoekt-webserver and zoekt-indexserver Grafana dashboards (#29768)
2022-01-14 18:17:07 +00:00
<p class="subtitle">percentage pods available</p>
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
monitoring: support multi-bound alerts, add lower-bound warnings for containers (#11082)
2020-06-16 07:09:25 +00:00
zoekt: combine zoekt-webserver and zoekt-indexserver Grafana dashboards (#29768)
2022-01-14 18:17:07 +00:00
- <span class="badge badge-critical">critical</span> zoekt: less than 90% percentage pods available for 10m0s
monitoring: support multi-bound alerts, add lower-bound warnings for containers (#11082)
2020-06-16 07:09:25 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: support multi-bound alerts, add lower-bound warnings for containers (#11082)
2020-06-16 07:09:25 +00:00
monitoring: forbid critical alerts with no solutions (#36423) Adds the restriction that critical alerts must have solutions defined as a hard requirement within the generator, and fixes all the outstanding issues: - Add PossibleSolutions for the pods_available_percentage and pg_up critical alerts (we have a similar solution doc in the container_missing alert) - Removes all the other critical alerts without possible solutions Co-authored-by: Michael Lin <mlzc@hey.com>
2022-06-02 15:51:43 +00:00
- Determine if the pod was OOM killed using `kubectl describe pod indexed-search` (look for `OOMKilled: true`) and, if so, consider increasing the memory limit in the relevant `Deployment.yaml`.
- Check the logs before the container restarted to see if there are `panic:` messages or similar using `kubectl logs -p indexed-search`.
zoekt: combine zoekt-webserver and zoekt-indexserver Grafana dashboards (#29768)
2022-01-14 18:17:07 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#zoekt-pods-available-percentage).
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
zoekt: combine zoekt-webserver and zoekt-indexserver Grafana dashboards (#29768)
2022-01-14 18:17:07 +00:00
"critical_zoekt_pods_available_percentage"
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
]
```
Monitoring: update owner for search platform (#59316) The 'search core' team is now called 'search platform'. --------- Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2024-01-04 20:55:41 +00:00
<sub>*Managed by the [Sourcegraph Search Platform team](https://handbook.sourcegraph.com/departments/engineering/teams/search/core).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
monitoring: extract into a submodule (#45786) This change extracts `monitoring` into a submodule for import in `sourcegraph/controller` (https://github.com/sourcegraph/controller/pull/195) so that we can generate dashboards for Cloud instances. These steps were required: 1. Initialize a `go.mod` in `monitoring` 2. Extract `dev/sg/internal/cliutil` into `lib` to avoid illegal imports from `monitoring` 3. Add local replaces to both `sourcegraph/sourcegraph` and `monitoring` 4. `go mod tidy` on all submodules 5. Update `go generate ./monitoring` commands to use `sg`, since the `go generate` command no longer works 6. Update `grafana/build.sh`, `prometheus/build.sh` to build the submodule 7. Amend linters to check for multiple `go.mod` files and ban imports of `github.com/sourcegraph/sourcegraph` 8. Update `sg generate go` to run in directories rather than from root The only caveat is that if you use VS code, you will now need to open `monitoring` in a separate workspace or similar, like with `lib`. Co-authored-by: Joe Chen <joe@sourcegraph.com>
2022-12-19 17:49:25 +00:00
Generated query for critical alert: `min((sum by (app) (up{app=~".*indexed-search"}) / count by (app) (up{app=~".*indexed-search"}) * 100) <= 90)`
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
</details>
monitoring: improve solutions formatting, add team information (#14878)
2020-10-20 05:11:20 +00:00
<br />
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: 'flatten' observables (reimplement #11985) (#17600) Re-introduce "flattening" of observables by wrapping queries in a min or max where appropriate (first added in #11985) to fix a regression introduced in #17599. Also add monitoring to ensure we have some visibility into rule failures, and reorganize Prometheus panels for uniformity. Co-authored-by: Gonzalo Peci <pecigonzalo@users.noreply.github.com>
2021-01-27 05:32:46 +00:00
## prometheus: prometheus_rule_eval_duration
monitoring: support multi-bound alerts, add lower-bound warnings for containers (#11082)
2020-06-16 07:09:25 +00:00
monitoring: 'flatten' observables (reimplement #11985) (#17600) Re-introduce "flattening" of observables by wrapping queries in a min or max where appropriate (first added in #11985) to fix a regression introduced in #17599. Also add monitoring to ensure we have some visibility into rule failures, and reorganize Prometheus panels for uniformity. Co-authored-by: Gonzalo Peci <pecigonzalo@users.noreply.github.com>
2021-01-27 05:32:46 +00:00
<p class="subtitle">average prometheus rule group evaluation duration over 10m by rule group</p>
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
monitoring: 'flatten' observables (reimplement #11985) (#17600) Re-introduce "flattening" of observables by wrapping queries in a min or max where appropriate (first added in #11985) to fix a regression introduced in #17599. Also add monitoring to ensure we have some visibility into rule failures, and reorganize Prometheus panels for uniformity. Co-authored-by: Gonzalo Peci <pecigonzalo@users.noreply.github.com>
2021-01-27 05:32:46 +00:00
- <span class="badge badge-warning">warning</span> prometheus: 30s+ average prometheus rule group evaluation duration over 10m by rule group
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
monitoring: 'flatten' observables (reimplement #11985) (#17600) Re-introduce "flattening" of observables by wrapping queries in a min or max where appropriate (first added in #11985) to fix a regression introduced in #17599. Also add monitoring to ensure we have some visibility into rule failures, and reorganize Prometheus panels for uniformity. Co-authored-by: Gonzalo Peci <pecigonzalo@users.noreply.github.com>
2021-01-27 05:32:46 +00:00
- Check the Container monitoring (not available on server) panels and try increasing resources for Prometheus if necessary.
- If the rule group taking a long time to evaluate belongs to `/sg_prometheus_addons`, try reducing the complexity of any custom Prometheus rules provided.
- If the rule group taking a long time to evaluate belongs to `/sg_config_prometheus`, please [open an issue](https://github.com/sourcegraph/sourcegraph/issues/new?assignees=&labels=&template=bug_report.md&title=).
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- More help interpreting this metric is available in the [dashboards reference](./dashboards.md#prometheus-prometheus-rule-eval-duration).
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
monitoring: 'flatten' observables (reimplement #11985) (#17600) Re-introduce "flattening" of observables by wrapping queries in a min or max where appropriate (first added in #11985) to fix a regression introduced in #17599. Also add monitoring to ensure we have some visibility into rule failures, and reorganize Prometheus panels for uniformity. Co-authored-by: Gonzalo Peci <pecigonzalo@users.noreply.github.com>
2021-01-27 05:32:46 +00:00
"warning_prometheus_prometheus_rule_eval_duration"
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
]
```
monitoring: test owners for valid Opsgenie teams and handbook pages (#59251) In INC-264 it seems that certain alerts - such as [zoekt: less than 90% percentage pods available for 10m0s](https://opsg.in/a/i/sourcegraph/178a626f-0f28-4295-bee9-84da988bb473-1703759057681) - don't seem to end up going anywhere because the ObservableOwner is defunct. This change adds _opt-in_ testing to report: 1. How many owners have valid Opsgenie teams 2. How many owners have valid handbook pages In addition, we collect ObservableOwners that pass the test and use it to generate configuration for `site.json` in Sourcegraph.com: https://github.com/sourcegraph/deploy-sourcegraph-cloud/pull/18338 - this helps ensure the list is valid and not deceptively high-coverage. The results are not great, but **enforcing** that owners are valid isn't currently in scope: ``` 6/10 ObservableOwners do not have valid Opsgenie teams 3/10 ObservableOwners do not point to valid handbook pages ``` I also removed some defunct/unused functionality/owners. ## Test plan To run these tests: ``` export OPSGENIE_API_KEY="..." go test -timeout 30s github.com/sourcegraph/sourcegraph/monitoring/monitoring -update -online ```
2023-12-29 22:07:35 +00:00
<sub>*Managed by the [Sourcegraph Infrastructure Org team](https://handbook.sourcegraph.com/departments/engineering/infrastructure).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
monitoring: extract into a submodule (#45786) This change extracts `monitoring` into a submodule for import in `sourcegraph/controller` (https://github.com/sourcegraph/controller/pull/195) so that we can generate dashboards for Cloud instances. These steps were required: 1. Initialize a `go.mod` in `monitoring` 2. Extract `dev/sg/internal/cliutil` into `lib` to avoid illegal imports from `monitoring` 3. Add local replaces to both `sourcegraph/sourcegraph` and `monitoring` 4. `go mod tidy` on all submodules 5. Update `go generate ./monitoring` commands to use `sg`, since the `go generate` command no longer works 6. Update `grafana/build.sh`, `prometheus/build.sh` to build the submodule 7. Amend linters to check for multiple `go.mod` files and ban imports of `github.com/sourcegraph/sourcegraph` 8. Update `sg generate go` to run in directories rather than from root The only caveat is that if you use VS code, you will now need to open `monitoring` in a separate workspace or similar, like with `lib`. Co-authored-by: Joe Chen <joe@sourcegraph.com>
2022-12-19 17:49:25 +00:00
Generated query for warning alert: `max((sum by (rule_group) (avg_over_time(prometheus_rule_group_last_duration_seconds[10m]))) >= 30)`
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
</details>
monitoring: improve solutions formatting, add team information (#14878)
2020-10-20 05:11:20 +00:00
<br />
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: 'flatten' observables (reimplement #11985) (#17600) Re-introduce "flattening" of observables by wrapping queries in a min or max where appropriate (first added in #11985) to fix a regression introduced in #17599. Also add monitoring to ensure we have some visibility into rule failures, and reorganize Prometheus panels for uniformity. Co-authored-by: Gonzalo Peci <pecigonzalo@users.noreply.github.com>
2021-01-27 05:32:46 +00:00
## prometheus: prometheus_rule_eval_failures
monitoring: migrate out-of-band alerts to the generator (#12391) Migrates all out-of-band alert rules defined by deploy-sourcegraph to the generator, such that each alert now has a panel as part of a service dashboard like all our other alerts, with a few changes and additions.
2020-07-28 02:17:56 +00:00
monitoring: 'flatten' observables (reimplement #11985) (#17600) Re-introduce "flattening" of observables by wrapping queries in a min or max where appropriate (first added in #11985) to fix a regression introduced in #17599. Also add monitoring to ensure we have some visibility into rule failures, and reorganize Prometheus panels for uniformity. Co-authored-by: Gonzalo Peci <pecigonzalo@users.noreply.github.com>
2021-01-27 05:32:46 +00:00
<p class="subtitle">failed prometheus rule evaluations over 5m by rule group</p>
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
monitoring: migrate out-of-band alerts to the generator (#12391) Migrates all out-of-band alert rules defined by deploy-sourcegraph to the generator, such that each alert now has a panel as part of a service dashboard like all our other alerts, with a few changes and additions.
2020-07-28 02:17:56 +00:00
monitoring: 'flatten' observables (reimplement #11985) (#17600) Re-introduce "flattening" of observables by wrapping queries in a min or max where appropriate (first added in #11985) to fix a regression introduced in #17599. Also add monitoring to ensure we have some visibility into rule failures, and reorganize Prometheus panels for uniformity. Co-authored-by: Gonzalo Peci <pecigonzalo@users.noreply.github.com>
2021-01-27 05:32:46 +00:00
- <span class="badge badge-warning">warning</span> prometheus: 0+ failed prometheus rule evaluations over 5m by rule group
monitoring: migrate out-of-band alerts to the generator (#12391) Migrates all out-of-band alert rules defined by deploy-sourcegraph to the generator, such that each alert now has a panel as part of a service dashboard like all our other alerts, with a few changes and additions.
2020-07-28 02:17:56 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: migrate out-of-band alerts to the generator (#12391) Migrates all out-of-band alert rules defined by deploy-sourcegraph to the generator, such that each alert now has a panel as part of a service dashboard like all our other alerts, with a few changes and additions.
2020-07-28 02:17:56 +00:00
monitoring: 'flatten' observables (reimplement #11985) (#17600) Re-introduce "flattening" of observables by wrapping queries in a min or max where appropriate (first added in #11985) to fix a regression introduced in #17599. Also add monitoring to ensure we have some visibility into rule failures, and reorganize Prometheus panels for uniformity. Co-authored-by: Gonzalo Peci <pecigonzalo@users.noreply.github.com>
2021-01-27 05:32:46 +00:00
- Check Prometheus logs for messages related to rule group evaluation (generally with log field `component="rule manager"`).
- If the rule group failing to evaluate belongs to `/sg_prometheus_addons`, ensure any custom Prometheus configuration provided is valid.
- If the rule group taking a long time to evaluate belongs to `/sg_config_prometheus`, please [open an issue](https://github.com/sourcegraph/sourcegraph/issues/new?assignees=&labels=&template=bug_report.md&title=).
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- More help interpreting this metric is available in the [dashboards reference](./dashboards.md#prometheus-prometheus-rule-eval-failures).
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
monitoring: 'flatten' observables (reimplement #11985) (#17600) Re-introduce "flattening" of observables by wrapping queries in a min or max where appropriate (first added in #11985) to fix a regression introduced in #17599. Also add monitoring to ensure we have some visibility into rule failures, and reorganize Prometheus panels for uniformity. Co-authored-by: Gonzalo Peci <pecigonzalo@users.noreply.github.com>
2021-01-27 05:32:46 +00:00
"warning_prometheus_prometheus_rule_eval_failures"
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
]
```
monitoring: migrate out-of-band alerts to the generator (#12391) Migrates all out-of-band alert rules defined by deploy-sourcegraph to the generator, such that each alert now has a panel as part of a service dashboard like all our other alerts, with a few changes and additions.
2020-07-28 02:17:56 +00:00
monitoring: test owners for valid Opsgenie teams and handbook pages (#59251) In INC-264 it seems that certain alerts - such as [zoekt: less than 90% percentage pods available for 10m0s](https://opsg.in/a/i/sourcegraph/178a626f-0f28-4295-bee9-84da988bb473-1703759057681) - don't seem to end up going anywhere because the ObservableOwner is defunct. This change adds _opt-in_ testing to report: 1. How many owners have valid Opsgenie teams 2. How many owners have valid handbook pages In addition, we collect ObservableOwners that pass the test and use it to generate configuration for `site.json` in Sourcegraph.com: https://github.com/sourcegraph/deploy-sourcegraph-cloud/pull/18338 - this helps ensure the list is valid and not deceptively high-coverage. The results are not great, but **enforcing** that owners are valid isn't currently in scope: ``` 6/10 ObservableOwners do not have valid Opsgenie teams 3/10 ObservableOwners do not point to valid handbook pages ``` I also removed some defunct/unused functionality/owners. ## Test plan To run these tests: ``` export OPSGENIE_API_KEY="..." go test -timeout 30s github.com/sourcegraph/sourcegraph/monitoring/monitoring -update -online ```
2023-12-29 22:07:35 +00:00
<sub>*Managed by the [Sourcegraph Infrastructure Org team](https://handbook.sourcegraph.com/departments/engineering/infrastructure).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
monitoring: extract into a submodule (#45786) This change extracts `monitoring` into a submodule for import in `sourcegraph/controller` (https://github.com/sourcegraph/controller/pull/195) so that we can generate dashboards for Cloud instances. These steps were required: 1. Initialize a `go.mod` in `monitoring` 2. Extract `dev/sg/internal/cliutil` into `lib` to avoid illegal imports from `monitoring` 3. Add local replaces to both `sourcegraph/sourcegraph` and `monitoring` 4. `go mod tidy` on all submodules 5. Update `go generate ./monitoring` commands to use `sg`, since the `go generate` command no longer works 6. Update `grafana/build.sh`, `prometheus/build.sh` to build the submodule 7. Amend linters to check for multiple `go.mod` files and ban imports of `github.com/sourcegraph/sourcegraph` 8. Update `sg generate go` to run in directories rather than from root The only caveat is that if you use VS code, you will now need to open `monitoring` in a separate workspace or similar, like with `lib`. Co-authored-by: Joe Chen <joe@sourcegraph.com>
2022-12-19 17:49:25 +00:00
Generated query for warning alert: `max((sum by (rule_group) (rate(prometheus_rule_evaluation_failures_total[5m]))) > 0)`
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
</details>
monitoring: improve solutions formatting, add team information (#14878)
2020-10-20 05:11:20 +00:00
<br />
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: 'flatten' observables (reimplement #11985) (#17600) Re-introduce "flattening" of observables by wrapping queries in a min or max where appropriate (first added in #11985) to fix a regression introduced in #17599. Also add monitoring to ensure we have some visibility into rule failures, and reorganize Prometheus panels for uniformity. Co-authored-by: Gonzalo Peci <pecigonzalo@users.noreply.github.com>
2021-01-27 05:32:46 +00:00
## prometheus: alertmanager_notification_latency
monitoring(prometheus): add new panels (#17246)
2021-01-15 04:12:37 +00:00
monitoring: 'flatten' observables (reimplement #11985) (#17600) Re-introduce "flattening" of observables by wrapping queries in a min or max where appropriate (first added in #11985) to fix a regression introduced in #17599. Also add monitoring to ensure we have some visibility into rule failures, and reorganize Prometheus panels for uniformity. Co-authored-by: Gonzalo Peci <pecigonzalo@users.noreply.github.com>
2021-01-27 05:32:46 +00:00
<p class="subtitle">alertmanager notification latency over 1m by integration</p>
monitoring(prometheus): add new panels (#17246)
2021-01-15 04:12:37 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
monitoring(prometheus): add new panels (#17246)
2021-01-15 04:12:37 +00:00
monitoring: 'flatten' observables (reimplement #11985) (#17600) Re-introduce "flattening" of observables by wrapping queries in a min or max where appropriate (first added in #11985) to fix a regression introduced in #17599. Also add monitoring to ensure we have some visibility into rule failures, and reorganize Prometheus panels for uniformity. Co-authored-by: Gonzalo Peci <pecigonzalo@users.noreply.github.com>
2021-01-27 05:32:46 +00:00
- <span class="badge badge-warning">warning</span> prometheus: 1s+ alertmanager notification latency over 1m by integration
monitoring(prometheus): add new panels (#17246)
2021-01-15 04:12:37 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring(prometheus): add new panels (#17246)
2021-01-15 04:12:37 +00:00
monitoring: 'flatten' observables (reimplement #11985) (#17600) Re-introduce "flattening" of observables by wrapping queries in a min or max where appropriate (first added in #11985) to fix a regression introduced in #17599. Also add monitoring to ensure we have some visibility into rule failures, and reorganize Prometheus panels for uniformity. Co-authored-by: Gonzalo Peci <pecigonzalo@users.noreply.github.com>
2021-01-27 05:32:46 +00:00
- Check the Container monitoring (not available on server) panels and try increasing resources for Prometheus if necessary.
Docs: update links to point to new site (#60381) We have a number of docs links in the product that point to the old doc site. Method: - Search the repo for `docs.sourcegraph.com` - Exclude the `doc/` dir, all test fixtures, and `CHANGELOG.md` - For each, replace `docs.sourcegraph.com` with `sourcegraph.com/docs` - Navigate to the resulting URL ensuring it's not a dead link, updating the URL if necessary Many of the URLs updated are just comments, but since I'm doing a manual audit of each URL anyways, I felt it was worth it to update these while I was at it.
2024-02-13 00:23:47 +00:00
- Ensure that your [`observability.alerts` configuration](https://sourcegraph.com/docs/admin/observability/alerting#setting-up-alerting) (in site configuration) is valid.
monitoring: 'flatten' observables (reimplement #11985) (#17600) Re-introduce "flattening" of observables by wrapping queries in a min or max where appropriate (first added in #11985) to fix a regression introduced in #17599. Also add monitoring to ensure we have some visibility into rule failures, and reorganize Prometheus panels for uniformity. Co-authored-by: Gonzalo Peci <pecigonzalo@users.noreply.github.com>
2021-01-27 05:32:46 +00:00
- Check if the relevant alert integration service is experiencing downtime or issues.
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#prometheus-alertmanager-notification-latency).
monitoring(prometheus): add new panels (#17246)
2021-01-15 04:12:37 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
monitoring: 'flatten' observables (reimplement #11985) (#17600) Re-introduce "flattening" of observables by wrapping queries in a min or max where appropriate (first added in #11985) to fix a regression introduced in #17599. Also add monitoring to ensure we have some visibility into rule failures, and reorganize Prometheus panels for uniformity. Co-authored-by: Gonzalo Peci <pecigonzalo@users.noreply.github.com>
2021-01-27 05:32:46 +00:00
"warning_prometheus_alertmanager_notification_latency"
monitoring(prometheus): add new panels (#17246)
2021-01-15 04:12:37 +00:00
]
```
monitoring: test owners for valid Opsgenie teams and handbook pages (#59251) In INC-264 it seems that certain alerts - such as [zoekt: less than 90% percentage pods available for 10m0s](https://opsg.in/a/i/sourcegraph/178a626f-0f28-4295-bee9-84da988bb473-1703759057681) - don't seem to end up going anywhere because the ObservableOwner is defunct. This change adds _opt-in_ testing to report: 1. How many owners have valid Opsgenie teams 2. How many owners have valid handbook pages In addition, we collect ObservableOwners that pass the test and use it to generate configuration for `site.json` in Sourcegraph.com: https://github.com/sourcegraph/deploy-sourcegraph-cloud/pull/18338 - this helps ensure the list is valid and not deceptively high-coverage. The results are not great, but **enforcing** that owners are valid isn't currently in scope: ``` 6/10 ObservableOwners do not have valid Opsgenie teams 3/10 ObservableOwners do not point to valid handbook pages ``` I also removed some defunct/unused functionality/owners. ## Test plan To run these tests: ``` export OPSGENIE_API_KEY="..." go test -timeout 30s github.com/sourcegraph/sourcegraph/monitoring/monitoring -update -online ```
2023-12-29 22:07:35 +00:00
<sub>*Managed by the [Sourcegraph Infrastructure Org team](https://handbook.sourcegraph.com/departments/engineering/infrastructure).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
monitoring: extract into a submodule (#45786) This change extracts `monitoring` into a submodule for import in `sourcegraph/controller` (https://github.com/sourcegraph/controller/pull/195) so that we can generate dashboards for Cloud instances. These steps were required: 1. Initialize a `go.mod` in `monitoring` 2. Extract `dev/sg/internal/cliutil` into `lib` to avoid illegal imports from `monitoring` 3. Add local replaces to both `sourcegraph/sourcegraph` and `monitoring` 4. `go mod tidy` on all submodules 5. Update `go generate ./monitoring` commands to use `sg`, since the `go generate` command no longer works 6. Update `grafana/build.sh`, `prometheus/build.sh` to build the submodule 7. Amend linters to check for multiple `go.mod` files and ban imports of `github.com/sourcegraph/sourcegraph` 8. Update `sg generate go` to run in directories rather than from root The only caveat is that if you use VS code, you will now need to open `monitoring` in a separate workspace or similar, like with `lib`. Co-authored-by: Joe Chen <joe@sourcegraph.com>
2022-12-19 17:49:25 +00:00
Generated query for warning alert: `max((sum by (integration) (rate(alertmanager_notification_latency_seconds_sum[1m]))) >= 1)`
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
</details>
monitoring(prometheus): add new panels (#17246)
2021-01-15 04:12:37 +00:00
<br />
monitoring: 'flatten' observables (reimplement #11985) (#17600) Re-introduce "flattening" of observables by wrapping queries in a min or max where appropriate (first added in #11985) to fix a regression introduced in #17599. Also add monitoring to ensure we have some visibility into rule failures, and reorganize Prometheus panels for uniformity. Co-authored-by: Gonzalo Peci <pecigonzalo@users.noreply.github.com>
2021-01-27 05:32:46 +00:00
## prometheus: alertmanager_notification_failures
monitoring(prometheus): add new panels (#17246)
2021-01-15 04:12:37 +00:00
monitoring: 'flatten' observables (reimplement #11985) (#17600) Re-introduce "flattening" of observables by wrapping queries in a min or max where appropriate (first added in #11985) to fix a regression introduced in #17599. Also add monitoring to ensure we have some visibility into rule failures, and reorganize Prometheus panels for uniformity. Co-authored-by: Gonzalo Peci <pecigonzalo@users.noreply.github.com>
2021-01-27 05:32:46 +00:00
<p class="subtitle">failed alertmanager notifications over 1m by integration</p>
monitoring(prometheus): add new panels (#17246)
2021-01-15 04:12:37 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
monitoring(prometheus): add new panels (#17246)
2021-01-15 04:12:37 +00:00
monitoring: 'flatten' observables (reimplement #11985) (#17600) Re-introduce "flattening" of observables by wrapping queries in a min or max where appropriate (first added in #11985) to fix a regression introduced in #17599. Also add monitoring to ensure we have some visibility into rule failures, and reorganize Prometheus panels for uniformity. Co-authored-by: Gonzalo Peci <pecigonzalo@users.noreply.github.com>
2021-01-27 05:32:46 +00:00
- <span class="badge badge-warning">warning</span> prometheus: 0+ failed alertmanager notifications over 1m by integration
monitoring(prometheus): add new panels (#17246)
2021-01-15 04:12:37 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring(prometheus): add new panels (#17246)
2021-01-15 04:12:37 +00:00
Docs: update links to point to new site (#60381) We have a number of docs links in the product that point to the old doc site. Method: - Search the repo for `docs.sourcegraph.com` - Exclude the `doc/` dir, all test fixtures, and `CHANGELOG.md` - For each, replace `docs.sourcegraph.com` with `sourcegraph.com/docs` - Navigate to the resulting URL ensuring it's not a dead link, updating the URL if necessary Many of the URLs updated are just comments, but since I'm doing a manual audit of each URL anyways, I felt it was worth it to update these while I was at it.
2024-02-13 00:23:47 +00:00
- Ensure that your [`observability.alerts` configuration](https://sourcegraph.com/docs/admin/observability/alerting#setting-up-alerting) (in site configuration) is valid.
monitoring: 'flatten' observables (reimplement #11985) (#17600) Re-introduce "flattening" of observables by wrapping queries in a min or max where appropriate (first added in #11985) to fix a regression introduced in #17599. Also add monitoring to ensure we have some visibility into rule failures, and reorganize Prometheus panels for uniformity. Co-authored-by: Gonzalo Peci <pecigonzalo@users.noreply.github.com>
2021-01-27 05:32:46 +00:00
- Check if the relevant alert integration service is experiencing downtime or issues.
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#prometheus-alertmanager-notification-failures).
monitoring(prometheus): add new panels (#17246)
2021-01-15 04:12:37 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
monitoring: 'flatten' observables (reimplement #11985) (#17600) Re-introduce "flattening" of observables by wrapping queries in a min or max where appropriate (first added in #11985) to fix a regression introduced in #17599. Also add monitoring to ensure we have some visibility into rule failures, and reorganize Prometheus panels for uniformity. Co-authored-by: Gonzalo Peci <pecigonzalo@users.noreply.github.com>
2021-01-27 05:32:46 +00:00
"warning_prometheus_alertmanager_notification_failures"
monitoring(prometheus): add new panels (#17246)
2021-01-15 04:12:37 +00:00
]
```
monitoring: test owners for valid Opsgenie teams and handbook pages (#59251) In INC-264 it seems that certain alerts - such as [zoekt: less than 90% percentage pods available for 10m0s](https://opsg.in/a/i/sourcegraph/178a626f-0f28-4295-bee9-84da988bb473-1703759057681) - don't seem to end up going anywhere because the ObservableOwner is defunct. This change adds _opt-in_ testing to report: 1. How many owners have valid Opsgenie teams 2. How many owners have valid handbook pages In addition, we collect ObservableOwners that pass the test and use it to generate configuration for `site.json` in Sourcegraph.com: https://github.com/sourcegraph/deploy-sourcegraph-cloud/pull/18338 - this helps ensure the list is valid and not deceptively high-coverage. The results are not great, but **enforcing** that owners are valid isn't currently in scope: ``` 6/10 ObservableOwners do not have valid Opsgenie teams 3/10 ObservableOwners do not point to valid handbook pages ``` I also removed some defunct/unused functionality/owners. ## Test plan To run these tests: ``` export OPSGENIE_API_KEY="..." go test -timeout 30s github.com/sourcegraph/sourcegraph/monitoring/monitoring -update -online ```
2023-12-29 22:07:35 +00:00
<sub>*Managed by the [Sourcegraph Infrastructure Org team](https://handbook.sourcegraph.com/departments/engineering/infrastructure).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
monitoring: extract into a submodule (#45786) This change extracts `monitoring` into a submodule for import in `sourcegraph/controller` (https://github.com/sourcegraph/controller/pull/195) so that we can generate dashboards for Cloud instances. These steps were required: 1. Initialize a `go.mod` in `monitoring` 2. Extract `dev/sg/internal/cliutil` into `lib` to avoid illegal imports from `monitoring` 3. Add local replaces to both `sourcegraph/sourcegraph` and `monitoring` 4. `go mod tidy` on all submodules 5. Update `go generate ./monitoring` commands to use `sg`, since the `go generate` command no longer works 6. Update `grafana/build.sh`, `prometheus/build.sh` to build the submodule 7. Amend linters to check for multiple `go.mod` files and ban imports of `github.com/sourcegraph/sourcegraph` 8. Update `sg generate go` to run in directories rather than from root The only caveat is that if you use VS code, you will now need to open `monitoring` in a separate workspace or similar, like with `lib`. Co-authored-by: Joe Chen <joe@sourcegraph.com>
2022-12-19 17:49:25 +00:00
Generated query for warning alert: `max((sum by (integration) (rate(alertmanager_notifications_failed_total[1m]))) > 0)`
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
</details>
monitoring(prometheus): add new panels (#17246)
2021-01-15 04:12:37 +00:00
<br />
## prometheus: prometheus_config_status
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
<p class="subtitle">prometheus configuration reload status</p>
monitoring(prometheus): add new panels (#17246)
2021-01-15 04:12:37 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
monitoring(prometheus): add new panels (#17246)
2021-01-15 04:12:37 +00:00
- <span class="badge badge-warning">warning</span> prometheus: less than 1 prometheus configuration reload status
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring(prometheus): add new panels (#17246)
2021-01-15 04:12:37 +00:00
- Check Prometheus logs for messages related to configuration loading.
Docs: update links to point to new site (#60381) We have a number of docs links in the product that point to the old doc site. Method: - Search the repo for `docs.sourcegraph.com` - Exclude the `doc/` dir, all test fixtures, and `CHANGELOG.md` - For each, replace `docs.sourcegraph.com` with `sourcegraph.com/docs` - Navigate to the resulting URL ensuring it's not a dead link, updating the URL if necessary Many of the URLs updated are just comments, but since I'm doing a manual audit of each URL anyways, I felt it was worth it to update these while I was at it.
2024-02-13 00:23:47 +00:00
- Ensure any [custom configuration you have provided Prometheus](https://sourcegraph.com/docs/admin/observability/metrics#prometheus-configuration) is valid.
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- More help interpreting this metric is available in the [dashboards reference](./dashboards.md#prometheus-prometheus-config-status).
monitoring(prometheus): add new panels (#17246)
2021-01-15 04:12:37 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"warning_prometheus_prometheus_config_status"
]
```
monitoring: test owners for valid Opsgenie teams and handbook pages (#59251) In INC-264 it seems that certain alerts - such as [zoekt: less than 90% percentage pods available for 10m0s](https://opsg.in/a/i/sourcegraph/178a626f-0f28-4295-bee9-84da988bb473-1703759057681) - don't seem to end up going anywhere because the ObservableOwner is defunct. This change adds _opt-in_ testing to report: 1. How many owners have valid Opsgenie teams 2. How many owners have valid handbook pages In addition, we collect ObservableOwners that pass the test and use it to generate configuration for `site.json` in Sourcegraph.com: https://github.com/sourcegraph/deploy-sourcegraph-cloud/pull/18338 - this helps ensure the list is valid and not deceptively high-coverage. The results are not great, but **enforcing** that owners are valid isn't currently in scope: ``` 6/10 ObservableOwners do not have valid Opsgenie teams 3/10 ObservableOwners do not point to valid handbook pages ``` I also removed some defunct/unused functionality/owners. ## Test plan To run these tests: ``` export OPSGENIE_API_KEY="..." go test -timeout 30s github.com/sourcegraph/sourcegraph/monitoring/monitoring -update -online ```
2023-12-29 22:07:35 +00:00
<sub>*Managed by the [Sourcegraph Infrastructure Org team](https://handbook.sourcegraph.com/departments/engineering/infrastructure).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
Generated query for warning alert: `min((prometheus_config_last_reload_successful) < 1)`
</details>
monitoring(prometheus): add new panels (#17246)
2021-01-15 04:12:37 +00:00
<br />
monitoring: 'flatten' observables (reimplement #11985) (#17600) Re-introduce "flattening" of observables by wrapping queries in a min or max where appropriate (first added in #11985) to fix a regression introduced in #17599. Also add monitoring to ensure we have some visibility into rule failures, and reorganize Prometheus panels for uniformity. Co-authored-by: Gonzalo Peci <pecigonzalo@users.noreply.github.com>
2021-01-27 05:32:46 +00:00
## prometheus: alertmanager_config_status
<p class="subtitle">alertmanager configuration reload status</p>
**Descriptions**
- <span class="badge badge-warning">warning</span> prometheus: less than 1 alertmanager configuration reload status
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: 'flatten' observables (reimplement #11985) (#17600) Re-introduce "flattening" of observables by wrapping queries in a min or max where appropriate (first added in #11985) to fix a regression introduced in #17599. Also add monitoring to ensure we have some visibility into rule failures, and reorganize Prometheus panels for uniformity. Co-authored-by: Gonzalo Peci <pecigonzalo@users.noreply.github.com>
2021-01-27 05:32:46 +00:00
Docs: update links to point to new site (#60381) We have a number of docs links in the product that point to the old doc site. Method: - Search the repo for `docs.sourcegraph.com` - Exclude the `doc/` dir, all test fixtures, and `CHANGELOG.md` - For each, replace `docs.sourcegraph.com` with `sourcegraph.com/docs` - Navigate to the resulting URL ensuring it's not a dead link, updating the URL if necessary Many of the URLs updated are just comments, but since I'm doing a manual audit of each URL anyways, I felt it was worth it to update these while I was at it.
2024-02-13 00:23:47 +00:00
- Ensure that your [`observability.alerts` configuration](https://sourcegraph.com/docs/admin/observability/alerting#setting-up-alerting) (in site configuration) is valid.
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- More help interpreting this metric is available in the [dashboards reference](./dashboards.md#prometheus-alertmanager-config-status).
monitoring: 'flatten' observables (reimplement #11985) (#17600) Re-introduce "flattening" of observables by wrapping queries in a min or max where appropriate (first added in #11985) to fix a regression introduced in #17599. Also add monitoring to ensure we have some visibility into rule failures, and reorganize Prometheus panels for uniformity. Co-authored-by: Gonzalo Peci <pecigonzalo@users.noreply.github.com>
2021-01-27 05:32:46 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"warning_prometheus_alertmanager_config_status"
]
```
monitoring: test owners for valid Opsgenie teams and handbook pages (#59251) In INC-264 it seems that certain alerts - such as [zoekt: less than 90% percentage pods available for 10m0s](https://opsg.in/a/i/sourcegraph/178a626f-0f28-4295-bee9-84da988bb473-1703759057681) - don't seem to end up going anywhere because the ObservableOwner is defunct. This change adds _opt-in_ testing to report: 1. How many owners have valid Opsgenie teams 2. How many owners have valid handbook pages In addition, we collect ObservableOwners that pass the test and use it to generate configuration for `site.json` in Sourcegraph.com: https://github.com/sourcegraph/deploy-sourcegraph-cloud/pull/18338 - this helps ensure the list is valid and not deceptively high-coverage. The results are not great, but **enforcing** that owners are valid isn't currently in scope: ``` 6/10 ObservableOwners do not have valid Opsgenie teams 3/10 ObservableOwners do not point to valid handbook pages ``` I also removed some defunct/unused functionality/owners. ## Test plan To run these tests: ``` export OPSGENIE_API_KEY="..." go test -timeout 30s github.com/sourcegraph/sourcegraph/monitoring/monitoring -update -online ```
2023-12-29 22:07:35 +00:00
<sub>*Managed by the [Sourcegraph Infrastructure Org team](https://handbook.sourcegraph.com/departments/engineering/infrastructure).*</sub>
monitoring: 'flatten' observables (reimplement #11985) (#17600) Re-introduce "flattening" of observables by wrapping queries in a min or max where appropriate (first added in #11985) to fix a regression introduced in #17599. Also add monitoring to ensure we have some visibility into rule failures, and reorganize Prometheus panels for uniformity. Co-authored-by: Gonzalo Peci <pecigonzalo@users.noreply.github.com>
2021-01-27 05:32:46 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
Generated query for warning alert: `min((alertmanager_config_last_reload_successful) < 1)`
</details>
monitoring: 'flatten' observables (reimplement #11985) (#17600) Re-introduce "flattening" of observables by wrapping queries in a min or max where appropriate (first added in #11985) to fix a regression introduced in #17599. Also add monitoring to ensure we have some visibility into rule failures, and reorganize Prometheus panels for uniformity. Co-authored-by: Gonzalo Peci <pecigonzalo@users.noreply.github.com>
2021-01-27 05:32:46 +00:00
<br />
## prometheus: prometheus_tsdb_op_failure
<p class="subtitle">prometheus tsdb failures by operation over 1m by operation</p>
**Descriptions**
- <span class="badge badge-warning">warning</span> prometheus: 0+ prometheus tsdb failures by operation over 1m by operation
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: 'flatten' observables (reimplement #11985) (#17600) Re-introduce "flattening" of observables by wrapping queries in a min or max where appropriate (first added in #11985) to fix a regression introduced in #17599. Also add monitoring to ensure we have some visibility into rule failures, and reorganize Prometheus panels for uniformity. Co-authored-by: Gonzalo Peci <pecigonzalo@users.noreply.github.com>
2021-01-27 05:32:46 +00:00
- Check Prometheus logs for messages related to the failing operation.
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#prometheus-prometheus-tsdb-op-failure).
monitoring: 'flatten' observables (reimplement #11985) (#17600) Re-introduce "flattening" of observables by wrapping queries in a min or max where appropriate (first added in #11985) to fix a regression introduced in #17599. Also add monitoring to ensure we have some visibility into rule failures, and reorganize Prometheus panels for uniformity. Co-authored-by: Gonzalo Peci <pecigonzalo@users.noreply.github.com>
2021-01-27 05:32:46 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"warning_prometheus_prometheus_tsdb_op_failure"
]
```
monitoring: test owners for valid Opsgenie teams and handbook pages (#59251) In INC-264 it seems that certain alerts - such as [zoekt: less than 90% percentage pods available for 10m0s](https://opsg.in/a/i/sourcegraph/178a626f-0f28-4295-bee9-84da988bb473-1703759057681) - don't seem to end up going anywhere because the ObservableOwner is defunct. This change adds _opt-in_ testing to report: 1. How many owners have valid Opsgenie teams 2. How many owners have valid handbook pages In addition, we collect ObservableOwners that pass the test and use it to generate configuration for `site.json` in Sourcegraph.com: https://github.com/sourcegraph/deploy-sourcegraph-cloud/pull/18338 - this helps ensure the list is valid and not deceptively high-coverage. The results are not great, but **enforcing** that owners are valid isn't currently in scope: ``` 6/10 ObservableOwners do not have valid Opsgenie teams 3/10 ObservableOwners do not point to valid handbook pages ``` I also removed some defunct/unused functionality/owners. ## Test plan To run these tests: ``` export OPSGENIE_API_KEY="..." go test -timeout 30s github.com/sourcegraph/sourcegraph/monitoring/monitoring -update -online ```
2023-12-29 22:07:35 +00:00
<sub>*Managed by the [Sourcegraph Infrastructure Org team](https://handbook.sourcegraph.com/departments/engineering/infrastructure).*</sub>
monitoring: 'flatten' observables (reimplement #11985) (#17600) Re-introduce "flattening" of observables by wrapping queries in a min or max where appropriate (first added in #11985) to fix a regression introduced in #17599. Also add monitoring to ensure we have some visibility into rule failures, and reorganize Prometheus panels for uniformity. Co-authored-by: Gonzalo Peci <pecigonzalo@users.noreply.github.com>
2021-01-27 05:32:46 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
Generated query for warning alert: `max((increase(label_replace({__name__=~"prometheus_tsdb_(.*)_failed_total"}, "operation", "$1", "__name__", "(.+)s_failed_total")[5m:1m])) > 0)`
</details>
monitoring: 'flatten' observables (reimplement #11985) (#17600) Re-introduce "flattening" of observables by wrapping queries in a min or max where appropriate (first added in #11985) to fix a regression introduced in #17599. Also add monitoring to ensure we have some visibility into rule failures, and reorganize Prometheus panels for uniformity. Co-authored-by: Gonzalo Peci <pecigonzalo@users.noreply.github.com>
2021-01-27 05:32:46 +00:00
<br />
monitoring(prometheus): add new panels (#17246)
2021-01-15 04:12:37 +00:00
## prometheus: prometheus_target_sample_exceeded
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
<p class="subtitle">prometheus scrapes that exceed the sample limit over 10m</p>
monitoring(prometheus): add new panels (#17246)
2021-01-15 04:12:37 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
monitoring(prometheus): add new panels (#17246)
2021-01-15 04:12:37 +00:00
- <span class="badge badge-warning">warning</span> prometheus: 0+ prometheus scrapes that exceed the sample limit over 10m
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring(prometheus): add new panels (#17246)
2021-01-15 04:12:37 +00:00
- Check Prometheus logs for messages related to target scrape failures.
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#prometheus-prometheus-target-sample-exceeded).
monitoring(prometheus): add new panels (#17246)
2021-01-15 04:12:37 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"warning_prometheus_prometheus_target_sample_exceeded"
]
```
monitoring: test owners for valid Opsgenie teams and handbook pages (#59251) In INC-264 it seems that certain alerts - such as [zoekt: less than 90% percentage pods available for 10m0s](https://opsg.in/a/i/sourcegraph/178a626f-0f28-4295-bee9-84da988bb473-1703759057681) - don't seem to end up going anywhere because the ObservableOwner is defunct. This change adds _opt-in_ testing to report: 1. How many owners have valid Opsgenie teams 2. How many owners have valid handbook pages In addition, we collect ObservableOwners that pass the test and use it to generate configuration for `site.json` in Sourcegraph.com: https://github.com/sourcegraph/deploy-sourcegraph-cloud/pull/18338 - this helps ensure the list is valid and not deceptively high-coverage. The results are not great, but **enforcing** that owners are valid isn't currently in scope: ``` 6/10 ObservableOwners do not have valid Opsgenie teams 3/10 ObservableOwners do not point to valid handbook pages ``` I also removed some defunct/unused functionality/owners. ## Test plan To run these tests: ``` export OPSGENIE_API_KEY="..." go test -timeout 30s github.com/sourcegraph/sourcegraph/monitoring/monitoring -update -online ```
2023-12-29 22:07:35 +00:00
<sub>*Managed by the [Sourcegraph Infrastructure Org team](https://handbook.sourcegraph.com/departments/engineering/infrastructure).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
Generated query for warning alert: `max((increase(prometheus_target_scrapes_exceeded_sample_limit_total[10m])) > 0)`
</details>
monitoring(prometheus): add new panels (#17246)
2021-01-15 04:12:37 +00:00
<br />
## prometheus: prometheus_target_sample_duplicate
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
<p class="subtitle">prometheus scrapes rejected due to duplicate timestamps over 10m</p>
monitoring(prometheus): add new panels (#17246)
2021-01-15 04:12:37 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
monitoring(prometheus): add new panels (#17246)
2021-01-15 04:12:37 +00:00
- <span class="badge badge-warning">warning</span> prometheus: 0+ prometheus scrapes rejected due to duplicate timestamps over 10m
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring(prometheus): add new panels (#17246)
2021-01-15 04:12:37 +00:00
- Check Prometheus logs for messages related to target scrape failures.
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#prometheus-prometheus-target-sample-duplicate).
monitoring(prometheus): add new panels (#17246)
2021-01-15 04:12:37 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"warning_prometheus_prometheus_target_sample_duplicate"
]
```
monitoring: test owners for valid Opsgenie teams and handbook pages (#59251) In INC-264 it seems that certain alerts - such as [zoekt: less than 90% percentage pods available for 10m0s](https://opsg.in/a/i/sourcegraph/178a626f-0f28-4295-bee9-84da988bb473-1703759057681) - don't seem to end up going anywhere because the ObservableOwner is defunct. This change adds _opt-in_ testing to report: 1. How many owners have valid Opsgenie teams 2. How many owners have valid handbook pages In addition, we collect ObservableOwners that pass the test and use it to generate configuration for `site.json` in Sourcegraph.com: https://github.com/sourcegraph/deploy-sourcegraph-cloud/pull/18338 - this helps ensure the list is valid and not deceptively high-coverage. The results are not great, but **enforcing** that owners are valid isn't currently in scope: ``` 6/10 ObservableOwners do not have valid Opsgenie teams 3/10 ObservableOwners do not point to valid handbook pages ``` I also removed some defunct/unused functionality/owners. ## Test plan To run these tests: ``` export OPSGENIE_API_KEY="..." go test -timeout 30s github.com/sourcegraph/sourcegraph/monitoring/monitoring -update -online ```
2023-12-29 22:07:35 +00:00
<sub>*Managed by the [Sourcegraph Infrastructure Org team](https://handbook.sourcegraph.com/departments/engineering/infrastructure).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
Generated query for warning alert: `max((increase(prometheus_target_scrapes_sample_duplicate_timestamp_total[10m])) > 0)`
</details>
monitoring(prometheus): add new panels (#17246)
2021-01-15 04:12:37 +00:00
<br />
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
## prometheus: container_cpu_usage
monitoring: migrate out-of-band alerts to the generator (#12391) Migrates all out-of-band alert rules defined by deploy-sourcegraph to the generator, such that each alert now has a panel as part of a service dashboard like all our other alerts, with a few changes and additions.
2020-07-28 02:17:56 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
<p class="subtitle">container cpu usage total (1m average) across all cores by instance</p>
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
monitoring: migrate out-of-band alerts to the generator (#12391) Migrates all out-of-band alert rules defined by deploy-sourcegraph to the generator, such that each alert now has a panel as part of a service dashboard like all our other alerts, with a few changes and additions.
2020-07-28 02:17:56 +00:00
monitoring: dashboards reference, refresh observability docs (#16939) - New monitoring definition field, 'Interpretation' - New generated dashboards reference - Grafana panels now link to alert solutions and dashboards reference - Refreshed observability docs Co-authored-by: uwedeportivo <534011+uwedeportivo@users.noreply.github.com>
2020-12-23 02:36:41 +00:00
- <span class="badge badge-warning">warning</span> prometheus: 99%+ container cpu usage total (1m average) across all cores by instance
monitoring: migrate out-of-band alerts to the generator (#12391) Migrates all out-of-band alert rules defined by deploy-sourcegraph to the generator, such that each alert now has a panel as part of a service dashboard like all our other alerts, with a few changes and additions.
2020-07-28 02:17:56 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: migrate out-of-band alerts to the generator (#12391) Migrates all out-of-band alert rules defined by deploy-sourcegraph to the generator, such that each alert now has a panel as part of a service dashboard like all our other alerts, with a few changes and additions.
2020-07-28 02:17:56 +00:00
- **Kubernetes:** Consider increasing CPU limits in the the relevant `Deployment.yaml`.
- **Docker Compose:** Consider increasing `cpus:` of the prometheus container in `docker-compose.yml`.
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#prometheus-container-cpu-usage).
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
monitoring: migrate out-of-band alerts to the generator (#12391) Migrates all out-of-band alert rules defined by deploy-sourcegraph to the generator, such that each alert now has a panel as part of a service dashboard like all our other alerts, with a few changes and additions.
2020-07-28 02:17:56 +00:00
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
```json
"observability.silenceAlerts": [
"warning_prometheus_container_cpu_usage"
]
```
monitoring: test owners for valid Opsgenie teams and handbook pages (#59251) In INC-264 it seems that certain alerts - such as [zoekt: less than 90% percentage pods available for 10m0s](https://opsg.in/a/i/sourcegraph/178a626f-0f28-4295-bee9-84da988bb473-1703759057681) - don't seem to end up going anywhere because the ObservableOwner is defunct. This change adds _opt-in_ testing to report: 1. How many owners have valid Opsgenie teams 2. How many owners have valid handbook pages In addition, we collect ObservableOwners that pass the test and use it to generate configuration for `site.json` in Sourcegraph.com: https://github.com/sourcegraph/deploy-sourcegraph-cloud/pull/18338 - this helps ensure the list is valid and not deceptively high-coverage. The results are not great, but **enforcing** that owners are valid isn't currently in scope: ``` 6/10 ObservableOwners do not have valid Opsgenie teams 3/10 ObservableOwners do not point to valid handbook pages ``` I also removed some defunct/unused functionality/owners. ## Test plan To run these tests: ``` export OPSGENIE_API_KEY="..." go test -timeout 30s github.com/sourcegraph/sourcegraph/monitoring/monitoring -update -online ```
2023-12-29 22:07:35 +00:00
<sub>*Managed by the [Sourcegraph Infrastructure Org team](https://handbook.sourcegraph.com/departments/engineering/infrastructure).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
Generated query for warning alert: `max((cadvisor_container_cpu_usage_percentage_total{name=~"^prometheus.*"}) >= 99)`
</details>
monitoring: improve solutions formatting, add team information (#14878)
2020-10-20 05:11:20 +00:00
<br />
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
## prometheus: container_memory_usage
monitoring: migrate out-of-band alerts to the generator (#12391) Migrates all out-of-band alert rules defined by deploy-sourcegraph to the generator, such that each alert now has a panel as part of a service dashboard like all our other alerts, with a few changes and additions.
2020-07-28 02:17:56 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
<p class="subtitle">container memory usage by instance</p>
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
monitoring: migrate out-of-band alerts to the generator (#12391) Migrates all out-of-band alert rules defined by deploy-sourcegraph to the generator, such that each alert now has a panel as part of a service dashboard like all our other alerts, with a few changes and additions.
2020-07-28 02:17:56 +00:00
monitoring: dashboards reference, refresh observability docs (#16939) - New monitoring definition field, 'Interpretation' - New generated dashboards reference - Grafana panels now link to alert solutions and dashboards reference - Refreshed observability docs Co-authored-by: uwedeportivo <534011+uwedeportivo@users.noreply.github.com>
2020-12-23 02:36:41 +00:00
- <span class="badge badge-warning">warning</span> prometheus: 99%+ container memory usage by instance
monitoring: migrate out-of-band alerts to the generator (#12391) Migrates all out-of-band alert rules defined by deploy-sourcegraph to the generator, such that each alert now has a panel as part of a service dashboard like all our other alerts, with a few changes and additions.
2020-07-28 02:17:56 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: migrate out-of-band alerts to the generator (#12391) Migrates all out-of-band alert rules defined by deploy-sourcegraph to the generator, such that each alert now has a panel as part of a service dashboard like all our other alerts, with a few changes and additions.
2020-07-28 02:17:56 +00:00
- **Kubernetes:** Consider increasing memory limit in relevant `Deployment.yaml`.
- **Docker Compose:** Consider increasing `memory:` of prometheus container in `docker-compose.yml`.
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#prometheus-container-memory-usage).
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"warning_prometheus_container_memory_usage"
]
```
monitoring: migrate out-of-band alerts to the generator (#12391) Migrates all out-of-band alert rules defined by deploy-sourcegraph to the generator, such that each alert now has a panel as part of a service dashboard like all our other alerts, with a few changes and additions.
2020-07-28 02:17:56 +00:00
monitoring: test owners for valid Opsgenie teams and handbook pages (#59251) In INC-264 it seems that certain alerts - such as [zoekt: less than 90% percentage pods available for 10m0s](https://opsg.in/a/i/sourcegraph/178a626f-0f28-4295-bee9-84da988bb473-1703759057681) - don't seem to end up going anywhere because the ObservableOwner is defunct. This change adds _opt-in_ testing to report: 1. How many owners have valid Opsgenie teams 2. How many owners have valid handbook pages In addition, we collect ObservableOwners that pass the test and use it to generate configuration for `site.json` in Sourcegraph.com: https://github.com/sourcegraph/deploy-sourcegraph-cloud/pull/18338 - this helps ensure the list is valid and not deceptively high-coverage. The results are not great, but **enforcing** that owners are valid isn't currently in scope: ``` 6/10 ObservableOwners do not have valid Opsgenie teams 3/10 ObservableOwners do not point to valid handbook pages ``` I also removed some defunct/unused functionality/owners. ## Test plan To run these tests: ``` export OPSGENIE_API_KEY="..." go test -timeout 30s github.com/sourcegraph/sourcegraph/monitoring/monitoring -update -online ```
2023-12-29 22:07:35 +00:00
<sub>*Managed by the [Sourcegraph Infrastructure Org team](https://handbook.sourcegraph.com/departments/engineering/infrastructure).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
Generated query for warning alert: `max((cadvisor_container_memory_usage_percentage_total{name=~"^prometheus.*"}) >= 99)`
</details>
monitoring: improve solutions formatting, add team information (#14878)
2020-10-20 05:11:20 +00:00
<br />
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: improve long-term provisioning alerts aggregation (#12778)
2020-08-08 06:44:33 +00:00
## prometheus: provisioning_container_cpu_usage_long_term
monitoring: migrate out-of-band alerts to the generator (#12391) Migrates all out-of-band alert rules defined by deploy-sourcegraph to the generator, such that each alert now has a panel as part of a service dashboard like all our other alerts, with a few changes and additions.
2020-07-28 02:17:56 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
<p class="subtitle">container cpu usage total (90th percentile over 1d) across all cores by instance</p>
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
monitoring: migrate out-of-band alerts to the generator (#12391) Migrates all out-of-band alert rules defined by deploy-sourcegraph to the generator, such that each alert now has a panel as part of a service dashboard like all our other alerts, with a few changes and additions.
2020-07-28 02:17:56 +00:00
monitoring: dashboards reference, refresh observability docs (#16939) - New monitoring definition field, 'Interpretation' - New generated dashboards reference - Grafana panels now link to alert solutions and dashboards reference - Refreshed observability docs Co-authored-by: uwedeportivo <534011+uwedeportivo@users.noreply.github.com>
2020-12-23 02:36:41 +00:00
- <span class="badge badge-warning">warning</span> prometheus: 80%+ container cpu usage total (90th percentile over 1d) across all cores by instance for 336h0m0s
monitoring: migrate out-of-band alerts to the generator (#12391) Migrates all out-of-band alert rules defined by deploy-sourcegraph to the generator, such that each alert now has a panel as part of a service dashboard like all our other alerts, with a few changes and additions.
2020-07-28 02:17:56 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: migrate out-of-band alerts to the generator (#12391) Migrates all out-of-band alert rules defined by deploy-sourcegraph to the generator, such that each alert now has a panel as part of a service dashboard like all our other alerts, with a few changes and additions.
2020-07-28 02:17:56 +00:00
Remove container_*_usage_7d_low alerts (#16164)
2020-11-25 18:19:33 +00:00
- **Kubernetes:** Consider increasing CPU limits in the `Deployment.yaml` for the prometheus service.
- **Docker Compose:** Consider increasing `cpus:` of the prometheus container in `docker-compose.yml`.
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#prometheus-provisioning-container-cpu-usage-long-term).
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
monitoring: migrate out-of-band alerts to the generator (#12391) Migrates all out-of-band alert rules defined by deploy-sourcegraph to the generator, such that each alert now has a panel as part of a service dashboard like all our other alerts, with a few changes and additions.
2020-07-28 02:17:56 +00:00
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
```json
"observability.silenceAlerts": [
monitoring: improve long-term provisioning alerts aggregation (#12778)
2020-08-08 06:44:33 +00:00
"warning_prometheus_provisioning_container_cpu_usage_long_term"
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
]
```
monitoring: test owners for valid Opsgenie teams and handbook pages (#59251) In INC-264 it seems that certain alerts - such as [zoekt: less than 90% percentage pods available for 10m0s](https://opsg.in/a/i/sourcegraph/178a626f-0f28-4295-bee9-84da988bb473-1703759057681) - don't seem to end up going anywhere because the ObservableOwner is defunct. This change adds _opt-in_ testing to report: 1. How many owners have valid Opsgenie teams 2. How many owners have valid handbook pages In addition, we collect ObservableOwners that pass the test and use it to generate configuration for `site.json` in Sourcegraph.com: https://github.com/sourcegraph/deploy-sourcegraph-cloud/pull/18338 - this helps ensure the list is valid and not deceptively high-coverage. The results are not great, but **enforcing** that owners are valid isn't currently in scope: ``` 6/10 ObservableOwners do not have valid Opsgenie teams 3/10 ObservableOwners do not point to valid handbook pages ``` I also removed some defunct/unused functionality/owners. ## Test plan To run these tests: ``` export OPSGENIE_API_KEY="..." go test -timeout 30s github.com/sourcegraph/sourcegraph/monitoring/monitoring -update -online ```
2023-12-29 22:07:35 +00:00
<sub>*Managed by the [Sourcegraph Infrastructure Org team](https://handbook.sourcegraph.com/departments/engineering/infrastructure).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
Generated query for warning alert: `max((quantile_over_time(0.9, cadvisor_container_cpu_usage_percentage_total{name=~"^prometheus.*"}[1d])) >= 80)`
</details>
monitoring: improve solutions formatting, add team information (#14878)
2020-10-20 05:11:20 +00:00
<br />
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: improve long-term provisioning alerts aggregation (#12778)
2020-08-08 06:44:33 +00:00
## prometheus: provisioning_container_memory_usage_long_term
monitoring: migrate out-of-band alerts to the generator (#12391) Migrates all out-of-band alert rules defined by deploy-sourcegraph to the generator, such that each alert now has a panel as part of a service dashboard like all our other alerts, with a few changes and additions.
2020-07-28 02:17:56 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
<p class="subtitle">container memory usage (1d maximum) by instance</p>
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
monitoring: migrate out-of-band alerts to the generator (#12391) Migrates all out-of-band alert rules defined by deploy-sourcegraph to the generator, such that each alert now has a panel as part of a service dashboard like all our other alerts, with a few changes and additions.
2020-07-28 02:17:56 +00:00
monitoring: dashboards reference, refresh observability docs (#16939) - New monitoring definition field, 'Interpretation' - New generated dashboards reference - Grafana panels now link to alert solutions and dashboards reference - Refreshed observability docs Co-authored-by: uwedeportivo <534011+uwedeportivo@users.noreply.github.com>
2020-12-23 02:36:41 +00:00
- <span class="badge badge-warning">warning</span> prometheus: 80%+ container memory usage (1d maximum) by instance for 336h0m0s
monitoring: migrate out-of-band alerts to the generator (#12391) Migrates all out-of-band alert rules defined by deploy-sourcegraph to the generator, such that each alert now has a panel as part of a service dashboard like all our other alerts, with a few changes and additions.
2020-07-28 02:17:56 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: migrate out-of-band alerts to the generator (#12391) Migrates all out-of-band alert rules defined by deploy-sourcegraph to the generator, such that each alert now has a panel as part of a service dashboard like all our other alerts, with a few changes and additions.
2020-07-28 02:17:56 +00:00
Remove container_*_usage_7d_low alerts (#16164)
2020-11-25 18:19:33 +00:00
- **Kubernetes:** Consider increasing memory limits in the `Deployment.yaml` for the prometheus service.
- **Docker Compose:** Consider increasing `memory:` of the prometheus container in `docker-compose.yml`.
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#prometheus-provisioning-container-memory-usage-long-term).
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
monitoring: improve long-term provisioning alerts aggregation (#12778)
2020-08-08 06:44:33 +00:00
"warning_prometheus_provisioning_container_memory_usage_long_term"
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
]
```
monitoring: migrate out-of-band alerts to the generator (#12391) Migrates all out-of-band alert rules defined by deploy-sourcegraph to the generator, such that each alert now has a panel as part of a service dashboard like all our other alerts, with a few changes and additions.
2020-07-28 02:17:56 +00:00
monitoring: test owners for valid Opsgenie teams and handbook pages (#59251) In INC-264 it seems that certain alerts - such as [zoekt: less than 90% percentage pods available for 10m0s](https://opsg.in/a/i/sourcegraph/178a626f-0f28-4295-bee9-84da988bb473-1703759057681) - don't seem to end up going anywhere because the ObservableOwner is defunct. This change adds _opt-in_ testing to report: 1. How many owners have valid Opsgenie teams 2. How many owners have valid handbook pages In addition, we collect ObservableOwners that pass the test and use it to generate configuration for `site.json` in Sourcegraph.com: https://github.com/sourcegraph/deploy-sourcegraph-cloud/pull/18338 - this helps ensure the list is valid and not deceptively high-coverage. The results are not great, but **enforcing** that owners are valid isn't currently in scope: ``` 6/10 ObservableOwners do not have valid Opsgenie teams 3/10 ObservableOwners do not point to valid handbook pages ``` I also removed some defunct/unused functionality/owners. ## Test plan To run these tests: ``` export OPSGENIE_API_KEY="..." go test -timeout 30s github.com/sourcegraph/sourcegraph/monitoring/monitoring -update -online ```
2023-12-29 22:07:35 +00:00
<sub>*Managed by the [Sourcegraph Infrastructure Org team](https://handbook.sourcegraph.com/departments/engineering/infrastructure).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
Generated query for warning alert: `max((max_over_time(cadvisor_container_memory_usage_percentage_total{name=~"^prometheus.*"}[1d])) >= 80)`
</details>
monitoring: improve solutions formatting, add team information (#14878)
2020-10-20 05:11:20 +00:00
<br />
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: improve long-term provisioning alerts aggregation (#12778)
2020-08-08 06:44:33 +00:00
## prometheus: provisioning_container_cpu_usage_short_term
monitoring: migrate out-of-band alerts to the generator (#12391) Migrates all out-of-band alert rules defined by deploy-sourcegraph to the generator, such that each alert now has a panel as part of a service dashboard like all our other alerts, with a few changes and additions.
2020-07-28 02:17:56 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
<p class="subtitle">container cpu usage total (5m maximum) across all cores by instance</p>
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
monitoring: migrate out-of-band alerts to the generator (#12391) Migrates all out-of-band alert rules defined by deploy-sourcegraph to the generator, such that each alert now has a panel as part of a service dashboard like all our other alerts, with a few changes and additions.
2020-07-28 02:17:56 +00:00
monitoring: dashboards reference, refresh observability docs (#16939) - New monitoring definition field, 'Interpretation' - New generated dashboards reference - Grafana panels now link to alert solutions and dashboards reference - Refreshed observability docs Co-authored-by: uwedeportivo <534011+uwedeportivo@users.noreply.github.com>
2020-12-23 02:36:41 +00:00
- <span class="badge badge-warning">warning</span> prometheus: 90%+ container cpu usage total (5m maximum) across all cores by instance for 30m0s
monitoring: migrate out-of-band alerts to the generator (#12391) Migrates all out-of-band alert rules defined by deploy-sourcegraph to the generator, such that each alert now has a panel as part of a service dashboard like all our other alerts, with a few changes and additions.
2020-07-28 02:17:56 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: migrate out-of-band alerts to the generator (#12391) Migrates all out-of-band alert rules defined by deploy-sourcegraph to the generator, such that each alert now has a panel as part of a service dashboard like all our other alerts, with a few changes and additions.
2020-07-28 02:17:56 +00:00
- **Kubernetes:** Consider increasing CPU limits in the the relevant `Deployment.yaml`.
- **Docker Compose:** Consider increasing `cpus:` of the prometheus container in `docker-compose.yml`.
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#prometheus-provisioning-container-cpu-usage-short-term).
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
monitoring: migrate out-of-band alerts to the generator (#12391) Migrates all out-of-band alert rules defined by deploy-sourcegraph to the generator, such that each alert now has a panel as part of a service dashboard like all our other alerts, with a few changes and additions.
2020-07-28 02:17:56 +00:00
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
```json
"observability.silenceAlerts": [
monitoring: improve long-term provisioning alerts aggregation (#12778)
2020-08-08 06:44:33 +00:00
"warning_prometheus_provisioning_container_cpu_usage_short_term"
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
]
```
monitoring: test owners for valid Opsgenie teams and handbook pages (#59251) In INC-264 it seems that certain alerts - such as [zoekt: less than 90% percentage pods available for 10m0s](https://opsg.in/a/i/sourcegraph/178a626f-0f28-4295-bee9-84da988bb473-1703759057681) - don't seem to end up going anywhere because the ObservableOwner is defunct. This change adds _opt-in_ testing to report: 1. How many owners have valid Opsgenie teams 2. How many owners have valid handbook pages In addition, we collect ObservableOwners that pass the test and use it to generate configuration for `site.json` in Sourcegraph.com: https://github.com/sourcegraph/deploy-sourcegraph-cloud/pull/18338 - this helps ensure the list is valid and not deceptively high-coverage. The results are not great, but **enforcing** that owners are valid isn't currently in scope: ``` 6/10 ObservableOwners do not have valid Opsgenie teams 3/10 ObservableOwners do not point to valid handbook pages ``` I also removed some defunct/unused functionality/owners. ## Test plan To run these tests: ``` export OPSGENIE_API_KEY="..." go test -timeout 30s github.com/sourcegraph/sourcegraph/monitoring/monitoring -update -online ```
2023-12-29 22:07:35 +00:00
<sub>*Managed by the [Sourcegraph Infrastructure Org team](https://handbook.sourcegraph.com/departments/engineering/infrastructure).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
Generated query for warning alert: `max((max_over_time(cadvisor_container_cpu_usage_percentage_total{name=~"^prometheus.*"}[5m])) >= 90)`
</details>
monitoring: improve solutions formatting, add team information (#14878)
2020-10-20 05:11:20 +00:00
<br />
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: improve long-term provisioning alerts aggregation (#12778)
2020-08-08 06:44:33 +00:00
## prometheus: provisioning_container_memory_usage_short_term
monitoring: migrate out-of-band alerts to the generator (#12391) Migrates all out-of-band alert rules defined by deploy-sourcegraph to the generator, such that each alert now has a panel as part of a service dashboard like all our other alerts, with a few changes and additions.
2020-07-28 02:17:56 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
<p class="subtitle">container memory usage (5m maximum) by instance</p>
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
monitoring: migrate out-of-band alerts to the generator (#12391) Migrates all out-of-band alert rules defined by deploy-sourcegraph to the generator, such that each alert now has a panel as part of a service dashboard like all our other alerts, with a few changes and additions.
2020-07-28 02:17:56 +00:00
monitoring: dashboards reference, refresh observability docs (#16939) - New monitoring definition field, 'Interpretation' - New generated dashboards reference - Grafana panels now link to alert solutions and dashboards reference - Refreshed observability docs Co-authored-by: uwedeportivo <534011+uwedeportivo@users.noreply.github.com>
2020-12-23 02:36:41 +00:00
- <span class="badge badge-warning">warning</span> prometheus: 90%+ container memory usage (5m maximum) by instance
monitoring: migrate out-of-band alerts to the generator (#12391) Migrates all out-of-band alert rules defined by deploy-sourcegraph to the generator, such that each alert now has a panel as part of a service dashboard like all our other alerts, with a few changes and additions.
2020-07-28 02:17:56 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: migrate out-of-band alerts to the generator (#12391) Migrates all out-of-band alert rules defined by deploy-sourcegraph to the generator, such that each alert now has a panel as part of a service dashboard like all our other alerts, with a few changes and additions.
2020-07-28 02:17:56 +00:00
- **Kubernetes:** Consider increasing memory limit in relevant `Deployment.yaml`.
- **Docker Compose:** Consider increasing `memory:` of prometheus container in `docker-compose.yml`.
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#prometheus-provisioning-container-memory-usage-short-term).
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
monitoring: improve long-term provisioning alerts aggregation (#12778)
2020-08-08 06:44:33 +00:00
"warning_prometheus_provisioning_container_memory_usage_short_term"
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
]
```
monitoring: test owners for valid Opsgenie teams and handbook pages (#59251) In INC-264 it seems that certain alerts - such as [zoekt: less than 90% percentage pods available for 10m0s](https://opsg.in/a/i/sourcegraph/178a626f-0f28-4295-bee9-84da988bb473-1703759057681) - don't seem to end up going anywhere because the ObservableOwner is defunct. This change adds _opt-in_ testing to report: 1. How many owners have valid Opsgenie teams 2. How many owners have valid handbook pages In addition, we collect ObservableOwners that pass the test and use it to generate configuration for `site.json` in Sourcegraph.com: https://github.com/sourcegraph/deploy-sourcegraph-cloud/pull/18338 - this helps ensure the list is valid and not deceptively high-coverage. The results are not great, but **enforcing** that owners are valid isn't currently in scope: ``` 6/10 ObservableOwners do not have valid Opsgenie teams 3/10 ObservableOwners do not point to valid handbook pages ``` I also removed some defunct/unused functionality/owners. ## Test plan To run these tests: ``` export OPSGENIE_API_KEY="..." go test -timeout 30s github.com/sourcegraph/sourcegraph/monitoring/monitoring -update -online ```
2023-12-29 22:07:35 +00:00
<sub>*Managed by the [Sourcegraph Infrastructure Org team](https://handbook.sourcegraph.com/departments/engineering/infrastructure).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
Generated query for warning alert: `max((max_over_time(cadvisor_container_memory_usage_percentage_total{name=~"^prometheus.*"}[5m])) >= 90)`
</details>
monitoring: improve solutions formatting, add team information (#14878)
2020-10-20 05:11:20 +00:00
<br />
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: improve containers metrics (#34808) part of - https://github.com/sourcegraph/sourcegraph/issues/33437 - https://github.com/sourcegraph/sourcegraph/issues/33438 This PR adds the following - A shared observable metric to indicate how many times a container (or child processes) has been oom kill - It's added to provisioning indicators since it's an indicator of underprovisioned resources - A new dashboard that shows resources usage of all containers at a glance - It contains two groups, one for all containers and one for containers that may have potential scaling issues - the first group is extremely noisy so it's hidden by default - the second group provides a more focused experience for site-admin to quickly find problematic resources <!-- > ~it looks like the alert table will be rendered regardless, which is pretty annoying (I guess the implementation of the monitoring stack is based on the assumption that one service/container will have its dashboard)~ __Questions & Notes__ - ~Should I make the new dashboard filterable by container name? like I only want to see the graph of `frontend`. Is that something that's valuable?~ - ~Any other metrics you all would like to see?~ - graph for total available resources on the VM compared to usage from https://github.com/sourcegraph/sourcegraph/issues/33437 is not really doable since we do not have `node-exporter` in anyone of our distro. We would have to tackle this from https://github.com/sourcegraph/customer/issues/806 --> ## Test plan connect to `demo`'s prom ```sh gcloud compute start-iap-tunnel default-$NEW_DEPLOYMENT-instance 9090 --local-host-port=localhost:4445 --zone us-central1-f --project $PROJECT_PREFIX-$CUSTOMER ``` or use dogfood ```sh kubectl port-forward svc/prometheus 4445:30090 ``` Update `dev/grafana/all/datasources.yaml` ```yml # Configuration for all non-Linux platforms. apiVersion: 1 datasources: - name: Prometheus type: prometheus access: proxy url: http://docker.for.mac.localhost:4445 # use the remote prometheus instance on demo instead of the local one isDefault: true editable: false - name: Jaeger type: jaeger access: proxy url: http://host.docker.internal:16686/-/debug/jaeger - name: Loki type: loki access: proxy url: http://host.docker.internal:3100 ``` Start monitoring stack ```sh sg run grafana ``` see screenshot > the first group is hidden by default ![CleanShot 2022-05-04 at 10 48 26](https://user-images.githubusercontent.com/8373004/166748470-22460552-6e51-4b29-a26b-4c5adbe1c404.png) ![CleanShot 2022-05-04 at 10 48 03](https://user-images.githubusercontent.com/8373004/166748396-1d410c35-e5fe-45e2-83bd-418d845413cb.png) ![CleanShot 2022-05-04 at 11 25 13](https://user-images.githubusercontent.com/8373004/166801266-21e51b68-65fd-4cb7-97de-83520e31e498.png)
2022-05-09 23:56:47 +00:00
## prometheus: container_oomkill_events_total
<p class="subtitle">container OOMKILL events total by instance</p>
**Descriptions**
- <span class="badge badge-warning">warning</span> prometheus: 1+ container OOMKILL events total by instance
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: improve containers metrics (#34808) part of - https://github.com/sourcegraph/sourcegraph/issues/33437 - https://github.com/sourcegraph/sourcegraph/issues/33438 This PR adds the following - A shared observable metric to indicate how many times a container (or child processes) has been oom kill - It's added to provisioning indicators since it's an indicator of underprovisioned resources - A new dashboard that shows resources usage of all containers at a glance - It contains two groups, one for all containers and one for containers that may have potential scaling issues - the first group is extremely noisy so it's hidden by default - the second group provides a more focused experience for site-admin to quickly find problematic resources <!-- > ~it looks like the alert table will be rendered regardless, which is pretty annoying (I guess the implementation of the monitoring stack is based on the assumption that one service/container will have its dashboard)~ __Questions & Notes__ - ~Should I make the new dashboard filterable by container name? like I only want to see the graph of `frontend`. Is that something that's valuable?~ - ~Any other metrics you all would like to see?~ - graph for total available resources on the VM compared to usage from https://github.com/sourcegraph/sourcegraph/issues/33437 is not really doable since we do not have `node-exporter` in anyone of our distro. We would have to tackle this from https://github.com/sourcegraph/customer/issues/806 --> ## Test plan connect to `demo`'s prom ```sh gcloud compute start-iap-tunnel default-$NEW_DEPLOYMENT-instance 9090 --local-host-port=localhost:4445 --zone us-central1-f --project $PROJECT_PREFIX-$CUSTOMER ``` or use dogfood ```sh kubectl port-forward svc/prometheus 4445:30090 ``` Update `dev/grafana/all/datasources.yaml` ```yml # Configuration for all non-Linux platforms. apiVersion: 1 datasources: - name: Prometheus type: prometheus access: proxy url: http://docker.for.mac.localhost:4445 # use the remote prometheus instance on demo instead of the local one isDefault: true editable: false - name: Jaeger type: jaeger access: proxy url: http://host.docker.internal:16686/-/debug/jaeger - name: Loki type: loki access: proxy url: http://host.docker.internal:3100 ``` Start monitoring stack ```sh sg run grafana ``` see screenshot > the first group is hidden by default ![CleanShot 2022-05-04 at 10 48 26](https://user-images.githubusercontent.com/8373004/166748470-22460552-6e51-4b29-a26b-4c5adbe1c404.png) ![CleanShot 2022-05-04 at 10 48 03](https://user-images.githubusercontent.com/8373004/166748396-1d410c35-e5fe-45e2-83bd-418d845413cb.png) ![CleanShot 2022-05-04 at 11 25 13](https://user-images.githubusercontent.com/8373004/166801266-21e51b68-65fd-4cb7-97de-83520e31e498.png)
2022-05-09 23:56:47 +00:00
- **Kubernetes:** Consider increasing memory limit in relevant `Deployment.yaml`.
- **Docker Compose:** Consider increasing `memory:` of prometheus container in `docker-compose.yml`.
- More help interpreting this metric is available in the [dashboards reference](./dashboards.md#prometheus-container-oomkill-events-total).
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"warning_prometheus_container_oomkill_events_total"
]
```
monitoring: test owners for valid Opsgenie teams and handbook pages (#59251) In INC-264 it seems that certain alerts - such as [zoekt: less than 90% percentage pods available for 10m0s](https://opsg.in/a/i/sourcegraph/178a626f-0f28-4295-bee9-84da988bb473-1703759057681) - don't seem to end up going anywhere because the ObservableOwner is defunct. This change adds _opt-in_ testing to report: 1. How many owners have valid Opsgenie teams 2. How many owners have valid handbook pages In addition, we collect ObservableOwners that pass the test and use it to generate configuration for `site.json` in Sourcegraph.com: https://github.com/sourcegraph/deploy-sourcegraph-cloud/pull/18338 - this helps ensure the list is valid and not deceptively high-coverage. The results are not great, but **enforcing** that owners are valid isn't currently in scope: ``` 6/10 ObservableOwners do not have valid Opsgenie teams 3/10 ObservableOwners do not point to valid handbook pages ``` I also removed some defunct/unused functionality/owners. ## Test plan To run these tests: ``` export OPSGENIE_API_KEY="..." go test -timeout 30s github.com/sourcegraph/sourcegraph/monitoring/monitoring -update -online ```
2023-12-29 22:07:35 +00:00
<sub>*Managed by the [Sourcegraph Infrastructure Org team](https://handbook.sourcegraph.com/departments/engineering/infrastructure).*</sub>
monitoring: improve containers metrics (#34808) part of - https://github.com/sourcegraph/sourcegraph/issues/33437 - https://github.com/sourcegraph/sourcegraph/issues/33438 This PR adds the following - A shared observable metric to indicate how many times a container (or child processes) has been oom kill - It's added to provisioning indicators since it's an indicator of underprovisioned resources - A new dashboard that shows resources usage of all containers at a glance - It contains two groups, one for all containers and one for containers that may have potential scaling issues - the first group is extremely noisy so it's hidden by default - the second group provides a more focused experience for site-admin to quickly find problematic resources <!-- > ~it looks like the alert table will be rendered regardless, which is pretty annoying (I guess the implementation of the monitoring stack is based on the assumption that one service/container will have its dashboard)~ __Questions & Notes__ - ~Should I make the new dashboard filterable by container name? like I only want to see the graph of `frontend`. Is that something that's valuable?~ - ~Any other metrics you all would like to see?~ - graph for total available resources on the VM compared to usage from https://github.com/sourcegraph/sourcegraph/issues/33437 is not really doable since we do not have `node-exporter` in anyone of our distro. We would have to tackle this from https://github.com/sourcegraph/customer/issues/806 --> ## Test plan connect to `demo`'s prom ```sh gcloud compute start-iap-tunnel default-$NEW_DEPLOYMENT-instance 9090 --local-host-port=localhost:4445 --zone us-central1-f --project $PROJECT_PREFIX-$CUSTOMER ``` or use dogfood ```sh kubectl port-forward svc/prometheus 4445:30090 ``` Update `dev/grafana/all/datasources.yaml` ```yml # Configuration for all non-Linux platforms. apiVersion: 1 datasources: - name: Prometheus type: prometheus access: proxy url: http://docker.for.mac.localhost:4445 # use the remote prometheus instance on demo instead of the local one isDefault: true editable: false - name: Jaeger type: jaeger access: proxy url: http://host.docker.internal:16686/-/debug/jaeger - name: Loki type: loki access: proxy url: http://host.docker.internal:3100 ``` Start monitoring stack ```sh sg run grafana ``` see screenshot > the first group is hidden by default ![CleanShot 2022-05-04 at 10 48 26](https://user-images.githubusercontent.com/8373004/166748470-22460552-6e51-4b29-a26b-4c5adbe1c404.png) ![CleanShot 2022-05-04 at 10 48 03](https://user-images.githubusercontent.com/8373004/166748396-1d410c35-e5fe-45e2-83bd-418d845413cb.png) ![CleanShot 2022-05-04 at 11 25 13](https://user-images.githubusercontent.com/8373004/166801266-21e51b68-65fd-4cb7-97de-83520e31e498.png)
2022-05-09 23:56:47 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
monitoring: extract into a submodule (#45786) This change extracts `monitoring` into a submodule for import in `sourcegraph/controller` (https://github.com/sourcegraph/controller/pull/195) so that we can generate dashboards for Cloud instances. These steps were required: 1. Initialize a `go.mod` in `monitoring` 2. Extract `dev/sg/internal/cliutil` into `lib` to avoid illegal imports from `monitoring` 3. Add local replaces to both `sourcegraph/sourcegraph` and `monitoring` 4. `go mod tidy` on all submodules 5. Update `go generate ./monitoring` commands to use `sg`, since the `go generate` command no longer works 6. Update `grafana/build.sh`, `prometheus/build.sh` to build the submodule 7. Amend linters to check for multiple `go.mod` files and ban imports of `github.com/sourcegraph/sourcegraph` 8. Update `sg generate go` to run in directories rather than from root The only caveat is that if you use VS code, you will now need to open `monitoring` in a separate workspace or similar, like with `lib`. Co-authored-by: Joe Chen <joe@sourcegraph.com>
2022-12-19 17:49:25 +00:00
Generated query for warning alert: `max((max by (name) (container_oom_events_total{name=~"^prometheus.*"})) >= 1)`
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
</details>
monitoring: improve containers metrics (#34808) part of - https://github.com/sourcegraph/sourcegraph/issues/33437 - https://github.com/sourcegraph/sourcegraph/issues/33438 This PR adds the following - A shared observable metric to indicate how many times a container (or child processes) has been oom kill - It's added to provisioning indicators since it's an indicator of underprovisioned resources - A new dashboard that shows resources usage of all containers at a glance - It contains two groups, one for all containers and one for containers that may have potential scaling issues - the first group is extremely noisy so it's hidden by default - the second group provides a more focused experience for site-admin to quickly find problematic resources <!-- > ~it looks like the alert table will be rendered regardless, which is pretty annoying (I guess the implementation of the monitoring stack is based on the assumption that one service/container will have its dashboard)~ __Questions & Notes__ - ~Should I make the new dashboard filterable by container name? like I only want to see the graph of `frontend`. Is that something that's valuable?~ - ~Any other metrics you all would like to see?~ - graph for total available resources on the VM compared to usage from https://github.com/sourcegraph/sourcegraph/issues/33437 is not really doable since we do not have `node-exporter` in anyone of our distro. We would have to tackle this from https://github.com/sourcegraph/customer/issues/806 --> ## Test plan connect to `demo`'s prom ```sh gcloud compute start-iap-tunnel default-$NEW_DEPLOYMENT-instance 9090 --local-host-port=localhost:4445 --zone us-central1-f --project $PROJECT_PREFIX-$CUSTOMER ``` or use dogfood ```sh kubectl port-forward svc/prometheus 4445:30090 ``` Update `dev/grafana/all/datasources.yaml` ```yml # Configuration for all non-Linux platforms. apiVersion: 1 datasources: - name: Prometheus type: prometheus access: proxy url: http://docker.for.mac.localhost:4445 # use the remote prometheus instance on demo instead of the local one isDefault: true editable: false - name: Jaeger type: jaeger access: proxy url: http://host.docker.internal:16686/-/debug/jaeger - name: Loki type: loki access: proxy url: http://host.docker.internal:3100 ``` Start monitoring stack ```sh sg run grafana ``` see screenshot > the first group is hidden by default ![CleanShot 2022-05-04 at 10 48 26](https://user-images.githubusercontent.com/8373004/166748470-22460552-6e51-4b29-a26b-4c5adbe1c404.png) ![CleanShot 2022-05-04 at 10 48 03](https://user-images.githubusercontent.com/8373004/166748396-1d410c35-e5fe-45e2-83bd-418d845413cb.png) ![CleanShot 2022-05-04 at 11 25 13](https://user-images.githubusercontent.com/8373004/166801266-21e51b68-65fd-4cb7-97de-83520e31e498.png)
2022-05-09 23:56:47 +00:00
<br />
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
## prometheus: pods_available_percentage
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
<p class="subtitle">percentage pods available</p>
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
monitoring: dashboards reference, refresh observability docs (#16939) - New monitoring definition field, 'Interpretation' - New generated dashboards reference - Grafana panels now link to alert solutions and dashboards reference - Refreshed observability docs Co-authored-by: uwedeportivo <534011+uwedeportivo@users.noreply.github.com>
2020-12-23 02:36:41 +00:00
- <span class="badge badge-critical">critical</span> prometheus: less than 90% percentage pods available for 10m0s
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
monitoring: forbid critical alerts with no solutions (#36423) Adds the restriction that critical alerts must have solutions defined as a hard requirement within the generator, and fixes all the outstanding issues: - Add PossibleSolutions for the pods_available_percentage and pg_up critical alerts (we have a similar solution doc in the container_missing alert) - Removes all the other critical alerts without possible solutions Co-authored-by: Michael Lin <mlzc@hey.com>
2022-06-02 15:51:43 +00:00
- Determine if the pod was OOM killed using `kubectl describe pod prometheus` (look for `OOMKilled: true`) and, if so, consider increasing the memory limit in the relevant `Deployment.yaml`.
- Check the logs before the container restarted to see if there are `panic:` messages or similar using `kubectl logs -p prometheus`.
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#prometheus-pods-available-percentage).
monitoring: encourage silencing, render entry for alerts w/o solutions (#12731)
2020-08-06 08:20:47 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"critical_prometheus_pods_available_percentage"
]
```
monitoring: migrate out-of-band alerts to the generator (#12391) Migrates all out-of-band alert rules defined by deploy-sourcegraph to the generator, such that each alert now has a panel as part of a service dashboard like all our other alerts, with a few changes and additions.
2020-07-28 02:17:56 +00:00
monitoring: test owners for valid Opsgenie teams and handbook pages (#59251) In INC-264 it seems that certain alerts - such as [zoekt: less than 90% percentage pods available for 10m0s](https://opsg.in/a/i/sourcegraph/178a626f-0f28-4295-bee9-84da988bb473-1703759057681) - don't seem to end up going anywhere because the ObservableOwner is defunct. This change adds _opt-in_ testing to report: 1. How many owners have valid Opsgenie teams 2. How many owners have valid handbook pages In addition, we collect ObservableOwners that pass the test and use it to generate configuration for `site.json` in Sourcegraph.com: https://github.com/sourcegraph/deploy-sourcegraph-cloud/pull/18338 - this helps ensure the list is valid and not deceptively high-coverage. The results are not great, but **enforcing** that owners are valid isn't currently in scope: ``` 6/10 ObservableOwners do not have valid Opsgenie teams 3/10 ObservableOwners do not point to valid handbook pages ``` I also removed some defunct/unused functionality/owners. ## Test plan To run these tests: ``` export OPSGENIE_API_KEY="..." go test -timeout 30s github.com/sourcegraph/sourcegraph/monitoring/monitoring -update -online ```
2023-12-29 22:07:35 +00:00
<sub>*Managed by the [Sourcegraph Infrastructure Org team](https://handbook.sourcegraph.com/departments/engineering/infrastructure).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
monitoring: extract into a submodule (#45786) This change extracts `monitoring` into a submodule for import in `sourcegraph/controller` (https://github.com/sourcegraph/controller/pull/195) so that we can generate dashboards for Cloud instances. These steps were required: 1. Initialize a `go.mod` in `monitoring` 2. Extract `dev/sg/internal/cliutil` into `lib` to avoid illegal imports from `monitoring` 3. Add local replaces to both `sourcegraph/sourcegraph` and `monitoring` 4. `go mod tidy` on all submodules 5. Update `go generate ./monitoring` commands to use `sg`, since the `go generate` command no longer works 6. Update `grafana/build.sh`, `prometheus/build.sh` to build the submodule 7. Amend linters to check for multiple `go.mod` files and ban imports of `github.com/sourcegraph/sourcegraph` 8. Update `sg generate go` to run in directories rather than from root The only caveat is that if you use VS code, you will now need to open `monitoring` in a separate workspace or similar, like with `lib`. Co-authored-by: Joe Chen <joe@sourcegraph.com>
2022-12-19 17:49:25 +00:00
Generated query for critical alert: `min((sum by (app) (up{app=~".*prometheus"}) / count by (app) (up{app=~".*prometheus"}) * 100) <= 90)`
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
</details>
monitoring: improve solutions formatting, add team information (#14878)
2020-10-20 05:11:20 +00:00
<br />
monitoring: generator docs, move definitions to subpackage (#16214) * 📖 Rendered monitoring package reference * ⭐ Monitoring generator highlights + intro to dev * ➡️ Migrate all definitions to a separate package for clarity and ease of documentation
2020-11-28 06:04:42 +00:00
executors: fix and move no-forward-progress alert (#40746)
2022-08-24 12:11:38 +00:00
## executor: executor_handlers
executors: alert when no jobs processed but queue > 0 (#40570) Co-authored-by: Erik Seliger <erikseliger@me.com>
2022-08-22 12:56:47 +00:00
executors: fix and move no-forward-progress alert (#40746)
2022-08-24 12:11:38 +00:00
<p class="subtitle">executor active handlers</p>
executors: alert when no jobs processed but queue > 0 (#40570) Co-authored-by: Erik Seliger <erikseliger@me.com>
2022-08-22 12:56:47 +00:00
**Descriptions**
executors: fix and move no-forward-progress alert (#40746)
2022-08-24 12:11:38 +00:00
- <span class="badge badge-critical">critical</span> executor: 0 active executor handlers and > 0 queue size for 5m0s
executors: alert when no jobs processed but queue > 0 (#40570) Co-authored-by: Erik Seliger <erikseliger@me.com>
2022-08-22 12:56:47 +00:00
**Next steps**
- Check to see the state of any compute VMs, they may be taking longer than expected to boot.
- Make sure the executors appear under Site Admin > Executors.
- Check the Grafana dashboard section for APIClient, it should do frequent requests to Dequeue and Heartbeat and those must not fail.
executors: fix and move no-forward-progress alert (#40746)
2022-08-24 12:11:38 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#executor-executor-handlers).
executors: alert when no jobs processed but queue > 0 (#40570) Co-authored-by: Erik Seliger <erikseliger@me.com>
2022-08-22 12:56:47 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
executors: fix and move no-forward-progress alert (#40746)
2022-08-24 12:11:38 +00:00
"critical_executor_executor_handlers"
executors: alert when no jobs processed but queue > 0 (#40570) Co-authored-by: Erik Seliger <erikseliger@me.com>
2022-08-22 12:56:47 +00:00
]
```
<sub>*Managed by the [Sourcegraph Code intelligence team](https://handbook.sourcegraph.com/departments/engineering/teams/code-intelligence).*</sub>
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
monitoring: extract into a submodule (#45786) This change extracts `monitoring` into a submodule for import in `sourcegraph/controller` (https://github.com/sourcegraph/controller/pull/195) so that we can generate dashboards for Cloud instances. These steps were required: 1. Initialize a `go.mod` in `monitoring` 2. Extract `dev/sg/internal/cliutil` into `lib` to avoid illegal imports from `monitoring` 3. Add local replaces to both `sourcegraph/sourcegraph` and `monitoring` 4. `go mod tidy` on all submodules 5. Update `go generate ./monitoring` commands to use `sg`, since the `go generate` command no longer works 6. Update `grafana/build.sh`, `prometheus/build.sh` to build the submodule 7. Amend linters to check for multiple `go.mod` files and ban imports of `github.com/sourcegraph/sourcegraph` 8. Update `sg generate go` to run in directories rather than from root The only caveat is that if you use VS code, you will now need to open `monitoring` in a separate workspace or similar, like with `lib`. Co-authored-by: Joe Chen <joe@sourcegraph.com>
2022-12-19 17:49:25 +00:00
Custom query for critical alert: `min(((sum(src_executor_processor_handlers{sg_job=~"^sourcegraph-executors.*"}) or vector(0)) == 0 and (sum by (queue) (src_executor_total{job=~"^sourcegraph-executors.*"})) > 0) <= 0)`
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
</details>
executors: alert when no jobs processed but queue > 0 (#40570) Co-authored-by: Erik Seliger <erikseliger@me.com>
2022-08-22 12:56:47 +00:00
<br />
codeintel: alert when all executor jobs are failing (#38767) Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2022-07-19 15:10:40 +00:00
## executor: executor_processor_error_rate
executors: fix and move no-forward-progress alert (#40746)
2022-08-24 12:11:38 +00:00
<p class="subtitle">executor operation error rate over 5m</p>
codeintel: alert when all executor jobs are failing (#38767) Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2022-07-19 15:10:40 +00:00
**Descriptions**
monitoring: disable executors rate critial error (#47425)
2023-02-06 20:24:26 +00:00
- <span class="badge badge-warning">warning</span> executor: 100%+ executor operation error rate over 5m for 1h0m0s
codeintel: alert when all executor jobs are failing (#38767) Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2022-07-19 15:10:40 +00:00
**Next steps**
- Determine the cause of failure from the auto-indexing job logs in the site-admin page.
- This alert fires if all executor jobs have been failing for the past hour. The alert will continue for up
to 5 hours until the error rate is no longer 100%, even if there are no running jobs in that time, as the
problem is not know to be resolved until jobs start succeeding again.
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#executor-executor-processor-error-rate).
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
monitoring: disable executors rate critial error (#47425)
2023-02-06 20:24:26 +00:00
"warning_executor_executor_processor_error_rate"
codeintel: alert when all executor jobs are failing (#38767) Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2022-07-19 15:10:40 +00:00
]
```
<sub>*Managed by the [Sourcegraph Code intelligence team](https://handbook.sourcegraph.com/departments/engineering/teams/code-intelligence).*</sub>
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
monitoring: disable executors rate critial error (#47425)
2023-02-06 20:24:26 +00:00
Custom query for warning alert: `max((last_over_time(sum(increase(src_executor_processor_errors_total{sg_job=~"^sourcegraph-executors.*"}[5m]))[5h:]) / (last_over_time(sum(increase(src_executor_processor_total{sg_job=~"^sourcegraph-executors.*"}[5m]))[5h:]) + last_over_time(sum(increase(src_executor_processor_errors_total{sg_job=~"^sourcegraph-executors.*"}[5m]))[5h:])) * 100) >= 100)`
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
</details>
codeintel: alert when all executor jobs are failing (#38767) Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2022-07-19 15:10:40 +00:00
<br />
monitoring: Update dashboards (#23091)
2021-07-25 16:25:33 +00:00
## executor: go_goroutines
codeintel: Update metrics dashboards (#16806)
2020-12-18 17:57:28 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
<p class="subtitle">maximum active goroutines</p>
codeintel: Update metrics dashboards (#16806)
2020-12-18 17:57:28 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
codeintel: Update metrics dashboards (#16806)
2020-12-18 17:57:28 +00:00
monitoring: Update dashboards (#23091)
2021-07-25 16:25:33 +00:00
- <span class="badge badge-warning">warning</span> executor: 10000+ maximum active goroutines for 10m0s
codeintel: Update metrics dashboards (#16806)
2020-12-18 17:57:28 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
codeintel: Update metrics dashboards (#16806)
2020-12-18 17:57:28 +00:00
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- More help interpreting this metric is available in the [dashboards reference](./dashboards.md#executor-go-goroutines).
codeintel: Update metrics dashboards (#16806)
2020-12-18 17:57:28 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
monitoring: Update dashboards (#23091)
2021-07-25 16:25:33 +00:00
"warning_executor_go_goroutines"
codeintel: Update metrics dashboards (#16806)
2020-12-18 17:57:28 +00:00
]
```
monitoring: rename Cloud SaaS to IAM (#38268)
2022-07-07 02:13:27 +00:00
<sub>*Managed by the [Sourcegraph Code intelligence team](https://handbook.sourcegraph.com/departments/engineering/teams/code-intelligence).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
monitoring: extract into a submodule (#45786) This change extracts `monitoring` into a submodule for import in `sourcegraph/controller` (https://github.com/sourcegraph/controller/pull/195) so that we can generate dashboards for Cloud instances. These steps were required: 1. Initialize a `go.mod` in `monitoring` 2. Extract `dev/sg/internal/cliutil` into `lib` to avoid illegal imports from `monitoring` 3. Add local replaces to both `sourcegraph/sourcegraph` and `monitoring` 4. `go mod tidy` on all submodules 5. Update `go generate ./monitoring` commands to use `sg`, since the `go generate` command no longer works 6. Update `grafana/build.sh`, `prometheus/build.sh` to build the submodule 7. Amend linters to check for multiple `go.mod` files and ban imports of `github.com/sourcegraph/sourcegraph` 8. Update `sg generate go` to run in directories rather than from root The only caveat is that if you use VS code, you will now need to open `monitoring` in a separate workspace or similar, like with `lib`. Co-authored-by: Joe Chen <joe@sourcegraph.com>
2022-12-19 17:49:25 +00:00
Generated query for warning alert: `max((max by (sg_instance) (go_goroutines{sg_job=~".*sourcegraph-executors"})) >= 10000)`
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
</details>
codeintel: Update metrics dashboards (#16806)
2020-12-18 17:57:28 +00:00
<br />
monitoring: Update dashboards (#23091)
2021-07-25 16:25:33 +00:00
## executor: go_gc_duration_seconds
codeintel: Update metrics dashboards (#16806)
2020-12-18 17:57:28 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
<p class="subtitle">maximum go garbage collection duration</p>
codeintel: Update metrics dashboards (#16806)
2020-12-18 17:57:28 +00:00
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
**Descriptions**
codeintel: Update metrics dashboards (#16806)
2020-12-18 17:57:28 +00:00
monitoring: Update dashboards (#23091)
2021-07-25 16:25:33 +00:00
- <span class="badge badge-warning">warning</span> executor: 2s+ maximum go garbage collection duration
codeintel: Update metrics dashboards (#16806)
2020-12-18 17:57:28 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
codeintel: Update metrics dashboards (#16806)
2020-12-18 17:57:28 +00:00
monitoring: dashboards docs improvements (#24563) - Render Description more prominently because that's how panels are identified in dashboards - Render link to panel and dashboards - Render full query in collapsible section for reference - Show number of alerts and indicate that no alerts are defined more prominently - Alerts now always link to the panel docs - Fix code-insights owner slug that lead to handbook 404 - Fix some `MetricDescriptionRoot` using underscore instead of space
2021-09-02 20:05:36 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#executor-go-gc-duration-seconds).
codeintel: Update metrics dashboards (#16806)
2020-12-18 17:57:28 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
monitoring: Update dashboards (#23091)
2021-07-25 16:25:33 +00:00
"warning_executor_go_gc_duration_seconds"
codeintel: Update metrics dashboards (#16806)
2020-12-18 17:57:28 +00:00
]
```
monitoring: rename Cloud SaaS to IAM (#38268)
2022-07-07 02:13:27 +00:00
<sub>*Managed by the [Sourcegraph Code intelligence team](https://handbook.sourcegraph.com/departments/engineering/teams/code-intelligence).*</sub>
monitoring: explicitly point out team that owns alerts/panels (#17471)
2021-01-20 15:24:10 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
monitoring: extract into a submodule (#45786) This change extracts `monitoring` into a submodule for import in `sourcegraph/controller` (https://github.com/sourcegraph/controller/pull/195) so that we can generate dashboards for Cloud instances. These steps were required: 1. Initialize a `go.mod` in `monitoring` 2. Extract `dev/sg/internal/cliutil` into `lib` to avoid illegal imports from `monitoring` 3. Add local replaces to both `sourcegraph/sourcegraph` and `monitoring` 4. `go mod tidy` on all submodules 5. Update `go generate ./monitoring` commands to use `sg`, since the `go generate` command no longer works 6. Update `grafana/build.sh`, `prometheus/build.sh` to build the submodule 7. Amend linters to check for multiple `go.mod` files and ban imports of `github.com/sourcegraph/sourcegraph` 8. Update `sg generate go` to run in directories rather than from root The only caveat is that if you use VS code, you will now need to open `monitoring` in a separate workspace or similar, like with `lib`. Co-authored-by: Joe Chen <joe@sourcegraph.com>
2022-12-19 17:49:25 +00:00
Generated query for warning alert: `max((max by (sg_instance) (go_gc_duration_seconds{sg_job=~".*sourcegraph-executors"})) >= 2)`
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
</details>
codeintel: Update metrics dashboards (#16806)
2020-12-18 17:57:28 +00:00
<br />
RFC 619: Move upload commitgraph background job into the new job skeleton (#36112)
2022-05-27 14:27:11 +00:00
## codeintel-uploads: codeintel_commit_graph_queued_max_age
<p class="subtitle">repository queue longest time in queue</p>
**Descriptions**
monitoring: downgrade commit_graph staleness critical alert to warning (#49059) This alert has paged the Cloud team on several occasions, and doesn't seem immediately actionable - see https://github.com/sourcegraph/customer/issues/1988 and related discussions. tl;dr it seems that this most often happens when a few repos are too large or take too long to clone or work on, which isn't actionable enough to warrant a critical alert (all the recent occurrences indicate none of the relevant services are starved for resources) - this PR proposes that we downgrade it to a warning. ## Test plan CI
2023-03-10 01:43:15 +00:00
- <span class="badge badge-warning">warning</span> codeintel-uploads: 3600s+ repository queue longest time in queue
RFC 619: Move upload commitgraph background job into the new job skeleton (#36112)
2022-05-27 14:27:11 +00:00
monitoring: rebrand PossibleSolutions as NextSteps (#36500) Rebrands PossibleSolutions as NextSteps to lower the bar for this field (the instructions don't need to provide an immediate solution, just guide the next steps once an alert is received) To accommodate this, also makes the following changes: - Refined guidance for how to write NextSteps - alert_solutions.md -> alerts.md (+ redirect) - prom-wrapper templates now indicate next steps instead of solutions
2022-06-02 17:51:06 +00:00
**Next steps**
RFC 619: Move upload commitgraph background job into the new job skeleton (#36112)
2022-05-27 14:27:11 +00:00
- An alert here is generally indicative of either underprovisioned worker instance(s) and/or
an underprovisioned main postgres instance.
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#codeintel-uploads-codeintel-commit-graph-queued-max-age).
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
monitoring: downgrade commit_graph staleness critical alert to warning (#49059) This alert has paged the Cloud team on several occasions, and doesn't seem immediately actionable - see https://github.com/sourcegraph/customer/issues/1988 and related discussions. tl;dr it seems that this most often happens when a few repos are too large or take too long to clone or work on, which isn't actionable enough to warrant a critical alert (all the recent occurrences indicate none of the relevant services are starved for resources) - this PR proposes that we downgrade it to a warning. ## Test plan CI
2023-03-10 01:43:15 +00:00
"warning_codeintel-uploads_codeintel_commit_graph_queued_max_age"
RFC 619: Move upload commitgraph background job into the new job skeleton (#36112)
2022-05-27 14:27:11 +00:00
]
```
monitoring: rename Cloud SaaS to IAM (#38268)
2022-07-07 02:13:27 +00:00
<sub>*Managed by the [Sourcegraph Code intelligence team](https://handbook.sourcegraph.com/departments/engineering/teams/code-intelligence).*</sub>
RFC 619: Move upload commitgraph background job into the new job skeleton (#36112)
2022-05-27 14:27:11 +00:00
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
monitoring: downgrade commit_graph staleness critical alert to warning (#49059) This alert has paged the Cloud team on several occasions, and doesn't seem immediately actionable - see https://github.com/sourcegraph/customer/issues/1988 and related discussions. tl;dr it seems that this most often happens when a few repos are too large or take too long to clone or work on, which isn't actionable enough to warrant a critical alert (all the recent occurrences indicate none of the relevant services are starved for resources) - this PR proposes that we downgrade it to a warning. ## Test plan CI
2023-03-10 01:43:15 +00:00
Generated query for warning alert: `max((max(src_codeintel_commit_graph_queued_duration_seconds_total)) >= 3600)`
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
</details>
RFC 619: Move upload commitgraph background job into the new job skeleton (#36112)
2022-05-27 14:27:11 +00:00
<br />
monitoring/telemetry: un-hide panels, improve docstrings (#57740)
2023-10-19 19:52:21 +00:00
## telemetry: telemetry_gateway_exporter_queue_growth
usage-data: monitoring and alerts (#40286) * monitoring and alerts for usage data pipeline * fix duration
2022-08-12 15:16:37 +00:00
monitoring/telemetry: un-hide panels, improve docstrings (#57740)
2023-10-19 19:52:21 +00:00
<p class="subtitle">rate of growth of export queue over 30m</p>
usage-data: monitoring and alerts (#40286) * monitoring and alerts for usage data pipeline * fix duration
2022-08-12 15:16:37 +00:00
**Descriptions**
monitoring/telemetry: un-hide panels, improve docstrings (#57740)
2023-10-19 19:52:21 +00:00
- <span class="badge badge-warning">warning</span> telemetry: 1+ rate of growth of export queue over 30m for 1h0m0s
- <span class="badge badge-critical">critical</span> telemetry: 1+ rate of growth of export queue over 30m for 36h0m0s
usage-data: monitoring and alerts (#40286) * monitoring and alerts for usage data pipeline * fix duration
2022-08-12 15:16:37 +00:00
**Next steps**
monitoring/telemetry: un-hide panels, improve docstrings (#57740)
2023-10-19 19:52:21 +00:00
- Check the "number of events exported per batch over 30m" dashboard panel to see if export throughput is at saturation.
- Increase `TELEMETRY_GATEWAY_EXPORTER_EXPORT_BATCH_SIZE` to export more events per batch.
- Reduce `TELEMETRY_GATEWAY_EXPORTER_EXPORT_INTERVAL` to schedule more export jobs.
- See worker logs in the `worker.telemetrygateway-exporter` log scope for more details to see if any export errors are occuring - if logs only indicate that exports failed, reach out to Sourcegraph with relevant log entries, as this may be an issue in Sourcegraph`s Telemetry Gateway service.
- More help interpreting this metric is available in the [dashboards reference](./dashboards.md#telemetry-telemetry-gateway-exporter-queue-growth).
usage-data: monitoring and alerts (#40286) * monitoring and alerts for usage data pipeline * fix duration
2022-08-12 15:16:37 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
monitoring/telemetry: un-hide panels, improve docstrings (#57740)
2023-10-19 19:52:21 +00:00
"warning_telemetry_telemetry_gateway_exporter_queue_growth",
"critical_telemetry_telemetry_gateway_exporter_queue_growth"
usage-data: monitoring and alerts (#40286) * monitoring and alerts for usage data pipeline * fix duration
2022-08-12 15:16:37 +00:00
]
```
<sub>*Managed by the [Sourcegraph Data & Analytics team](https://handbook.sourcegraph.com/departments/engineering/teams/data-analytics).*</sub>
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
monitoring/telemetry: un-hide panels, improve docstrings (#57740)
2023-10-19 19:52:21 +00:00
Generated query for warning alert: `max((max(deriv(src_telemetrygatewayexporter_queue_size[30m]))) > 1)`
Generated query for critical alert: `max((max(deriv(src_telemetrygatewayexporter_queue_size[30m]))) > 1)`
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
</details>
usage-data: monitoring and alerts (#40286) * monitoring and alerts for usage data pipeline * fix duration
2022-08-12 15:16:37 +00:00
<br />
monitoring/telemetry: un-hide panels, improve docstrings (#57740)
2023-10-19 19:52:21 +00:00
## telemetry: telemetrygatewayexporter_exporter_errors_total
usage-data: monitoring and alerts (#40286) * monitoring and alerts for usage data pipeline * fix duration
2022-08-12 15:16:37 +00:00
monitoring/telemetry: un-hide panels, improve docstrings (#57740)
2023-10-19 19:52:21 +00:00
<p class="subtitle">events exporter operation errors every 30m</p>
usage-data: monitoring and alerts (#40286) * monitoring and alerts for usage data pipeline * fix duration
2022-08-12 15:16:37 +00:00
**Descriptions**
monitoring/telemetry: un-hide panels, improve docstrings (#57740)
2023-10-19 19:52:21 +00:00
- <span class="badge badge-warning">warning</span> telemetry: 0+ events exporter operation errors every 30m
usage-data: monitoring and alerts (#40286) * monitoring and alerts for usage data pipeline * fix duration
2022-08-12 15:16:37 +00:00
**Next steps**
monitoring/telemetry: un-hide panels, improve docstrings (#57740)
2023-10-19 19:52:21 +00:00
- Failures indicate that exporting of telemetry events from Sourcegraph are failing. This may affect the performance of the database as the backlog grows.
- See worker logs in the `worker.telemetrygateway-exporter` log scope for more details. If logs only indicate that exports failed, reach out to Sourcegraph with relevant log entries, as this may be an issue in Sourcegraph`s Telemetry Gateway service.
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#telemetry-telemetrygatewayexporter-exporter-errors-total).
usage-data: monitoring and alerts (#40286) * monitoring and alerts for usage data pipeline * fix duration
2022-08-12 15:16:37 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
monitoring/telemetry: un-hide panels, improve docstrings (#57740)
2023-10-19 19:52:21 +00:00
"warning_telemetry_telemetrygatewayexporter_exporter_errors_total"
usage-data: monitoring and alerts (#40286) * monitoring and alerts for usage data pipeline * fix duration
2022-08-12 15:16:37 +00:00
]
```
<sub>*Managed by the [Sourcegraph Data & Analytics team](https://handbook.sourcegraph.com/departments/engineering/teams/data-analytics).*</sub>
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
<details>
<summary>Technical details</summary>
monitoring/telemetry: un-hide panels, improve docstrings (#57740)
2023-10-19 19:52:21 +00:00
Generated query for warning alert: `max((sum(increase(src_telemetrygatewayexporter_exporter_errors_total{job=~"^worker.*"}[30m]))) > 0)`
monitoring: always render full generated alert query in docs (#41956) This effectively allows us to track rendered alerts in Git as well, and check how they have changed when the generator is changed, and could be a useful reference in general to have on hand.
2022-09-23 21:48:55 +00:00
</details>
usage-data: monitoring and alerts (#40286) * monitoring and alerts for usage data pipeline * fix duration
2022-08-12 15:16:37 +00:00
<br />
monitoring/telemetry: un-hide panels, improve docstrings (#57740)
2023-10-19 19:52:21 +00:00
## telemetry: telemetrygatewayexporter_queue_cleanup_errors_total
monitoring: add queue growth panel and alert (#57222)
2023-10-02 14:59:41 +00:00
monitoring/telemetry: un-hide panels, improve docstrings (#57740)
2023-10-19 19:52:21 +00:00
<p class="subtitle">export queue cleanup operation errors every 30m</p>
monitoring: add queue growth panel and alert (#57222)
2023-10-02 14:59:41 +00:00
**Descriptions**
monitoring/telemetry: un-hide panels, improve docstrings (#57740)
2023-10-19 19:52:21 +00:00
- <span class="badge badge-warning">warning</span> telemetry: 0+ export queue cleanup operation errors every 30m
monitoring: add queue growth panel and alert (#57222)
2023-10-02 14:59:41 +00:00
**Next steps**
monitoring/telemetry: un-hide panels, improve docstrings (#57740)
2023-10-19 19:52:21 +00:00
- Failures indicate that pruning of already-exported telemetry events from the database is failing. This may affect the performance of the database as the export queue table grows.
- See worker logs in the `worker.telemetrygateway-exporter` log scope for more details.
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#telemetry-telemetrygatewayexporter-queue-cleanup-errors-total).
monitoring: add queue growth panel and alert (#57222)
2023-10-02 14:59:41 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
monitoring/telemetry: un-hide panels, improve docstrings (#57740)
2023-10-19 19:52:21 +00:00
"warning_telemetry_telemetrygatewayexporter_queue_cleanup_errors_total"
monitoring: add queue growth panel and alert (#57222)
2023-10-02 14:59:41 +00:00
]
```
<sub>*Managed by the [Sourcegraph Data & Analytics team](https://handbook.sourcegraph.com/departments/engineering/teams/data-analytics).*</sub>
<details>
<summary>Technical details</summary>
monitoring/telemetry: un-hide panels, improve docstrings (#57740)
2023-10-19 19:52:21 +00:00
Generated query for warning alert: `max((sum(increase(src_telemetrygatewayexporter_queue_cleanup_errors_total{job=~"^worker.*"}[30m]))) > 0)`
monitoring: add queue growth panel and alert (#57222)
2023-10-02 14:59:41 +00:00
</details>
<br />
monitoring/telemetry: un-hide panels, improve docstrings (#57740)
2023-10-19 19:52:21 +00:00
## telemetry: telemetrygatewayexporter_queue_metrics_reporter_errors_total
monitoring: add telemetrygatewayexporter panels, improve metrics (#57171) Part of https://github.com/sourcegraph/sourcegraph/issues/56970 - this adds some dashboards for the export side of things, as well as improves the existing metrics. Only includes warnings. ## Test plan Had to test locally only because I ended up changing the metrics a bit, but validated that the queue size metric works in S2. Testing locally: ```yaml # sg.config.overwrite.yaml env: TELEMETRY_GATEWAY_EXPORTER_EXPORT_INTERVAL: "30s" TELEMETRY_GATEWAY_EXPORTER_EXPORTED_EVENTS_RETENTION: "5m" TELEMETRY_GATEWAY_EXPORTER_QUEUE_CLEANUP_INTERVAL: "10m" ``` ``` sg start sg start monitoring ``` Do lots of searches to generate events. Note `telemetry-export` feature flag must be enabled Data is not realistic because of the super high interval I configured for testing, but it shows that things work: ![image](https://github.com/sourcegraph/sourcegraph/assets/23356519/c44cd60e-514e-4b62-a6b6-890582d8059c)
2023-09-29 17:10:07 +00:00
monitoring/telemetry: un-hide panels, improve docstrings (#57740)
2023-10-19 19:52:21 +00:00
<p class="subtitle">export backlog metrics reporting operation errors every 30m</p>
monitoring: add telemetrygatewayexporter panels, improve metrics (#57171) Part of https://github.com/sourcegraph/sourcegraph/issues/56970 - this adds some dashboards for the export side of things, as well as improves the existing metrics. Only includes warnings. ## Test plan Had to test locally only because I ended up changing the metrics a bit, but validated that the queue size metric works in S2. Testing locally: ```yaml # sg.config.overwrite.yaml env: TELEMETRY_GATEWAY_EXPORTER_EXPORT_INTERVAL: "30s" TELEMETRY_GATEWAY_EXPORTER_EXPORTED_EVENTS_RETENTION: "5m" TELEMETRY_GATEWAY_EXPORTER_QUEUE_CLEANUP_INTERVAL: "10m" ``` ``` sg start sg start monitoring ``` Do lots of searches to generate events. Note `telemetry-export` feature flag must be enabled Data is not realistic because of the super high interval I configured for testing, but it shows that things work: ![image](https://github.com/sourcegraph/sourcegraph/assets/23356519/c44cd60e-514e-4b62-a6b6-890582d8059c)
2023-09-29 17:10:07 +00:00
**Descriptions**
monitoring/telemetry: un-hide panels, improve docstrings (#57740)
2023-10-19 19:52:21 +00:00
- <span class="badge badge-warning">warning</span> telemetry: 0+ export backlog metrics reporting operation errors every 30m
monitoring: add telemetrygatewayexporter panels, improve metrics (#57171) Part of https://github.com/sourcegraph/sourcegraph/issues/56970 - this adds some dashboards for the export side of things, as well as improves the existing metrics. Only includes warnings. ## Test plan Had to test locally only because I ended up changing the metrics a bit, but validated that the queue size metric works in S2. Testing locally: ```yaml # sg.config.overwrite.yaml env: TELEMETRY_GATEWAY_EXPORTER_EXPORT_INTERVAL: "30s" TELEMETRY_GATEWAY_EXPORTER_EXPORTED_EVENTS_RETENTION: "5m" TELEMETRY_GATEWAY_EXPORTER_QUEUE_CLEANUP_INTERVAL: "10m" ``` ``` sg start sg start monitoring ``` Do lots of searches to generate events. Note `telemetry-export` feature flag must be enabled Data is not realistic because of the super high interval I configured for testing, but it shows that things work: ![image](https://github.com/sourcegraph/sourcegraph/assets/23356519/c44cd60e-514e-4b62-a6b6-890582d8059c)
2023-09-29 17:10:07 +00:00
**Next steps**
monitoring/telemetry: un-hide panels, improve docstrings (#57740)
2023-10-19 19:52:21 +00:00
- Failures indicate that reporting of telemetry events metrics is failing. This may affect the reliability of telemetry events export metrics.
monitoring: add telemetrygatewayexporter panels, improve metrics (#57171) Part of https://github.com/sourcegraph/sourcegraph/issues/56970 - this adds some dashboards for the export side of things, as well as improves the existing metrics. Only includes warnings. ## Test plan Had to test locally only because I ended up changing the metrics a bit, but validated that the queue size metric works in S2. Testing locally: ```yaml # sg.config.overwrite.yaml env: TELEMETRY_GATEWAY_EXPORTER_EXPORT_INTERVAL: "30s" TELEMETRY_GATEWAY_EXPORTER_EXPORTED_EVENTS_RETENTION: "5m" TELEMETRY_GATEWAY_EXPORTER_QUEUE_CLEANUP_INTERVAL: "10m" ``` ``` sg start sg start monitoring ``` Do lots of searches to generate events. Note `telemetry-export` feature flag must be enabled Data is not realistic because of the super high interval I configured for testing, but it shows that things work: ![image](https://github.com/sourcegraph/sourcegraph/assets/23356519/c44cd60e-514e-4b62-a6b6-890582d8059c)
2023-09-29 17:10:07 +00:00
- See worker logs in the `worker.telemetrygateway-exporter` log scope for more details.
monitoring/telemetry: un-hide panels, improve docstrings (#57740)
2023-10-19 19:52:21 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#telemetry-telemetrygatewayexporter-queue-metrics-reporter-errors-total).
monitoring: add telemetrygatewayexporter panels, improve metrics (#57171) Part of https://github.com/sourcegraph/sourcegraph/issues/56970 - this adds some dashboards for the export side of things, as well as improves the existing metrics. Only includes warnings. ## Test plan Had to test locally only because I ended up changing the metrics a bit, but validated that the queue size metric works in S2. Testing locally: ```yaml # sg.config.overwrite.yaml env: TELEMETRY_GATEWAY_EXPORTER_EXPORT_INTERVAL: "30s" TELEMETRY_GATEWAY_EXPORTER_EXPORTED_EVENTS_RETENTION: "5m" TELEMETRY_GATEWAY_EXPORTER_QUEUE_CLEANUP_INTERVAL: "10m" ``` ``` sg start sg start monitoring ``` Do lots of searches to generate events. Note `telemetry-export` feature flag must be enabled Data is not realistic because of the super high interval I configured for testing, but it shows that things work: ![image](https://github.com/sourcegraph/sourcegraph/assets/23356519/c44cd60e-514e-4b62-a6b6-890582d8059c)
2023-09-29 17:10:07 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
monitoring/telemetry: un-hide panels, improve docstrings (#57740)
2023-10-19 19:52:21 +00:00
"warning_telemetry_telemetrygatewayexporter_queue_metrics_reporter_errors_total"
monitoring: add telemetrygatewayexporter panels, improve metrics (#57171) Part of https://github.com/sourcegraph/sourcegraph/issues/56970 - this adds some dashboards for the export side of things, as well as improves the existing metrics. Only includes warnings. ## Test plan Had to test locally only because I ended up changing the metrics a bit, but validated that the queue size metric works in S2. Testing locally: ```yaml # sg.config.overwrite.yaml env: TELEMETRY_GATEWAY_EXPORTER_EXPORT_INTERVAL: "30s" TELEMETRY_GATEWAY_EXPORTER_EXPORTED_EVENTS_RETENTION: "5m" TELEMETRY_GATEWAY_EXPORTER_QUEUE_CLEANUP_INTERVAL: "10m" ``` ``` sg start sg start monitoring ``` Do lots of searches to generate events. Note `telemetry-export` feature flag must be enabled Data is not realistic because of the super high interval I configured for testing, but it shows that things work: ![image](https://github.com/sourcegraph/sourcegraph/assets/23356519/c44cd60e-514e-4b62-a6b6-890582d8059c)
2023-09-29 17:10:07 +00:00
]
```
<sub>*Managed by the [Sourcegraph Data & Analytics team](https://handbook.sourcegraph.com/departments/engineering/teams/data-analytics).*</sub>
<details>
<summary>Technical details</summary>
monitoring/telemetry: un-hide panels, improve docstrings (#57740)
2023-10-19 19:52:21 +00:00
Generated query for warning alert: `max((sum(increase(src_telemetrygatewayexporter_queue_metrics_reporter_errors_total{job=~"^worker.*"}[30m]))) > 0)`
monitoring: add telemetrygatewayexporter panels, improve metrics (#57171) Part of https://github.com/sourcegraph/sourcegraph/issues/56970 - this adds some dashboards for the export side of things, as well as improves the existing metrics. Only includes warnings. ## Test plan Had to test locally only because I ended up changing the metrics a bit, but validated that the queue size metric works in S2. Testing locally: ```yaml # sg.config.overwrite.yaml env: TELEMETRY_GATEWAY_EXPORTER_EXPORT_INTERVAL: "30s" TELEMETRY_GATEWAY_EXPORTER_EXPORTED_EVENTS_RETENTION: "5m" TELEMETRY_GATEWAY_EXPORTER_QUEUE_CLEANUP_INTERVAL: "10m" ``` ``` sg start sg start monitoring ``` Do lots of searches to generate events. Note `telemetry-export` feature flag must be enabled Data is not realistic because of the super high interval I configured for testing, but it shows that things work: ![image](https://github.com/sourcegraph/sourcegraph/assets/23356519/c44cd60e-514e-4b62-a6b6-890582d8059c)
2023-09-29 17:10:07 +00:00
</details>
<br />
monitoring/telemetry: un-hide panels, improve docstrings (#57740)
2023-10-19 19:52:21 +00:00
## telemetry: telemetry_job_error_rate
monitoring: add telemetrygatewayexporter panels, improve metrics (#57171) Part of https://github.com/sourcegraph/sourcegraph/issues/56970 - this adds some dashboards for the export side of things, as well as improves the existing metrics. Only includes warnings. ## Test plan Had to test locally only because I ended up changing the metrics a bit, but validated that the queue size metric works in S2. Testing locally: ```yaml # sg.config.overwrite.yaml env: TELEMETRY_GATEWAY_EXPORTER_EXPORT_INTERVAL: "30s" TELEMETRY_GATEWAY_EXPORTER_EXPORTED_EVENTS_RETENTION: "5m" TELEMETRY_GATEWAY_EXPORTER_QUEUE_CLEANUP_INTERVAL: "10m" ``` ``` sg start sg start monitoring ``` Do lots of searches to generate events. Note `telemetry-export` feature flag must be enabled Data is not realistic because of the super high interval I configured for testing, but it shows that things work: ![image](https://github.com/sourcegraph/sourcegraph/assets/23356519/c44cd60e-514e-4b62-a6b6-890582d8059c)
2023-09-29 17:10:07 +00:00
monitoring/telemetry: un-hide panels, improve docstrings (#57740)
2023-10-19 19:52:21 +00:00
<p class="subtitle">usage data exporter operation error rate over 5m</p>
monitoring: add telemetrygatewayexporter panels, improve metrics (#57171) Part of https://github.com/sourcegraph/sourcegraph/issues/56970 - this adds some dashboards for the export side of things, as well as improves the existing metrics. Only includes warnings. ## Test plan Had to test locally only because I ended up changing the metrics a bit, but validated that the queue size metric works in S2. Testing locally: ```yaml # sg.config.overwrite.yaml env: TELEMETRY_GATEWAY_EXPORTER_EXPORT_INTERVAL: "30s" TELEMETRY_GATEWAY_EXPORTER_EXPORTED_EVENTS_RETENTION: "5m" TELEMETRY_GATEWAY_EXPORTER_QUEUE_CLEANUP_INTERVAL: "10m" ``` ``` sg start sg start monitoring ``` Do lots of searches to generate events. Note `telemetry-export` feature flag must be enabled Data is not realistic because of the super high interval I configured for testing, but it shows that things work: ![image](https://github.com/sourcegraph/sourcegraph/assets/23356519/c44cd60e-514e-4b62-a6b6-890582d8059c)
2023-09-29 17:10:07 +00:00
**Descriptions**
monitoring/telemetry: un-hide panels, improve docstrings (#57740)
2023-10-19 19:52:21 +00:00
- <span class="badge badge-warning">warning</span> telemetry: 0%+ usage data exporter operation error rate over 5m for 30m0s
monitoring: add telemetrygatewayexporter panels, improve metrics (#57171) Part of https://github.com/sourcegraph/sourcegraph/issues/56970 - this adds some dashboards for the export side of things, as well as improves the existing metrics. Only includes warnings. ## Test plan Had to test locally only because I ended up changing the metrics a bit, but validated that the queue size metric works in S2. Testing locally: ```yaml # sg.config.overwrite.yaml env: TELEMETRY_GATEWAY_EXPORTER_EXPORT_INTERVAL: "30s" TELEMETRY_GATEWAY_EXPORTER_EXPORTED_EVENTS_RETENTION: "5m" TELEMETRY_GATEWAY_EXPORTER_QUEUE_CLEANUP_INTERVAL: "10m" ``` ``` sg start sg start monitoring ``` Do lots of searches to generate events. Note `telemetry-export` feature flag must be enabled Data is not realistic because of the super high interval I configured for testing, but it shows that things work: ![image](https://github.com/sourcegraph/sourcegraph/assets/23356519/c44cd60e-514e-4b62-a6b6-890582d8059c)
2023-09-29 17:10:07 +00:00
**Next steps**
monitoring/telemetry: un-hide panels, improve docstrings (#57740)
2023-10-19 19:52:21 +00:00
- Involved cloud team to inspect logs of the managed instance to determine error sources.
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#telemetry-telemetry-job-error-rate).
monitoring: add telemetrygatewayexporter panels, improve metrics (#57171) Part of https://github.com/sourcegraph/sourcegraph/issues/56970 - this adds some dashboards for the export side of things, as well as improves the existing metrics. Only includes warnings. ## Test plan Had to test locally only because I ended up changing the metrics a bit, but validated that the queue size metric works in S2. Testing locally: ```yaml # sg.config.overwrite.yaml env: TELEMETRY_GATEWAY_EXPORTER_EXPORT_INTERVAL: "30s" TELEMETRY_GATEWAY_EXPORTER_EXPORTED_EVENTS_RETENTION: "5m" TELEMETRY_GATEWAY_EXPORTER_QUEUE_CLEANUP_INTERVAL: "10m" ``` ``` sg start sg start monitoring ``` Do lots of searches to generate events. Note `telemetry-export` feature flag must be enabled Data is not realistic because of the super high interval I configured for testing, but it shows that things work: ![image](https://github.com/sourcegraph/sourcegraph/assets/23356519/c44cd60e-514e-4b62-a6b6-890582d8059c)
2023-09-29 17:10:07 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
monitoring/telemetry: un-hide panels, improve docstrings (#57740)
2023-10-19 19:52:21 +00:00
"warning_telemetry_telemetry_job_error_rate"
monitoring: add telemetrygatewayexporter panels, improve metrics (#57171) Part of https://github.com/sourcegraph/sourcegraph/issues/56970 - this adds some dashboards for the export side of things, as well as improves the existing metrics. Only includes warnings. ## Test plan Had to test locally only because I ended up changing the metrics a bit, but validated that the queue size metric works in S2. Testing locally: ```yaml # sg.config.overwrite.yaml env: TELEMETRY_GATEWAY_EXPORTER_EXPORT_INTERVAL: "30s" TELEMETRY_GATEWAY_EXPORTER_EXPORTED_EVENTS_RETENTION: "5m" TELEMETRY_GATEWAY_EXPORTER_QUEUE_CLEANUP_INTERVAL: "10m" ``` ``` sg start sg start monitoring ``` Do lots of searches to generate events. Note `telemetry-export` feature flag must be enabled Data is not realistic because of the super high interval I configured for testing, but it shows that things work: ![image](https://github.com/sourcegraph/sourcegraph/assets/23356519/c44cd60e-514e-4b62-a6b6-890582d8059c)
2023-09-29 17:10:07 +00:00
]
```
<sub>*Managed by the [Sourcegraph Data & Analytics team](https://handbook.sourcegraph.com/departments/engineering/teams/data-analytics).*</sub>
<details>
<summary>Technical details</summary>
monitoring/telemetry: un-hide panels, improve docstrings (#57740)
2023-10-19 19:52:21 +00:00
Generated query for warning alert: `max((sum by (op) (increase(src_telemetry_job_errors_total{job=~"^worker.*"}[5m])) / (sum by (op) (increase(src_telemetry_job_total{job=~"^worker.*"}[5m])) + sum by (op) (increase(src_telemetry_job_errors_total{job=~"^worker.*"}[5m]))) * 100) > 0)`
monitoring: add telemetrygatewayexporter panels, improve metrics (#57171) Part of https://github.com/sourcegraph/sourcegraph/issues/56970 - this adds some dashboards for the export side of things, as well as improves the existing metrics. Only includes warnings. ## Test plan Had to test locally only because I ended up changing the metrics a bit, but validated that the queue size metric works in S2. Testing locally: ```yaml # sg.config.overwrite.yaml env: TELEMETRY_GATEWAY_EXPORTER_EXPORT_INTERVAL: "30s" TELEMETRY_GATEWAY_EXPORTER_EXPORTED_EVENTS_RETENTION: "5m" TELEMETRY_GATEWAY_EXPORTER_QUEUE_CLEANUP_INTERVAL: "10m" ``` ``` sg start sg start monitoring ``` Do lots of searches to generate events. Note `telemetry-export` feature flag must be enabled Data is not realistic because of the super high interval I configured for testing, but it shows that things work: ![image](https://github.com/sourcegraph/sourcegraph/assets/23356519/c44cd60e-514e-4b62-a6b6-890582d8059c)
2023-09-29 17:10:07 +00:00
</details>
<br />
monitoring/telemetry: un-hide panels, improve docstrings (#57740)
2023-10-19 19:52:21 +00:00
## telemetry: telemetry_job_utilized_throughput
monitoring: add telemetrygatewayexporter panels, improve metrics (#57171) Part of https://github.com/sourcegraph/sourcegraph/issues/56970 - this adds some dashboards for the export side of things, as well as improves the existing metrics. Only includes warnings. ## Test plan Had to test locally only because I ended up changing the metrics a bit, but validated that the queue size metric works in S2. Testing locally: ```yaml # sg.config.overwrite.yaml env: TELEMETRY_GATEWAY_EXPORTER_EXPORT_INTERVAL: "30s" TELEMETRY_GATEWAY_EXPORTER_EXPORTED_EVENTS_RETENTION: "5m" TELEMETRY_GATEWAY_EXPORTER_QUEUE_CLEANUP_INTERVAL: "10m" ``` ``` sg start sg start monitoring ``` Do lots of searches to generate events. Note `telemetry-export` feature flag must be enabled Data is not realistic because of the super high interval I configured for testing, but it shows that things work: ![image](https://github.com/sourcegraph/sourcegraph/assets/23356519/c44cd60e-514e-4b62-a6b6-890582d8059c)
2023-09-29 17:10:07 +00:00
monitoring/telemetry: un-hide panels, improve docstrings (#57740)
2023-10-19 19:52:21 +00:00
<p class="subtitle">utilized percentage of maximum throughput</p>
monitoring: add telemetrygatewayexporter panels, improve metrics (#57171) Part of https://github.com/sourcegraph/sourcegraph/issues/56970 - this adds some dashboards for the export side of things, as well as improves the existing metrics. Only includes warnings. ## Test plan Had to test locally only because I ended up changing the metrics a bit, but validated that the queue size metric works in S2. Testing locally: ```yaml # sg.config.overwrite.yaml env: TELEMETRY_GATEWAY_EXPORTER_EXPORT_INTERVAL: "30s" TELEMETRY_GATEWAY_EXPORTER_EXPORTED_EVENTS_RETENTION: "5m" TELEMETRY_GATEWAY_EXPORTER_QUEUE_CLEANUP_INTERVAL: "10m" ``` ``` sg start sg start monitoring ``` Do lots of searches to generate events. Note `telemetry-export` feature flag must be enabled Data is not realistic because of the super high interval I configured for testing, but it shows that things work: ![image](https://github.com/sourcegraph/sourcegraph/assets/23356519/c44cd60e-514e-4b62-a6b6-890582d8059c)
2023-09-29 17:10:07 +00:00
**Descriptions**
monitoring/telemetry: un-hide panels, improve docstrings (#57740)
2023-10-19 19:52:21 +00:00
- <span class="badge badge-warning">warning</span> telemetry: 90%+ utilized percentage of maximum throughput for 30m0s
monitoring: add telemetrygatewayexporter panels, improve metrics (#57171) Part of https://github.com/sourcegraph/sourcegraph/issues/56970 - this adds some dashboards for the export side of things, as well as improves the existing metrics. Only includes warnings. ## Test plan Had to test locally only because I ended up changing the metrics a bit, but validated that the queue size metric works in S2. Testing locally: ```yaml # sg.config.overwrite.yaml env: TELEMETRY_GATEWAY_EXPORTER_EXPORT_INTERVAL: "30s" TELEMETRY_GATEWAY_EXPORTER_EXPORTED_EVENTS_RETENTION: "5m" TELEMETRY_GATEWAY_EXPORTER_QUEUE_CLEANUP_INTERVAL: "10m" ``` ``` sg start sg start monitoring ``` Do lots of searches to generate events. Note `telemetry-export` feature flag must be enabled Data is not realistic because of the super high interval I configured for testing, but it shows that things work: ![image](https://github.com/sourcegraph/sourcegraph/assets/23356519/c44cd60e-514e-4b62-a6b6-890582d8059c)
2023-09-29 17:10:07 +00:00
**Next steps**
monitoring/telemetry: un-hide panels, improve docstrings (#57740)
2023-10-19 19:52:21 +00:00
- Throughput utilization is high. This could be a signal that this instance is producing too many events for the export job to keep up. Configure more throughput using the maxBatchSize option.
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#telemetry-telemetry-job-utilized-throughput).
monitoring: add telemetrygatewayexporter panels, improve metrics (#57171) Part of https://github.com/sourcegraph/sourcegraph/issues/56970 - this adds some dashboards for the export side of things, as well as improves the existing metrics. Only includes warnings. ## Test plan Had to test locally only because I ended up changing the metrics a bit, but validated that the queue size metric works in S2. Testing locally: ```yaml # sg.config.overwrite.yaml env: TELEMETRY_GATEWAY_EXPORTER_EXPORT_INTERVAL: "30s" TELEMETRY_GATEWAY_EXPORTER_EXPORTED_EVENTS_RETENTION: "5m" TELEMETRY_GATEWAY_EXPORTER_QUEUE_CLEANUP_INTERVAL: "10m" ``` ``` sg start sg start monitoring ``` Do lots of searches to generate events. Note `telemetry-export` feature flag must be enabled Data is not realistic because of the super high interval I configured for testing, but it shows that things work: ![image](https://github.com/sourcegraph/sourcegraph/assets/23356519/c44cd60e-514e-4b62-a6b6-890582d8059c)
2023-09-29 17:10:07 +00:00
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
monitoring/telemetry: un-hide panels, improve docstrings (#57740)
2023-10-19 19:52:21 +00:00
"warning_telemetry_telemetry_job_utilized_throughput"
monitoring: add telemetrygatewayexporter panels, improve metrics (#57171) Part of https://github.com/sourcegraph/sourcegraph/issues/56970 - this adds some dashboards for the export side of things, as well as improves the existing metrics. Only includes warnings. ## Test plan Had to test locally only because I ended up changing the metrics a bit, but validated that the queue size metric works in S2. Testing locally: ```yaml # sg.config.overwrite.yaml env: TELEMETRY_GATEWAY_EXPORTER_EXPORT_INTERVAL: "30s" TELEMETRY_GATEWAY_EXPORTER_EXPORTED_EVENTS_RETENTION: "5m" TELEMETRY_GATEWAY_EXPORTER_QUEUE_CLEANUP_INTERVAL: "10m" ``` ``` sg start sg start monitoring ``` Do lots of searches to generate events. Note `telemetry-export` feature flag must be enabled Data is not realistic because of the super high interval I configured for testing, but it shows that things work: ![image](https://github.com/sourcegraph/sourcegraph/assets/23356519/c44cd60e-514e-4b62-a6b6-890582d8059c)
2023-09-29 17:10:07 +00:00
]
```
<sub>*Managed by the [Sourcegraph Data & Analytics team](https://handbook.sourcegraph.com/departments/engineering/teams/data-analytics).*</sub>
<details>
<summary>Technical details</summary>
monitoring/telemetry: un-hide panels, improve docstrings (#57740)
2023-10-19 19:52:21 +00:00
Generated query for warning alert: `max((rate(src_telemetry_job_total{op="SendEvents"}[1h]) / on () group_right () src_telemetry_job_max_throughput * 100) > 90)`
monitoring: add telemetrygatewayexporter panels, improve metrics (#57171) Part of https://github.com/sourcegraph/sourcegraph/issues/56970 - this adds some dashboards for the export side of things, as well as improves the existing metrics. Only includes warnings. ## Test plan Had to test locally only because I ended up changing the metrics a bit, but validated that the queue size metric works in S2. Testing locally: ```yaml # sg.config.overwrite.yaml env: TELEMETRY_GATEWAY_EXPORTER_EXPORT_INTERVAL: "30s" TELEMETRY_GATEWAY_EXPORTER_EXPORTED_EVENTS_RETENTION: "5m" TELEMETRY_GATEWAY_EXPORTER_QUEUE_CLEANUP_INTERVAL: "10m" ``` ``` sg start sg start monitoring ``` Do lots of searches to generate events. Note `telemetry-export` feature flag must be enabled Data is not realistic because of the super high interval I configured for testing, but it shows that things work: ![image](https://github.com/sourcegraph/sourcegraph/assets/23356519/c44cd60e-514e-4b62-a6b6-890582d8059c)
2023-09-29 17:10:07 +00:00
</details>
<br />
otel: add collector dashboard (#45009) * add initial dashboard for otel * add failed sent dashboard * extra panels * use sum and rate for resource queries * review comments * add warning alerts * Update monitoring/definitions/otel_collector.go * review comments * run go generate * Update monitoring/definitions/otel_collector.go Co-authored-by: Robert Lin <robert@bobheadxi.dev> * Update monitoring/definitions/otel_collector.go Co-authored-by: Robert Lin <robert@bobheadxi.dev> * Update monitoring/definitions/otel_collector.go Co-authored-by: Robert Lin <robert@bobheadxi.dev> * Update monitoring/definitions/otel_collector.go Co-authored-by: Robert Lin <robert@bobheadxi.dev> * Update monitoring/definitions/otel_collector.go Co-authored-by: Robert Lin <robert@bobheadxi.dev> * Update monitoring/definitions/otel_collector.go Co-authored-by: Robert Lin <robert@bobheadxi.dev> * review comments * review feedback also drop two panels * remove brackets in metrics * update docs * fix goimport * gogenerate Co-authored-by: Robert Lin <robert@bobheadxi.dev> Co-authored-by: Jean-Hadrien Chabran <jh@chabran.fr>
2022-12-19 12:18:51 +00:00
## otel-collector: otel_span_refused
<p class="subtitle">spans refused per receiver</p>
**Descriptions**
- <span class="badge badge-warning">warning</span> otel-collector: 1+ spans refused per receiver for 5m0s
**Next steps**
- Check logs of the collector and configuration of the receiver
- More help interpreting this metric is available in the [dashboards reference](./dashboards.md#otel-collector-otel-span-refused).
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"warning_otel-collector_otel_span_refused"
]
```
monitoring: test owners for valid Opsgenie teams and handbook pages (#59251) In INC-264 it seems that certain alerts - such as [zoekt: less than 90% percentage pods available for 10m0s](https://opsg.in/a/i/sourcegraph/178a626f-0f28-4295-bee9-84da988bb473-1703759057681) - don't seem to end up going anywhere because the ObservableOwner is defunct. This change adds _opt-in_ testing to report: 1. How many owners have valid Opsgenie teams 2. How many owners have valid handbook pages In addition, we collect ObservableOwners that pass the test and use it to generate configuration for `site.json` in Sourcegraph.com: https://github.com/sourcegraph/deploy-sourcegraph-cloud/pull/18338 - this helps ensure the list is valid and not deceptively high-coverage. The results are not great, but **enforcing** that owners are valid isn't currently in scope: ``` 6/10 ObservableOwners do not have valid Opsgenie teams 3/10 ObservableOwners do not point to valid handbook pages ``` I also removed some defunct/unused functionality/owners. ## Test plan To run these tests: ``` export OPSGENIE_API_KEY="..." go test -timeout 30s github.com/sourcegraph/sourcegraph/monitoring/monitoring -update -online ```
2023-12-29 22:07:35 +00:00
<sub>*Managed by the [Sourcegraph Infrastructure Org team](https://handbook.sourcegraph.com/departments/engineering/infrastructure).*</sub>
otel: add collector dashboard (#45009) * add initial dashboard for otel * add failed sent dashboard * extra panels * use sum and rate for resource queries * review comments * add warning alerts * Update monitoring/definitions/otel_collector.go * review comments * run go generate * Update monitoring/definitions/otel_collector.go Co-authored-by: Robert Lin <robert@bobheadxi.dev> * Update monitoring/definitions/otel_collector.go Co-authored-by: Robert Lin <robert@bobheadxi.dev> * Update monitoring/definitions/otel_collector.go Co-authored-by: Robert Lin <robert@bobheadxi.dev> * Update monitoring/definitions/otel_collector.go Co-authored-by: Robert Lin <robert@bobheadxi.dev> * Update monitoring/definitions/otel_collector.go Co-authored-by: Robert Lin <robert@bobheadxi.dev> * Update monitoring/definitions/otel_collector.go Co-authored-by: Robert Lin <robert@bobheadxi.dev> * review comments * review feedback also drop two panels * remove brackets in metrics * update docs * fix goimport * gogenerate Co-authored-by: Robert Lin <robert@bobheadxi.dev> Co-authored-by: Jean-Hadrien Chabran <jh@chabran.fr>
2022-12-19 12:18:51 +00:00
<details>
<summary>Technical details</summary>
monitoring: extract into a submodule (#45786) This change extracts `monitoring` into a submodule for import in `sourcegraph/controller` (https://github.com/sourcegraph/controller/pull/195) so that we can generate dashboards for Cloud instances. These steps were required: 1. Initialize a `go.mod` in `monitoring` 2. Extract `dev/sg/internal/cliutil` into `lib` to avoid illegal imports from `monitoring` 3. Add local replaces to both `sourcegraph/sourcegraph` and `monitoring` 4. `go mod tidy` on all submodules 5. Update `go generate ./monitoring` commands to use `sg`, since the `go generate` command no longer works 6. Update `grafana/build.sh`, `prometheus/build.sh` to build the submodule 7. Amend linters to check for multiple `go.mod` files and ban imports of `github.com/sourcegraph/sourcegraph` 8. Update `sg generate go` to run in directories rather than from root The only caveat is that if you use VS code, you will now need to open `monitoring` in a separate workspace or similar, like with `lib`. Co-authored-by: Joe Chen <joe@sourcegraph.com>
2022-12-19 17:49:25 +00:00
Generated query for warning alert: `max((sum by (receiver) (rate(otelcol_receiver_refused_spans[1m]))) > 1)`
otel: add collector dashboard (#45009) * add initial dashboard for otel * add failed sent dashboard * extra panels * use sum and rate for resource queries * review comments * add warning alerts * Update monitoring/definitions/otel_collector.go * review comments * run go generate * Update monitoring/definitions/otel_collector.go Co-authored-by: Robert Lin <robert@bobheadxi.dev> * Update monitoring/definitions/otel_collector.go Co-authored-by: Robert Lin <robert@bobheadxi.dev> * Update monitoring/definitions/otel_collector.go Co-authored-by: Robert Lin <robert@bobheadxi.dev> * Update monitoring/definitions/otel_collector.go Co-authored-by: Robert Lin <robert@bobheadxi.dev> * Update monitoring/definitions/otel_collector.go Co-authored-by: Robert Lin <robert@bobheadxi.dev> * Update monitoring/definitions/otel_collector.go Co-authored-by: Robert Lin <robert@bobheadxi.dev> * review comments * review feedback also drop two panels * remove brackets in metrics * update docs * fix goimport * gogenerate Co-authored-by: Robert Lin <robert@bobheadxi.dev> Co-authored-by: Jean-Hadrien Chabran <jh@chabran.fr>
2022-12-19 12:18:51 +00:00
</details>
<br />
## otel-collector: otel_span_export_failures
<p class="subtitle">span export failures by exporter</p>
**Descriptions**
- <span class="badge badge-warning">warning</span> otel-collector: 1+ span export failures by exporter for 5m0s
**Next steps**
- Check the configuration of the exporter and if the service being exported is up
- More help interpreting this metric is available in the [dashboards reference](./dashboards.md#otel-collector-otel-span-export-failures).
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"warning_otel-collector_otel_span_export_failures"
]
```
monitoring: test owners for valid Opsgenie teams and handbook pages (#59251) In INC-264 it seems that certain alerts - such as [zoekt: less than 90% percentage pods available for 10m0s](https://opsg.in/a/i/sourcegraph/178a626f-0f28-4295-bee9-84da988bb473-1703759057681) - don't seem to end up going anywhere because the ObservableOwner is defunct. This change adds _opt-in_ testing to report: 1. How many owners have valid Opsgenie teams 2. How many owners have valid handbook pages In addition, we collect ObservableOwners that pass the test and use it to generate configuration for `site.json` in Sourcegraph.com: https://github.com/sourcegraph/deploy-sourcegraph-cloud/pull/18338 - this helps ensure the list is valid and not deceptively high-coverage. The results are not great, but **enforcing** that owners are valid isn't currently in scope: ``` 6/10 ObservableOwners do not have valid Opsgenie teams 3/10 ObservableOwners do not point to valid handbook pages ``` I also removed some defunct/unused functionality/owners. ## Test plan To run these tests: ``` export OPSGENIE_API_KEY="..." go test -timeout 30s github.com/sourcegraph/sourcegraph/monitoring/monitoring -update -online ```
2023-12-29 22:07:35 +00:00
<sub>*Managed by the [Sourcegraph Infrastructure Org team](https://handbook.sourcegraph.com/departments/engineering/infrastructure).*</sub>
otel: add collector dashboard (#45009) * add initial dashboard for otel * add failed sent dashboard * extra panels * use sum and rate for resource queries * review comments * add warning alerts * Update monitoring/definitions/otel_collector.go * review comments * run go generate * Update monitoring/definitions/otel_collector.go Co-authored-by: Robert Lin <robert@bobheadxi.dev> * Update monitoring/definitions/otel_collector.go Co-authored-by: Robert Lin <robert@bobheadxi.dev> * Update monitoring/definitions/otel_collector.go Co-authored-by: Robert Lin <robert@bobheadxi.dev> * Update monitoring/definitions/otel_collector.go Co-authored-by: Robert Lin <robert@bobheadxi.dev> * Update monitoring/definitions/otel_collector.go Co-authored-by: Robert Lin <robert@bobheadxi.dev> * Update monitoring/definitions/otel_collector.go Co-authored-by: Robert Lin <robert@bobheadxi.dev> * review comments * review feedback also drop two panels * remove brackets in metrics * update docs * fix goimport * gogenerate Co-authored-by: Robert Lin <robert@bobheadxi.dev> Co-authored-by: Jean-Hadrien Chabran <jh@chabran.fr>
2022-12-19 12:18:51 +00:00
<details>
<summary>Technical details</summary>
monitoring: extract into a submodule (#45786) This change extracts `monitoring` into a submodule for import in `sourcegraph/controller` (https://github.com/sourcegraph/controller/pull/195) so that we can generate dashboards for Cloud instances. These steps were required: 1. Initialize a `go.mod` in `monitoring` 2. Extract `dev/sg/internal/cliutil` into `lib` to avoid illegal imports from `monitoring` 3. Add local replaces to both `sourcegraph/sourcegraph` and `monitoring` 4. `go mod tidy` on all submodules 5. Update `go generate ./monitoring` commands to use `sg`, since the `go generate` command no longer works 6. Update `grafana/build.sh`, `prometheus/build.sh` to build the submodule 7. Amend linters to check for multiple `go.mod` files and ban imports of `github.com/sourcegraph/sourcegraph` 8. Update `sg generate go` to run in directories rather than from root The only caveat is that if you use VS code, you will now need to open `monitoring` in a separate workspace or similar, like with `lib`. Co-authored-by: Joe Chen <joe@sourcegraph.com>
2022-12-19 17:49:25 +00:00
Generated query for warning alert: `max((sum by (exporter) (rate(otelcol_exporter_send_failed_spans[1m]))) > 1)`
otel: add collector dashboard (#45009) * add initial dashboard for otel * add failed sent dashboard * extra panels * use sum and rate for resource queries * review comments * add warning alerts * Update monitoring/definitions/otel_collector.go * review comments * run go generate * Update monitoring/definitions/otel_collector.go Co-authored-by: Robert Lin <robert@bobheadxi.dev> * Update monitoring/definitions/otel_collector.go Co-authored-by: Robert Lin <robert@bobheadxi.dev> * Update monitoring/definitions/otel_collector.go Co-authored-by: Robert Lin <robert@bobheadxi.dev> * Update monitoring/definitions/otel_collector.go Co-authored-by: Robert Lin <robert@bobheadxi.dev> * Update monitoring/definitions/otel_collector.go Co-authored-by: Robert Lin <robert@bobheadxi.dev> * Update monitoring/definitions/otel_collector.go Co-authored-by: Robert Lin <robert@bobheadxi.dev> * review comments * review feedback also drop two panels * remove brackets in metrics * update docs * fix goimport * gogenerate Co-authored-by: Robert Lin <robert@bobheadxi.dev> Co-authored-by: Jean-Hadrien Chabran <jh@chabran.fr>
2022-12-19 12:18:51 +00:00
</details>
<br />
monitoring: update otel dashboard metrics (#55899) * add metrics to OTEL * monitoring: update otel dashboard metrics * create a seperate group for Queue Length * Update monitoring/definitions/otel_collector.go Co-authored-by: William Bezuidenhout <william.bezuidenhout@sourcegraph.com> * update auto gen doc * update auto gen alerts doc --------- Co-authored-by: William Bezuidenhout <william.bezuidenhout@sourcegraph.com>
2023-08-17 01:03:12 +00:00
## otel-collector: otelcol_exporter_enqueue_failed_spans
<p class="subtitle">exporter enqueue failed spans</p>
**Descriptions**
- <span class="badge badge-warning">warning</span> otel-collector: 0+ exporter enqueue failed spans for 5m0s
**Next steps**
- Check the configuration of the exporter and if the service being exported is up. This may be caused by a queue full of unsettled elements, so you may need to decrease your sending rate or horizontally scale collectors.
- More help interpreting this metric is available in the [dashboards reference](./dashboards.md#otel-collector-otelcol-exporter-enqueue-failed-spans).
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"warning_otel-collector_otelcol_exporter_enqueue_failed_spans"
]
```
monitoring: test owners for valid Opsgenie teams and handbook pages (#59251) In INC-264 it seems that certain alerts - such as [zoekt: less than 90% percentage pods available for 10m0s](https://opsg.in/a/i/sourcegraph/178a626f-0f28-4295-bee9-84da988bb473-1703759057681) - don't seem to end up going anywhere because the ObservableOwner is defunct. This change adds _opt-in_ testing to report: 1. How many owners have valid Opsgenie teams 2. How many owners have valid handbook pages In addition, we collect ObservableOwners that pass the test and use it to generate configuration for `site.json` in Sourcegraph.com: https://github.com/sourcegraph/deploy-sourcegraph-cloud/pull/18338 - this helps ensure the list is valid and not deceptively high-coverage. The results are not great, but **enforcing** that owners are valid isn't currently in scope: ``` 6/10 ObservableOwners do not have valid Opsgenie teams 3/10 ObservableOwners do not point to valid handbook pages ``` I also removed some defunct/unused functionality/owners. ## Test plan To run these tests: ``` export OPSGENIE_API_KEY="..." go test -timeout 30s github.com/sourcegraph/sourcegraph/monitoring/monitoring -update -online ```
2023-12-29 22:07:35 +00:00
<sub>*Managed by the [Sourcegraph Infrastructure Org team](https://handbook.sourcegraph.com/departments/engineering/infrastructure).*</sub>
monitoring: update otel dashboard metrics (#55899) * add metrics to OTEL * monitoring: update otel dashboard metrics * create a seperate group for Queue Length * Update monitoring/definitions/otel_collector.go Co-authored-by: William Bezuidenhout <william.bezuidenhout@sourcegraph.com> * update auto gen doc * update auto gen alerts doc --------- Co-authored-by: William Bezuidenhout <william.bezuidenhout@sourcegraph.com>
2023-08-17 01:03:12 +00:00
<details>
<summary>Technical details</summary>
Generated query for warning alert: `max((sum by (exporter) (rate(otelcol_exporter_enqueue_failed_spans{job=~"^.*"}[1m]))) > 0)`
</details>
<br />
## otel-collector: otelcol_processor_dropped_spans
<p class="subtitle">spans dropped per processor per minute</p>
**Descriptions**
- <span class="badge badge-warning">warning</span> otel-collector: 0+ spans dropped per processor per minute for 5m0s
**Next steps**
- Check the configuration of the processor
- More help interpreting this metric is available in the [dashboards reference](./dashboards.md#otel-collector-otelcol-processor-dropped-spans).
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"warning_otel-collector_otelcol_processor_dropped_spans"
]
```
monitoring: test owners for valid Opsgenie teams and handbook pages (#59251) In INC-264 it seems that certain alerts - such as [zoekt: less than 90% percentage pods available for 10m0s](https://opsg.in/a/i/sourcegraph/178a626f-0f28-4295-bee9-84da988bb473-1703759057681) - don't seem to end up going anywhere because the ObservableOwner is defunct. This change adds _opt-in_ testing to report: 1. How many owners have valid Opsgenie teams 2. How many owners have valid handbook pages In addition, we collect ObservableOwners that pass the test and use it to generate configuration for `site.json` in Sourcegraph.com: https://github.com/sourcegraph/deploy-sourcegraph-cloud/pull/18338 - this helps ensure the list is valid and not deceptively high-coverage. The results are not great, but **enforcing** that owners are valid isn't currently in scope: ``` 6/10 ObservableOwners do not have valid Opsgenie teams 3/10 ObservableOwners do not point to valid handbook pages ``` I also removed some defunct/unused functionality/owners. ## Test plan To run these tests: ``` export OPSGENIE_API_KEY="..." go test -timeout 30s github.com/sourcegraph/sourcegraph/monitoring/monitoring -update -online ```
2023-12-29 22:07:35 +00:00
<sub>*Managed by the [Sourcegraph Infrastructure Org team](https://handbook.sourcegraph.com/departments/engineering/infrastructure).*</sub>
monitoring: update otel dashboard metrics (#55899) * add metrics to OTEL * monitoring: update otel dashboard metrics * create a seperate group for Queue Length * Update monitoring/definitions/otel_collector.go Co-authored-by: William Bezuidenhout <william.bezuidenhout@sourcegraph.com> * update auto gen doc * update auto gen alerts doc --------- Co-authored-by: William Bezuidenhout <william.bezuidenhout@sourcegraph.com>
2023-08-17 01:03:12 +00:00
<details>
<summary>Technical details</summary>
Generated query for warning alert: `max((sum by (processor) (rate(otelcol_processor_dropped_spans[1m]))) > 0)`
</details>
<br />
otel: add collector dashboard (#45009) * add initial dashboard for otel * add failed sent dashboard * extra panels * use sum and rate for resource queries * review comments * add warning alerts * Update monitoring/definitions/otel_collector.go * review comments * run go generate * Update monitoring/definitions/otel_collector.go Co-authored-by: Robert Lin <robert@bobheadxi.dev> * Update monitoring/definitions/otel_collector.go Co-authored-by: Robert Lin <robert@bobheadxi.dev> * Update monitoring/definitions/otel_collector.go Co-authored-by: Robert Lin <robert@bobheadxi.dev> * Update monitoring/definitions/otel_collector.go Co-authored-by: Robert Lin <robert@bobheadxi.dev> * Update monitoring/definitions/otel_collector.go Co-authored-by: Robert Lin <robert@bobheadxi.dev> * Update monitoring/definitions/otel_collector.go Co-authored-by: Robert Lin <robert@bobheadxi.dev> * review comments * review feedback also drop two panels * remove brackets in metrics * update docs * fix goimport * gogenerate Co-authored-by: Robert Lin <robert@bobheadxi.dev> Co-authored-by: Jean-Hadrien Chabran <jh@chabran.fr>
2022-12-19 12:18:51 +00:00
## otel-collector: container_cpu_usage
<p class="subtitle">container cpu usage total (1m average) across all cores by instance</p>
**Descriptions**
- <span class="badge badge-warning">warning</span> otel-collector: 99%+ container cpu usage total (1m average) across all cores by instance
**Next steps**
- **Kubernetes:** Consider increasing CPU limits in the the relevant `Deployment.yaml`.
- **Docker Compose:** Consider increasing `cpus:` of the otel-collector container in `docker-compose.yml`.
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#otel-collector-container-cpu-usage).
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"warning_otel-collector_container_cpu_usage"
]
```
monitoring: test owners for valid Opsgenie teams and handbook pages (#59251) In INC-264 it seems that certain alerts - such as [zoekt: less than 90% percentage pods available for 10m0s](https://opsg.in/a/i/sourcegraph/178a626f-0f28-4295-bee9-84da988bb473-1703759057681) - don't seem to end up going anywhere because the ObservableOwner is defunct. This change adds _opt-in_ testing to report: 1. How many owners have valid Opsgenie teams 2. How many owners have valid handbook pages In addition, we collect ObservableOwners that pass the test and use it to generate configuration for `site.json` in Sourcegraph.com: https://github.com/sourcegraph/deploy-sourcegraph-cloud/pull/18338 - this helps ensure the list is valid and not deceptively high-coverage. The results are not great, but **enforcing** that owners are valid isn't currently in scope: ``` 6/10 ObservableOwners do not have valid Opsgenie teams 3/10 ObservableOwners do not point to valid handbook pages ``` I also removed some defunct/unused functionality/owners. ## Test plan To run these tests: ``` export OPSGENIE_API_KEY="..." go test -timeout 30s github.com/sourcegraph/sourcegraph/monitoring/monitoring -update -online ```
2023-12-29 22:07:35 +00:00
<sub>*Managed by the [Sourcegraph Infrastructure Org team](https://handbook.sourcegraph.com/departments/engineering/infrastructure).*</sub>
otel: add collector dashboard (#45009) * add initial dashboard for otel * add failed sent dashboard * extra panels * use sum and rate for resource queries * review comments * add warning alerts * Update monitoring/definitions/otel_collector.go * review comments * run go generate * Update monitoring/definitions/otel_collector.go Co-authored-by: Robert Lin <robert@bobheadxi.dev> * Update monitoring/definitions/otel_collector.go Co-authored-by: Robert Lin <robert@bobheadxi.dev> * Update monitoring/definitions/otel_collector.go Co-authored-by: Robert Lin <robert@bobheadxi.dev> * Update monitoring/definitions/otel_collector.go Co-authored-by: Robert Lin <robert@bobheadxi.dev> * Update monitoring/definitions/otel_collector.go Co-authored-by: Robert Lin <robert@bobheadxi.dev> * Update monitoring/definitions/otel_collector.go Co-authored-by: Robert Lin <robert@bobheadxi.dev> * review comments * review feedback also drop two panels * remove brackets in metrics * update docs * fix goimport * gogenerate Co-authored-by: Robert Lin <robert@bobheadxi.dev> Co-authored-by: Jean-Hadrien Chabran <jh@chabran.fr>
2022-12-19 12:18:51 +00:00
<details>
<summary>Technical details</summary>
Generated query for warning alert: `max((cadvisor_container_cpu_usage_percentage_total{name=~"^otel-collector.*"}) >= 99)`
</details>
<br />
## otel-collector: container_memory_usage
<p class="subtitle">container memory usage by instance</p>
**Descriptions**
- <span class="badge badge-warning">warning</span> otel-collector: 99%+ container memory usage by instance
**Next steps**
- **Kubernetes:** Consider increasing memory limit in relevant `Deployment.yaml`.
- **Docker Compose:** Consider increasing `memory:` of otel-collector container in `docker-compose.yml`.
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#otel-collector-container-memory-usage).
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"warning_otel-collector_container_memory_usage"
]
```
monitoring: test owners for valid Opsgenie teams and handbook pages (#59251) In INC-264 it seems that certain alerts - such as [zoekt: less than 90% percentage pods available for 10m0s](https://opsg.in/a/i/sourcegraph/178a626f-0f28-4295-bee9-84da988bb473-1703759057681) - don't seem to end up going anywhere because the ObservableOwner is defunct. This change adds _opt-in_ testing to report: 1. How many owners have valid Opsgenie teams 2. How many owners have valid handbook pages In addition, we collect ObservableOwners that pass the test and use it to generate configuration for `site.json` in Sourcegraph.com: https://github.com/sourcegraph/deploy-sourcegraph-cloud/pull/18338 - this helps ensure the list is valid and not deceptively high-coverage. The results are not great, but **enforcing** that owners are valid isn't currently in scope: ``` 6/10 ObservableOwners do not have valid Opsgenie teams 3/10 ObservableOwners do not point to valid handbook pages ``` I also removed some defunct/unused functionality/owners. ## Test plan To run these tests: ``` export OPSGENIE_API_KEY="..." go test -timeout 30s github.com/sourcegraph/sourcegraph/monitoring/monitoring -update -online ```
2023-12-29 22:07:35 +00:00
<sub>*Managed by the [Sourcegraph Infrastructure Org team](https://handbook.sourcegraph.com/departments/engineering/infrastructure).*</sub>
otel: add collector dashboard (#45009) * add initial dashboard for otel * add failed sent dashboard * extra panels * use sum and rate for resource queries * review comments * add warning alerts * Update monitoring/definitions/otel_collector.go * review comments * run go generate * Update monitoring/definitions/otel_collector.go Co-authored-by: Robert Lin <robert@bobheadxi.dev> * Update monitoring/definitions/otel_collector.go Co-authored-by: Robert Lin <robert@bobheadxi.dev> * Update monitoring/definitions/otel_collector.go Co-authored-by: Robert Lin <robert@bobheadxi.dev> * Update monitoring/definitions/otel_collector.go Co-authored-by: Robert Lin <robert@bobheadxi.dev> * Update monitoring/definitions/otel_collector.go Co-authored-by: Robert Lin <robert@bobheadxi.dev> * Update monitoring/definitions/otel_collector.go Co-authored-by: Robert Lin <robert@bobheadxi.dev> * review comments * review feedback also drop two panels * remove brackets in metrics * update docs * fix goimport * gogenerate Co-authored-by: Robert Lin <robert@bobheadxi.dev> Co-authored-by: Jean-Hadrien Chabran <jh@chabran.fr>
2022-12-19 12:18:51 +00:00
<details>
<summary>Technical details</summary>
Generated query for warning alert: `max((cadvisor_container_memory_usage_percentage_total{name=~"^otel-collector.*"}) >= 99)`
</details>
<br />
## otel-collector: pods_available_percentage
<p class="subtitle">percentage pods available</p>
**Descriptions**
- <span class="badge badge-critical">critical</span> otel-collector: less than 90% percentage pods available for 10m0s
**Next steps**
- Determine if the pod was OOM killed using `kubectl describe pod otel-collector` (look for `OOMKilled: true`) and, if so, consider increasing the memory limit in the relevant `Deployment.yaml`.
- Check the logs before the container restarted to see if there are `panic:` messages or similar using `kubectl logs -p otel-collector`.
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#otel-collector-pods-available-percentage).
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"critical_otel-collector_pods_available_percentage"
]
```
monitoring: test owners for valid Opsgenie teams and handbook pages (#59251) In INC-264 it seems that certain alerts - such as [zoekt: less than 90% percentage pods available for 10m0s](https://opsg.in/a/i/sourcegraph/178a626f-0f28-4295-bee9-84da988bb473-1703759057681) - don't seem to end up going anywhere because the ObservableOwner is defunct. This change adds _opt-in_ testing to report: 1. How many owners have valid Opsgenie teams 2. How many owners have valid handbook pages In addition, we collect ObservableOwners that pass the test and use it to generate configuration for `site.json` in Sourcegraph.com: https://github.com/sourcegraph/deploy-sourcegraph-cloud/pull/18338 - this helps ensure the list is valid and not deceptively high-coverage. The results are not great, but **enforcing** that owners are valid isn't currently in scope: ``` 6/10 ObservableOwners do not have valid Opsgenie teams 3/10 ObservableOwners do not point to valid handbook pages ``` I also removed some defunct/unused functionality/owners. ## Test plan To run these tests: ``` export OPSGENIE_API_KEY="..." go test -timeout 30s github.com/sourcegraph/sourcegraph/monitoring/monitoring -update -online ```
2023-12-29 22:07:35 +00:00
<sub>*Managed by the [Sourcegraph Infrastructure Org team](https://handbook.sourcegraph.com/departments/engineering/infrastructure).*</sub>
otel: add collector dashboard (#45009) * add initial dashboard for otel * add failed sent dashboard * extra panels * use sum and rate for resource queries * review comments * add warning alerts * Update monitoring/definitions/otel_collector.go * review comments * run go generate * Update monitoring/definitions/otel_collector.go Co-authored-by: Robert Lin <robert@bobheadxi.dev> * Update monitoring/definitions/otel_collector.go Co-authored-by: Robert Lin <robert@bobheadxi.dev> * Update monitoring/definitions/otel_collector.go Co-authored-by: Robert Lin <robert@bobheadxi.dev> * Update monitoring/definitions/otel_collector.go Co-authored-by: Robert Lin <robert@bobheadxi.dev> * Update monitoring/definitions/otel_collector.go Co-authored-by: Robert Lin <robert@bobheadxi.dev> * Update monitoring/definitions/otel_collector.go Co-authored-by: Robert Lin <robert@bobheadxi.dev> * review comments * review feedback also drop two panels * remove brackets in metrics * update docs * fix goimport * gogenerate Co-authored-by: Robert Lin <robert@bobheadxi.dev> Co-authored-by: Jean-Hadrien Chabran <jh@chabran.fr>
2022-12-19 12:18:51 +00:00
<details>
<summary>Technical details</summary>
monitoring: extract into a submodule (#45786) This change extracts `monitoring` into a submodule for import in `sourcegraph/controller` (https://github.com/sourcegraph/controller/pull/195) so that we can generate dashboards for Cloud instances. These steps were required: 1. Initialize a `go.mod` in `monitoring` 2. Extract `dev/sg/internal/cliutil` into `lib` to avoid illegal imports from `monitoring` 3. Add local replaces to both `sourcegraph/sourcegraph` and `monitoring` 4. `go mod tidy` on all submodules 5. Update `go generate ./monitoring` commands to use `sg`, since the `go generate` command no longer works 6. Update `grafana/build.sh`, `prometheus/build.sh` to build the submodule 7. Amend linters to check for multiple `go.mod` files and ban imports of `github.com/sourcegraph/sourcegraph` 8. Update `sg generate go` to run in directories rather than from root The only caveat is that if you use VS code, you will now need to open `monitoring` in a separate workspace or similar, like with `lib`. Co-authored-by: Joe Chen <joe@sourcegraph.com>
2022-12-19 17:49:25 +00:00
Generated query for critical alert: `min((sum by (app) (up{app=~".*otel-collector"}) / count by (app) (up{app=~".*otel-collector"}) * 100) <= 90)`
otel: add collector dashboard (#45009) * add initial dashboard for otel * add failed sent dashboard * extra panels * use sum and rate for resource queries * review comments * add warning alerts * Update monitoring/definitions/otel_collector.go * review comments * run go generate * Update monitoring/definitions/otel_collector.go Co-authored-by: Robert Lin <robert@bobheadxi.dev> * Update monitoring/definitions/otel_collector.go Co-authored-by: Robert Lin <robert@bobheadxi.dev> * Update monitoring/definitions/otel_collector.go Co-authored-by: Robert Lin <robert@bobheadxi.dev> * Update monitoring/definitions/otel_collector.go Co-authored-by: Robert Lin <robert@bobheadxi.dev> * Update monitoring/definitions/otel_collector.go Co-authored-by: Robert Lin <robert@bobheadxi.dev> * Update monitoring/definitions/otel_collector.go Co-authored-by: Robert Lin <robert@bobheadxi.dev> * review comments * review feedback also drop two panels * remove brackets in metrics * update docs * fix goimport * gogenerate Co-authored-by: Robert Lin <robert@bobheadxi.dev> Co-authored-by: Jean-Hadrien Chabran <jh@chabran.fr>
2022-12-19 12:18:51 +00:00
</details>
<br />
conf: add metric and associated alert if clients fail to update site configuration within 5 minutes (#57682)
2023-10-18 23:53:55 +00:00
## embeddings: embeddings_site_configuration_duration_since_last_successful_update_by_instance
<p class="subtitle">maximum duration since last successful site configuration update (all "embeddings" instances)</p>
**Descriptions**
- <span class="badge badge-critical">critical</span> embeddings: 300s+ maximum duration since last successful site configuration update (all "embeddings" instances)
**Next steps**
- This indicates that one or more "embeddings" instances have not successfully updated the site configuration in over 5 minutes. This could be due to networking issues between services or problems with the site configuration service itself.
- Check for relevant errors in the "embeddings" logs, as well as frontend`s logs.
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#embeddings-embeddings-site-configuration-duration-since-last-successful-update-by-instance).
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"critical_embeddings_embeddings_site_configuration_duration_since_last_successful_update_by_instance"
]
```
monitoring: test owners for valid Opsgenie teams and handbook pages (#59251) In INC-264 it seems that certain alerts - such as [zoekt: less than 90% percentage pods available for 10m0s](https://opsg.in/a/i/sourcegraph/178a626f-0f28-4295-bee9-84da988bb473-1703759057681) - don't seem to end up going anywhere because the ObservableOwner is defunct. This change adds _opt-in_ testing to report: 1. How many owners have valid Opsgenie teams 2. How many owners have valid handbook pages In addition, we collect ObservableOwners that pass the test and use it to generate configuration for `site.json` in Sourcegraph.com: https://github.com/sourcegraph/deploy-sourcegraph-cloud/pull/18338 - this helps ensure the list is valid and not deceptively high-coverage. The results are not great, but **enforcing** that owners are valid isn't currently in scope: ``` 6/10 ObservableOwners do not have valid Opsgenie teams 3/10 ObservableOwners do not point to valid handbook pages ``` I also removed some defunct/unused functionality/owners. ## Test plan To run these tests: ``` export OPSGENIE_API_KEY="..." go test -timeout 30s github.com/sourcegraph/sourcegraph/monitoring/monitoring -update -online ```
2023-12-29 22:07:35 +00:00
<sub>*Managed by the [Sourcegraph Infrastructure Org team](https://handbook.sourcegraph.com/departments/engineering/infrastructure).*</sub>
conf: add metric and associated alert if clients fail to update site configuration within 5 minutes (#57682)
2023-10-18 23:53:55 +00:00
<details>
<summary>Technical details</summary>
monitoring: fix alert definition for site configuration by adding scrape job label (#59687) We discovered recently that the definition for the alert that fires if the site configuration hasn't been fetched within 5 minutes strips out the regex that targets individual services (since it uses a grafana variable). This means that every instance of this alert will fire if any individual service trips over this threshold. This PR fixes the issue by adding a new `job` filter for this alert that targets only the services that have that Prometheus scrape target name. This works around the previous issue by using a fixed value for the `job` value instead of a dynamic grafana value. The value of the job filter generally looks like `job=~.*$container_name` (following the strategy from https://sourcegraph.com/github.com/sourcegraph/sourcegraph@9a780f2e694238b5326e3e121d6a1828463001b9/-/blob/monitoring/monitoring/monitoring.go?L161 ) unless I noticed that there was different logic in the existing dashboard for the services. Ex: - `frontend`: already used `job=~"(sourcegraph-)?frontend"` for some metrics, so I used it again here - `worker`: `already used `job=~"^worker.*"` in some metrics, so I used it again and standarized the other existing panels to use the same shared variable ## Test plan I eyeballed the generated alert.md and dashboards.md to verify that my changes looked correct (that is, my refactors resulted in either no diff, or that the diff I generated still looked like valid regex).
2024-01-17 23:19:54 +00:00
Generated query for critical alert: `max((max(max_over_time(src_conf_client_time_since_last_successful_update_seconds{job=~".*embeddings"}[1m]))) >= 300)`
conf: add metric and associated alert if clients fail to update site configuration within 5 minutes (#57682)
2023-10-18 23:53:55 +00:00
</details>
<br />
embeddings: container metrics (#48897) Adds basic resource metrics for embeddings container. ## Test plan * Manual testing.
2023-03-08 09:42:10 +00:00
## embeddings: mean_blocked_seconds_per_conn_request
<p class="subtitle">mean blocked seconds per conn request</p>
**Descriptions**
monitoring: relax mean_blocked_seconds_per_conn_request alerts (#59507) https://github.com/sourcegraph/sourcegraph/pull/59284 dramatically reduced the `mean_blocked_seconds_per_conn_request` issues we've been seeing, but overall delays are still higher, even with generally healthy Cloud SQL resource utilization. <img width="1630" alt="image" src="https://github.com/sourcegraph/sourcegraph/assets/23356519/91615471-5187-4d15-83e7-5cc94595303c"> Spot-checking the spikes in load in Cloud SQL, it seems that there is a variety of causes for each spike (analytics workloads, Cody Gateway syncs, code intel workloads, gitserver things, `ListSourcegraphDotComIndexableRepos` etc) so I'm chalking this up to "expected". Since this alert is seen firing on a Cloud instance, let's just relax it for now so that it only fires a critical alert on very significant delays.
2024-01-11 01:14:28 +00:00
- <span class="badge badge-warning">warning</span> embeddings: 0.1s+ mean blocked seconds per conn request for 10m0s
- <span class="badge badge-critical">critical</span> embeddings: 0.5s+ mean blocked seconds per conn request for 10m0s
embeddings: container metrics (#48897) Adds basic resource metrics for embeddings container. ## Test plan * Manual testing.
2023-03-08 09:42:10 +00:00
**Next steps**
- Increase SRC_PGSQL_MAX_OPEN together with giving more memory to the database if needed
Docs: update links to point to new site (#60381) We have a number of docs links in the product that point to the old doc site. Method: - Search the repo for `docs.sourcegraph.com` - Exclude the `doc/` dir, all test fixtures, and `CHANGELOG.md` - For each, replace `docs.sourcegraph.com` with `sourcegraph.com/docs` - Navigate to the resulting URL ensuring it's not a dead link, updating the URL if necessary Many of the URLs updated are just comments, but since I'm doing a manual audit of each URL anyways, I felt it was worth it to update these while I was at it.
2024-02-13 00:23:47 +00:00
- Scale up Postgres memory/cpus - [see our scaling guide](https://sourcegraph.com/docs/admin/config/postgres-conf)
monitoring: relax mean_blocked_seconds_per_conn_request alerts (#59507) https://github.com/sourcegraph/sourcegraph/pull/59284 dramatically reduced the `mean_blocked_seconds_per_conn_request` issues we've been seeing, but overall delays are still higher, even with generally healthy Cloud SQL resource utilization. <img width="1630" alt="image" src="https://github.com/sourcegraph/sourcegraph/assets/23356519/91615471-5187-4d15-83e7-5cc94595303c"> Spot-checking the spikes in load in Cloud SQL, it seems that there is a variety of causes for each spike (analytics workloads, Cody Gateway syncs, code intel workloads, gitserver things, `ListSourcegraphDotComIndexableRepos` etc) so I'm chalking this up to "expected". Since this alert is seen firing on a Cloud instance, let's just relax it for now so that it only fires a critical alert on very significant delays.
2024-01-11 01:14:28 +00:00
- If using GCP Cloud SQL, check for high lock waits or CPU usage in query insights
embeddings: container metrics (#48897) Adds basic resource metrics for embeddings container. ## Test plan * Manual testing.
2023-03-08 09:42:10 +00:00
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#embeddings-mean-blocked-seconds-per-conn-request).
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"warning_embeddings_mean_blocked_seconds_per_conn_request",
"critical_embeddings_mean_blocked_seconds_per_conn_request"
]
```
monitoring: assign obvious owners to DatabaseConnectionsMonitoringGroup (#58474) updates `DatabaseConnectionsMonitoringGroup` to accept an owner - before it was just hardcoded to `ObservableOwnerDevOps`, which is not very helpful. This assigns some of the obvious service owners: 1. Source: gitserver, repo-updater 2. Cody: embeddings (but should eventually be @sourcegraph/search-platform, along with all embeddings alerts: https://github.com/sourcegraph/sourcegraph/pull/58474#issuecomment-1821505062) Source is an active owner based on [thread](https://sourcegraph.slack.com/archives/C0652SSUA20/p1700592165408089?thread_ts=1700549423.860019&cid=C0652SSUA20), and Cody is a fairly recent addition so hopefully it's valid. I'm not sure the Search one is still up-to-date, so I didn't change some of the obvious search services - for now, these still point to DevOps as they did before. If it becomes problematic we can revisit later.
2023-11-22 01:06:53 +00:00
<sub>*Managed by the [Sourcegraph Cody team](https://handbook.sourcegraph.com/departments/engineering/teams/cody).*</sub>
embeddings: container metrics (#48897) Adds basic resource metrics for embeddings container. ## Test plan * Manual testing.
2023-03-08 09:42:10 +00:00
<details>
<summary>Technical details</summary>
monitoring: relax mean_blocked_seconds_per_conn_request alerts (#59507) https://github.com/sourcegraph/sourcegraph/pull/59284 dramatically reduced the `mean_blocked_seconds_per_conn_request` issues we've been seeing, but overall delays are still higher, even with generally healthy Cloud SQL resource utilization. <img width="1630" alt="image" src="https://github.com/sourcegraph/sourcegraph/assets/23356519/91615471-5187-4d15-83e7-5cc94595303c"> Spot-checking the spikes in load in Cloud SQL, it seems that there is a variety of causes for each spike (analytics workloads, Cody Gateway syncs, code intel workloads, gitserver things, `ListSourcegraphDotComIndexableRepos` etc) so I'm chalking this up to "expected". Since this alert is seen firing on a Cloud instance, let's just relax it for now so that it only fires a critical alert on very significant delays.
2024-01-11 01:14:28 +00:00
Generated query for warning alert: `max((sum by (app_name, db_name) (increase(src_pgsql_conns_blocked_seconds{app_name="embeddings"}[5m])) / sum by (app_name, db_name) (increase(src_pgsql_conns_waited_for{app_name="embeddings"}[5m]))) >= 0.1)`
embeddings: container metrics (#48897) Adds basic resource metrics for embeddings container. ## Test plan * Manual testing.
2023-03-08 09:42:10 +00:00
monitoring: relax mean_blocked_seconds_per_conn_request alerts (#59507) https://github.com/sourcegraph/sourcegraph/pull/59284 dramatically reduced the `mean_blocked_seconds_per_conn_request` issues we've been seeing, but overall delays are still higher, even with generally healthy Cloud SQL resource utilization. <img width="1630" alt="image" src="https://github.com/sourcegraph/sourcegraph/assets/23356519/91615471-5187-4d15-83e7-5cc94595303c"> Spot-checking the spikes in load in Cloud SQL, it seems that there is a variety of causes for each spike (analytics workloads, Cody Gateway syncs, code intel workloads, gitserver things, `ListSourcegraphDotComIndexableRepos` etc) so I'm chalking this up to "expected". Since this alert is seen firing on a Cloud instance, let's just relax it for now so that it only fires a critical alert on very significant delays.
2024-01-11 01:14:28 +00:00
Generated query for critical alert: `max((sum by (app_name, db_name) (increase(src_pgsql_conns_blocked_seconds{app_name="embeddings"}[5m])) / sum by (app_name, db_name) (increase(src_pgsql_conns_waited_for{app_name="embeddings"}[5m]))) >= 0.5)`
embeddings: container metrics (#48897) Adds basic resource metrics for embeddings container. ## Test plan * Manual testing.
2023-03-08 09:42:10 +00:00
</details>
<br />
## embeddings: container_cpu_usage
<p class="subtitle">container cpu usage total (1m average) across all cores by instance</p>
**Descriptions**
- <span class="badge badge-warning">warning</span> embeddings: 99%+ container cpu usage total (1m average) across all cores by instance
**Next steps**
- **Kubernetes:** Consider increasing CPU limits in the the relevant `Deployment.yaml`.
- **Docker Compose:** Consider increasing `cpus:` of the embeddings container in `docker-compose.yml`.
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#embeddings-container-cpu-usage).
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"warning_embeddings_container_cpu_usage"
]
```
<sub>*Managed by the [Sourcegraph Cody team](https://handbook.sourcegraph.com/departments/engineering/teams/cody).*</sub>
<details>
<summary>Technical details</summary>
Generated query for warning alert: `max((cadvisor_container_cpu_usage_percentage_total{name=~"^embeddings.*"}) >= 99)`
</details>
<br />
## embeddings: container_memory_usage
<p class="subtitle">container memory usage by instance</p>
**Descriptions**
- <span class="badge badge-warning">warning</span> embeddings: 99%+ container memory usage by instance
**Next steps**
- **Kubernetes:** Consider increasing memory limit in relevant `Deployment.yaml`.
- **Docker Compose:** Consider increasing `memory:` of embeddings container in `docker-compose.yml`.
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#embeddings-container-memory-usage).
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"warning_embeddings_container_memory_usage"
]
```
<sub>*Managed by the [Sourcegraph Cody team](https://handbook.sourcegraph.com/departments/engineering/teams/cody).*</sub>
<details>
<summary>Technical details</summary>
Generated query for warning alert: `max((cadvisor_container_memory_usage_percentage_total{name=~"^embeddings.*"}) >= 99)`
</details>
<br />
## embeddings: provisioning_container_cpu_usage_long_term
<p class="subtitle">container cpu usage total (90th percentile over 1d) across all cores by instance</p>
**Descriptions**
- <span class="badge badge-warning">warning</span> embeddings: 80%+ container cpu usage total (90th percentile over 1d) across all cores by instance for 336h0m0s
**Next steps**
- **Kubernetes:** Consider increasing CPU limits in the `Deployment.yaml` for the embeddings service.
- **Docker Compose:** Consider increasing `cpus:` of the embeddings container in `docker-compose.yml`.
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#embeddings-provisioning-container-cpu-usage-long-term).
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"warning_embeddings_provisioning_container_cpu_usage_long_term"
]
```
<sub>*Managed by the [Sourcegraph Cody team](https://handbook.sourcegraph.com/departments/engineering/teams/cody).*</sub>
<details>
<summary>Technical details</summary>
Generated query for warning alert: `max((quantile_over_time(0.9, cadvisor_container_cpu_usage_percentage_total{name=~"^embeddings.*"}[1d])) >= 80)`
</details>
<br />
## embeddings: provisioning_container_memory_usage_long_term
<p class="subtitle">container memory usage (1d maximum) by instance</p>
**Descriptions**
- <span class="badge badge-warning">warning</span> embeddings: 80%+ container memory usage (1d maximum) by instance for 336h0m0s
**Next steps**
- **Kubernetes:** Consider increasing memory limits in the `Deployment.yaml` for the embeddings service.
- **Docker Compose:** Consider increasing `memory:` of the embeddings container in `docker-compose.yml`.
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#embeddings-provisioning-container-memory-usage-long-term).
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"warning_embeddings_provisioning_container_memory_usage_long_term"
]
```
<sub>*Managed by the [Sourcegraph Cody team](https://handbook.sourcegraph.com/departments/engineering/teams/cody).*</sub>
<details>
<summary>Technical details</summary>
Generated query for warning alert: `max((max_over_time(cadvisor_container_memory_usage_percentage_total{name=~"^embeddings.*"}[1d])) >= 80)`
</details>
<br />
## embeddings: provisioning_container_cpu_usage_short_term
<p class="subtitle">container cpu usage total (5m maximum) across all cores by instance</p>
**Descriptions**
- <span class="badge badge-warning">warning</span> embeddings: 90%+ container cpu usage total (5m maximum) across all cores by instance for 30m0s
**Next steps**
- **Kubernetes:** Consider increasing CPU limits in the the relevant `Deployment.yaml`.
- **Docker Compose:** Consider increasing `cpus:` of the embeddings container in `docker-compose.yml`.
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#embeddings-provisioning-container-cpu-usage-short-term).
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"warning_embeddings_provisioning_container_cpu_usage_short_term"
]
```
<sub>*Managed by the [Sourcegraph Cody team](https://handbook.sourcegraph.com/departments/engineering/teams/cody).*</sub>
<details>
<summary>Technical details</summary>
Generated query for warning alert: `max((max_over_time(cadvisor_container_cpu_usage_percentage_total{name=~"^embeddings.*"}[5m])) >= 90)`
</details>
<br />
## embeddings: provisioning_container_memory_usage_short_term
<p class="subtitle">container memory usage (5m maximum) by instance</p>
**Descriptions**
- <span class="badge badge-warning">warning</span> embeddings: 90%+ container memory usage (5m maximum) by instance
**Next steps**
- **Kubernetes:** Consider increasing memory limit in relevant `Deployment.yaml`.
- **Docker Compose:** Consider increasing `memory:` of embeddings container in `docker-compose.yml`.
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#embeddings-provisioning-container-memory-usage-short-term).
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"warning_embeddings_provisioning_container_memory_usage_short_term"
]
```
<sub>*Managed by the [Sourcegraph Cody team](https://handbook.sourcegraph.com/departments/engineering/teams/cody).*</sub>
<details>
<summary>Technical details</summary>
Generated query for warning alert: `max((max_over_time(cadvisor_container_memory_usage_percentage_total{name=~"^embeddings.*"}[5m])) >= 90)`
</details>
<br />
## embeddings: container_oomkill_events_total
<p class="subtitle">container OOMKILL events total by instance</p>
**Descriptions**
- <span class="badge badge-warning">warning</span> embeddings: 1+ container OOMKILL events total by instance
**Next steps**
- **Kubernetes:** Consider increasing memory limit in relevant `Deployment.yaml`.
- **Docker Compose:** Consider increasing `memory:` of embeddings container in `docker-compose.yml`.
- More help interpreting this metric is available in the [dashboards reference](./dashboards.md#embeddings-container-oomkill-events-total).
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"warning_embeddings_container_oomkill_events_total"
]
```
<sub>*Managed by the [Sourcegraph Cody team](https://handbook.sourcegraph.com/departments/engineering/teams/cody).*</sub>
<details>
<summary>Technical details</summary>
Generated query for warning alert: `max((max by (name) (container_oom_events_total{name=~"^embeddings.*"})) >= 1)`
</details>
<br />
## embeddings: go_goroutines
<p class="subtitle">maximum active goroutines</p>
**Descriptions**
- <span class="badge badge-warning">warning</span> embeddings: 10000+ maximum active goroutines for 10m0s
**Next steps**
- More help interpreting this metric is available in the [dashboards reference](./dashboards.md#embeddings-go-goroutines).
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"warning_embeddings_go_goroutines"
]
```
<sub>*Managed by the [Sourcegraph Cody team](https://handbook.sourcegraph.com/departments/engineering/teams/cody).*</sub>
<details>
<summary>Technical details</summary>
Generated query for warning alert: `max((max by (instance) (go_goroutines{job=~".*embeddings"})) >= 10000)`
</details>
<br />
## embeddings: go_gc_duration_seconds
<p class="subtitle">maximum go garbage collection duration</p>
**Descriptions**
- <span class="badge badge-warning">warning</span> embeddings: 2s+ maximum go garbage collection duration
**Next steps**
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#embeddings-go-gc-duration-seconds).
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"warning_embeddings_go_gc_duration_seconds"
]
```
<sub>*Managed by the [Sourcegraph Cody team](https://handbook.sourcegraph.com/departments/engineering/teams/cody).*</sub>
<details>
<summary>Technical details</summary>
Generated query for warning alert: `max((max by (instance) (go_gc_duration_seconds{job=~".*embeddings"})) >= 2)`
</details>
<br />
## embeddings: pods_available_percentage
<p class="subtitle">percentage pods available</p>
**Descriptions**
- <span class="badge badge-critical">critical</span> embeddings: less than 90% percentage pods available for 10m0s
**Next steps**
- Determine if the pod was OOM killed using `kubectl describe pod embeddings` (look for `OOMKilled: true`) and, if so, consider increasing the memory limit in the relevant `Deployment.yaml`.
- Check the logs before the container restarted to see if there are `panic:` messages or similar using `kubectl logs -p embeddings`.
- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#embeddings-pods-available-percentage).
- **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert:
```json
"observability.silenceAlerts": [
"critical_embeddings_pods_available_percentage"
]
```
<sub>*Managed by the [Sourcegraph Cody team](https://handbook.sourcegraph.com/departments/engineering/teams/cody).*</sub>
<details>
<summary>Technical details</summary>
Generated query for critical alert: `min((sum by (app) (up{app=~".*embeddings"}) / count by (app) (up{app=~".*embeddings"}) * 100) <= 90)`
</details>
<br />
Reference in New Issue Copy Permalink