90th percentile blob load latency over 10m. The 90th percentile of API calls to the blob route in the frontend API is at 5 seconds or more, meaning calls to the blob route, are slow to return a response. The blob API route provides the files and code snippets that the UI displays. When this alert fires, the UI will likely experience delays loading files and code snippets. It is likely that the gitserver and/or frontend services are experiencing issues, leading to slower responses.
+90th percentile blob load latency over 10m
**Descriptions** -- critical frontend: 5s+ 90th percentile blob load latency over 10m. The 90th percentile of API calls to the blob route in the frontend API is at 5 seconds or more, meaning calls to the blob route, are slow to return a response. The blob API route provides the files and code snippets that the UI displays. When this alert fires, the UI will likely experience delays loading files and code snippets. It is likely that the gitserver and/or frontend services are experiencing issues, leading to slower responses. +- critical frontend: 5s+ 90th percentile blob load latency over 10m **Next steps** -- Confirm that the Sourcegraph frontend has enough CPU/memory using the provisioning panels. +- When this alert fires, calls to the blob route are slow to return a response. The UI will likely experience delays loading files and code snippets. It is likely that the gitserver and/or frontend services are experiencing issues, leading to slower responses. +- Confirm that the Sourcegraph gitserver and frontend services have enough CPU/memory using the provisioning panels. - Trace a request to see what the slowest part is: https://docs.sourcegraph.com/admin/observability/tracing - Check that gitserver containers have enough CPU/memory and are not getting throttled. -- Learn more about the related dashboard panel in the [dashboards reference](./dashboards.md#frontend-blob-load-latency). +- More help interpreting this metric is available in the [dashboards reference](./dashboards.md#frontend-blob-load-latency). - **Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration and set a reminder to re-evaluate the alert: ```json diff --git a/doc/admin/observability/dashboards.md b/doc/admin/observability/dashboards.md index 8337050af43..6a5530f3535 100644 --- a/doc/admin/observability/dashboards.md +++ b/doc/admin/observability/dashboards.md @@ -149,7 +149,9 @@ Query: `histogram_quantile(0.9, sum by(le) (rate(src_http_request_duration_secon #### frontend: blob_load_latency -90th percentile blob load latency over 10m. The 90th percentile of API calls to the blob route in the frontend API is at 5 seconds or more, meaning calls to the blob route, are slow to return a response. The blob API route provides the files and code snippets that the UI displays. When this alert fires, the UI will likely experience delays loading files and code snippets. It is likely that the gitserver and/or frontend services are experiencing issues, leading to slower responses.
+90th percentile blob load latency over 10m
+ +- The blob API route provides the files and code snippets that the UI displays. Refer to the [alerts reference](./alerts.md#frontend-blob-load-latency) for 1 alert related to this panel. diff --git a/monitoring/definitions/frontend.go b/monitoring/definitions/frontend.go index 58d89af83c0..0ac43a650a4 100644 --- a/monitoring/definitions/frontend.go +++ b/monitoring/definitions/frontend.go @@ -162,13 +162,17 @@ func Frontend() *monitoring.Dashboard { }, { Name: "blob_load_latency", - Description: "90th percentile blob load latency over 10m. The 90th percentile of API calls to the blob route in the frontend API is at 5 seconds or more, meaning calls to the blob route, are slow to return a response. The blob API route provides the files and code snippets that the UI displays. When this alert fires, the UI will likely experience delays loading files and code snippets. It is likely that the gitserver and/or frontend services are experiencing issues, leading to slower responses.", + Description: "90th percentile blob load latency over 10m", Query: `histogram_quantile(0.9, sum by(le) (rate(src_http_request_duration_seconds_bucket{route="blob"}[10m])))`, Critical: monitoring.Alert().GreaterOrEqual(5), Panel: monitoring.Panel().LegendFormat("latency").Unit(monitoring.Seconds), Owner: monitoring.ObservableOwnerSource, + Interpretation: ` + - The blob API route provides the files and code snippets that the UI displays. + `, NextSteps: ` - - Confirm that the Sourcegraph frontend has enough CPU/memory using the provisioning panels. + - When this alert fires, calls to the blob route are slow to return a response. The UI will likely experience delays loading files and code snippets. It is likely that the gitserver and/or frontend services are experiencing issues, leading to slower responses. + - Confirm that the Sourcegraph gitserver and frontend services have enough CPU/memory using the provisioning panels. - Trace a request to see what the slowest part is: https://docs.sourcegraph.com/admin/observability/tracing - Check that gitserver containers have enough CPU/memory and are not getting throttled. `, diff --git a/monitoring/monitoring/monitoring.go b/monitoring/monitoring/monitoring.go index 63d369b3abc..4b13b78bc64 100644 --- a/monitoring/monitoring/monitoring.go +++ b/monitoring/monitoring/monitoring.go @@ -596,6 +596,12 @@ func (o Observable) validate(variables []ContainerVariable) error { if len(o.Description) == 0 { return errors.New("Description must be set") } + // Avoid paragraphs in the brief description + const maxDescriptionLength = 120 + if len(o.Description) > maxDescriptionLength { + return errors.Newf("Description must be under %d characters (current length: %d) - to provide more context, use Interpretation and NextSteps fields instead", + maxDescriptionLength, len(o.Description)) + } if first, second := string([]rune(o.Description)[0]), string([]rune(o.Description)[1]); first != strings.ToLower(first) && second == strings.ToLower(second) { return errors.Errorf("Description must be lowercase except for acronyms; found \"%s\"", o.Description) }