sourcegraph/doc/admin/observability/metrics.md

Metrics

Sourcegraph uses Prometheus for metrics and Grafana for metrics dashboards.

If you're using the Kubernetes cluster deployment option, see the Prometheus README for more information.

Prometheus

Prometheus is a monitoring tool that collects application- and system-level metrics over time and makes these accessible through a query language and simple UI.

Accessing Prometheus

Most of the time, Sourcegraph site admins will monitor key metrics through the Grafana UI, rather than through Prometheus directly. Grafana provides the dashboards that monitor the standard metrics that indicate the health of the instance. Only if an admin wants to write a novel metrics formula or query do they need to access the Prometheus UI.

If you are using single-container Sourcegraph, you will need to restart the Sourcegraph container with a flag --publish 9090:9090 in the docker run command. Subsequently, you can access Prometheus at http://localhost:9090.

If you are using the Sourcegraph Kubernetes Cluster, port-forward the Prometheus service:

kubectl port-forward svc/prometheus 9090:30090

Configuration

Sourcegraph runs a slightly customized image of Prometheus, which packages a standard Prometheus installation together with rules files and target files tailored to Sourcegraph.

A directory can be mounted at /sg_prometheus_add_ons. It can contain additional config files of two types:

• rule files which must have the suffix _rules.yml in their filename (ie gitserver_rules.yml)
• target files which must have the suffix _targets.yml in their filename (ie local_targets.yml)

Rule files and target files must use the latest Prometheus 2.x syntax.

The environment variable PROMETHEUS_ADDITIONAL_FLAGS can be used to pass on additional flags to the prometheus executable running in the container.

Grafana

Site admins can view the monitoring dashboards on a Sourcegraph instance:

1. Go to User menu > Site admin.
2. Open the Monitoring page (left sidebar). The URL is https://sourcegraph.example.com/-/debug/grafana/?orgId=1.
3. Read the Sourcegraph Grafana dashboard descriptions before exploring the dashboards.

NOTE: There is a known issue where attempting to edit a dashboard will result in a 403 response with "invalid CSRF token". As a workaround, site admins can connect to Grafana directly (described below) to edit the dashboards.

Accessing Grafana directly

Follow the instructions below to access Grafana directly, and add, modify and delete your own dashboards and panels.

Kubernetes

If you're using the Kubernetes cluster deployment option, you can access Grafana directly using Kubernetes port forwarding to your local machine:

kubectl port-forward svc/grafana 3370:30070

Now visit http://localhost:3370/-/debug/grafana.

Single-container server deployments

For simplicity, Grafana does not require authentication, as the port binding of 3370 is restricted to connections from localhost only.

Therefore, if accessing Grafana locally, the URL will be http://localhost:3370/-/debug/grafana. If Sourcegraph is deployed to a remote server, then access via an SSH tunnel using a tool such as sshuttle is required to establish a secure connection to Grafana. To access the remote server using sshuttle from your local machine:

sshuttle -r user@host 0/0

Then simply visit http://host:3370 in your browser.

Configuration

Sourcegraph runs a slightly customized image of Grafana, which includes a standard Grafana installation initialized with Sourcegraph-specific dashboard definitions.

NOTE: Our Grafana instance runs in anonymous mode with all authentication turned off. Please be careful when exposing it to external traffic.

A directory containing dashboard JSON specifications can be mounted in the Docker container at /sg_grafana_additional_dashboards. Changes to files in that directory will be detected automatically while Grafana is running.

More behavior can be controlled with environmental variables.

FAQ

Can I consume Sourcegraph's Prometheus metrics in my own monitoring system (Datadog, New Relic, etc.)?

It is technically possible to consume all of Sourcegraph's Prometheus metrics in any external monitoring system that supports Prometheus scraping (both Datadog and New Relic support this). With that said, we would advise against it because Sourcegraph is a very complex system and defining all of the thresholds and alerting rules that you will need to ensure Sourcegraph is healthy would be tedious and difficult.

One of the primary benefits of using Sourcegraph's builtin Prometheus and Grafana monitoring is that you get builtin dashboards and alerting thresholds out-of-the-box, and as Sourcegraph's internals change with each update you can rest assured the metrics and information you are monitoring is up-to-date.

What we usually see people do instead is either configure Sourcegraph's Grafana monitoring to send alerts to your own PagerDuty, Slack, email, etc. or sometimes query Sourcegraph's monitoring via HTTP to find out what alerts are firing in order to pipe that into your own custom monitoring solution.