mirror of https://github.com/sourcegraph/sourcegraph.git synced 2026-02-06 18:31:54 +00:00

History

Noah S-C 67f30a9d7a chore(ci): rework build-tracker to use redis instead of in-memory store of build results (#64304 ) Currently, build-tracker keeps track of consecutive build failures through an in-memory store of failed builds. As this gets deployed more frequently on MSP, we lose state more frequently which would result in incorrect results. Instead, we can use redis as our external store as well as for locking using redsync ## Test plan Unit tests have been updated, but proper testing will require live traffic ## Changelog <!-- OPTIONAL; info at https://www.notion.so/sourcegraph/Writing-a-changelog-entry-dd997f411d524caabf0d8d38a24a878c -->		2024-08-13 10:32:09 +00:00
..
build	chore(ci): rework build-tracker to use redis instead of in-memory store of build results (#64304 )	2024-08-13 10:32:09 +00:00
config	bazel: transcribe test ownership to bazel tags (#62664 )	2024-05-16 15:51:16 +01:00
notify	build-tracker: remove old links (#63065 )	2024-06-04 12:03:58 +01:00
.gitignore	feat(build-tracker): Adds a build-tracking service to provide better notifications about build failures (#39355 )	2022-08-02 17:25:39 +00:00
background.go	chore(ci): rework build-tracker to use redis instead of in-memory store of build results (#64304 )	2024-08-13 10:32:09 +00:00
bigquery.go	build-tracker: include timestamp in agent state change events (#62670 )	2024-05-14 18:03:17 +00:00
BUILD.bazel	chore(ci): rework build-tracker to use redis instead of in-memory store of build results (#64304 )	2024-08-13 10:32:09 +00:00
image_test.yaml	build-tracker: deploy to MSP (#61510 )	2024-04-03 11:49:57 +01:00
integration_test.go	chore(ci): rework build-tracker to use redis instead of in-memory store of build results (#64304 )	2024-08-13 10:32:09 +00:00
main_test.go	ci: use build creator when the build is a release build (#61545 )	2024-04-03 16:52:41 +02:00
main.go	chore(ci): rework build-tracker to use redis instead of in-memory store of build results (#64304 )	2024-08-13 10:32:09 +00:00
mocks_test.go	build-tracker: emit agent state-change webhook events to BigQuery (#62598 )	2024-05-12 16:20:04 +02:00
README.md	chore(ci): rework build-tracker to use redis instead of in-memory store of build results (#64304 )	2024-08-13 10:32:09 +00:00
server_test.go	chore(ci): rework build-tracker to use redis instead of in-memory store of build results (#64304 )	2024-08-13 10:32:09 +00:00
util_test.go	chore(ci): rework build-tracker to use redis instead of in-memory store of build results (#64304 )	2024-08-13 10:32:09 +00:00

README.md

BUILD TRACKER

Build Tracker is a server that listens for build events from Buildkite, stores them in Redis and sends notifications about builds if they've failed.

The server currently listens for two events:

build.finished
job.finished

For each job.finished event that is received, the corresponding build is updated with the job that has finished. On receipt of a build.finished event, the server will determine if the build has failed by going through all the contained jobs of the build. If one or more jobs have indeed failed, a notification will be sent over slack. As well as this, the server will trigger a Buildkite job to process CI and Bazel data for the build for analytics purposes.

Deployment infrastructure

Build Tracker is deployed in MSP. See the auto-generated Notion doc for details around accessing the environment and observability systems.

It is fine to wipe Redis if there are any issues stemming from data inconsistencies, redsync lock problems etc.

Notification testing

To test the notifications that get sent over slack you can pass the flag -RunSlackIntegrationTest as part of your test invocation, with some required configuration:

export SLACK_TOKEN='my valid token'
export BUILDKITE_WEBHOOK_TOKEN='optional'
export GITHUB_TOKEN='optional'
go test . -RunSlackIntegrationTest

You can enable Slack client debugging by exporting the following environment variable BUILD_TRACKER_SLACK_DEBUG=1