20b858f6c3
Closes [DINF-51](https://linear.app/sourcegraph/issue/DINF-51/failed-back-compat-doesnt-count-towards-branch-locking-quota) ## Context If a back-compat step on main fails, the build is marked as having failed. However, we don't treat that as a failure in build-tracker, resulting in no #buildkite-main post and not counting towards failed build quota for locking main. The reason why this was happening is that the Backcompat build wasn't linked to the main Sourcegraph build in anyway. However, when a backcompat fails the main build reflects the status of this failure, but we do not use this field when determining the status of a build, so it doesn't work for our use case.  We [instead do a walkthrough of all the jobs associated with a build to figure out](https://sourcegraph.sourcegraph.com/github.com/sourcegraph/sourcegraph/-/blob/dev/build-tracker/main.go?L349-372) if the build has failed, fixed or is passing. With this logic, it means we have to link the steps from child builds that a particular build triggers to it's parent. ## Test plan * Create a build that'll have backcompat failing * The build tracker event associated with the main build will be reported with a state of failed to buildkite.  ## Changelog <!-- OPTIONAL; info at https://www.notion.so/sourcegraph/Writing-a-changelog-entry-dd997f411d524caabf0d8d38a24a878c --> |
||
|---|---|---|
| .. | ||
| testdata/TestRepoBranchLocker | ||
| branch_test.go | ||
| branch.go | ||
| BUILD.bazel | ||
| checker_test.go | ||
| checker.go | ||
| failures_test.go | ||
| failures.go | ||
| main.go | ||
| OWNERS | ||
| README.md | ||
| run-check.sh | ||
| slack_test.go | ||
| slack.go | ||
buildchecker 
buildchecker is designed to respond to periods of consecutive build failures on a Buildkite pipeline.
Owned by the DevInfra team.
More documentation for Sourcegraph teammates is available in theCI incidents playbook.
Buildchecker is deployed as a GitHub Action.
Usage
Available commands:
Check
Checks for a series of build failures that exceed the configured threshold, locks the target branch, and posts various updates to Slack.
go run ./dev/buildchecker/ check # directly
./dev/buildchecker/run-check.sh # using wrapper script
Also see the buildchecker GitHub Action workflow where buildchecker check is run on an automated basis.
History
Writes aggregated historical data, including the builds it finds, to a few files.
go run ./dev/buildchecker \
-buildkite.token=$BUILDKITE_TOKEN \
-builds.write-to=".tmp/builds.json" \
-csv.write-to=".tmp/" \
-failures.timeout=999 \
-created.from="2021-08-01" \
history
To load builds from a file instead of fetching from Buildkite, use -builds.load-from="$FILE".
You can also send metrics to Honeycomb with -honeycomb.dataset and -honeycomb.token:
go run ./dev/buildchecker \
-builds.load-from=".tmp/builds.json" \
-failures.timeout=999 \
-created.from="2021-08-01" \
-honeycomb.dataset="buildkite-history" \
-honeycomb.token=$HONEYCOMB_TOKEN \
history
Tokens
Buildkite API token
Required for all buildchecker commands, except for buildchecker history -load-from.
- Go over your personal settings
- Create a new token with the following permissions:
- check
sourcegraphorganization read_buildsread_pipelines
Development
branch_test.gocontains integration tests against the GitHub API. Normally runs against recordings intestdata- to updatetestdata, run the tests with the-updateflag.- All other tests are strictly unit tests.