artin/sourcegraph
1
0
Fork 0
mirror of https://github.com/sourcegraph/sourcegraph.git synced 2026-02-06 17:51:57 +00:00
Code Issues Packages Projects Releases Wiki Activity

doc: Add Flaky Test bug template, and link to it from the doc. (#21236)

Browse Source 
- Add a GitHub issue template to capture information about flaky tests that have been disabled.
- Link to the template from the "Flaky test" section of the Testing Principles page.
This commit is contained in:
Patrick Dubroy 2021-05-21 14:25:16 +02:00 committed by GitHub
parent 128483c3d6
commit 5798547834
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
2 changed files with 20 additions and 2 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

18
.github/ISSUE_TEMPLATE/flaky_test.md vendored Normal file
View File

@ -0,0 +1,18 @@
---
name: Flaky Test
about: Capture information about a flaky test that has been disabled.
title: 'Flake: $TEST_NAME disabled'
labels:
- 'testing'
- 'flake'
assignees: ''
---
- **Name of test:** <!-- Name of the test that was disabled -->
- **Example failure:** <!-- Buildkite link to an example faiure -->
- **PR**: <!-- Link to PR that disabled the test. E.g., #1234 >
#### Additional details
<!-- Notes and/or screenshot describing the problem -->

4
doc/dev/background-information/testing_principles.md
View File

@ -19,7 +19,7 @@ Engineers should budget an appropriate amount of time for writing tests when mak
## Flaky tests
A *flaky* test is defined as a test that is unreliable or non-deterministic, meaning that it doesn't consistently produce the same pass/fail result.
A *flaky* test is defined as a test that is unreliable or non-deterministic, i.e. it exhibits both a passing and a failing result with the same code.
Typical reasons why a test may be flaky:
@ -31,7 +31,7 @@ Typical reasons why a test may be flaky:
We do not tolerate flaky tests of any kind. Any engineer that sees a flaky test should immediately:
1. Open a PR to disable the flaky test.
1. Open an issue to re-enable the flaky test, assign it to the most likely owner, and add it to the current release milestone.
1. Open an issue to re-enable the flaky test (use the [Flaky Test template](https://github.com/sourcegraph/sourcegraph/issues/new?assignees=&labels=&template=flaky_test.md&title=Flake%3A+%24TEST_NAME+disabled)), and assign it to the most likely owner, and add it to the current release milestone.
If the build or test infrastructure itself is flaky, then [open an issue](https://github.com/sourcegraph/sourcegraph/issues/new?labels=team/distribution) and notify the [distribution team](https://about.sourcegraph.com/handbook/engineering/distribution#contact).