mirror of
https://github.com/sourcegraph/sourcegraph.git
synced 2026-02-06 19:21:50 +00:00
doc/dev: feature CI in quickstart, move development to separate page (#34528)
This commit is contained in:
Robert Lin
GitHub
parent
6385c871b6
commit
a643cdcfac
167
doc/dev/background-information/ci/development.md
Normal file
167
doc/dev/background-information/ci/development.md
Normal file
@ -0,0 +1,167 @@
|
||||
# Continuous integration development
|
||||
|
||||
This document covers information about contributing to [Sourcegraph's continuous integration tools](./index.md).
|
||||
|
||||
## Pipeline generator
|
||||
|
||||
The source code of [Sourcegraph's Buildkite pipelines](./index.md#buildkite-pipelines) generator is in [`/enterprise/dev/ci`](https://sourcegraph.com/github.com/sourcegraph/sourcegraph@main/-/tree/enterprise/dev/ci).
|
||||
Internally, the pipeline generator determines what gets run over contributions based on:
|
||||
|
||||
1. [Run types](#run-types), determined by branch naming conventions, tags, and environment variables
|
||||
2. [Diff types](#diff-types), determined by what files have been changed in a given branch
|
||||
|
||||
The above factors are then used to determine the appropriate [operations](#operations), composed of [step options](#step-options), that translate into steps in the resulting pipeline.
|
||||
|
||||
> WARNING: Sourcegraph's pipeline generator and its generated output are under the [Sourcegraph Enterprise license](https://github.com/sourcegraph/sourcegraph/blob/main/LICENSE.enterprise).
|
||||
|
||||
### Run types
|
||||
|
||||
<div class="embed">
|
||||
<iframe src="https://sourcegraph.com/embed/notebooks/Tm90ZWJvb2s6MTU5"
|
||||
style="width:100%;height:720px" frameborder="0" sandbox="allow-scripts allow-same-origin allow-popups">
|
||||
</iframe>
|
||||
</div>
|
||||
|
||||
### Diff types
|
||||
|
||||
<div class="embed">
|
||||
<iframe src="https://sourcegraph.com/embed/notebooks/Tm90ZWJvb2s6MTYw"
|
||||
style="width:100%;height:720px" frameborder="0" sandbox="allow-scripts allow-same-origin allow-popups">
|
||||
</iframe>
|
||||
</div>
|
||||
|
||||
### Operations
|
||||
|
||||
<div class="embed">
|
||||
<iframe src="https://sourcegraph.com/embed/notebooks/Tm90ZWJvb2s6MTYx"
|
||||
style="width:100%;height:720px" frameborder="0" sandbox="allow-scripts allow-same-origin allow-popups">
|
||||
</iframe>
|
||||
</div>
|
||||
|
||||
#### Developing PR checks
|
||||
|
||||
To create a new check that can run on pull requests on relevant files, refer to how [diff types](#diff-types) work to get started.
|
||||
|
||||
Then, you can add a new check to [`CoreTestOperations`](https://sourcegraph.com/search?q=context:global+repo:%5Egithub%5C.com/sourcegraph/sourcegraph%24+file:%5Eenterprise/dev/ci/internal/ci+CoreTestOperations+type:symbol+&patternType=literal).
|
||||
Make sure to follow the best practices outlined in docstring.
|
||||
|
||||
For more advanced pipelines, see [Run types](#run-types).
|
||||
|
||||
### Step options
|
||||
|
||||
> NOTE: Coming soon!
|
||||
|
||||
#### Creating annotations
|
||||
|
||||
Annotations get rendered in the Buildkite UI to present the viewer notices about the build.
|
||||
The pipeline generator provides an API for this that, at a high level, works like this:
|
||||
|
||||
1. In your script, leave a file in `./annotations`:
|
||||
|
||||
```sh
|
||||
if [ $EXIT_CODE -ne 0 ]; then
|
||||
echo -e "$OUT" >./annotations/docsite
|
||||
fi
|
||||
```
|
||||
|
||||
2. In your pipeline operation, replace the usual `bk.Cmd` with `bk.AnnotatedCmd`:
|
||||
|
||||
```go
|
||||
pipeline.AddStep(":memo: Check and build docsite",
|
||||
bk.AnnotatedCmd("./dev/check/docsite.sh", bk.AnnotatedCmdOpts{
|
||||
Annotations: &bk.AnnotationOpts{},
|
||||
}))
|
||||
```
|
||||
|
||||
3. That's it!
|
||||
|
||||
For more details about best practices and additional features and capabilities, please refer to [the `bk.AnnotatedCmd` docstring](https://sourcegraph.com/search?q=context:global+repo:%5Egithub%5C.com/sourcegraph/sourcegraph%24+file:%5Eenterprise/dev/ci/internal/buildkite+AnnotatedCmd+type:symbol&patternType=literal).
|
||||
|
||||
#### Caching build artefacts
|
||||
|
||||
For caching artefacts in steps to speed up steps, see [How to cache CI artefacts](../../how-to/cache_ci_artefacts.md).
|
||||
|
||||
### Observability
|
||||
|
||||
> NOTE: Sourcegraph teammates should refer to the [CI incidents playbook](https://handbook.sourcegraph.com/departments/product-engineering/engineering/process/incidents/playbooks/ci#scenarios) for help managing issues with [pipeline health](./index.md#pipeline-health).
|
||||
|
||||
#### Pipeline command tracing
|
||||
|
||||
Every successful build of the `sourcegraph/sourcegraph` repository comes with an annotation pointing at the full trace of the build on [Honeycomb.io](https://honeycomb.io).
|
||||
See the [Buildkite board on Honeycomb](https://ui.honeycomb.io/sourcegraph/board/sqPvYj5BXNy/Buildkite) for an overview.
|
||||
|
||||
Individual commands are tracked from the perspective of a given [step](#step-options):
|
||||
|
||||
```go
|
||||
pipeline.AddStep(":memo: Check and build docsite", /* ... */)
|
||||
```
|
||||
|
||||
Will result in a single trace span for the `./dev/check/docsite.sh` script. But the following will have individual trace spans for each `yarn` commands:
|
||||
|
||||
```go
|
||||
pipeline.AddStep(fmt.Sprintf(":%s: Puppeteer tests for %s extension", browser, browser),
|
||||
// ...
|
||||
bk.Cmd("yarn --frozen-lockfile --network-timeout 60000"),
|
||||
bk.Cmd("yarn workspace @sourcegraph/browser -s run build"),
|
||||
bk.Cmd("yarn run cover-browser-integration"),
|
||||
bk.Cmd("yarn nyc report -r json"),
|
||||
bk.Cmd("dev/ci/codecov.sh -c -F typescript -F integration"),
|
||||
```
|
||||
|
||||
Therefore, it's beneficial for tracing purposes to split the step in multiple commands, if possible.
|
||||
|
||||
#### Test analytics
|
||||
|
||||
Our test analytics is currently powered by a Buildkite beta feature for analysing individual tests across builds called [Buildkite Analytics](https://buildkite.com/test-analytics).
|
||||
This tool enables us to observe the evolution of each individual test on the following metrics: duration and flakiness.
|
||||
|
||||
Browse the [dashboard](https://buildkite.com/organizations/sourcegraph/analytics) to explore the metrics and optionally set monitors that will alert if a given test or a test suite is deviating from its historical duration or flakiness.
|
||||
|
||||
In order to track a new test suite, test results must be converted to JUnit XML reports and uploaded to Buildkite.
|
||||
The pipeline generator provides an API for this that, at a high level, works like this:
|
||||
|
||||
1. In your script, leave your JUnit XML test report in `./test-reports`
|
||||
2. [Create a new Test Suite](https://buildkite.com/organizations/sourcegraph/analytics/suites/new) in the Buildkite Analytics UI.
|
||||
3. In your pipeline operation, replace the usual `bk.Cmd` with `bk.AnnotatedCmd`:
|
||||
|
||||
```go
|
||||
pipeline.AddStep(":jest::globe_with_meridians: Test",
|
||||
withYarnCache(),
|
||||
bk.AnnotatedCmd("dev/ci/yarn-test.sh client/web", bk.AnnotatedCmdOpts{
|
||||
TestReports: &bk.TestReportOpts{/* ... */},
|
||||
}),
|
||||
```
|
||||
|
||||
4. That's it!
|
||||
|
||||
For more details about best practices and additional features and capabilities, please refer to [the `bk.AnnotatedCmd` docstring](https://sourcegraph.com/search?q=context:global+repo:%5Egithub%5C.com/sourcegraph/sourcegraph%24+file:%5Eenterprise/dev/ci/internal/buildkite+AnnotatedCmd+type:symbol&patternType=literal).
|
||||
|
||||
> WARNING: The Buildkite API is not finalized and neither are the configuration options for `TestReportOpts`.
|
||||
> To get started with Buildkite Analytics please reach out to the `#dev-experience` channel for assistance.
|
||||
|
||||
### Buildkite infrastructure
|
||||
|
||||
Our continuous integration system is composed of two parts, a central server controled by Buildkite and agents that are operated by Sourcegraph within our own infrastructure.
|
||||
In order to provide strong isolation across builds, to prevent a previous build to create any effect on the next one, our agents are stateless jobs.
|
||||
|
||||
When a build is dispatched by Buildkite, each individual job will be assigned to an agent in a pristine state. Each agent will execute its assigned job, automatically report back to Buildkite and finally shuts itself down. A fresh agent will then be created and will stand in line for the next job.
|
||||
|
||||
This means that our agents are totally **stateless**, exactly like the runners used in GitHub actions.
|
||||
|
||||
Also see [Flaky infrastructure](#flaky-infrastructure), [Continous integration infrastructure](https://handbook.sourcegraph.com/departments/product-engineering/engineering/tools/infrastructure/ci), and the [Continuous integration changelog](https://handbook.sourcegraph.com/departments/product-engineering/engineering/tools/infrastructure/ci/changelog).
|
||||
|
||||
#### Pipeline setup
|
||||
|
||||
To set up Buildkite to use the rendered pipeline, add the following step in the [pipeline settings](https://buildkite.com/sourcegraph/sourcegraph/settings):
|
||||
|
||||
```shell
|
||||
go run ./enterprise/dev/ci/gen-pipeline.go | buildkite-agent pipeline upload
|
||||
```
|
||||
|
||||
#### Managing secrets
|
||||
|
||||
The term _secret_ refers to authentication credentials like passwords, API keys, tokens, etc. which are used to access a particular service. Our CI pipeline must never leak secrets:
|
||||
|
||||
- to add a secret, use the Secret Manager on Google Cloud and then inject it at deployment time as an environment variable in the CI agents, which will make it available to every step.
|
||||
- use an environment variable name with one of the following suffixes to ensure it gets redacted in the logs: `*_PASSWORD, *_SECRET, *_TOKEN, *_ACCESS_KEY, *_SECRET_KEY, *_CREDENTIALS`
|
||||
- while environment variables can be assigned when declaring steps, they should never be used for secrets, because they won't get redacted, even if they match one of the above patterns.
|
||||
@ -2,16 +2,36 @@
|
||||
|
||||
<span class="badge badge-note">SOC2/GN-105</span> <span class="badge badge-note">SOC2/GN-106</span>
|
||||
|
||||
Sourcegraph uses a continuous integration and delivery tool, [Buildkite](#buildkite-pipelines), to help ensure a [consistent](#pipeline-health) build, test and deploy process. Software changes are systematically required to complete all steps within the continuous integration tool workflow prior to production deployment, in addition to being [peer reviewed](../pull_request_reviews.md).
|
||||
Sourcegraph uses a continuous integration (CI) and delivery tool, [Buildkite](#buildkite-pipelines), to help ensure a [consistent](#pipeline-health) build, test and deploy process. Software changes are systematically required to complete all steps within the continuous integration tool workflow prior to production deployment, in addition to being [peer reviewed](../pull_request_reviews.md).
|
||||
|
||||
Sourcegraph also maintains a variety of tooling on [GitHub Actions](#github-actions) for continuous integration and repository maintainence purposes.
|
||||
|
||||
> NOTE: To learn more about testing in particular, see our [testing principles](../testing_principles.md).
|
||||
|
||||
<div class="getting-started">
|
||||
<a href="#buildkite-pipelines" class="btn" alt="Buildkite pipelines">
|
||||
<span>★ Buildkite pipelines</span>
|
||||
</br>
|
||||
An introduction to Sourcegraph's Buildkite pipelines.
|
||||
</a>
|
||||
|
||||
<a href="./reference" class="btn" alt="Pipeline references">
|
||||
<span>Pipelines reference</span>
|
||||
</br>
|
||||
A complete Buildkite pipeline reference covering all the different pipeline types.
|
||||
</a>
|
||||
|
||||
<a href="./development" class="btn" alt="Development">
|
||||
<span>Development</span>
|
||||
</br>
|
||||
Contribute changes to Sourcegraph's Buildkite pipelines!
|
||||
</a>
|
||||
</div>
|
||||
|
||||
## Buildkite pipelines
|
||||
|
||||
[Tests](../../how-to/testing.md) are automatically run in our [various Buildkite pipelines](https://buildkite.com/sourcegraph) when you push your changes to GitHub.
|
||||
Pipeline steps are generated using the [pipeline generator](https://sourcegraph.com/github.com/sourcegraph/sourcegraph@main/-/tree/enterprise/dev/ci).
|
||||
[Tests](../../how-to/testing.md) are automatically run in our [various Buildkite pipelines](https://buildkite.com/sourcegraph) when you push your changes (i.e. when you run `git push`) to the `sourcegraph/sourcegraph` GitHub repository.
|
||||
Pipeline steps are generated on the fly using the [pipeline generator](https://sourcegraph.com/github.com/sourcegraph/sourcegraph@main/-/tree/enterprise/dev/ci).
|
||||
|
||||
To see what checks will get run against your current branch, use [`sg`](../../setup/quickstart.md):
|
||||
|
||||
@ -22,9 +42,9 @@ sg ci preview
|
||||
A complete reference of all available pipeline types and steps is available in the generated [Pipeline reference](./reference.md).
|
||||
You can also see these docs locally with `sg ci docs`.
|
||||
|
||||
You can also request builds for your changes for you builds using `sg ci build`.
|
||||
You can also request builds manually for your builds using `sg ci build`.
|
||||
|
||||
To learn about making changes to our Buildkite pipelines, see [Pipeline development](#pipeline-development).
|
||||
To learn about making changes to our Buildkite pipelines, see [Pipeline development](./development.md).
|
||||
|
||||
### Pipeline steps
|
||||
|
||||
@ -138,171 +158,13 @@ Also see [Buildkite infrastructure](#buildkite-infrastructure).
|
||||
|
||||
### Pipeline development
|
||||
|
||||
The source code of the pipeline generator is in [`/enterprise/dev/ci`](https://sourcegraph.com/github.com/sourcegraph/sourcegraph@main/-/tree/enterprise/dev/ci).
|
||||
Internally, the pipeline generator determines what gets run over contributions based on:
|
||||
|
||||
1. [Run types](#run-types), determined by branch naming conventions, tags, and environment variables
|
||||
2. [Diff types](#diff-types), determined by what files have been changed in a given branch
|
||||
|
||||
The above factors are then used to determine the appropriate [operations](#operations), composed of [step options](#step-options), that translate into steps in the resulting pipeline.
|
||||
|
||||
> WARNING: Sourcegraph's pipeline generator and its generated output are under the [Sourcegraph Enterprise license](https://github.com/sourcegraph/sourcegraph/blob/main/LICENSE.enterprise).
|
||||
|
||||
#### Run types
|
||||
|
||||
<div class="embed">
|
||||
<iframe src="https://sourcegraph.com/embed/notebooks/Tm90ZWJvb2s6MTU5"
|
||||
style="width:100%;height:720px" frameborder="0" sandbox="allow-scripts allow-same-origin allow-popups">
|
||||
</iframe>
|
||||
</div>
|
||||
|
||||
#### Diff types
|
||||
|
||||
<div class="embed">
|
||||
<iframe src="https://sourcegraph.com/embed/notebooks/Tm90ZWJvb2s6MTYw"
|
||||
style="width:100%;height:720px" frameborder="0" sandbox="allow-scripts allow-same-origin allow-popups">
|
||||
</iframe>
|
||||
</div>
|
||||
|
||||
#### Operations
|
||||
|
||||
<div class="embed">
|
||||
<iframe src="https://sourcegraph.com/embed/notebooks/Tm90ZWJvb2s6MTYx"
|
||||
style="width:100%;height:720px" frameborder="0" sandbox="allow-scripts allow-same-origin allow-popups">
|
||||
</iframe>
|
||||
</div>
|
||||
|
||||
##### Developing PR checks
|
||||
|
||||
To create a new check that can run on pull requests on relevant files, refer to how [diff types](#diff-types) work to get started.
|
||||
|
||||
Then, you can add a new check to [`CoreTestOperations`](https://sourcegraph.com/search?q=context:global+repo:%5Egithub%5C.com/sourcegraph/sourcegraph%24+file:%5Eenterprise/dev/ci/internal/ci+CoreTestOperations+type:symbol+&patternType=literal).
|
||||
Make sure to follow the best practices outlined in docstring.
|
||||
|
||||
For more advanced pipelines, see [Run types](#run-types).
|
||||
|
||||
#### Step options
|
||||
|
||||
> NOTE: Coming soon!
|
||||
|
||||
##### Creating annotations
|
||||
|
||||
Annotations get rendered in the Buildkite UI to present the viewer notices about the build.
|
||||
The pipeline generator provides an API for this that, at a high level, works like this:
|
||||
|
||||
1. In your script, leave a file in `./annotations`:
|
||||
|
||||
```sh
|
||||
if [ $EXIT_CODE -ne 0 ]; then
|
||||
echo -e "$OUT" >./annotations/docsite
|
||||
fi
|
||||
```
|
||||
|
||||
2. In your pipeline operation, replace the usual `bk.Cmd` with `bk.AnnotatedCmd`:
|
||||
|
||||
```go
|
||||
pipeline.AddStep(":memo: Check and build docsite",
|
||||
bk.AnnotatedCmd("./dev/check/docsite.sh", bk.AnnotatedCmdOpts{
|
||||
Annotations: &bk.AnnotationOpts{},
|
||||
}))
|
||||
```
|
||||
|
||||
3. That's it!
|
||||
|
||||
For more details about best practices and additional features and capabilities, please refer to [the `bk.AnnotatedCmd` docstring](https://sourcegraph.com/search?q=context:global+repo:%5Egithub%5C.com/sourcegraph/sourcegraph%24+file:%5Eenterprise/dev/ci/internal/buildkite+AnnotatedCmd+type:symbol&patternType=literal).
|
||||
|
||||
##### Caching build artefacts
|
||||
|
||||
For caching artefacts in steps to speed up steps, see [How to cache CI artefacts](../../how-to/cache_ci_artefacts.md).
|
||||
|
||||
#### Observability
|
||||
|
||||
##### Pipeline command tracing
|
||||
|
||||
Every successful build of the `sourcegraph/sourcegraph` repository comes with an annotation pointing at the full trace of the build on [Honeycomb.io](https://honeycomb.io).
|
||||
See the [Buildkite board on Honeycomb](https://ui.honeycomb.io/sourcegraph/board/sqPvYj5BXNy/Buildkite) for an overview.
|
||||
|
||||
Individual commands are tracked from the perspective of a given [step](#step-options):
|
||||
|
||||
```go
|
||||
pipeline.AddStep(":memo: Check and build docsite", /* ... */)
|
||||
```
|
||||
|
||||
Will result in a single trace span for the `./dev/check/docsite.sh` script. But the following will have individual trace spans for each `yarn` commands:
|
||||
|
||||
```go
|
||||
pipeline.AddStep(fmt.Sprintf(":%s: Puppeteer tests for %s extension", browser, browser),
|
||||
// ...
|
||||
bk.Cmd("yarn --frozen-lockfile --network-timeout 60000"),
|
||||
bk.Cmd("yarn workspace @sourcegraph/browser -s run build"),
|
||||
bk.Cmd("yarn run cover-browser-integration"),
|
||||
bk.Cmd("yarn nyc report -r json"),
|
||||
bk.Cmd("dev/ci/codecov.sh -c -F typescript -F integration"),
|
||||
```
|
||||
|
||||
Therefore, it's beneficial for tracing purposes to split the step in multiple commands, if possible.
|
||||
|
||||
##### Test analytics
|
||||
|
||||
Our test analytics is currently powered by a Buildkite beta feature for analysing individual tests across builds called [Buildkite Analytics](https://buildkite.com/test-analytics).
|
||||
This tool enables us to observe the evolution of each individual test on the following metrics: duration and flakiness.
|
||||
|
||||
Browse the [dashboard](https://buildkite.com/organizations/sourcegraph/analytics) to explore the metrics and optionally set monitors that will alert if a given test or a test suite is deviating from its historical duration or flakiness.
|
||||
|
||||
In order to track a new test suite, test results must be converted to JUnit XML reports and uploaded to Buildkite.
|
||||
The pipeline generator provides an API for this that, at a high level, works like this:
|
||||
|
||||
1. In your script, leave your JUnit XML test report in `./test-reports`
|
||||
2. [Create a new Test Suite](https://buildkite.com/organizations/sourcegraph/analytics/suites/new) in the Buildkite Analytics UI.
|
||||
3. In your pipeline operation, replace the usual `bk.Cmd` with `bk.AnnotatedCmd`:
|
||||
|
||||
```go
|
||||
pipeline.AddStep(":jest::globe_with_meridians: Test",
|
||||
withYarnCache(),
|
||||
bk.AnnotatedCmd("dev/ci/yarn-test.sh client/web", bk.AnnotatedCmdOpts{
|
||||
TestReports: &bk.TestReportOpts{/* ... */},
|
||||
}),
|
||||
```
|
||||
|
||||
4. That's it!
|
||||
|
||||
For more details about best practices and additional features and capabilities, please refer to [the `bk.AnnotatedCmd` docstring](https://sourcegraph.com/search?q=context:global+repo:%5Egithub%5C.com/sourcegraph/sourcegraph%24+file:%5Eenterprise/dev/ci/internal/buildkite+AnnotatedCmd+type:symbol&patternType=literal).
|
||||
|
||||
> WARNING: The Buildkite API is not finalized and neither are the configuration options for `TestReportOpts`.
|
||||
> To get started with Buildkite Analytics please reach out to the `#dev-experience` channel for assistance.
|
||||
|
||||
### Buildkite infrastructure
|
||||
|
||||
Our continuous integration system is composed of two parts, a central server controled by Buildkite and agents that are operated by Sourcegraph within our own infrastructure.
|
||||
In order to provide strong isolation across builds, to prevent a previous build to create any effect on the next one, our agents are stateless jobs.
|
||||
|
||||
When a build is dispatched by Buildkite, each individual job will be assigned to an agent in a pristine state. Each agent will execute its assigned job, automatically report back to Buildkite and finally shuts itself down. A fresh agent will then be created and will stand in line for the next job.
|
||||
|
||||
This means that our agents are totally **stateless**, exactly like the runners used in GitHub actions.
|
||||
|
||||
Also see [Flaky infrastructure](#flaky-infrastructure), [Continous integration infrastructure](https://handbook.sourcegraph.com/departments/product-engineering/engineering/tools/infrastructure/ci), and the [Continuous integration changelog](https://handbook.sourcegraph.com/departments/product-engineering/engineering/tools/infrastructure/ci/changelog).
|
||||
|
||||
#### Pipeline setup
|
||||
|
||||
To set up Buildkite to use the rendered pipeline, add the following step in the [pipeline settings](https://buildkite.com/sourcegraph/sourcegraph/settings):
|
||||
|
||||
```shell
|
||||
go run ./enterprise/dev/ci/gen-pipeline.go | buildkite-agent pipeline upload
|
||||
```
|
||||
|
||||
#### Managing secrets
|
||||
|
||||
The term _secret_ refers to authentication credentials like passwords, API keys, tokens, etc. which are used to access a particular service. Our CI pipeline must never leak secrets:
|
||||
|
||||
- to add a secret, use the Secret Manager on Google Cloud and then inject it at deployment time as an environment variable in the CI agents, which will make it available to every step.
|
||||
- use an environment variable name with one of the following suffixes to ensure it gets redacted in the logs: `*_PASSWORD, *_SECRET, *_TOKEN, *_ACCESS_KEY, *_SECRET_KEY, *_CREDENTIALS`
|
||||
- while environment variables can be assigned when declaring steps, they should never be used for secrets, because they won't get redacted, even if they match one of the above patterns.
|
||||
See [Pipeline development](./development.md) to get started with contributing to our Buildkite pipelines!
|
||||
|
||||
## GitHub Actions
|
||||
|
||||
### `buildchecker`
|
||||
|
||||
[](https://github.com/sourcegraph/sourcegraph/actions/workflows/buildchecker.yml)
|
||||
[](https://github.com/sourcegraph/sourcegraph/actions/workflows/buildchecker.yml) [](https://github.com/sourcegraph/sourcegraph/actions/workflows/buildchecker-history.yml)
|
||||
|
||||
[`buildchecker`](https://github.com/sourcegraph/sourcegraph/actions/workflows/buildchecker.yml), our [branch lock management tool](#branch-locks), runs in GitHub actions - see the [workflow specification](https://github.com/sourcegraph/sourcegraph/blob/main/.github/workflows/buildchecker.yml).
|
||||
|
||||
|
||||
@ -2,11 +2,11 @@
|
||||
|
||||
This guide documents how to cache build artefacts in order to speed up build times in Sourcegraph's [Buildkite CI pipelines](../background-information/ci/index.md#buildkite-pipelines).
|
||||
|
||||
> NOTE: Before getting started, we recommend familiarize yourself with [Pipeline development](../background-information/ci/index.md#pipeline-development) and [Buildkite infrastructure](../background-information/ci/index.md#buildkite-infrastructure).
|
||||
> NOTE: Before getting started, we recommend familiarize yourself with [Pipeline development](../background-information/ci/development.md) and [Buildkite infrastructure](../background-information/ci/development.md#buildkite-infrastructure).
|
||||
|
||||
## The need for caching
|
||||
|
||||
Because [Buildkite agents are stateless](../background-information/ci/index.md#buildkite-infrastructure) and start with a blank slate, this means that all dependencies and binaries have to rebuild on each job. This is the price to pay for complete isolation from one job to the other.
|
||||
Because [Buildkite agents are stateless](../background-information/ci/development.md#buildkite-infrastructure) and start with a blank slate, this means that all dependencies and binaries have to rebuild on each job. This is the price to pay for complete isolation from one job to the other.
|
||||
|
||||
A common strategy to address this problem of having to rebuild everything is to store objects that are commonly reused accross jobs and to download them again rather than rebuilding everything from scratch.
|
||||
|
||||
@ -22,7 +22,7 @@ In order to determine what we can cache and when to do it, we need to make sure
|
||||
|
||||
## How to write a step that caches an artefact?
|
||||
|
||||
In the [CI pipeline generator](../background-information/ci/index.md#pipeline-development), when defining a step you can use the `buildkite.Cache()` function to define what needs to be cached and under which key to store it.
|
||||
In the [CI pipeline generator](../background-information/ci/development.md), when defining a step you can use the `buildkite.Cache()` function to define what needs to be cached and under which key to store it.
|
||||
|
||||
For example: we want to cache the `node_modules` folder to avoid dowloading again all dependencies for the front-end.
|
||||
|
||||
|
||||
@ -98,4 +98,5 @@ sg start monitoring
|
||||
Here are some additional resources to help you go further:
|
||||
|
||||
- [Troubleshooting local development](troubleshooting.md)
|
||||
- [Continuous integration](../background-information/ci/index.md)
|
||||
- [Background information](../background-information/index.md) for more context on various topics.
|
||||
|
||||
Loading…
Reference in New Issue
Block a user