c9e903ed39
* executor: Add audit log mode
Adds a mode on request of a customer that logs ALL the things the executor does.
Essentially, we're dumping the whole job payload, which contains all the relevant information to be able to fully replicate what users did.
Here's an example:
```
[batches-exe...r] WARN executor_processor.Handle worker/handler.go:98 Received new job to process {"handle": {"jobID": 5, "repositoryName": "github.com/k3s-io/k3s", "commit": "6d77b7a9204ebe40c53425ce4bc82c1df456e911", "jobPayload": "{\"version\":2,\"id\":5,\"token\":\"57627701c5480c22b832e361b7e4e84a07803e13\",\"repositoryName\":\"github.com/k3s-io/k3s\",\"repositoryDirectory\":\"repository\",\"commit\":\"6d77b7a9204ebe40c53425ce4bc82c1df456e911\",\"fetchTags\":false,\"shallowClone\":true,\"sparseCheckout\":null,\"files\":{\"input.json\":{\"content\":\"eyJCYXRjaENoYW5nZUF0dHJpYnV0ZXMiOnsiTmFtZSI6InRlc3QtbG9ncyIsIkRlc2NyaXB0aW9uIjoiQWRkIEhlbGxvIFdvcmxkIHRvIFJFQURNRXMifSwicmVwb3NpdG9yeSI6eyJpZCI6IlVtVndiM05wZEc5eWVUb3hNdz09IiwibmFtZSI6ImdpdGh1Yi5jb20vazNzLWlvL2szcyJ9LCJicmFuY2giOnsibmFtZSI6InJlZnMvaGVhZHMvbWFzdGVyIiwidGFyZ2V0Ijp7Im9pZCI6IjZkNzdiN2E5MjA0ZWJlNDBjNTM0MjVjZTRiYzgyYzFkZjQ1NmU5MTEifX0sInBhdGgiOiIiLCJvbmx5RmV0Y2hXb3Jrc3BhY2UiOmZhbHNlLCJzdGVwcyI6W3sicnVuIjoiZWNobyBJIGFtIGV2aWwgfCB0ZWUgLWEgJChmaW5kIC1uYW1lIFJFQURNRS5tZCkiLCJjb250YWluZXIiOiJ1YnVudHU6MTguMDQiLCJlbnYiOnt9fV0sInNlYXJjaFJlc3VsdFBhdGhzIjpbIlJFQURNRS5tZCJdLCJjYWNoZWRTdGVwUmVzdWx0Rm91bmQiOmZhbHNlLCJjYWNoZWRTdGVwUmVzdWx0Ijp7ImNoYW5nZWRGaWxlcyI6eyJtb2RpZmllZCI6bnVsbCwiYWRkZWQiOm51bGwsImRlbGV0ZWQiOm51bGwsInJlbmFtZWQiOm51bGx9LCJzdGRvdXQiOiIiLCJzdGRlcnIiOiIiLCJzdGVwSW5kZXgiOjAsImRpZmYiOiIiLCJvdXRwdXRzIjpudWxsfSwic2tpcHBlZFN0ZXBzIjp7fX0=\",\"modifiedAt\":\"0001-01-01T00:00:00Z\"}},\"dockerSteps\":null,\"cliSteps\":[{\"key\":\"batch-exec\",\"command\":[\"batch\",\"exec\",\"-f\",\"input.json\",\"-repo\",\"repository\",\"-tmp\",\".src-tmp\",\"-binaryDiffs\"],\"dir\":\".\",\"env\":[]}],\"redactedValues\":{},\"dockerAuthConfig\":{}}"}}
```
Where the base64 encoded file content contains (and might get corrupted from redaction) the following _unredacted_ file contents:
```
{
"BatchChangeAttributes": {
"Name": "test-logs",
"Description": "Add Hello World to READMEs"
},
"repository": {
"id": "UmVwb3NpdG9yeToxMw==",
"name": "github.com/k3s-io/k3s"
},
"branch": {
"name": "refs/heads/master",
"target": { "oid": "6d77b7a9204ebe40c53425ce4bc82c1df456e911" }
},
"path": "",
"onlyFetchWorkspace": false,
"steps": [
{
"run": "echo I am evil | tee -a $(find -name README.md)",
"container": "ubuntu:18.04",
"env": {}
}
],
"searchResultPaths": ["README.md"],
"cachedStepResultFound": false,
"cachedStepResult": {
"changedFiles": {
"modified": null,
"added": null,
"deleted": null,
"renamed": null
},
"stdout": "",
"stderr": "",
"stepIndex": 0,
"diff": "",
"outputs": null
},
"skippedSteps": {}
}
```
## Test plan
Manual.
* More structured logging
|
||
|---|---|---|
| .. | ||
| docker-mirror | ||
| internal | ||
| kubernetes | ||
| vm-image | ||
| _binary.push.sh | ||
| BUILD.bazel | ||
| ci-should-rebuild.sh | ||
| image_test.yaml | ||
| main.go | ||
| README.md | ||
Executor
The executor service polls the public frontend API for work to perform. The executor will pull a job from a particular queue (configured via the envvar EXECUTOR_QUEUE_NAME), then performs the job by running a sequence of docker and src-cli commands. This service is horizontally scalable.
Since executors and Sourcegraph are separate deployments, our agreement is to support 1 minor version divergence for now. See this example for more details:
| Sourcegraph version | Executor version | Ok |
|---|---|---|
| 3.43.0 | 3.43.* | ✅ |
| 3.43.3 | 3.43.* | ✅ |
| 3.43.0 | 3.44.* | ✅ |
| 3.43.0 | 3.42.* | ✅ |
| 3.43.0 | 3.41.* | 🚫 |
| 3.43.0 | 3.45.* | 🚫 |
See the executor queue for a complete list of queues.
Building and releasing
Building and releasing is handled automatically by the CI pipeline.
Binary
The executor binary is simply built with bazel build //cmd/executor:executor.
For publishing it, see bazel run //cmd/executor:binary.push:
- In every scenario, the binary will be uploaded to
gcs://sourcegraph-artifacts/executors/$GIT_COMMIT/. - If the current branch is
mainwhen this target is run, it will also be copied over togcs://sourcegraph-artifacts/executors/latest. - If the env var
EXECUTOR_IS_TAGGED_RELEASEis set to true, it will also be copied over togcs://sourcegraph-artifacts/executors/$BUILDKITE_TAG.
VM image
The VM Image is built with packer, but it also uses an OCI image as a base for Firecracker, //docker-images/executor-vm:image_tarball which it depends on. That OCI image is a legacy image, see docker-images/executor-vm/README.md.
Because we're producing an AMI for both AWS and GCP, there are two steps involved:
bazel run //cmd/executor/vm-image:ami.buildcreates the AMI and names it according to the CI runtype.bazel run //cmd/executor/vm-image:ami.pushtakes the AMIs from above and publish them (adjust perms, naming).
While gcloud is provided by Bazel, AWS cli is expected to be available on the host running Bazel.
Building AMIs on GCP is rather quick, but it's notoriously slow on AWS (about 20m) so we use target-determinator to detect when to rebuild the image. See ci-should-rebuild.sh, which is used by the pipeline generator to skip building it if we detect that nothing changed since the parent commit.
Docker Mirror
As with the VM image, we're producing an AMI for both AWS and GCP, there are two steps involved:
bazel run //cmd/executor/docker-mirror:ami.buildcreates the AMI and names it according to the CI runtype.bazel run //cmd/executor/docker-mirror:ami.pushtakes the AMIs from above and publish them (adjust perms, naming).
While gcloud is provided by Bazel, AWS cli is expected to be available on the host running Bazel.