For Executors on Native Kubernetes deployments, the option to run jobs
in a single pod has been available since Native Kubernetes has been
around.
The purpose of running jobs in a single pod is:
1. Efficiency. Jobs require three steps at least, and without specifying
a single pod, that requires spinning up three pods.
2. Security. For Batch Changes, when jobs are run across several pods,
`git`'s `safe.directory` must be set to avoid untrusted users or
processes injecting code or an attack. Running the job in one pod
removes the need for `safe.directory`.
3. Usability. Because of the need to set `safe.directory`, `root` access
to write to `git`'s global config is required, which means that many
times special configurations and sign-offs from security teams must be
used for Batch Change setups.
This PR takes a step toward using single pod jobs only in enabling them
by default instead of requiring an environment variable to enable them.
The same environment variable that was used to enable them -
`KUBERNETES_SINGLE_JOB_POD` - is still available to disable them by
setting it to `false`.
## Test plan
Bazel and CI for now
## Changelog
This change extracts the unrelated transitive upgrades of
https://github.com/sourcegraph/sourcegraph/pull/63171 (CORE-177) into a
separate PR. I'm making this because @unknwon ran into issues with the
exact same dependencies in
https://github.com/sourcegraph/sourcegraph/pull/63171#issuecomment-2157694545.
The change consists of upgrades to:
- `google.golang.org/grpc` - there's a deprecation of `grpc.DialContext`
that we agreed in #63171 to keep for now.
- removing our `replace` directive on `github.com/prometheus/common` and
upgrading it. This is safe to do because our Alertmanager version is
already way ahead, and the reason this has a `replace` is outdated now.
## Test plan
CI, nothing blows up on `sg start` and I can click around and do a bit
of searching
With this patch, the `errors.HasType` API behaves similar to `Is` and `As`,
where it checks the full error tree instead of just checking a linearized version
of it, as cockroachdb/errors's `HasType` implementation does not respect
multi-errors.
As a consequence, a bunch of relationships between HasType and Is/As that
you'd intuitively expect to hold are now true; see changes to `invariants_test.go`.
This PR is a result/followup of the improvements we've made in the [SAMS repo](https://github.com/sourcegraph/sourcegraph-accounts/pull/199) that allows call sites to pass down a context (primarily to indicate deadline, and of course, cancellation if desired) and collects the error returned from `background.Routine`s `Stop` method.
Note that I did not adopt returning error from `Stop` method because I realize in monorepo, the more common (and arguably the desired) pattern is to hang on the call of `Start` method until `Stop` is called, so it is meaningless to collect errors from `Start` methods as return values anyway, and doing that would also complicate the design and semantics more than necessary.
All usages of the the `background.Routine` and `background.CombinedRoutines` are updated, I DID NOT try to interpret the code logic and make anything better other than fixing compile and test errors.
The only file that contains the core change is the [`lib/background/background.go`](https://github.com/sourcegraph/sourcegraph/pull/62136/files#diff-65c3228388620e91f8c22d91c18faac3f985fc67d64b08612df18fa7c04fafcd).
This fixes an issue that is generating failures on repos with spaces in their names.
Previously, we were passing the script name as a shell command with -c. However, that means it's subject to shell escaping. If we instead pass the file as a script for sh to run, it avoids the need to escape the file name and also reduces the surface area for injection attacks.
* k8s: update deps and fix breaks
* appliance: Add internal spec of config
Add an internal spec of Sourcegraph to be used for user config and state
in the appliance.
* cmd/appliance: Add boilerplate and stub service
* Fix the bazel deps
* fix missing err returns
* Use 'MainWithoutConfig
* Add readme with basic info
observation.TestContextTB is better to use since your logs will be
scoped to your test and it will use a more pedantic prometheus registry.
To be honest TestContext should be removed but this is the first step.
This is a mechanical change. I replaced "&observation.TestContext" with
"observation.TestContextTB(t)". I then undid the change each time it
caused a compilation error (was only a handful of times).
Test Plan: go test
Now that we've updated to Go 1.22, we don't need to copy loop variables before
using them in goroutines.
I found these using the regex searches `go func\(\w+` and `\.Go(func\(\w+`. I
also simplified some non-loop vars when it made sense.
## Test plan
Straight refactor, covered by existing tests
Adds a new:
- gazelle generator
- rule + rule targets + catchall target
for generating go-mockgen mocks & testing for their being up-to-date.
Each go_mockgen macro invocation adds targets for generating mocks, copying to the source tree, as well as testing whether the current source tree mocks are up-to-date.
How to use this: `bazel run //dev:go_mockgen` for the catch-all, or `bazel run //some/target:generate_mocks` for an individual package, and `bazel test //some/target:generate_mocks_tests` to test for up-to-date-ness. There is no catch-all for testing
This currently uses a fork of go-mockgen, with an open PR for upstream here: https://github.com/derision-test/go-mockgen/pull/50.
Closes https://github.com/sourcegraph/sourcegraph/issues/60099
## Test plan
Extensive testing during development, including the following cases:
- Deleting a generated file and its entry in a go_library/go_test `srcs` attribute list and then re-running `sg bazel configure`
- Adding a non-existent output directory to mockgen.test.yaml and running the bash one-liner emitted to prepare the workspace for rerunning `sg bazel configure`
The existing config tests a lot of existing paths anyway (creating mocks for a 3rd party library's interface, entries for a given output file in >1 config file etc)
* executor: Add audit log mode
Adds a mode on request of a customer that logs ALL the things the executor does.
Essentially, we're dumping the whole job payload, which contains all the relevant information to be able to fully replicate what users did.
Here's an example:
```
[batches-exe...r] WARN executor_processor.Handle worker/handler.go:98 Received new job to process {"handle": {"jobID": 5, "repositoryName": "github.com/k3s-io/k3s", "commit": "6d77b7a9204ebe40c53425ce4bc82c1df456e911", "jobPayload": "{\"version\":2,\"id\":5,\"token\":\"57627701c5480c22b832e361b7e4e84a07803e13\",\"repositoryName\":\"github.com/k3s-io/k3s\",\"repositoryDirectory\":\"repository\",\"commit\":\"6d77b7a9204ebe40c53425ce4bc82c1df456e911\",\"fetchTags\":false,\"shallowClone\":true,\"sparseCheckout\":null,\"files\":{\"input.json\":{\"content\":\"eyJCYXRjaENoYW5nZUF0dHJpYnV0ZXMiOnsiTmFtZSI6InRlc3QtbG9ncyIsIkRlc2NyaXB0aW9uIjoiQWRkIEhlbGxvIFdvcmxkIHRvIFJFQURNRXMifSwicmVwb3NpdG9yeSI6eyJpZCI6IlVtVndiM05wZEc5eWVUb3hNdz09IiwibmFtZSI6ImdpdGh1Yi5jb20vazNzLWlvL2szcyJ9LCJicmFuY2giOnsibmFtZSI6InJlZnMvaGVhZHMvbWFzdGVyIiwidGFyZ2V0Ijp7Im9pZCI6IjZkNzdiN2E5MjA0ZWJlNDBjNTM0MjVjZTRiYzgyYzFkZjQ1NmU5MTEifX0sInBhdGgiOiIiLCJvbmx5RmV0Y2hXb3Jrc3BhY2UiOmZhbHNlLCJzdGVwcyI6W3sicnVuIjoiZWNobyBJIGFtIGV2aWwgfCB0ZWUgLWEgJChmaW5kIC1uYW1lIFJFQURNRS5tZCkiLCJjb250YWluZXIiOiJ1YnVudHU6MTguMDQiLCJlbnYiOnt9fV0sInNlYXJjaFJlc3VsdFBhdGhzIjpbIlJFQURNRS5tZCJdLCJjYWNoZWRTdGVwUmVzdWx0Rm91bmQiOmZhbHNlLCJjYWNoZWRTdGVwUmVzdWx0Ijp7ImNoYW5nZWRGaWxlcyI6eyJtb2RpZmllZCI6bnVsbCwiYWRkZWQiOm51bGwsImRlbGV0ZWQiOm51bGwsInJlbmFtZWQiOm51bGx9LCJzdGRvdXQiOiIiLCJzdGRlcnIiOiIiLCJzdGVwSW5kZXgiOjAsImRpZmYiOiIiLCJvdXRwdXRzIjpudWxsfSwic2tpcHBlZFN0ZXBzIjp7fX0=\",\"modifiedAt\":\"0001-01-01T00:00:00Z\"}},\"dockerSteps\":null,\"cliSteps\":[{\"key\":\"batch-exec\",\"command\":[\"batch\",\"exec\",\"-f\",\"input.json\",\"-repo\",\"repository\",\"-tmp\",\".src-tmp\",\"-binaryDiffs\"],\"dir\":\".\",\"env\":[]}],\"redactedValues\":{},\"dockerAuthConfig\":{}}"}}
```
Where the base64 encoded file content contains (and might get corrupted from redaction) the following _unredacted_ file contents:
```
{
"BatchChangeAttributes": {
"Name": "test-logs",
"Description": "Add Hello World to READMEs"
},
"repository": {
"id": "UmVwb3NpdG9yeToxMw==",
"name": "github.com/k3s-io/k3s"
},
"branch": {
"name": "refs/heads/master",
"target": { "oid": "6d77b7a9204ebe40c53425ce4bc82c1df456e911" }
},
"path": "",
"onlyFetchWorkspace": false,
"steps": [
{
"run": "echo I am evil | tee -a $(find -name README.md)",
"container": "ubuntu:18.04",
"env": {}
}
],
"searchResultPaths": ["README.md"],
"cachedStepResultFound": false,
"cachedStepResult": {
"changedFiles": {
"modified": null,
"added": null,
"deleted": null,
"renamed": null
},
"stdout": "",
"stderr": "",
"stepIndex": 0,
"diff": "",
"outputs": null
},
"skippedSteps": {}
}
```
## Test plan
Manual.
* More structured logging
From https://github.com/sourcegraph/sourcegraph/pull/59170#discussion_r1435025135
## Test plan
Bazel build attempt using smithy-go/ptr: `dev/linters/depguard/depguard.go:7:2: import 'github.com/aws/smithy-go/ptr' is not allowed from list 'Main': use github.com/sourcegraph/sourcegraph/lib/pointers instead (depguard)`
Cody no longer needs it and it is obsolete now!
Since App added a non-insignificant amount of new concepts and alternative code paths, I decided to take some time and remove it from out codebase.
This PR removes ~21k lines of code. If we ever want parts of single binary (app), the redis kv alternatives, or the release pipeline for a native mac app back, we can look back at this PR and revert parts of it, but maintaining 21k lines of code and many code paths for which I had to delete a surprisingly small amount of tests justifies this move for me very well.
Technically, to some extent SG App and Cody App both still existed in the codebase, but we don't distribute either of them anymore, so IMO we shouldn't keep this weight in our code.
So.. here we go.
This should not affect any of the existing deployments, we only remove functionality that was special-cased for app.
* log: remove use of description paramter in Scoped
* temporarily point to sglog branch
* bazel configure + gazelle
* remove additional use of description param
* use latest versions of zoekt,log,mountinfo
* go.mod
support single-program execution
Now, `sg start single-program` starts a single-binary local dev server. This is similar to Cody app, but instead of using a Tauri desktop app UI and limiting to only Cody-related functionality, it runs a full Sourcegraph instance and lets you access it through your web browser. It is useful for local dev because it's less resource-intensive and has faster recompile/relink times than `sg start` (which runs many processes).
We seem to have lost this special build tag somewhere in migrations, causing the bundled-executor to no longer have the shell runtime code in it.
Co-authored-by: davejrt <davetry@gmail.com>
This is a mechanical move to get the executor out of the enterprise/cmd
directory. Eventually, this directory should disappear, this is another
step towards that.
This does not change anything about how it's licensed.
## Test plan
CI is still passing, local executor starts up.
