OpenTelemetry versioning scheme is a bit convoluted but this upgrades
all OpenTelemetry-related packages together to around v1.16 (for Go SDK
packages) and v0.81 (for Collector packages, and the Collector itself) -
thankfully there were fewer breaking changes this time around. Big diff
is almost entirely `deps.bzl`
One of the more significant changes is we now have access to the
[finalized OpenTelemetry Metrics
API](https://github.com/open-telemetry/opentelemetry-go/releases/tag/v1.16.0),
which will come in handy for
https://github.com/sourcegraph/sourcegraph/issues/53775, as well as
various Collector improvements from recent releases:
https://github.com/open-telemetry/opentelemetry-collector-contrib/releases
I skimmed through the recent breaking changes and I don't think there's
anything that affects us
Update:
https://github.com/sourcegraph/sourcegraph/pull/54969#issuecomment-1638557302
## Test plan
```
sg run otel-collector jaeger
sg start
```
See v0.81 collector running:
```
[ otel-collector] 2023-07-14T21:37:54.709Z info jaegerexporter@v0.81.0/exporter.go:173 State of the connection with the Jaeger Collector backend{"kind": "exporter", "data_type": "traces", "name": "jaeger", "state": "READY"}
```
Do a trace, get a result from multiple services:
---------
Co-authored-by: Will Dollman <will.dollman@sourcegraph.com>
Co-authored-by: davejrt <davetry@gmail.com>
Main motivation is to see the effect on performance for attribution
search.
Note that this included a bump in the otel version used since zoekt is
using a new version. This had a bit of a cascading effect on our third
party deps since they removed a package. So this bumps most 3rd party
packages that directly interacted with otel.
The changed commits are
7643f3b313...45f608ff95
- a176bde1a3 go get -u -t ./...
- e2e8aede00 Fix template documentation comments
- 25c1ea5177 all: observe missing Stats RegexpsConsidered and FlushReason
- 9abbb8b0d3 zoekt-indexserver: Prevent invalid config from causing an NPE
- 008a775ba8 zoekt-indexserver: use value format directive for bad conf warning
- f9d3a0e2e4 zoekt: add fgprof for full profiling
- 3d0bdd5c9c remove ngram offset code
- 45f608ff95 sort ngrams before looking them up
Test Plan: tested in the zoekt repo. Our CI will handle the dep updates.
I eyeballed them and they all look low risk.
Co-authored-by: Jean-Hadrien Chabran <jh@chabran.fr>
- bumps `rules_go` to `0.40.1` (didn't go for `0.41.0` yet, the breaking
change with google apis is a bit messy to deal with, can be handled in a
second time).
- Because of an update to the crypto libs, we can't compare
authenticators directly anymore. Instead I updated the test code to
compare their `.Hash()`, which is semantically equivalent.
- Added that particular test to the flakes, to green the back compat
tests.
- Big thanks to @camdencheek for fixing the issues with the Go compiler
and the `execabs` mess which requires absolute paths, which needed to be
fixed, as Go 1.20.X doesn't ship with precompiled code for the stdlib
anymore.
## Test plan
<!-- All pull requests REQUIRE a test plan:
https://docs.sourcegraph.com/dev/background-information/testing_principles
-->
CI
---------
Co-authored-by: Camden Cheek <camden@ccheek.com>
This combines enterprise and oss repo-updater into a single package,
removing the enterprise setup hook.
## Test plan
Starts up locally, also CI tests.
`--excluded` flag doesn't exist, but `--except` does
## Test plan
```
sourcegraph on wb/sg/typo [$?] via 🐹 v1.19.8 took 10s
❯ go run ./dev/sg start --except symbols
ℹ️ We found some changes in dev-private that you're missing out on! If you want the new changes, 'cd ../dev-private' and then do a 'git stash' and a 'git pull'!
✅ Done checking dev-private changes
💡 Running 4 checks...
✅ Check "docker" success!
✅ Check "redis" success!
✅ Check "postgres" success!
✅ Check "git" success!
Skipping command symbols since it's in --except.
```
<!-- All pull requests REQUIRE a test plan:
https://docs.sourcegraph.com/dev/background-information/testing_principles
-->
Fixes the cadvisor entry point, which was being split onto new lines,
meaning we were dropping some metrics like memory as reported by
@michaellzc.
As part of the testing I tried just downloading the binary from github,
rather than building it from scratch. That didn't turn out to the final
solution so that's an OPTIONAL leave in, but to me seems like an easier
approach than compiling ourselves.
With a rebuild of the base image we're also now running cadvisor 0.47.3
Old (working):
```
"Entrypoint": [
....
"-enable_metrics=cpu,diskIO,memory,network",
....
],
```
New (broken):
```
"Entrypoint": [
"/usr/bin/cadvisor",
....
"-enable_metrics=cpu",
"diskIO",
"memory",
"network",
....
```
## Test plan
Tested on scaletesting.sgdev.org
<img width="1121" alt="image"
src="https://github.com/sourcegraph/sourcegraph/assets/2067825/bedfbe7e-b83c-4a09-8527-5143412cb533">
vs sourcegraph.com
<img width="692" alt="image"
src="https://github.com/sourcegraph/sourcegraph/assets/2067825/0b221909-8c80-4665-9e28-59ba6e6a4722">
Minor improvements I noticed while updating this file. Not sure if this
is in good style for bazel files, so interested in feedback.
Test Plan: rely on CI, didn't test locally.
This allows you to run `sg start` but exclude commands, one off, without
having to modify `sg.config.yaml`. Helpful when debugging or when trying
to get something working while a single command is not compiling.
Usage:
- `sg start --only frontend,worker`
- `sg start --except worker,gitserver-0`
## Test plan
1. Create `sg.config.overwrite.yaml`:
```yaml
defaultCommandset: testing
commandsets:
testing:
commands:
- frontend
- worker
- repo-updater
- gitserver-0
- gitserver-1
```
2. Run `go run ./dev/sg start --only frontend,worker`
3. Run `go run ./dev/sg start --except repo-updater,gitserver-1`
---------
Co-authored-by: Dave Try <davetry@gmail.com>
Everything is enterprise licensed now, so this doesn't change anything
license-wise. Just a small step towards getting rid of the enterprise
directory.
## Test plan
CI.
It seems we were using a weird version of throttled which I don't think
was actually based on the v2 series correctly. This updates us to the
latest version and moves us away from the removed struct GCRARateLimiter
and the deprecated interface GCRAStore.
I came across this while attempting to update deps and getting failing
compiles.
Test Plan: we have unit tests for this, so CI.
Ports our build of prometheus-gcp to wolfi inline with our other images.
## Test plan
Built previously and has been tested on a clouddev instance with help
from @danieldides with no issues.
Cody code is being moved to https://github.com/sourcegraph/cody. This PR
removes the moved code and configures this repository to use the
`@sourcegraph/cody-{shared,ui}` packages published by the new separate
Cody repository.
## Test plan
Ensure that the new separate Cody repository tests pass. Ensure that
this repository's tests pass now that it uses 2 npm packages published
by the new separate Cody repository.
1. Determines the next available RFC ID
2. Creates the RFC document from the template
3. Updates the content of the RFC document to match the RFC title
4. Remove the reminder to update the titles (as we just did it)
If private:
5. Creates a breadcrumb file in the public RFC pointing to the private
RFC.
> TODO(nelsona): Add --type <type> and support other kinds of RFC
formats (right now, it only supports the problem solution)
## Test plan
(as it requires Google Drive credentials) Execute manually:
1. `sg rfc create "<some public RFC title>"`. Observe that a new doc was
created in the public RFC folder
2. `sg rfc --private create "<some private RFC title>"`. Observe that:
* A new RFC doc is created in the private RFC folder
* A breadcrumb document pointing to the private RFC is created in the
public RFC folder
Resolves#25440
Companion handbook change
https://github.com/sourcegraph/handbook/pull/7288
---------
Co-authored-by: William Bezuidenhout <william.bezuidenhout@sourcegraph.com>
Co-authored-by: Thorsten Ball <mrnugget@gmail.com>
This marks everyone as optional when sending calendar events related to the release.
## Test plan
<!-- All pull requests REQUIRE a test plan: https://docs.sourcegraph.com/dev/background-information/testing_principles -->
Run `pnpm run release tracking:timeline ` with `{dryRun.calendar}` set to `true` and the events generated should have an extra field of `optional: true`
Replaces an explicit `go:generate` call to `stringer` by its Bazel
counterpart.
This is a part of the effort to remove all `go:generate` statements and
to have them handled by Bazel. See tracked issue for more details.
## Test plan
<!-- All pull requests REQUIRE a test plan:
https://docs.sourcegraph.com/dev/background-information/testing_principles
-->
CI
@efritz noticed that `sg run frontend` always pick the Bazel variant,
which at the moment triggers a bundle build, which is really really
slow.
This was not how it was when we first introduced the Bazel variants of
the commands, but I missed the consequences of refactoring how we handle
the assets a while ago, which created this situation.
This PR provides a bandage, which I hope will be ok with you @efritz
while we fix the root cause for this.
## Test plan
<!-- All pull requests REQUIRE a test plan:
https://docs.sourcegraph.com/dev/background-information/testing_principles
-->
CI + locally tested.
Movin enterprise codeintel stuff (and ~two others that had to be dragged
along) out of `enterprise/internal` and into `internal` as part of the
shift towards enterprise-only
## Test plan
Successfully built frontend with bazel, CI will check the rest
😎 no logic changed, just shufflin things around
This introduces the `bazel-do` runtype, which provides a
`bazel-do/[name]` branch scheme, where you can pass in the commit
description a bazel command to be run.
Diagnosing flaky tests, or simply testing something in CI can a bit
cumbersome if you don't know your way around the pipeline. This PR fixes
that by providing a way to set-up a build for your branch that will run
a single bazel command and nothing else.
Classic use cases can be:
- diagnosing a flaky test
- just testing something and you want the faster builds from the CI
- ensuring that your new code builds remotely without having to wait for
the whole build.
How to use: create a `bazel-do/my-thing` branch and put `!bazel test
//something` in the last commit description.
I plan to follow up with a `bazel ci bazel test //foobar` that will
create the branch and the commit for you, as a convenience.
Security: only Sourcegraph teammates can build this, and it only
triggers on branch named `bazel-do`.
## Test plan
<!-- All pull requests REQUIRE a test plan:
https://docs.sourcegraph.com/dev/background-information/testing_principles
-->
Did a few manual tests in CI, they worked perfectly.
---------
Co-authored-by: Will Dollman <will.dollman@sourcegraph.com>
Update SRC endpoint for release Batch Change.
## Test plan
<!-- All pull requests REQUIRE a test plan: https://docs.sourcegraph.com/dev/background-information/testing_principles -->
* Create a batch change with the release tool, the batch change should be created on `sourcegraph.sourcegraph.com`
This removes an extra `semaphore.Release` (it's already deferred in that
function) that was causing a panic locally in `sg` and misleading me on
the underlying cause.
This PR is on top of #54517
---
This PR rework how we handle the docsite and some of its related tasks:
- While we _could_ avoid committing the generated files to disk, the way
that docsite currently works, requires to have all files available in
git, so the docsite can serve various versions.
- `bazel run //doc/cli/references:write_doc_files` handles this (and
it's umbreall `//dev:write_all_generated`.
- Its generation is now handled by `//doc/cli/references:generate_doc`
and its associated test.
- `sg run docsite` now calls `bazel run //doc:serve`, we don't need to
install the docsite directly anymore (but it's available under `bazel
run //dev/tools:docsite` if needed).
As a byproduct, we can now run `src-cli` with `bazel run
//dev/tools:src-cli` which will pick the correct binary for your machine
transparently.
## Test plan
<!-- All pull requests REQUIRE a test plan:
https://docs.sourcegraph.com/dev/background-information/testing_principles
-->
Locally tested + CI
Creates a `go_stringer` macro and replace lsif code that used a
`go:generate` + stringer. Because this generates Go code, it's still
better to have the generated code versioned, as it doesn't require
everyone to configure `gopls` so it plays nicely with Bazel.
This also introduces a `//dev:write_all` target that calls all known
targets that write back to the source tree.
## Test plan
<!-- All pull requests REQUIRE a test plan:
https://docs.sourcegraph.com/dev/background-information/testing_principles
-->
CI
As we transitioned to rules_oci + wolfi, the custom dockerfile checks
are not necessary anymore. Hadolint is still there, will address it in
another PR that introduces pre-commit.
## Test plan
<!-- All pull requests REQUIRE a test plan:
https://docs.sourcegraph.com/dev/background-information/testing_principles
-->
CI
Because we at least need one file to be present or the embed won't work,
this replaces CHANGELOG.md by CONTRIBUTING.md as a placeholder. This
will prevent PRs updating the changelog from invalidating the cache for
nothing for these slow targets.
## Test plan
<!-- All pull requests REQUIRE a test plan:
https://docs.sourcegraph.com/dev/background-information/testing_principles
-->
CI
This removes the linter that checks for incorrectly formatted
`go:generate` directives from `sg lint` and replace it with a proper
linter handled by `nogo`.
## Test plan
<!-- All pull requests REQUIRE a test plan:
https://docs.sourcegraph.com/dev/background-information/testing_principles
-->
Locally tested + CI
This endpoint was unused which was why it was removed in Zoekt. The
changes from Zoekt included in this dep bump:
- 1686b50d50 indexserver: remove unused GetRepoRank
- 7078a585a9 shards: populate RepoList.Stats.Repos
- b9e6d9433e zoekt-indexserver: Check stderr for git fetch
- 93f7b0c983 matchtree: capture Stats before pruning
- 7643f3b313 matchiter: capture metric NgramLookups
Test Plan: CI
This PR moves the target release for back compat tests to 5.1 and fixes
the various issues that arose.
- We don't need to handle otel breaking updates, as 5.1 and HEAD are on
the same version
- We now need to explicitly remove `/client/*` as we're properly
targeting the `v5.1.0` tag and it doesn't contain my crude patches
anymore
- Bumped pinned back compat scip deps, as it frequently breaks due to
having a lot of API changes across time
- Added a new patch, to handle the consequences of having removed the
client tree: previous release didn't have the refactored ui/assets
explicit oss/enterprise targets and thus didn't explicitly depend on the
client. Now we do, so it needed some patching to make the build working.
- The buildkite command itself now substracts a few steps that are not
playing very well with the backcompat approach, I'm not sure to
understand why, the the `//cmd/migrator:schema_descriptions` `genrule`
doesn't like very much being built in a different place. I suspect
that's because the tree is one folder below, but it's harder to patch
this than it is to simply remove it from the test targets entirely.
- And finally, the buildkite command also filters things to only run go
tests.
## Test plan
<!-- All pull requests REQUIRE a test plan:
https://docs.sourcegraph.com/dev/background-information/testing_principles
-->
CI + local fun
