A few `x_defs` attributes were missing on binaries, which is now fixed.
Also moved to stamping from `go_library` rules to `go_binaries` to ease
caching.
## Test plan
<!-- All pull requests REQUIRE a test plan:
https://docs.sourcegraph.com/dev/background-information/testing_principles
-->
CI + main-dry-run + locally tested + `strings github-proxy | grep
version` on `us.gcr.io/sourcegraph-dev/github-proxy:323c450504f8`
Having the default tags handled makes it confusing as we're still in a
transition phase and having two places to handle those is error prone.
## Test plan
<!-- All pull requests REQUIRE a test plan:
https://docs.sourcegraph.com/dev/background-information/testing_principles
-->
local testing + CI
The previous approach to enable race detection was too radical and
accidently led to build our binaries with the race flage enabled, which
caused issues when building images down the line.
This happened because putting a `test --something` in bazelrc also sets
it on `build` which is absolutely not what we wanted. Usually folks get
this one working by having a `--stamp` config setting that fixes this
when releasing binaries, which we don't at this stage, as we're still
learning Bazel.
Luckily, this was caught swiftly. The current approach insteads takes a
more granular approach, which makes the `go_test` rule uses our own
variant, which injects the `race = "on"` attribute, but only on
`go_test`.
## Test plan
<!-- All pull requests REQUIRE a test plan:
https://docs.sourcegraph.com/dev/background-information/testing_principles
-->
CI, being a main-dry-run, this will cover the container building jobs,
which were the ones failing.
---------
Co-authored-by: Alex Ostrikov <alex.ostrikov@sourcegraph.com>
This PR ships our freshly rewritten container images built with
rules_oci and Wolfi, which for now will only be used on S2.
*What is this about*
This work is the conjunction of [hardening container
images](https://github.com/orgs/sourcegraph/projects/302?pane=issue&itemId=25019223)
and fully building our container images with Bazel.
* All base images are now distroless, based on Wolfi, meaning we fully
control every little package version and we won't be subject anymore to
Alpine maintainers dropping a postgres version for example.
* Container images are now built with `rules_oci`, meaning we don't have
Dockerfile anymore, but instead created through [Bazel
rules](https://sourcegraph.sourcegraph.com/github.com/sourcegraph/sourcegraph@bzl/oci_wolfi/-/blob/enterprise/cmd/gitserver/BUILD.bazel).
Don't be scared, while this will look a bit strange to you at first,
it's much saner and simpler to do than our Dockerfiles and their muddy
shell scripts calling themselves in cascade.
:spiral_note_pad: *Plan*:
*1/ (NOW) We merge our branch on `main` today, here is what it does
change for you 👇:skin-tone-3::*
* On `main`:
* It will introduce a new job on `main` _Bazel Push_, which will push
those new images on our registries with all tags prefixed by `bazel-`.
* These new images will be picked up by S2 and S2 only.
* The existing jobs building docker images and pushing them will stay in
place until we have QA'ed them enough and are confident to roll them out
on Dotcom.
* Because we'll be building both images, there will be more jobs running
on `main`, but this should not affect the wall clock time.
* On all branches (so your PRs and `main`)
* The _Bazel Test_ job will now run: Backend Integration Tests, E2E
Tests and CodeIntel QA
* This will increase the duration of your test jobs in PRs, but as we
haven't removed yet the `sg lint` step, it should not affect too much
the wall clock time of your PRs.
* But it will also increase your confidence toward your changes, as the
coverage will vastly increased compared to before.
* If you have ongoing branches which are affecting the docker images
(like adding a new binary, like the recent `scip-tags`, reach us out on
#job-fair-bazel so we can help you to port your changes. It's much much
simpler than before, but it's going to be unfamiliar to you).
* If something goes awfully wrong, we'll rollback and update this
thread.
*2/ (EOW / Early next week) Once we're confident enough with what we saw
on S2, we'll roll the new images on Dotcom.*
* After the first successful deploy and a few sanity checks, we will
drop the old images building jobs.
* At this point, we'll reach out to all TLs asking for their help to
exercise all features of our product to ensure we catch any potential
breakage.
## Test plan
<!-- All pull requests REQUIRE a test plan:
https://docs.sourcegraph.com/dev/background-information/testing_principles
-->
* We tested our new images on `scale-testing` and it worked.
* The new container building rules comes with _container tests_ which
ensures that produced images are containing and configured with what
should be in there:
[example](https://sourcegraph.sourcegraph.com/github.com/sourcegraph/sourcegraph@bzl/oci_wolfi/-/blob/enterprise/cmd/gitserver/image_test.yaml)
.
---------
Co-authored-by: Dave Try <davetry@gmail.com>
Co-authored-by: Will Dollman <will.dollman@sourcegraph.com>
Addresses https://github.com/sourcegraph/sourcegraph/issues/52420
Previously when App shut down it was leaving behind postgres processes
thats caused issues due to the socket from the previous instance
remaining. This is manifest to the user as a app with a blank screen
that wouldn't launch.
There were several issues
1. The github proxy was calling `os.Exit` on the first shutdown signal
received. This is an issue for app because all services are running in
the same parent process it would terminate the app before any other
services or postgres could properly close.
2. Once the `os.Exit` issue was resolved the single binary would never
terminate because it was paused on an empty select. This replaces the
empty select with a 2nd wait group that waits for the termination of
each service that was started. The app will exit now but leaves a race
condition on shutdown of the services and postgres.
3. To resolve the race condition I removed postgres from being monitored
as a background process and instead return a cleanup function from
`Init` so that balance can be restored. Init is responsible for starting
postgres before any service and now it can inform the caller how to
clean up anything it needs to.
As we're using
https://github.com/GoogleContainerTools/container-structure-test when
building images with Bazel, we can write tests that ensures that the
binaries we produced are executable on the current platform.
The code using this feature is not present in this PR, as it makes it
much more readable for everyone to ship this independently.
We may still be bitten by `init()` functions, but we'll set low timeouts
on those container structure tests to ensure it stays quick.
## Test plan
<!-- All pull requests REQUIRE a test plan:
https://docs.sourcegraph.com/dev/background-information/testing_principles
-->
```
~/work/other jh/sanity_check $ SANITY_CHECK=true ./bazel-bin/docker-images/syntax-highlighter/syntect_server
Sanity check passed, exiting without error
~/work/other U jh/sanity_check $ ./bazel-bin/docker-images/syntax-highlighter/syntect_server
## Embedded themes:
- `InspiredGitHub`
- `Monokai`
- `Solarized (dark)`
# ...
```
```
~/work/other jh/sanity_check $ SANITY_CHECK=true bazel-bin/cmd/worker/worker_/worker
Sanity check passed, exiting without error
```
This PR contains all the fixes and tweaks I've made during the process
of getting the wolfi-test cloud instance working.
There's a lot going on in this PR:
Packages:
* Build comby from source - the published binary isn't compatible with
Arch/Wolfi's libpcre library path
* Build cadvisor from source - may not be necessary, but done as a
debugging step and gives us more arch flexibility
* Package s3proxy
* Update p4-fusion dependency SHA
* Bump all packages to trigger a rebuild with latest version of melange.
The old version didn't seem to pick up on dylibs properly
* Fix opentelemetry-collector binary path
Base images:
* Fix cadvisor base image
* Fix redis UID+GIDs
* Add missing p4cli package to repo-updater image
* Remove nodejs from server image (no longer required by code-intel)
Other fixes:
* Update build-wolfi.sh scripts to match the current Bazelified build.sh
scripts
* Fix `server` image builds
* Fix zoekt-webserver CMD
* Fix postgres locale + default config + add a patch for
`unix_socket_directories`
* Fix `symbols` go-build-wolfi scripts
## Things to do before merging
- [ ] Validate the new postgres patch-conf.sh script with delivery - will do post-merge
- [x] Remove all DO NOT MERGE comments
## Test plan
<!-- All pull requests REQUIRE a test plan:
https://docs.sourcegraph.com/dev/background-information/testing_principles
-->
- [x] Manual testing on wolfi-test cloud instance
- [x] Manual testing on scaletesting instance
Renaming main entrypoints which are confusingly marked as "deprecated",
yet are not deprecated from what I've been informed 🤷
## Test plan
No logic changes, just renaming functions
Reintroduces the same changes as
https://github.com/sourcegraph/sourcegraph/pull/51104 minus
syntax-highlighter which we're unable to compile with the right
toolchain at the moment.
Tested as a full main-dry-run, as well as running the stack with compose
and checking indexing and syntax-highlighting.
Executors are also built correctly.
## Test plan
CI + manual test via compose.
---------
Co-authored-by: Jean-Hadrien Chabran <jh@chabran.fr>
Build the remaining docker images with bazel
Depends on:
https://github.com/sourcegraph/sourcegraph/pull/49228
## Test plan
CI
---------
Co-authored-by: Jean-Hadrien Chabran <jh@chabran.fr>
Co-authored-by: William Bezuidenhout <william.bezuidenhout@sourcegraph.com>
* internal: add service and singleprogram packages
* sg.config.yaml: add single-binary build targets
* internal/env: add a function for clearing environ cache
* internal/{workerutil,metrics}: add a hack to allow running 2 executors in the same process
* internal/conf: add single-program deploy type
* internal/singleprogram: clarify security
* cmd/sourcegraph-oss: add initial single-binary main (will not build yet)
* enterprise/cmd/sourcegraph: initial enterprise single-binary
* Add multi-platform builds for single-program
* single-binary: correctly build JS artifacts into binary
* license_finder licenses add github.com/xi2/xz "Public domain"
* internal/service/svcmain: correctly initialize logger for DeprecatedSingleServiceMain
* worker: refactor to new service pattern
* cmd/github-proxy: refactor to use new service pattern
* symbols: refactor to use new service pattern
* gitserver: refactor to user new service pattern
* searcher: refactor to use new service pattern
* gitserver: refactor to use new service pattern
* repo-updater: refactor to use new service pattern
* frontend: refactor to use new service pattern
* executor: refactor to use new service pattern
* internal/symbols: use new LoadConfig pattern
* precise-code-intel-worker: refactor to use new service pattern
* internal/symbols: load config for tests
* cmd/repo-updater: remove LoadConfig approach
* cmd/symbols: workaround env var conflict with searcher
* executor: internal: add workaround to allow running 2 instances in same process
* executors: add EXECUTOR_QUEUE_DISABLE_ACCESS_TOKEN for single-binary and dev deployments only
* single-binary: use EXECUTOR_QUEUE_DISABLE_ACCESS_TOKEN
* extsvc/github: fix default value for single-program deploy type
* single-binary: stop relying on a local ctags image
* single-binary: use unix sockets for postgres
* release App snapshots in CI when pushed to app/release-snapshot branch
* internal/service/svcmain: update TODO comment
* executor: correct DEPLOY_TYPE check
* dev/check: allow single-binary to import dbconn
* executor: remove accidental reliance on dbconn package
* executor: improve error logging when running commands (#46546)
* executor: improve error logging when running commands
* executor: do not attempt std config validation running e.g. install cmd
* executor: do not pull in the conf package / frontend reliance
* ci: executors: correct site config for passwordless auth
* server: fix bug where github-proxy would try to be a conf server
* CI: executors: fix integration test passwordless auth
* executors: allow passwordless auth in sourcegraph/server for testing
* repo-updater: fix enterprise init (caused regression in repository syncing)
Signed-off-by: Stephen Gutekanst <stephen@sourcegraph.com>
Co-authored-by: Peter Guy <peter.guy@sourcegraph.com>
Co-authored-by: Quinn Slack <quinn@slack.org>
This is a copy-and-paste of the github-proxy main function to a shared package so that github-proxy can be invoked from a different entrypoint. This is how it's done for several other programs (frontend, repo-updater, gitserver, etc.). There are no logic changes.
We updated sourcegraph/alpine-3.14 today in 3b3879b9 to remove a bunch
of CVEs in bind-tools. This commit makes it so we use the new image.
Test Plan: CI works and has far less security reports.
A repo label is set on httptrace metrics, but it only tracks a small set of hardcoded repositories that have not been changed for at least 4 years. There do not appear to be any references to this label in any dashboard either.
origin is another label untouched for 4+ years, and is always set to 'unknown' in s2. Tracking references also reveals two metrics that have no references in dashboards.
Swap out the OpenTracing HTTP middleware with otelhttp, with a default constructor provided in a new package internal/instrumentation. Also adds deprecation notices on internal/trace/ot, and updates deprecation notices in internal/trace so that they render properly in VS Code.
We have a 3.14 image available, but when we introduced it we ran into problems
using it. This updates all images to use the newer version of alpine.
ruplacer --go \
sourcegraph/alpine-3.12:142406_2022-04-14_8836ac3499f4@sha256:4681a48d1fb9a73fef1b540c08b3411f797351bbeda749f5dca21213a1e71526 \
sourcegraph/alpine-3.14:142406_2022-04-14_8836ac3499f4@sha256:2a2d1cbaec78882661fe1aa5b0a4af0c23a37be2ea9ff8aadc2da5b80852c233
I also had to pin postgresql in the server image since we can't upgrade it
otherwise customer upgrades fail (incompatible on disk data). Additionally we
had to include libstc++ and libgcc for tree sitter.
Risks: I am relying on our CI to catch problems with this upgrade. The main
risks here are us not testing things which exercise non-go binaries that rely
on shared libraries. What comes to mind is things like p4 and tree-sitter, I
am unsure if they get properly exercised. However, I think now is a good time
to land a change like this since we have a lot of time until branch cut.
Test Plan: main dry run on CI to test docker image building and integration
testing.
