* wip
* gitserver (mostly) wolfi 4 bazel
* the big heck of all things
* Add rules_apko lock translation rules to WORKSPACE
* Call apko_repositories() more
* fix rules_apko to handle our shorter repo urls
* fix workspace from rebase, and missing locks
* visibility on wolfi_base_image
* hand-fix a lock coz apko lock is 🅱️roken
* remove chainguard repo+keyring from base
* update locks
* add chainguard repo+keychain to single server manifest
* unrelated fixes, server+grafana still h*cked
* fix postgres-exporter
* the big fix
* aws lib got bumped?
* downgrade sso-oidc? idk
* ignore wolfi locks from prettier
* dynamically do the locks with a reporule
* document and make nice :nails:
* bazel run @rules_apko//apko patch
* Fix .typo.typo
* Update tooling for end-to-end Bazel images (#61106)
* Update sg wolfi image to build using Bazel
* bazel run @rules_apko//apko patch
* Fix .typo.typo
* Add update-images and implement apko YAML change monitoring
* Use bazel apko and add support for additional repos
* Refactor sg wolfi
* Rework wolfi base image auto-update pipeline
* sg bazel configure
* [rough] Add --check flag to sg wolfi lock
* Refactor sg wolfi lock --check
* Simplify check and update apko lock hash operations
* Fix resolveImagePath when running in bazel
* Fixup logic error in CheckApkoLockHashes
* Tweak DoBaseImageBuild output
* Remove debug output
* Fix sg wolfi lock --check behaviour for all images
* Replace base image build step with apko lock --check
* Remove debug line
* Minor fixups for CI step
* Wrap with AnnotatedCmd
* Fixup annotation
* Update apko lockfiles
* Allow additional repos to be passed
* Update build-base-image.sh with bazel + add back to pipeline
* Ensure that modified base images are rebuilt
* Solve bazelception
* Remove timestamp for bit-level reproducibility
* Skip local keygen when running on buildkite
* Add workaround for lack of local repo support in rules_apko
* Run apkoOps first as it's quick and might fail
* Remove blocking allBaseImagesBuilt step
* Remove unused promethus-gcp image
* Add special cases to resolveImagePath
* Cleanly handle case where no bazel build path exists
This could happen in cases where a base image is only used outside of sourcegraph/sourcegraph,
or if you've added a new base image config but haven't added the associated Bazel scaffolding
* Add debugging around failing docker builds
* More debugging
* Normalise apko_lockfile to match repo.bzl
* Fixup apko docker call
* Try passing imageconfigdir differently to docker
* Run ls in different container
* Soft-fail when using legacy build in Buildkite
* Add missing include
* Workaround for building sourcegraph and sourcegraph-dev
* Add postgresql-client package to server
This contains createdb, which was recently moved from postgresql
* Inflate postgres-12-codeinsights image to avoid rules_apko errors
* Remove update line from yaml files
* Fix issue caused by moving base sourcegraph image
* Remove apk-tools from server
* Update lockfiles
* Address review feedback
* Remove debug lines
* fix unbound var
---------
Co-authored-by: Noah Santschi-Cooney <noah@santschi-cooney.ch>
* go mod tidy + gazelle-update-repos after merging main
* Use aspect bazel cache
* Use Aspect bazel caching when calling bazel in bash and sg
* Append annotation
* Run apko lock on aspect agent
* Remove base image builds
Discussion in https://sourcegraph.slack.com/archives/C05EVRLQEUR/p1712307465660509
* Remove unused functionality
* Update BaseImageConfig comments
* Rewrite wolfi-images/README.md
* Add .apko/range.sh to .gitattributes
* Remove "wolfi" from :base_image and :base_tarball targets
* remove allowlist extras from debugging
* Tweak user instructions around package testing
* Add agent healthcheck to buildkite scripts
* prettier
* sg bazel configure
* bazel run //:gazelle-update-repos
---------
Co-authored-by: Noah Santschi-Cooney <noah@santschi-cooney.ch>
Co-authored-by: Noah S-C <noah@sourcegraph.com>
* wip
* gitserver (mostly) wolfi 4 bazel
* the big heck of all things
* Add rules_apko lock translation rules to WORKSPACE
* Call apko_repositories() more
* fix rules_apko to handle our shorter repo urls
* fix workspace from rebase, and missing locks
* visibility on wolfi_base_image
* hand-fix a lock coz apko lock is 🅱️roken
* remove chainguard repo+keyring from base
* update locks
* add chainguard repo+keychain to single server manifest
* unrelated fixes, server+grafana still h*cked
* fix postgres-exporter
* the big fix
* aws lib got bumped?
* downgrade sso-oidc? idk
* ignore wolfi locks from prettier
* dynamically do the locks with a reporule
* document and make nice :nails:
* bazel run @rules_apko//apko patch
* Fix .typo.typo
* Update tooling for end-to-end Bazel images (#61106)
* Update sg wolfi image to build using Bazel
* bazel run @rules_apko//apko patch
* Fix .typo.typo
* Add update-images and implement apko YAML change monitoring
* Use bazel apko and add support for additional repos
* Refactor sg wolfi
* Rework wolfi base image auto-update pipeline
* sg bazel configure
* [rough] Add --check flag to sg wolfi lock
* Refactor sg wolfi lock --check
* Simplify check and update apko lock hash operations
* Fix resolveImagePath when running in bazel
* Fixup logic error in CheckApkoLockHashes
* Tweak DoBaseImageBuild output
* Remove debug output
* Fix sg wolfi lock --check behaviour for all images
* Replace base image build step with apko lock --check
* Remove debug line
* Minor fixups for CI step
* Wrap with AnnotatedCmd
* Fixup annotation
* Update apko lockfiles
* Allow additional repos to be passed
* Update build-base-image.sh with bazel + add back to pipeline
* Ensure that modified base images are rebuilt
* Solve bazelception
* Remove timestamp for bit-level reproducibility
* Skip local keygen when running on buildkite
* Add workaround for lack of local repo support in rules_apko
* Run apkoOps first as it's quick and might fail
* Remove blocking allBaseImagesBuilt step
* Remove unused promethus-gcp image
* Add special cases to resolveImagePath
* Cleanly handle case where no bazel build path exists
This could happen in cases where a base image is only used outside of sourcegraph/sourcegraph,
or if you've added a new base image config but haven't added the associated Bazel scaffolding
* Add debugging around failing docker builds
* More debugging
* Normalise apko_lockfile to match repo.bzl
* Fixup apko docker call
* Try passing imageconfigdir differently to docker
* Run ls in different container
* Soft-fail when using legacy build in Buildkite
* Add missing include
* Workaround for building sourcegraph and sourcegraph-dev
* Add postgresql-client package to server
This contains createdb, which was recently moved from postgresql
* Inflate postgres-12-codeinsights image to avoid rules_apko errors
* Remove update line from yaml files
* Fix issue caused by moving base sourcegraph image
* Remove apk-tools from server
* Update lockfiles
* Address review feedback
* Remove debug lines
* fix unbound var
---------
Co-authored-by: Noah Santschi-Cooney <noah@santschi-cooney.ch>
* go mod tidy + gazelle-update-repos after merging main
* Use aspect bazel cache
* Use Aspect bazel caching when calling bazel in bash and sg
* Append annotation
* Run apko lock on aspect agent
* Remove base image builds
Discussion in https://sourcegraph.slack.com/archives/C05EVRLQEUR/p1712307465660509
* Remove unused functionality
* Update BaseImageConfig comments
* Rewrite wolfi-images/README.md
* Add .apko/range.sh to .gitattributes
* Remove "wolfi" from :base_image and :base_tarball targets
* remove allowlist extras from debugging
* Tweak user instructions around package testing
* Add agent healthcheck to buildkite scripts
* prettier
---------
Co-authored-by: Noah Santschi-Cooney <noah@santschi-cooney.ch>
Co-authored-by: Noah S-C <noah@sourcegraph.com>
This adds a generated markdown reference that can serve as a source-of-truth for exactly what data we do and don't export from Sourcegraph instances.
Table output is a little funky, unless https://github.com/pseudomuto/protoc-gen-doc/pull/520 gets landed one day (for that new `br` template function)
---------
Co-authored-by: William Bezuidenhout <william.bezuidenhout@sourcegraph.com>
This change only renames internal references and the LLM-proxy image
itself.
Will follow up with another PR to rename licensing, database tables, and
the GraphQL queries/mutations.
Paired with https://github.com/sourcegraph/infrastructure/pull/5010
Part of #52258
## Test plan
CI, `sg start dotcom`, `sg run llm-proxy`
Adds a generated client for talking to Sourcegraph.com's GraphQL API.
This is mostly copy-pasta from [the generated client we set up for
Sourcegraph
Cloud](https://github.com/sourcegraph/controller/tree/main/internal/srcgql),
but simplified for use here. Example usage:
```go
c := dotcom.NewClient(sourcegraphToken)
resp, err := dotcom.CheckAccessToken(ctx, c, licenseToken)
if err != nil {
log.Fatal(err)
}
println(resp.GetDotcom().ProductSubscriptionByAccessToken.LlmProxyAccess.Enabled)
```
Why a generated client? The API surface between LLM-proxy and dotcom
will hopefully be small, but a generated client is IMO still a far
superior experience than writing the query in a Go string, hand-crafting
the appropriate structs and hoping for the best. This gives us robust
typechecking, boilerplate generation, and [nice pathways to
mocking](https://github.com/sourcegraph/controller/blob/main/internal/srcgql/gqltest/gqltest.go)
Some of the copy-pasta adds tracing instrumentation - will follow up to
add an exporter for it.
Part of https://github.com/sourcegraph/sourcegraph/issues/50726
## Test plan
<!-- All pull requests REQUIRE a test plan:
https://docs.sourcegraph.com/dev/background-information/testing_principles
-->
n/a - this setup is pretty tried and tested in Cloud, and this PR
doesn't add any actual usages yet
---------
Co-authored-by: Valery Bugakov <skymk1@gmail.com>
Last time we changed the zoekt module name it broke the wire protocol
for zoekt. This now includes one more commit such that the wire protocol
is unchanged:
- c9182fcd2a rpc: use old module name when registering gob values
Revert chain:
- #40404
- #40353
Test Plan: Ran a local dev server with a hardcoded zoekt-webserver
pinned to the old version (the google/zoekt module). indexed search was
failing. Then upgraded the gomod version and it started working.
This reverts commit 6d376d871a.
See #inc-134-search-on-sourcegraphcom-fails-with-errors in Slack.
Current suspicion is that it causes gob-encoding/decoding problems in
production if the code below is not in sync with the zoekt cluster code.
We have updated the module name in the zoekt repo to our own repository name.
This means we can remove the replace directive and just rely on normal go
tooling.
Note: at the same time we rename the default branch for zoekt from master to
main.
Test Plan: master dry run in CI.
* Add diff for a TypeScript file to trigger CI job
* Rename job
* Delete unnecessary TypeScript file after validating CI job
* Remove unused "LSIF Typed" code
This code got moved to a separate repo github.com/sourcegraph/scip.
* gomod: update zoekt
Includes the following commits
- 9d90c1c gitindex: allocate buffer of correct size
- aa0e667 zoekt: verbose flag prints shard metadata
- ddddd93 gitindex: do not resolve HEAD ref
- 8c0aee7 build: handle nil Options in largeFilesFlag
- 36a801c indexserver: use temp directory on same mount as indexes
- 070c191 indexserver: align flag defaults with dockerfile
- 35dabc7 indexserver: optionally shallow clone and use zoekt-git-index
- f89b8e3 indexserver: inline CmdArgs into archiveIndex
- ef52f4b indexserver: consistently use a pointer to indexArgs
- e407d68 indexserver: store root in indexArgs
- 70628e0 indexserver: factor out index logic
- 4961ee9 indexserver: add cli flags to debug indexing and listing
- 4abd60f indexserver: log success
- 605d890 indexserver: metrics track if we skipped indexing
- 1a62bb8 indexserver: rename metric vars to have metric prefix
- a2233e1 indexserver: check incremental in process
- ddccdf9 indexserver: use build.Options to generate flags
- 8463ade Do not ignore large files.
* zoekt: set cpu_fraction flag for indexserver
The default was 0.25, but was bumped to 1.0. This bump was done to align
the flag defaults with how we run indexserver in cluster
environments. We update our dev and single image environments to use the
same value. Those environmemts don't have resource isolation so we need
to continue using 0.25 rather than allowing indexing to use all cores.
* all: build and include zoekt-git-index
If you are indexing multiple branches this command is required. This
adds the command to our dev and single docker image environments. The
docker images for cluster already include the command (see zoekt repo).
* gofmt
This prevents us needing to include the dependencies in our go.mod. Additionally
right now dlv has some conflicts in its dependencies with what we use, leading
to build errors. We can install the same version of dlv by just using "go get"
more like how we did it before go modules.
It depends on GPL code. Lets make it clear we do not pull in GPL code, even if
it is only for our tooling. golanglint-ci is currently disabled on CI, so we
just remove it. When we want to add it back we need to use the same pattern as
we use for some other dev tools (IE download the release rather than compile the
tool)
This uses the same pattern we use for docsite and jaeger. This avoids including
all the dependencies of caddy as well as avoiding the need to build it.
txeh can be replaced with a grep and an append to /etc/hosts, so is an
unneccessary dependency. Additionally, we install it via sudo which causes
issues with your go cache. You end up writing entries into your go cache as
root, breaking your cache when used as yourself.
docsite brought in a lot of dependencies. It is faster to just fetch prebuilt
binaries. This is some tech debt I have been wanting to do for a long time,
especially since it often caused problem when doing go mod update.
We use a branch of godockerize which generates a build.sh and Dockerfile instead
of directly building the docker image.
https://github.com/sourcegraph/godockerize/tree/gen-build
This was run on all main entrypoints (excluding a few which already have a
build.sh / shouldn't be docker images):
go list -f '{{ if (eq .Name "main") }}{{.ImportPath}}{{ end }}' ./... \
| grep cmd | grep -v goreman | grep -v 'schemadoc' | grep -v /server \
| while read pkg; do godockerize build $pkg; done
The build.sh should likely be refactored into using a shared shell script. But
for now this gives us a nice and easy to change replacement.
This commit also removes godockerize from CI and our go.mod.
* go get github.com/sourcegraph/docsite@master
* check for broken links, etc., in documentation in CI
Running `docsite check` at the top level also runs the checker.
* use .Critical accessors; use conf.Unified type
* pkg/conf: add unified configuration types
* {pkg/conf,cmd/...}: rewrite conf package to be database-backed
* pkg/legacyconf: import from sourcegraph@v2.13.6 pkg/conf/ (and reduce)
* legacyschema: import from sourcegraph@v2.13.6 schema/ (and reduce)
* add dev config override support
* cmd/management-console/assets: add assets package for packing web app code
* cmd/management-console/web: initial webapp implementation
* schema: split site.schema.json into site and critical configuration portions
* schema: critical: add gitlab as valid auth.providers key (broken in master)
* web: update to reflect site config changes
* cmd/management-console: initial backend implementation
* cmd/management-console: add Go backend
* cmd/management-console: add TS frontend
* web: site-admin: add management console password alert
* cmd/management-console/internal/tlscertgen: package for TLS certificate generation
* schema: update docs
* upgrade deasync
* pkg/conf: add proper config defaults for each deployment type
* vfsgendev installation fix
* mgmt console web package-lock.json change
* assets doc.go + gitignore change
* gofmt assets
* dev/check: use dev build tag
Some things such as assets are only defined behind a dev build tag OR
after generating something via `go generate`. Since go generate has not
run yet, I am opting to use the `dev` build tag here.
* cmd/management-console/auth_test.go: simplify test
* fix go assets
* pkg/conf/confdefaults
* pkg/conf: validation test fix
* web: update to reflect site configuration changes
* expose management console port in docker run commands
* management-console: go: return concrete type to /update requests, not type that can arbitrarily change in future
* management-console: web: fix saving ID bug + properly display errors
* pkg/db/confdb: return an error if the creator is not up to date
* web mgmt console error handling
* mgmt console go backend ErrNewerEdit
* pkg/conf: fix zero configuration check (marshaled JSON is "{}")
* add linter for accidental transitive imports of pkg/conf in mgmt console
* enterprise/dev/ci: fix building of non-enterprise docker-images branches
* NOCHANGELOG
* git rm -r vendor
* ci: Remove go mod vendor check
* all: remove direct uses of vendor
* all: Specify GO111MODULE=on and GOBIN for generate
* ci: Enable go modules when generating pipeline
* ci: Enable go modules for every pipeline step
* ci: Use count flag to force test run instead of disabling GOCACHE
* ci: Use next branch of honnef.co/go/tools
master currently does not support go modules, but next should.
* all: Update to pass next branch of staticcheck
