### The Old
Replaces `aspect configure` (used via `bazel configure`) which had a few issues:
- Uses host Go toolchain instead of bazel-managed one
- Didnt use our //:gazelle binary, which is configured with `rules_buf` gazelle rules
- Not able to configure other gazelle rules outside of the fixed set that comes with aspect cli
### The New
`sg bazel configure` and its targets (currently `builds`, `godeps` and `rustdeps`). Passing none only runs `sg bazel configure builds`, the pseudo-target `all` runs all of them.
`sg bazel [other arg]` passes to `bazel` underneath, as a convenience option for if people get more used to running `sg bazel ...` than `bazel` directly (or from shell history)
```
$ bazel-bin/dev/sg/sg_/sg bazel help configure
NAME:
sg bazel configure - Wrappers around some commands to generate various files required by Bazel
USAGE:
sg bazel configure [category...]
DESCRIPTION:
For convenience, a number of Bazel commands are wrapped by this command to update various files required by Bazel.
Available categories:
- builds: updates BUILD.bazel files for Go & Typescript targets.
- godeps: updates the bazel Go dependency targets based on go.mod changes.
- rustdeps: updates the cargo bazel lockfile.
- all: catch-all for the above
If no categories are referenced, then 'builds' is assumed as the default.
OPTIONS:
--help, -h show help
```
```
$ bazel-bin/dev/sg/sg_/sg bazel help
Additional commands from sg:
configure Wrappers around some commands to generate various files required by Bazel
[bazel release 7.0.0- (@non-git)]
Usage: bazel <command> <options> ...
Available commands:
analyze-profile Analyzes build profile data.
aquery Analyzes the given targets and queries the action graph.
build Builds the specified targets.
canonicalize-flags Canonicalizes a list of bazel options.
clean Removes output files and optionally stops the server.
coverage Generates code coverage report for specified test targets.
cquery Loads, analyzes, and queries the specified targets w/ configurations.
dump Dumps the internal state of the bazel server process.
fetch Fetches external repositories that are prerequisites to the targets.
help Prints help for commands, or the index.
info Displays runtime info about the bazel server.
license Prints the license of this software.
mobile-install Installs targets to mobile devices.
mod Queries the Bzlmod external dependency graph
print_action Prints the command line args for compiling a file.
query Executes a dependency graph query.
run Runs the specified target.
shutdown Stops the bazel server.
sync Syncs all repositories specified in the workspace file
test Builds and runs the specified test targets.
version Prints version information for bazel.
Getting more help:
bazel help <command>
Prints help and options for <command>.
bazel help startup_options
Options for the JVM hosting bazel.
bazel help target-syntax
Explains the syntax for specifying targets.
bazel help info-keys
Displays a list of keys used by the info command.
```
## Test plan
`bazel run //dev/sg:sg -- bazel configure` and others
As detailed in the shell hook:
Without this, zig cc is forced to rebuild on every sandboxed GoLink action, which adds ~1m of time to GoLink actions. The reason it's on _every_ GoLink action is because sandboxes are ephemeral and don't persist non-mounted paths between actions.
We're already doing this for `darwin-docker` config, so this only affects us nixos users
---
Smuggling in a libtool fix for macos that would otherwise yield the following error when building rust with bazel:
<details>
<summary>Details</summary>
```
cargo:rerun-if-env-changed=AR_aarch64-apple-darwin
AR_aarch64-apple-darwin = None
cargo:rerun-if-env-changed=AR_aarch64_apple_darwin
AR_aarch64_apple_darwin = None
cargo:rerun-if-env-changed=HOST_AR
HOST_AR = None
cargo:rerun-if-env-changed=AR
AR = Some("/nix/store/42yck6r7y2jhcrd0ay0glz30w6pw4wzh-libtool-2.4.7")
cargo:rerun-if-env-changed=ARFLAGS_aarch64-apple-darwin
ARFLAGS_aarch64-apple-darwin = None
cargo:rerun-if-env-changed=ARFLAGS_aarch64_apple_darwin
ARFLAGS_aarch64_apple_darwin = None
cargo:rerun-if-env-changed=HOST_ARFLAGS
HOST_ARFLAGS = None
cargo:rerun-if-env-changed=ARFLAGS
ARFLAGS = None
running: ZERO_AR_DATE="1" "/nix/store/42yck6r7y2jhcrd0ay0glz30w6pw4wzh-libtool-2.4.7" "cq" "/private/var/tmp/_bazel_noah/dcf2fbfa8ce2981c9fc4201fa6327d3b/sandbox/darwin-sandbox/6091/execroot/__main__/bazel-out/darwin_arm64-fastbuild/bin/external/crate_index__tree-sitter-kotlin-0.2.11/tree-sitter-kotlin_build_script.out_dir/libparser.a" "/private/var/tmp/_bazel_noah/dcf2fbfa8ce2981c9fc4201fa6327d3b/sandbox/darwin-sandbox/6091/execroot/__main__/bazel-out/darwin_arm64-fastbuild/bin/external/crate_index__tree-sitter-kotlin-0.2.11/tree-sitter-kotlin_build_script.out_dir/src/parser.o" "/private/var/tmp/_bazel_noah/dcf2fbfa8ce2981c9fc4201fa6327d3b/sandbox/darwin-sandbox/6091/execroot/__main__/bazel-out/darwin_arm64-fastbuild/bin/external/crate_index__tree-sitter-kotlin-0.2.11/tree-sitter-kotlin_build_script.out_dir/src/scanner.o"
--stderr:
error occurred: Command ZERO_AR_DATE="1" "/nix/store/42yck6r7y2jhcrd0ay0glz30w6pw4wzh-libtool-2.4.7" "cq" "/private/var/tmp/_bazel_noah/dcf2fbfa8ce2981c9fc4201fa6327d3b/sandbox/darwin-sandbox/6091/execroot/__main__/bazel-out/darwin_arm64-fastbuild/bin/external/crate_index__tree-sitter-kotlin-0.2.11/tree-sitter-kotlin_build_script.out_dir/libparser.a" "/private/var/tmp/_bazel_noah/dcf2fbfa8ce2981c9fc4201fa6327d3b/sandbox/darwin-sandbox/6091/execroot/__main__/bazel-out/darwin_arm64-fastbuild/bin/external/crate_index__tree-sitter-kotlin-0.2.11/tree-sitter-kotlin_build_script.out_dir/src/parser.o" "/private/var/tmp/_bazel_noah/dcf2fbfa8ce2981c9fc4201fa6327d3b/sandbox/darwin-sandbox/6091/execroot/__main__/bazel-out/darwin_arm64-fastbuild/bin/external/crate_index__tree-sitter-kotlin-0.2.11/tree-sitter-kotlin_build_script.out_dir/src/scanner.o" with args "" failed to start: Os { code: 13, kind: PermissionDenied, message: "Permission denied" }
```
</details>
## Test plan
Observing `bazel build //dev/sg` times
Reimplement fetching & building code-intel-extension bundle as a bazel `http_archive` repository & refactored `js_run_binary`. This should allow the extensions bundle to be cached in the repository cache, instead of being affected by changing action environments
## Test plan
`bazel build //client/browser:code-intel-extensions` and `cd client/browser && pnpm run create-source-zip` run without error
Host clang for bazel reason:
```
ERROR: /Users/noah/Sourcegraph/sourcegraph/BUILD.bazel:305:5: GoLink sg_nogo_actual_/sg_nogo_actual [for tool] failed: (Exit 1): builder failed: error executing GoLink command (from target //:sg_nogo_actual) bazel-out/darwin_arm64-opt-exec-ST-1a88b5d644a4/bin/external/go_sdk/builder_reset/builder '-param=bazel-out/darwin_arm64-opt-exec-ST-1a88b5d644a4/bin/sg_nogo_actual_/sg_nogo_actual-0.params' -- -extld ... (remaining 6 arguments skipped)
Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
external/go_sdk/pkg/tool/darwin_arm64/link: running external/local_config_cc/cc_wrapper.sh failed: exit status 1
ld: framework not found CoreFoundation
clang: error: linker command failed with exit code 1 (use -v to see invocation)
```
clang 11 for shell reason: https://github.com/NixOS/nixpkgs/issues/166205
## Test plan
`./dev/scip-ctags-install.sh` and `bazel build //:gazelle`
Due to a newly enabled (in bazel 7) `--incompatible_sandbox_hermetic_tmp`, and nix setting TMP/TMPDIR to something below /tmp, we encounter https://github.com/bazelbuild/bazel/issues/5900
## Test plan
`nix develop --command bazel build //dev/sg` is successfull
standard bash is a non-interactive shell meant for a build environment instead of an interactive environment.
Also bumping nixpkgs while we're here
## Test plan
run `bash` in `nix develop`
Use [esbuild](https://esbuild.github.io/) instead of Webpack for builds of `client/web`, for faster builds (dev and prod) and greater dev-prod parity. This PR completely removes all use of Webpack in this repository.
`client/web` is the last build target that still uses Webpack; all others have been recently migrated to esbuild. Most devs here have been using esbuild for local dev of `client/web` for the last 6-12 months anyway. The change here is that now our production builds will be built by esbuild.
All sg commands, integration/e2e tests, etc., continue to work as-is. The bundlesize report will take a while to stabilize because the new build products use different filenames.
## Benchmarks
Running `pnpm run generate && time pnpm -C client/web run task:gulp webBuild` and taking the `time` output from the last command:
- Webpack: 62.5s
- esbuild: 6.7s
Note: This understates esbuild's victory for 2 reasons: (1) because esbuild is building both the main and embed entrypoints, whereas Webpack only builds the main entrypoint in this benchmark) and (2) because a lot of it is in the fixed startup time of `gulp`; esbuild incremental rebuilds during local dev only take ~1s.
## Notes
We no longer use Babel to produce web builds (we use esbuild), so we don't need any Babel plugins that optimize the output or improve browser compatibility. Right now, Babel is only used by Jest (for tests) and by Bazel as an intermediate step.
It has been broken for me since the nix changes a few weeks ago. I've
mostly been devving on linux so I've been ignoring it.
Test Plan: nix develop on my mbp and linux
* Bump NodeJS to v18
v16 is completely without support now, so finally making the switch to v18!
sg start and building the server image still seem to work - so I'm assuming this is fine.
Bazel question: Where does bazel get it's nodejs version from? Is this covered in this diff?
* update rules_js,rules_ts for node 18
* add rules_ts bazelrc config
* update test timeouts
* use 6.0.1 rules_nodejs and 18.17.1 nodejs tls
---------
Co-authored-by: William Bezuidenhout <william.bezuidenhout@sourcegraph.com>
In our flake we expose a bunch of statically compiled binaries. We
introduced this to help us ship those binaries that are quite hard to
build outside of nix. With our recent re-organization we started using
them in the devenv. The downside of this is we now have to do things
like recompile ocaml. Even on my beast linux desktop this feels like
overkill.
This commit switches us to using the normal nix packages. Additionally
it:
- removes p4-fusion from our devenv since no nix users dev with it.
- re-introduces the universal-ctags wrapper
Test Plan: checked ctags in the devenv:
``` shell
nix develop
file $CTAGS_COMMAND
which universal-ctags
```
I had some bazel tests commands which failed to find the "find" binary.
Additionally it was not respecting my configured PG* settings. This
makes it so we pass those envvars on. To be honest, I'm surprised this
isn't the default in a devenv.
Test Plan: bazel test //internal/database:database_test
Re-organizing a couple of things to make the individual files nicer,
consolidated some things in ctags.nix, extracted forEachSystem stuff to
flake.nix instead of the individual files, overlay for our specific
nodejs requirements, some more comments to stuff
## Test plan
N/A, nix stuff
Uses zig-cc as a hermetic c/c++ toolchain, and wraps bazel in an FHS
environment for nixos. This lets us drop some hacks and workarounds
needed to address gaps in hermeticity in some bazel-provisioned tools
(mostly around FHS env requirements)
## Test plan
Built :allthethings: many times over many days. Not tested on darwin but
we're deferring entirely to bazelisk there anyways
We only did this for bazel but not ibazel. Found this due to having
bazel build work for me but not ibazel.
Test Plan: sg run syntax-highlighter works on nix darwin
Can't explain why but for some reason I'm not getting libstdc++ issues
in the bazel tools such as `build-runfiles`, so lets drop it so we can
use the binary cache instead of building locally
## Test plan
N/A, nix stuff
Allows formatting everything with `nix fmt`
[nil](https://github.com/oxalica/nil) appears more maintained and higher
quality than rnix-lsp
## Test plan
N/A, nix stuff
Bazel complains when theres even a patch mismatch (unstable is on 6.1.2,
.bazelversion says 6.1.1). Might mask issues e.g. if unstable moves to
7.0.0 before we move there, but we'll likely see that fail audibly then
## Test plan
N/A, nix stuff
Some sprinkling of special sauce for the NixOS (and nix shell) users in
the house. Much care has been taken so that there should be no
interference with anyone running bazel on a non-nixos machine, and that
it also works as (mostly) expected for nix shell users on macos.
## Test plan
Ran main dry-run, all green, so it shouldnt interfere with anyone on a
non-nixos machine
Some sg run targets expect bazelisk to be on the path. So we go from
only exposing it as "bazel" to putting it on the PATH as both bazel and
bazelisk.
Test Plan: sg start enterprise-bazel works
All but 2 of the linters are running in bazel now. The two linters that
are missing are:
* goimports - have to execute a binary which then modifies the source
which is not very bazelish
* unconvert - the whole package is private and pretty much wants you to
execute it as a binary ... which is unbazelish
## Test plan
Check that the go linters are not running in CI
<!-- All pull requests REQUIRE a test plan:
https://docs.sourcegraph.com/dev/background-information/testing_principles
-->
This is the result of William and I debugging getting bazel to work on
my nix-darwin managed M2 macbook to work with bazel. This is the output
of a full morning of work :)
The root cause of the issue we ran into was we could not get bazel to
compile protobuf, it would fail to find the C++ std headers. We had many
misteps here, it turned out that unsetting CC and CXX would lead to
bazel using the system installed toolchain. We are unsure why we can't
use nix's clang-wrapped, maybe it has to do with that respecting special
NIX_ envvars for building nix packages?
This might break bazel on NixOS, but it wasn't working for me anyways. I
imagine it should continue to work on nix on linux.
Test Plan: "bazel test //cmd/frontend/internal/search:search_test" and
"bazel configure" works.
Very logically (not), the existing method doesnt actually work. This resulted in there being both nodejs18 and nodejs16 at work, with nodejs18 being used when running sg start (as evident by the warning log).
## Test plan
Checked the paths:
```
$ cat (which pnpm)
...
exec -a "$0" "/nix/store/w8vn7zcdjk9bx0qcfw349v6bhf362zag-pnpm-7.24.2/bin/.pnpm-wrapped" "$@"
$ cat /nix/store/w8vn7zcdjk9bx0qcfw349v6bhf362zag-pnpm-7.24.2/bin/.pnpm-wrapped
#!/nix/store/48qwp6zx3gm8b89m812vn8h63kmnqb12-nodejs-16.19.0/bin/node
...
$
```
Had to move to unstable for this, but everything seems to be working. I
wonder if we should introduce both stable and unstable pkgs? IE it kind
of sucks needing to download 4G from the cache every update (which I
think is mostly bazel?).
Test Plan: nix develop
Things seem to be progressing fast, lets get bazel installed for the nix
users. One downside is this adds a huge amount of stuff to fetch from
the cache (I'm looking at you java headless sdk).
Test Plan: nix develop followed by bazel version working.
We add the optional dependency git-lfs to any image gitserver may run
on. This is still an experimental feature, but is a requirement when
enabled.
Additionally we include it in the nix devenv since its cheap to add.
Note: git-lfs is a relatively small go binary and does not require
specific versions of git. Additionally git will not automatically use
it, so its presence will not change behaviour.
Test Plan: CI docker image dry run
