Tired of seeing the go toolchain being easier to use than nix.
Test Plan: nix develop on linux amd64 and macbook arm64 followed by
running "go test ./internal/search" working. Also confirming that "go
env GOROOT" points into the nix store.
Turns out we can do this after all! And thankfully so, because our buildkite runners have an older glibc version than what Nix was building our bins against which was resulting in `dropdb: /lib/x86_64-linux-gnu/libc.so.6: version GLIBC_2.32' not found (required by dropdb)` when bazel was running in buildkite
## Test plan
Much local `nix build`'ing, `ldd` and `patchelf`'ing
Lets get rid of PG_UTILS_PATH! :allthethings: 🎉
## Test plan
CI and a _lot_ of `nix build .#pg-utils && ldd ./result/bin/pg_dump`
also tested e2e with the following diff: 61e0eb0d55
Using my fork with open PR: https://github.com/NixOS/nixpkgs/pull/295615
Im needing 7.1 sooner than later, given its new features and this repo also using 7.1 (so all new features are fair game for it!)
Cached version is pushed to sourcegraph-noah.cachix.org, which I've added at the top here so you too can avoid building it locally 🙂
## Test plan
Tested locally 😎
See https://github.com/orgs/Homebrew/discussions/4686#discussioncomment-6628463
Thanks @jac !
## Test plan
- Ensure CI builds correctly
<!-- All pull requests REQUIRE a test plan: https://docs.sourcegraph.com/dev/background-information/testing_principles
Why does it matter?
These test plans are there to demonstrate that are following industry standards which are important or critical for our customers.
They might be read by customers or an auditor. There are meant be simple and easy to read. Simply explain what you did to ensure
your changes are correct!
Here are a non exhaustive list of test plan examples to help you:
- Making changes on a given feature or component:
- "Covered by existing tests" or "CI" for the shortest possible plan if there is zero ambiguity
- "Added new tests"
- "Manually tested" (if non trivial, share some output, logs, or screenshot)
- Updating docs:
- "previewed locally"
- share a screenshot if you want to be thorough
- Updating deps, that would typically fail immediately in CI if incorrect
- "CI"
- "locally tested"
-->
As detailed in the shell hook:
Without this, zig cc is forced to rebuild on every sandboxed GoLink action, which adds ~1m of time to GoLink actions. The reason it's on _every_ GoLink action is because sandboxes are ephemeral and don't persist non-mounted paths between actions.
We're already doing this for `darwin-docker` config, so this only affects us nixos users
---
Smuggling in a libtool fix for macos that would otherwise yield the following error when building rust with bazel:
<details>
<summary>Details</summary>
```
cargo:rerun-if-env-changed=AR_aarch64-apple-darwin
AR_aarch64-apple-darwin = None
cargo:rerun-if-env-changed=AR_aarch64_apple_darwin
AR_aarch64_apple_darwin = None
cargo:rerun-if-env-changed=HOST_AR
HOST_AR = None
cargo:rerun-if-env-changed=AR
AR = Some("/nix/store/42yck6r7y2jhcrd0ay0glz30w6pw4wzh-libtool-2.4.7")
cargo:rerun-if-env-changed=ARFLAGS_aarch64-apple-darwin
ARFLAGS_aarch64-apple-darwin = None
cargo:rerun-if-env-changed=ARFLAGS_aarch64_apple_darwin
ARFLAGS_aarch64_apple_darwin = None
cargo:rerun-if-env-changed=HOST_ARFLAGS
HOST_ARFLAGS = None
cargo:rerun-if-env-changed=ARFLAGS
ARFLAGS = None
running: ZERO_AR_DATE="1" "/nix/store/42yck6r7y2jhcrd0ay0glz30w6pw4wzh-libtool-2.4.7" "cq" "/private/var/tmp/_bazel_noah/dcf2fbfa8ce2981c9fc4201fa6327d3b/sandbox/darwin-sandbox/6091/execroot/__main__/bazel-out/darwin_arm64-fastbuild/bin/external/crate_index__tree-sitter-kotlin-0.2.11/tree-sitter-kotlin_build_script.out_dir/libparser.a" "/private/var/tmp/_bazel_noah/dcf2fbfa8ce2981c9fc4201fa6327d3b/sandbox/darwin-sandbox/6091/execroot/__main__/bazel-out/darwin_arm64-fastbuild/bin/external/crate_index__tree-sitter-kotlin-0.2.11/tree-sitter-kotlin_build_script.out_dir/src/parser.o" "/private/var/tmp/_bazel_noah/dcf2fbfa8ce2981c9fc4201fa6327d3b/sandbox/darwin-sandbox/6091/execroot/__main__/bazel-out/darwin_arm64-fastbuild/bin/external/crate_index__tree-sitter-kotlin-0.2.11/tree-sitter-kotlin_build_script.out_dir/src/scanner.o"
--stderr:
error occurred: Command ZERO_AR_DATE="1" "/nix/store/42yck6r7y2jhcrd0ay0glz30w6pw4wzh-libtool-2.4.7" "cq" "/private/var/tmp/_bazel_noah/dcf2fbfa8ce2981c9fc4201fa6327d3b/sandbox/darwin-sandbox/6091/execroot/__main__/bazel-out/darwin_arm64-fastbuild/bin/external/crate_index__tree-sitter-kotlin-0.2.11/tree-sitter-kotlin_build_script.out_dir/libparser.a" "/private/var/tmp/_bazel_noah/dcf2fbfa8ce2981c9fc4201fa6327d3b/sandbox/darwin-sandbox/6091/execroot/__main__/bazel-out/darwin_arm64-fastbuild/bin/external/crate_index__tree-sitter-kotlin-0.2.11/tree-sitter-kotlin_build_script.out_dir/src/parser.o" "/private/var/tmp/_bazel_noah/dcf2fbfa8ce2981c9fc4201fa6327d3b/sandbox/darwin-sandbox/6091/execroot/__main__/bazel-out/darwin_arm64-fastbuild/bin/external/crate_index__tree-sitter-kotlin-0.2.11/tree-sitter-kotlin_build_script.out_dir/src/scanner.o" with args "" failed to start: Os { code: 13, kind: PermissionDenied, message: "Permission denied" }
```
</details>
## Test plan
Observing `bazel build //dev/sg` times
standard bash is a non-interactive shell meant for a build environment instead of an interactive environment.
Also bumping nixpkgs while we're here
## Test plan
run `bash` in `nix develop`
Currently on non-nixos hosts we will append to .bazelrc-nix everytime
the shell hook runs. This leads to the file growing quite large, I am
unsure on the impact on bazel. This was an accidental regression when we
removed bazel from the nix darwin environment.
Test Plan: ran nix develop and I didn't have duplicated lines.
This is the latest release from December 2022. I see important bug fixes
between our tagged version and this release.
A follow up commit needs to update the bazel shas once CI has built the
artifacts on main.
Test Plan: CI
* only upload when a change to ctags nix is pushed to main
* change to trigger workflow
* use apple specific shasum
* binaries get uploaded when branch matches
* use github.head_ref
* use ref_name for push events
* add short sha to filename
We want to get newer stuff in the devshell, but keep a stabler pin for the static binaries (until darwin stdenv stabilizes, most notably the big brain work over at https://github.com/NixOS/nixpkgs/pull/256590)
## Test plan
N/A nix stuff :clueless:
Use nix to compile and upload universal-ctags to a gcloud bucket. We then pull this down using a bazel http_file and can run it with bazel also. Removed all of the old bash logic that was hard to follow and maintain also.
---------
Co-authored-by: Noah Santschi-Cooney <noah@santschi-cooney.ch>
Co-authored-by: William Bezuidenhout <william.bezuidenhout@sourcegraph.com>
I had some bazel tests commands which failed to find the "find" binary.
Additionally it was not respecting my configured PG* settings. This
makes it so we pass those envvars on. To be honest, I'm surprised this
isn't the default in a devenv.
Test Plan: bazel test //internal/database:database_test
Re-organizing a couple of things to make the individual files nicer,
consolidated some things in ctags.nix, extracted forEachSystem stuff to
flake.nix instead of the individual files, overlay for our specific
nodejs requirements, some more comments to stuff
## Test plan
N/A, nix stuff
Uses zig-cc as a hermetic c/c++ toolchain, and wraps bazel in an FHS
environment for nixos. This lets us drop some hacks and workarounds
needed to address gaps in hermeticity in some bazel-provisioned tools
(mostly around FHS env requirements)
## Test plan
Built :allthethings: many times over many days. Not tested on darwin but
we're deferring entirely to bazelisk there anyways
When using nix-ld instead of symlinking /lib64/ld-linux-x86-64.so.2 to
glibc's ld.so, we need to pass through NIX_LD and NIX_LD_LIBRARY_PATH
for it to work. cargo-bazel doesnt pass through stuff by default, so we
have to do it ourselves
## Test plan
N/A, nix stuff
This is to avoid the macos firewall popups. Also its just better to use
localhost for local dev.
Test Plan: Killed the running server then ensured the started server was
listening on localhost
Some sprinkling of special sauce for the NixOS (and nix shell) users in
the house. Much care has been taken so that there should be no
interference with anyone running bazel on a non-nixos machine, and that
it also works as (mostly) expected for nix shell users on macos.
## Test plan
Ran main dry-run, all green, so it shouldnt interfere with anyone on a
non-nixos machine
## How to build
`nix build .#ctags .#p4-fusion .#comby` with outputs in
`./result{,-1,-2}/bin`
## Static? Prove it
Linux:
```
$ ldd ./result/bin/comby ./result-1/bin/p4-fusion ./result-2/bin/ctags
./result/bin/comby:
not a dynamic executable
./result-1/bin/p4-fusion:
not a dynamic executable
./result-2/bin/ctags:
not a dynamic executable
```
Darwin (libiconv and libxml are in dylib cache, libSystem.B cannot be
statically linked, and the rest are core system libraries):
```
otool -L ./result-1/bin/p4-fusion ./result/bin/comby ./result-2/bin/ctags
./result-1/bin/p4-fusion:
/System/Library/Frameworks/CFNetwork.framework/Versions/A/CFNetwork (compatibility version 1.0.0, current version 1209.1.0)
/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 1770.255.0)
/System/Library/Frameworks/Cocoa.framework/Versions/A/Cocoa (compatibility version 1.0.0, current version 23.0.0)
/System/Library/Frameworks/Security.framework/Versions/A/Security (compatibility version 1.0.0, current version 59754.60.13)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1292.60.1)
./result/bin/comby:
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1292.60.1)
./result-2/bin/ctags:
@rpath/libxml2.2.dylib (compatibility version 13.0.0, current version 13.3.0)
@rpath/libiconv.dylib (compatibility version 7.0.0, current version 7.0.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1292.60.1)
```
Review with whitespace changes hidden
## Test plan
Built many many times, checked ldd & otool many many times
Some of our tests seem to depend on this environment variable being set.
It not being set doesn't lead to them failing, but rather a lot of log
spam. Setting this avoids the logspam.
Test Plan: go test ./internal/repos has less noise
