This is in preperation for the 5.3 release.
19fa44cea9...245e0cebf7
- fdc144f8fd all: gofmt -s -w .
- 746f38849d matchtree: fix panic for missing files
- 62017762fc gomod: update mountinfo to latest
- 0ddb91fea5 all: use stdlib slices package
- 7ec3d8e8bf gomod: update grpc for GHSA-m425-mq94-257g
- 245e0cebf7 gomod: update otel and circl for CVEs
Test Plan: CI and the relevant CVEs we solved no longer appear when
running "trivy fs go.mod".
Last time we changed the zoekt module name it broke the wire protocol
for zoekt. This now includes one more commit such that the wire protocol
is unchanged:
- c9182fcd2a rpc: use old module name when registering gob values
Revert chain:
- #40404
- #40353
Test Plan: Ran a local dev server with a hardcoded zoekt-webserver
pinned to the old version (the google/zoekt module). indexed search was
failing. Then upgraded the gomod version and it started working.
This reverts commit 6d376d871a.
See #inc-134-search-on-sourcegraphcom-fails-with-errors in Slack.
Current suspicion is that it causes gob-encoding/decoding problems in
production if the code below is not in sync with the zoekt cluster code.
We have updated the module name in the zoekt repo to our own repository name.
This means we can remove the replace directive and just rely on normal go
tooling.
Note: at the same time we rename the default branch for zoekt from master to
main.
Test Plan: master dry run in CI.
This greatly speeds up the script since it avoids the use of using
GOPROXY=direct. It also allows us to inline the suggested PR and commit
descriptions. The intention here is we can then start using this output
to improve the automated PRs created by the bot.
Additionally I changed the command we test to be an package which is
faster to compile.
This is some over engineering thanks to me having only 1h before a
chore.
Included zoekt changes:
- 16b6b7e50c all: remove datadog support
- f7f68fd7f3 ci: cleanup whitespace
Test Plan: ran the script and included the update to gomod here.
Includes the following commits. In particular we are interested to see
if our commit which removes python from the image removes a CVE related
to expat.
- 9d1b56e change findShard implementation to match on IDs, not names
- f6ba702 ctags: bump version to latest
- 840d5e4 docs: document JSON API
- 5f70fa0 fix: sourcegraphFake only checks SG_PRIVATE
- abedc19 introduce BLOCK_PROFILE_RATE environment variable to control the sampling rate of Go's block profiler
- acb6f19 use the ctags build script from sourcegraph
- 9e4c394 indexserver: track number of repository options we fetch
- 6b1df4f build ctags on alpine 3.11
- 65df08f specify docker tag versions
- b87be87 debug: add more facilities for debugging
- d19dfcd remove zoekt-merge-small.py
- 27e54f1 Add support for JSON responses in the webserver
- 574b805 indexserver: return uint from MaybeRemoveMissing
- 512672d indexserver: track num of repos we stop tracking
- 5bbbd4e merging: run vacuum every 24h instead of every hour
- d11a068 merging: remove obsolete maxSize
- 8ec61a4 merging: support unsetting tombstones during cleanup
Test Plan: main dry run for integration tests and to inspect the trivy
report to see if expat CVE is gone.
Quite a few changes since our last bump:
- ae47cfc indexserver: ensure indexed for unchanged repositories
- 905148e indexserver: set and read Config Fingerprint
- 8cc3582 indexserver: Sourcegraph interface is just List
- d89a313 indexserver: move CloneURL into IndexOptions
- bfeddd7 indexserver: delete meta file after merging
- cd787b0 update README to reflect we are the defacto source
- f8682c1 shards: stream results per repo for compound shards
- ba7cd37 indexserver: merge in order of priority
- 004c956 shards: send max pending priority while searching
- df63d76 indexserver: don't delete shards if merge didn't run
- 4c69cfb indexserver: automate merging
- 9d85eab indexserver: update merge policy
- 0e81eb3 shards: set SearchResult.Progress.Priority
- c79ecf9 shards: only `observeMetrics` when `r.err == nil`
- f493172 shards: buffer search channel
We also update the update script to suggest a command which strips out
the PR numbers from the commit message. This is done since GitHub
interprets them as PRs in the SG repo, but they are in the zoekt repo.
I ran into issues setting up Sourcegraph on NixOS, because NixOS, like
some other distros, doesn't have a /bin/bash. We already use /usr/bin/env
in many of our scripts, so this improves consistency, too.
We also update go-git and go-ctags like we did in zoekt.
- 0ff6088 indexserver: send sigquit before sigkill to stalled index
- c8b1e5f ctags: timeout parse requests after a minute
- d50ed97 Merge remote-tracking branch 'gerrit/master'
- e056565 eval: fix bug where /(foo){2,}/ was equivalent to /foo/
- 6e5e87c ngramoffset: add a dense asciiNgramOffset mapper and a combiner.
- a6adac0 all: rely on priority in RawConfig only
- a6a7c57 Merge remote-tracking branch 'gerrit/master'
- 5118231 indexbuilder: remove subrepo branch validation
- f560be7 Guard against invalid slice bound
- 0a98eee indexbuilder: avoid writing root in SubRepoMap
- 253a9c5 index: pre-allocate size of ngram map
- d5ee8b0 index: read in ngrams in own function
- 68ffcf0 web: serve robots.txt, disallowing /search
- 0d5782d Revert "add back subrepo branch validation"
This commit bumps the version of zoekt to include one new commit:
https://github.com/sourcegraph/zoekt/commit/4c31790 shards: introduce a
scheduler abstraction.
We enable this scheduler in development mode.
This converts the API into a batch API by writing a JSON blob per "repo"
query parameter in the order they are specified.
Each zoekt indexserver currently queries this endpoint with a hardcoded
concurrency of 32. This can cause many many HTTP requests, which on a
linux machine without tuning can lead to socket errors. By converting
this into a batch API we reduce the number of connections as well as
moving the responsbility of concurrency to the frontend which has more
knowledge of load/etc.
We also remove backwards compatibility support where repo wasn't
specified. Repo has been specified for multiple releases now.
See 6ed80895db...a99799fdc0
- a99799f ctags: handle fatal ctags errors
- 5fcad78 ctags: skip commands that are too large
- cf36be0 ctags: use file basename
- d4fecb4 indexserver: timeout if no output for 5m
- 34a0dd6 indexserver: coarse 1h timeout on index job
- fdc9c2c Merge remote-tracking branch 'gerrit/master'
- b48e35d shards: a progress message every 10s when loading
- 4481f7b shards: throttle loading in watcher
* gomod: update zoekt
Includes the following commits
- 9d90c1c gitindex: allocate buffer of correct size
- aa0e667 zoekt: verbose flag prints shard metadata
- ddddd93 gitindex: do not resolve HEAD ref
- 8c0aee7 build: handle nil Options in largeFilesFlag
- 36a801c indexserver: use temp directory on same mount as indexes
- 070c191 indexserver: align flag defaults with dockerfile
- 35dabc7 indexserver: optionally shallow clone and use zoekt-git-index
- f89b8e3 indexserver: inline CmdArgs into archiveIndex
- ef52f4b indexserver: consistently use a pointer to indexArgs
- e407d68 indexserver: store root in indexArgs
- 70628e0 indexserver: factor out index logic
- 4961ee9 indexserver: add cli flags to debug indexing and listing
- 4abd60f indexserver: log success
- 605d890 indexserver: metrics track if we skipped indexing
- 1a62bb8 indexserver: rename metric vars to have metric prefix
- a2233e1 indexserver: check incremental in process
- ddccdf9 indexserver: use build.Options to generate flags
- 8463ade Do not ignore large files.
* zoekt: set cpu_fraction flag for indexserver
The default was 0.25, but was bumped to 1.0. This bump was done to align
the flag defaults with how we run indexserver in cluster
environments. We update our dev and single image environments to use the
same value. Those environmemts don't have resource isolation so we need
to continue using 0.25 rather than allowing indexing to use all cores.
* all: build and include zoekt-git-index
If you are indexing multiple branches this command is required. This
adds the command to our dev and single docker image environments. The
docker images for cluster already include the command (see zoekt repo).
* gofmt
We update our zoekt wrapper to understand a replica number. If
DEV_SEARCH_SHARDING is set we then start up two replicas locally, each
pointing to `~/.sourcegraph/zoekt/index-$replica`.
