mirror of
https://github.com/sourcegraph/sourcegraph.git
synced 2026-02-06 15:31:48 +00:00
cd38adb4a7
Fixes SPLF-119 This adds a background job to Search Jobs that periodically scans for finished jobs to aggregate the status, upload logs, and clean up the tables. This drastically reduces the size of the tables and improves the performance of the Search Jobs GQL API. For example, with this change, a finished search job on .com only has 1 entry in the database, whereas before it could have several millions if we searched each repository. Notes: - the diff seems larger than it actually is. I left a couple of comments to help the reviewers. ## Test plan: - new unit tests - manual testing: I ran a couple of search jobs locally (with the janitor job interval set to 1 min) and checked that - logs are uploaded to `blobstore-go/buckets/search-jobs` - repo jobs are deleted from `exhaustive_repo_jobs` - logs are served from the blobstore after the janitor ran - downloading logs while the job is running still works ## Changelog The new background job drastically reduces the size of the `exhaustive_*` tables and improves performance of the Search Jobs GQL API. |
||
|---|---|---|
| .. | ||
| service | ||
| store | ||
| types | ||
| BUILD.bazel | ||
| conf_test.go | ||
| conf.go | ||