sourcegraph/cmd/syntactic-code-intel-worker
Anton Sviridov 2821447ae5
Syntactic indexing: enqueuer and scheduler (#62485)
This PR introduces three main components:

    Enqueuer – a thin layer responsible for actually inserting the syntactic indexing records into the database.
    Scheduler - a service that
        Identifies repositories that haven't been processed in a while
        Identifies policies that match those repositories (policies that have syntactic indexing enabled)
        Identifies commits that match any of the policies
        And finally, enqueues the jobs to index the discovered repositories and commits
    Scheduler job – a periodic routine that triggers Scheduler on with specified interval. This job runs as part of the main Worker service, and only schedules jobs if the experimental syntactic indexing feature is enabled.

Refactoring:

    Making some methods public in policies.Service to make it easier to test logic that depends on glob matching of repository names (this matching requires a separate state to be updated)
    Extracting some test utilities into a separate package

---------

Co-authored-by: Varun Gandhi <varun.gandhi@sourcegraph.com>
2024-05-31 10:25:30 +01:00
..
shared Syntactic indexing: enqueuer and scheduler (#62485) 2024-05-31 10:25:30 +01:00
BUILD.bazel bazel: transcribe test ownership to bazel tags (#62664) 2024-05-16 15:51:16 +01:00
CODENOTIFY Batch indexing: syntactic codeintel worker scaffolding (#59747) 2024-01-24 13:04:20 +00:00
image_test.yaml highlighter: Rename scip-treesitter-cli -> scip-syntax (#60077) 2024-02-05 13:15:09 +01:00
main.go Batch indexing: syntactic codeintel worker scaffolding (#59747) 2024-01-24 13:04:20 +00:00
README.md Batch indexing: syntactic codeintel worker scaffolding (#59747) 2024-01-24 13:04:20 +00:00

Syntactic code intel worker

🚧 WORK IN PROGRESS 🚧

Stateless service that handles generating SCIP data for codebases using Tree-sitter for powering syntax-based code navigation.

Design docs (Sourcegraph internal)