artin/sourcegraph
1
0
Fork 0
mirror of https://github.com/sourcegraph/sourcegraph.git synced 2026-02-06 17:31:43 +00:00
Code Issues Packages Projects Releases Wiki Activity
850291aa58
sourcegraph/internal/database
History
Anton Sviridov 2821447ae5
Syntactic indexing: enqueuer and scheduler (#62485) 
This PR introduces three main components:

    Enqueuer – a thin layer responsible for actually inserting the syntactic indexing records into the database.
    Scheduler - a service that
        Identifies repositories that haven't been processed in a while
        Identifies policies that match those repositories (policies that have syntactic indexing enabled)
        Identifies commits that match any of the policies
        And finally, enqueues the jobs to index the discovered repositories and commits
    Scheduler job – a periodic routine that triggers Scheduler on with specified interval. This job runs as part of the main Worker service, and only schedules jobs if the experimental syntactic indexing feature is enabled.

Refactoring:

    Making some methods public in policies.Service to make it easier to test logic that depends on glob matching of repository names (this matching requires a separate state to be updated)
    Extracting some test utilities into a separate package

---------

Co-authored-by: Varun Gandhi <varun.gandhi@sourcegraph.com>
2024-05-31 10:25:30 +01:00
..
basestore Replace all traditional for-loops (#60988) 2024-03-11 16:05:47 +02:00
batch Replace all traditional for-loops (#60988) 2024-03-11 16:05:47 +02:00
connections all: use observation.TestContextTB instead of TestContext (#61751) 2024-04-10 14:07:39 +02:00
dbcache Replace all traditional for-loops (#60988) 2024-03-11 16:05:47 +02:00
dbconn feat/enterprise-portal: DB layer for {Get/List}CodyGatewayAccess (#62706) 2024-05-22 12:56:59 -07:00
dbmocks Remove localclone store (#61994) 2024-04-29 23:51:26 +02:00
dbtest Syntactic indexing: enqueuer and scheduler (#62485) 2024-05-31 10:25:30 +01:00
dbutil codenotify: dr. fritz is kil (#57313) 2023-10-03 16:19:55 +00:00
fakedb Replace all traditional for-loops (#60988) 2024-03-11 16:05:47 +02:00
locker Chore: construct logger inside dbtest.NewDB (#57549) 2023-10-11 20:41:11 -05:00
migration lib/background: upgrade Routine interface with context and errors (#62136) 2024-05-24 10:04:55 -04:00
postgresdsn codenotify: dr. fritz is kil (#57313) 2023-10-03 16:19:55 +00:00
access_requests_test.go Improve cursor based pagination implementation (#58766) 2023-12-05 13:06:43 +01:00
access_requests.go [Request access] Add capturing of decision maker admin user id (#48496) 2023-03-02 12:31:39 +03:00
access_tokens_test.go Update Access token defaults (#60190) 2024-02-05 21:28:06 -05:00
access_tokens.go New setting to specify the maximum number of active access tokens a user is allowed (#59731) 2024-01-23 13:42:12 -05:00
assigned_owners_test.go Chore: construct logger inside dbtest.NewDB (#57549) 2023-10-11 20:41:11 -05:00
assigned_owners.go logger: update log lib and remove use of description (#57690) 2023-10-18 17:29:08 +02:00
assigned_teams_test.go Chore: construct logger inside dbtest.NewDB (#57549) 2023-10-11 20:41:11 -05:00
assigned_teams.go logger: update log lib and remove use of description (#57690) 2023-10-18 17:29:08 +02:00
authenticator_test.go Standardize database encryption (#40050) 2022-08-08 20:29:09 +00:00
authenticator.go encryption: Lazily decrypt batch_changes_site_credentials (#40228) 2022-08-15 15:58:13 +00:00
authz_test.go Remove watcher for permission mapper (#60809) 2024-03-12 23:12:36 +01:00
authz.go Remove watcher for permission mapper (#60809) 2024-03-12 23:12:36 +01:00
bitbucket_project_permissions_test.go Chore: construct logger inside dbtest.NewDB (#57549) 2023-10-11 20:41:11 -05:00
bitbucket_project_permissions.go internal/database: Remove unused transact func (#47312) 2023-02-02 11:41:30 +00:00
BUILD.bazel chore: Move helper function to database package (#62543) 2024-05-08 17:25:01 +02:00
code_hosts_test.go Chore: construct logger inside dbtest.NewDB (#57549) 2023-10-11 20:41:11 -05:00
code_hosts.go Periodic Rate Limit config job (#55558) 2023-08-21 20:41:40 +00:00
code_monitor_action_jobs_test.go Backend: remove EnterpriseDB (#54699) 2023-07-06 20:03:31 -06:00
code_monitor_action_jobs.go Backend: remove EnterpriseDB (#54699) 2023-07-06 20:03:31 -06:00
code_monitor_emails_test.go monitors: show every users' code monitors to admins (#54981) 2023-07-24 12:11:15 +00:00
code_monitor_emails.go monitors: show every users' code monitors to admins (#54981) 2023-07-24 12:11:15 +00:00
code_monitor_last_searched_test.go Chore: construct logger inside dbtest.NewDB (#57549) 2023-10-11 20:41:11 -05:00
code_monitor_last_searched.go Backend: remove EnterpriseDB (#54699) 2023-07-06 20:03:31 -06:00
code_monitor_monitors.go code-monitors: handle soft-deleted users (#60405) 2024-02-13 14:30:39 +01:00
code_monitor_queries_test.go Backend: remove EnterpriseDB (#54699) 2023-07-06 20:03:31 -06:00
code_monitor_queries.go monitors: show every users' code monitors to admins (#54981) 2023-07-24 12:11:15 +00:00
code_monitor_recipient_test.go Backend: remove EnterpriseDB (#54699) 2023-07-06 20:03:31 -06:00
code_monitor_recipients.go Backend: remove EnterpriseDB (#54699) 2023-07-06 20:03:31 -06:00
code_monitor_slack_webhook_test.go Chore: construct logger inside dbtest.NewDB (#57549) 2023-10-11 20:41:11 -05:00
code_monitor_slack_webhook.go monitors: show every users' code monitors to admins (#54981) 2023-07-24 12:11:15 +00:00
code_monitor_test.go Backend: remove EnterpriseDB (#54699) 2023-07-06 20:03:31 -06:00
code_monitor_trigger_jobs_test.go code monitors: don't fail job if repo empty (#61367) 2024-04-02 09:30:59 +02:00
code_monitor_trigger_jobs.go code monitors: don't fail job if repo empty (#61367) 2024-04-02 09:30:59 +02:00
code_monitor_webhook_test.go Chore: construct logger inside dbtest.NewDB (#57549) 2023-10-11 20:41:11 -05:00
code_monitor_webhook.go monitors: show every users' code monitors to admins (#54981) 2023-07-24 12:11:15 +00:00
code_monitors.go Move record encrypter to worker package (#61900) 2024-04-16 10:16:58 +02:00
CODENOTIFY chore: update Codenotify subscriptions for Joe (#46978) 2023-01-26 11:27:00 +00:00
codeowners_test.go Replace all traditional for-loops (#60988) 2024-03-11 16:05:47 +02:00
codeowners.go Backend: remove EnterpriseDB (#54699) 2023-07-06 20:03:31 -06:00
conf_test.go Improve cursor based pagination implementation (#58766) 2023-12-05 13:06:43 +01:00
conf.go logger: update log lib and remove use of description (#57690) 2023-10-18 17:29:08 +02:00
database_test.go Chore: construct logger inside dbtest.NewDB (#57549) 2023-10-11 20:41:11 -05:00
database.go Remove localclone store (#61994) 2024-04-29 23:51:26 +02:00
dbstore_db_test.go
doc.go
err_test.go
event_logs_scrape_state_own.go own: integrate signal configurations into resolvers and indexers (#52179) 2023-05-19 10:56:47 -07:00
event_logs_test.go Replace all traditional for-loops (#60988) 2024-03-11 16:05:47 +02:00
event_logs.go fix: update links for dev docs (#62758) 2024-05-17 13:47:34 +02:00
executor_secret_access_logs_test.go Chore: construct logger inside dbtest.NewDB (#57549) 2023-10-11 20:41:11 -05:00
executor_secret_access_logs.go Database: convert Transact to WithTransact for stores that don't use their transactions (#47119) 2023-01-31 08:45:05 -07:00
executor_secrets_internal_test.go Chore: construct logger inside dbtest.NewDB (#57549) 2023-10-11 20:41:11 -05:00
executor_secrets_test.go executor: Fix global secrets across namespaces colliding (#61045) 2024-03-13 13:39:52 +01:00
executor_secrets.go executor: Fix global secrets across namespaces colliding (#61045) 2024-03-13 13:39:52 +01:00
executors_test.go chore: Use min/max builtins over math.Max/math.Min (#62233) 2024-04-29 14:27:28 -04:00
executors.go Executors: enable dequeueing and heartbeating for multiple queues (#52016) 2023-06-04 13:25:05 +00:00
external_accounts_test.go Speed up some slow DB tests (#59131) 2023-12-20 17:42:23 +00:00
external_accounts.go Using new function for security even logs (#59245) 2024-01-03 17:24:44 -05:00
external_services_external_test.go [fix] GH App Auth Validator missing db (#55994) 2023-08-21 11:26:14 +02:00
external_services_test.go dotcom: MockSourcegraphDotComMode requires a T for cleanup (#61172) 2024-03-14 20:27:21 +00:00
external_services.go Remove watcher for permission mapper (#60809) 2024-03-12 23:12:36 +01:00
feature_flags_test.go Feature flags: relax some constraints (#61343) 2024-03-25 10:39:01 -06:00
feature_flags.go Feature flags: relax some constraints (#61343) 2024-03-25 10:39:01 -06:00
gitserver_repos_test.go Remove clone progress DB experiment (#61615) 2024-04-10 21:00:22 +02:00
gitserver_repos.go gitserver: Move repo timestamp updates to caller (#62610) 2024-05-13 14:50:48 +02:00
global_state_test.go Chore: construct logger inside dbtest.NewDB (#57549) 2023-10-11 20:41:11 -05:00
global_state.go jscontext: Use faster, non-blocking method to determine initialized state (#49336) 2023-03-15 01:38:57 +01:00
helpers_test.go chore: Move helper function to database package (#62543) 2024-05-08 17:25:01 +02:00
helpers.go feat: Add new GraphQL API for getting occurrences (#62531) 2024-05-24 09:20:52 +08:00
insights.go Backend: remove enterprise/internal (#54872) 2023-07-12 22:51:02 +02:00
main_test.go deps: upgrade github.com/sourcegraph/log (#41058) 2022-09-01 08:01:59 -07:00
mockerr.go permissions-center: add cancelPermissionsSyncJob mutation. (#48900) 2023-03-08 19:30:57 +04:00
namespace_permissions_test.go Chore: construct logger inside dbtest.NewDB (#57549) 2023-10-11 20:41:11 -05:00
namespace_permissions.go rbac: automate generation of namespaces & actions (#50855) 2023-05-10 18:47:23 +02:00
namespaces_test.go Chore: construct logger inside dbtest.NewDB (#57549) 2023-10-11 20:41:11 -05:00
namespaces.go Database: convert Transact to WithTransact for stores that don't use their transactions (#47119) 2023-01-31 08:45:05 -07:00
org_invitations_test.go Chore: construct logger inside dbtest.NewDB (#57549) 2023-10-11 20:41:11 -05:00
org_invitations.go Database: convert Transact to WithTransact for stores that don't use their transactions (#47119) 2023-01-31 08:45:05 -07:00
org_members_db_test.go Chore: construct logger inside dbtest.NewDB (#57549) 2023-10-11 20:41:11 -05:00
org_members.go users: Remove searchable attribute (#57278) 2023-10-04 15:50:34 +02:00
orgs_test.go Remove unnecessary dotcom checks (#61904) 2024-04-16 13:21:49 +02:00
orgs.go Remove unnecessary dotcom checks (#61904) 2024-04-16 13:21:49 +02:00
outbound_webhook_jobs_test.go [chore] Refactored to use common pointers helper functions (#53441) 2023-06-14 16:48:44 +02:00
outbound_webhook_jobs.go webhooks: wait to enqueue webhook jobs until transaction completes (#49791) 2023-04-01 20:52:55 -07:00
outbound_webhook_logs_test.go Speed up some slow DB tests (#59131) 2023-12-20 17:42:23 +00:00
outbound_webhook_logs.go Database: convert Transact to WithTransact for stores that don't use their transactions (#47119) 2023-01-31 08:45:05 -07:00
outbound_webhooks_test.go Speed up some slow DB tests (#59131) 2023-12-20 17:42:23 +00:00
outbound_webhooks.go database: add outbound webhook tables (#46007) 2023-01-17 16:08:40 -08:00
own_signal_configurations_test.go Chore: construct logger inside dbtest.NewDB (#57549) 2023-10-11 20:41:11 -05:00
own_signal_configurations.go own: integrate signal configurations into resolvers and indexers (#52179) 2023-05-19 10:56:47 -07:00
ownership_stats_test.go chore: remove gotest.tools/assert (#58899) 2023-12-12 12:01:12 -08:00
ownership_stats.go own: don't count excluded paths in ownership analytics. (#54034) 2023-06-26 16:09:19 +04:00
permission_sync_code_host_state_test.go permissions-center: add is_partial_success permissions sync job property. (#50114) 2023-03-30 16:55:06 +04:00
permission_sync_code_host_state.go permissions-center: add is_partial_success permissions sync job property. (#50114) 2023-03-30 16:55:06 +04:00
permission_sync_jobs_test.go Replace all traditional for-loops (#60988) 2024-03-11 16:05:47 +02:00
permission_sync_jobs.go [fix] permission sync is now scheduled correctly for new users (#54258) 2023-06-27 12:46:26 +02:00
permissions_test.go Chore: construct logger inside dbtest.NewDB (#57549) 2023-10-11 20:41:11 -05:00
permissions.go rbac: automate generation of namespaces & actions (#50855) 2023-05-10 18:47:23 +02:00
perms_store_test.go Remove watcher for permission mapper (#60809) 2024-03-12 23:12:36 +01:00
perms_store.go Check for scheduled or processing jobs when scheduling permission syncs (#61024) 2024-03-12 15:04:52 +02:00
phabricator_test.go Chore: construct logger inside dbtest.NewDB (#57549) 2023-10-11 20:41:11 -05:00
phabricator.go Database: convert Transact to WithTransact for stores that don't use their transactions (#47119) 2023-01-31 08:45:05 -07:00
recent_contribution_signal_test.go Chore: construct logger inside dbtest.NewDB (#57549) 2023-10-11 20:41:11 -05:00
recent_contribution_signal.go Chore: remove a handful of unused code (#55686) 2023-08-09 12:52:55 -06:00
recent_view_signal_test.go Speed up some slow DB tests (#59131) 2023-12-20 17:42:23 +00:00
recent_view_signal.go Speed up some slow DB tests (#59131) 2023-12-20 17:42:23 +00:00
redis_key_value_test.go Chore: construct logger inside dbtest.NewDB (#57549) 2023-10-11 20:41:11 -05:00
redis_key_value.go redispool: use postgres for redispool.Store in App (#47188) 2023-02-03 12:59:05 +00:00
repo_commits_changelists_test.go Chore: construct logger inside dbtest.NewDB (#57549) 2023-10-11 20:41:11 -05:00
repo_commits_changelists.go repository changelist resolver: correctly handle changelist id not found (#56646) 2023-09-15 14:08:47 +00:00
repo_kvps_test.go Chore: construct logger inside dbtest.NewDB (#57549) 2023-10-11 20:41:11 -05:00
repo_kvps.go Repo metadata: remove stray printlns (#55135) 2023-07-20 04:43:28 +00:00
repo_paths_test.go Chore: construct logger inside dbtest.NewDB (#57549) 2023-10-11 20:41:11 -05:00
repo_paths.go Replace all traditional for-loops (#60988) 2024-03-11 16:05:47 +02:00
repo_statistics_test.go repo statistics: coalesce when compacting table (#58710) 2023-12-01 16:34:33 +00:00
repo_statistics.go repo statistics: coalesce when compacting table (#58710) 2023-12-01 16:34:33 +00:00
repos_perm_test.go Remove watcher for permission mapper (#60809) 2024-03-12 23:12:36 +01:00
repos_perm.go Remove watcher for permission mapper (#60809) 2024-03-12 23:12:36 +01:00
repos_test.go gitserver: Implement Perforce changelist ID mapper as standalone job (#62868) 2024-05-27 10:00:31 +08:00
repos.go gitserver: Implement Perforce changelist ID mapper as standalone job (#62868) 2024-05-27 10:00:31 +08:00
role_permissions_test.go Replace all traditional for-loops (#60988) 2024-03-11 16:05:47 +02:00
role_permissions.go rbac: don't include OWNERSHIP permissions to USER role by default. (#53121) 2023-06-08 19:03:26 +04:00
roles_test.go Chore: construct logger inside dbtest.NewDB (#57549) 2023-10-11 20:41:11 -05:00
roles.go rbac: change role name to citext (#49109) 2023-03-10 17:36:11 +01:00
saved_searches_test.go Chore: construct logger inside dbtest.NewDB (#57549) 2023-10-11 20:41:11 -05:00
saved_searches.go Tracing: final cleanups (#54694) 2023-07-13 10:16:11 +02:00
schema.codeinsights.json bzl: rework migration schemas generation (#57511) 2023-10-10 17:19:47 +02:00
schema.codeinsights.md Chore: remove some uses of log15 (#57519) 2023-10-10 16:27:52 -06:00
schema.codeintel.json bzl: rework migration schemas generation (#57511) 2023-10-10 17:19:47 +02:00
schema.codeintel.md Chore: remove some uses of log15 (#57519) 2023-10-10 16:27:52 -06:00
schema.json search jobs: add index on state column (#62145) 2024-04-25 09:17:09 +02:00
schema.md search jobs: add index on state column (#62145) 2024-04-25 09:17:09 +02:00
search_contexts_test.go Chore: construct logger inside dbtest.NewDB (#57549) 2023-10-11 20:41:11 -05:00
search_contexts.go search contexts: allow setting a default context and use it on load (#45387) 2022-12-12 09:39:53 -08:00
security_event_logs_test.go Fix: ensure event.Argument is json (#59861) 2024-01-25 15:07:34 +00:00
security_event_logs.go sec: emit security event for RBAC changes (#61787) 2024-04-11 17:49:23 +01:00
settings_test.go Chore: construct logger inside dbtest.NewDB (#57549) 2023-10-11 20:41:11 -05:00
settings.go Tracing: final cleanups (#54694) 2023-07-13 10:16:11 +02:00
sub_repo_perms_store_test.go Chore: construct logger inside dbtest.NewDB (#57549) 2023-10-11 20:41:11 -05:00
sub_repo_perms_store.go clarify wording for error message (#58501) 2023-11-22 15:55:21 +00:00
survey_responses_test.go Chore: construct logger inside dbtest.NewDB (#57549) 2023-10-11 20:41:11 -05:00
survey_responses.go chore: load user only once from the database in UserResolver. (#50228) 2023-04-03 15:51:32 +04:00
teams_test.go Replace all traditional for-loops (#60988) 2024-03-11 16:05:47 +02:00
teams.go handle deleted users when listing team members (#59471) 2024-01-10 23:12:30 +00:00
telemetry_export_store_test.go lib/telemetrygateway: publish all non-Sourcegraph-specific Telemetry Gateway bindings (#62061) 2024-04-22 14:36:46 -07:00
telemetry_export_store.go lib/telemetrygateway: publish all non-Sourcegraph-specific Telemetry Gateway bindings (#62061) 2024-04-22 14:36:46 -07:00
temporary_settings_test.go Chore: construct logger inside dbtest.NewDB (#57549) 2023-10-11 20:41:11 -05:00
temporary_settings.go database: remove dbutil-based constructors (#36210) 2022-05-30 23:48:20 +00:00
user_credentials_test.go Chore: construct logger inside dbtest.NewDB (#57549) 2023-10-11 20:41:11 -05:00
user_credentials.go Database: convert Transact to WithTransact for stores that don't use their transactions (#47119) 2023-01-31 08:45:05 -07:00
user_emails_test.go Chore: construct logger inside dbtest.NewDB (#57549) 2023-10-11 20:41:11 -05:00
user_emails.go KS/set first verified email to primary (#55950) 2023-08-23 10:21:08 +03:00
user_roles_test.go Chore: construct logger inside dbtest.NewDB (#57549) 2023-10-11 20:41:11 -05:00
user_roles.go rbac: implement mutation for assigning roles to a user (#47717) 2023-03-09 13:03:06 +01:00
users_builtin_auth_test.go Refactor ExternalAccountsStore functions (#57509) 2023-10-13 10:41:10 +02:00
users_test.go Close existing access requests when user is created (#59113) 2024-01-10 12:27:13 -08:00
users.go Cleanup Cody Pro pre-feb release stuff (#60940) 2024-03-08 18:16:28 +05:30
util_test.go Chore: replace pointer helpers with pointers package (#55687) 2023-08-09 19:11:06 +00:00
webhook_logs_test.go Chore: construct logger inside dbtest.NewDB (#57549) 2023-10-11 20:41:11 -05:00
webhook_logs.go Add webhook logging to new webhooks handler (#43446) 2022-10-27 09:58:37 -06:00
webhooks_test.go Chore: construct logger inside dbtest.NewDB (#57549) 2023-10-11 20:41:11 -05:00
webhooks.go Remove all unused code specified by linter warnings (#47015) 2023-01-27 13:17:36 +02:00
zoekt_repos_test.go Replace all traditional for-loops (#60988) 2024-03-11 16:05:47 +02:00
zoekt_repos.go search: use ReposMap for ListAllIndexed (#54886) 2023-07-13 06:59:39 +02:00