sourcegraph/internal/database
William Bezuidenhout 7dbfb87a11
repos: log corruption events per repo (#45667)
* return a bool for checkMaybeCorrupt

* store plumbing for SetCorruptedAt

* debug

* move to markIfCorrupted + test

* add additional checks for repo corruption

* add test cases for server repo corruption detection

* add corrupted_at field for sql mapping

* fix err not being returned

* ensure corrupted at gets mapped

* test cases to ensure corrupted at gets cleared

* add migration

* update types for CorrruptedAt

* fix sql query and remove field for now

* run go generate

* Update internal/database/gitserver_repos_test.go

Co-authored-by: Erik Seliger <erikseliger@me.com>

* Update internal/database/gitserver_repos_test.go

Co-authored-by: Erik Seliger <erikseliger@me.com>

* Update cmd/gitserver/server/cleanup.go

Co-authored-by: Erik Seliger <erikseliger@me.com>

* review comments

* review feedback

- check for err
- remove reason
- set default timestamp to 0 value

* tests pass 🎉

* reset stiched migration graph

* go gen magic

* move docstring

* check for err in corruption check

* fix tests 🤞

* add corruption_lod column and rename migration

* fix migration scripts

* go gen

* change corruption_log column type to jsonb

* update db mocks

* add logCorruption method

- log repo corruption into a JSON array
- limit repo corruption log to 10 elements
- db tests

* fix wording and remove test

* add CorruptionLog to ignored fields

* add more ignores

* remove comment

* review feedback

* log git corrupt stderr
* cap corruption reason to 1mb
* rename CorruptionLog -> CorruptionLogs
* rename corruption_log -> corruption_logs

* Update internal/database/gitserver_repos_test.go

Co-authored-by: Thorsten Ball <mrnugget@gmail.com>

* Fixes

Fix rename in cmp IgnoreFields
Fix malformed SQL

* fix test

* add comment to trigger sonar cloud analysis

* review comments

append to json array in postgres

* add comment about json query

* comments and fixes

* Update cmd/gitserver/server/server_test.go

Co-authored-by: Thorsten Ball <mrnugget@gmail.com>

* Fix indentation

* review feedback

- Make corrupted_at column nullable
- Check err returned from LogCorruption call
- Add condition to make sure we only log more
  repo corruptions if there repo isn't currently
  corrupt.
- Run sg generate

* Fix test

- update corrupted_at also when using GitserverRepo store update
- clear corrupted_at status during batch sync update

* Update cmd/gitserver/server/server_test.go

Co-authored-by: Thorsten Ball <mrnugget@gmail.com>

* store plumbing for SetCorruptedAt

* debug

* move to markIfCorrupted + test

* add additional checks for repo corruption

* add test cases for server repo corruption detection

* add migration

* update types for CorrruptedAt

* fix sql query and remove field for now

* review comments

* review feedback

- check for err
- remove reason
- set default timestamp to 0 value

* reset stiched migration graph

* add corruption_lod column and rename migration

* go gen

* add logCorruption method

- log repo corruption into a JSON array
- limit repo corruption log to 10 elements
- db tests

* fix wording and remove test

* Fix more indentation 🤦🏻

* review feedback: update test

* run sg generate again

* test fixes

- warn instead of return an error during cleanup
- delete repos so that the name can be reused

* fix bug

- repo was corrupt but did not set the reason so the repo never got
  cleaned up

* review feedback

- sql query indentation

* review feedback

- use assert module

* do not clear corruption status on err

* add test for LastError and corruptedAt

- fix SQL issue on LastError

Co-authored-by: Erik Seliger <erikseliger@me.com>
Co-authored-by: Thorsten Ball <mrnugget@gmail.com>
2023-01-09 17:24:32 +02:00
..
basestore Update Store First Scanners to return data from 1st row instead of the last (#46133) 2023-01-04 15:06:24 -05:00
batch database: Add WithInserterForIdentifiers (#45588) 2022-12-13 02:07:57 +01:00
connections observability: build+propagate hierarchical observation contexts for telemetry provenance (#42765) 2022-12-05 21:52:30 +00:00
dbcache debt: Remove code related to user public repos (#44793) 2022-11-25 10:17:39 +01:00
dbconn otelsql: mute less useful spans (#44798) 2022-11-24 19:06:26 +01:00
dbtest codeintel: Fix test database at revision for oobmigrations (#46141) 2023-01-05 16:27:31 -06:00
dbutil chore: move all NullxxxColumn functions to dbutil package (#42882) 2022-10-13 11:27:10 +01:00
locker observability: build+propagate hierarchical observation contexts for telemetry provenance (#42765) 2022-12-05 21:52:30 +00:00
migration codeintel: Fix test database at revision for oobmigrations (#46141) 2023-01-05 16:27:31 -06:00
postgresdsn postgresdsn: add a test case for search_path (#40731) 2022-08-23 11:54:15 +00:00
access_tokens_test.go Add security events for access token deletions (#43680) 2022-11-03 09:54:27 +00:00
access_tokens.go Add security events for access token deletions (#43680) 2022-11-03 09:54:27 +00:00
authenticator_test.go Standardize database encryption (#40050) 2022-08-08 20:29:09 +00:00
authenticator.go encryption: Lazily decrypt batch_changes_site_credentials (#40228) 2022-08-15 15:58:13 +00:00
authz.go admin-analytics: add user administration backend API endpoints (#39926) 2022-08-05 17:18:21 +06:00
bitbucket_project_permissions_test.go dbconn: Modify query text to tag source, other metadata (#42588) 2022-10-13 12:34:37 +00:00
bitbucket_project_permissions.go dbconn: Modify query text to tag source, other metadata (#42588) 2022-10-13 12:34:37 +00:00
CODENOTIFY rbac: Create store methods for roles and permissions (Part 3) (#45752) 2022-12-16 16:27:32 +01:00
conf_test.go internal: Persist author in site config (#46150) 2023-01-09 12:14:06 +05:30
conf.go internal: Persist author in site config (#46150) 2023-01-09 12:14:06 +05:30
database_test.go database: fix wrong assertions in nested-transactions test (#40012) 2022-08-05 14:38:37 +00:00
database.go authz: Move sub-repo perms to enterprise (#45659) 2022-12-28 10:44:07 +02:00
dbstore_db_test.go Remove dbtesting package (#28426) 2021-12-02 08:24:03 -07:00
doc.go db: rename internal/db package to internal/database (#17607) 2021-01-25 20:24:15 +04:00
encryption_tables.go executors: Implement stores for secrets (#43257) 2022-10-24 23:44:15 +02:00
encryption_test.go Standardize database encryption (#40050) 2022-08-08 20:29:09 +00:00
encryption_utils.go Standardize database encryption (#40050) 2022-08-08 20:29:09 +00:00
encryption.go Standardize database encryption (#40050) 2022-08-08 20:29:09 +00:00
err_test.go all: use any instead of interface{} (#35102) 2022-05-09 10:59:39 +02:00
event_logs_test.go Clean linting errors from unparam (#45548) 2022-12-13 11:18:25 +01:00
event_logs.go Clean linting errors from unparam (#45548) 2022-12-13 11:18:25 +01:00
executor_secret_access_logs_test.go codeintel: first implementation of auto-indexing secrets (#45580) 2022-12-15 22:32:16 +00:00
executor_secret_access_logs.go codeintel: first implementation of auto-indexing secrets (#45580) 2022-12-15 22:32:16 +00:00
executor_secrets_test.go Implement UI components for executor secrets (#43942) 2022-11-10 20:33:42 +01:00
executor_secrets.go codeintel: first implementation of auto-indexing secrets (#45580) 2022-12-15 22:32:16 +00:00
executors_test.go executors: Undo service-ification for now (#43001) 2022-10-14 19:00:57 +01:00
executors.go database: implement ShareableStore on stores that accept them (#45673) 2022-12-14 13:03:27 -08:00
external_accounts_test.go Remove GH App linked accounts implementation (#44841) 2022-11-28 16:28:46 +02:00
external_accounts.go linters: re-enable nolintlint and staticcheck SA1019 (#45847) 2022-12-20 14:24:44 +00:00
external_services_test.go sg hacking hour offsite: removing cloud v1 code (#44824) 2022-12-07 11:14:43 +01:00
external_services.go linters: re-enable nolintlint and staticcheck SA1019 (#45847) 2022-12-20 14:24:44 +00:00
feature_flags_test.go Feature Flags: reduce DB roundtrips (#45747) 2022-12-15 16:23:21 -07:00
feature_flags.go Feature Flags: reduce DB roundtrips (#45747) 2022-12-15 16:23:21 -07:00
gen.go sg migration: Remove schemadoc (#35905) 2022-05-24 16:19:19 +00:00
gen.sh migration: Improve drift schema fetching (#37043) 2022-06-27 16:06:42 -05:00
github_app_helper.go internal/database/github_app_helper.go: Indent code (#43999) 2022-11-07 17:39:09 +05:30
gitserver_localclone_jobs_test.go logging(gitstart): migrate internal/database to lib/log (#36466) 2022-06-20 23:04:01 +00:00
gitserver_localclone_jobs.go dbconn: Modify query text to tag source, other metadata (#42588) 2022-10-13 12:34:37 +00:00
gitserver_repos_test.go repos: log corruption events per repo (#45667) 2023-01-09 17:24:32 +02:00
gitserver_repos.go repos: log corruption events per repo (#45667) 2023-01-09 17:24:32 +02:00
global_state_test.go global state: Rewrite store (#38781) 2022-07-15 09:56:53 -05:00
global_state.go dbconn: Modify query text to tag source, other metadata (#42588) 2022-10-13 12:34:37 +00:00
helpers.go [Pagination] add new connection resolver (#45167) 2022-12-13 20:58:41 +05:30
main_test.go deps: upgrade github.com/sourcegraph/log (#41058) 2022-09-01 08:01:59 -07:00
mockerr.go db: rename internal/db package to internal/database (#17607) 2021-01-25 20:24:15 +04:00
mocks_temp.go repos: log corruption events per repo (#45667) 2023-01-09 17:24:32 +02:00
namespaces_test.go logging(gitstart): migrate internal/database to lib/log (#36466) 2022-06-20 23:04:01 +00:00
namespaces.go dbconn: Modify query text to tag source, other metadata (#42588) 2022-10-13 12:34:37 +00:00
oauth_token_helper_test.go Add refreshable token interface for GitHub and GitLab (#42629) 2022-10-17 16:30:37 +02:00
oauth_token_helper.go Add refreshable token interface for GitHub and GitLab (#42629) 2022-10-17 16:30:37 +02:00
org_invitations_test.go logging(gitstart): migrate internal/database to lib/log (#36466) 2022-06-20 23:04:01 +00:00
org_invitations.go DB Backend: embed dbutil.DB in TransactableHandle (#37165) 2022-06-14 17:45:10 +00:00
org_members_db_test.go logging(gitstart): migrate internal/database to lib/log (#36466) 2022-06-20 23:04:01 +00:00
org_members.go DB Backend: embed dbutil.DB in TransactableHandle (#37165) 2022-06-14 17:45:10 +00:00
org_stats_test.go logging(gitstart): migrate internal/database to lib/log (#36466) 2022-06-20 23:04:01 +00:00
org_stats.go DB Backend: embed dbutil.DB in TransactableHandle (#37165) 2022-06-14 17:45:10 +00:00
orgs_test.go debt: Remove NamespaceUserID and NamespaceOrgID from external services (#44992) 2022-12-05 12:36:02 +01:00
orgs.go debt: Remove NamespaceUserID and NamespaceOrgID from external services (#44992) 2022-12-05 12:36:02 +01:00
permissions_test.go rbac: update database permissions on startup (#46053) 2023-01-05 17:39:03 +01:00
permissions.go rbac: update database permissions on startup (#46053) 2023-01-05 17:39:03 +01:00
phabricator_test.go Add tests for our Phabricator store (#41570) 2022-09-19 11:26:37 +02:00
phabricator.go Add tests for our Phabricator store (#41570) 2022-09-19 11:26:37 +02:00
repo_kvps_test.go Search backend: add support for arbitrary repo key-value pairs (#40112) 2022-08-11 15:17:33 +00:00
repo_kvps.go Search backend: add support for arbitrary repo key-value pairs (#40112) 2022-08-11 15:17:33 +00:00
repo_statistics_test.go repo stats tables: fix migration to only update function (#41038) 2022-08-30 13:23:00 +02:00
repo_statistics.go database: implement ShareableStore on stores that accept them (#45673) 2022-12-14 13:03:27 -08:00
repos_perm_test.go sg hacking hour offsite: removing cloud v1 code (#44824) 2022-12-07 11:14:43 +01:00
repos_perm.go Simplify authzQuery (#43798) 2022-11-07 14:39:37 +02:00
repos_test.go repos: log corruption events per repo (#45667) 2023-01-09 17:24:32 +02:00
repos.go linters: re-enable nolintlint and staticcheck SA1019 (#45847) 2022-12-20 14:24:44 +00:00
role_permissions_test.go rbac: Create store methods for roles and permissions (Part 4) (#45791) 2022-12-19 18:16:31 +01:00
role_permissions.go rbac: Create store methods for roles and permissions (Part 4) (#45791) 2022-12-19 18:16:31 +01:00
roles_test.go rbac: Create store methods for roles and permissions (Part 1) (#45658) 2022-12-15 00:04:44 +01:00
roles.go rbac: Create store methods for roles and permissions (Part 2) (#45680) 2022-12-16 09:57:02 +01:00
saved_searches_test.go logging(gitstart): migrate internal/database to lib/log (#36466) 2022-06-20 23:04:01 +00:00
saved_searches.go Add pagination to saved searches pages (#45705) 2023-01-05 21:11:42 +05:30
schema.codeinsights.json insights: add base for data retention worker (#46082) 2023-01-04 14:07:25 +00:00
schema.codeinsights.md insights: add base for data retention worker (#46082) 2023-01-04 14:07:25 +00:00
schema.codeintel.json codeintel: Clean up triggers (#45681) 2022-12-15 08:58:38 -06:00
schema.codeintel.md codeintel: Clean up triggers (#45681) 2022-12-15 08:58:38 -06:00
schema.json repos: log corruption events per repo (#45667) 2023-01-09 17:24:32 +02:00
schema.md repos: log corruption events per repo (#45667) 2023-01-09 17:24:32 +02:00
search_contexts_test.go search contexts: allow setting a default context and use it on load (#45387) 2022-12-12 09:39:53 -08:00
search_contexts.go search contexts: allow setting a default context and use it on load (#45387) 2022-12-12 09:39:53 -08:00
security_event_logs_test.go security_event_logs: set AnonymousUserID to internal for internal actors (#45309) 2022-12-07 08:32:46 -08:00
security_event_logs.go security_event_logs: set AnonymousUserID to internal for internal actors (#45309) 2022-12-07 08:32:46 -08:00
settings_test.go remove many removed and unused fields from settings and site config schemas (#46045) 2023-01-02 12:12:30 -10:00
settings.go Revive #42039 (#43168) 2022-10-19 17:57:01 +02:00
survey_responses_test.go NPS Survey: Update use cases to be a free-form field (#38387) 2022-07-08 14:13:42 +01:00
survey_responses.go NPS Survey: Update use cases to be a free-form field (#38387) 2022-07-08 14:13:42 +01:00
temporary_settings_test.go logging(gitstart): migrate internal/database to lib/log (#36466) 2022-06-20 23:04:01 +00:00
temporary_settings.go database: remove dbutil-based constructors (#36210) 2022-05-30 23:48:20 +00:00
test_util.go Standardize database encryption (#40050) 2022-08-08 20:29:09 +00:00
testing.go all: use any instead of interface{} (#35102) 2022-05-09 10:59:39 +02:00
user_credentials_test.go encryption: Lazily decrypt user_credentials (#40177) 2022-08-15 14:47:23 +00:00
user_credentials.go chore: remove duplicate dbutil.Scanner interfaces (#45630) 2022-12-13 17:06:47 +00:00
user_emails_test.go useremail: Remove dotcom special cases (#44814) 2022-12-06 13:01:17 +01:00
user_emails.go useremail: Remove dotcom special cases (#44814) 2022-12-06 13:01:17 +01:00
user_roles_test.go rbac: Create store methods for roles and permissions (Part 4) (#45791) 2022-12-19 18:16:31 +01:00
user_roles.go rbac: Create store methods for roles and permissions (Part 4) (#45791) 2022-12-19 18:16:31 +01:00
users_builtin_auth_test.go logging(gitstart): migrate internal/database to lib/log (#36466) 2022-06-20 23:04:01 +00:00
users_test.go database: UsersListOptions.Query searches for ID (#45334) 2022-12-07 13:08:16 +00:00
users.go database: implement ShareableStore on stores that accept them (#45673) 2022-12-14 13:03:27 -08:00
webhook_logs_test.go webhooks: add name column to webhooks table. (#44756) 2022-11-23 17:17:14 +04:00
webhook_logs.go Add webhook logging to new webhooks handler (#43446) 2022-10-27 09:58:37 -06:00
webhooks_test.go Update the webhooks update GraphQL endpoint (#45207) 2022-12-07 10:42:27 +02:00
webhooks.go Update the webhooks update GraphQL endpoint (#45207) 2022-12-07 10:42:27 +02:00
zoekt_repos_test.go Add zoekt_repos table, populate it in worker, add endpoint for Zoekt to update (#43289) 2022-10-27 14:07:19 +02:00
zoekt_repos.go Add zoekt_repos table, populate it in worker, add endpoint for Zoekt to update (#43289) 2022-10-27 14:07:19 +02:00