mirror of
https://github.com/sourcegraph/sourcegraph.git
synced 2026-02-06 15:51:43 +00:00
Code AI platform with Code Search & Cody
28348e7c80
Adds a new `postgreSQL.logicalReplication` configuration to allow MSP to generate prerequisite setup for integration with Datastream: https://cloud.google.com/datastream/docs/sources-postgresql. Integration with Datastream allows the Data Analytics team to self-serve data enrichment needs for the Telemetry V2 pipeline. Enabling this feature entails downtime (Cloud SQL instance restart), so enabling the logical replication feature at the Cloud SQL level (`cloudsql.logical_decoding`) is gated behind `postgreSQL.logicalReplication: {}`. Setting up the required stuff in Postgres is a bit complicated, requiring 3 Postgres provider instances: 1. The default admin one, authenticated with our admin user 2. New: a workload identity provider, using https://github.com/cyrilgdn/terraform-provider-postgresql/pull/448 / https://github.com/sourcegraph/managed-services-platform-cdktf/pull/11. This is required for creating a publication on selected tables, which requires being owner of said table. Because tables are created by application using e.g. auto-migrate, the workload identity is always the table owner, so we need to impersonate the IAM user 3. New: a "replication user" which is created with the replication permission. Replication seems to not be a propagated permission so we need a role/user that has replication enabled. A bit more context scattered here and there in the docstrings. Beyond the Postgres configuration we also introduce some additional resources to enable easy Datastream configuration: 1. Datastream Private Connection, which peers to the service private network 2. Cloud SQL Proxy VM, which only allows connections to `:5432` from the range specified in 1, allowing a connection to the Cloud SQL instance 2. Datastream Connection Profile attached to 1 From there, data team can click-ops or manage the Datastream Stream and BigQuery destination on their own. Closes CORE-165 Closes CORE-212 Sample config: ```yaml resources: postgreSQL: databases: - "primary" logicalReplication: publications: - name: testing database: primary tables: - users ``` ## Test plan https://github.com/sourcegraph/managed-services/pull/1569 ## Changelog - MSP services can now configure `postgreSQL.logicalReplication` to enable Data Analytics team to replicate selected database tables into BigQuery. |
||
|---|---|---|
| .apko | ||
| .aspect | ||
| .buildkite | ||
| .github | ||
| .vscode | ||
| client | ||
| cmd | ||
| dev | ||
| doc | ||
| docker-images | ||
| internal | ||
| lib | ||
| migrations | ||
| monitoring | ||
| schema | ||
| testing | ||
| third_party | ||
| third-party-licenses | ||
| tools | ||
| ui/assets | ||
| wolfi-images | ||
| wolfi-packages | ||
| .bazel_fix_commands.json | ||
| .bazelignore | ||
| .bazeliskrc | ||
| .bazelrc | ||
| .bazelversion | ||
| .dockerignore | ||
| .editorconfig | ||
| .eslintrc.js | ||
| .gitattributes | ||
| .gitignore | ||
| .graphqlrc.yml | ||
| .hadolint.yaml | ||
| .mailmap | ||
| .mocharc.js | ||
| .npmrc | ||
| .pre-commit-config.yaml | ||
| .prettierignore | ||
| .stylelintignore | ||
| .stylelintrc.json | ||
| .swcrc | ||
| .tool-versions | ||
| .trivyignore | ||
| BUILD.bazel | ||
| CHANGELOG.md | ||
| CODENOTIFY | ||
| CONTRIBUTING.md | ||
| deps.bzl | ||
| doc.go | ||
| eslint-relative-formatter.js | ||
| flake.lock | ||
| flake.nix | ||
| gen.go | ||
| go.mod | ||
| go.sum | ||
| graphql-schema-linter.config.js | ||
| LICENSE | ||
| LICENSE.enterprise | ||
| linter_deps.bzl | ||
| mockgen.temp.yaml | ||
| mockgen.test.yaml | ||
| mockgen.yaml | ||
| nogo_config.json | ||
| package.json | ||
| pnpm-lock.yaml | ||
| pnpm-workspace.yaml | ||
| postcss.config.js | ||
| prettier.config.js | ||
| README.md | ||
| release.yaml | ||
| renovate.json | ||
| SECURITY.md | ||
| service-catalog.yaml | ||
| sg.config.yaml | ||
| shell.nix | ||
| stamp_tags.bzl | ||
| tsconfig.base.json | ||
| tsconfig.json | ||
| vitest.shared.ts | ||
| vitest.workspace.ts | ||
| WORKSPACE | ||
Docs •
Contributing •
Twitter •
Discord
Sourcegraph makes it easy to read, write, and fix code—even in big, complex codebases.
- Code search: Search all of your repositories across all branches and all code hosts.
- Code intelligence: Navigate code, find references, see code owners, trace history, and more.
- Fix and refactor: Roll out large-scale changes to many repositories at once and track big migrations.
Getting started
Development
Refer to the Developing Sourcegraph guide to get started.
Documentation
The doc directory has additional documentation for developing and understanding Sourcegraph:
- Architecture: high-level architecture
- Database setup: database best practices
- Go style guide
- Documentation style guide
- GraphQL API: useful tips when modifying the GraphQL API
- Contributing
License
This repository contains primarily non-OSS-licensed files. See LICENSE.
Copyright (c) 2018-present Sourcegraph Inc.