This PR has 3 major changes:
1. Provisioning a Cloud SQL PostgreSQL instance: `resource.postgreSQL`
2. Updates to the MSP runtime for connecting to the database
3. Publishable `msp-example` for the `msp-testbed` deployment, for testing integrations and other MSP features, and adds an example Gorm setup
Closes https://github.com/sourcegraph/sourcegraph/issues/56848. For more context, see go/msp
### `resource.postgreSQL`
This change adds `resource.postgreSQL` to the spec for provisioning a Cloud SQL database. Example:
```yaml
resources:
postgreSQL:
databases: ["primary"]
cpu: 1
memoryGB: 4
```
When configured, a Cloud SQL instance is provisioned, and the Cloud Run workload SA is granted access to it. The specified `databases` are also created. `cpu` and `memoryGB` are used to generate the custom VM to provision the instance with. A lot of the base configuration for the database instance are taken directly from what we use in Cloud: 9cac1b1aa0/internal/resource/sql/sql.go (L116), and is similar to the Redis setup, in that we need the private peering VPC and install additional certs on the Cloud Run instance.
### MSP runtime
`service.Contract` now has a helper, `GetPostgreSQLDB(database)`, for generating the appropriate configuration for creating a connection to a given database and returning a `sql.DB`, which most libraries have adapters for. The helper sets up `pgx.ConnConfig` that uses workload identity to connect directly to Cloud SQL, following the guide here: https://github.com/GoogleCloudPlatform/cloud-sql-go-connector?tab=readme-ov-file#automatic-iam-database-authentication
In local dev, `PGDSN` can be used to hardcode an appropriate value.
### `msp-example`
I've moved the MSP example from #56846 to the top-level `cmd/msp-example,` and set up some guidance for publishing it. This service now also includes a simple database connection and setup using Gorm, used by https://github.com/sourcegraph/accounts.sourcegraph.com, and can be used for testing, e.g. https://github.com/sourcegraph/managed-services/pull/87
## Test plan
- [x] Deploy https://github.com/sourcegraph/managed-services/pull/87, which will connect to the database and set up some tables
- [x] Review diff: https://github.com/sourcegraph/managed-services/actions/runs/6951246277
- [x] Tear down database in https://github.com/sourcegraph/managed-services/pull/87, with caveat: requires state surgery. Should be rare enough (we might prefer to tear down environments entirely) - see docstrings in package `postgresqlroles`
---------
Co-authored-by: Michael Lin <mlzc@hey.com>
Co-authored-by: Joe Chen <joe@sourcegraph.com>
This change:
1. Adds support for `kind: job` services, and a new specification field `cron:` for cron-job-related options - see demo in https://github.com/sourcegraph/managed-services/pull/42. Closes#57100
2. Adds automatic support for private image registries operated by Sourcegraph in GCP - when you provide a `build.image` that points to a GCR or Artifact Registry image, we automatically grant the Cloud Run robot account various object/artifact-reading permissions to the target GCP project specified in the image repo name.
A few larger refactors were needed to make this happen:
1. In `dev/managedservicesplatform/internal/stack/cloudrun`, we now have 2 internal packages that implement an abstraction for a "Cloud Run resource builder", so that we can generalize the configuration of both services and jobs
2. Service-specific and job-specific configuration now lives in embedded structs that group them by purpose, and more configuration options are now nil-able so that we can have configuration exclusive to services/jobs
This change adds:
- telemetry export background jobs: flagged behind `TELEMETRY_GATEWAY_EXPORTER_EXPORT_ADDR`, default empty => disabled
- telemetry redaction: configured in package `internal/telemetry/sensitivemetadataallowlist`
- telemetry-gateway service receiving events and forwarding it to a pub/sub topic (or just logging it, as configured in local dev)
- utilities for easily creating an event recorder: `internal/telemetry/telemetryrecorder`
Notes:
- all changes are feature-flagged to some degree, off by default, so the merge should be fairly low-risk.
- we decided that transmitting the full license key continues to be the way to go. we transmit it once per stream and attach it on all events in the telemetry-gateway. there is no auth mechanism at the moment
- GraphQL return type `EventLog.Source` is now a plain string instead of string enum. This should not be a breaking change in our clients, but must be made so that our generated V2 events do not break requesting of event logs
Stacked on https://github.com/sourcegraph/sourcegraph/pull/56520
Closes https://github.com/sourcegraph/sourcegraph/issues/56289
Closes https://github.com/sourcegraph/sourcegraph/issues/56287
## Test plan
Add an override to make the export super frequent:
```
env:
TELEMETRY_GATEWAY_EXPORTER_EXPORT_INTERVAL: "10s"
TELEMETRY_GATEWAY_EXPORTER_EXPORTED_EVENTS_RETENTION: "5m"
```
Start sourcegraph:
```
sg start
```
Enable `telemetry-export` featureflag (from https://github.com/sourcegraph/sourcegraph/pull/56520)
Emit some events in GraphQL:
```gql
mutation {
telemetry {
recordEvents(events:[{
feature:"foobar"
action:"view"
source:{
client:"WEB"
}
parameters:{
version:0
}
}]) {
alwaysNil
}
}
```
See series of log events:
```
[ worker] INFO worker.telemetrygateway-exporter telemetrygatewayexporter/telemetrygatewayexporter.go:61 Telemetry Gateway export enabled - initializing background routines
[ worker] INFO worker.telemetrygateway-exporter telemetrygatewayexporter/exporter.go:99 exporting events {"maxBatchSize": 10000, "count": 1}
[telemetry-g...y] INFO telemetry-gateway.pubsub pubsub/topic.go:115 Publish {"TraceId": "7852903434f0d2f647d397ee83b4009b", "SpanId": "8d945234bccf319b", "message": "{\"event\":{\"id\":\"dc96ae84-4ac4-4760-968f-0a0307b8bb3d\",\"timestamp\":\"2023-09-19T01:57:13.590266Z\",\"feature\":\"foobar\", ....
```
Build:
```
export VERSION="insiders"
bazel run //cmd/telemetry-gateway:candidate_push --config darwin-docker --stamp --workspace_status_command=./dev/bazel_stamp_vars.sh -- --tag $VERSION --repository us.gcr.io/sourcegraph-dev/telemetry-gateway
```
Deploy: https://github.com/sourcegraph/managed-services/pull/7
Add override:
```yaml
env:
# Port required. TODO: What's the best way to provide gRPC addresses, such that a
# localhost address is also possible?
TELEMETRY_GATEWAY_EXPORTER_EXPORT_ADDR: "https://telemetry-gateway.sgdev.org:443"
```
Repeat the above (`sg start` and emit some events):
```
[ worker] INFO worker.telemetrygateway-exporter telemetrygatewayexporter/exporter.go:94 exporting events {"maxBatchSize": 10000, "count": 6}
[ worker] INFO worker.telemetrygateway-exporter telemetrygatewayexporter/exporter.go:113 events exported {"maxBatchSize": 10000, "succeeded": 6}
[ worker] INFO worker.telemetrygateway-exporter telemetrygatewayexporter/exporter.go:94 exporting events {"maxBatchSize": 10000, "count": 1}
[ worker] INFO worker.telemetrygateway-exporter telemetrygatewayexporter/exporter.go:113 events exported {"maxBatchSize": 10000, "succeeded": 1}
```
The eventual goal is to be able to generate a spec somewhat close to the infrastructure definitions for Cody Gateway today based on the way we generate infrastructure for Cloud - see go/rfc-msp . This PR is a first-step prototype to demonstrate CDKTF integration, draft out the sketch as proposed in go/rfc-msp, and lay out some basic architecture for the stuff. A barebones CDKTF setup that imitates ~most of the Cody Gateway setup can be generated and interacted with (see test plan), but has not been applied live anywhere - a few components are still missing, and a migration plan needs to be determined.
The current goal is to land this PR as-is and iterate from here. The new command set `sg msp` is only included with the `msp` build tag, so it won't be distributed just yet. To install a version of `sg` that includes a functional `sg msp` command, run:
```
go build -tags=msp -o=./sg ./dev/sg && ./sg install -f -p=false
```
In a vanilla build of `sg`, you'll get an error when trying to use `sg msp`:
## Binary size
Thanks to Michael's improvements to generated CDKTF Go modules, we are not looking at a 500MB increase in `sg` binary sizes, but we are still faced with a ~20% increase at the moment:
```sh
# Fresh sg installation
$ curl --proto '=https' --tlsv1.2 -sSLf https://install.sg.dev | sh
$ du -h $(which sg)
132M /Users/robert@sourcegraph.com/.sg/sg
# MSP-enabled sg build
$ go build -tags=msp -o=./sg ./dev/sg && du -h ./sg
161M ./sg
```
However, the _default_ build of `sg` continues to not include `msp`, so the binary size remains the same:
```sh
$ go build -o=./sg ./dev/sg && du -h ./sg
132M ./sg
```
## Architecture
See `dev/managedservicesplatform/internal/stack/README.md` and `dev/managedservicesplatform/internal/resource/README.md` for some rudimentary docs on some CDKTF conventions.
## Specifications
Set up initial specifications here: https://github.com/sourcegraph/managed-services
## Test plan
Currently:
```
go build -tags=msp -o=./sg ./dev/sg && ./sg msp generate -o=./tmp ./dev/sg/msp/example/example-service.service.yaml dev
```
And check out the `./tmp` output. Each module is plan-able out of the box right now, as we default to a local Terraform backend:
```
cd ./tmp
asdf install # a .tool-versions is generated
cd stacks/project
terraform init
terraform plan
```
---------
Co-authored-by: Joe Chen <joe@sourcegraph.com>
* feat: send test email with non-saved configuration
* fix comment on Deref function
* bazil
* change copy based on PR feedback
* Update cmd/frontend/graphqlbackend/send_test_email.go
Co-authored-by: Erik Seliger <erikseliger@me.com>
* Update cmd/frontend/graphqlbackend/schema.graphql
Co-authored-by: Erik Seliger <erikseliger@me.com>
* move args to a named type
* bidirectional updates to json based on forms
and forms based on json...
This allows us to flip between the tabs and keep consistent
in both places. Change to the form is immediately rendered on the
JSON tab. Change on the JSON tab is immediately rendered in a form.
* fix bad merge from main
* fix my event update loop mess
* graphql schema linter
* eslint
---------
Co-authored-by: Erik Seliger <erikseliger@me.com>
Originally I started working on this because of
[comment on another
PR](https://github.com/sourcegraph/sourcegraph/pull/53373#discussion_r1228058455).
I quickly wrote an implementation of Ptr and NonZeroPtr. Then I started
refactoring existing code only to find out that @efritz already beat me
to it. But Erics implementation was hidden in
internal/codeintel/resolvers/utils.go
Moved all of the Ptr functions to `pointers` package instead and
refactored all existing code that I could find to use it instead of
redefining the same functions all the time.
Usage is mostly in tests, so hopefully the impact is not as huge as the
diff size might suggest.
## Test plan
A whole lot of unit tests.
