For simple, MVP integrations with Telemetry Gateway, and also as a
reference. See example - this can be consumed with:
```go
import telemetrygatewayv1 "github.com/sourcegraph/sourcegraph/lib/telemetrygateway/v1"
```
## Test plan
n/a
- Updates enry to include languages from Linguist v7.29.0 such as Mojo etc.
- Updates auto-complete filters in frontend code.
- Updates Zoekt to pick up newer version with bumped enry dep.
- Updates language extension overrides to avoid ambiguity for `.json` and `.yml`.
- Updates snapshot tests.
Closes CORE-99, closes CORE-176
This PR is based off (and was also served as PoC of) [RFC 962: MSP IAM
framework](https://docs.google.com/document/d/1ItJlQnpR5AHbrfAholZqjH8-8dPF1iQcKh99gE6SSjs/edit).
It comes with two main parts:
1. The initial version of the MSP IAM SDK:
`lib/managedservicesplatform/iam`
- Embeds the [OpenFGA server
implementation](https://github.com/openfga/openfga/tree/main/pkg/server)
and exposes the a `ClientV1` for interacting with it.
- Automagically manages the both MSP IAM's and OpenFGA's database
migrations upon initializing the `ClientV1`.
- Ensures the specified OpenFGA's store and automatization model DSL
exists.
- Utility types and helpers to avoid easy mistakes (i.e. make the
relation tuples a bit more strongly-typed).
- Decided to put all types and pre-defined values together to simulate a
"central registry" and acting as a forcing function for services to form
some sort of convention. Then when we migrate the OpenFGA server to a
separate standalone service, it will be less headache about
consolidating similar meaning types/relations but different string
literals.
1. The first use case of the MSP IAM:
`cmd/enterprise-portal/internal/subscriptionsservice`
- Added/updated RPCs:
- Listing enterprise subscriptions via permissions
- Update enterprise subscriptions to assign instance domains
- Update enterprise subscriptions membership to assign roles (and
permissions)
- A database table for enterprise subscriptions, only storing the extra
instance domains as Enterprise Portal is not the
writeable-source-of-truth.
## Other minor changes
- Moved `internal/redislock` to `lib/redislock` to be used in MSP IAM
SDK.
- Call `createdb ...` as part of `enterprise-portal` install script in
`sg.config.yaml` (`msp_iam` database is a hard requirement of MSP IAM
framework).
## Test plan
Tested with gRPC UI:
- `UpdateEnterpriseSubscription` to assign an instance domain
- `UpdateEnterpriseSubscriptionMembership` to assign roles
- `ListEnterpriseSubscriptions`:
- List by subscription ID
- List by instance domain
- List by view cody analytics permissions
---------
Co-authored-by: Robert Lin <robert@bobheadxi.dev>
Based on the [discussion in Slack](https://sourcegraph.slack.com/archives/C02UC4WUX1Q/p1717067410264679),
thought it would be useful to solidify the invariants that hold
between errors.As, errors.Is and errors.HasType
(and provide counter-examples for things which you think
ought to hold but don't actually hold) in code itself, instead of
just via docs. So this PR adds property-based tests for
quickly generating different kinds of error shapes, and check
which invariants hold and which ones don't.
It also adds better documentation for errors.Is and errors.As
See [RFC 885 Sourcegraph Enterprise Portal (go/enterprise-portal)](https://docs.google.com/document/d/1tiaW1IVKm_YSSYhH-z7Q8sv4HSO_YJ_Uu6eYDjX7uU4/edit#heading=h.tdaxc5h34u7q) - closes CORE-6. The only files requiring in-depth review are the `.proto` files, as everything else is generated:
- `lib/enterpriseportal/subscriptions/v1/subscriptions.proto`
- `lib/enterpriseportal/codyaccess/v1/codyaccess.proto`
This PR only introduces API definitions - implementation will come as subsequent PRs, tracked in the ["Launch Enterprise Portal" Linear project](https://linear.app/sourcegraph/project/launch-sourcegraph-enterprise-portal-ee5d9ea105c2).
Before reviewing the diffs, **please review this PR description in depth**.
### Design goals
This initial schema aims to help achieve CORE-97 by adding our initial "get subscription Cody access", as well our general Stage 1 goal of providing read-only access to our existing enterprise subscription mechanisms. In doing so, we can start to reshape the API in a way that accommodates future growth and addresses some debt we have accumulated over time, before the Stage 2 goal of having the new Enterprise Portal be the source-of-truth for all things subscriptions.
I am also aiming for a conservative approach with the Cody Gateway access related RPCs, to ease migration risks and allow for Cody teams to follow up quickly with more drastic changes in a V2 of the service after a Core-Services-driven migration to use the new service: https://github.com/sourcegraph/sourcegraph/pull/62263#issuecomment-2101874114
### Design overview
- **Multiple services**: Enterprise Portal aims to be the home of most Enterprise-related subscription and access management, but each component should be defined as a separate service to maintain clear boundaries between "core" capabilities and future extensions. One problem we see in the `dotcom { productSubscriptions }` is the embedding of additional concepts like Cody Gateway access makes the API surface unwieldy and brittle, and encourages an internal design that bundles everything together (the `product_subscriptions` table has 10 `cody_gateway_*` columns today). More concretely, this PR designs 2 services that Enterprise Portal will implement:
- `EnterprisePortalSubscriptionsService` (`subscriptions.proto`): subscriptions and licenses CRUD
- `EnterprisePortalCodyGatewayService` (`codygateway.proto`): Enterprise Cody Gateway access
- **Multiple protocols**: We use [ConnectRPC](https://connectrpc.com/) to generate traditional gRPC handlers for service-to-service use, but also a plain HTTP/1 "REST"-ish protocol (the ["Connect Protocol"](https://connectrpc.com/docs/protocol)) that works for web clients and simple integrations. Go bindings for the Connect protocol are generated into the `v1connect` subpackages.
- **Future licensing model/mechanism changes**: The _Subscription_ model is designed to remain static, but _Licenses_ are designed to accommodate future changes -`EnterpriseSubscriptionLicenseType` and `EnterpriseSubscriptionLicense` in this PR describe only the current type of license, referred to as "classic licenses", but we can extend this in the future for e.g. new models (refreshable licenses?) or new products (Cody-only? PLG enterprise?), or existing problems (test instance licenses?)
- **Granular history**: Instead of a `createdAt`, `isArchived`, `revokedAt` and and so on, the new API defines Kubernetes-style `conditions` for licenses and subscriptions to describe creation, archival, and revocation events respectively, and can be more flexibly extended for future events and a lightweight audit log of major changes to a subscription or license. In particular, `revokedAt` already has a `revokedReason` - this allows us to extend these important events with additional metadata in a flexible manner.
- **Pagination**: I couldn't find a shared internal or off-the-shelf representation of pagination attributes, but each `List*` RPC describes `page_size`, `page_token`, and `next_page_token`
- **Querying/filtering**: I couldn't find a strong standard for this either, but in general:
- `Get*` accepts `query` that is a `oneof`, with the goal of providing exact matches only.
- `List*` accepts `repeated filter`, where each `filter` is a `oneof` a set of strategies relevant to a particular `List*` RPC. Multiple filters are treated as `AND`-concatenated.
Some major changes from the existing model:
- **Downgrade the concept of "subscription access token"**: this was built for Cody Gateway but I am not sure it has aged well, as the mechanism is still tied to individual licenses, did not find new non-Cody-Gateway use cases (except for license checks, though those do not require an "access token" model either), and today are still not "true" access tokens as they cannot be expired/managed properly. This PR relegates the concept to remain Cody-specific as it effectively is today so that we might be able to introduce a better subscription-wide model if the use case arises. Over time, we may want to make this even more opaque, relying entirely on zero-config instead (generating from license keys).
- **Subscriptions are no longer attached to a single dotcom user**: Most of these users today are not real users anyway, as our license creation process asks that you create a fake user account (["User account: [...] We create a company-level account for this."](https://handbook.sourcegraph.com/departments/technical-success/ce/process/license_keys/#license-key-mechanics)). The new API removes the concept entirely, in favour of a true user access management system in CORE-102.
- **Database/GraphQL IDs** are no longer exposed - we use external, prefixed UUIDs for representing entities over APIs in a universal manner.
- **Per-subscription Cody Gateway access no longer exposes `allowed models`**: I suggested this to @rafax in light of recent problems with propagating new models to Enterprise customers. He agreed that the general product direction is "model options as a selling point" - it no longer makes sense to configure these at a per-subscription level. Instead, the Cody Gateway service should configure globally allowed models directly, and each Sourcegraph instance can determine what models they trust. If we really need this back we can add it later, but for now I think this removal is the right direction.
### Direct translations
`cmd/cody-gateway/internal/dotcom/operations.graphql` defines our key dependencies for achieving CORE-97. The concepts referred in `operations.graphql` translate to this new API as follows:
- `dotcom { productSubscriptionByAccessToken(accessToken) }`: `codygateway.v1.GetCodyGatewayAccess({ access_token })`
- `dotcom { productSubscriptions }`: `codygateway.v1.ListCodyGatewayAccess()`
- `fragment ProductSubscriptionState`:
- `id`: **n/a**
- `uuid`: `subscriptions.v1.EnterpriseSubscription.id`
- `account { username }`: `subscriptions.v1.EnterpriseSubscription.display_name`
- `isArchived`: `subscriptions.v1.EnterpriseSubscription.conditions`
- `codyGatewayAccess { ... }`: **separate RPC to `codygateway.v1.GetCodyGatewayAccess`**
- `activeLicense { ... }`: **separate RPC to `subscriptions.v1.ListEnterpriseSubscriptionLicenses`**
### Why `lib/enterpriseportal`?
We recently had to move another Telemetry Gateway to `lib`: #62061. Inevitably, there will be services that live outside the monorepo that want to integrate with Enterprise Portal (one is on our roadmap: Cody Analytics in https://github.com/sourcegraph/cody-analytics). This allows us to share generated bindings and some useful helpers, while keeping things in the monorepo.
### Implications for Cody Clients
For now (and in the future), nothing is likely to change. Here's how I imagine things playing out:
```mermaid
graph TD
cc["Cody Clients"] -- unified API --> cg[services like Cody Gateway]
cg -- PLG users --> ssc[Self-Serve Cody]
cg -- Enterprise users --> ep[Enterprise Portal]
```
## Test plan
CI passes, the schemas can be generated by hand:
```
sg gen buf \
lib/enterpriseportal/subscriptions/v1/buf.gen.yaml \
lib/enterpriseportal/codyaccess/v1/buf.gen.yaml
```
---------
Co-authored-by: Joe Chen <joe@sourcegraph.com>
Co-authored-by: Chris Smith <chrsmith@users.noreply.github.com>
Migrates the Telemetry Gateway:
1. Service specification
2. Generated Go bindings
3. UUID constructor
into an exported `lib/telemetrygateway` package for internal and external consumption. See https://github.com/sourcegraph/sourcegraph/issues/61489 for use cases. This allows MSP services to more easily start publishing events for to Telemetry Gateway, and adds no new dependencies to `lib`.
Splits Sourcegraph-specific functionality that used to live in the `telemetrygateway/v1` package to:
1. `internal/telemetrygateway`: backcompat testing
2. `internal/telmeetrygateway/event`: event constructors (collapsing into parent caused import cycle)
I've left README + a stub service spec in the old package to redirect visitors from outdated links.
Closes https://github.com/sourcegraph/sourcegraph/issues/61489
## Test plan
```
sg start
```
watch for successful export logs from `telemetrygatewayexporter`
Camden added 4fea34868d to the tool so that we don't have to have a functioning build when mocks need to be regenerated, which can sometimes cause a cyclic issue.
### Test plan
Ran generate and it worked.
To determine the icon to show for a file, we need to know
the language. The code previously relied on the getFileInfo
API which did this using file extensions, leading to
incorrect results for header files with the '.h' extension,
which could have either C, C++ or Objective-C code.
Instead, we now get the language from the server,
where the go-enry library can optionally use the file
contents to return a more appropriate result.
Consequently, we can remove the getFileInfo API altogether.
Updates both the React and Svelte versions of the code.
Other changes in this patch:
- I've added icons for a bunch of languages which were missing.
- I've centralized the icon lists (and the bulk of the related logic) for
both React and Svelte in one place.
We use github.com/stretchr/testify pretty much universally except for some odd edge cases. This makes VSCode turn up inconsistent package recommendations on auto-import.
This change:
1. Renames `lib/managedservicesplatform/service` to `lib/managedservicesplatform/runtime` - I've been referring to it as runtime, and we use the word "service" a lot already, so this makes it easier to refer to and is more descriptive.
2. Makes everything under `lib/managedservicesplatform` a separate submodule from `lib` - the MSP runtime contributes a lot of dependencies to `lib`, and I don't think we should keep bloating `lib` to add more functionality to MSP.
3. Adds OpenTelemetry tracing and metrics configuration similar to the setup used by Cody Gateway and Telemetry Gateway
This PR has 3 major changes:
1. Provisioning a Cloud SQL PostgreSQL instance: `resource.postgreSQL`
2. Updates to the MSP runtime for connecting to the database
3. Publishable `msp-example` for the `msp-testbed` deployment, for testing integrations and other MSP features, and adds an example Gorm setup
Closes https://github.com/sourcegraph/sourcegraph/issues/56848. For more context, see go/msp
### `resource.postgreSQL`
This change adds `resource.postgreSQL` to the spec for provisioning a Cloud SQL database. Example:
```yaml
resources:
postgreSQL:
databases: ["primary"]
cpu: 1
memoryGB: 4
```
When configured, a Cloud SQL instance is provisioned, and the Cloud Run workload SA is granted access to it. The specified `databases` are also created. `cpu` and `memoryGB` are used to generate the custom VM to provision the instance with. A lot of the base configuration for the database instance are taken directly from what we use in Cloud: 9cac1b1aa0/internal/resource/sql/sql.go (L116), and is similar to the Redis setup, in that we need the private peering VPC and install additional certs on the Cloud Run instance.
### MSP runtime
`service.Contract` now has a helper, `GetPostgreSQLDB(database)`, for generating the appropriate configuration for creating a connection to a given database and returning a `sql.DB`, which most libraries have adapters for. The helper sets up `pgx.ConnConfig` that uses workload identity to connect directly to Cloud SQL, following the guide here: https://github.com/GoogleCloudPlatform/cloud-sql-go-connector?tab=readme-ov-file#automatic-iam-database-authentication
In local dev, `PGDSN` can be used to hardcode an appropriate value.
### `msp-example`
I've moved the MSP example from #56846 to the top-level `cmd/msp-example,` and set up some guidance for publishing it. This service now also includes a simple database connection and setup using Gorm, used by https://github.com/sourcegraph/accounts.sourcegraph.com, and can be used for testing, e.g. https://github.com/sourcegraph/managed-services/pull/87
## Test plan
- [x] Deploy https://github.com/sourcegraph/managed-services/pull/87, which will connect to the database and set up some tables
- [x] Review diff: https://github.com/sourcegraph/managed-services/actions/runs/6951246277
- [x] Tear down database in https://github.com/sourcegraph/managed-services/pull/87, with caveat: requires state surgery. Should be rare enough (we might prefer to tear down environments entirely) - see docstrings in package `postgresqlroles`
---------
Co-authored-by: Michael Lin <mlzc@hey.com>
Co-authored-by: Joe Chen <joe@sourcegraph.com>
Part of #56846 - initializes a `lib/managedservicesplatform` package that can be imported from other repositories. See `lib/managedservicesplatform/example` for details. It doesn't do much, but emulates Sourcegraph's internal service stuff somewhat, just stripped down significantly (we only need to support a single service per binary), and using generics.
To support the new Accounts service, I've also moved parts of background routine management into `lib/background`, aliased from the original `internal/goroutine` package to maintain compatibility - the utilities here are a bit hefty to copy-paste, and I think it's useful to share this.
## Test plan
```
PORT=8080 VARIABLE=24 go run ./managedservicesplatform/example
```
```
curl localhost:8080
```
I noticed a compile error in the lib package that was caused by an old version of sourcegraph/log. This fixes it. This also runs go mod tidy, which seems to have cleaned up the file a lot.
When auto-upgrade intent is signaled, we want the frontend to do the
migration instead of migrator. As by default, migrator runs up (unless
manually changed to something else or invoked), we check before that
command runs whether we should auto-upgrade, and exiting with 0 if yes
## Test plan
Tested locally in docker-compose setup
Did you know: all args to `errors.Newf`, `errors.Wrapf`, etc are
considered sensitive, and redacted before errors are reported to Sentry?
That's where all the error reports that look useless come from:
```
x
x
x
x
x
```
This change documents how this works based on cockroachdb error docs,
and also registers a set of arg types that can automatically be
considered safe based on what is configured in cockroachdb itself
(basically most primitive types except string, such as ints for status
codes and whatnot).
Aside: PII is important to consider as we build out multi-tenant
services like Cody Gateway, and important if we want to consider using
nice third-party tools like Honeycomb/Sentry for Cloud instances in the
future.
Related: https://github.com/sourcegraph/sourcegraph/issues/51998
## Test plan
CI
- Write Rust one-for-one protocol compatible replacement for
universal-ctags in JSON streaming mode (scip-ctags)
- Use tree-sitter to generate scip symbols, then emit those through
scip-ctags
- These symbols will be reused for Cody context
- Ensure code is built with musl libc
## Test plan
Unit and snapshot tests in the Rust symbol generation code - verified
working in the symbols sidebar and symbol search for enabled languages.
---------
Co-authored-by: TJ DeVries <devries.timothyj@gmail.com>
Co-authored-by: William Bezuidenhout <william.bezuidenhout@sourcegraph.com>
Co-authored-by: Eric Fritz <eric@eric-fritz.com>
Co-authored-by: Eric Fritz <eric@sourcegraph.com>
Co-authored-by: Jean-Hadrien Chabran <jh@chabran.fr>
- Write Rust one-for-one protocol compatible replacement for
universal-ctags in JSON streaming mode (scip-ctags)
- Use tree-sitter to generate scip symbols, then emit those through
scip-ctags
- These symbols will be reused for Cody context
Currently, only zig is enabled (so other languages should remain unaffected by this change).
We will add other languages throughout the next week as we're able to check them off.
## Test plan
Unit and snapshot tests in the Rust symbol generation code - verified
working in the symbols sidebar and symbol search for enabled languages.
---------
Co-authored-by: SuperAuguste <19855629+SuperAuguste@users.noreply.github.com>
Co-authored-by: William Bezuidenhout <william.bezuidenhout@sourcegraph.com>
Co-authored-by: Eric Fritz <eric@eric-fritz.com>
Co-authored-by: Eric Fritz <eric@sourcegraph.com>
Due to an issue with Autogold and Bazel, we have to migrate all
`autogold` tests to use the `v2`. This draft PR loses some annotations
describing the test cases, but I'll update that after once I have
validate that we can get those to work with Bazel.
Why is this a huge PR and not a couple of smaller ones: Because this PR
needed a few manual fixes, it was simpler to grind through the changes
after running comby locally. I have attached the config below so you can
review what I used instead.
@chwarwick the code insights tests were heavily relying on the attached
description on `autogold.Want` to provide context, I manually ported
them by adding a new `name string` field on the test cases.
Comby config I used:
```
[update-imports]
match="\"github.com/hexops/autogold\""
rewrite="\"github.com/hexops/autogold/v2\""
[update-api-want]
match="autogold.Want(:[desc], :[v])"
rewrite="autogold.Expect(:[v])"
[update-api-equal]
match="autogold.Equal(:[v])"
rewrite="autogold.ExpectFile(:[v])"
```
```
$ comby -config ../autogold-comby.toml -matcher .go -exclude-dir vendor,node_modules -in-place
```
I then followed up with some manual fixes for the test cases that were
making the assumption that `want.Name()` exists.
## Test plan
<!-- All pull requests REQUIRE a test plan:
https://docs.sourcegraph.com/dev/background-information/testing_principles
-->
Green go tests.
---------
Co-authored-by: davejrt <davetry@gmail.com>
Patches can contain non UTF-8 characters (who knew). Since we JSON-encode the patch as a string field, we lose that encoding over the wire to Sourcegraph from src-cli and also when we interact with executors. This PR adds support for a byte slice encoded (so base64 over the wire) mode from Sourcegraph 4.4 onwards, which retains original encoding.
Executors also now support binary files for VirtualMachineFiles. The most of this diff is restoring backwards compatibility with old src-cli and executors, the main change is string -> []byte.
Closes#44743
Move sg's docgen code into lib/cliutil/docgen for sharing with the mi CLI. In the future we can move other useful utilities for urfave/cli and CLIs in general into this set of subpackages for use in mi.
This adds an experimental code path that I will use to test a docker-only execution mode for server-side batch changes. This code path is never executed for customers until we make the switch when we deem it ready. This will allow me to dogfood this while it's not available to customer instances yet.
Ultimately, the goal of this is to make executors simply be "the job runner platform through a generic interface". Today, this depends on src-cli to do a good bunch of the work. This is a blocker for going full docker-based with executors, which will ultimately be a requirement on the road to k8s-based executors.
As this removes the dependency on src-cli, nothing but the job interface and API endpoints tie executor and Sourcegraph instance together. Ultimately, this will allow us to support larger version spans between the two (pending executors going GA and being feature-complete).
Known issues/limitations:
Steps skipped in between steps that run don't work yet
Skipping steps dynamically is inefficient as we cannot tell the executor to skip a step IF X, so we replace the script by exit 0
It is unclear if all variants of file mounts still work. Basic cases do work. Files used to be read-only in src-cli, they aren't now, but content is still reset in between steps.
The assumption that everything operates in /work is broken here, because we need to use what executors give us to persist out-of-repo state in between containers (like the step result from the previous step)
It is unclear if workspace mounts work
Cache keys are not correctly computed if using workspace mounts - the metadataretriever is nil
We still use log outputs to transfer the AfterStepResults to the Sourcegraph instance, this should finally become an artifact instead. Then, we don't have to rely on the execution_log_entires anymore and can theoretically prune those after some time. This column is currently growing indefinitely.
It depends on tee being available in the docker images to capture the cmd.stdout/cmd.stderr properly for template variable rendering
Env-vars are not rendered in their evaluated form post-execution
File permissions are unclear and might be similarly broken to how they are now - or even worse
Disclaimer: It's not feature complete today! But it is also not hitting any default code paths either. As development on this goes on, we can eventually remove the feature flag and run the new job format on all instances. This PR handles fallback of rendering old records correctly in the UI already.
This should resolve some issues we are seeing with terminal detection that causes users of lib/output to hang when running in Buildkite environments, which also means we can remove the ForceDarkBackground workaround in sg.
Turns out there's an Jaeger propagator we're supposed to use for the opentelemetry library. After enabling it and using the composite propagator on both ends of the tracer, we now have linked traces to Zoekt's opentelemetry Jaeger implementation again in Jaeger with opentelemetry enabled.
Enables log sampling by default for Sourcegraph components that use the new internal logging library - the first 100 identical log entries per second will always be output, but thereafter only every 100th identical message will be output. It can be configured for each service using the environment variables SRC_LOG_SAMPLING_INITIAL and SRC_LOG_SAMPLING_THEREAFTER, and if SRC_LOG_SAMPLING_INITIAL is set to 0 or -1 the sampling will be disabled entirely.
We extend internal/tracer to support a new tracer type, "opentelemetry", that sends everything in OpenTelemetry format using the OpenTracing bridge. See https://github.com/sourcegraph/sourcegraph/pull/37984 for more details.
