Adds a super simple E2E test suite that must be run with `sg test
enterprise-portal-e2e` against a locally running Enterprise Portal
instance. This is not intended to be filled with super granular
assertions - it simply tests that everything is wired up correctly and
runs end-to-end.
Caught at ~3 issues with this already, amended various downstack PRs
with the fix 😆
Closes
https://linear.app/sourcegraph/issue/CORE-229/enterprise-portal-basic-manual-e2e-testing-setup
## Test plan
```
sg start dotcom
sg test enterprise-portal-e2e
```
No additional configuration required, the defaults work as-is
In https://github.com/sourcegraph/sourcegraph/pull/63946 we reworked how
we handle assets, but we overlooked updating the target in
`sg.config.yaml`.
## Test plan
CI + locally tested (made some changes on the home page, got reloaded as
expected)
Fixes srch-867
This commit adds a new command, `web-sveltekit-prod`, which simplies
builds the SvelteKit app and exits.
This command is now run by default when using `sg start`, so the assets
will be available locally.
Note however that the build process is quite slow, which is why I didn't
make it rebuild on file changes. Using the vite dev server, e.g. via `sg
start enterprise-sveltekit`, delivers a much better experience.
To make the integration a bit more ergonomic, I made additional changes:
- When run via `sg` (determined by checking `DEPLOY_TYPE`) compiler
warnings are not printed. That reduces noise in the `sg` output.
- I made similar changes to the svelte check bazel target to make it
easier to spot actuall errors (warnings are still important to fix but
since they don't cause the target to fail they just make it difficult to
spot the actual error).
- Since I finally found out about the `onwarn` handler, I made it so
that warnings about missing declarations are ignored. These warnings
occur for every icon because the Svelte compiler doesn't respect ambient
d.ts files.
- And since I made icon warning related changes I also documented a bit
how icons work in the app.
## Test plan
- `sg start` runs the `web-sveltekit-prod` command and the Svelte app is
served when going to `https://sourcegraph.test:3443/search`.
- The output of `bazel test //client/web-sveltekit:svelte-check` now
only contains errors.
- The output of `pnpm dev` (vite devserver) shows warnings that are not
related to icons.
Now that https://github.com/sourcegraph/sourcegraph/pull/64339 is
merged, we can tell users more about what to expect with `sg start
single-program-experimental-blame-sqs`. And as it's been in flight for a
while now, it's safe to say that's it's time to give it a shorter name
😊.
So it's been renamed from `single-program-experimental-blame-sqs` to
`minimal`. And to ensure nobody is getting confused, a `deprecated`
attribute has been added on the command sets, which is used here to
indicate that the new alternative is `sg start`.
❓ Thoughts about `sg start minimal`? `sg start single` perhaps?
Running the old commandset: (ignore the yellow message, that's just a
local warning from `sg`)
Running the new commandset, with the preamble explaining what to expect:
## Test plan
Locally tested.
There currently isn't a 'one step' way to start a local Sourcegraph
instance with the SvelteKit app.
This commit adds `sg start enterprise-sveltekit` to fix that.
The changes in https://github.com/sourcegraph/sourcegraph/pull/64272
allow us to run the vite dev server in front of the Sourcegraph
instance, instead of building the assets and having them served by
frontend.
This gives us the benefit of hot module reloading and in general seems
to be a less fragile approach.
It's basically the same what we do with the React app in development
mode.
## Test plan
`sg start enterprise-sveltekit` starts the vite dev server as well as
the sourcegraph instance. Navigating to
`https://sourcegraph.test:3443/search` opens the Svelte app (when
enabled and logged in). Making a change in a source file updates the web
page immediately.
`sg start web-sveltekit-standalone` still works
2nd attempt of #63111, a follow up
https://github.com/sourcegraph/sourcegraph/pull/63085
rules_oci 2.0 brings a lot of performance improvement around oci_image
and oci_pull, which will benefit Sourcegraph. It will also make RBE
faster and have less load on remote cache.
However, 2.0 makes some breaking changes like
- oci_tarball's default output is no longer a tarball
- oci_image no longer compresses layers that are uncompressed, somebody
has to make sure all `pkg_tar` targets have a `compression` attribute
set to compress it beforehand.
- there is no curl fallback, but this is fine for sourcegraph as it
already uses bazel 7.1.
I checked all targets that use oci_tarball as much as i could to make
sure nothing depends on the default tarball output of oci_tarball. there
was one target which used the default output which i put a TODO for
somebody else (somebody who is more on top of the repo) to tackle
**later**.
## Test plan
CI. Also run delivery on this PR (don't land those changes)
---------
Co-authored-by: Noah Santschi-Cooney <noah@santschi-cooney.ch>
Some commands like the
[`batcheshelper-builder`](https://sourcegraph.sourcegraph.com/github.com/sourcegraph/sourcegraph/-/blob/sg.config.yaml?L821)
aren't long running commands.
This command is used to build and load an image into docker. The `cmd`
section returns an `exit 0`. This behavior combined with
`continueWatchOnExit` results in an infinite loop where the process is
continually restarted because `sg` doesn't know that the process has
finished executing and isn't a long-running process.
https://github.com/user-attachments/assets/e7a027a1-6f93-403f-9240-6a791255fba9
An example of the behavior is shown below as running `sg start batches`
results in the `batcheshelper-builder` command continually restarted.
The fix is quite simple, we return an empty receiver channel when the
process is done executing so that `sg` knows it's done and doesn't
restart the command unless there's a change.
## Test plan
* Manual testing with `go run ./dev/sg start batches` doesn't result in
an infinite loop anymore.
* Add unit tests
## Changelog
Changes in client-side code should not result in expensive rebuilds of
`frontend` locally when using bazel with `sg start`. For some reason,
iBazel still thinks it needs to be rebuilt (possibly it uses `query`
instead of `cquery` and therefore doesnt resolve `selects`), but bazel
underneath determines nothing needs to be rebuilt so theres very little
cost
## Test plan
Command set with just `frontend` in a `bazelCommands` section and then
`sg start <command set>`
## Changelog
<!-- OPTIONAL; info at
https://www.notion.so/sourcegraph/Writing-a-changelog-entry-dd997f411d524caabf0d8d38a24a878c
-->
- The `internal/uploadstore` package is renamed to `object` indicating
that it is meant to provide a generic object storage wrapper.
- The `search/exhaustive/uploadstore` package is used in very few places
so I've merged into the `internal/search` package similar to
`internal/embeddings`.
There are a few reasons to do the renaming.
1. The word `upload` in a more general context is ambiguous (just in
`internal/`) - in the codeintel context, it means "SCIP index" but it
can also be interpreted generically ("upload of _some_ data").
2. Better readability - `object.Storage` is much shorter than
`uploadstore.Store`. Additionally, we use the term `Store` A LOT
in the codebase, and usually, these refer to wrappers over some
tables in some DB.
Making things worse, some of our code also has:
```
uploadsstore
"github.com/sourcegraph/sourcegraph/internal/codeintel/uploads/internal/store"
```
And code which says `uploadsstore.Store` (notice the extra `s` 😢), which
is actually a wrapper over some key DB tables like `lsif_uploads`.
Prototype what it might be like to have `sg` run `sourcegraph-accounts`.
I think this would be a huge improvement over the multi-step,
many-things-to-install process that currently exists for SAMS.
There are a couple of quirks, but I think I'll leave this here for now,
as this is part https://linear.app/sourcegraph/issue/CORE-220, a
timeboxed spike into polishing some local-dev DX for SAMS.
Also see https://github.com/sourcegraph/sourcegraph-accounts/pull/262
## Test plan
```
sg start sourcegraph-accounts
```
Go to 127.0.0.1:9991 and try to log in
This commit adds the React version of the cody chat sidebar to the
Svelte web app. I used @vovakulikov's work in the React app as guidance.
I'm sure we'll have to do follow up work, but it's a start.
This PR also fixes `sg start web-sveltekit-standalone` (because I
originally thought that running it from `sourcegraph.test` is necessary
to make auth work).
## Test plan
Manual testing.
https://github.com/sourcegraph/sourcegraph/assets/179026/3fa3f2ea-b23e-44ca-a75a-0089bf07fb2b
As titled - in deployments we ran this by hand already. We use `|| true`
to make the forced migration non-blocking if it fails in case it just
works, though in local testing it seems that command is idempotent
anyway
See https://github.com/sourcegraph/devx-support/issues/1070
## Test plan
```
sg run enterprise-portal
```
```
sql -d sourcegraph -c 'ALTER TABLE enterprise_portal_subscriptions ALTER COLUMN id TYPE uuid USING id::uuid;'
```
This makes it easier to run Sourcegraph in local dev by compiling a few
key services (frontend, searcher, repo-updater, gitserver, and worker)
into a single Go binary and running that.
Compared to `sg start` (which compiles and runs ~10 services), it's
faster to start up (by ~10% or a few seconds), takes a lot less memory
and CPU when running, has less log noise, and rebuilds faster. It is
slower to recompile for changes just to `frontend` because it needs to
link in more code on each recompile, but it's faster for most other Go
changes that require recompilation of multiple services.
This is only intended for local dev as a convenience. There may be
different behavior in this mode that could result in problems when your
code runs in the normal deployment. Usually our e2e tests should catch
this, but to be safe, you should run in the usual mode if you are making
sensitive cross-service changes.
Partially reverts "svcmain: Simplify service setup (#61903)" (commit
9541032292).
## Test plan
Existing tests cover any regressions to existing behavior. This new
behavior is for local dev only.
Closes CORE-99, closes CORE-176
This PR is based off (and was also served as PoC of) [RFC 962: MSP IAM
framework](https://docs.google.com/document/d/1ItJlQnpR5AHbrfAholZqjH8-8dPF1iQcKh99gE6SSjs/edit).
It comes with two main parts:
1. The initial version of the MSP IAM SDK:
`lib/managedservicesplatform/iam`
- Embeds the [OpenFGA server
implementation](https://github.com/openfga/openfga/tree/main/pkg/server)
and exposes the a `ClientV1` for interacting with it.
- Automagically manages the both MSP IAM's and OpenFGA's database
migrations upon initializing the `ClientV1`.
- Ensures the specified OpenFGA's store and automatization model DSL
exists.
- Utility types and helpers to avoid easy mistakes (i.e. make the
relation tuples a bit more strongly-typed).
- Decided to put all types and pre-defined values together to simulate a
"central registry" and acting as a forcing function for services to form
some sort of convention. Then when we migrate the OpenFGA server to a
separate standalone service, it will be less headache about
consolidating similar meaning types/relations but different string
literals.
1. The first use case of the MSP IAM:
`cmd/enterprise-portal/internal/subscriptionsservice`
- Added/updated RPCs:
- Listing enterprise subscriptions via permissions
- Update enterprise subscriptions to assign instance domains
- Update enterprise subscriptions membership to assign roles (and
permissions)
- A database table for enterprise subscriptions, only storing the extra
instance domains as Enterprise Portal is not the
writeable-source-of-truth.
## Other minor changes
- Moved `internal/redislock` to `lib/redislock` to be used in MSP IAM
SDK.
- Call `createdb ...` as part of `enterprise-portal` install script in
`sg.config.yaml` (`msp_iam` database is a hard requirement of MSP IAM
framework).
## Test plan
Tested with gRPC UI:
- `UpdateEnterpriseSubscription` to assign an instance domain
- `UpdateEnterpriseSubscriptionMembership` to assign roles
- `ListEnterpriseSubscriptions`:
- List by subscription ID
- List by instance domain
- List by view cody analytics permissions
---------
Co-authored-by: Robert Lin <robert@bobheadxi.dev>
`sg start dotcom` still fired up the `docsite` command, which is
unecessary at this point. This PR removes it.
## Test plan
Locally tested.
<!-- All pull requests REQUIRE a test plan:
https://docs-legacy.sourcegraph.com/dev/background-information/testing_principles
-->
## Changelog
<!--
1. Ensure your pull request title is formatted as: $type($domain): $what
2. Add bullet list items for each additional detail you want to cover
(see example below)
3. You can edit this after the pull request was merged, as long as
release shipping it hasn't been promoted to the public.
4. For more information, please see this how-to
https://www.notion.so/sourcegraph/Writing-a-changelog-entry-dd997f411d524caabf0d8d38a24a878c?
Audience: TS/CSE > Customers > Teammates (in that order).
Cheat sheet: $type = chore|fix|feat $domain:
source|search|ci|release|plg|cody|local|...
-->
<!--
Example:
Title: fix(search): parse quotes with the appropriate context
Changelog section:
## Changelog
- When a quote is used with regexp pattern type, then ...
- Refactored underlying code.
-->
Follow up https://github.com/sourcegraph/sourcegraph/pull/63085
rules_oci 2.0 brings a lot of performance improvement around oci_image
and oci_pull, which will benefit sourcegraph. It will also make RBE
faster and have less load on remote cache.
However, 2.0 makes some breaking changes like
- oci_tarball's default output is no longer a tarball
- oci_image no longer compresses layers that are uncompressed, somebody
has to make sure all `pkg_tar` targets have a `compression` attribute
set to compress it beforehand.
- there is no curl fallback, but this is fine for sourcegraph as it
already uses bazel 7.1.
I checked all targets that use oci_tarball as much as i could to make
sure nothing depends on the default tarball output of oci_tarball. there
was one target which used the default output which i put a TODO for
somebody else (somebody who is more on top of the repo) to tackle later.
## Test plan
I am assuming that the repo has enough tests to catch potential problems
on CI. Also somebody who knows the repo better should double check my
changes.
---------
Co-authored-by: Noah Santschi-Cooney <noah@santschi-cooney.ch>
Migrates Cody Gateway to use the new Enterprise Portal's "read-only"
APIs. For the most part, this is an in-place replacement - a lot of the
diff is in testing and minor changes. Some changes, such as the removal
of model allowlists, were made down the PR stack in
https://github.com/sourcegraph/sourcegraph/pull/62911.
At a high level, we replace the data requested by
`cmd/cody-gateway/internal/dotcom/operations.graphql` and replace it
with Enterprise Portal RPCs:
- `codyaccessv1.GetCodyGatewayAccess`
- `codyaccessv1.ListCodyGatewayAccesses`
Use cases that previously required retrieving the active license tags
now:
1. Use the display name provided by the Cody Access API
https://github.com/sourcegraph/sourcegraph/pull/62968
2. Depend on the connected Enterprise Portal dev instance to only return
dev subscriptions https://github.com/sourcegraph/sourcegraph/pull/62966
Closes https://linear.app/sourcegraph/issue/CORE-98
Related to https://linear.app/sourcegraph/issue/CORE-135
(https://github.com/sourcegraph/sourcegraph/pull/62909,
https://github.com/sourcegraph/sourcegraph/pull/62911)
Related to https://linear.app/sourcegraph/issue/CORE-97
## Local development
This change also adds Enterprise Portal to `sg start dotcom`. For local
development, we set up Cody Gateway to connect to Enterprise Portal such
that zero configuration is needed - all the required secrets are sourced
from the `sourcegrah-local-dev` GCP project automatically when you run
`sg start dotcom`, and local Cody Gateway will talk to local Enterprise
Portal to do the Enterprise subscriptions sync.
This is actually an upgrade from the current experience where you need
to provide Cody Gateway a Sourcegraph user access token to test
Enterprise locally, though the Sourcegraph user access token is still
required for the PLG actor source.
The credential is configured in
https://console.cloud.google.com/security/secret-manager/secret/SG_LOCAL_DEV_SAMS_CLIENT_SECRET/overview?project=sourcegraph-local-dev,
and I've included documentation in the secret annotation about what it
is for and what to do with it:
## Rollout plan
I will open PRs to set up the necessary configuration for Cody Gateway
dev and prod. Once reviews taper down I'll cut an image from this branch
and deploy it to Cody Gateway dev, and monitor it closely + do some
manual testing. Once verified, I'll land this change and monitor a
rollout to production.
Cody Gateway dev SAMS client:
https://github.com/sourcegraph/infrastructure/pull/6108
Cody Gateway prod SAMS client update (this one already exists):
```
accounts=> UPDATE idp_clients
SET scopes = scopes || '["enterprise_portal::subscription::read", "enterprise_portal::codyaccess::read"]'::jsonb
WHERE id = 'sams_cid_018ea062-479e-7342-9473-66645e616cbf';
UPDATE 1
accounts=> select name, scopes from idp_clients WHERE name = 'Cody Gateway (prod)';
name | scopes
---------------------+----------------------------------------------------------------------------------------------------------------------------------
Cody Gateway (prod) | ["openid", "profile", "email", "offline_access", "enterprise_portal::subscription::read", "enterprise_portal::codyaccess::read"]
(1 row)
```
Configuring the target Enterprise Portal instances:
https://github.com/sourcegraph/infrastructure/pull/6127
## Test plan
Start the new `dotcom` runset, now including Enterprise Portal, and
observe logs from both `enterprise-portal` and `cody-gateway`:
```
sg start dotcom
```
I reused the test plan from
https://github.com/sourcegraph/sourcegraph/pull/62911: set up Cody
Gateway external dependency secrets, then set up an enterprise
subscription + license with a high seat count (for a high quota), and
force a Cody Gateway sync:
```
curl -v -H 'Authorization: bearer sekret' http://localhost:9992/-/actor/sync-all-sources
```
This should indicate the new sync against "local dotcom" fetches the
correct number of actors and whatnot.
Using the local enterprise subscription's access token, we run the QA
test suite:
```sh
$ bazel test --runs_per_test=2 --test_output=all //cmd/cody-gateway/qa:qa_test --test_env=E2E_GATEWAY_ENDPOINT=http://localhost:9992 --test_env=E2E_GATEWAY_TOKEN=$TOKEN
INFO: Analyzed target //cmd/cody-gateway/qa:qa_test (0 packages loaded, 0 targets configured).
INFO: From Testing //cmd/cody-gateway/qa:qa_test (run 1 of 2):
==================== Test output for //cmd/cody-gateway/qa:qa_test (run 1 of 2):
PASS
================================================================================
INFO: From Testing //cmd/cody-gateway/qa:qa_test (run 2 of 2):
==================== Test output for //cmd/cody-gateway/qa:qa_test (run 2 of 2):
PASS
================================================================================
INFO: Found 1 test target...
Target //cmd/cody-gateway/qa:qa_test up-to-date:
bazel-bin/cmd/cody-gateway/qa/qa_test_/qa_test
Aspect @@rules_rust//rust/private:clippy.bzl%rust_clippy_aspect of //cmd/cody-gateway/qa:qa_test up-to-date (nothing to build)
Aspect @@rules_rust//rust/private:rustfmt.bzl%rustfmt_aspect of //cmd/cody-gateway/qa:qa_test up-to-date (nothing to build)
INFO: Elapsed time: 13.653s, Critical Path: 13.38s
INFO: 7 processes: 1 internal, 6 darwin-sandbox.
INFO: Build completed successfully, 7 total actions
//cmd/cody-gateway/qa:qa_test PASSED in 11.7s
Stats over 2 runs: max = 11.7s, min = 11.7s, avg = 11.7s, dev = 0.0s
Executed 1 out of 1 test: 1 test passes.
```
Part of CORE-99
This PR scaffolds the database schema and code structure based on
[CORE-99
comment](https://linear.app/sourcegraph/issue/CORE-99/enterprise-portal-design-sams-user-to-subscription-rpcs#comment-8105ac31)
with some modifications. See inline comments for more elaborations.
- It uses GORM's ONLY for auto migration, just to kick things off, we
may migrate to file-based migration like we are planning for SAMS.
- It then uses the `*pgxpool.Pool` as the DB interface for executing
business logic queries.
Additionally, refactored `subscriptionsservice/v1.go` to use a `Store`
that provide single interface for accessing data(base), as we have been
doing in SAMS and SSC.
## Test plan
Enterprise Portal starts locally, and database is initialized:
This runs playwright tests with bazel. This changes how the
app is served in the tests, specifically playwright will intercept all
network calls to the local server and serve the static assets directly
or serve root index.html file if nothing is matched.
---------
Co-authored-by: bahrmichael <michael.bahr@sourcegraph.com>
Co-authored-by: Jean-Hadrien Chabran <jh@chabran.fr>
Co-authored-by: Michael Bahr <1830132+bahrmichael@users.noreply.github.com>
Co-authored-by: Jean-Hadrien Chabran <jean-hadrien.chabran@sourcegraph.com>
Co-authored-by: Camden Cheek <camden@ccheek.com>
This modifies our pattern for shipping a svelte-enabled HTML document by
injecting the svelte-specific things into our primary `app.html` rather
than trying to make `app.prod.html` match the behavior of the frontend
`app.html`.
I could not figure out a "blessed" way to integrate this into the build
system, so I went for the hacky way: render `app.prod.html` in a format
that's easily parseable, then parse it in `frontend` and inject it into
our `app.html` with the standard go template patterns.
This change makes Cody Gateway always apply a wildcard model allowlist,
irrespective of what the configured model allowlist is for an Enterprise
subscription is in dotcom (see #62909).
The next PR in the stack,
https://github.com/sourcegraph/sourcegraph/pull/62912, makes the GraphQL
queries return similar results, and removes model allowlists from the
subscription management UI.
Closes https://linear.app/sourcegraph/issue/CORE-135
### Context
In https://sourcegraph.slack.com/archives/C05SZB829D0/p1715638980052279
we shared a decision we landed on as part of #62263:
> Ignoring (then removing) per-subscription model allowlists: As part of
the API discussions, we've also surfaced some opportunities for
improvements - to make it easier to roll out new models to Enterprise,
we're not including per-subscription model allowlists in the new API,
and as part of the Cody Gateway migration (by end-of-June), we will
update Cody Gateway to stop enforcing per-subscription model allowlists.
Cody Gateway will still retain a Cody-Gateway-wide model allowlist.
[@chrsmith](https://sourcegraph.slack.com/team/U061QHKUBJ8) is working
on a broader design here and will have more to share on this later.
This means there is one less thing for us to migrate as part of
https://github.com/sourcegraph/sourcegraph/pull/62934, and avoids the
need to add an API field that will be removed shortly post-migration.
As part of this, rolling out new models to Enterprise customers no
longer require additional code/override changes.
## Test plan
Set up Cody Gateway locally as documented, then `sg start dotcom`. Set
up an enterprise subscription + license with a high seat count (for a
high quota), and force a Cody Gateway sync:
```
curl -v -H 'Authorization: bearer sekret' http://localhost:9992/-/actor/sync-all-sources
```
Verify we are using wildcard allowlist:
```sh
$ redis-cli -p 6379 get 'v2:product-subscriptions:v2:slk_...'
"{\"key\":\"slk_...\",\"id\":\"6ad033f4-c6da-43a9-95ef-f653bf59aaac\",\"name\":\"bobheadxi\",\"accessEnabled\":true,\"endpointAccess\":{\"/v1/attribution\":true},\"rateLimits\":{\"chat_completions\":{\"allowedModels\":[\"*\"],\"limit\":660,\"interval\":86400000000000,\"concurrentRequests\":330,\"concurrentRequestsInterval\":10000000000},\"code_completions\":{\"allowedModels\":[\"*\"],\"limit\":66000,\"interval\":86400000000000,\"concurrentRequests\":33000,\"concurrentRequestsInterval\":10000000000},\"embeddings\":{\"allowedModels\":[\"*\"],\"limit\":220000000,\"interval\":86400000000000,\"concurrentRequests\":110000000,\"concurrentRequestsInterval\":10000000000}},\"lastUpdated\":\"2024-05-24T20:28:58.283296Z\"}"
```
Using the local enterprise subscription's access token, we run the QA
test suite:
```sh
$ bazel test --runs_per_test=2 --test_output=all //cmd/cody-gateway/qa:qa_test --test_env=E2E_GATEWAY_ENDPOINT=http://localhost:9992 --test_env=E2E_GATEWAY_TOKEN=$TOKEN
INFO: Analyzed target //cmd/cody-gateway/qa:qa_test (0 packages loaded, 0 targets configured).
INFO: From Testing //cmd/cody-gateway/qa:qa_test (run 1 of 2):
==================== Test output for //cmd/cody-gateway/qa:qa_test (run 1 of 2):
PASS
================================================================================
INFO: From Testing //cmd/cody-gateway/qa:qa_test (run 2 of 2):
==================== Test output for //cmd/cody-gateway/qa:qa_test (run 2 of 2):
PASS
================================================================================
INFO: Found 1 test target...
Target //cmd/cody-gateway/qa:qa_test up-to-date:
bazel-bin/cmd/cody-gateway/qa/qa_test_/qa_test
Aspect @@rules_rust//rust/private:clippy.bzl%rust_clippy_aspect of //cmd/cody-gateway/qa:qa_test up-to-date (nothing to build)
Aspect @@rules_rust//rust/private:rustfmt.bzl%rustfmt_aspect of //cmd/cody-gateway/qa:qa_test up-to-date (nothing to build)
INFO: Elapsed time: 13.653s, Critical Path: 13.38s
INFO: 7 processes: 1 internal, 6 darwin-sandbox.
INFO: Build completed successfully, 7 total actions
//cmd/cody-gateway/qa:qa_test PASSED in 11.7s
Stats over 2 runs: max = 11.7s, min = 11.7s, avg = 11.7s, dev = 0.0s
Executed 1 out of 1 test: 1 test passes.
```
During manual testing for https://github.com/sourcegraph/sourcegraph/pull/62934, I started realizing that I would run into database errors:
```
failed to deallocate cached statement(s): conn busy
```
Turns out, `pgx.Conn` is meant for non-concurrent use. What we really want is a connection pool, with an off-the-shelf offering from `pgxpool`.
## Test plan
Integration tests pass, now with more cases using `t.Parallel()`. Also ran a quick sanity check by hand:
```
sg start
```
```
for i in {1..10}; do curl --header "Content-Type: application/json" --header 'authorization: bearer $SAMS_TOKEN' --data '{"filters":[{"filter":{"is_archived":false}}]}' \
http://localhost:6081/enterpriseportal.subscriptions.v1.SubscriptionsService/ListEnterpriseSubscriptionLicenses & ; done
```
## Changelog
- The MSP runtime `lib/managedservicesplatform/contract.Contract`'s `ConnectToDatabase(...)` has been renamed to `GetConnectionPool(...)`, and now returns a `*pgxpool.Pool` instead of a `*pgx.Conn`
- The MSP runtime `lib/managedservicesplatform/cloudsql` helper library's `Connect(...)` has been renamed to `GetConnectionPool(...)`, and now returns a `*pgxpool.Pool` instead of a `*pgx.Conn`
Since we are reading from the dotcom database for the time being, the dev and prod Enterprise Portal instances serve the same data. This change adds a configuration so that we can set up the dev instance to only serve data from licenses with the dev or internal tag, toggled by `DOTCOM_INCLUDE_PRODUCTION_LICENSES`
This will become a non-issue for Enterprise Portal when we have our own database in https://linear.app/sourcegraph/issue/CORE-100, but for now this helps make sure that the dev instance only serves dev data. This is fairly important since we grant Enterprise Portal scopes more liberally in accounts.sgdev.org.
## Test plan
Updated integration tests to use the dev-only mode.
Also tested locally: with `DOTCOM_INCLUDE_PRODUCTION_LICENSES=false`, endpoints return nothing, while with `DOTCOM_INCLUDE_PRODUCTION_LICENSES=true`, endpoints return the expected data.
Part of CORE-112. We need to implement the `CodyAccess` service proposed in https://github.com/sourcegraph/sourcegraph/pull/62263, so that Cody Gateway can depend on it as we start a transition over to Enterprise Portal as the source-or-truth for Cody Gateway access; see the [Linear project](https://linear.app/sourcegraph/project/kr-launch-enterprise-portal-for-cody-gateway-and-cody-analytics-ee5d9ea105c2/overview). This PR implements the data layer by reading directly from the Sourcegraph.com Cloud SQL database, and a subsequent PR https://github.com/sourcegraph/sourcegraph/pull/62771 will expose this via the API and also implement auth; nothing in this PR is used yet.
Most things in this PR will be undone by the end of a [follow-up project](https://linear.app/sourcegraph/project/kr-enterprise-portal-manages-all-enterprise-subscriptions-12f1d5047bd2/overview) tentatively slated for completion by end-of-August.
### Query
I've opted to write a new query specifically to fetch the data required to fulfill the proposed `CodyAccess` RPCs; the existing queries fetch a lot more than is strictly needed, and often make multiple round trips to the database. The new query fetches everything it needs for get/list in a single round trip.
`EXPLAIN ANALYZE` of the new list-all query against the Sourcegraph.com production database indicates this is likely performant enough for our internal-only use cases, especially as this will only be around for a few months.
```
QUERY PLAN
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------
HashAggregate (cost=1610.56..1629.45 rows=1511 width=121) (actual time=23.358..24.921 rows=1512 loops=1)
Group Key: ps.id
-> Hash Left Join (cost=958.18..1585.58 rows=1999 width=1094) (actual time=8.258..12.255 rows=2748 loops=1)
Hash Cond: (ps.id = active_license.product_subscription_id)
-> Hash Right Join (cost=67.00..689.14 rows=1999 width=956) (actual time=1.098..3.970 rows=2748 loops=1)
Hash Cond: (product_licenses.product_subscription_id = ps.id)
-> Seq Scan on product_licenses (cost=0.00..616.88 rows=1999 width=919) (actual time=0.015..1.769 rows=2002 loops=1)
Filter: (access_token_enabled IS TRUE)
Rows Removed by Filter: 1789
-> Hash (cost=48.11..48.11 rows=1511 width=53) (actual time=1.055..1.056 rows=1512 loops=1)
Buckets: 2048 Batches: 1 Memory Usage: 93kB
-> Seq Scan on product_subscriptions ps (cost=0.00..48.11 rows=1511 width=53) (actual time=0.016..0.552 rows=1512 loops=1)
-> Hash (cost=874.39..874.39 rows=1343 width=154) (actual time=7.123..7.125 rows=1343 loops=1)
Buckets: 2048 Batches: 1 Memory Usage: 248kB
-> Subquery Scan on active_license (cost=842.02..874.39 rows=1343 width=154) (actual time=5.425..6.461 rows=1343 loops=1)
-> Unique (cost=842.02..860.96 rows=1343 width=162) (actual time=5.422..6.268 rows=1343 loops=1)
-> Sort (cost=842.02..851.49 rows=3788 width=162) (actual time=5.421..5.719 rows=3791 loops=1)
Sort Key: product_licenses_1.product_subscription_id, product_licenses_1.created_at DESC
Sort Method: quicksort Memory: 1059kB
-> Seq Scan on product_licenses product_licenses_1 (cost=0.00..616.88 rows=3788 width=162) (actual time=0.003..1.872 rows=3791 loops=1)
Planning Time: 2.266 ms
Execution Time: 28.568 ms
```
We noted the lack of index on `product_livenses.subscription_id`, but it doesn't seem to be an issue at this scale, so I've left it as is.
### Pagination
After discussing with Erik, we decided there is no need to implement pagination for the list-all RPC yet; a rough upper bound of 1kb per subscription * 1511 rows (see `EXPLAIN ANALYZE` above) is 1.5MB, which is well below the per-message limits we have set for Sourcegraph-internal traffic (40MB), and below the [default 4MB limit](https://pkg.go.dev/google.golang.org/grpc#MaxRecvMsgSize) as well. In https://github.com/sourcegraph/sourcegraph/pull/62771 providing pagination parameters will result in a `CodeUnimplemented` error.
We can figure out how we want to implement pagination as part of the [follow-up project](https://linear.app/sourcegraph/project/kr-enterprise-portal-manages-all-enterprise-subscriptions-12f1d5047bd2/overview) to migrate the data to an Enterprise-Portal-owned database.
### Testing
A good chunk of this PR's changes are exposing a small set of `cmd/frontend` internals **for testing** via the new `cmd/frontend/dotcomproductsubscriptiontest`:
- seeding test databases with subscriptions and licenses
- for "regression testing" the new read queries by validating what the new read queries get, against what the existing GraphQL resolvers resolve to. This is important because the GraphQL resolvers has a lot of the override logic
See `TestGetCodyGatewayAccessAttributes` for how all this is used.
<img width="799" alt="image" src="https://github.com/sourcegraph/sourcegraph/assets/23356519/af4d0c1e-c9a9-448a-9b8e-0f328688a75a">
There is also some hackery involved in setting up a `pgx/v5` connection used in MSP from the `sql.DB` + `pgx/v4` stuff used by `dbtest`; see `newTestDotcomReader` docstrings for details.
## Test plan
```
go test -v ./cmd/enterprise-portal/internal/dotcomdb
```
---
Co-authored-by: Chris Smith <chrsmith@users.noreply.github.com>
Co-authored-by: Joe Chen <joe@sourcegraph.com>
This removes qdrant from this codebase entirely.
All the docker images, dependencies, (dead) usage in code.
My understanding is that we don't use this feature and never properly rolled it out.
Test plan:
CI passes and code review from owners.
Exports package `github.com/sourcegraph/sourcegraph/lib/managedservicesplatform/runtime/contract` to expose previously-internal env-configuration (the "MSP contract") loading, helpers, and configuration. This allows programs that aren't 100% in the MSP runtime ecosystem yet (hopefully reducing future barrier to entry), or are in particular scenarios where they might not want to use runtime ([e.g. MSP jobs which doesn't have proper runtime support yet](https://sourcegraph.slack.com/archives/C06062P5TS5/p1711663052720289)), to integrate with MSP-provisioned infrastructure like BigQuery, PostgreSQL, and even Sentry etc. directly.
The recommended path will still be to build a `runtime`-compliant interface, but it can be a hassle to migrate sometimes, and as mentioned, we don't properly support jobs yet.
Simple example:
```go
// Parse the environment into an Env instance.
e, _ := contract.ParseEnv([]string{"MSP=true"})
// Extract Contract instance from Env configuration.
c := contract.New(logger, service, e)
// Also load other custom configuration here from Env you want here
// Check for errors on Env retrieval (missing/invalid values, etc.)
if err := e.Validate(); err != nil { ... }
// Use Contract helpers and configuration values
writer, _ := c.BigQuery.GetTableWriter(ctx, "my-table")
writer.Write(...)
```
## Test plan
There are no functionality changes, only code moving. But, ran some sanity checks:
```
sg run msp-example
sg run telemetry-gateway
sg run pings
```
---------
Co-authored-by: Chris Smith <chrsmith@users.noreply.github.com>
`bazel build` on percy mocha targets (such as //client/web/src/integration:integration-tests) no longer result in actually running the test!
Originally, we used `js_run_binary` with `build_test` as `js_test` doesnt support stamping, and we need to be able to read volatile variables for percy.
Then, we worked around https://github.com/bazelbuild/bazel/issues/16231 in https://github.com/sourcegraph/sourcegraph/pull/58505 by not explicitly depending on the stamp variables, but exploiting a bit of a hack to read them anyways (will this work with RBE?)
Now, given that we're not explicitly stamping and still using the hack, we can use `js_test` instead, to avoid having the tests run as part of `bazel build`, instead only when we run `bazel test` (as is good 😌)
It is apparently possible to work around https://github.com/bazelbuild/bazel/issues/16231 when using disk/remote caches, but only for local builds (so no remote builds) according to [the following comment](https://github.com/bazelbuild/bazel/issues/16231#issuecomment-1772835555), but we would still need:
1. `js_test` to support stamping and
2. this workaround to also apply to remote execution (as we're considering that once its supported in Aspect Workflows)
todo: update doc/dev/background-information/bazel/web_overview.md in new docs repo
## Test plan
CI 🎉
