Searcher doesn't speak to the database nor has it for a long time. See
https://github.com/sourcegraph/sourcegraph/pull/61463
Test Plan: The following command is empty
go run ./dev/depgraph/ summary internal/database | grep 'cmd/searcher'
Part of
[GRAPH-759](https://linear.app/sourcegraph/issue/GRAPH-759/issue-with-apex-extension-not-appearing-for-langapex)
Linguist only supports a subset of the file extensions often used for
the Apex programming languages. This PR adds support for the main set
commonly used.
**Key changes**
1. Adds all extensions for Apex
2. Update our logic to handle multiple extensions for one language
3. Update tests to ensure we only manually map languages if they don't
exist OR have different extensions in go-enry (prevents us from
duplicating entries completely from go-enry)
## Test plan
- [x] Update unit tests
- [x] Validate locally by testing the language filter
If you look at the code for `bulkMonikerResultsQuery` and
`minimalBulkMonikerResultsQuery`, you'll see that the last
returned value is the `document_path`. The value returned
was directly used an `UploadRootRelPath` elsewhere. So it
makes sense to rename the field from URI to `DocumentPath`.
## Test plan
Covered by existing tests
For simple, MVP integrations with Telemetry Gateway, and also as a
reference. See example - this can be consumed with:
```go
import telemetrygatewayv1 "github.com/sourcegraph/sourcegraph/lib/telemetrygateway/v1"
```
## Test plan
n/a
Implements upsert for all the subscriptions fields in the DB client. As
part of this I generalized the logic for building upsert DB interactions
into a new `upsert` package, because this pattern is a common one we'll
need to implement to maintain various AIP-update-compliant endpoints,
which specifies various upsert behaviours: https://google.aip.dev/134
Part of CORE-216
Part of CORE-156
## Test plan
Integration tests against DB
This is part of the Keyword GA Project.
Batch Changes uses Sourcegraph queries to define the list of repositories on which the batch change will run.
With this change we default to pattern type "keyword" instead of "standard".
To make this a backward compatible change, we also introduce a version identifier to batch specs. Authors can specify `version: 2` in the spec, in which case we default to pattern type "keyword". Existing specs (without a specified version) and specs with `version: 1` will keep using pattern type "standard".
Notes:
- Corresponding doc update [PR](https://github.com/sourcegraph/docs/pull/477)
- We don't have a query input field, but instead the query is defined in a batch spec YAML. It didn't feel right to edit the YAML and append "patternType: " on save, which is what we do for Code Monitors and Insights.
- I misuse the pattern type query parameter to effectively override the version. Once we introduce "V4" we should come back here and clean up. I left a TODO in the code.
Test plan:
- New and updated unit tests
- manual testing
- new batch changes use `version: 2` by default.
- using an unsupported version returns an error
- I ran various "on:" queries to verify that version 2 uses keyword search and version 1 uses standard search.
Fixes DINF-82; This was very much a rabbithole. A few things:
- The race that @bobheadxi mentioned here
https://github.com/sourcegraph/sourcegraph/pull/63405#discussion_r1648180713
wasn't from `*output.Output` being unsafe, but `outputtest.Buffer` as it
happened again (see
[DINF-82](https://linear.app/sourcegraph/issue/DINF-82/devsgsg-test-failed-with-a-detected-race-condition))
- There something messed up with `cmds.start()`, which sometimes ends up
printing the command output _after_ the exit message instead of before.
- The crude `sort.Strings(want|have)` that was there already fixes that.
- And without the sleep, it's possible to read the output from the
`outputtest.Buffer` before the command outputs get written to it.
- The `time.Sleep(300 * time.Milliseconds)` _mitigates/hides_ that
problem.
At least, this shouldn't blow up in CI and buys us time to fix the whole
thing. We're tracking this in DINF-104. And out of 200 runs, I also
stumbled on a race in `progress_tty`, tracked in DINF-105 (that packages
is originally meant to be used by `src-cli` and was re-used for `sg` 3
years ago).
I'm pretty unhappy about the solution, but a bandage is better than
nothing. While ideally, we should really reconsider dropping
`std.Output` entirely in `sg` and use the good stuff from
github.com/charmbracelet instead because we don't want to spend too much
time on arcane terminal things ourselves, I'm much more about concerned
the concurrency issues mentioned above.
## Test plan
CI + `sg bazel test //dev/sg:sg_test --runs_per_test=100`
Part of https://linear.app/sourcegraph/issue/CORE-201. The RPC will be
backed by https://github.com/sourcegraph/sourcegraph/pull/63528
The `uint64` for ratelimits is a backwards-compatible change, per proto
docs:
> int32, uint32, int64, uint64, and bool are all compatible – this means
you can change a field from one of these types to another without
breaking forwards- or backwards-compatibility.
## Test plan
CI
Closes https://linear.app/sourcegraph/issue/CORE-199. AIP generally
implies `Update` RPCs are authoritative, which means that we should be
deleting all roles memberships not provided to
`UpdateEnterpriseSubscriptionMembership`. Most important outcome here is
that we can actually remove roles from users by assigning them an empty
role set `[]`
Later we can add a "get roles" RPC to safely make these updates, and
introduce a purely additive RPC if needed. It's not a huge deal right
now because we only have 1 role ("customer admin")
Also removes the deprecated value from
https://github.com/sourcegraph/sourcegraph/pull/63501.
## Test plan
Unit tests, expanded with better table-driven cases and expanded
assertions
Studying Joe's work a bit more in depth I noticed that our API
representation of this role ("Cody Analytics admin") does not line up
with our internal representation ("customer admin").
Since we're already here, it's probably better to just align on
"customer admin" as the role everywhere, and figure out more granular
roles if we need it later.
Once it's rolled out and usages are migrated
(https://github.com/sourcegraph/cody-analytics/pull/83), we can remove
the deprecated enum entirely
(https://github.com/sourcegraph/sourcegraph/pull/63502)
## Test plan
CI
Follow-up to #63448 - we now get Redis spans, but not the database
operations that happen throughout a migration. Maybe this will do the
thing?
## Test plan
n/a
The use of different types makes it clear which kind of path is needed
in which place. This also makes the CodeNavService layering clearer;
it has the responsibility of taking in RepoRelPaths and correctly interfacing
with LsifStore, which deals in UploadRelPath values.
There's a confusing notion of "archived license" that really means
"archived subscription", which is problematic because "can an archived
subscription, have valid licenses, in a world where revoked licenses
exist?"
IMO archiving a subscription should immediately and permanently revoke
all its associated licenses, per discussion in
https://github.com/sourcegraph/sourcegraph/pull/63330#discussion_r1645333457.
This means we can remove all notion of "archived license" - when looking
at licenses, they're only revoked, or not revoked.
⚠️ These RPCs are not used anywhere yet so this is a safe breaking
change.
## Test plan
CI
The runtime `Env` currently lacks a convenient method for getting a
floating point env var. This PR adds a `GetFloat`method which will parse
a float64 from a given env var
Part of CORE-181
## Test plan
locally tested
values such as `10.2` and `infinity` were correctly parsed
- Updates enry to include languages from Linguist v7.29.0 such as Mojo etc.
- Updates auto-complete filters in frontend code.
- Updates Zoekt to pick up newer version with bumped enry dep.
- Updates language extension overrides to avoid ambiguity for `.json` and `.yml`.
- Updates snapshot tests.
Closes CORE-99, closes CORE-176
This PR is based off (and was also served as PoC of) [RFC 962: MSP IAM
framework](https://docs.google.com/document/d/1ItJlQnpR5AHbrfAholZqjH8-8dPF1iQcKh99gE6SSjs/edit).
It comes with two main parts:
1. The initial version of the MSP IAM SDK:
`lib/managedservicesplatform/iam`
- Embeds the [OpenFGA server
implementation](https://github.com/openfga/openfga/tree/main/pkg/server)
and exposes the a `ClientV1` for interacting with it.
- Automagically manages the both MSP IAM's and OpenFGA's database
migrations upon initializing the `ClientV1`.
- Ensures the specified OpenFGA's store and automatization model DSL
exists.
- Utility types and helpers to avoid easy mistakes (i.e. make the
relation tuples a bit more strongly-typed).
- Decided to put all types and pre-defined values together to simulate a
"central registry" and acting as a forcing function for services to form
some sort of convention. Then when we migrate the OpenFGA server to a
separate standalone service, it will be less headache about
consolidating similar meaning types/relations but different string
literals.
1. The first use case of the MSP IAM:
`cmd/enterprise-portal/internal/subscriptionsservice`
- Added/updated RPCs:
- Listing enterprise subscriptions via permissions
- Update enterprise subscriptions to assign instance domains
- Update enterprise subscriptions membership to assign roles (and
permissions)
- A database table for enterprise subscriptions, only storing the extra
instance domains as Enterprise Portal is not the
writeable-source-of-truth.
## Other minor changes
- Moved `internal/redislock` to `lib/redislock` to be used in MSP IAM
SDK.
- Call `createdb ...` as part of `enterprise-portal` install script in
`sg.config.yaml` (`msp_iam` database is a hard requirement of MSP IAM
framework).
## Test plan
Tested with gRPC UI:
- `UpdateEnterpriseSubscription` to assign an instance domain
- `UpdateEnterpriseSubscriptionMembership` to assign roles
- `ListEnterpriseSubscriptions`:
- List by subscription ID
- List by instance domain
- List by view cody analytics permissions
---------
Co-authored-by: Robert Lin <robert@bobheadxi.dev>
Enterprise Portal equivalent of the `archiveProductSubscription` GraphQL
mutation. It's not a "delete" operation - we retain the record forever -
so it's idempotent, as an archive on an already-archived subscription
can return OK. If we want to add a hard delete we should add a separate
one based more closely on https://google.aip.dev/135.
It's not part of "update" because it's closer to a "delete" than
anything else, and currently cannot be undone.
Closes CORE-153
## Test plan
CI
Enterprise Portal equivalent of the `revokeLicense` GraphQL mutation.
It's not a "delete" operation - we currently retain the record forever -
so it's idempotent, as the revocation of an already-revoked license can
return OK. If we want to add a hard delete we should add a separate one
based more closely on https://google.aip.dev/135.
Related: https://github.com/sourcegraph/sourcegraph/pull/63330
Closes CORE-153
## Test plan
CI
Defines an RPC for creating licenses for a subscription in Enterprise
Portal (which will replace the existing
`generateProductLicenseForSubscription` GraphQL mutation). Because of
#63317 we will no longer accept Salesforce details as part of the input
for creating licenses. Most everything else stays the same.
https://google.aip.dev/133 says we must name the parent subscription ID
as the `parent` parameter. But, we already have a `subscription_id` in
`EnterpriseSubscriptionLicense`, which feels more descriptive, so I've
opted not to use `parent`... but I'm open to suggestions otherwise 🤷
Closes CORE-150, implementation tracked in CORE-157
## Test plan
CI
Today Salesforce "subscription ID" and "opportunity ID" are attached at
the license level. They should really be on the Enterprise subscription
level instead, which are parent entities of licenses. As we move forward
with CORE-149, I think this is a good chance to introduce this change in
our schema, and migrate the currently per-license information to the
subscription level when we move things into Enterprise Portal.
## Test plan
CI
Proposed RPC definition for creating an Enterprise Subscription, which
is currently the `createProductSubscription` GraphQL mutation, via
Enterprise Portal. The RPC follows patterns suggested for "create" RPCs
in https://google.aip.dev/133.
One thing we didn't account for yet is the way AIP suggests handling IDs
and names. It raises some interesting points about user-provided ID
components in https://google.aip.dev/133#user-specified-ids - I'm pretty
sure we don't want to introduce a user-controlled component to the
currently randomized subscription ID, _but_ we may want to introduce a
"slug" identifier of sorts that must be unique. The default value can
use something like https://pkg.go.dev/github.com/gosimple/slug to
convert the display name. This would help us ensure duplicate
subscriptions for the same customer aren't created.
Closes CORE-149. Implementation is tracked in a follow-up (CORE-155) -
for now this will just return "unimplemented".
## Test plan
CI
Proposed RPC definition for updating Cody Gateway access, which is
currently part of the `updateProductSubscription` GraphQL mutation, via
Enterprise Portal. The RPC follows patterns suggested for "update" RPCs
in https://google.aip.dev/134.
As an example, to use `UpdateCodyGatewayAccess` to update a
subscription's code completions limit:
```ts
UpdateCodyGatewayAccess({
access: {
subscription_id: "es_...",
code_completions: { limit: 420 },
},
update_mask: {
paths: ["code_completions.limit"],
},
})
```
Closes CORE-154. Implementation is tracked in a follow-up (CORE-159) -
for now this will just return "unimplemented".
I've also removed the "draft state" docstrings.
## Test plan
CI
This patch does a few things:
- Adds `go-enry` packages to depguard, so that people do not
accidentally use enry APIs instead of the corresponding APIs
in the `languages` package.
- Adds more tests for different functions in the languages package
to ensure mutual consistency in how language<->extension mappings
are handled.
- Adds tests for enry upgrades
- Adds comments with IDs so that related parts in the code can be
pieced together easily
Refactors usage of teestore to use new default `telemetry.EventsStore`
constructor. Default middleware can be added to the constructor to more
easily make transformations to server side generated events.
Closes CORE-123
## Test plan
CI
Adds support to the lang file filter for Magik language (`lang: Magik`).
In order to do that we add wrappers around go-enry and update search
code to use them. This provides flexibility for us in the future to
support other languages that are not in Linguist as well.
Migrates Cody Gateway to use the new Enterprise Portal's "read-only"
APIs. For the most part, this is an in-place replacement - a lot of the
diff is in testing and minor changes. Some changes, such as the removal
of model allowlists, were made down the PR stack in
https://github.com/sourcegraph/sourcegraph/pull/62911.
At a high level, we replace the data requested by
`cmd/cody-gateway/internal/dotcom/operations.graphql` and replace it
with Enterprise Portal RPCs:
- `codyaccessv1.GetCodyGatewayAccess`
- `codyaccessv1.ListCodyGatewayAccesses`
Use cases that previously required retrieving the active license tags
now:
1. Use the display name provided by the Cody Access API
https://github.com/sourcegraph/sourcegraph/pull/62968
2. Depend on the connected Enterprise Portal dev instance to only return
dev subscriptions https://github.com/sourcegraph/sourcegraph/pull/62966
Closes https://linear.app/sourcegraph/issue/CORE-98
Related to https://linear.app/sourcegraph/issue/CORE-135
(https://github.com/sourcegraph/sourcegraph/pull/62909,
https://github.com/sourcegraph/sourcegraph/pull/62911)
Related to https://linear.app/sourcegraph/issue/CORE-97
## Local development
This change also adds Enterprise Portal to `sg start dotcom`. For local
development, we set up Cody Gateway to connect to Enterprise Portal such
that zero configuration is needed - all the required secrets are sourced
from the `sourcegrah-local-dev` GCP project automatically when you run
`sg start dotcom`, and local Cody Gateway will talk to local Enterprise
Portal to do the Enterprise subscriptions sync.
This is actually an upgrade from the current experience where you need
to provide Cody Gateway a Sourcegraph user access token to test
Enterprise locally, though the Sourcegraph user access token is still
required for the PLG actor source.
The credential is configured in
https://console.cloud.google.com/security/secret-manager/secret/SG_LOCAL_DEV_SAMS_CLIENT_SECRET/overview?project=sourcegraph-local-dev,
and I've included documentation in the secret annotation about what it
is for and what to do with it:
## Rollout plan
I will open PRs to set up the necessary configuration for Cody Gateway
dev and prod. Once reviews taper down I'll cut an image from this branch
and deploy it to Cody Gateway dev, and monitor it closely + do some
manual testing. Once verified, I'll land this change and monitor a
rollout to production.
Cody Gateway dev SAMS client:
https://github.com/sourcegraph/infrastructure/pull/6108
Cody Gateway prod SAMS client update (this one already exists):
```
accounts=> UPDATE idp_clients
SET scopes = scopes || '["enterprise_portal::subscription::read", "enterprise_portal::codyaccess::read"]'::jsonb
WHERE id = 'sams_cid_018ea062-479e-7342-9473-66645e616cbf';
UPDATE 1
accounts=> select name, scopes from idp_clients WHERE name = 'Cody Gateway (prod)';
name | scopes
---------------------+----------------------------------------------------------------------------------------------------------------------------------
Cody Gateway (prod) | ["openid", "profile", "email", "offline_access", "enterprise_portal::subscription::read", "enterprise_portal::codyaccess::read"]
(1 row)
```
Configuring the target Enterprise Portal instances:
https://github.com/sourcegraph/infrastructure/pull/6127
## Test plan
Start the new `dotcom` runset, now including Enterprise Portal, and
observe logs from both `enterprise-portal` and `cody-gateway`:
```
sg start dotcom
```
I reused the test plan from
https://github.com/sourcegraph/sourcegraph/pull/62911: set up Cody
Gateway external dependency secrets, then set up an enterprise
subscription + license with a high seat count (for a high quota), and
force a Cody Gateway sync:
```
curl -v -H 'Authorization: bearer sekret' http://localhost:9992/-/actor/sync-all-sources
```
This should indicate the new sync against "local dotcom" fetches the
correct number of actors and whatnot.
Using the local enterprise subscription's access token, we run the QA
test suite:
```sh
$ bazel test --runs_per_test=2 --test_output=all //cmd/cody-gateway/qa:qa_test --test_env=E2E_GATEWAY_ENDPOINT=http://localhost:9992 --test_env=E2E_GATEWAY_TOKEN=$TOKEN
INFO: Analyzed target //cmd/cody-gateway/qa:qa_test (0 packages loaded, 0 targets configured).
INFO: From Testing //cmd/cody-gateway/qa:qa_test (run 1 of 2):
==================== Test output for //cmd/cody-gateway/qa:qa_test (run 1 of 2):
PASS
================================================================================
INFO: From Testing //cmd/cody-gateway/qa:qa_test (run 2 of 2):
==================== Test output for //cmd/cody-gateway/qa:qa_test (run 2 of 2):
PASS
================================================================================
INFO: Found 1 test target...
Target //cmd/cody-gateway/qa:qa_test up-to-date:
bazel-bin/cmd/cody-gateway/qa/qa_test_/qa_test
Aspect @@rules_rust//rust/private:clippy.bzl%rust_clippy_aspect of //cmd/cody-gateway/qa:qa_test up-to-date (nothing to build)
Aspect @@rules_rust//rust/private:rustfmt.bzl%rustfmt_aspect of //cmd/cody-gateway/qa:qa_test up-to-date (nothing to build)
INFO: Elapsed time: 13.653s, Critical Path: 13.38s
INFO: 7 processes: 1 internal, 6 darwin-sandbox.
INFO: Build completed successfully, 7 total actions
//cmd/cody-gateway/qa:qa_test PASSED in 11.7s
Stats over 2 runs: max = 11.7s, min = 11.7s, avg = 11.7s, dev = 0.0s
Executed 1 out of 1 test: 1 test passes.
```
Part of CORE-99
This PR scaffolds the database schema and code structure based on
[CORE-99
comment](https://linear.app/sourcegraph/issue/CORE-99/enterprise-portal-design-sams-user-to-subscription-rpcs#comment-8105ac31)
with some modifications. See inline comments for more elaborations.
- It uses GORM's ONLY for auto migration, just to kick things off, we
may migrate to file-based migration like we are planning for SAMS.
- It then uses the `*pgxpool.Pool` as the DB interface for executing
business logic queries.
Additionally, refactored `subscriptionsservice/v1.go` to use a `Store`
that provide single interface for accessing data(base), as we have been
doing in SAMS and SSC.
## Test plan
Enterprise Portal starts locally, and database is initialized:
Part of CORE-99
After some heavy discussion, we settled on "Roles + Permissions" (that
is a whole different large topic), but as far as the RPCs here
concerned, using "scopes" here is not appropriate.
With this patch, the `errors.HasType` API behaves similar to `Is` and `As`,
where it checks the full error tree instead of just checking a linearized version
of it, as cockroachdb/errors's `HasType` implementation does not respect
multi-errors.
As a consequence, a bunch of relationships between HasType and Is/As that
you'd intuitively expect to hold are now true; see changes to `invariants_test.go`.
Part of CORE-99
This PR adds new RPCs and fields for add and get/list subscription
domain and members, prepare to be used by Cody Analytics related
operations.
This is a simplified version of what we have for in-Sourcegraph db
connections as well. They're not directly compatible because we use
`pgx` hooks.
## Test plan
n/a
