This is a partial revert commit of
cbd12608b5.
We added support for NoTransaction but it isn't needed anymore. In fact
avoiding transactions leads to issues like poisoning connections.
Test Plan: CI
We hit a deadlock when deploying this migration to s2. This is because
within our transaction of the migration we likely didn't obtain table
locks in the same order as a transaction in our application code.
So this commit introduces a new migration metadata field
"noTransaction". The documentation for migrator says you should create a
migration per needed transactions. However, this would require us to
create 100s of migrations. We believe the better approach is introducing
this field and barely advertising it.
When reading the code which actually runs migrations, there is no extra
logic done outside of BEGIN; run_migration; COMMIT; so this change is
safe.
We update the migrations to avoid duplicating the function name we
introduce in case something goes wrong (now that the transaction could
leak out the function name).
Test Plan: The actual migrations are tested by go test. I added a test
assertion that we don't call Transact, but to be honest that is super
sketchy. However, we couldn't actually find any test fixtures which
actually run against the DB. So that would require a much deeper
investment for how simple the code change is.
Co-authored-by: Erik Seliger <erikseliger@me.com>
Sourcegraph's GitHub discussions have been defunct for awhile, but links
to them persisted in READMEs and documentation.
## Test plan
Documentation update only.
2nd attempt of #63111, a follow up
https://github.com/sourcegraph/sourcegraph/pull/63085
rules_oci 2.0 brings a lot of performance improvement around oci_image
and oci_pull, which will benefit Sourcegraph. It will also make RBE
faster and have less load on remote cache.
However, 2.0 makes some breaking changes like
- oci_tarball's default output is no longer a tarball
- oci_image no longer compresses layers that are uncompressed, somebody
has to make sure all `pkg_tar` targets have a `compression` attribute
set to compress it beforehand.
- there is no curl fallback, but this is fine for sourcegraph as it
already uses bazel 7.1.
I checked all targets that use oci_tarball as much as i could to make
sure nothing depends on the default tarball output of oci_tarball. there
was one target which used the default output which i put a TODO for
somebody else (somebody who is more on top of the repo) to tackle
**later**.
## Test plan
CI. Also run delivery on this PR (don't land those changes)
---------
Co-authored-by: Noah Santschi-Cooney <noah@santschi-cooney.ch>
- Update the "getting started" link in the quickstart guide to point to
the correct URL
- Remove the outdated developer help links as they are no longer
relevant
## Test plan
Test the links manually
Since we don't do the enterprise/oss split anymore, this global package
is no longer required and we can move the code to where it's actually
used.
Test plan: Go compiler doesn't complain, and integration tests are still
passing.
Removes the `sg telemetry` command that pertains to the legacy V1
exporter that is specific to Cloud instances.
I got asked about this recently, and especially with the new `sg
analytics` for usage of the `sg` CLI, this has the potential to be
pretty confusing.
Part of https://linear.app/sourcegraph/issue/CORE-104
## Test plan
n/a
## Changelog
- `sg`: the deprecated `sg telemetry` command for allowlisting export of
V1 telemetry from Cloud instances has been removed. Use telemetry V2
instead.
Historically, sourcegraph.com has been the only instance. It was
connected to GitHub.com and GitLab.com only.
Configuration should be as simple as possible, and we wanted everyone to
try it on any repo. So public repos were added on-demand when browsed
from these code hosts.
Since, dotcom is no longer the only instance, and this is a special case
that only exists for sourcegraph.com.
This causes a bunch of additional complexity and various extra code
paths that we don't test well enough today.
We want to make dotcom simpler to understand, so we've made the decision
to disable that feature, and instead we will maintain a list of
repositories that we have on the instance.
We already disallowed several repos half a year ago, by restricting size
of repos with few stars heavily.
This is basically just a continuation of that.
In the diff, you'll mostly find deletions. This PR does not do much
other than removing the code paths that were only enabled in dotcom mode
in the repo syncer, and then removes code that became unused as a result
of that.
## Test plan
Ran a dotcom mode instance locally, it did not behave differently than a
regular instance wrt. repo cloning.
We will need to verify during the rollout that we're not suddenly
hitting code paths that don't scale to the dotcom size.
## Changelog
Dotcom no longer clones repos on demand.
Closes CORE-99, closes CORE-176
This PR is based off (and was also served as PoC of) [RFC 962: MSP IAM
framework](https://docs.google.com/document/d/1ItJlQnpR5AHbrfAholZqjH8-8dPF1iQcKh99gE6SSjs/edit).
It comes with two main parts:
1. The initial version of the MSP IAM SDK:
`lib/managedservicesplatform/iam`
- Embeds the [OpenFGA server
implementation](https://github.com/openfga/openfga/tree/main/pkg/server)
and exposes the a `ClientV1` for interacting with it.
- Automagically manages the both MSP IAM's and OpenFGA's database
migrations upon initializing the `ClientV1`.
- Ensures the specified OpenFGA's store and automatization model DSL
exists.
- Utility types and helpers to avoid easy mistakes (i.e. make the
relation tuples a bit more strongly-typed).
- Decided to put all types and pre-defined values together to simulate a
"central registry" and acting as a forcing function for services to form
some sort of convention. Then when we migrate the OpenFGA server to a
separate standalone service, it will be less headache about
consolidating similar meaning types/relations but different string
literals.
1. The first use case of the MSP IAM:
`cmd/enterprise-portal/internal/subscriptionsservice`
- Added/updated RPCs:
- Listing enterprise subscriptions via permissions
- Update enterprise subscriptions to assign instance domains
- Update enterprise subscriptions membership to assign roles (and
permissions)
- A database table for enterprise subscriptions, only storing the extra
instance domains as Enterprise Portal is not the
writeable-source-of-truth.
## Other minor changes
- Moved `internal/redislock` to `lib/redislock` to be used in MSP IAM
SDK.
- Call `createdb ...` as part of `enterprise-portal` install script in
`sg.config.yaml` (`msp_iam` database is a hard requirement of MSP IAM
framework).
## Test plan
Tested with gRPC UI:
- `UpdateEnterpriseSubscription` to assign an instance domain
- `UpdateEnterpriseSubscriptionMembership` to assign roles
- `ListEnterpriseSubscriptions`:
- List by subscription ID
- List by instance domain
- List by view cody analytics permissions
---------
Co-authored-by: Robert Lin <robert@bobheadxi.dev>
It seems many of our doc links for code hosts are broken in production
due to a url changed from external_services to code_hosts. I did a find
an replace to update all the ones I could find.
This adds `Inter` as the preferred for `--font-family-base` in the
Svelte webapp. Previously, we were using system fonts for our base font,
which meant our web app looked different depending on the OS+browser.
Instead of depending on the Google Fonts CDN, this uses Fontsource,
which let us easily self-host Google Fonts (and other OSS fonts) just by
importing an NPM package. No committing font files to git, and no
dependency or connection to a 3rd-party server. The font itself weighs
~40KB for the full, variable-weight font.
Contributes to SRCH-445
With these additions, the instructions on how to run Cody Gateway
locally should be correct now.
## Test plan
The proposed changes work for me locally.
Follow up https://github.com/sourcegraph/sourcegraph/pull/63085
rules_oci 2.0 brings a lot of performance improvement around oci_image
and oci_pull, which will benefit sourcegraph. It will also make RBE
faster and have less load on remote cache.
However, 2.0 makes some breaking changes like
- oci_tarball's default output is no longer a tarball
- oci_image no longer compresses layers that are uncompressed, somebody
has to make sure all `pkg_tar` targets have a `compression` attribute
set to compress it beforehand.
- there is no curl fallback, but this is fine for sourcegraph as it
already uses bazel 7.1.
I checked all targets that use oci_tarball as much as i could to make
sure nothing depends on the default tarball output of oci_tarball. there
was one target which used the default output which i put a TODO for
somebody else (somebody who is more on top of the repo) to tackle later.
## Test plan
I am assuming that the repo has enough tests to catch potential problems
on CI. Also somebody who knows the repo better should double check my
changes.
---------
Co-authored-by: Noah Santschi-Cooney <noah@santschi-cooney.ch>
The first attempt didn't work as there are other exit conditions for the
stream version than just calling RecvMsg until io.EOF. I found that gRPC
has a callback for onFinish, and this seems to work properly locally.
See commit number 2 for the diff over the initial implementation.
## Test plan
Verified locally that all connection counts drop to zero eventually.
This reverts commit 9185da3c3e.
Noticed there are some bad callers in worker and symbols that don't
properly return a connection. Will need to investigate and fix that
first.
## Test plan
Worked before, CI passes.
This used to exist in the HTTP world, and we currently have zero
safeguards to prevent clients from making one billion requests
concurrently.
Until we invest more into server-side rate limiting, or per tenant rate
limiting, we reintroduce this limiter, to prevent resource usage spikes.
Test plan:
Added a test suite.
---------
Co-authored-by: Geoffrey Gilmore <geoffrey@sourcegraph.com>
Since we split out this service, we lost a few metrics on call counts and latencies.
This PR adds them back.
Closes#62785
Test plan:
Ran the dashboards locally and they return data. These dashboards are a 1:1 replica of the git service observability.
This PR refactors a batch changes example that uses `patterntype:structural` to
use regex instead. We no longer expose `patterntype:structural` by default, and
want to push users towards other pattern types.
* Add container test for s3proxy script
* Add explanatory comments
* Fixup s3proxy container structure test
* Update docs to mention a caveat of the setup steps
* Fix hostname
Turns out I blindly copy-pasted this from elsewhere and the graphs always said
GraphQL operations, which is wrong and confused people.
Test plan:
Reads better now.
This PR adds additional observation tools and warning logs for git commands that required a lot of memory.
That should help us better identify where potential for OOMs exists and what endpoints could benefit from optimization.
```
[ gitserver-0] WARN gitserver.cleanup gitcli/command.go:307 High memory usage exec request {"TraceId": "f70c73e500ed7831207ce9a7c6dc63fb", "SpanId": "705d1dcfd0b44a06", "ev.Fields": {"exit_status": "0", "cmd_duration_ms": "1944", "user_time": "234.915ms", "cmd_ru_minflt": "10231", "cmd_ru_majflt": "7", "duration_ms": "1944", "trace": "https://sourcegraph.test:3443/-/debug/jaeger/trace/f70c73e500ed7831207ce9a7c6dc63fb", "cmd_ru_maxrss_kib": "160672", "actor": "0", "traceID": "f70c73e500ed7831207ce9a7c6dc63fb", "repo": "github.com/sourcegraph/sourcegraph", "args": "[git commit-graph write --reachable --changed-paths --size-multiple=4 --split]", "system_time": "1.679428s", "cmd_ru_inblock": "0", "cmd_ru_oublock": "0", "cmd": "commit-graph"}}
```
Test plan:
Tried this locally using some command I know will use a bunch of memory, see test output above.
Migrates the Telemetry Gateway:
1. Service specification
2. Generated Go bindings
3. UUID constructor
into an exported `lib/telemetrygateway` package for internal and external consumption. See https://github.com/sourcegraph/sourcegraph/issues/61489 for use cases. This allows MSP services to more easily start publishing events for to Telemetry Gateway, and adds no new dependencies to `lib`.
Splits Sourcegraph-specific functionality that used to live in the `telemetrygateway/v1` package to:
1. `internal/telemetrygateway`: backcompat testing
2. `internal/telmeetrygateway/event`: event constructors (collapsing into parent caused import cycle)
I've left README + a stub service spec in the old package to redirect visitors from outdated links.
Closes https://github.com/sourcegraph/sourcegraph/issues/61489
## Test plan
```
sg start
```
watch for successful export logs from `telemetrygatewayexporter`
This removes qdrant from this codebase entirely.
All the docker images, dependencies, (dead) usage in code.
My understanding is that we don't use this feature and never properly rolled it out.
Test plan:
CI passes and code review from owners.
* create base skeleton for sg cloud deploy
* move sanitasion of branch name to happen inside images
* Add cloud api client for ephemeral
Created package sg/internal/cloud which connects to the cloud api.
From the examples it looks like we need to impersonate a service account
to get a token. For now I am just print the token from gcloud but will
sync with the cloud team on this tomorrow
* active cloud-ephemeral runtype by env var
* test kick off of cloud ephemeral build
* check if commit exists remotely before building
* add CLOUD_EPHEMERAL env to pipline env
* push with respec always and use force with lease
* use pending for longer running ops
* add cloud-api as approved dependency
