Commit Graph

33 Commits

Author SHA1 Message Date
Robert Lin
557b4df0ed
chore/deps: upgrade grpc, prometheus/common (#63328)
This change extracts the unrelated transitive upgrades of
https://github.com/sourcegraph/sourcegraph/pull/63171 (CORE-177) into a
separate PR. I'm making this because @unknwon ran into issues with the
exact same dependencies in
https://github.com/sourcegraph/sourcegraph/pull/63171#issuecomment-2157694545.

The change consists of upgrades to:

- `google.golang.org/grpc` - there's a deprecation of `grpc.DialContext`
that we agreed in #63171 to keep for now.
- removing our `replace` directive on `github.com/prometheus/common` and
upgrading it. This is safe to do because our Alertmanager version is
already way ahead, and the reason this has a `replace` is outdated now.

## Test plan

CI, nothing blows up on `sg start` and I can click around and do a bit
of searching
2024-06-19 09:55:44 -04:00
Robert Lin
704b36a143
feat/enterprise-portal: ConnectRPC layer for {Get/List}CodyGatewayAccess (#62771)
This PR exposes the data layer implemented in https://github.com/sourcegraph/sourcegraph/pull/62706 via the Enterprise Portal API. We register the services proposed in #62263 and also set up tooling like gRPC UI locally for DX.

Auth is via SAMS M2M; https://github.com/sourcegraph/sourcegraph-accounts-sdk-go/pull/28 and https://github.com/sourcegraph/sourcegraph-accounts/pull/227 rolls out the new scopes, and https://github.com/sourcegraph/managed-services/pull/1474 adds credentials for the enterprise-portal-dev deployment.

Closes CORE-112

## Test plan

https://github.com/sourcegraph/sourcegraph/pull/62706 has extensive testing of the data layer, and this PR expands on it a little bit. I tested the RPC layer by hand:

Create SAMS client for Enterprise Portal Dev in **accounts.sgdev.org**:

```sh
curl -s -X POST \
        -H "Authorization: Bearer $MANAGEMENT_SECRET" \
        https://accounts.sgdev.org/api/management/v1/identity-provider/clients \
--data '{"name": "enterprise-portal-dev", "scopes": [], "redirect_uris": ["https://enterprise-portal.sgdev.org"]}' | jq
```

Configure `sg.config.overwrite.yaml`

```yaml
  enterprise-portal:
    env:
      SRC_LOG_LEVEL: debug
      # sams-dev
      SAMS_URL: https://accounts.sgdev.org
      ENTERPRISE_PORTAL_SAMS_CLIENT_ID: "sams_cid_..."
      ENTERPRISE_PORTAL_SAMS_CLIENT_SECRET: "sams_cs_..."
```

Create a test client (later, we will do the same thing for Cody Gateway), also in **accounts.sgdev.org**:

```sh
curl -s -X POST \
        -H "Authorization: Bearer $MANAGEMENT_SECRET" \
        https://accounts.sgdev.org/api/management/v1/identity-provider/clients \
--data '{"name": "enterprise-portal-dev-reader", "scopes": ["enterprise_portal::codyaccess::read", "enterprise_portal::subscription::read"], "redirect_uris": ["https://enterprise-portal.sgdev.org"]}' | jq
```

Then:

```
sg run enterprise-portal
```

Navigate to the locally-enabled gRPC debug UI at http://localhost:6081/debug/grcpui, using https://github.com/sourcegraph/sourcegraph/pull/62883 to get an access token from our test client to add in the request metadata:

```sh
sg sams create-client-token -s 'enterprise_portal::codyaccess::read'
```

I'm using some local subscriptions I've made previously in `sg start dotcom`:

![image](https://github.com/sourcegraph/sourcegraph/assets/23356519/a55c6f0d-b0ae-4e68-8e4c-ccb6e2cc442d)

![image](https://github.com/sourcegraph/sourcegraph/assets/23356519/19d18104-1051-4a82-abe0-58010dd13a27)

Without a valid authorization header:

![image](https://github.com/sourcegraph/sourcegraph/assets/23356519/c9cf4c89-9902-48f8-ac41-daf9a63ca789)

Verified a lookup using the returned access tokens also works

---------

Co-authored-by: Jean-Hadrien Chabran <jh@chabran.fr>
Co-authored-by: Joe Chen <joe@sourcegraph.com>
2024-05-27 13:39:57 -07:00
Joe Chen
2589fef13e
lib/background: upgrade Routine interface with context and errors (#62136)
This PR is a result/followup of the improvements we've made in the [SAMS repo](https://github.com/sourcegraph/sourcegraph-accounts/pull/199) that allows call sites to pass down a context (primarily to indicate deadline, and of course, cancellation if desired) and collects the error returned from `background.Routine`s `Stop` method.

Note that I did not adopt returning error from `Stop` method because I realize in monorepo, the more common (and arguably the desired) pattern is to hang on the call of `Start` method until `Stop` is called, so it is meaningless to collect errors from `Start` methods as return values anyway, and doing that would also complicate the design and semantics more than necessary.

All usages of the the `background.Routine` and `background.CombinedRoutines` are updated, I DID NOT try to interpret the code logic and make anything better other than fixing compile and test errors.

The only file that contains the core change is the [`lib/background/background.go`](https://github.com/sourcegraph/sourcegraph/pull/62136/files#diff-65c3228388620e91f8c22d91c18faac3f985fc67d64b08612df18fa7c04fafcd).
2024-05-24 10:04:55 -04:00
Noah S-C
9b6ba7741e
bazel: transcribe test ownership to bazel tags (#62664) 2024-05-16 15:51:16 +01:00
William Bezuidenhout
1ae6cc6bfd
logger: update log lib and remove use of description (#57690)
* log: remove use of description paramter in Scoped

* temporarily point to sglog branch

* bazel configure + gazelle

* remove additional use of description param

* use latest versions of zoekt,log,mountinfo

* go.mod
2023-10-18 17:29:08 +02:00
Geoffrey Gilmore
3cb0b12b11
grpc: put grpcwebui behind environment variable (#56314) 2023-08-31 15:13:16 +01:00
Robert Lin
61ce005b83
internal/trace: remove net/trace integration (#53795)
Follow-up from
https://github.com/sourcegraph/sourcegraph/pull/53435#issuecomment-1597050970
and INC-214 - this removes `net/trace` instrumentation entirely in the
next release.

It seems [this gets low usage and may be difficult to
use](https://github.com/sourcegraph/sourcegraph/pull/53795#pullrequestreview-1489890896),
and it seems to introduce some risk and overhead. This will also make it
easier to drop our internal trace wrapper entirely and/or integrate it
as part of OpenTelemetry's APIs instead of having a proprietary way of
doing traces.

## Test plan

Already tested in https://github.com/sourcegraph/sourcegraph/pull/53435
2023-06-21 18:35:07 +00:00
Robert Lin
a346366d61
trace: add SRC_ENABLE_NET_TRACE to disable net/trace (#53435)
This change adds `SRC_ENABLE_NET_TRACE`, which if set to `false` will
disable `net/trace` support in `internal/trace` and our debug routers.
The plan is to just set this to `false` in Sourcegraph.com
(https://github.com/sourcegraph/deploy-sourcegraph-cloud/pull/17937)
until we can absolutely confirm that `net/trace` is an issue - it's also
possible this is only problematic for Sourcegraph.com levels of traffic.
Follow-ups will be in
[INC-214](https://sourcegraph.slack.com/archives/C05CGC5UL3E).

Context: today we noticed unreliability in Sourcegraph.com,
[INC-214](https://sourcegraph.slack.com/archives/C05CGC5UL3E), where a
lot of requests would get no responses. We diagnosed the issue to be a
frontend pod that had a huge goroutine leak, causing unreliability in
sourcegraph.com for requests routed to the affected pod. Removing the
affected pod immediately resolved the stability issues.

<img width="300" alt="image"
src="https://github.com/sourcegraph/sourcegraph/assets/23356519/b158305c-8724-463c-b5e5-0c20a3ea9d72">
<img width="500" alt="image"
src="https://github.com/sourcegraph/sourcegraph/assets/23356519/8780f3ca-9f0d-496e-9daf-f7e46d2c969a">

It appears this may be caused by increased usage of `internal/trace`,
which [creates a trace with
`golang.org/x/net/trace`](https://sourcegraph.com/github.com/sourcegraph/sourcegraph@f3b0276dfe24018db5f76d5d8d430db98e59f2dd/-/blob/internal/trace/tracer.go?L32-33),
which then [acquires a global
mutex](https://sourcegraph.com/go/golang.org/x/net/-/blob/trace/trace.go?L369-380).
Cloud Profile indicates this trend started ~June 5, which _somewhat_
correlates with a series of changes that removed OpenTracing usages and
replaced them with OpenTelemetry/`internal/trace`, the latter of which
ends up in this code path.

<img width="979" alt="image"
src="https://github.com/sourcegraph/sourcegraph/assets/23356519/304c6de7-6d0c-4a29-936f-b56405f8cae8">
<img width="1001" alt="image"
src="https://github.com/sourcegraph/sourcegraph/assets/23356519/48a281db-df81-4e3e-8455-90844ee008a4">

See incident discussion thread for more details:
https://sourcegraph.slack.com/archives/C05CGC5UL3E/p1686712614273539?thread_ts=1686710733.750869&cid=C05CGC5UL3E

## Test plan

`sg start`, code review, CI
2023-06-14 05:20:48 +00:00
Geoffrey Gilmore
5d2b832db5
Revert "Revert "grpc: create interceptors to log / chart errors that seem like they come from go-grpc (#51749) (#52083)" (#52090) 2023-05-17 20:04:10 +00:00
Geoffrey Gilmore
e14f783c7e
Revert "grpc: create interceptors to log / chart errors that seem like they come from go-grpc (#51749)" (#52083) 2023-05-17 16:50:16 +00:00
Geoffrey Gilmore
520eaf0310
grpc: create interceptors to log / chart errors that seem like they come from go-grpc (#51749) 2023-05-17 09:14:07 -07:00
Geoffrey Gilmore
834f893d18
grpc: add support for frontend's indexed-search internal API methods (#47710)
Co-authored-by: Camden Cheek <camden@ccheek.com>
2023-03-08 02:15:54 +00:00
Dave Try
83746c1167
bazel: update build files (#47574)
Updates the BUILD.bazel files as well as various deps after main drifted
from our last working config.

Most notably the move from `proto` to `v1` in `internal/symbols`,
`internal/searcher` and `internal/own/codeowners`.

Gazelle fixed almost everything with the exception of a few things. I
ran `bazel run //:gazelle` again after `bazel run //:gazelle --
update-repos` which appended a bunch of new rules, instead of replacing
the old ones.


## Test plan

```bazel build //...```

https://buildkite.com/sourcegraph/sourcegraph/builds/198456

---------

Co-authored-by: Jean-Hadrien Chabran <jh@chabran.fr>
2023-02-14 15:36:39 +01:00
Camden Cheek
4d3db96e92
grpc gitserver exec (#47290) 2023-02-09 09:52:32 +01:00
Jean-Hadrien Chabran
9c481afcb4
Bazel: build //cmd/... (+ rules for buf and fix grpc proto compiler) (#47270)
* Add buf rules + fix grpc proto generation

* Build all the cmd targets
2023-02-01 13:14:33 -05:00
Geoffrey Gilmore
245f89d08f
debugserver: add /debug/grpcui route for exposing GRPCUI for symbols and searcher (#47128) 2023-01-31 20:56:22 +00:00
Jean-Hadrien Chabran
bc5490c4bb
bazel: introduce build files for Go (#46770) 2023-01-23 14:00:01 +01:00
Joe Chen
4d85487c1e
debugserver: remove DB interface for debug dump (#38140) 2022-07-04 16:22:04 +08:00
Joe Chen
67f2ce8349
repo-updater: allow filter debug dumper by name (#37915) 2022-06-29 16:13:23 +08:00
Keegan Carruthers-Smith
18f487ccaa
all: use any instead of interface{} (#35102)
Now that we require go1.18, we can use a builtin type alias for
"interface{}": "any". I've updated all code except lib, which can be
imported by external modules. That is currently pinned at go1.16.
Additionally I had to rerun generate since rewriting generated go code
will fail CI.

  find -name '*.go' | xargs gofmt -s -r 'interface{} -> any' -w
  git checkout lib
  go generate ./...

Test Plan: CI will exercise that the code still compiles. Otherwise this
is a noop.
2022-05-09 10:59:39 +02:00
Eric Fritz
7c111fecff
debugserver: Add /healthz and /ready endpoints (#19846) 2021-04-08 20:21:29 +00:00
Robert Lin
de2a805a55
monitoring: version metric, annotations on versions and alerts (#17198)
* All services now export a Gauge, `src_service_metadata`, indicating their Sourcegraph version (and possibly other metadata in the future)
* Add a (currently disabled by default) annotation layer for each service that adds an annotation per version/instance when version labels change.
* Add a (currently disabled by default) annotation layer for each service that shows all service alerts
2021-01-13 00:14:18 +08:00
Eric Fritz
e0eb8937ca
chore: Use httpserver package in frontend startup (#15707) 2020-11-18 15:30:27 -06:00
flying-robot
c488cdd6d0
debugserver: handle wildcard pprof requests (#15587)
@tsenart and I found while pairing that some pprof endpoints in debugserver would result in a 404. We traced it back and found that the router had switched[0] from the stdlib to gorilla/mux. The latter is explicit in its route handling, so shifting to `PathPrefix` and moving it further down the list will fix the problem.

On main:

```
$ curl -s -o /dev/null -w "%{http_code}" http://sourcegraph.test:6074/debug/pprof/goroutine\?debug\=1
404
$ curl -s -o /dev/null -w "%{http_code}" http://sourcegraph.test:6074/debug/pprof/mutex\?debug\=1
404
...
```

With this change:

```
$ curl -s -o /dev/null -w "%{http_code}" http://sourcegraph.test:6074/debug/pprof/goroutine\?debug\=1
200
$ curl -s -o /dev/null -w "%{http_code}" http://sourcegraph.test:6074/debug/pprof/mutex\?debug\=1
200
...
```

[0] https://github.com/sourcegraph/sourcegraph/pull/15282
2020-11-09 12:52:25 -05:00
Eric Fritz
66ec98093c
debugserver: Use httpserver (#15282) 2020-11-02 13:47:53 -06:00
Eric Fritz
ade9cd29f1
chore: Add additional utils for interacting with goroutines (#14840) 2020-10-19 10:15:54 -05:00
Keegan Carruthers-Smith
54a3283247
all: add keegancsmith to CODENOTIFY for many pkgs (#14241)
Random assortment of packages I am interested in being notified about.
2020-09-29 11:03:53 +02:00
Keegan Carruthers-Smith
a718276cbd
debugserver: add fgprof (#11748)
pprof which picks up IO. See README for motivation, seems very
useful. Adding it to debugserver so all our go services built in this
repo will get it for free.
2020-06-26 21:35:48 +02:00
ᴜɴᴋɴᴡᴏɴ
adf60bd73f
authz: add debug endpoint for PermsSyncer (#9046)
* Add debug endpoint for PermsSyncer

* Add size

* Update comment
2020-03-17 20:10:20 +08:00
uwedeportivo
d028a71506
app/debug: fixes no allowed in traces and restores headers link (#8879)
* app/debug: fixes no allowed in traces and restores headers link

* fix my unit test yay
2020-03-07 12:03:55 -08:00
Keegan Carruthers-Smith
8bf5e54ea2
all: ignore most warnings from errcheck (#7817)
These are mostly calls to Write that we can't do anything with if they
fail. Logging the failures is usually noisy since the most common reason for
failure is the client has gone away. This just marks us ignoring the error as
explicit.
2020-01-16 22:57:56 +02:00
Rijnard van Tonder
d03b895185
tidy: convert Fprintf to Fprintln when possible (#7174) 2019-12-12 12:46:02 -07:00
Nick Snyder
85fba22315
Rename pkg to internal (#5898)
* git mv pkg internal

* fix imports in Go files

find ./ -type f -name '*.go' -exec sed -i '' -e 's/sourcegraph\/pkg\//sourcegraph\/internal\//g' {} \;

* fix imports in scripts

find ./ -type f -name '*.sh' -exec sed -i '' -e 's/sourcegraph\/pkg\//sourcegraph\/internal\//g' {} \;

* fix launch.sh

* go fmt ./...

* fix CODEOWNERS
2019-10-07 15:36:41 -07:00