Commit Graph

51 Commits

Author SHA1 Message Date
Erik Seliger
83d0f6876c
dotcom: Remove on-demand cloning of repositories (#63321)
Historically, sourcegraph.com has been the only instance. It was
connected to GitHub.com and GitLab.com only.
Configuration should be as simple as possible, and we wanted everyone to
try it on any repo. So public repos were added on-demand when browsed
from these code hosts.

Since, dotcom is no longer the only instance, and this is a special case
that only exists for sourcegraph.com.
This causes a bunch of additional complexity and various extra code
paths that we don't test well enough today.

We want to make dotcom simpler to understand, so we've made the decision
to disable that feature, and instead we will maintain a list of
repositories that we have on the instance.
We already disallowed several repos half a year ago, by restricting size
of repos with few stars heavily.
This is basically just a continuation of that.

In the diff, you'll mostly find deletions. This PR does not do much
other than removing the code paths that were only enabled in dotcom mode
in the repo syncer, and then removes code that became unused as a result
of that.

## Test plan

Ran a dotcom mode instance locally, it did not behave differently than a
regular instance wrt. repo cloning.
We will need to verify during the rollout that we're not suddenly
hitting code paths that don't scale to the dotcom size.

## Changelog

Dotcom no longer clones repos on demand.
2024-06-26 14:53:14 -07:00
Matthew Manela
92b8ffb8e1
fix(Source): Fix documentation URLs for code hosts help pages (#63274)
It seems many of our doc links for code hosts are broken in production
due to a url changed from external_services to code_hosts. I did a find
an replace to update all the ones I could find.
2024-06-17 14:32:46 -04:00
Petri-Johan Last
710c734021
Add 'internal' case for repositoryQuery (#62266) 2024-05-02 15:32:39 +02:00
Camden Cheek
1ead945267
Docs: update links to point to new site (#60381)
We have a number of docs links in the product that point to the old doc site. 

Method:
- Search the repo for `docs.sourcegraph.com`
- Exclude the `doc/` dir, all test fixtures, and `CHANGELOG.md`
- For each, replace `docs.sourcegraph.com` with `sourcegraph.com/docs`
- Navigate to the resulting URL ensuring it's not a dead link, updating the URL if necessary

Many of the URLs updated are just comments, but since I'm doing a manual audit of each URL anyways, I felt it was worth it to update these while I was at it.
2024-02-13 00:23:47 +00:00
Thorsten Ball
80dfe159f3
github star/size exclusion: incorporate feedback from code review (#58398) 2023-11-17 10:21:24 +01:00
Thorsten Ball
8f0c04090c
github: add ability to exclude repositories based on size & stars (#58377) 2023-11-16 19:03:24 +01:00
Petri-Johan Last
d271395d10
Add GitHub Auth config options to manage internal repos (#56677) 2023-10-04 16:43:40 +02:00
Petri-Johan Last
a1460312c8
[GitHub App] Connections can clone all installation repositories (#53869) 2023-06-23 14:29:07 +02:00
Milan Freml
d648db4b23
[github apps] Add new code host connection screen for ghapps (#52012)
## Description

Adds specific changes to support github app code host connections in
create and edit code host connections screens.

Added:
- github app selector in the edit screen
- github app installation selector in edit screen
- github app instructions which are different than standard ghe
instructions
- validation of JSON schema for code host connections and disabling the
`Add connection` or `Update configuration` buttons at the bottom of the
page if the JSONC doc does not adhere to the schema

## Video

[Loom
video](https://www.loom.com/share/48898711281b4f369ba58613c5c26ff9)

## Test plan

Tested locally, only UI changes.
2023-05-18 13:02:15 +02:00
Petri-Johan Last
7f90642849
[github app] Add GitHub App authenticators (#50963) 2023-04-24 09:05:50 +02:00
Petri-Johan Last
1eee46e679
[cleanup] Remove old GitHub App code (#50933) 2023-04-21 07:27:27 +02:00
Kelli Rockwell
06fb837f1a
webhooks: create new namespace for incoming + outgoing (#49570)
Co-authored-by: Alex Ostrikov <alex.ostrikov@sourcegraph.com>
2023-03-17 16:52:18 -07:00
Milan Freml
ea59a0c996
[docs] fixed papercuts in new permission docs (#48470)
## Description

- Fixed some glaring issues with whitespace not being good
- Fixed sidebar saying `Repository Permissions` instead of `Repository
permissions`
- Fixed links to permissions docs page from everywhere
- Made SLAs more prominent and defined SLAs for all 3 types of getting
permissions to sourcegraph

Old version:
https://docs.sourcegraph.com/admin/permissions

Version from this PR:
https://docs.sourcegraph.com/@milan_perms_docs_fixes/admin/permissions

## Test plan

Just docs or copy changes, so eyeballing. And verifying if the links
work.
2023-03-02 08:48:33 +01:00
Ryan Slade
45d1e14c43
webhooks: Add deprecation notices (#45478)
Co-authored-by: Thorsten Ball <mrnugget@gmail.com>
2022-12-12 10:23:32 +01:00
Joe Chen
014e8742fe
schema: add "pending" field to GitHub code host connection (#32433) 2022-03-14 14:13:07 +08:00
Joe Chen
7c13225fca
[CLOUD-203] Handle GitHub App setup callback (#29467)
Co-authored-by: Milan Freml <kopancek@users.noreply.github.com>
Co-authored-by: Rafał Gajdulewicz <rafax@users.noreply.github.com>
2022-01-13 12:07:23 +08:00
Warren Gifford
2a226213bb
Clarify auth.providers is in site configuration (#28435)
* Clarify auth.providers is in site configuration

* improved added link

* needed to gen schema
2021-12-07 19:15:51 +00:00
Robert Lin
e34980d87e
authz/github: do not recommend groupsCacheTTL, document allowGroupsPermissionsSync (#24528) 2021-09-02 14:44:25 +00:00
davejrt
3e433c24bc
doc: add guidance about GitHub token permissions for caching (#24332)
* update schema with example for docs

* add note about token

* add token scope for permissions caching

* prettier

* schema update

* update repo token scope in schema

* Update doc/admin/repo/permissions.md

Co-authored-by: Robert Lin <robert@bobheadxi.dev>

Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2021-08-26 14:41:28 +00:00
Robert Lin
9917519a92
authz/github: sync user permissions from team/org perms caches (#23978)
Extend the GitHub provider's FetchUserPermsByToken to use cached groups (orgs or teams) permissions to reduce API calls required to do a full permissions sync. Also introduces options for performing permission syncs that can be provided at various levels of permissions syncing. For now, this is only added to user permissions, and is primarily used to provide an InvalidateCaches flag that signals the Provider to invalidate caches encountered during the sync, which is currently only used by webhook handlers to invalidate caches when a team/org-related event is received. The ScheduleUserPermissionsSync mutation and some webhook handlers also take advantage of this.

Co-authored-by: ᴜɴᴋɴᴡᴏɴ <joe@sourcegraph.com>
2021-08-24 22:14:07 -04:00
Ryan Slade
b30f570813
doc: Update comments regarding cloud_default (#20056) 2021-04-19 10:26:34 +02:00
Indradhanush Gupta
6483387afc
Add docs to explain burst settings for ratelimiter config
Co-authored-by: Ryan Slade <ryanslade@gmail.com>
2021-04-12 16:19:04 +05:30
Ryan Slade
3d5548947c
dev: Easier development in sourcegraph.com mode (#18172)
When running with SOURCEGRAPHDOTCOM_MODE=true set we require that at
least one GitHub and GitLab external service have their cloud_default
flags set.

This change allows us to enforce this by setting the CloudDefault field
on config in dev-private.
2021-02-12 09:19:14 +02:00
Ryan Slade
e549ca30f2
db: Use cloud_default column (#17519)
This change just switches to using the new cloud_default column on the external_services table instead of external service config to choose our "default" sources on cloud.
2021-01-22 10:19:39 +02:00
Ryan Slade
e49a57f0e2
cloud: Explicitly set global external service (#15748)
Add CloudGlobal flag to GitHub and GitLab external services

Validate that only one site admin owned service on cloud can have this flag set as true.

A followup PR will use these values one this PR has been deployed and the appropriate external services have been flagged.
2020-11-13 16:09:06 +02:00
ᴜɴᴋɴᴡᴏɴ
7295c5d2c6
schema: remove deprecated authz fields (#13395) 2020-08-27 15:38:35 +08:00
ᴜɴᴋɴᴡᴏɴ
806465d881
authz: remove on-demand permissions fetching (#12319)
Co-authored-by: Ryan Slade <ryanslade@gmail.com>
2020-07-22 09:25:45 +08:00
Erik Seliger
58407d02ad
Display webhook URL on external services page (#10634) 2020-05-14 14:38:23 +02:00
Ryan Slade
45b2997b5b
repo-updater: Add configurable rate limiter for GitHub (#9658) 2020-04-09 10:05:52 +02:00
Thorsten Ball
04d115f1d5
Document which GitHub token scopes are required (#9399)
* Document which GitHub token scopes are required

This fixes #9398.

* Move explanation of GitHub token scopes around
2020-03-30 17:16:40 +02:00
Keegan Carruthers-Smith
6c47cf2e4a
repo-updater: exclude archived setting for GitHub (#9003)
This follows on from our exclude fork work to allow excluding archived. This was
very recently requested by a customer, so would be nice to get out in 3.14.

I didn't add any end to end tests. It feels a bit overkill. Rather I think I
will look into making our exclude code more generic / unit testable in a future
commit.
2020-03-13 14:52:30 +02:00
Tomás Senart
38518b5cb7
repo-updater: Add support for excluding forks from GitHub (#8974)
* repo-updater: Add support for excluding forks from GitHub

This commit is an alternative implementation of #8861 that re-uses our
existing `exclude` setting.

* fixup! CHANGELOG

* fixup format
2020-03-12 21:47:36 +01:00
Beyang Liu
64dc9a75fd
doc: update description for gitlab and github authz ttl (#8037) 2020-01-27 09:58:53 -08:00
Beyang Liu
4b4a0507f6
Link to JSON escaping tool for fields with multiline values, fix #4695 2019-12-04 15:53:21 -08:00
Tomás Senart
0abe3dad51
a8n: Upsert changeset events from GitHub webhooks (#5913)
* a8n: Add support for GitHub webhooks

* fixup! Change doc explaining GitHub webhook events
2019-10-14 14:58:13 +02:00
无闻
92ff1b784b
schema: fix typo GitLab -> GitHub (#4814) 2019-07-08 09:45:46 -07:00
Thorsten Ball
4041c605ad
schema: Validate GH & BBS config have repos OR repositoryQuery (#4685)
* schema: Validate GH & BBS config have repos OR repositoryQuery

This fixes #4322 by changing the validation of external service
configurations for GitHub and Bitbucket Server to validate that either
repositoryQuery OR repos are set.

* Remove GH & BBS SetDefaultRepositoryQueryMigrations
2019-06-28 14:06:34 +02:00
Keegan Carruthers-Smith
2fda851347
repo-updater: Exclude pattern for GitHub (#4647) 2019-06-24 19:24:18 +02:00
Kevin Zheng
3d1ac5ea12
add orgs option to github extsvc config (#4266)
This change adds the `orgs` config option to GitHub external service config. This will allow site admins to explicitly select organizations to sync all repositories from as an alternative to `org:<org name>` in the `repositoryQuery` config option.
2019-06-05 12:28:19 -07:00
Quinn Slack
82776aed75
do not show errors for trailing commas in web JSON editors (#4100)
fix #4008

All JSON documents we use (for settings, site config, external services, and extension manifests) support trailing commas on the frontend and backend. However, an upgrade to monaco-editor unexpectedly caused the Monaco JSON editors in our UI to show red squiggly errors for trailing commas.

The (undocumented) way to do this is to add `allowComments` to the root of the JSON Schema. It is not sufficient to set `allowComments: true` in a `monaco.languages.json.jsonDefaults.setDiagnosticsOptions` call. I found this secret fix by finding https://github.com/microsoft/vscode/issues/19992#issuecomment-347124969 and then inferring what changes that might entail, to discover `allowComments` in the JSON Schema is responsible for ignoring trailing comma errors.
2019-05-16 23:29:12 -07:00
Keegan Carruthers-Smith
a730d9955d
docs: Expand on GitHub exclude and repos fields. (#3435)
The documentation for the GitHub external service is a bit sparse for exclude
since it doesn't explain what you can put in the exclude field. By including
examples (and expanding on the description) in the schema it becomes better
documented both in our docs and in the editor.
2019-04-16 15:44:27 +02:00
Tomás Senart
312ae8f239
schema: Make (repository|project)Query required (#3344)
* schema: Make (repository|project)Query required

We previously assumed that the default field would populate the
respective setting when unmarshaled, but this is not the case, so we
need to make this field required so that the migrations setting these
fields to ["none"] don't run one new configs.

Part of #2025

* fixup! Fix tests

* fixup! STYLE
2019-04-11 16:48:08 +02:00
Beyang Liu
95f9ffbc1f
Add quick configure buttons to more external services (#2913) 2019-03-29 11:28:23 -07:00
Tomás Senart
ed0676ad4a
repo-updater: Implement GitLab exclude feature (#3014)
* repo-updater: Implement GitLab exclude feature

This commit implements the `gitlab.exclude` feature.

Part of #2025

* fixup! Remove name conflict of json schema types

* Update CHANGELOG.md

Co-Authored-By: tsenart <tsenart@gmail.com>

* Update CHANGELOG
2019-03-29 14:46:17 +01:00
Tomás Senart
0f0f24f4b1
repo-updater: Implement GitHub repo enabled state deprecation migration (#2887)
* repo-updater: Factor out Sources type

* repo-updater: Add Filter method to Repos

* repo-updater: Move ExternalServicesFromSources to Sources type

* repo-updater: First draft of GithubReposMigrationToNewSyncer

* schema: Deprecate initialRepositoryEnablement

* repo-updater: Simplify Sourcer

* repo-updater: Define URN method on ExternalService

* repo-updater: Second iteration of migration

* repo-updater: Initial TestGithubReposEnabledStateDeprecationMigration

* jsonc: Add Remove, Edit and Format functions

* repo-updater: Reuse jsonc.{Edit,Remove}

* repo-updater: Implement ExcludeGithubRepos and IncludeGithubRepos

* repo-updater: Test ExternalService.IncludeGithubRepos

* repo-updater: Reuse ExternalService.{Include,Exclude}GithubRepos

* repo-updater: Refactor FakeSource to take in an ExternalService

* repo-updater: Gather FakeSource's external service

* repo-updater: Fix {Include,Exclude}GithubRepos

* repo-updater: Test GithubReposEnabledStateDeprecationMigration

* repo-updater: Github repos are always enabled

* repo-updater: Pass Sourcer to repos.NewSyncer

* repo-updater: Remove unused function

* repo-updater: Upper-case external services

* repo-updater: Update updated_at column in migrations

* repo-updater: Run GithubReposEnabledStateDeprecationMigration on startup

* repo-updater: Generate valid config

* repo-updater: Reuse FakeClock

* fixup! Integration test GithubReposEnabledStateDeprecationMigration

* fixup! Keep using InitialRepositoryEnablement for old syncer

* repo-updater: Skip empty config

* migrations: Check external service's config is not empty

* WIP: Debug logging for CI

* fixup! Skip injected Sources

* fixup! Lower case string inclusion test

* fixup! Remove wrong import

* fixup! Test missing sources

* repo-updater: Test case for noop on enabled and sourced repos

* fixup! Generalize isSerializationError with isRetryable

* fixup! Return -1 on error

* fixup! Skip running migration for now

* fixup! Improve algorithmic complexity
2019-03-26 19:33:59 +01:00
Quinn Slack
d2b0714b29
clarify GitHub ext svc docs, standardize on terms (#2952)
- fix #2946 by implying that no "repo" scope is needed in the example JSON and explicitly mentioning it in the JSON Schema docs (the explicit mention is omitted from the example JSON for brevity, so that the "token" property is above the fold on an average-sized screen)
- use "repository" not "repo" per https://docs.sourcegraph.com/dev/style_guide
2019-03-23 23:28:57 +01:00
Tomás Senart
da3743ece3
Revert "Revert "external services: Change default github.repositoryQuery to ["none"] (#2792)" (#2890)" (#2891)
This reverts the reverted commit 2da1d2f449.
2019-03-21 16:57:00 +01:00
Tomás Senart
2da1d2f449
Revert "external services: Change default github.repositoryQuery to ["none"] (#2792)" (#2890)
This reverts commit 1bc0f70827.
2019-03-21 16:15:08 +01:00
Tomás Senart
1bc0f70827
external services: Change default github.repositoryQuery to ["none"] (#2792)
* repo-updater: Factor out common assertions

* repo-updater: Implement GithubSetDefaultRepositoryQueryMigration

* external services: Change default github.repositoryQuery to none

* Update CHANGELOG

* fixup! Fix JSON schema validation

* repo-updater: Fix rebase auto-merge fuckup

* fixup! Add another test case

* fixup! Set updated_at column in migration
2019-03-21 15:26:33 +01:00
Tomás Senart
64f342f8f8
repo-updater: Implement GitHub external service exclude list (#2695)
* schema: Add exclude list to GitHubConnection

* repo-updater: Initial GithubSource_ListRepos test

* repo-updater: Rename test-fixtures to testdata

* repo-updater: Write failing test for github exclude list

* repo-updater: Implement exclude feature

* Update CHANGELOG

* fixup! Pretty JSON

* fixup! Pretty JSON in Go

* fixup! Better CHANGELOG

* fixup! Support github.exclude in old syncer

* fixup! Use file variable

* fixup! Update changelog

* fixup! Address PR comments

* fixup! Handle Github repo name case insensitivity
2019-03-13 14:00:20 +01:00