artin/sourcegraph
1
0
Fork 0
mirror of https://github.com/sourcegraph/sourcegraph.git synced 2026-02-06 13:51:46 +00:00
Code Issues Packages Projects Releases Wiki Activity
5de9993d88
sourcegraph/sg.config.yaml

1476 lines
43 KiB
YAML
Raw Normal View History

Add link to docs for how to customize sg for local development (#37608)
2022-06-23 12:35:39 +00:00
# Documentation for how to override sg configuration for local development:
# https://github.com/sourcegraph/sourcegraph/blob/main/doc/dev/background-information/sg/index.md#configuration
Move sg.config.yaml to repo root & allow private overwrites (#19487) This moves the sg.config.yaml to the repository root so that one can just use `sg start` and `sg run gitserver` etc. without having to specify the `-conf` flag. It also adds support for a local `.gitignore`d `sg.config.overwrites.yaml` file that users can add to the repository. If it exists, it will then get merged with the basic configuration when `sg` starts. What's the usecase? Custom command sets, custom env vars, etc. that you don't want to commit to the repository.
2021-03-29 08:14:53 +00:00
env:
sg: add ability to define `checks` per `commandset` (#23020) This allows us to check whether services are running before we boot up the environment. This should reduce the number of "why is service X failing?". At least that's the hope. The changes to the config file format are backwards compatible. See the `UnmarshalYAML` method on `Commandset`. It also updates the checks to make sure that they work for users of the docker-compose setup.
2021-07-21 15:37:24 +00:00
PGPORT: 5432
PGHOST: localhost
PGUSER: sourcegraph
PGPASSWORD: sourcegraph
PGDATABASE: sourcegraph
PGSSLMODE: disable
migration: Rough sketch of migration runner validation (#28824)
2021-12-10 13:59:33 +00:00
SG_DEV_MIGRATE_ON_APPLICATION_STARTUP: 'true'
dev: listen on 127.0.0.1 in dev (INSECURE_DEV=true) (#34714) This PR sets INSECURE_DEV (maybe rename in a followup?) in sg.config.yaml by default, which has most services listening on 127.0.0.1 instead of all interfaces. Any services not yet configured on INSECURE_DEV has been updated to listen on 127.0.0.1 if the envvar is set. Also removes all the MacOS firewall stuff in sg See https://github.com/sourcegraph/sourcegraph/issues/29908 for more details
2022-04-29 19:13:57 +00:00
INSECURE_DEV: true
Update web-standalone success banner to be informative and green (#29443) * Update web-standalone success banner to be informative and green
2022-01-06 11:49:45 +00:00
Move sg.config.yaml to repo root & allow private overwrites (#19487) This moves the sg.config.yaml to the repository root so that one can just use `sg start` and `sg run gitserver` etc. without having to specify the `-conf` flag. It also adds support for a local `.gitignore`d `sg.config.overwrites.yaml` file that users can add to the repository. If it exists, it will then get merged with the basic configuration when `sg` starts. What's the usecase? Custom command sets, custom env vars, etc. that you don't want to commit to the repository.
2021-03-29 08:14:53 +00:00
SRC_REPOS_DIR: $HOME/.sourcegraph/repos
SRC_LOG_LEVEL: info
SRC_LOG_FORMAT: condensed
observation: disable trace logs in dev (#34703) Explicit logging of trace logs was added because Datadog does not support OT/OTel trace logs/events at INFO level because we do not want these logs to be lost if we have no other way of seeing trace log events. This PR disables it for dev environments, and optionally for other environments, because it can get a bit noisy.
2022-04-29 15:40:09 +00:00
SRC_TRACE_LOG: false
logging: Print iTerm link to the log's source file:line in VS Code (#30439)
2022-02-01 20:55:40 +00:00
# Set this to true to show an iTerm link to the file:line where the log message came from
SRC_LOG_SOURCE_LINK: false
sg config: use two gitserver instances by default (#42749) We talked about this in repo-management & IAM and we think we should increase our dogfood consumption. This is one step in that direction: most of our customers run mulitple gitserver instances. This helps us make sure we dogfood that. ATTENTION: this will lead to some repositories being recloned/reindexed locally.
2022-10-10 13:26:58 +00:00
# Use two gitserver instances in local dev
SRC_GIT_SERVER_1: 127.0.0.1:3501
SRC_GIT_SERVER_2: 127.0.0.1:3502
SRC_GIT_SERVERS: 127.0.0.1:3501 127.0.0.1:3502
Move sg.config.yaml to repo root & allow private overwrites (#19487) This moves the sg.config.yaml to the repository root so that one can just use `sg start` and `sg run gitserver` etc. without having to specify the `-conf` flag. It also adds support for a local `.gitignore`d `sg.config.overwrites.yaml` file that users can add to the repository. If it exists, it will then get merged with the basic configuration when `sg` starts. What's the usecase? Custom command sets, custom env vars, etc. that you don't want to commit to the repository.
2021-03-29 08:14:53 +00:00
# Enable sharded indexed search mode:
INDEXED_SEARCH_SERVERS: localhost:3070 localhost:3071
sg: Always set GO111MODULE=on (#23658) This is set in dev/start, so to be consistent across dev/start and sg, we should set it globally here, too.
2021-08-06 08:51:09 +00:00
GO111MODULE: 'on'
Move sg.config.yaml to repo root & allow private overwrites (#19487) This moves the sg.config.yaml to the repository root so that one can just use `sg start` and `sg run gitserver` etc. without having to specify the `-conf` flag. It also adds support for a local `.gitignore`d `sg.config.overwrites.yaml` file that users can add to the repository. If it exists, it will then get merged with the basic configuration when `sg` starts. What's the usecase? Custom command sets, custom env vars, etc. that you don't want to commit to the repository.
2021-03-29 08:14:53 +00:00
DEPLOY_TYPE: dev
sg: Fix configuration for code intel stack (#21532) Updates `sg.config.yaml` to fix booting up the code intel stack.
2021-05-31 07:51:39 +00:00
SRC_HTTP_ADDR: ':3082'
Move sg.config.yaml to repo root & allow private overwrites (#19487) This moves the sg.config.yaml to the repository root so that one can just use `sg start` and `sg run gitserver` etc. without having to specify the `-conf` flag. It also adds support for a local `.gitignore`d `sg.config.overwrites.yaml` file that users can add to the repository. If it exists, it will then get merged with the basic configuration when `sg` starts. What's the usecase? Custom command sets, custom env vars, etc. that you don't want to commit to the repository.
2021-03-29 08:14:53 +00:00
GITHUB_BASE_URL: http://127.0.0.1:3180
# I don't think we even need to set these?
SEARCHER_URL: http://127.0.0.1:3181
REPO_UPDATER_URL: http://127.0.0.1:3182
REDIS_ENDPOINT: 127.0.0.1:6379
SYMBOLS_URL: http://localhost:3184
embeddings: searcher and indexer (#48017) # High-level architecture overview <img width="2231" alt="Screenshot 2023-02-24 at 15 13 59" src="https://user-images.githubusercontent.com/6417322/221200130-53c1ff25-4c47-4532-885f-5c4f9dadb05e.png"> # Embeddings Really quickly: embeddings are a semantic representation of text. Embeddings are usually floating-point vectors with 256+ elements. The neat thing about embeddings is that they allow us to search over textual information using a semantic correlation between the query and the text, not just syntactic (matching keywords). In this PR, we implemented an embedding service that will allow us to do semantic code search over repositories in Sourcegraph. So, for example, you'll be able to ask, "how do access tokens work in Sourcegraph", and it will give you a list of the closest matching code files. Additionally, we build a context detection service powered by embeddings. In chat applications, it is important to know whether the user's message requires additional context. We have to differentiate between two cases: the user asks a general question about the codebase, or the user references something in the existing conversation. In the latter case, including the context would ruin the flow of the conversation, and the chatbot would most likely return a confusing answer. We determine whether a query _does not_ require additional context using two approaches: 1. We check if the query contains well-known phrases that would indicate the user is referencing the existing conversation (e.g., translate previous, change that) 1. We have a static dataset of messages that require context and a dataset of messages that do not. We embed both datasets, and then, using embedding similarity, we can check which set is more similar to the query. ## GraphQL API We add four new resolvers to the GraphQL API: ```graphql extend type Query { embeddingsSearch(repo: ID!, query: String!, codeResultsCount: Int!, textResultsCount: Int!): EmbeddingsSearchResults! isContextRequiredForQuery(query: String!): Boolean! } extend type Mutation { scheduleRepositoriesForEmbedding(repoNames: [String!]!): EmptyResponse! scheduleContextDetectionForEmbedding: EmptyResponse! } ``` - `embeddingsSearch` performs embeddings search over the repo embeddings and returns the specified number of results - `isContextRequiredForQuery` determines whether the given query requires additional context - `scheduleRepositoriesForEmbedding` schedules a repo embedding background job - `scheduleContextDetectionForEmbedding` schedules a context detection embedding background job that embeds a static dataset of messages. ## Repo embedding background job Embedding a repository is implemented as a background job. The background job handler receives the repository and the revision, which should be embedded. Handler then gathers a list of files from the gitserver and excludes files >1MB in size. The list of files is split into code and text files (.md, .txt), and we build a separate embedding index for both. We split them because in a combined index, the text files always tended to feature as top results and didn't leave any room for code files. Once we have the list of files, the procedure is as follows: - For each file - Get file contents from gitserver - Check if the file is embeddable (is not autogenerated, is large enough, does not have long lines) - Split the file into embeddable chunks - Embed the file chunks using an external embedding service (defined in site config) - Add embedded file chunks and metadata to the index - Metadata contains the file name, the start line, and the end line of the chunk - Once all files are processed, the index is marshaled into JSON and stored in Cloud storage (GCS, S3) ### Site config changes As mentioned, we use a configurable external embedding API that does the actual text -> vector embedding part. Ideally, this allows us to swap embedding providers in the future. ```json "embeddings": { "description": "Configuration for embeddings service.", "type": "object", "required": ["enabled", "dimensions", "model", "accessToken", "url"], "properties": { "enabled": { "description": "Toggles whether embedding service is enabled.", "type": "boolean", "default": false }, "dimensions": { "description": "The dimensionality of the embedding vectors.", "type": "integer", "minimum": 0 }, "model": { "description": "The model used for embedding.", "type": "string" }, "accessToken": { "description": "The access token used to authenticate with the external embedding API service.", "type": "string" }, "url": { "description": "The url to the external embedding API service.", "type": "string", "format": "uri" } } } ``` ## Repo embeddings search The repo embeddings search is implemented in its own service. When a user queries a repo using embeddings search, the following happens: - Download the repo embedding index from blob storage and cache it in memory - We cache up to 5 embedding indexes in memory - Embed the query and use the embedded query vector to find similar code and text file metadata in the embedding index - Query gitserver for the actual file contents - Return the results ## Interesting files - [Similarity search](https://github.com/sourcegraph/sourcegraph/pull/48017/files#diff-102cc83520004eb0e2795e49bc435c5142ca555189b1db3a52bbf1ffb82fa3c6) - [Repo embedding job handler](https://github.com/sourcegraph/sourcegraph/pull/48017/files#diff-c345f373f426398beb4b9cd5852ba862a2718687882db2a8b2d9c7fbb5f1dc52) - [External embedding api client](https://github.com/sourcegraph/sourcegraph/pull/48017/files#diff-ad1e7956f518e4bcaee17dd9e7ac04a5e090c00d970fcd273919e887e1d2cf8f) - [Embedding a repo](https://github.com/sourcegraph/sourcegraph/pull/48017/files#diff-1f35118727128095b7816791b6f0a2e0e060cddee43d25102859b8159465585c) - [Embeddings searcher service](https://github.com/sourcegraph/sourcegraph/pull/48017/files#diff-5b20f3e7ef87041daeeaef98b58ebf7388519cedcdfc359dc5e6d4e0b021472e) - [Embeddings search](https://github.com/sourcegraph/sourcegraph/pull/48017/files#diff-79f95b9cc3f1ef39c1a0b88015bd9cd6c19c30a8d4c147409f1b8e8cd9462ea1) - [Repo embedding index cache management](https://github.com/sourcegraph/sourcegraph/pull/48017/files#diff-8a41f7dec31054889dbf86e97c52223d5636b4d408c6b375bcfc09160a8b70f8) - [GraphQL resolvers](https://github.com/sourcegraph/sourcegraph/pull/48017/files#diff-9b30a0b5efcb63e2f4611b99ab137fbe09629a769a4f30d10a1b2da41a01d21f) ## Test plan - Start by filling out the `embeddings` object in the site config (let me know if you need an API key) - Start the embeddings service using `sg start embeddings` - Go to the `/api/console` page and schedule a repo embedding job and a context detection embedding job: ```graphql mutation { scheduleRepositoriesForEmbedding(repoNames: ["github.com/sourcegraph/handbook"]) { __typename } scheduleContextDetectionForEmbedding { __typename } } ``` - Once both are finished, you should be able to query the repo embedding index, and determine whether context is need for a given query: ```graphql query { isContextRequiredForQuery(query: "how do access tokens work") embeddingsSearch( repo: "UmVwb3NpdG9yeToy", # github.com/sourcegraph/handbook GQL ID query: "how do access tokens work", codeResultsCount: 5, textResultsCount: 5) { codeResults { fileName content } textResults { fileName content } } } ```
2023-03-01 09:50:12 +00:00
EMBEDDINGS_URL: http://localhost:9991
Move sg.config.yaml to repo root & allow private overwrites (#19487) This moves the sg.config.yaml to the repository root so that one can just use `sg start` and `sg run gitserver` etc. without having to specify the `-conf` flag. It also adds support for a local `.gitignore`d `sg.config.overwrites.yaml` file that users can add to the repository. If it exists, it will then get merged with the basic configuration when `sg` starts. What's the usecase? Custom command sets, custom env vars, etc. that you don't want to commit to the repository.
2021-03-29 08:14:53 +00:00
SRC_SYNTECT_SERVER: http://localhost:9238
SRC_FRONTEND_INTERNAL: localhost:3090
GRAFANA_SERVER_URL: http://localhost:3370
PROMETHEUS_URL: http://localhost:9090
JAEGER_SERVER_URL: http://localhost:16686
lib/log: initial implementation (#33956) This introduces a new logging library we plan to enforce as the standard for logging at Sourcegraph - see https://github.com/sourcegraph/sourcegraph/issues/33192 for more details. Co-authored-by: Eric Fritz <eric@eric-fritz.com>
2022-04-21 19:58:50 +00:00
SRC_DEVELOPMENT: 'true'
monitoring: Database connections monitoring (#22570) * Revert "Revert "dbconn: Better SQL pool stats (#22523)"" This reverts commit 18c8696687a4e6e5cbed6d4115baae4fdbde31d3. * Remove dependency on sqlstats * codeintel: Avoid duplicate app name * dbconn: Parameterise dbname for Prometheus label * monitoring: Database connections monitoring * Add CHANGELOG entry * fixup! Add PR link to CHANGELOG * fixup! Fix insights test * fixup! Introduce dbconn.Opts for readability
2021-07-05 10:24:20 +00:00
SRC_PROF_HTTP: ''
SRC_PROF_SERVICES: |
[
sg: shorten prefix for commands name (#37437)
2022-06-20 17:57:54 +00:00
{ "Name": "oss-frontend", "Host": "127.0.0.1:6063" },
monitoring: Database connections monitoring (#22570) * Revert "Revert "dbconn: Better SQL pool stats (#22523)"" This reverts commit 18c8696687a4e6e5cbed6d4115baae4fdbde31d3. * Remove dependency on sqlstats * codeintel: Avoid duplicate app name * dbconn: Parameterise dbname for Prometheus label * monitoring: Database connections monitoring * Add CHANGELOG entry * fixup! Add PR link to CHANGELOG * fixup! Fix insights test * fixup! Introduce dbconn.Opts for readability
2021-07-05 10:24:20 +00:00
{ "Name": "frontend", "Host": "127.0.0.1:6063" },
sg config: use two gitserver instances by default (#42749) We talked about this in repo-management & IAM and we think we should increase our dogfood consumption. This is one step in that direction: most of our customers run mulitple gitserver instances. This helps us make sure we dogfood that. ATTENTION: this will lead to some repositories being recloned/reindexed locally.
2022-10-10 13:26:58 +00:00
{ "Name": "gitserver-0", "Host": "127.0.0.1:3551" },
{ "Name": "gitserver-1", "Host": "127.0.0.1:3552" },
authz: Move sub-repo perms to enterprise (#45659)
2022-12-28 08:44:07 +00:00
{ "Name": "oss-gitserver-0", "Host": "127.0.0.1:3551" },
{ "Name": "oss-gitserver-1", "Host": "127.0.0.1:3552" },
monitoring: Database connections monitoring (#22570) * Revert "Revert "dbconn: Better SQL pool stats (#22523)"" This reverts commit 18c8696687a4e6e5cbed6d4115baae4fdbde31d3. * Remove dependency on sqlstats * codeintel: Avoid duplicate app name * dbconn: Parameterise dbname for Prometheus label * monitoring: Database connections monitoring * Add CHANGELOG entry * fixup! Add PR link to CHANGELOG * fixup! Fix insights test * fixup! Introduce dbconn.Opts for readability
2021-07-05 10:24:20 +00:00
{ "Name": "searcher", "Host": "127.0.0.1:6069" },
Fix observability scraping in local dev (#39502) The binary names didn't match with whats in the PROF port list, so it didn't work. I've fixed it and aligned them more closely in filenames, too.
2022-07-27 15:21:29 +00:00
{ "Name": "oss-symbols", "Host": "127.0.0.1:6071" },
monitoring: Database connections monitoring (#22570) * Revert "Revert "dbconn: Better SQL pool stats (#22523)"" This reverts commit 18c8696687a4e6e5cbed6d4115baae4fdbde31d3. * Remove dependency on sqlstats * codeintel: Avoid duplicate app name * dbconn: Parameterise dbname for Prometheus label * monitoring: Database connections monitoring * Add CHANGELOG entry * fixup! Add PR link to CHANGELOG * fixup! Fix insights test * fixup! Introduce dbconn.Opts for readability
2021-07-05 10:24:20 +00:00
{ "Name": "symbols", "Host": "127.0.0.1:6071" },
sg: shorten prefix for commands name (#37437)
2022-06-20 17:57:54 +00:00
{ "Name": "oss-repo-updater", "Host": "127.0.0.1:6074" },
monitoring: Database connections monitoring (#22570) * Revert "Revert "dbconn: Better SQL pool stats (#22523)"" This reverts commit 18c8696687a4e6e5cbed6d4115baae4fdbde31d3. * Remove dependency on sqlstats * codeintel: Avoid duplicate app name * dbconn: Parameterise dbname for Prometheus label * monitoring: Database connections monitoring * Add CHANGELOG entry * fixup! Add PR link to CHANGELOG * fixup! Fix insights test * fixup! Introduce dbconn.Opts for readability
2021-07-05 10:24:20 +00:00
{ "Name": "repo-updater", "Host": "127.0.0.1:6074" },
sg: shorten prefix for commands name (#37437)
2022-06-20 17:57:54 +00:00
{ "Name": "codeintel-worker", "Host": "127.0.0.1:6088" },
{ "Name": "oss-worker", "Host": "127.0.0.1:6089" },
sg: Fix prometheus scraping for local dev (#43703) This was missing github-proxy and the new dual deployment of gitserver.
2022-10-31 17:22:27 +00:00
{ "Name": "github-proxy", "Host": "127.0.0.1:6090" },
monitoring: Database connections monitoring (#22570) * Revert "Revert "dbconn: Better SQL pool stats (#22523)"" This reverts commit 18c8696687a4e6e5cbed6d4115baae4fdbde31d3. * Remove dependency on sqlstats * codeintel: Avoid duplicate app name * dbconn: Parameterise dbname for Prometheus label * monitoring: Database connections monitoring * Add CHANGELOG entry * fixup! Add PR link to CHANGELOG * fixup! Fix insights test * fixup! Introduce dbconn.Opts for readability
2021-07-05 10:24:20 +00:00
{ "Name": "worker", "Host": "127.0.0.1:6089" },
Push executor metrics (#36969)
2022-08-03 10:08:04 +00:00
{ "Name": "worker-executors", "Host": "127.0.0.1:6996" },
sg: shorten prefix for commands name (#37437)
2022-06-20 17:57:54 +00:00
{ "Name": "zoekt-index-0", "Host": "127.0.0.1:6072" },
{ "Name": "zoekt-index-1", "Host": "127.0.0.1:6073" },
{ "Name": "zoekt-web-0", "Host": "127.0.0.1:3070", "DefaultPath": "/debug/requests/" },
{ "Name": "zoekt-web-1", "Host": "127.0.0.1:3071", "DefaultPath": "/debug/requests/" }
monitoring: Database connections monitoring (#22570) * Revert "Revert "dbconn: Better SQL pool stats (#22523)"" This reverts commit 18c8696687a4e6e5cbed6d4115baae4fdbde31d3. * Remove dependency on sqlstats * codeintel: Avoid duplicate app name * dbconn: Parameterise dbname for Prometheus label * monitoring: Database connections monitoring * Add CHANGELOG entry * fixup! Add PR link to CHANGELOG * fixup! Fix insights test * fixup! Introduce dbconn.Opts for readability
2021-07-05 10:24:20 +00:00
]
Move sg.config.yaml to repo root & allow private overwrites (#19487) This moves the sg.config.yaml to the repository root so that one can just use `sg start` and `sg run gitserver` etc. without having to specify the `-conf` flag. It also adds support for a local `.gitignore`d `sg.config.overwrites.yaml` file that users can add to the repository. If it exists, it will then get merged with the basic configuration when `sg` starts. What's the usecase? Custom command sets, custom env vars, etc. that you don't want to commit to the repository.
2021-03-29 08:14:53 +00:00
# Settings/config
SITE_CONFIG_FILE: ./dev/site-config.json
SITE_CONFIG_ALLOW_EDITS: true
GLOBAL_SETTINGS_FILE: ./dev/global-settings.json
GLOBAL_SETTINGS_ALLOW_EDITS: true
Fix `sg run-set enterprise` not starting on Linux (#19999) Before this change `sg run-set enterprise` would fail under Linux: ``` [enterprise-frontend] Failed to connect to codeintel database: DB not available: failed to connect to `host=127.0.0.1 user=mrnugget database=sourcegraph`: server error (FATAL: password authentication failed for user "mrnugget" (SQLSTATE 28P01)) -------------------------------------------------------------------------------- Failed to run enterprise-frontend: exited with 1 -------------------------------------------------------------------------------- error: failed to run enterprise-frontend ``` I suspect the reason it worked on macOS is that Postgres.app sets other defaults (database name, user name, etc.) In any case: it's good to make this explicit.
2021-04-14 10:53:56 +00:00
# Point codeintel to the `frontend` database in development
CODEINTEL_PGPORT: $PGPORT
CODEINTEL_PGHOST: $PGHOST
CODEINTEL_PGUSER: $PGUSER
CODEINTEL_PGPASSWORD: $PGPASSWORD
CODEINTEL_PGDATABASE: $PGDATABASE
CODEINTEL_PGSSLMODE: $PGSSLMODE
CODEINTEL_PGDATASOURCE: $PGDATASOURCE
CODEINTEL_PG_ALLOW_SINGLE_DB: true
web: add web-app server for development and production builds (#20126) This PR introduces an independent server for the web application, which can use any deployed API instance as a backend.
2021-05-07 11:01:10 +00:00
# Required for `frontend` and `web` commands
SOURCEGRAPH_HTTPS_DOMAIN: sourcegraph.test
SOURCEGRAPH_HTTPS_PORT: 3443
# Required for `web` commands
Node: double max_old_space_size (#26365)
2021-10-20 14:07:10 +00:00
NODE_OPTIONS: '--max_old_space_size=8192'
web: add web-app server for development and production builds (#20126) This PR introduces an independent server for the web application, which can use any deployed API instance as a backend.
2021-05-07 11:01:10 +00:00
# Default `NODE_ENV` to `development`
NODE_ENV: development
sg: Fix configuration for code intel stack (#21532) Updates `sg.config.yaml` to fix booting up the code intel stack.
2021-05-31 07:51:39 +00:00
# Required for codeintel uploadstore
PRECISE_CODE_INTEL_UPLOAD_AWS_ENDPOINT: http://localhost:9000
all: introduce sourcegraph/blobstore; add option to cmd/server (#44255) * all: add new sg blobstore backend to supersede minio * cmd/server: optionally run sg blobstore backend over minio * lsifuploadstore: add minio back to help text * sg generate * uploadstore: add DeleteObjects / NewListObjectsV2Paginator to s3API interface * uploadstore: regenerate mocks * uploadstore: add gcsAPI.Objects listing API * uploadstore: regenerate mocks * uploadstore: add Store.ExpireObjects stub implementation (thanks eric) * uploadstore: implement ExpireObjects in GCS/S3/Lazy interfaces * worker: integrate lsifuploadstore expirer * uploadstore: update test API calls * uploadstore: add TODO for myself * uploadstore: batch S3 object deletions * uploadstore: use operation logger instead of log15 * cmd/server: use sglog * cmd/server: fix image build Signed-off-by: Stephen Gutekanst <stephen@sourcegraph.com>
2022-11-30 23:04:48 +00:00
PRECISE_CODE_INTEL_UPLOAD_BACKEND: blobstore
sg: Fix configuration for code intel stack (#21532) Updates `sg.config.yaml` to fix booting up the code intel stack.
2021-05-31 07:51:39 +00:00
Cody: locally configure embedding upload URL (#50143) This config is required when running `sg start embeddings`, since the embeddings job uploads the embedding vectors to blobstore.
2023-04-03 15:50:06 +00:00
# Required for embeddings job upload
EMBEDDINGS_UPLOAD_AWS_ENDPOINT: http://localhost:9000
sg: Fix configuration for code intel stack (#21532) Updates `sg.config.yaml` to fix booting up the code intel stack.
2021-05-31 07:51:39 +00:00
# Disable auto-indexing the CNCF repo group (this only works in Cloud)
# This setting will be going away soon
DISABLE_CNCF: notonmybox
sg: use same PG envvar copy for code insights (#45204) Directly constructing a DSN is incorrect, since the user may have just specified a DSN. Instead we need to copy over from the same format that the user specified for connecting to frontend. This is the same approach use by codeintel. Test Plan: sg start enterprise-codeinsights works on my machine (did not before).
2022-12-06 03:41:14 +00:00
# Point code insights to the `frontend` database in development
CODEINSIGHTS_PGPORT: $PGPORT
CODEINSIGHTS_PGHOST: $PGHOST
CODEINSIGHTS_PGUSER: $PGUSER
CODEINSIGHTS_PGPASSWORD: $PGPASSWORD
CODEINSIGHTS_PGDATABASE: $PGDATABASE
CODEINSIGHTS_PGSSLMODE: $PGSSLMODE
CODEINSIGHTS_PGDATASOURCE: $PGDATASOURCE
# Disable code insights by default
sg: hacking hour with lots of YAML (#21250) This PR is a collection of all the things we did in the `sg` hacking hour just now: - Clean up the README so that the TODOs are up to date - Add definitions for `minio`, `codeinsights-db`, `executor-queue`, `precise-code-intel-worker` and `codeintel-executor` - Make it possible to launch Sourcegraph with codeinsights!! See the `sg.config.yaml` for which env vars to set and then run `sg run-set enterprise-codeinsights`
2021-05-21 16:01:11 +00:00
DB_STARTUP_TIMEOUT: 120s # codeinsights-db needs more time to start in some instances.
DISABLE_CODE_INSIGHTS_HISTORICAL: true
DISABLE_CODE_INSIGHTS: true
debug: OpenTelemetry http/json tunnel on /-/debug/otlp (#39453) Adds an OpenTelemetry protocol (OTLP) adapter tunnel through the frontend (backend), /-/debug/otlp, that accepts http/json OTLP requests and sends them to either a grpc or http/json OpenTelemetry collector (depending on the configured protocol). We build an adapter because the OpenTelemetry collector is gRPC by default in most of its guidance, while browser environments can only support the http/json protocol. This allows admins to use their preferred OpenTelemetry backend of choice without worrying about compatibility with the http/json protocol. This is done by (ab)using OpenTelemetry collector (not the Go API) packages to partially implement an OpenTelemetry receiver on /-/debug/otlp - see the otlpadapter package for more details.
2022-08-02 14:09:41 +00:00
# # OpenTelemetry in dev - use single http/json endpoint
# OTEL_EXPORTER_OTLP_ENDPOINT: http://127.0.0.1:4318
# OTEL_EXPORTER_OTLP_PROTOCOL: http/json
Move sg.config.yaml to repo root & allow private overwrites (#19487) This moves the sg.config.yaml to the repository root so that one can just use `sg start` and `sg run gitserver` etc. without having to specify the `-conf` flag. It also adds support for a local `.gitignore`d `sg.config.overwrites.yaml` file that users can add to the repository. If it exists, it will then get merged with the basic configuration when `sg` starts. What's the usecase? Custom command sets, custom env vars, etc. that you don't want to commit to the repository.
2021-03-29 08:14:53 +00:00
commands:
dev/sg: add 'sg run server' alias (#38256) Convenient for running end-to-end tests the same way we run them in CI.
2022-07-06 17:09:41 +00:00
server:
sg: Add description field to sg commands (#39240)
2022-07-26 14:51:04 +00:00
description: Run an all-in-one sourcegraph/server image
dev/sg: add 'sg run server' alias (#38256) Convenient for running end-to-end tests the same way we run them in CI.
2022-07-06 17:09:41 +00:00
cmd: ./dev/run-server-image.sh
env:
TAG: insiders
CLEAN: 'true'
DATA: '/tmp/sourcegraph-data'
URL: 'http://localhost:7080'
sg: shorten prefix for commands name (#37437)
2022-06-20 17:57:54 +00:00
oss-frontend:
cmd: .bin/oss-frontend
sg: respect $DELVE (#25812)
2021-10-09 01:47:08 +00:00
install: |
if [ -n "$DELVE" ]; then
export GCFLAGS='all=-N -l'
fi
sg: shorten prefix for commands name (#37437)
2022-06-20 17:57:54 +00:00
go build -gcflags="$GCFLAGS" -o .bin/oss-frontend github.com/sourcegraph/sourcegraph/cmd/frontend
sg: fix sg start oss (#38409)
2022-07-07 13:08:06 +00:00
checkBinary: .bin/oss-frontend
Move sg.config.yaml to repo root & allow private overwrites (#19487) This moves the sg.config.yaml to the repository root so that one can just use `sg start` and `sg run gitserver` etc. without having to specify the `-conf` flag. It also adds support for a local `.gitignore`d `sg.config.overwrites.yaml` file that users can add to the repository. If it exists, it will then get merged with the basic configuration when `sg` starts. What's the usecase? Custom command sets, custom env vars, etc. that you don't want to commit to the repository.
2021-03-29 08:14:53 +00:00
env:
CONFIGURATION_MODE: server
USE_ENHANCED_LANGUAGE_DETECTION: false
Add WEBPACK_DEV_SERVER to frontend procs in sg.config.yaml (#19923) Fix `sg start` throwing a 500 error because the paths to the assets in `app.html` were wrong.
2021-04-12 12:14:41 +00:00
# frontend processes need this to be so that the paths to the assets are rendered correctly
WEBPACK_DEV_SERVER: 1
Move sg.config.yaml to repo root & allow private overwrites (#19487) This moves the sg.config.yaml to the repository root so that one can just use `sg start` and `sg run gitserver` etc. without having to specify the `-conf` flag. It also adds support for a local `.gitignore`d `sg.config.overwrites.yaml` file that users can add to the repository. If it exists, it will then get merged with the basic configuration when `sg` starts. What's the usecase? Custom command sets, custom env vars, etc. that you don't want to commit to the repository.
2021-03-29 08:14:53 +00:00
watch:
sg: Fix configuration for code intel stack (#21532) Updates `sg.config.yaml` to fix booting up the code intel stack.
2021-05-31 07:51:39 +00:00
- lib
Move sg.config.yaml to repo root & allow private overwrites (#19487) This moves the sg.config.yaml to the repository root so that one can just use `sg start` and `sg run gitserver` etc. without having to specify the `-conf` flag. It also adds support for a local `.gitignore`d `sg.config.overwrites.yaml` file that users can add to the repository. If it exists, it will then get merged with the basic configuration when `sg` starts. What's the usecase? Custom command sets, custom env vars, etc. that you don't want to commit to the repository.
2021-03-29 08:14:53 +00:00
- internal
- cmd/frontend
sg: shorten prefix for commands name (#37437)
2022-06-20 17:57:54 +00:00
frontend:
sg: Add description field to sg commands (#39240)
2022-07-26 14:51:04 +00:00
description: Enterprise frontend
sg: allow running enterprise env with `sg`, flesh out TODO/Inspiration/Quickstart/Hacking in README (#19643) This updates the `sg.config.yaml` so that `sg run-set enterprise` boots up the enterprise environment. It also updates the README to contain more TODOs, an Inspiration section, an updated Quickstart/Installation section and a Hacking section.
2021-04-01 11:59:48 +00:00
cmd: |
# TODO: This should be fixed
export SOURCEGRAPH_LICENSE_GENERATION_KEY=$(cat ../dev-private/enterprise/dev/test-license-generation-key.pem)
dev/sg: fix enterprise-e2e set with env hackery (#38327)
2022-07-06 21:38:47 +00:00
# If EXTSVC_CONFIG_FILE is *unset*, set a default.
export EXTSVC_CONFIG_FILE=${EXTSVC_CONFIG_FILE-'../dev-private/enterprise/dev/external-services-config.json'}
Fix observability scraping in local dev (#39502) The binary names didn't match with whats in the PROF port list, so it didn't work. I've fixed it and aligned them more closely in filenames, too.
2022-07-27 15:21:29 +00:00
.bin/frontend
sg: respect $DELVE (#25812)
2021-10-09 01:47:08 +00:00
install: |
if [ -n "$DELVE" ]; then
export GCFLAGS='all=-N -l'
fi
Fix observability scraping in local dev (#39502) The binary names didn't match with whats in the PROF port list, so it didn't work. I've fixed it and aligned them more closely in filenames, too.
2022-07-27 15:21:29 +00:00
go build -gcflags="$GCFLAGS" -o .bin/frontend github.com/sourcegraph/sourcegraph/enterprise/cmd/frontend
checkBinary: .bin/frontend
sg: allow running enterprise env with `sg`, flesh out TODO/Inspiration/Quickstart/Hacking in README (#19643) This updates the `sg.config.yaml` so that `sg run-set enterprise` boots up the enterprise environment. It also updates the README to contain more TODOs, an Inspiration section, an updated Quickstart/Installation section and a Hacking section.
2021-04-01 11:59:48 +00:00
env:
CONFIGURATION_MODE: server
USE_ENHANCED_LANGUAGE_DETECTION: false
ENTERPRISE: 1
SITE_CONFIG_FILE: '../dev-private/enterprise/dev/site-config.json'
sg: set `SITE_CONFIG_ESCAPE_HATCH_PATH` to not be under home root (#25393)
2021-09-27 14:34:19 +00:00
SITE_CONFIG_ESCAPE_HATCH_PATH: '$HOME/.sourcegraph/site-config.json'
Add WEBPACK_DEV_SERVER to frontend procs in sg.config.yaml (#19923) Fix `sg start` throwing a 500 error because the paths to the assets in `app.html` were wrong.
2021-04-12 12:14:41 +00:00
# frontend processes need this to be so that the paths to the assets are rendered correctly
WEBPACK_DEV_SERVER: 1
Move sg.config.yaml to repo root & allow private overwrites (#19487) This moves the sg.config.yaml to the repository root so that one can just use `sg start` and `sg run gitserver` etc. without having to specify the `-conf` flag. It also adds support for a local `.gitignore`d `sg.config.overwrites.yaml` file that users can add to the repository. If it exists, it will then get merged with the basic configuration when `sg` starts. What's the usecase? Custom command sets, custom env vars, etc. that you don't want to commit to the repository.
2021-03-29 08:14:53 +00:00
watch:
sg: Fix configuration for code intel stack (#21532) Updates `sg.config.yaml` to fix booting up the code intel stack.
2021-05-31 07:51:39 +00:00
- lib
Move sg.config.yaml to repo root & allow private overwrites (#19487) This moves the sg.config.yaml to the repository root so that one can just use `sg start` and `sg run gitserver` etc. without having to specify the `-conf` flag. It also adds support for a local `.gitignore`d `sg.config.overwrites.yaml` file that users can add to the repository. If it exists, it will then get merged with the basic configuration when `sg` starts. What's the usecase? Custom command sets, custom env vars, etc. that you don't want to commit to the repository.
2021-03-29 08:14:53 +00:00
- internal
- cmd/frontend
sg: Fix configuration for code intel stack (#21532) Updates `sg.config.yaml` to fix booting up the code intel stack.
2021-05-31 07:51:39 +00:00
- enterprise/internal
Move sg.config.yaml to repo root & allow private overwrites (#19487) This moves the sg.config.yaml to the repository root so that one can just use `sg start` and `sg run gitserver` etc. without having to specify the `-conf` flag. It also adds support for a local `.gitignore`d `sg.config.overwrites.yaml` file that users can add to the repository. If it exists, it will then get merged with the basic configuration when `sg` starts. What's the usecase? Custom command sets, custom env vars, etc. that you don't want to commit to the repository.
2021-03-29 08:14:53 +00:00
- enterprise/cmd/frontend
sg config: use two gitserver instances by default (#42749) We talked about this in repo-management & IAM and we think we should increase our dogfood consumption. This is one step in that direction: most of our customers run mulitple gitserver instances. This helps us make sure we dogfood that. ATTENTION: this will lead to some repositories being recloned/reindexed locally.
2022-10-10 13:26:58 +00:00
gitserver-template: &gitserver_template
Move sg.config.yaml to repo root & allow private overwrites (#19487) This moves the sg.config.yaml to the repository root so that one can just use `sg start` and `sg run gitserver` etc. without having to specify the `-conf` flag. It also adds support for a local `.gitignore`d `sg.config.overwrites.yaml` file that users can add to the repository. If it exists, it will then get merged with the basic configuration when `sg` starts. What's the usecase? Custom command sets, custom env vars, etc. that you don't want to commit to the repository.
2021-03-29 08:14:53 +00:00
cmd: .bin/gitserver
sg: respect $DELVE (#25812)
2021-10-09 01:47:08 +00:00
install: |
if [ -n "$DELVE" ]; then
export GCFLAGS='all=-N -l'
fi
authz: Move sub-repo perms to enterprise (#45659)
2022-12-28 08:44:07 +00:00
go build -gcflags="$GCFLAGS" -o .bin/gitserver github.com/sourcegraph/sourcegraph/enterprise/cmd/gitserver
sg: do not restart binary if it didn't change (#21509) This adds the `checkBinary` property to `sg.config.yaml` which allows `sg` to check whether the specified binary actually changed after rebuilding a command. In practice this means that multiple file modifications that result in the same binary do not lead to multiple restarts.
2021-06-02 09:38:06 +00:00
checkBinary: .bin/gitserver
sg config: use two gitserver instances by default (#42749) We talked about this in repo-management & IAM and we think we should increase our dogfood consumption. This is one step in that direction: most of our customers run mulitple gitserver instances. This helps us make sure we dogfood that. ATTENTION: this will lead to some repositories being recloned/reindexed locally.
2022-10-10 13:26:58 +00:00
env: &gitserverenv
Move sg.config.yaml to repo root & allow private overwrites (#19487) This moves the sg.config.yaml to the repository root so that one can just use `sg start` and `sg run gitserver` etc. without having to specify the `-conf` flag. It also adds support for a local `.gitignore`d `sg.config.overwrites.yaml` file that users can add to the repository. If it exists, it will then get merged with the basic configuration when `sg` starts. What's the usecase? Custom command sets, custom env vars, etc. that you don't want to commit to the repository.
2021-03-29 08:14:53 +00:00
HOSTNAME: 127.0.0.1:3178
watch:
sg: Fix configuration for code intel stack (#21532) Updates `sg.config.yaml` to fix booting up the code intel stack.
2021-05-31 07:51:39 +00:00
- lib
Move sg.config.yaml to repo root & allow private overwrites (#19487) This moves the sg.config.yaml to the repository root so that one can just use `sg start` and `sg run gitserver` etc. without having to specify the `-conf` flag. It also adds support for a local `.gitignore`d `sg.config.overwrites.yaml` file that users can add to the repository. If it exists, it will then get merged with the basic configuration when `sg` starts. What's the usecase? Custom command sets, custom env vars, etc. that you don't want to commit to the repository.
2021-03-29 08:14:53 +00:00
- internal
- cmd/gitserver
authz: Move sub-repo perms to enterprise (#45659)
2022-12-28 08:44:07 +00:00
- enterprise/internal
- enterprise/cmd/gitserver
oss-gitserver-template: &oss_gitserver_template
cmd: .bin/oss-gitserver
install: |
if [ -n "$DELVE" ]; then
export GCFLAGS='all=-N -l'
fi
go build -gcflags="$GCFLAGS" -o .bin/oss-gitserver github.com/sourcegraph/sourcegraph/cmd/gitserver
checkBinary: .bin/oss-gitserver
env: &oss_gitserverenv
HOSTNAME: 127.0.0.1:3178
watch:
- lib
- internal
- cmd/gitserver
oss-gitserver-0:
<<: *oss_gitserver_template
env:
<<: *gitserverenv
singleprogram: use localhost for gitserver (#47366) * singleprogram: use localhost for gitserver Using the hostname isn't gaurenteed to work, it depends on how the machine is configured. For example on my linux desktop this fails. We only start the services on localhost and we use ipv4 localhost address for all other services. Additionally introduce GITSERVER_EXTERNAL_ADDR. Treating the hostname as an address is incorrect. We instead use a more common pattern of having an environment variable which specifies the reachable address for the gitserver service. Test Plan: sg start app and sg start work
2023-02-03 12:32:08 +00:00
GITSERVER_EXTERNAL_ADDR: 127.0.0.1:3501
authz: Move sub-repo perms to enterprise (#45659)
2022-12-28 08:44:07 +00:00
GITSERVER_ADDR: 127.0.0.1:3501
SRC_REPOS_DIR: $HOME/.sourcegraph/repos_1
SRC_PROF_HTTP: 127.0.0.1:3551
oss-gitserver-1:
<<: *oss_gitserver_template
env:
<<: *oss_gitserverenv
singleprogram: use localhost for gitserver (#47366) * singleprogram: use localhost for gitserver Using the hostname isn't gaurenteed to work, it depends on how the machine is configured. For example on my linux desktop this fails. We only start the services on localhost and we use ipv4 localhost address for all other services. Additionally introduce GITSERVER_EXTERNAL_ADDR. Treating the hostname as an address is incorrect. We instead use a more common pattern of having an environment variable which specifies the reachable address for the gitserver service. Test Plan: sg start app and sg start work
2023-02-03 12:32:08 +00:00
GITSERVER_EXTERNAL_ADDR: 127.0.0.1:3502
authz: Move sub-repo perms to enterprise (#45659)
2022-12-28 08:44:07 +00:00
GITSERVER_ADDR: 127.0.0.1:3502
SRC_REPOS_DIR: $HOME/.sourcegraph/repos_2
SRC_PROF_HTTP: 127.0.0.1:3552
Move sg.config.yaml to repo root & allow private overwrites (#19487) This moves the sg.config.yaml to the repository root so that one can just use `sg start` and `sg run gitserver` etc. without having to specify the `-conf` flag. It also adds support for a local `.gitignore`d `sg.config.overwrites.yaml` file that users can add to the repository. If it exists, it will then get merged with the basic configuration when `sg` starts. What's the usecase? Custom command sets, custom env vars, etc. that you don't want to commit to the repository.
2021-03-29 08:14:53 +00:00
sg config: use two gitserver instances by default (#42749) We talked about this in repo-management & IAM and we think we should increase our dogfood consumption. This is one step in that direction: most of our customers run mulitple gitserver instances. This helps us make sure we dogfood that. ATTENTION: this will lead to some repositories being recloned/reindexed locally.
2022-10-10 13:26:58 +00:00
# This is only here to stay backwards-compatible with people's custom
# `sg.config.overwrite.yaml` files
gitserver:
<<: *gitserver_template
gitserver-0:
<<: *gitserver_template
env:
<<: *gitserverenv
singleprogram: use localhost for gitserver (#47366) * singleprogram: use localhost for gitserver Using the hostname isn't gaurenteed to work, it depends on how the machine is configured. For example on my linux desktop this fails. We only start the services on localhost and we use ipv4 localhost address for all other services. Additionally introduce GITSERVER_EXTERNAL_ADDR. Treating the hostname as an address is incorrect. We instead use a more common pattern of having an environment variable which specifies the reachable address for the gitserver service. Test Plan: sg start app and sg start work
2023-02-03 12:32:08 +00:00
GITSERVER_EXTERNAL_ADDR: 127.0.0.1:3501
sg config: use two gitserver instances by default (#42749) We talked about this in repo-management & IAM and we think we should increase our dogfood consumption. This is one step in that direction: most of our customers run mulitple gitserver instances. This helps us make sure we dogfood that. ATTENTION: this will lead to some repositories being recloned/reindexed locally.
2022-10-10 13:26:58 +00:00
GITSERVER_ADDR: 127.0.0.1:3501
SRC_REPOS_DIR: $HOME/.sourcegraph/repos_1
sg: Fix prometheus scraping for local dev (#43703) This was missing github-proxy and the new dual deployment of gitserver.
2022-10-31 17:22:27 +00:00
SRC_PROF_HTTP: 127.0.0.1:3551
sg config: use two gitserver instances by default (#42749) We talked about this in repo-management & IAM and we think we should increase our dogfood consumption. This is one step in that direction: most of our customers run mulitple gitserver instances. This helps us make sure we dogfood that. ATTENTION: this will lead to some repositories being recloned/reindexed locally.
2022-10-10 13:26:58 +00:00
gitserver-1:
<<: *gitserver_template
env:
<<: *gitserverenv
singleprogram: use localhost for gitserver (#47366) * singleprogram: use localhost for gitserver Using the hostname isn't gaurenteed to work, it depends on how the machine is configured. For example on my linux desktop this fails. We only start the services on localhost and we use ipv4 localhost address for all other services. Additionally introduce GITSERVER_EXTERNAL_ADDR. Treating the hostname as an address is incorrect. We instead use a more common pattern of having an environment variable which specifies the reachable address for the gitserver service. Test Plan: sg start app and sg start work
2023-02-03 12:32:08 +00:00
GITSERVER_EXTERNAL_ADDR: 127.0.0.1:3502
sg config: use two gitserver instances by default (#42749) We talked about this in repo-management & IAM and we think we should increase our dogfood consumption. This is one step in that direction: most of our customers run mulitple gitserver instances. This helps us make sure we dogfood that. ATTENTION: this will lead to some repositories being recloned/reindexed locally.
2022-10-10 13:26:58 +00:00
GITSERVER_ADDR: 127.0.0.1:3502
SRC_REPOS_DIR: $HOME/.sourcegraph/repos_2
sg: Fix prometheus scraping for local dev (#43703) This was missing github-proxy and the new dual deployment of gitserver.
2022-10-31 17:22:27 +00:00
SRC_PROF_HTTP: 127.0.0.1:3552
sg config: use two gitserver instances by default (#42749) We talked about this in repo-management & IAM and we think we should increase our dogfood consumption. This is one step in that direction: most of our customers run mulitple gitserver instances. This helps us make sure we dogfood that. ATTENTION: this will lead to some repositories being recloned/reindexed locally.
2022-10-10 13:26:58 +00:00
Move sg.config.yaml to repo root & allow private overwrites (#19487) This moves the sg.config.yaml to the repository root so that one can just use `sg start` and `sg run gitserver` etc. without having to specify the `-conf` flag. It also adds support for a local `.gitignore`d `sg.config.overwrites.yaml` file that users can add to the repository. If it exists, it will then get merged with the basic configuration when `sg` starts. What's the usecase? Custom command sets, custom env vars, etc. that you don't want to commit to the repository.
2021-03-29 08:14:53 +00:00
github-proxy:
cmd: .bin/github-proxy
sg: respect $DELVE (#25812)
2021-10-09 01:47:08 +00:00
install: |
if [ -n "$DELVE" ]; then
export GCFLAGS='all=-N -l'
fi
go build -gcflags="$GCFLAGS" -o .bin/github-proxy github.com/sourcegraph/sourcegraph/cmd/github-proxy
sg: do not restart binary if it didn't change (#21509) This adds the `checkBinary` property to `sg.config.yaml` which allows `sg` to check whether the specified binary actually changed after rebuilding a command. In practice this means that multiple file modifications that result in the same binary do not lead to multiple restarts.
2021-06-02 09:38:06 +00:00
checkBinary: .bin/github-proxy
Move sg.config.yaml to repo root & allow private overwrites (#19487) This moves the sg.config.yaml to the repository root so that one can just use `sg start` and `sg run gitserver` etc. without having to specify the `-conf` flag. It also adds support for a local `.gitignore`d `sg.config.overwrites.yaml` file that users can add to the repository. If it exists, it will then get merged with the basic configuration when `sg` starts. What's the usecase? Custom command sets, custom env vars, etc. that you don't want to commit to the repository.
2021-03-29 08:14:53 +00:00
watch:
sg: Fix configuration for code intel stack (#21532) Updates `sg.config.yaml` to fix booting up the code intel stack.
2021-05-31 07:51:39 +00:00
- lib
- internal
Move sg.config.yaml to repo root & allow private overwrites (#19487) This moves the sg.config.yaml to the repository root so that one can just use `sg start` and `sg run gitserver` etc. without having to specify the `-conf` flag. It also adds support for a local `.gitignore`d `sg.config.overwrites.yaml` file that users can add to the repository. If it exists, it will then get merged with the basic configuration when `sg` starts. What's the usecase? Custom command sets, custom env vars, etc. that you don't want to commit to the repository.
2021-03-29 08:14:53 +00:00
- cmd/github-proxy
sg: shorten prefix for commands name (#37437)
2022-06-20 17:57:54 +00:00
oss-worker:
cmd: .bin/oss-worker
sg: respect $DELVE (#25812)
2021-10-09 01:47:08 +00:00
install: |
if [ -n "$DELVE" ]; then
export GCFLAGS='all=-N -l'
fi
sg: shorten prefix for commands name (#37437)
2022-06-20 17:57:54 +00:00
go build -gcflags="$GCFLAGS" -o .bin/oss-worker github.com/sourcegraph/sourcegraph/cmd/worker
worker: Add skeleton service (#21768)
2021-06-04 19:48:13 +00:00
watch:
- lib
- internal
- cmd/worker
sg: shorten prefix for commands name (#37437)
2022-06-20 17:57:54 +00:00
worker:
insights: Background process to check for a license and freeze/unfreeze insights (#32418)
2022-03-15 19:41:16 +00:00
cmd: |
export SOURCEGRAPH_LICENSE_GENERATION_KEY=$(cat ../dev-private/enterprise/dev/test-license-generation-key.pem)
.bin/worker
sg: respect $DELVE (#25812)
2021-10-09 01:47:08 +00:00
install: |
if [ -n "$DELVE" ]; then
export GCFLAGS='all=-N -l'
fi
go build -gcflags="$GCFLAGS" -o .bin/worker github.com/sourcegraph/sourcegraph/enterprise/cmd/worker
worker: Add skeleton service (#21768)
2021-06-04 19:48:13 +00:00
watch:
- lib
- internal
- enterprise/internal
- cmd/worker
- enterprise/cmd/worker
sg: shorten prefix for commands name (#37437)
2022-06-20 17:57:54 +00:00
oss-repo-updater:
cmd: .bin/oss-repo-updater
sg: respect $DELVE (#25812)
2021-10-09 01:47:08 +00:00
install: |
if [ -n "$DELVE" ]; then
export GCFLAGS='all=-N -l'
fi
sg: shorten prefix for commands name (#37437)
2022-06-20 17:57:54 +00:00
go build -gcflags="$GCFLAGS" -o .bin/oss-repo-updater github.com/sourcegraph/sourcegraph/cmd/repo-updater
checkBinary: .bin/oss-repo-updater
Move sg.config.yaml to repo root & allow private overwrites (#19487) This moves the sg.config.yaml to the repository root so that one can just use `sg start` and `sg run gitserver` etc. without having to specify the `-conf` flag. It also adds support for a local `.gitignore`d `sg.config.overwrites.yaml` file that users can add to the repository. If it exists, it will then get merged with the basic configuration when `sg` starts. What's the usecase? Custom command sets, custom env vars, etc. that you don't want to commit to the repository.
2021-03-29 08:14:53 +00:00
watch:
sg: Fix configuration for code intel stack (#21532) Updates `sg.config.yaml` to fix booting up the code intel stack.
2021-05-31 07:51:39 +00:00
- lib
Move sg.config.yaml to repo root & allow private overwrites (#19487) This moves the sg.config.yaml to the repository root so that one can just use `sg start` and `sg run gitserver` etc. without having to specify the `-conf` flag. It also adds support for a local `.gitignore`d `sg.config.overwrites.yaml` file that users can add to the repository. If it exists, it will then get merged with the basic configuration when `sg` starts. What's the usecase? Custom command sets, custom env vars, etc. that you don't want to commit to the repository.
2021-03-29 08:14:53 +00:00
- internal
- cmd/repo-updater
sg: shorten prefix for commands name (#37437)
2022-06-20 17:57:54 +00:00
repo-updater:
licensing: fix setup for creating license keys locally and for testing features in a local instance (#41150) Fix function to generate license key; add env var to the repo-updater
2022-09-01 19:27:23 +00:00
cmd: |
export SOURCEGRAPH_LICENSE_GENERATION_KEY=$(cat ../dev-private/enterprise/dev/test-license-generation-key.pem)
.bin/repo-updater
sg: respect $DELVE (#25812)
2021-10-09 01:47:08 +00:00
install: |
if [ -n "$DELVE" ]; then
export GCFLAGS='all=-N -l'
fi
Fix observability scraping in local dev (#39502) The binary names didn't match with whats in the PROF port list, so it didn't work. I've fixed it and aligned them more closely in filenames, too.
2022-07-27 15:21:29 +00:00
go build -gcflags="$GCFLAGS" -o .bin/repo-updater github.com/sourcegraph/sourcegraph/enterprise/cmd/repo-updater
checkBinary: .bin/repo-updater
Move sg.config.yaml to repo root & allow private overwrites (#19487) This moves the sg.config.yaml to the repository root so that one can just use `sg start` and `sg run gitserver` etc. without having to specify the `-conf` flag. It also adds support for a local `.gitignore`d `sg.config.overwrites.yaml` file that users can add to the repository. If it exists, it will then get merged with the basic configuration when `sg` starts. What's the usecase? Custom command sets, custom env vars, etc. that you don't want to commit to the repository.
2021-03-29 08:14:53 +00:00
env:
sg: allow running enterprise env with `sg`, flesh out TODO/Inspiration/Quickstart/Hacking in README (#19643) This updates the `sg.config.yaml` so that `sg run-set enterprise` boots up the enterprise environment. It also updates the README to contain more TODOs, an Inspiration section, an updated Quickstart/Installation section and a Hacking section.
2021-04-01 11:59:48 +00:00
ENTERPRISE: 1
Move sg.config.yaml to repo root & allow private overwrites (#19487) This moves the sg.config.yaml to the repository root so that one can just use `sg start` and `sg run gitserver` etc. without having to specify the `-conf` flag. It also adds support for a local `.gitignore`d `sg.config.overwrites.yaml` file that users can add to the repository. If it exists, it will then get merged with the basic configuration when `sg` starts. What's the usecase? Custom command sets, custom env vars, etc. that you don't want to commit to the repository.
2021-03-29 08:14:53 +00:00
watch:
sg: Fix configuration for code intel stack (#21532) Updates `sg.config.yaml` to fix booting up the code intel stack.
2021-05-31 07:51:39 +00:00
- lib
Move sg.config.yaml to repo root & allow private overwrites (#19487) This moves the sg.config.yaml to the repository root so that one can just use `sg start` and `sg run gitserver` etc. without having to specify the `-conf` flag. It also adds support for a local `.gitignore`d `sg.config.overwrites.yaml` file that users can add to the repository. If it exists, it will then get merged with the basic configuration when `sg` starts. What's the usecase? Custom command sets, custom env vars, etc. that you don't want to commit to the repository.
2021-03-29 08:14:53 +00:00
- internal
- cmd/repo-updater
sg: Fix configuration for code intel stack (#21532) Updates `sg.config.yaml` to fix booting up the code intel stack.
2021-05-31 07:51:39 +00:00
- enterprise/internal
Move sg.config.yaml to repo root & allow private overwrites (#19487) This moves the sg.config.yaml to the repository root so that one can just use `sg start` and `sg run gitserver` etc. without having to specify the `-conf` flag. It also adds support for a local `.gitignore`d `sg.config.overwrites.yaml` file that users can add to the repository. If it exists, it will then get merged with the basic configuration when `sg` starts. What's the usecase? Custom command sets, custom env vars, etc. that you don't want to commit to the repository.
2021-03-29 08:14:53 +00:00
- enterprise/cmd/repo-updater
sg: shorten prefix for commands name (#37437)
2022-06-20 17:57:54 +00:00
oss-symbols:
cmd: .bin/oss-symbols
Move sg.config.yaml to repo root & allow private overwrites (#19487) This moves the sg.config.yaml to the repository root so that one can just use `sg start` and `sg run gitserver` etc. without having to specify the `-conf` flag. It also adds support for a local `.gitignore`d `sg.config.overwrites.yaml` file that users can add to the repository. If it exists, it will then get merged with the basic configuration when `sg` starts. What's the usecase? Custom command sets, custom env vars, etc. that you don't want to commit to the repository.
2021-03-29 08:14:53 +00:00
install: |
sg: respect $DELVE (#25812)
2021-10-09 01:47:08 +00:00
if [ -n "$DELVE" ]; then
export GCFLAGS='all=-N -l'
fi
Move sg.config.yaml to repo root & allow private overwrites (#19487) This moves the sg.config.yaml to the repository root so that one can just use `sg start` and `sg run gitserver` etc. without having to specify the `-conf` flag. It also adds support for a local `.gitignore`d `sg.config.overwrites.yaml` file that users can add to the repository. If it exists, it will then get merged with the basic configuration when `sg` starts. What's the usecase? Custom command sets, custom env vars, etc. that you don't want to commit to the repository.
2021-03-29 08:14:53 +00:00
./cmd/symbols/build-ctags.sh &&
sg: shorten prefix for commands name (#37437)
2022-06-20 17:57:54 +00:00
go build -gcflags="$GCFLAGS" -o .bin/oss-symbols github.com/sourcegraph/sourcegraph/cmd/symbols
sg: fix sg start oss (#38409)
2022-07-07 13:08:06 +00:00
checkBinary: .bin/oss-symbols
Move sg.config.yaml to repo root & allow private overwrites (#19487) This moves the sg.config.yaml to the repository root so that one can just use `sg start` and `sg run gitserver` etc. without having to specify the `-conf` flag. It also adds support for a local `.gitignore`d `sg.config.overwrites.yaml` file that users can add to the repository. If it exists, it will then get merged with the basic configuration when `sg` starts. What's the usecase? Custom command sets, custom env vars, etc. that you don't want to commit to the repository.
2021-03-29 08:14:53 +00:00
env:
dev: add installer script for building a local universal-ctag (#45198)
2022-12-06 17:32:21 +00:00
CTAGS_COMMAND: dev/universal-ctags-dev
Move sg.config.yaml to repo root & allow private overwrites (#19487) This moves the sg.config.yaml to the repository root so that one can just use `sg start` and `sg run gitserver` etc. without having to specify the `-conf` flag. It also adds support for a local `.gitignore`d `sg.config.overwrites.yaml` file that users can add to the repository. If it exists, it will then get merged with the basic configuration when `sg` starts. What's the usecase? Custom command sets, custom env vars, etc. that you don't want to commit to the repository.
2021-03-29 08:14:53 +00:00
CTAGS_PROCESSES: 2
watch:
sg: Fix configuration for code intel stack (#21532) Updates `sg.config.yaml` to fix booting up the code intel stack.
2021-05-31 07:51:39 +00:00
- lib
Move sg.config.yaml to repo root & allow private overwrites (#19487) This moves the sg.config.yaml to the repository root so that one can just use `sg start` and `sg run gitserver` etc. without having to specify the `-conf` flag. It also adds support for a local `.gitignore`d `sg.config.overwrites.yaml` file that users can add to the repository. If it exists, it will then get merged with the basic configuration when `sg` starts. What's the usecase? Custom command sets, custom env vars, etc. that you don't want to commit to the repository.
2021-03-29 08:14:53 +00:00
- internal
- cmd/symbols
sg: shorten prefix for commands name (#37437)
2022-06-20 17:57:54 +00:00
symbols:
Fix observability scraping in local dev (#39502) The binary names didn't match with whats in the PROF port list, so it didn't work. I've fixed it and aligned them more closely in filenames, too.
2022-07-27 15:21:29 +00:00
cmd: .bin/symbols
codeintel: Rockskip for symbols (#28719)
2022-03-03 04:13:28 +00:00
install: |
if [ -n "$DELVE" ]; then
export GCFLAGS='all=-N -l'
fi
./cmd/symbols/build-ctags.sh &&
Fix observability scraping in local dev (#39502) The binary names didn't match with whats in the PROF port list, so it didn't work. I've fixed it and aligned them more closely in filenames, too.
2022-07-27 15:21:29 +00:00
go build -gcflags="$GCFLAGS" -o .bin/symbols github.com/sourcegraph/sourcegraph/enterprise/cmd/symbols
checkBinary: .bin/symbols
codeintel: Rockskip for symbols (#28719)
2022-03-03 04:13:28 +00:00
env:
dev: add installer script for building a local universal-ctag (#45198)
2022-12-06 17:32:21 +00:00
CTAGS_COMMAND: dev/universal-ctags-dev
codeintel: Rockskip for symbols (#28719)
2022-03-03 04:13:28 +00:00
CTAGS_PROCESSES: 2
USE_ROCKSKIP: 'false'
watch:
- lib
- internal
- cmd/symbols
- enterprise/cmd/symbols
- enterprise/internal/rockskip
embeddings: searcher and indexer (#48017) # High-level architecture overview <img width="2231" alt="Screenshot 2023-02-24 at 15 13 59" src="https://user-images.githubusercontent.com/6417322/221200130-53c1ff25-4c47-4532-885f-5c4f9dadb05e.png"> # Embeddings Really quickly: embeddings are a semantic representation of text. Embeddings are usually floating-point vectors with 256+ elements. The neat thing about embeddings is that they allow us to search over textual information using a semantic correlation between the query and the text, not just syntactic (matching keywords). In this PR, we implemented an embedding service that will allow us to do semantic code search over repositories in Sourcegraph. So, for example, you'll be able to ask, "how do access tokens work in Sourcegraph", and it will give you a list of the closest matching code files. Additionally, we build a context detection service powered by embeddings. In chat applications, it is important to know whether the user's message requires additional context. We have to differentiate between two cases: the user asks a general question about the codebase, or the user references something in the existing conversation. In the latter case, including the context would ruin the flow of the conversation, and the chatbot would most likely return a confusing answer. We determine whether a query _does not_ require additional context using two approaches: 1. We check if the query contains well-known phrases that would indicate the user is referencing the existing conversation (e.g., translate previous, change that) 1. We have a static dataset of messages that require context and a dataset of messages that do not. We embed both datasets, and then, using embedding similarity, we can check which set is more similar to the query. ## GraphQL API We add four new resolvers to the GraphQL API: ```graphql extend type Query { embeddingsSearch(repo: ID!, query: String!, codeResultsCount: Int!, textResultsCount: Int!): EmbeddingsSearchResults! isContextRequiredForQuery(query: String!): Boolean! } extend type Mutation { scheduleRepositoriesForEmbedding(repoNames: [String!]!): EmptyResponse! scheduleContextDetectionForEmbedding: EmptyResponse! } ``` - `embeddingsSearch` performs embeddings search over the repo embeddings and returns the specified number of results - `isContextRequiredForQuery` determines whether the given query requires additional context - `scheduleRepositoriesForEmbedding` schedules a repo embedding background job - `scheduleContextDetectionForEmbedding` schedules a context detection embedding background job that embeds a static dataset of messages. ## Repo embedding background job Embedding a repository is implemented as a background job. The background job handler receives the repository and the revision, which should be embedded. Handler then gathers a list of files from the gitserver and excludes files >1MB in size. The list of files is split into code and text files (.md, .txt), and we build a separate embedding index for both. We split them because in a combined index, the text files always tended to feature as top results and didn't leave any room for code files. Once we have the list of files, the procedure is as follows: - For each file - Get file contents from gitserver - Check if the file is embeddable (is not autogenerated, is large enough, does not have long lines) - Split the file into embeddable chunks - Embed the file chunks using an external embedding service (defined in site config) - Add embedded file chunks and metadata to the index - Metadata contains the file name, the start line, and the end line of the chunk - Once all files are processed, the index is marshaled into JSON and stored in Cloud storage (GCS, S3) ### Site config changes As mentioned, we use a configurable external embedding API that does the actual text -> vector embedding part. Ideally, this allows us to swap embedding providers in the future. ```json "embeddings": { "description": "Configuration for embeddings service.", "type": "object", "required": ["enabled", "dimensions", "model", "accessToken", "url"], "properties": { "enabled": { "description": "Toggles whether embedding service is enabled.", "type": "boolean", "default": false }, "dimensions": { "description": "The dimensionality of the embedding vectors.", "type": "integer", "minimum": 0 }, "model": { "description": "The model used for embedding.", "type": "string" }, "accessToken": { "description": "The access token used to authenticate with the external embedding API service.", "type": "string" }, "url": { "description": "The url to the external embedding API service.", "type": "string", "format": "uri" } } } ``` ## Repo embeddings search The repo embeddings search is implemented in its own service. When a user queries a repo using embeddings search, the following happens: - Download the repo embedding index from blob storage and cache it in memory - We cache up to 5 embedding indexes in memory - Embed the query and use the embedded query vector to find similar code and text file metadata in the embedding index - Query gitserver for the actual file contents - Return the results ## Interesting files - [Similarity search](https://github.com/sourcegraph/sourcegraph/pull/48017/files#diff-102cc83520004eb0e2795e49bc435c5142ca555189b1db3a52bbf1ffb82fa3c6) - [Repo embedding job handler](https://github.com/sourcegraph/sourcegraph/pull/48017/files#diff-c345f373f426398beb4b9cd5852ba862a2718687882db2a8b2d9c7fbb5f1dc52) - [External embedding api client](https://github.com/sourcegraph/sourcegraph/pull/48017/files#diff-ad1e7956f518e4bcaee17dd9e7ac04a5e090c00d970fcd273919e887e1d2cf8f) - [Embedding a repo](https://github.com/sourcegraph/sourcegraph/pull/48017/files#diff-1f35118727128095b7816791b6f0a2e0e060cddee43d25102859b8159465585c) - [Embeddings searcher service](https://github.com/sourcegraph/sourcegraph/pull/48017/files#diff-5b20f3e7ef87041daeeaef98b58ebf7388519cedcdfc359dc5e6d4e0b021472e) - [Embeddings search](https://github.com/sourcegraph/sourcegraph/pull/48017/files#diff-79f95b9cc3f1ef39c1a0b88015bd9cd6c19c30a8d4c147409f1b8e8cd9462ea1) - [Repo embedding index cache management](https://github.com/sourcegraph/sourcegraph/pull/48017/files#diff-8a41f7dec31054889dbf86e97c52223d5636b4d408c6b375bcfc09160a8b70f8) - [GraphQL resolvers](https://github.com/sourcegraph/sourcegraph/pull/48017/files#diff-9b30a0b5efcb63e2f4611b99ab137fbe09629a769a4f30d10a1b2da41a01d21f) ## Test plan - Start by filling out the `embeddings` object in the site config (let me know if you need an API key) - Start the embeddings service using `sg start embeddings` - Go to the `/api/console` page and schedule a repo embedding job and a context detection embedding job: ```graphql mutation { scheduleRepositoriesForEmbedding(repoNames: ["github.com/sourcegraph/handbook"]) { __typename } scheduleContextDetectionForEmbedding { __typename } } ``` - Once both are finished, you should be able to query the repo embedding index, and determine whether context is need for a given query: ```graphql query { isContextRequiredForQuery(query: "how do access tokens work") embeddingsSearch( repo: "UmVwb3NpdG9yeToy", # github.com/sourcegraph/handbook GQL ID query: "how do access tokens work", codeResultsCount: 5, textResultsCount: 5) { codeResults { fileName content } textResults { fileName content } } } ```
2023-03-01 09:50:12 +00:00
embeddings:
Add env variable to embeddings cmd in sg.config.yaml (#49973)
2023-03-24 17:11:38 +00:00
cmd: |
export SOURCEGRAPH_LICENSE_GENERATION_KEY=$(cat ../dev-private/enterprise/dev/test-license-generation-key.pem)
.bin/embeddings
embeddings: searcher and indexer (#48017) # High-level architecture overview <img width="2231" alt="Screenshot 2023-02-24 at 15 13 59" src="https://user-images.githubusercontent.com/6417322/221200130-53c1ff25-4c47-4532-885f-5c4f9dadb05e.png"> # Embeddings Really quickly: embeddings are a semantic representation of text. Embeddings are usually floating-point vectors with 256+ elements. The neat thing about embeddings is that they allow us to search over textual information using a semantic correlation between the query and the text, not just syntactic (matching keywords). In this PR, we implemented an embedding service that will allow us to do semantic code search over repositories in Sourcegraph. So, for example, you'll be able to ask, "how do access tokens work in Sourcegraph", and it will give you a list of the closest matching code files. Additionally, we build a context detection service powered by embeddings. In chat applications, it is important to know whether the user's message requires additional context. We have to differentiate between two cases: the user asks a general question about the codebase, or the user references something in the existing conversation. In the latter case, including the context would ruin the flow of the conversation, and the chatbot would most likely return a confusing answer. We determine whether a query _does not_ require additional context using two approaches: 1. We check if the query contains well-known phrases that would indicate the user is referencing the existing conversation (e.g., translate previous, change that) 1. We have a static dataset of messages that require context and a dataset of messages that do not. We embed both datasets, and then, using embedding similarity, we can check which set is more similar to the query. ## GraphQL API We add four new resolvers to the GraphQL API: ```graphql extend type Query { embeddingsSearch(repo: ID!, query: String!, codeResultsCount: Int!, textResultsCount: Int!): EmbeddingsSearchResults! isContextRequiredForQuery(query: String!): Boolean! } extend type Mutation { scheduleRepositoriesForEmbedding(repoNames: [String!]!): EmptyResponse! scheduleContextDetectionForEmbedding: EmptyResponse! } ``` - `embeddingsSearch` performs embeddings search over the repo embeddings and returns the specified number of results - `isContextRequiredForQuery` determines whether the given query requires additional context - `scheduleRepositoriesForEmbedding` schedules a repo embedding background job - `scheduleContextDetectionForEmbedding` schedules a context detection embedding background job that embeds a static dataset of messages. ## Repo embedding background job Embedding a repository is implemented as a background job. The background job handler receives the repository and the revision, which should be embedded. Handler then gathers a list of files from the gitserver and excludes files >1MB in size. The list of files is split into code and text files (.md, .txt), and we build a separate embedding index for both. We split them because in a combined index, the text files always tended to feature as top results and didn't leave any room for code files. Once we have the list of files, the procedure is as follows: - For each file - Get file contents from gitserver - Check if the file is embeddable (is not autogenerated, is large enough, does not have long lines) - Split the file into embeddable chunks - Embed the file chunks using an external embedding service (defined in site config) - Add embedded file chunks and metadata to the index - Metadata contains the file name, the start line, and the end line of the chunk - Once all files are processed, the index is marshaled into JSON and stored in Cloud storage (GCS, S3) ### Site config changes As mentioned, we use a configurable external embedding API that does the actual text -> vector embedding part. Ideally, this allows us to swap embedding providers in the future. ```json "embeddings": { "description": "Configuration for embeddings service.", "type": "object", "required": ["enabled", "dimensions", "model", "accessToken", "url"], "properties": { "enabled": { "description": "Toggles whether embedding service is enabled.", "type": "boolean", "default": false }, "dimensions": { "description": "The dimensionality of the embedding vectors.", "type": "integer", "minimum": 0 }, "model": { "description": "The model used for embedding.", "type": "string" }, "accessToken": { "description": "The access token used to authenticate with the external embedding API service.", "type": "string" }, "url": { "description": "The url to the external embedding API service.", "type": "string", "format": "uri" } } } ``` ## Repo embeddings search The repo embeddings search is implemented in its own service. When a user queries a repo using embeddings search, the following happens: - Download the repo embedding index from blob storage and cache it in memory - We cache up to 5 embedding indexes in memory - Embed the query and use the embedded query vector to find similar code and text file metadata in the embedding index - Query gitserver for the actual file contents - Return the results ## Interesting files - [Similarity search](https://github.com/sourcegraph/sourcegraph/pull/48017/files#diff-102cc83520004eb0e2795e49bc435c5142ca555189b1db3a52bbf1ffb82fa3c6) - [Repo embedding job handler](https://github.com/sourcegraph/sourcegraph/pull/48017/files#diff-c345f373f426398beb4b9cd5852ba862a2718687882db2a8b2d9c7fbb5f1dc52) - [External embedding api client](https://github.com/sourcegraph/sourcegraph/pull/48017/files#diff-ad1e7956f518e4bcaee17dd9e7ac04a5e090c00d970fcd273919e887e1d2cf8f) - [Embedding a repo](https://github.com/sourcegraph/sourcegraph/pull/48017/files#diff-1f35118727128095b7816791b6f0a2e0e060cddee43d25102859b8159465585c) - [Embeddings searcher service](https://github.com/sourcegraph/sourcegraph/pull/48017/files#diff-5b20f3e7ef87041daeeaef98b58ebf7388519cedcdfc359dc5e6d4e0b021472e) - [Embeddings search](https://github.com/sourcegraph/sourcegraph/pull/48017/files#diff-79f95b9cc3f1ef39c1a0b88015bd9cd6c19c30a8d4c147409f1b8e8cd9462ea1) - [Repo embedding index cache management](https://github.com/sourcegraph/sourcegraph/pull/48017/files#diff-8a41f7dec31054889dbf86e97c52223d5636b4d408c6b375bcfc09160a8b70f8) - [GraphQL resolvers](https://github.com/sourcegraph/sourcegraph/pull/48017/files#diff-9b30a0b5efcb63e2f4611b99ab137fbe09629a769a4f30d10a1b2da41a01d21f) ## Test plan - Start by filling out the `embeddings` object in the site config (let me know if you need an API key) - Start the embeddings service using `sg start embeddings` - Go to the `/api/console` page and schedule a repo embedding job and a context detection embedding job: ```graphql mutation { scheduleRepositoriesForEmbedding(repoNames: ["github.com/sourcegraph/handbook"]) { __typename } scheduleContextDetectionForEmbedding { __typename } } ``` - Once both are finished, you should be able to query the repo embedding index, and determine whether context is need for a given query: ```graphql query { isContextRequiredForQuery(query: "how do access tokens work") embeddingsSearch( repo: "UmVwb3NpdG9yeToy", # github.com/sourcegraph/handbook GQL ID query: "how do access tokens work", codeResultsCount: 5, textResultsCount: 5) { codeResults { fileName content } textResults { fileName content } } } ```
2023-03-01 09:50:12 +00:00
install: |
if [ -n "$DELVE" ]; then
export GCFLAGS='all=-N -l'
fi
go build -gcflags="$GCFLAGS" -o .bin/embeddings github.com/sourcegraph/sourcegraph/enterprise/cmd/embeddings
checkBinary: .bin/embeddings
watch:
- lib
- internal
- enterprise/cmd/embeddings
- enterprise/internal/embeddings
Move sg.config.yaml to repo root & allow private overwrites (#19487) This moves the sg.config.yaml to the repository root so that one can just use `sg start` and `sg run gitserver` etc. without having to specify the `-conf` flag. It also adds support for a local `.gitignore`d `sg.config.overwrites.yaml` file that users can add to the repository. If it exists, it will then get merged with the basic configuration when `sg` starts. What's the usecase? Custom command sets, custom env vars, etc. that you don't want to commit to the repository.
2021-03-29 08:14:53 +00:00
searcher:
cmd: .bin/searcher
sg: respect $DELVE (#25812)
2021-10-09 01:47:08 +00:00
install: |
if [ -n "$DELVE" ]; then
export GCFLAGS='all=-N -l'
fi
go build -gcflags="$GCFLAGS" -o .bin/searcher github.com/sourcegraph/sourcegraph/cmd/searcher
sg: do not restart binary if it didn't change (#21509) This adds the `checkBinary` property to `sg.config.yaml` which allows `sg` to check whether the specified binary actually changed after rebuilding a command. In practice this means that multiple file modifications that result in the same binary do not lead to multiple restarts.
2021-06-02 09:38:06 +00:00
checkBinary: .bin/searcher
Move sg.config.yaml to repo root & allow private overwrites (#19487) This moves the sg.config.yaml to the repository root so that one can just use `sg start` and `sg run gitserver` etc. without having to specify the `-conf` flag. It also adds support for a local `.gitignore`d `sg.config.overwrites.yaml` file that users can add to the repository. If it exists, it will then get merged with the basic configuration when `sg` starts. What's the usecase? Custom command sets, custom env vars, etc. that you don't want to commit to the repository.
2021-03-29 08:14:53 +00:00
watch:
sg: Fix configuration for code intel stack (#21532) Updates `sg.config.yaml` to fix booting up the code intel stack.
2021-05-31 07:51:39 +00:00
- lib
Move sg.config.yaml to repo root & allow private overwrites (#19487) This moves the sg.config.yaml to the repository root so that one can just use `sg start` and `sg run gitserver` etc. without having to specify the `-conf` flag. It also adds support for a local `.gitignore`d `sg.config.overwrites.yaml` file that users can add to the repository. If it exists, it will then get merged with the basic configuration when `sg` starts. What's the usecase? Custom command sets, custom env vars, etc. that you don't want to commit to the repository.
2021-03-29 08:14:53 +00:00
- internal
- cmd/searcher
caddy:
sg: Results of hack session with Erik (#20703) High output hacking: * Fixed env var expansion to fix zoekt-indexserver not being able to index repositories * Add ignoreStdout/ignoreStderr options to ignore command output * Add awesome logo * Add install script Fixes #20683
2021-05-06 14:58:47 +00:00
ignoreStdout: true
ignoreStderr: true
dev/sg: fix macos firewall exceptions (#34475)
2022-04-26 06:35:54 +00:00
cmd: .bin/caddy_${CADDY_VERSION} run --watch --config=dev/Caddyfile
Download caddy and unarchive it (#35426)
2022-05-23 07:58:53 +00:00
install_func: installCaddy
Move sg.config.yaml to repo root & allow private overwrites (#19487) This moves the sg.config.yaml to the repository root so that one can just use `sg start` and `sg run gitserver` etc. without having to specify the `-conf` flag. It also adds support for a local `.gitignore`d `sg.config.overwrites.yaml` file that users can add to the repository. If it exists, it will then get merged with the basic configuration when `sg` starts. What's the usecase? Custom command sets, custom env vars, etc. that you don't want to commit to the repository.
2021-03-29 08:14:53 +00:00
env:
Bump caddy version (#26329) This version distributes a binary for ARM based macs. Also, I fixed the script used in sg to allow upgrades of caddy.
2021-10-19 22:06:22 +00:00
CADDY_VERSION: 2.4.5
Move sg.config.yaml to repo root & allow private overwrites (#19487) This moves the sg.config.yaml to the repository root so that one can just use `sg start` and `sg run gitserver` etc. without having to specify the `-conf` flag. It also adds support for a local `.gitignore`d `sg.config.overwrites.yaml` file that users can add to the repository. If it exists, it will then get merged with the basic configuration when `sg` starts. What's the usecase? Custom command sets, custom env vars, etc. that you don't want to commit to the repository.
2021-03-29 08:14:53 +00:00
sg: shorten prefix for commands name (#37437)
2022-06-20 17:57:54 +00:00
oss-web:
sg: Add description field to sg commands (#39240)
2022-07-26 14:51:04 +00:00
description: Open source version of the web app
sg: Do not run gulp with --silent option (#24494) I suspect that when `--silent` is used, `gulp` also doesn't print errors on stderr. And since the output has become much less noisy, I think it's fine to re-enable it.
2021-09-01 08:41:58 +00:00
cmd: ./node_modules/.bin/gulp --color dev
web: migrate from `yarn` to `pnpm` (#46143) Bazel's rules_js rely on the pnpm package manager. To simplify the integration, we're migrating to pnpm from our current package manager — yarn. Another reason to migrate is that pnpm is cool and fast. 😉
2023-01-12 03:50:09 +00:00
install: pnpm install
Move sg.config.yaml to repo root & allow private overwrites (#19487) This moves the sg.config.yaml to the repository root so that one can just use `sg start` and `sg run gitserver` etc. without having to specify the `-conf` flag. It also adds support for a local `.gitignore`d `sg.config.overwrites.yaml` file that users can add to the repository. If it exists, it will then get merged with the basic configuration when `sg` starts. What's the usecase? Custom command sets, custom env vars, etc. that you don't want to commit to the repository.
2021-03-29 08:14:53 +00:00
sg: shorten prefix for commands name (#37437)
2022-06-20 17:57:54 +00:00
web:
sg: Add description field to sg commands (#39240)
2022-07-26 14:51:04 +00:00
description: Enterprise version of the web app
sg: Do not run gulp with --silent option (#24494) I suspect that when `--silent` is used, `gulp` also doesn't print errors on stderr. And since the output has become much less noisy, I think it's fine to re-enable it.
2021-09-01 08:41:58 +00:00
cmd: ./node_modules/.bin/gulp --color dev
web: migrate from `yarn` to `pnpm` (#46143) Bazel's rules_js rely on the pnpm package manager. To simplify the integration, we're migrating to pnpm from our current package manager — yarn. Another reason to migrate is that pnpm is cool and fast. 😉
2023-01-12 03:50:09 +00:00
install: pnpm install
sg: allow running enterprise env with `sg`, flesh out TODO/Inspiration/Quickstart/Hacking in README (#19643) This updates the `sg.config.yaml` so that `sg run-set enterprise` boots up the enterprise environment. It also updates the README to contain more TODOs, an Inspiration section, an updated Quickstart/Installation section and a Hacking section.
2021-04-01 11:59:48 +00:00
env:
ENTERPRISE: 1
debug: OpenTelemetry http/json tunnel on /-/debug/otlp (#39453) Adds an OpenTelemetry protocol (OTLP) adapter tunnel through the frontend (backend), /-/debug/otlp, that accepts http/json OTLP requests and sends them to either a grpc or http/json OpenTelemetry collector (depending on the configured protocol). We build an adapter because the OpenTelemetry collector is gRPC by default in most of its guidance, while browser environments can only support the http/json protocol. This allows admins to use their preferred OpenTelemetry backend of choice without worrying about compatibility with the http/json protocol. This is done by (ab)using OpenTelemetry collector (not the Go API) packages to partially implement an OpenTelemetry receiver on /-/debug/otlp - see the otlpadapter package for more details.
2022-08-02 14:09:41 +00:00
ENABLE_OPEN_TELEMETRY: true
web: add web-app server for development and production builds (#20126) This PR introduces an independent server for the web application, which can use any deployed API instance as a backend.
2021-05-07 11:01:10 +00:00
Rename web-standalone to web-standalone-http for http-only (#26495)
2021-10-22 10:30:29 +00:00
web-standalone-http:
sg: Add description field to sg commands (#39240)
2022-07-26 14:51:04 +00:00
description: Standalone web frontend (dev) with API proxy to a configurable URL
web: migrate from `yarn` to `pnpm` (#46143) Bazel's rules_js rely on the pnpm package manager. To simplify the integration, we're migrating to pnpm from our current package manager — yarn. Another reason to migrate is that pnpm is cool and fast. 😉
2023-01-12 03:50:09 +00:00
cmd: pnpm --filter @sourcegraph/web serve:dev --color
Add yarn generate to sg start web-standalone (#38277)
2022-07-06 09:51:02 +00:00
install: |
web: migrate from `yarn` to `pnpm` (#46143) Bazel's rules_js rely on the pnpm package manager. To simplify the integration, we're migrating to pnpm from our current package manager — yarn. Another reason to migrate is that pnpm is cool and fast. 😉
2023-01-12 03:50:09 +00:00
pnpm install
pnpm generate
web: add web-app server for development and production builds (#20126) This PR introduces an independent server for the web application, which can use any deployed API instance as a backend.
2021-05-07 11:01:10 +00:00
env:
WEBPACK_SERVE_INDEX: true
Code Insights: Create insight example section (#30300) * Restructure view components for future reuses * Implement code insights example layouts * Add storybook story * Update sg config yaml URL var
2022-01-28 14:15:56 +00:00
SOURCEGRAPH_API_URL: https://k8s.sgdev.org
web: add web-app server for development and production builds (#20126) This PR introduces an independent server for the web application, which can use any deployed API instance as a backend.
2021-05-07 11:01:10 +00:00
Rename web-standalone to web-standalone-http for http-only (#26495)
2021-10-22 10:30:29 +00:00
web-standalone-http-prod:
sg: Add description field to sg commands (#39240)
2022-07-26 14:51:04 +00:00
description: Standalone web frontend (production) with API proxy to a configurable URL
web: migrate from `yarn` to `pnpm` (#46143) Bazel's rules_js rely on the pnpm package manager. To simplify the integration, we're migrating to pnpm from our current package manager — yarn. Another reason to migrate is that pnpm is cool and fast. 😉
2023-01-12 03:50:09 +00:00
cmd: pnpm --filter @sourcegraph/web serve:prod
install: pnpm --filter @sourcegraph/web run build
web: add web-app server for development and production builds (#20126) This PR introduces an independent server for the web application, which can use any deployed API instance as a backend.
2021-05-07 11:01:10 +00:00
env:
NODE_ENV: production
WEBPACK_SERVE_INDEX: true
SOURCEGRAPH_API_URL: https://k8s.sgdev.org
sg: improve `sg` web testsuites (#41488)
2022-09-09 09:36:10 +00:00
web-integration-build:
[SG-22831] - Use `production` build for frontend integration tests (#46495)
2023-01-30 06:51:24 +00:00
description: Build development web application for integration tests
web: migrate from `yarn` to `pnpm` (#46143) Bazel's rules_js rely on the pnpm package manager. To simplify the integration, we're migrating to pnpm from our current package manager — yarn. Another reason to migrate is that pnpm is cool and fast. 😉
2023-01-12 03:50:09 +00:00
cmd: pnpm --filter @sourcegraph/web run build
sg: improve `sg` web testsuites (#41488)
2022-09-09 09:36:10 +00:00
env:
ENTERPRISE: 1
INTEGRATION_TESTS: true
[SG-22831] - Use `production` build for frontend integration tests (#46495)
2023-01-30 06:51:24 +00:00
web-integration-build-prod:
description: Build production web application for integration tests
cmd: pnpm --filter @sourcegraph/web run build
env:
ENTERPRISE: 1
INTEGRATION_TESTS: true
NODE_ENV: production
Move sg.config.yaml to repo root & allow private overwrites (#19487) This moves the sg.config.yaml to the repository root so that one can just use `sg start` and `sg run gitserver` etc. without having to specify the `-conf` flag. It also adds support for a local `.gitignore`d `sg.config.overwrites.yaml` file that users can add to the repository. If it exists, it will then get merged with the basic configuration when `sg` starts. What's the usecase? Custom command sets, custom env vars, etc. that you don't want to commit to the repository.
2021-03-29 08:14:53 +00:00
docsite:
sg: Add description field to sg commands (#39240)
2022-07-26 14:51:04 +00:00
description: Docsite instance serving the docs
sg: conditionally install docsite, add docsite check (#23606)
2021-08-05 08:19:45 +00:00
cmd: .bin/docsite_${DOCSITE_VERSION} -config doc/docsite.json serve -http=localhost:5080
sg: Add description field to sg commands (#39240)
2022-07-26 14:51:04 +00:00
install_func: 'installDocsite'
Move sg.config.yaml to repo root & allow private overwrites (#19487) This moves the sg.config.yaml to the repository root so that one can just use `sg start` and `sg run gitserver` etc. without having to specify the `-conf` flag. It also adds support for a local `.gitignore`d `sg.config.overwrites.yaml` file that users can add to the repository. If it exists, it will then get merged with the basic configuration when `sg` starts. What's the usecase? Custom command sets, custom env vars, etc. that you don't want to commit to the repository.
2021-03-29 08:14:53 +00:00
env:
chore: bump docsite version in `sg.config.yaml` (#50054)
2023-03-28 10:21:52 +00:00
DOCSITE_VERSION: v1.9.2 # Update in all places (including outside this repo)
Move sg.config.yaml to repo root & allow private overwrites (#19487) This moves the sg.config.yaml to the repository root so that one can just use `sg start` and `sg run gitserver` etc. without having to specify the `-conf` flag. It also adds support for a local `.gitignore`d `sg.config.overwrites.yaml` file that users can add to the repository. If it exists, it will then get merged with the basic configuration when `sg` starts. What's the usecase? Custom command sets, custom env vars, etc. that you don't want to commit to the repository.
2021-03-29 08:14:53 +00:00
sg: rename syntect_server to syntax-highlighter in dev (#25402)
2021-09-28 04:18:12 +00:00
syntax-highlighter:
sg: Results of hack session with Erik (#20703) High output hacking: * Fixed env var expansion to fix zoekt-indexserver not being able to index repositories * Add ignoreStdout/ignoreStderr options to ignore command output * Add awesome logo * Add install script Fixes #20683
2021-05-06 14:58:47 +00:00
ignoreStdout: true
ignoreStderr: true
Move sg.config.yaml to repo root & allow private overwrites (#19487) This moves the sg.config.yaml to the repository root so that one can just use `sg start` and `sg run gitserver` etc. without having to specify the `-conf` flag. It also adds support for a local `.gitignore`d `sg.config.overwrites.yaml` file that users can add to the repository. If it exists, it will then get merged with the basic configuration when `sg` starts. What's the usecase? Custom command sets, custom env vars, etc. that you don't want to commit to the repository.
2021-03-29 08:14:53 +00:00
cmd: |
sg: rename syntect_server to syntax-highlighter in dev (#25402)
2021-09-28 04:18:12 +00:00
docker run --name=syntax-highlighter --rm -p9238:9238 \
Move sg.config.yaml to repo root & allow private overwrites (#19487) This moves the sg.config.yaml to the repository root so that one can just use `sg start` and `sg run gitserver` etc. without having to specify the `-conf` flag. It also adds support for a local `.gitignore`d `sg.config.overwrites.yaml` file that users can add to the repository. If it exists, it will then get merged with the basic configuration when `sg` starts. What's the usecase? Custom command sets, custom env vars, etc. that you don't want to commit to the repository.
2021-03-29 08:14:53 +00:00
-e WORKERS=1 -e ROCKET_ADDRESS=0.0.0.0 \
sg: Change sg.config.yaml to point to insiders for syntax-highlighter (#31999) * sg: Change sg.config.yaml to point to insiders for syntax-highlighter * fixup: Also make sure to pull latest image * fixup: add -q
2022-03-01 18:19:16 +00:00
sourcegraph/syntax-highlighter:insiders
sg: rename syntect_server to syntax-highlighter in dev (#25402)
2021-09-28 04:18:12 +00:00
install: |
# Remove containers by the old name, too.
docker inspect syntect_server >/dev/null 2>&1 && docker rm -f syntect_server || true
docker inspect syntax-highlighter >/dev/null 2>&1 && docker rm -f syntax-highlighter || true
sg.config.yaml: Skip docker pull syntax-highligher if OFFLINE=true (#45161) Why: The docs currently suggest to comment out the docker pull command for the syntax-highlighter image if OFFLINE=true. Instead of making devs go through this process, skip the docker pull command if OFFLINE=true is already set. Better DevX. Win win. [1]: https://docs.sourcegraph.com/dev/background-information/sg#offline-development
2022-12-05 13:06:34 +00:00
# Pull syntax-highlighter latest insider image, only during install, but
# skip if OFFLINE=true is set.
if [[ "$OFFLINE" != "true" ]]; then
docker pull -q sourcegraph/syntax-highlighter:insiders
fi
Move sg.config.yaml to repo root & allow private overwrites (#19487) This moves the sg.config.yaml to the repository root so that one can just use `sg start` and `sg run gitserver` etc. without having to specify the `-conf` flag. It also adds support for a local `.gitignore`d `sg.config.overwrites.yaml` file that users can add to the repository. If it exists, it will then get merged with the basic configuration when `sg` starts. What's the usecase? Custom command sets, custom env vars, etc. that you don't want to commit to the repository.
2021-03-29 08:14:53 +00:00
sg: Fix configuration for code intel stack (#21532) Updates `sg.config.yaml` to fix booting up the code intel stack.
2021-05-31 07:51:39 +00:00
zoekt-indexserver-template: &zoekt_indexserver_template
Move sg.config.yaml to repo root & allow private overwrites (#19487) This moves the sg.config.yaml to the repository root so that one can just use `sg start` and `sg run gitserver` etc. without having to specify the `-conf` flag. It also adds support for a local `.gitignore`d `sg.config.overwrites.yaml` file that users can add to the repository. If it exists, it will then get merged with the basic configuration when `sg` starts. What's the usecase? Custom command sets, custom env vars, etc. that you don't want to commit to the repository.
2021-03-29 08:14:53 +00:00
cmd: |
sg: Fix PATH if it contains whitespace (#24187) Without this change I ran into problems where `env` would complain about a missing command because my `$PATH` contained a path with whitespace in it. [zoekt-indexserver-0] env: Fusion.app/Contents/Public:/Library/TeX/texbin:/Library/Apple/usr/bin:/Applications/Postgres.app/Contents/Versions/latest/bin:/Applications/kitty.app/Contents/MacOS: No such file or directory
2021-08-20 08:42:05 +00:00
env PATH="${PWD}/.bin:$PATH" .bin/zoekt-sourcegraph-indexserver \
Move sg.config.yaml to repo root & allow private overwrites (#19487) This moves the sg.config.yaml to the repository root so that one can just use `sg start` and `sg run gitserver` etc. without having to specify the `-conf` flag. It also adds support for a local `.gitignore`d `sg.config.overwrites.yaml` file that users can add to the repository. If it exists, it will then get merged with the basic configuration when `sg` starts. What's the usecase? Custom command sets, custom env vars, etc. that you don't want to commit to the repository.
2021-03-29 08:14:53 +00:00
-sourcegraph_url 'http://localhost:3090' \
sg: hacking hour with lots of YAML (#21250) This PR is a collection of all the things we did in the `sg` hacking hour just now: - Clean up the README so that the TODOs are up to date - Add definitions for `minio`, `codeinsights-db`, `executor-queue`, `precise-code-intel-worker` and `codeintel-executor` - Make it possible to launch Sourcegraph with codeinsights!! See the `sg.config.yaml` for which env vars to set and then run `sg run-set enterprise-codeinsights`
2021-05-21 16:01:11 +00:00
-index "$HOME/.sourcegraph/zoekt/index-$ZOEKT_NUM" \
-hostname "localhost:$ZOEKT_HOSTNAME_PORT" \
Move sg.config.yaml to repo root & allow private overwrites (#19487) This moves the sg.config.yaml to the repository root so that one can just use `sg start` and `sg run gitserver` etc. without having to specify the `-conf` flag. It also adds support for a local `.gitignore`d `sg.config.overwrites.yaml` file that users can add to the repository. If it exists, it will then get merged with the basic configuration when `sg` starts. What's the usecase? Custom command sets, custom env vars, etc. that you don't want to commit to the repository.
2021-03-29 08:14:53 +00:00
-interval 1m \
dev: Make zoekt services listen on 127.0.0.1 (#35047)
2022-05-06 11:37:11 +00:00
-listen "127.0.0.1:$ZOEKT_LISTEN_PORT" \
Move sg.config.yaml to repo root & allow private overwrites (#19487) This moves the sg.config.yaml to the repository root so that one can just use `sg start` and `sg run gitserver` etc. without having to specify the `-conf` flag. It also adds support for a local `.gitignore`d `sg.config.overwrites.yaml` file that users can add to the repository. If it exists, it will then get merged with the basic configuration when `sg` starts. What's the usecase? Custom command sets, custom env vars, etc. that you don't want to commit to the repository.
2021-03-29 08:14:53 +00:00
-cpu_fraction 0.25
install: |
mkdir -p .bin
export GOBIN="${PWD}/.bin"
unrevert "all: rename go module google/zoekt to sourcegraph/zoekt" (#40423) Last time we changed the zoekt module name it broke the wire protocol for zoekt. This now includes one more commit such that the wire protocol is unchanged: - c9182fcd2a rpc: use old module name when registering gob values Revert chain: - #40404 - #40353 Test Plan: Ran a local dev server with a hardcoded zoekt-webserver pinned to the old version (the google/zoekt module). indexed search was failing. Then upgraded the gomod version and it started working.
2022-08-17 22:30:49 +00:00
go install github.com/sourcegraph/zoekt/cmd/zoekt-archive-index
go install github.com/sourcegraph/zoekt/cmd/zoekt-git-index
go install github.com/sourcegraph/zoekt/cmd/zoekt-sourcegraph-indexserver
sg: do not restart binary if it didn't change (#21509) This adds the `checkBinary` property to `sg.config.yaml` which allows `sg` to check whether the specified binary actually changed after rebuilding a command. In practice this means that multiple file modifications that result in the same binary do not lead to multiple restarts.
2021-06-02 09:38:06 +00:00
checkBinary: .bin/zoekt-sourcegraph-indexserver
sg: hacking hour with lots of YAML (#21250) This PR is a collection of all the things we did in the `sg` hacking hour just now: - Clean up the README so that the TODOs are up to date - Add definitions for `minio`, `codeinsights-db`, `executor-queue`, `precise-code-intel-worker` and `codeintel-executor` - Make it possible to launch Sourcegraph with codeinsights!! See the `sg.config.yaml` for which env vars to set and then run `sg run-set enterprise-codeinsights`
2021-05-21 16:01:11 +00:00
env: &zoektenv
dev: add installer script for building a local universal-ctag (#45198)
2022-12-06 17:32:21 +00:00
CTAGS_COMMAND: dev/universal-ctags-dev
local dev: enable gRPC zoekt client communication (#49553)
2023-03-17 02:54:00 +00:00
GRPC_ENABLED: true
sg: allow running enterprise env with `sg`, flesh out TODO/Inspiration/Quickstart/Hacking in README (#19643) This updates the `sg.config.yaml` so that `sg run-set enterprise` boots up the enterprise environment. It also updates the README to contain more TODOs, an Inspiration section, an updated Quickstart/Installation section and a Hacking section.
2021-04-01 11:59:48 +00:00
sg: shorten prefix for commands name (#37437)
2022-06-20 17:57:54 +00:00
zoekt-index-0:
sg: Fix configuration for code intel stack (#21532) Updates `sg.config.yaml` to fix booting up the code intel stack.
2021-05-31 07:51:39 +00:00
<<: *zoekt_indexserver_template
sg: hacking hour with lots of YAML (#21250) This PR is a collection of all the things we did in the `sg` hacking hour just now: - Clean up the README so that the TODOs are up to date - Add definitions for `minio`, `codeinsights-db`, `executor-queue`, `precise-code-intel-worker` and `codeintel-executor` - Make it possible to launch Sourcegraph with codeinsights!! See the `sg.config.yaml` for which env vars to set and then run `sg run-set enterprise-codeinsights`
2021-05-21 16:01:11 +00:00
env:
<<: *zoektenv
ZOEKT_NUM: 0
ZOEKT_HOSTNAME_PORT: 3070
ZOEKT_LISTEN_PORT: 6072
sg: shorten prefix for commands name (#37437)
2022-06-20 17:57:54 +00:00
zoekt-index-1:
sg: Fix configuration for code intel stack (#21532) Updates `sg.config.yaml` to fix booting up the code intel stack.
2021-05-31 07:51:39 +00:00
<<: *zoekt_indexserver_template
Move sg.config.yaml to repo root & allow private overwrites (#19487) This moves the sg.config.yaml to the repository root so that one can just use `sg start` and `sg run gitserver` etc. without having to specify the `-conf` flag. It also adds support for a local `.gitignore`d `sg.config.overwrites.yaml` file that users can add to the repository. If it exists, it will then get merged with the basic configuration when `sg` starts. What's the usecase? Custom command sets, custom env vars, etc. that you don't want to commit to the repository.
2021-03-29 08:14:53 +00:00
env:
sg: hacking hour with lots of YAML (#21250) This PR is a collection of all the things we did in the `sg` hacking hour just now: - Clean up the README so that the TODOs are up to date - Add definitions for `minio`, `codeinsights-db`, `executor-queue`, `precise-code-intel-worker` and `codeintel-executor` - Make it possible to launch Sourcegraph with codeinsights!! See the `sg.config.yaml` for which env vars to set and then run `sg run-set enterprise-codeinsights`
2021-05-21 16:01:11 +00:00
<<: *zoektenv
ZOEKT_NUM: 1
ZOEKT_HOSTNAME_PORT: 3071
ZOEKT_LISTEN_PORT: 6073
Move sg.config.yaml to repo root & allow private overwrites (#19487) This moves the sg.config.yaml to the repository root so that one can just use `sg start` and `sg run gitserver` etc. without having to specify the `-conf` flag. It also adds support for a local `.gitignore`d `sg.config.overwrites.yaml` file that users can add to the repository. If it exists, it will then get merged with the basic configuration when `sg` starts. What's the usecase? Custom command sets, custom env vars, etc. that you don't want to commit to the repository.
2021-03-29 08:14:53 +00:00
sg: shorten prefix for commands name (#37437)
2022-06-20 17:57:54 +00:00
zoekt-web-template: &zoekt_webserver_template
Move sg.config.yaml to repo root & allow private overwrites (#19487) This moves the sg.config.yaml to the repository root so that one can just use `sg start` and `sg run gitserver` etc. without having to specify the `-conf` flag. It also adds support for a local `.gitignore`d `sg.config.overwrites.yaml` file that users can add to the repository. If it exists, it will then get merged with the basic configuration when `sg` starts. What's the usecase? Custom command sets, custom env vars, etc. that you don't want to commit to the repository.
2021-03-29 08:14:53 +00:00
install: |
mkdir -p .bin
unrevert "all: rename go module google/zoekt to sourcegraph/zoekt" (#40423) Last time we changed the zoekt module name it broke the wire protocol for zoekt. This now includes one more commit such that the wire protocol is unchanged: - c9182fcd2a rpc: use old module name when registering gob values Revert chain: - #40404 - #40353 Test Plan: Ran a local dev server with a hardcoded zoekt-webserver pinned to the old version (the google/zoekt module). indexed search was failing. Then upgraded the gomod version and it started working.
2022-08-17 22:30:49 +00:00
env GOBIN="${PWD}/.bin" go install github.com/sourcegraph/zoekt/cmd/zoekt-webserver
sg: do not restart binary if it didn't change (#21509) This adds the `checkBinary` property to `sg.config.yaml` which allows `sg` to check whether the specified binary actually changed after rebuilding a command. In practice this means that multiple file modifications that result in the same binary do not lead to multiple restarts.
2021-06-02 09:38:06 +00:00
checkBinary: .bin/zoekt-webserver
sg: allow running enterprise env with `sg`, flesh out TODO/Inspiration/Quickstart/Hacking in README (#19643) This updates the `sg.config.yaml` so that `sg run-set enterprise` boots up the enterprise environment. It also updates the README to contain more TODOs, an Inspiration section, an updated Quickstart/Installation section and a Hacking section.
2021-04-01 11:59:48 +00:00
env:
dev: enable OpenTelemetry by default in dev, improve tracer logging (#40946) Since observability components are not run by default in sg start, we also amend the error logging of the exporter to suppress issues unless observability.tracing.debug is enabled.
2022-08-29 16:42:36 +00:00
JAEGER_DISABLED: true
OPENTELEMETRY_DISABLED: false
zoekt: use GOGC=25 (#38708) After much experimentation we have a new value for GOGC, which is 25. We noticed a large regression in zoekts memory use when we switched to go1.18. So we reverted to go1.17 in zoekt releases and since then have been experimenting with different GOGC values. We have found the same stable heap size with GOGC=25 and have had this stable size for more than a week. Included zoekt changes: - 42276b7b70 docker: change our default GOGC to 25 - 267622ccb4 fix constant evaluation of empty repo set Test Plan: Lots of experimentation in our production work loads has been done. This is the outcome of that.
2022-07-14 11:08:58 +00:00
GOGC: 25
sg: allow running enterprise env with `sg`, flesh out TODO/Inspiration/Quickstart/Hacking in README (#19643) This updates the `sg.config.yaml` so that `sg run-set enterprise` boots up the enterprise environment. It also updates the README to contain more TODOs, an Inspiration section, an updated Quickstart/Installation section and a Hacking section.
2021-04-01 11:59:48 +00:00
sg: shorten prefix for commands name (#37437)
2022-06-20 17:57:54 +00:00
zoekt-web-0:
sg: Fix configuration for code intel stack (#21532) Updates `sg.config.yaml` to fix booting up the code intel stack.
2021-05-31 07:51:39 +00:00
<<: *zoekt_webserver_template
zoekt: set -indexserver_proxy for webserver (#44995) Relates to https://github.com/sourcegraph/zoekt/pull/487 We set the new flag for local dev and server.
2022-12-02 07:38:44 +00:00
cmd: env PATH="${PWD}/.bin:$PATH" .bin/zoekt-webserver -index "$HOME/.sourcegraph/zoekt/index-0" -pprof -rpc -indexserver_proxy -listen "127.0.0.1:3070"
sg: Fix configuration for code intel stack (#21532) Updates `sg.config.yaml` to fix booting up the code intel stack.
2021-05-31 07:51:39 +00:00
sg: shorten prefix for commands name (#37437)
2022-06-20 17:57:54 +00:00
zoekt-web-1:
sg: Fix configuration for code intel stack (#21532) Updates `sg.config.yaml` to fix booting up the code intel stack.
2021-05-31 07:51:39 +00:00
<<: *zoekt_webserver_template
zoekt: set -indexserver_proxy for webserver (#44995) Relates to https://github.com/sourcegraph/zoekt/pull/487 We set the new flag for local dev and server.
2022-12-02 07:38:44 +00:00
cmd: env PATH="${PWD}/.bin:$PATH" .bin/zoekt-webserver -index "$HOME/.sourcegraph/zoekt/index-1" -pprof -rpc -indexserver_proxy -listen "127.0.0.1:3071"
Move sg.config.yaml to repo root & allow private overwrites (#19487) This moves the sg.config.yaml to the repository root so that one can just use `sg start` and `sg run gitserver` etc. without having to specify the `-conf` flag. It also adds support for a local `.gitignore`d `sg.config.overwrites.yaml` file that users can add to the repository. If it exists, it will then get merged with the basic configuration when `sg` starts. What's the usecase? Custom command sets, custom env vars, etc. that you don't want to commit to the repository.
2021-03-29 08:14:53 +00:00
sg: shorten prefix for commands name (#37437)
2022-06-20 17:57:54 +00:00
codeintel-worker:
sg: set license generation key for codeintel-worker (#41302)
2022-09-05 10:16:27 +00:00
cmd: |
export SOURCEGRAPH_LICENSE_GENERATION_KEY=$(cat ../dev-private/enterprise/dev/test-license-generation-key.pem)
.bin/codeintel-worker
sg: hacking hour with lots of YAML (#21250) This PR is a collection of all the things we did in the `sg` hacking hour just now: - Clean up the README so that the TODOs are up to date - Add definitions for `minio`, `codeinsights-db`, `executor-queue`, `precise-code-intel-worker` and `codeintel-executor` - Make it possible to launch Sourcegraph with codeinsights!! See the `sg.config.yaml` for which env vars to set and then run `sg run-set enterprise-codeinsights`
2021-05-21 16:01:11 +00:00
install: |
sg: respect $DELVE (#25812)
2021-10-09 01:47:08 +00:00
if [ -n "$DELVE" ]; then
export GCFLAGS='all=-N -l'
fi
worker: actually track error metrics on worker handle error (#39978)
2022-08-04 21:21:45 +00:00
go build -gcflags="$GCFLAGS" -o .bin/codeintel-worker github.com/sourcegraph/sourcegraph/enterprise/cmd/precise-code-intel-worker
checkBinary: .bin/codeintel-worker
sg: hacking hour with lots of YAML (#21250) This PR is a collection of all the things we did in the `sg` hacking hour just now: - Clean up the README so that the TODOs are up to date - Add definitions for `minio`, `codeinsights-db`, `executor-queue`, `precise-code-intel-worker` and `codeintel-executor` - Make it possible to launch Sourcegraph with codeinsights!! See the `sg.config.yaml` for which env vars to set and then run `sg run-set enterprise-codeinsights`
2021-05-21 16:01:11 +00:00
watch:
sg: Fix configuration for code intel stack (#21532) Updates `sg.config.yaml` to fix booting up the code intel stack.
2021-05-31 07:51:39 +00:00
- lib
sg: hacking hour with lots of YAML (#21250) This PR is a collection of all the things we did in the `sg` hacking hour just now: - Clean up the README so that the TODOs are up to date - Add definitions for `minio`, `codeinsights-db`, `executor-queue`, `precise-code-intel-worker` and `codeintel-executor` - Make it possible to launch Sourcegraph with codeinsights!! See the `sg.config.yaml` for which env vars to set and then run `sg run-set enterprise-codeinsights`
2021-05-21 16:01:11 +00:00
- internal
sg: Fix configuration for code intel stack (#21532) Updates `sg.config.yaml` to fix booting up the code intel stack.
2021-05-31 07:51:39 +00:00
- enterprise/cmd/precise-code-intel-worker
sg: hacking hour with lots of YAML (#21250) This PR is a collection of all the things we did in the `sg` hacking hour just now: - Clean up the README so that the TODOs are up to date - Add definitions for `minio`, `codeinsights-db`, `executor-queue`, `precise-code-intel-worker` and `codeintel-executor` - Make it possible to launch Sourcegraph with codeinsights!! See the `sg.config.yaml` for which env vars to set and then run `sg run-set enterprise-codeinsights`
2021-05-21 16:01:11 +00:00
- enterprise/internal
- lib/codeintel
codeintel: Rockskip for symbols (#28719)
2022-03-03 04:13:28 +00:00
executor-template:
&executor_template # TMPDIR is set here so it's not set in the `install` process, which would trip up `go build`.
Add batches executor to dev env (#22578) * Add batch spec execution DB entity * Fixup migration sequence number * Add batches executor to dev env Co-authored-by: Erik Seliger <erikseliger@me.com>
2021-07-05 11:06:52 +00:00
cmd: |
env TMPDIR="$HOME/.sourcegraph/executor-temp" .bin/executor
sg: Fix configuration for code intel stack (#21532) Updates `sg.config.yaml` to fix booting up the code intel stack.
2021-05-31 07:51:39 +00:00
install: |
sg: respect $DELVE (#25812)
2021-10-09 01:47:08 +00:00
if [ -n "$DELVE" ]; then
export GCFLAGS='all=-N -l'
fi
chore: Bash bad (#42579)
2022-10-05 17:26:28 +00:00
go build -gcflags="$GCFLAGS" -o .bin/executor github.com/sourcegraph/sourcegraph/enterprise/cmd/executor
sg: do not restart binary if it didn't change (#21509) This adds the `checkBinary` property to `sg.config.yaml` which allows `sg` to check whether the specified binary actually changed after rebuilding a command. In practice this means that multiple file modifications that result in the same binary do not lead to multiple restarts.
2021-06-02 09:38:06 +00:00
checkBinary: .bin/executor
sg: hacking hour with lots of YAML (#21250) This PR is a collection of all the things we did in the `sg` hacking hour just now: - Clean up the README so that the TODOs are up to date - Add definitions for `minio`, `codeinsights-db`, `executor-queue`, `precise-code-intel-worker` and `codeintel-executor` - Make it possible to launch Sourcegraph with codeinsights!! See the `sg.config.yaml` for which env vars to set and then run `sg run-set enterprise-codeinsights`
2021-05-21 16:01:11 +00:00
env:
Fix add host gateway behavior for app (#50020) We don't only need to add this when the executor frontend URL points to it, but also if the site config is set to host.docker.internal, hence I made this an executor config flag instead. I think this should fix it, but `sg start app` is acting up for me so would appreciate if someone could verify that. Closes https://github.com/sourcegraph/sourcegraph/issues/50011
2023-03-28 21:18:56 +00:00
# Required for frontend and executor to communicate
EXECUTOR_FRONTEND_URL: http://localhost:3080
# Must match the secret defined in the site config.
EXECUTOR_FRONTEND_PASSWORD: hunter2hunter2hunter2
# Disable firecracker inside executor in dev
EXECUTOR_USE_FIRECRACKER: false
Add batches executor to dev env (#22578) * Add batch spec execution DB entity * Fixup migration sequence number * Add batches executor to dev env Co-authored-by: Erik Seliger <erikseliger@me.com>
2021-07-05 11:06:52 +00:00
EXECUTOR_QUEUE_NAME: TEMPLATE
sg: Fix configuration for code intel stack (#21532) Updates `sg.config.yaml` to fix booting up the code intel stack.
2021-05-31 07:51:39 +00:00
watch:
- lib
- internal
- enterprise/internal
- enterprise/cmd/executor
sg: hacking hour with lots of YAML (#21250) This PR is a collection of all the things we did in the `sg` hacking hour just now: - Clean up the README so that the TODOs are up to date - Add definitions for `minio`, `codeinsights-db`, `executor-queue`, `precise-code-intel-worker` and `codeintel-executor` - Make it possible to launch Sourcegraph with codeinsights!! See the `sg.config.yaml` for which env vars to set and then run `sg run-set enterprise-codeinsights`
2021-05-21 16:01:11 +00:00
Add batches executor to dev env (#22578) * Add batch spec execution DB entity * Fixup migration sequence number * Add batches executor to dev env Co-authored-by: Erik Seliger <erikseliger@me.com>
2021-07-05 11:06:52 +00:00
codeintel-executor:
<<: *executor_template
cmd: |
env TMPDIR="$HOME/.sourcegraph/indexer-temp" .bin/executor
env:
EXECUTOR_QUEUE_NAME: codeintel
executors: Reduce disk IO by attaching workspace as block device (#41335) Co-authored-by: Erik Seliger <erikseliger@me.com>
2022-09-21 19:39:50 +00:00
# If you want to use this, either start it with `sg run batches-executor-firecracker` or
# modify the `commandsets.batches` in your local `sg.config.overwrite.yaml`
codeintel-executor-firecracker:
<<: *executor_template
cmd: |
env TMPDIR="$HOME/.sourcegraph/codeintel-executor-temp" \
executors: Make setup simpler (#42026) Outside of our provided terraform modules, the setup of executors is very tedious and error prone today. This PR tries to address that by adding as much validation of the env as possible, adding hopefully helpful resolutions, and removing some of the manual steps that we did in the VM install script.
2022-10-05 13:32:30 +00:00
sudo --preserve-env=TMPDIR,EXECUTOR_QUEUE_NAME,EXECUTOR_FRONTEND_URL,EXECUTOR_FRONTEND_PASSWORD,EXECUTOR_USE_FIRECRACKER \
executors: Reduce disk IO by attaching workspace as block device (#41335) Co-authored-by: Erik Seliger <erikseliger@me.com>
2022-09-21 19:39:50 +00:00
.bin/executor
env:
EXECUTOR_USE_FIRECRACKER: true
EXECUTOR_QUEUE_NAME: codeintel
Add batches executor to dev env (#22578) * Add batch spec execution DB entity * Fixup migration sequence number * Add batches executor to dev env Co-authored-by: Erik Seliger <erikseliger@me.com>
2021-07-05 11:06:52 +00:00
batches-executor:
<<: *executor_template
cmd: |
env TMPDIR="$HOME/.sourcegraph/batches-executor-temp" .bin/executor
env:
Delete BatchSpecExecutions (#25178)
2021-09-22 10:03:57 +00:00
EXECUTOR_QUEUE_NAME: batches
ssbc: Implement a subset of resolvers (#25182)
2021-09-29 12:17:18 +00:00
EXECUTOR_MAXIMUM_NUM_JOBS: 8
Add batches executor to dev env (#22578) * Add batch spec execution DB entity * Fixup migration sequence number * Add batches executor to dev env Co-authored-by: Erik Seliger <erikseliger@me.com>
2021-07-05 11:06:52 +00:00
Experiment: Natively run SSBC in docker (#44034) This adds an experimental code path that I will use to test a docker-only execution mode for server-side batch changes. This code path is never executed for customers until we make the switch when we deem it ready. This will allow me to dogfood this while it's not available to customer instances yet. Ultimately, the goal of this is to make executors simply be "the job runner platform through a generic interface". Today, this depends on src-cli to do a good bunch of the work. This is a blocker for going full docker-based with executors, which will ultimately be a requirement on the road to k8s-based executors. As this removes the dependency on src-cli, nothing but the job interface and API endpoints tie executor and Sourcegraph instance together. Ultimately, this will allow us to support larger version spans between the two (pending executors going GA and being feature-complete). Known issues/limitations: Steps skipped in between steps that run don't work yet Skipping steps dynamically is inefficient as we cannot tell the executor to skip a step IF X, so we replace the script by exit 0 It is unclear if all variants of file mounts still work. Basic cases do work. Files used to be read-only in src-cli, they aren't now, but content is still reset in between steps. The assumption that everything operates in /work is broken here, because we need to use what executors give us to persist out-of-repo state in between containers (like the step result from the previous step) It is unclear if workspace mounts work Cache keys are not correctly computed if using workspace mounts - the metadataretriever is nil We still use log outputs to transfer the AfterStepResults to the Sourcegraph instance, this should finally become an artifact instead. Then, we don't have to rely on the execution_log_entires anymore and can theoretically prune those after some time. This column is currently growing indefinitely. It depends on tee being available in the docker images to capture the cmd.stdout/cmd.stderr properly for template variable rendering Env-vars are not rendered in their evaluated form post-execution File permissions are unclear and might be similarly broken to how they are now - or even worse Disclaimer: It's not feature complete today! But it is also not hitting any default code paths either. As development on this goes on, we can eventually remove the feature flag and run the new job format on all instances. This PR handles fallback of rendering old records correctly in the UI already.
2022-11-09 23:20:43 +00:00
# This tool rebuilds the batcheshelper image every time the source of it is changed.
batcheshelper-builder:
# Nothing to run for this, we just want to re-run the install script every time.
cmd: exit 0
install: ./enterprise/cmd/batcheshelper/build.sh
env:
IMAGE: sourcegraph/batcheshelper:insiders
# TODO: This is required but should only be set on M1 Macs.
PLATFORM: linux/arm64
watch:
- enterprise/cmd/batcheshelper
- lib/batches
continueWatchOnExit: true
sg: add run-set for core app (#23156)
2021-07-23 07:59:43 +00:00
# If you want to use this, either start it with `sg run batches-executor-firecracker` or
Stream ExecutionLogEntries from executor (#22884) Instead of writing all entries to the database once the job is finished we write them continuously. Co-authored-by: Thorsten Ball <mrnugget@gmail.com>
2021-07-22 15:09:36 +00:00
# modify the `commandsets.batches` in your local `sg.config.overwrite.yaml`
batches-executor-firecracker:
<<: *executor_template
cmd: |
env TMPDIR="$HOME/.sourcegraph/batches-executor-temp" \
executors: Make setup simpler (#42026) Outside of our provided terraform modules, the setup of executors is very tedious and error prone today. This PR tries to address that by adding as much validation of the env as possible, adding hopefully helpful resolutions, and removing some of the manual steps that we did in the VM install script.
2022-10-05 13:32:30 +00:00
sudo --preserve-env=TMPDIR,EXECUTOR_QUEUE_NAME,EXECUTOR_FRONTEND_URL,EXECUTOR_FRONTEND_PASSWORD,EXECUTOR_USE_FIRECRACKER \
Stream ExecutionLogEntries from executor (#22884) Instead of writing all entries to the database once the job is finished we write them continuously. Co-authored-by: Thorsten Ball <mrnugget@gmail.com>
2021-07-22 15:09:36 +00:00
.bin/executor
env:
EXECUTOR_USE_FIRECRACKER: true
Delete BatchSpecExecutions (#25178)
2021-09-22 10:03:57 +00:00
EXECUTOR_QUEUE_NAME: batches
Stream ExecutionLogEntries from executor (#22884) Instead of writing all entries to the database once the job is finished we write them continuously. Co-authored-by: Thorsten Ball <mrnugget@gmail.com>
2021-07-22 15:09:36 +00:00
all: introduce sourcegraph/blobstore; add option to cmd/server (#44255) * all: add new sg blobstore backend to supersede minio * cmd/server: optionally run sg blobstore backend over minio * lsifuploadstore: add minio back to help text * sg generate * uploadstore: add DeleteObjects / NewListObjectsV2Paginator to s3API interface * uploadstore: regenerate mocks * uploadstore: add gcsAPI.Objects listing API * uploadstore: regenerate mocks * uploadstore: add Store.ExpireObjects stub implementation (thanks eric) * uploadstore: implement ExpireObjects in GCS/S3/Lazy interfaces * worker: integrate lsifuploadstore expirer * uploadstore: update test API calls * uploadstore: add TODO for myself * uploadstore: batch S3 object deletions * uploadstore: use operation logger instead of log15 * cmd/server: use sglog * cmd/server: fix image build Signed-off-by: Stephen Gutekanst <stephen@sourcegraph.com>
2022-11-30 23:04:48 +00:00
blobstore:
dev: use new Go blobstore implementation in dev env (#50212) Today, Sourcegraph [ships the Java service s3proxy](https://github.com/sourcegraph/sourcegraph/tree/main/docker-images/blobstore) as our S3-compatible blob storage provider out-of-the-box. I have been working on [a Go implementation](https://github.com/sourcegraph/sourcegraph/tree/main/cmd/blobstore) which is generally nicer, reports errors in a more logical way, and just generally fits into Sourcegraph better. We already ship the new Go implementation as part of the Sourcegraph app (it is easier to distribute) and now with this PR we will also use the new Go implementation in all our dev environments, too. This is one step along the road to ultimately getting rid of s3proxy entirely: once we feel confident about running the new Go implementation in our dev environments, we can begin to ship the new implementation to all our enterprise customers as well. There are a few [small issues](https://github.com/sourcegraph/sourcegraph/issues/45594#issuecomment-1421935287) we should fix first. Depends on #50211 Fixes #49865 Helps #45594 ## Test plan This commit can be easily reverted. If issues arise, please feel free to revert it. To test this I ran the codeintel-qa test suite: ``` sg start enterprise-codeintel export SOURCEGRAPH_SUDO_TOKEN=<redacted> export SOURCEGRAPH_BASE_URL=http://localhost:3080 cd dev/codeintel-qa go run ./cmd/download go run ./cmd/upload go run ./cmd/query ```
2023-04-03 22:15:13 +00:00
cmd: .bin/blobstore
sg: hacking hour with lots of YAML (#21250) This PR is a collection of all the things we did in the `sg` hacking hour just now: - Clean up the README so that the TODOs are up to date - Add definitions for `minio`, `codeinsights-db`, `executor-queue`, `precise-code-intel-worker` and `codeintel-executor` - Make it possible to launch Sourcegraph with codeinsights!! See the `sg.config.yaml` for which env vars to set and then run `sg run-set enterprise-codeinsights`
2021-05-21 16:01:11 +00:00
install: |
dev: use new Go blobstore implementation in dev env (#50212) Today, Sourcegraph [ships the Java service s3proxy](https://github.com/sourcegraph/sourcegraph/tree/main/docker-images/blobstore) as our S3-compatible blob storage provider out-of-the-box. I have been working on [a Go implementation](https://github.com/sourcegraph/sourcegraph/tree/main/cmd/blobstore) which is generally nicer, reports errors in a more logical way, and just generally fits into Sourcegraph better. We already ship the new Go implementation as part of the Sourcegraph app (it is easier to distribute) and now with this PR we will also use the new Go implementation in all our dev environments, too. This is one step along the road to ultimately getting rid of s3proxy entirely: once we feel confident about running the new Go implementation in our dev environments, we can begin to ship the new implementation to all our enterprise customers as well. There are a few [small issues](https://github.com/sourcegraph/sourcegraph/issues/45594#issuecomment-1421935287) we should fix first. Depends on #50211 Fixes #49865 Helps #45594 ## Test plan This commit can be easily reverted. If issues arise, please feel free to revert it. To test this I ran the codeintel-qa test suite: ``` sg start enterprise-codeintel export SOURCEGRAPH_SUDO_TOKEN=<redacted> export SOURCEGRAPH_BASE_URL=http://localhost:3080 cd dev/codeintel-qa go run ./cmd/download go run ./cmd/upload go run ./cmd/query ```
2023-04-03 22:15:13 +00:00
# Ensure the old blobstore Docker container is not running
docker rm -f blobstore
if [ -n "$DELVE" ]; then
export GCFLAGS='all=-N -l'
fi
go build -gcflags="$GCFLAGS" -o .bin/blobstore github.com/sourcegraph/sourcegraph/cmd/blobstore
checkBinary: .bin/blobstore
watch:
- lib
- internal
- cmd/blobstore
sg: hacking hour with lots of YAML (#21250) This PR is a collection of all the things we did in the `sg` hacking hour just now: - Clean up the README so that the TODOs are up to date - Add definitions for `minio`, `codeinsights-db`, `executor-queue`, `precise-code-intel-worker` and `codeintel-executor` - Make it possible to launch Sourcegraph with codeinsights!! See the `sg.config.yaml` for which env vars to set and then run `sg run-set enterprise-codeinsights`
2021-05-21 16:01:11 +00:00
env:
dev: use new Go blobstore implementation in dev env (#50212) Today, Sourcegraph [ships the Java service s3proxy](https://github.com/sourcegraph/sourcegraph/tree/main/docker-images/blobstore) as our S3-compatible blob storage provider out-of-the-box. I have been working on [a Go implementation](https://github.com/sourcegraph/sourcegraph/tree/main/cmd/blobstore) which is generally nicer, reports errors in a more logical way, and just generally fits into Sourcegraph better. We already ship the new Go implementation as part of the Sourcegraph app (it is easier to distribute) and now with this PR we will also use the new Go implementation in all our dev environments, too. This is one step along the road to ultimately getting rid of s3proxy entirely: once we feel confident about running the new Go implementation in our dev environments, we can begin to ship the new implementation to all our enterprise customers as well. There are a few [small issues](https://github.com/sourcegraph/sourcegraph/issues/45594#issuecomment-1421935287) we should fix first. Depends on #50211 Fixes #49865 Helps #45594 ## Test plan This commit can be easily reverted. If issues arise, please feel free to revert it. To test this I ran the codeintel-qa test suite: ``` sg start enterprise-codeintel export SOURCEGRAPH_SUDO_TOKEN=<redacted> export SOURCEGRAPH_BASE_URL=http://localhost:3080 cd dev/codeintel-qa go run ./cmd/download go run ./cmd/upload go run ./cmd/query ```
2023-04-03 22:15:13 +00:00
BLOBSTORE_DATA_DIR: $HOME/.sourcegraph-dev/data/blobstore-go
sg: hacking hour with lots of YAML (#21250) This PR is a collection of all the things we did in the `sg` hacking hour just now: - Clean up the README so that the TODOs are up to date - Add definitions for `minio`, `codeinsights-db`, `executor-queue`, `precise-code-intel-worker` and `codeintel-executor` - Make it possible to launch Sourcegraph with codeinsights!! See the `sg.config.yaml` for which env vars to set and then run `sg run-set enterprise-codeinsights`
2021-05-21 16:01:11 +00:00
dev(sg): add 'run redis-postgres' via compose (#22921)
2021-07-19 16:15:00 +00:00
redis-postgres:
# Add the following overwrites to your sg.config.overwrite.yaml to use the docker-compose
# database:
#
# env:
Fix PGHOST env var in sg.config.yaml (#36485) POSTGRES_HOST doesn't seem to work when running `sg run redis-postgres`: $ POSTGRES_HOST=localhost PGPASSWORD=sourcegraph PGUSER=sourcegraph psql psql: error: connection to server on socket "/tmp/.s.PGSQL.5432" failed: No such file or directory Is the server running locally and accepting connections on that socket? $ PGHOST=localhost PGPASSWORD=sourcegraph PGUSER=sourcegraph psql Null display is "[NULL]". Expanded display is used automatically. psql (14.1, server 12.7) Type "help" for help. sourcegraph@localhost:5432 sourcegraph#
2022-06-02 16:06:54 +00:00
# PGHOST: localhost
dev(sg): add 'run redis-postgres' via compose (#22921)
2021-07-19 16:15:00 +00:00
# PGPASSWORD: sourcegraph
# PGUSER: sourcegraph
#
# You could also add an overwrite to add `redis-postgres` to the relevant command set(s).
sg: Add description field to sg commands (#39240)
2022-07-26 14:51:04 +00:00
description: Dockerized version of redis and postgres
dev/sg: fix redis-postgres persistence (#25476)
2021-09-29 11:57:55 +00:00
cmd: docker-compose -f dev/redis-postgres.yml up $COMPOSE_ARGS
env:
COMPOSE_ARGS: --force-recreate
dev(sg): add 'run redis-postgres' via compose (#22921)
2021-07-19 16:15:00 +00:00
sg: migrate jaeger and grafana to sg.config.yaml (#23413) The other two are just placeholders and will be migrated later.
2021-08-02 08:25:39 +00:00
jaeger:
cmd: |
dev/sg: instrument with OpenTelemetry traces, submit to Honeycomb (#38889) Migrates sg to directly use OpenTelemetry tracing to track events, and updates sg analytics submit to submit traces to Honeycomb. The data we end up with is quite similar to the data we had in OkayHQ, but this time as proper traces - the analytics API is a bit more complicated than before, but I think the flexibility gains here are worth it, and in practice few parts of sg will require very thorough instrumentation. We also provide a constructor through which spans should be created, which appends a standardized category attribute and returns a wrapped span with shortcuts for common events Note that I have not implemented an automatic way to get the Honeycomb token yet. Cool side effect: you can run sg start otel, OTEL_EXPORTER_OTLP_ENDPOINT=http://127.0.0.1:4317, and submit your traces to a local Jaeger instance instead
2022-07-20 04:34:16 +00:00
echo "Jaeger will be available on http://localhost:16686/-/debug/jaeger/search"
sg: simplify jaeger binary naming (#34524)
2022-04-27 07:26:00 +00:00
.bin/jaeger-all-in-one-${JAEGER_VERSION} --log-level ${JAEGER_LOG_LEVEL}
sg: add install func for Jaeger (#36726) * sg: add install func for Jaeger * Update dev/sg/internal/run/run.go Co-authored-by: Thorsten Ball <mrnugget@gmail.com> Co-authored-by: Thorsten Ball <mrnugget@gmail.com>
2022-06-07 15:46:15 +00:00
install_func: installJaeger
sg: migrate jaeger and grafana to sg.config.yaml (#23413) The other two are just placeholders and will be migrated later.
2021-08-02 08:25:39 +00:00
env:
dev/sg: instrument with OpenTelemetry traces, submit to Honeycomb (#38889) Migrates sg to directly use OpenTelemetry tracing to track events, and updates sg analytics submit to submit traces to Honeycomb. The data we end up with is quite similar to the data we had in OkayHQ, but this time as proper traces - the analytics API is a bit more complicated than before, but I think the flexibility gains here are worth it, and in practice few parts of sg will require very thorough instrumentation. We also provide a constructor through which spans should be created, which appends a standardized category attribute and returns a wrapped span with shortcuts for common events Note that I have not implemented an automatic way to get the Honeycomb token yet. Cool side effect: you can run sg start otel, OTEL_EXPORTER_OTLP_ENDPOINT=http://127.0.0.1:4317, and submit your traces to a local Jaeger instance instead
2022-07-20 04:34:16 +00:00
JAEGER_VERSION: 1.36.0
sg: migrate jaeger and grafana to sg.config.yaml (#23413) The other two are just placeholders and will be migrated later.
2021-08-02 08:25:39 +00:00
JAEGER_DISK: $HOME/.sourcegraph-dev/data/jaeger
sg: correctly download jaeger (#26984) Currently jaeger just doesn't start in our dev environment since it used "version" as an envvar. Additionally, we didn't notice this was broken since "-e" bash option was not set. This is now set. We should consider it for all shell commands in sg. The log level is also lowered for Jaeger, since it is quiet spammy.
2021-11-03 10:15:18 +00:00
JAEGER_LOG_LEVEL: error
sg: migrate jaeger and grafana to sg.config.yaml (#23413) The other two are just placeholders and will be migrated later.
2021-08-02 08:25:39 +00:00
QUERY_BASE_PATH: /-/debug/jaeger
grafana:
cmd: |
sg: use `uname` instead of `$OSTYPE` to check for OS (#40602) sg: fallback OS check
2022-08-19 11:55:58 +00:00
if [[ $(uname) == "Linux" ]]; then
sg: Migrate prometheus to sg.config.yaml (#23447)
2021-08-02 11:28:54 +00:00
# Linux needs an extra arg to support host.internal.docker, which is how grafana connects
# to the prometheus backend.
ADD_HOST_FLAG="--add-host=host.docker.internal:host-gateway"
# Docker users on Linux will generally be using direct user mapping, which
# means that they'll want the data in the volume mount to be owned by the
# same user as is running this script. Fortunately, the Grafana container
# doesn't really care what user it runs as, so long as it can write to
# /var/lib/grafana.
DOCKER_USER="--user=$UID"
fi
sg: migrate jaeger and grafana to sg.config.yaml (#23413) The other two are just placeholders and will be migrated later.
2021-08-02 08:25:39 +00:00
echo "Grafana: serving on http://localhost:${PORT}"
echo "Grafana: note that logs are piped to ${GRAFANA_LOG_FILE}"
docker run --rm ${DOCKER_USER} \
--name=${CONTAINER} \
--cpus=1 \
--memory=1g \
-p 0.0.0.0:3370:3370 ${ADD_HOST_FLAG} \
-v "${GRAFANA_DISK}":/var/lib/grafana \
-v "$(pwd)"/dev/grafana/all:/sg_config_grafana/provisioning/datasources \
monitoring: link directly to panels and dashsboards from alerts panels (#24920) * alerts panels can now be clicked to go directly to relevant panels and dashboards * filtering on alerts tables * place custom template vars at end for consistency * deprecate custom annotations - unlikely this will be used
2021-09-14 18:36:04 +00:00
sourcegraph/grafana:dev >"${GRAFANA_LOG_FILE}" 2>&1
sg: migrate jaeger and grafana to sg.config.yaml (#23413) The other two are just placeholders and will be migrated later.
2021-08-02 08:25:39 +00:00
install: |
sg: Migrate prometheus to sg.config.yaml (#23447)
2021-08-02 11:28:54 +00:00
mkdir -p "${GRAFANA_DISK}"
sg: migrate jaeger and grafana to sg.config.yaml (#23413) The other two are just placeholders and will be migrated later.
2021-08-02 08:25:39 +00:00
mkdir -p "$(dirname ${GRAFANA_LOG_FILE})"
worker: actually track error metrics on worker handle error (#39978)
2022-08-04 21:21:45 +00:00
export CACHE=true
sg: migrate jaeger and grafana to sg.config.yaml (#23413) The other two are just placeholders and will be migrated later.
2021-08-02 08:25:39 +00:00
docker inspect $CONTAINER >/dev/null 2>&1 && docker rm -f $CONTAINER
monitoring: link directly to panels and dashsboards from alerts panels (#24920) * alerts panels can now be clicked to go directly to relevant panels and dashboards * filtering on alerts tables * place custom template vars at end for consistency * deprecate custom annotations - unlikely this will be used
2021-09-14 18:36:04 +00:00
./docker-images/grafana/build.sh
sg: migrate jaeger and grafana to sg.config.yaml (#23413) The other two are just placeholders and will be migrated later.
2021-08-02 08:25:39 +00:00
env:
GRAFANA_DISK: $HOME/.sourcegraph-dev/data/grafana
# Log file location: since we log outside of the Docker container, we should
# log somewhere that's _not_ ~/.sourcegraph-dev/data/grafana, since that gets
# volume mounted into the container and therefore has its own ownership
# semantics.
# Now for the actual logging. Grafana's output gets sent to stdout and stderr.
# We want to capture that output, but because it's fairly noisy, don't want to
# display it in the normal case.
sg: Migrate prometheus to sg.config.yaml (#23447)
2021-08-02 11:28:54 +00:00
GRAFANA_LOG_FILE: $HOME/.sourcegraph-dev/logs/grafana/grafana.log
Add Redis monitoring dashboard (#26969) Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2021-11-08 17:09:12 +00:00
IMAGE: sourcegraph/grafana:dev
sg: migrate jaeger and grafana to sg.config.yaml (#23413) The other two are just placeholders and will be migrated later.
2021-08-02 08:25:39 +00:00
CONTAINER: grafana
PORT: 3370
# docker containers must access things via docker host on non-linux platforms
codeintel: Rockskip for symbols (#28719)
2022-03-03 04:13:28 +00:00
DOCKER_USER: ''
ADD_HOST_FLAG: ''
monitoring: link directly to panels and dashsboards from alerts panels (#24920) * alerts panels can now be clicked to go directly to relevant panels and dashboards * filtering on alerts tables * place custom template vars at end for consistency * deprecate custom annotations - unlikely this will be used
2021-09-14 18:36:04 +00:00
CACHE: false
sg: migrate jaeger and grafana to sg.config.yaml (#23413) The other two are just placeholders and will be migrated later.
2021-08-02 08:25:39 +00:00
prometheus:
sg: Migrate prometheus to sg.config.yaml (#23447)
2021-08-02 11:28:54 +00:00
cmd: |
sg: use `uname` instead of `$OSTYPE` to check for OS (#40602) sg: fallback OS check
2022-08-19 11:55:58 +00:00
if [[ $(uname) == "Linux" ]]; then
sg: Migrate prometheus to sg.config.yaml (#23447)
2021-08-02 11:28:54 +00:00
DOCKER_USER="--user=$UID"
# Frontend generally runs outside of Docker, so to access it we need to be
# able to access ports on the host. --net=host is a very dirty way of
# enabling this.
DOCKER_NET="--net=host"
SRC_FRONTEND_INTERNAL="localhost:3090"
fi
echo "Prometheus: serving on http://localhost:${PORT}"
echo "Prometheus: note that logs are piped to ${PROMETHEUS_LOG_FILE}"
docker run --rm ${DOCKER_NET} ${DOCKER_USER} \
--name=${CONTAINER} \
--cpus=1 \
--memory=4g \
-p 0.0.0.0:9090:9090 \
-v "${PROMETHEUS_DISK}":/prometheus \
-v "$(pwd)/${CONFIG_DIR}":/sg_prometheus_add_ons \
-e SRC_FRONTEND_INTERNAL="${SRC_FRONTEND_INTERNAL}" \
-e DISABLE_SOURCEGRAPH_CONFIG="${DISABLE_SOURCEGRAPH_CONFIG:-""}" \
-e DISABLE_ALERTMANAGER="${DISABLE_ALERTMANAGER:-""}" \
prometheus: disable web.enable-lifecycle and web.enable-admin-api outside of dev (#46393) Closes https://github.com/sourcegraph/security-issues/issues/329 - we only need the reload API in development, in deployments the Prometheus configuration should never change.
2023-01-13 01:44:57 +00:00
-e PROMETHEUS_ADDITIONAL_FLAGS="--web.enable-lifecycle --web.enable-admin-api" \
sg: Migrate prometheus to sg.config.yaml (#23447)
2021-08-02 11:28:54 +00:00
${IMAGE} >"${PROMETHEUS_LOG_FILE}" 2>&1
install: |
mkdir -p "${PROMETHEUS_DISK}"
mkdir -p "$(dirname ${PROMETHEUS_LOG_FILE})"
docker inspect $CONTAINER >/dev/null 2>&1 && docker rm -f $CONTAINER
sg: use `uname` instead of `$OSTYPE` to check for OS (#40602) sg: fallback OS check
2022-08-19 11:55:58 +00:00
if [[ $(uname) == "Linux" ]]; then
sg: Migrate prometheus to sg.config.yaml (#23447)
2021-08-02 11:28:54 +00:00
PROM_TARGETS="dev/prometheus/linux/prometheus_targets.yml"
fi
cp ${PROM_TARGETS} "${CONFIG_DIR}"/prometheus_targets.yml
CACHE=true ./docker-images/prometheus/build.sh
env:
PROMETHEUS_DISK: $HOME/.sourcegraph-dev/data/prometheus
# See comment above for `grafana`
PROMETHEUS_LOG_FILE: $HOME/.sourcegraph-dev/logs/prometheus/prometheus.log
IMAGE: sourcegraph/prometheus:dev
CONTAINER: prometheus
PORT: 9090
CONFIG_DIR: docker-images/prometheus/config
codeintel: Rockskip for symbols (#28719)
2022-03-03 04:13:28 +00:00
DOCKER_USER: ''
DOCKER_NET: ''
sg: Migrate prometheus to sg.config.yaml (#23447)
2021-08-02 11:28:54 +00:00
PROM_TARGETS: dev/prometheus/all/prometheus_targets.yml
SRC_FRONTEND_INTERNAL: host.docker.internal:3090
codeintel: Rockskip for symbols (#28719)
2022-03-03 04:13:28 +00:00
ADD_HOST_FLAG: ''
sg: Migrate prometheus to sg.config.yaml (#23447)
2021-08-02 11:28:54 +00:00
DISABLE_SOURCEGRAPH_CONFIG: false
sg: migrate jaeger and grafana to sg.config.yaml (#23413) The other two are just placeholders and will be migrated later.
2021-08-02 08:25:39 +00:00
postgres_exporter:
cmd: ./dev/postgres_exporter.sh
sg: support reloading+retrying of non-long-running commands (#23895) Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2021-08-13 17:50:27 +00:00
monitoring-generator:
monitoring: extract into a submodule (#45786) This change extracts `monitoring` into a submodule for import in `sourcegraph/controller` (https://github.com/sourcegraph/controller/pull/195) so that we can generate dashboards for Cloud instances. These steps were required: 1. Initialize a `go.mod` in `monitoring` 2. Extract `dev/sg/internal/cliutil` into `lib` to avoid illegal imports from `monitoring` 3. Add local replaces to both `sourcegraph/sourcegraph` and `monitoring` 4. `go mod tidy` on all submodules 5. Update `go generate ./monitoring` commands to use `sg`, since the `go generate` command no longer works 6. Update `grafana/build.sh`, `prometheus/build.sh` to build the submodule 7. Amend linters to check for multiple `go.mod` files and ban imports of `github.com/sourcegraph/sourcegraph` 8. Update `sg generate go` to run in directories rather than from root The only caveat is that if you use VS code, you will now need to open `monitoring` in a separate workspace or similar, like with `lib`. Co-authored-by: Joe Chen <joe@sourcegraph.com>
2022-12-19 17:49:25 +00:00
cmd: (cd monitoring/ && go generate ./... )
sg: support reloading+retrying of non-long-running commands (#23895) Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2021-08-13 17:50:27 +00:00
env:
RELOAD: true
watch:
- monitoring
continueWatchOnExit: true
dev/sg: introduce 'sg ci logs' with Loki support (#25835) Introduce `sg ci logs` to pull job logs from a build's (by default) failed jobs for you to grep, pipe to another file, or do whatever manipulation you want with it. `sg ci logs` also includes support for pushing logs into a Loki instance for aggregation and querying, for example: sg run loki grafana # grafana and loki sg ci logs --out=http://127.0.0.1:3100 # push job logs to a Loki instance A long-term goal of the Loki support introduced here is to have failed builds automatically upload logs for query. This is part of https://github.com/sourcegraph/sourcegraph/issues/25562 - learn more about these efforts in RFC 496: https://github.com/sourcegraph/sourcegraph/issues/25768 Co-authored-by: Jean Hadrien Chabran <jh@chabran.fr>
2021-10-12 15:14:50 +00:00
loki:
cmd: |
sg ci logs: fix ci branch, improve error handling (#27145) - fix `getCIBranch` always failing in CI - better message for missing bk timestamps - continue pushing logs on error - set additional branch label from `BUILDKITE_BRANCH` (only, for now)
2021-11-05 23:16:36 +00:00
echo "Loki: serving on http://localhost:3100"
dev/sg: introduce 'sg ci logs' with Loki support (#25835) Introduce `sg ci logs` to pull job logs from a build's (by default) failed jobs for you to grep, pipe to another file, or do whatever manipulation you want with it. `sg ci logs` also includes support for pushing logs into a Loki instance for aggregation and querying, for example: sg run loki grafana # grafana and loki sg ci logs --out=http://127.0.0.1:3100 # push job logs to a Loki instance A long-term goal of the Loki support introduced here is to have failed builds automatically upload logs for query. This is part of https://github.com/sourcegraph/sourcegraph/issues/25562 - learn more about these efforts in RFC 496: https://github.com/sourcegraph/sourcegraph/issues/25768 Co-authored-by: Jean Hadrien Chabran <jh@chabran.fr>
2021-10-12 15:14:50 +00:00
echo "Loki: note that logs are piped to ${LOKI_LOG_FILE}"
docker run --rm --name=loki \
-p 3100:3100 -v $LOKI_DISK:/loki \
index.docker.io/grafana/loki:$LOKI_VERSION >"${LOKI_LOG_FILE}" 2>&1
install: |
mkdir -p "${LOKI_DISK}"
mkdir -p "$(dirname ${LOKI_LOG_FILE})"
docker pull index.docker.io/grafana/loki:$LOKI_VERSION
env:
LOKI_DISK: $HOME/.sourcegraph-dev/data/loki
codeintel: Rockskip for symbols (#28719)
2022-03-03 04:13:28 +00:00
LOKI_VERSION: '2.3.0'
dev/sg: introduce 'sg ci logs' with Loki support (#25835) Introduce `sg ci logs` to pull job logs from a build's (by default) failed jobs for you to grep, pipe to another file, or do whatever manipulation you want with it. `sg ci logs` also includes support for pushing logs into a Loki instance for aggregation and querying, for example: sg run loki grafana # grafana and loki sg ci logs --out=http://127.0.0.1:3100 # push job logs to a Loki instance A long-term goal of the Loki support introduced here is to have failed builds automatically upload logs for query. This is part of https://github.com/sourcegraph/sourcegraph/issues/25562 - learn more about these efforts in RFC 496: https://github.com/sourcegraph/sourcegraph/issues/25768 Co-authored-by: Jean Hadrien Chabran <jh@chabran.fr>
2021-10-12 15:14:50 +00:00
LOKI_LOG_FILE: $HOME/.sourcegraph-dev/logs/loki/loki.log
internal/tracer: add OpenTelemetry tracer (#37984) We extend internal/tracer to support a new tracer type, "opentelemetry", that sends everything in OpenTelemetry format using the OpenTracing bridge. See https://github.com/sourcegraph/sourcegraph/pull/37984 for more details.
2022-07-05 18:28:15 +00:00
otel-collector:
docker-images: publish sourcegraph/opentelemetry-collector with bundled config (#39002) Co-authored-by: Keegan Carruthers-Smith <keegan.csmith@gmail.com>
2022-07-20 19:06:23 +00:00
install: docker-images/opentelemetry-collector/build.sh
sg: Add description field to sg commands (#39240)
2022-07-26 14:51:04 +00:00
description: OpenTelemetry collector
internal/tracer: add OpenTelemetry tracer (#37984) We extend internal/tracer to support a new tracer type, "opentelemetry", that sends everything in OpenTelemetry format using the OpenTracing bridge. See https://github.com/sourcegraph/sourcegraph/pull/37984 for more details.
2022-07-05 18:28:15 +00:00
cmd: |
JAEGER_HOST='host.docker.internal'
sg: use `uname` instead of `$OSTYPE` to check for OS (#40602) sg: fallback OS check
2022-08-19 11:55:58 +00:00
if [[ $(uname) == "Linux" ]]; then
internal/tracer: add OpenTelemetry tracer (#37984) We extend internal/tracer to support a new tracer type, "opentelemetry", that sends everything in OpenTelemetry format using the OpenTracing bridge. See https://github.com/sourcegraph/sourcegraph/pull/37984 for more details.
2022-07-05 18:28:15 +00:00
# Jaeger generally runs outside of Docker, so to access it we need to be
# able to access ports on the host, because the Docker host only exists on
# MacOS. --net=host is a very dirty way of enabling this.
DOCKER_NET="--net=host"
JAEGER_HOST="localhost"
fi
docker-images: publish sourcegraph/opentelemetry-collector with bundled config (#39002) Co-authored-by: Keegan Carruthers-Smith <keegan.csmith@gmail.com>
2022-07-20 19:06:23 +00:00
docker container rm otel-collector
docker run --rm --name=otel-collector $DOCKER_NET $DOCKER_ARGS \
otel: add tailsamplingprocessor to collector build (#42444) Co-authored-by: William Bezuidenhout <william.bezuidenhout@sourcegraph.com>
2022-10-18 10:23:21 +00:00
-p 4317:4317 -p 4318:4318 -p 55679:55679 -p 55670:55670 \
otel: add collector dashboard (#45009) * add initial dashboard for otel * add failed sent dashboard * extra panels * use sum and rate for resource queries * review comments * add warning alerts * Update monitoring/definitions/otel_collector.go * review comments * run go generate * Update monitoring/definitions/otel_collector.go Co-authored-by: Robert Lin <robert@bobheadxi.dev> * Update monitoring/definitions/otel_collector.go Co-authored-by: Robert Lin <robert@bobheadxi.dev> * Update monitoring/definitions/otel_collector.go Co-authored-by: Robert Lin <robert@bobheadxi.dev> * Update monitoring/definitions/otel_collector.go Co-authored-by: Robert Lin <robert@bobheadxi.dev> * Update monitoring/definitions/otel_collector.go Co-authored-by: Robert Lin <robert@bobheadxi.dev> * Update monitoring/definitions/otel_collector.go Co-authored-by: Robert Lin <robert@bobheadxi.dev> * review comments * review feedback also drop two panels * remove brackets in metrics * update docs * fix goimport * gogenerate Co-authored-by: Robert Lin <robert@bobheadxi.dev> Co-authored-by: Jean-Hadrien Chabran <jh@chabran.fr>
2022-12-19 12:18:51 +00:00
-p 8888:8888 \
internal/tracer: add OpenTelemetry tracer (#37984) We extend internal/tracer to support a new tracer type, "opentelemetry", that sends everything in OpenTelemetry format using the OpenTracing bridge. See https://github.com/sourcegraph/sourcegraph/pull/37984 for more details.
2022-07-05 18:28:15 +00:00
-e JAEGER_HOST=$JAEGER_HOST \
docker-images: publish sourcegraph/opentelemetry-collector with bundled config (#39002) Co-authored-by: Keegan Carruthers-Smith <keegan.csmith@gmail.com>
2022-07-20 19:06:23 +00:00
-e HONEYCOMB_API_KEY=$HONEYCOMB_API_KEY \
-e HONEYCOMB_DATASET=$HONEYCOMB_DATASET \
$IMAGE --config "/etc/otel-collector/$CONFIGURATION_FILE"
internal/tracer: add OpenTelemetry tracer (#37984) We extend internal/tracer to support a new tracer type, "opentelemetry", that sends everything in OpenTelemetry format using the OpenTracing bridge. See https://github.com/sourcegraph/sourcegraph/pull/37984 for more details.
2022-07-05 18:28:15 +00:00
env:
docker-images: publish sourcegraph/opentelemetry-collector with bundled config (#39002) Co-authored-by: Keegan Carruthers-Smith <keegan.csmith@gmail.com>
2022-07-20 19:06:23 +00:00
IMAGE: sourcegraph/opentelemetry-collector:dev
# Overwrite the following in sg.config.overwrite.yaml, based on which collector
# config you are using - see docker-images/opentelemetry-collector for more details.
CONFIGURATION_FILE: 'configs/jaeger.yaml'
# HONEYCOMB_API_KEY: ''
# HONEYCOMB_DATASET: ''
internal/tracer: add OpenTelemetry tracer (#37984) We extend internal/tracer to support a new tracer type, "opentelemetry", that sends everything in OpenTelemetry format using the OpenTracing bridge. See https://github.com/sourcegraph/sourcegraph/pull/37984 for more details.
2022-07-05 18:28:15 +00:00
sg: Add storybook command (#26271) I found it convenient to have all commands in one tool, so I added an alias for `yarn storybook` to sg. If you don't think this should be in here, I'll add it to my local overrides instead.
2021-10-19 13:11:21 +00:00
storybook:
web: migrate from `yarn` to `pnpm` (#46143) Bazel's rules_js rely on the pnpm package manager. To simplify the integration, we're migrating to pnpm from our current package manager — yarn. Another reason to migrate is that pnpm is cool and fast. 😉
2023-01-12 03:50:09 +00:00
cmd: pnpm storybook
install: pnpm install
sg: Add storybook command (#26271) I found it convenient to have all commands in one tool, so I added an alias for `yarn storybook` to sg. If you don't think this should be in here, I'll add it to my local overrides instead.
2021-10-19 13:11:21 +00:00
sg: add debug-env command (#25509) To debug environment you can run `sg run debug-env` to see the computed environment.
2021-09-30 07:23:24 +00:00
# This will execute `env`, a utility to print the process environment. Can
# be used to debug which global vars `sg` uses.
debug-env:
sg: Add description field to sg commands (#39240)
2022-07-26 14:51:04 +00:00
description: Debug env vars
sg: add debug-env command (#25509) To debug environment you can run `sg run debug-env` to see the computed environment.
2021-09-30 07:23:24 +00:00
cmd: env
Add browser extension sg cli support (#27276) * Add 'sg test bext|bext-integration|bext-e2e' to run corresponding tests * Add 'sg run bext' to start bext in dev mode
2021-11-10 09:26:15 +00:00
bext:
web: migrate from `yarn` to `pnpm` (#46143) Bazel's rules_js rely on the pnpm package manager. To simplify the integration, we're migrating to pnpm from our current package manager — yarn. Another reason to migrate is that pnpm is cool and fast. 😉
2023-01-12 03:50:09 +00:00
cmd: pnpm --filter @sourcegraph/browser dev
install: pnpm install
Add browser extension sg cli support (#27276) * Add 'sg test bext|bext-integration|bext-e2e' to run corresponding tests * Add 'sg run bext' to start bext in dev mode
2021-11-10 09:26:15 +00:00
Sourcegraph App (single-binary branch) (#46547) * internal: add service and singleprogram packages * sg.config.yaml: add single-binary build targets * internal/env: add a function for clearing environ cache * internal/{workerutil,metrics}: add a hack to allow running 2 executors in the same process * internal/conf: add single-program deploy type * internal/singleprogram: clarify security * cmd/sourcegraph-oss: add initial single-binary main (will not build yet) * enterprise/cmd/sourcegraph: initial enterprise single-binary * Add multi-platform builds for single-program * single-binary: correctly build JS artifacts into binary * license_finder licenses add github.com/xi2/xz "Public domain" * internal/service/svcmain: correctly initialize logger for DeprecatedSingleServiceMain * worker: refactor to new service pattern * cmd/github-proxy: refactor to use new service pattern * symbols: refactor to use new service pattern * gitserver: refactor to user new service pattern * searcher: refactor to use new service pattern * gitserver: refactor to use new service pattern * repo-updater: refactor to use new service pattern * frontend: refactor to use new service pattern * executor: refactor to use new service pattern * internal/symbols: use new LoadConfig pattern * precise-code-intel-worker: refactor to use new service pattern * internal/symbols: load config for tests * cmd/repo-updater: remove LoadConfig approach * cmd/symbols: workaround env var conflict with searcher * executor: internal: add workaround to allow running 2 instances in same process * executors: add EXECUTOR_QUEUE_DISABLE_ACCESS_TOKEN for single-binary and dev deployments only * single-binary: use EXECUTOR_QUEUE_DISABLE_ACCESS_TOKEN * extsvc/github: fix default value for single-program deploy type * single-binary: stop relying on a local ctags image * single-binary: use unix sockets for postgres * release App snapshots in CI when pushed to app/release-snapshot branch * internal/service/svcmain: update TODO comment * executor: correct DEPLOY_TYPE check * dev/check: allow single-binary to import dbconn * executor: remove accidental reliance on dbconn package * executor: improve error logging when running commands (#46546) * executor: improve error logging when running commands * executor: do not attempt std config validation running e.g. install cmd * executor: do not pull in the conf package / frontend reliance * ci: executors: correct site config for passwordless auth * server: fix bug where github-proxy would try to be a conf server * CI: executors: fix integration test passwordless auth * executors: allow passwordless auth in sourcegraph/server for testing * repo-updater: fix enterprise init (caused regression in repository syncing) Signed-off-by: Stephen Gutekanst <stephen@sourcegraph.com> Co-authored-by: Peter Guy <peter.guy@sourcegraph.com> Co-authored-by: Quinn Slack <quinn@slack.org>
2023-01-20 00:35:39 +00:00
sourcegraph:
description: Single program (Go static binary) distribution
cmd: |
redispool: use postgres for redispool.Store in App (#47188) We introduce a new table "redis_key_value" which is written to by KeyValue. The DB backed KeyValue store needs to wait for the database to be ready. However, Store is a package global. As such we use a pattern were we register the DB into redispool once the DB is ready. Before that all the KeyValue operations will return an error. Test Plan: go test and manual testing of Sourcegraph App.
2023-02-03 12:59:05 +00:00
unset SRC_GIT_SERVERS INDEXED_SEARCH_SERVERS REDIS_ENDPOINT
Sourcegraph App (single-binary branch) (#46547) * internal: add service and singleprogram packages * sg.config.yaml: add single-binary build targets * internal/env: add a function for clearing environ cache * internal/{workerutil,metrics}: add a hack to allow running 2 executors in the same process * internal/conf: add single-program deploy type * internal/singleprogram: clarify security * cmd/sourcegraph-oss: add initial single-binary main (will not build yet) * enterprise/cmd/sourcegraph: initial enterprise single-binary * Add multi-platform builds for single-program * single-binary: correctly build JS artifacts into binary * license_finder licenses add github.com/xi2/xz "Public domain" * internal/service/svcmain: correctly initialize logger for DeprecatedSingleServiceMain * worker: refactor to new service pattern * cmd/github-proxy: refactor to use new service pattern * symbols: refactor to use new service pattern * gitserver: refactor to user new service pattern * searcher: refactor to use new service pattern * gitserver: refactor to use new service pattern * repo-updater: refactor to use new service pattern * frontend: refactor to use new service pattern * executor: refactor to use new service pattern * internal/symbols: use new LoadConfig pattern * precise-code-intel-worker: refactor to use new service pattern * internal/symbols: load config for tests * cmd/repo-updater: remove LoadConfig approach * cmd/symbols: workaround env var conflict with searcher * executor: internal: add workaround to allow running 2 instances in same process * executors: add EXECUTOR_QUEUE_DISABLE_ACCESS_TOKEN for single-binary and dev deployments only * single-binary: use EXECUTOR_QUEUE_DISABLE_ACCESS_TOKEN * extsvc/github: fix default value for single-program deploy type * single-binary: stop relying on a local ctags image * single-binary: use unix sockets for postgres * release App snapshots in CI when pushed to app/release-snapshot branch * internal/service/svcmain: update TODO comment * executor: correct DEPLOY_TYPE check * dev/check: allow single-binary to import dbconn * executor: remove accidental reliance on dbconn package * executor: improve error logging when running commands (#46546) * executor: improve error logging when running commands * executor: do not attempt std config validation running e.g. install cmd * executor: do not pull in the conf package / frontend reliance * ci: executors: correct site config for passwordless auth * server: fix bug where github-proxy would try to be a conf server * CI: executors: fix integration test passwordless auth * executors: allow passwordless auth in sourcegraph/server for testing * repo-updater: fix enterprise init (caused regression in repository syncing) Signed-off-by: Stephen Gutekanst <stephen@sourcegraph.com> Co-authored-by: Peter Guy <peter.guy@sourcegraph.com> Co-authored-by: Quinn Slack <quinn@slack.org>
2023-01-20 00:35:39 +00:00
# TODO: This should be fixed
export SOURCEGRAPH_LICENSE_GENERATION_KEY=$(cat ../dev-private/enterprise/dev/test-license-generation-key.pem)
# If EXTSVC_CONFIG_FILE is *unset*, set a default.
export EXTSVC_CONFIG_FILE=${EXTSVC_CONFIG_FILE-'../dev-private/enterprise/dev/external-services-config.json'}
.bin/sourcegraph
install: |
if [ -n "$DELVE" ]; then
export GCFLAGS='all=-N -l'
fi
Replace single-program with app in some config files (#49091) Follow-up to #49064 The recent change broke `sg start app` and this fixes it ## Test plan - `sg start app` works again <!-- All pull requests REQUIRE a test plan: https://docs.sourcegraph.com/dev/background-information/testing_principles -->
2023-03-10 12:36:20 +00:00
go build -gcflags="$GCFLAGS" -ldflags="-X github.com/sourcegraph/sourcegraph/internal/conf/deploy.forceType=app" -o .bin/sourcegraph github.com/sourcegraph/sourcegraph/enterprise/cmd/sourcegraph
Sourcegraph App (single-binary branch) (#46547) * internal: add service and singleprogram packages * sg.config.yaml: add single-binary build targets * internal/env: add a function for clearing environ cache * internal/{workerutil,metrics}: add a hack to allow running 2 executors in the same process * internal/conf: add single-program deploy type * internal/singleprogram: clarify security * cmd/sourcegraph-oss: add initial single-binary main (will not build yet) * enterprise/cmd/sourcegraph: initial enterprise single-binary * Add multi-platform builds for single-program * single-binary: correctly build JS artifacts into binary * license_finder licenses add github.com/xi2/xz "Public domain" * internal/service/svcmain: correctly initialize logger for DeprecatedSingleServiceMain * worker: refactor to new service pattern * cmd/github-proxy: refactor to use new service pattern * symbols: refactor to use new service pattern * gitserver: refactor to user new service pattern * searcher: refactor to use new service pattern * gitserver: refactor to use new service pattern * repo-updater: refactor to use new service pattern * frontend: refactor to use new service pattern * executor: refactor to use new service pattern * internal/symbols: use new LoadConfig pattern * precise-code-intel-worker: refactor to use new service pattern * internal/symbols: load config for tests * cmd/repo-updater: remove LoadConfig approach * cmd/symbols: workaround env var conflict with searcher * executor: internal: add workaround to allow running 2 instances in same process * executors: add EXECUTOR_QUEUE_DISABLE_ACCESS_TOKEN for single-binary and dev deployments only * single-binary: use EXECUTOR_QUEUE_DISABLE_ACCESS_TOKEN * extsvc/github: fix default value for single-program deploy type * single-binary: stop relying on a local ctags image * single-binary: use unix sockets for postgres * release App snapshots in CI when pushed to app/release-snapshot branch * internal/service/svcmain: update TODO comment * executor: correct DEPLOY_TYPE check * dev/check: allow single-binary to import dbconn * executor: remove accidental reliance on dbconn package * executor: improve error logging when running commands (#46546) * executor: improve error logging when running commands * executor: do not attempt std config validation running e.g. install cmd * executor: do not pull in the conf package / frontend reliance * ci: executors: correct site config for passwordless auth * server: fix bug where github-proxy would try to be a conf server * CI: executors: fix integration test passwordless auth * executors: allow passwordless auth in sourcegraph/server for testing * repo-updater: fix enterprise init (caused regression in repository syncing) Signed-off-by: Stephen Gutekanst <stephen@sourcegraph.com> Co-authored-by: Peter Guy <peter.guy@sourcegraph.com> Co-authored-by: Quinn Slack <quinn@slack.org>
2023-01-20 00:35:39 +00:00
checkBinary: .bin/sourcegraph
env:
ENTERPRISE: 1
SITE_CONFIG_FILE: '../dev-private/enterprise/dev/site-config.json'
SITE_CONFIG_ESCAPE_HATCH_PATH: '$HOME/.sourcegraph/site-config.json'
WEBPACK_DEV_SERVER: 1
watch:
- cmd
- enterprise
- internal
- lib
- schema
sourcegraph-oss:
description: Single program (Go static binary) distribution, OSS variant
cmd: |
unset SRC_GIT_SERVERS INDEXED_SEARCH_SERVERS
.bin/sourcegraph-oss
install: |
if [ -n "$DELVE" ]; then
export GCFLAGS='all=-N -l'
fi
Replace single-program with app in some config files (#49091) Follow-up to #49064 The recent change broke `sg start app` and this fixes it ## Test plan - `sg start app` works again <!-- All pull requests REQUIRE a test plan: https://docs.sourcegraph.com/dev/background-information/testing_principles -->
2023-03-10 12:36:20 +00:00
go build -gcflags="$GCFLAGS" -ldflags="-X github.com/sourcegraph/sourcegraph/internal/conf/deploy.forceType=app" -o .bin/sourcegraph-oss github.com/sourcegraph/sourcegraph/cmd/sourcegraph-oss
Sourcegraph App (single-binary branch) (#46547) * internal: add service and singleprogram packages * sg.config.yaml: add single-binary build targets * internal/env: add a function for clearing environ cache * internal/{workerutil,metrics}: add a hack to allow running 2 executors in the same process * internal/conf: add single-program deploy type * internal/singleprogram: clarify security * cmd/sourcegraph-oss: add initial single-binary main (will not build yet) * enterprise/cmd/sourcegraph: initial enterprise single-binary * Add multi-platform builds for single-program * single-binary: correctly build JS artifacts into binary * license_finder licenses add github.com/xi2/xz "Public domain" * internal/service/svcmain: correctly initialize logger for DeprecatedSingleServiceMain * worker: refactor to new service pattern * cmd/github-proxy: refactor to use new service pattern * symbols: refactor to use new service pattern * gitserver: refactor to user new service pattern * searcher: refactor to use new service pattern * gitserver: refactor to use new service pattern * repo-updater: refactor to use new service pattern * frontend: refactor to use new service pattern * executor: refactor to use new service pattern * internal/symbols: use new LoadConfig pattern * precise-code-intel-worker: refactor to use new service pattern * internal/symbols: load config for tests * cmd/repo-updater: remove LoadConfig approach * cmd/symbols: workaround env var conflict with searcher * executor: internal: add workaround to allow running 2 instances in same process * executors: add EXECUTOR_QUEUE_DISABLE_ACCESS_TOKEN for single-binary and dev deployments only * single-binary: use EXECUTOR_QUEUE_DISABLE_ACCESS_TOKEN * extsvc/github: fix default value for single-program deploy type * single-binary: stop relying on a local ctags image * single-binary: use unix sockets for postgres * release App snapshots in CI when pushed to app/release-snapshot branch * internal/service/svcmain: update TODO comment * executor: correct DEPLOY_TYPE check * dev/check: allow single-binary to import dbconn * executor: remove accidental reliance on dbconn package * executor: improve error logging when running commands (#46546) * executor: improve error logging when running commands * executor: do not attempt std config validation running e.g. install cmd * executor: do not pull in the conf package / frontend reliance * ci: executors: correct site config for passwordless auth * server: fix bug where github-proxy would try to be a conf server * CI: executors: fix integration test passwordless auth * executors: allow passwordless auth in sourcegraph/server for testing * repo-updater: fix enterprise init (caused regression in repository syncing) Signed-off-by: Stephen Gutekanst <stephen@sourcegraph.com> Co-authored-by: Peter Guy <peter.guy@sourcegraph.com> Co-authored-by: Quinn Slack <quinn@slack.org>
2023-01-20 00:35:39 +00:00
checkBinary: .bin/sourcegraph-oss
env:
WEBPACK_DEV_SERVER: 1
watch:
- cmd
- internal
- schema
bazel: add command sets that uses bazel under the hood (#48460) This PRs introduces `bazelCommand`(s) in `sg.config.yaml` to handle starting those services, plus command sets for both oss and enterprise. `go run ./dev/sg start enterprise-bazel` will start most of the services with bazel and will keep using normal commands for the others (caddy, etc ...). The code powering those is built on top of `startedCmd` which I don't want to touch until we have a bit more understanding how all of this behaves. Once we have that, we can start refactoring this so it's more straightforward. Running the command set with bazel is bit verbose, you'll see two new "services" being ran: - `[ bazel]` which is ran at the very beginning and the equivalent of the old `install` part of the commands, except it does all of them at once. Once it's complete, it will stop exit nicely. - `[ iBazel]` which handles rebuilding the services in the background. The only thing it will do is to watch the FS based on the bazel rules and the depencency tree to understand what exactly need to be rebuilt. Once it detects some changes, it will overwrite the existing binary that was previously built by the first step (`[ bazel]`). The bazel commands will detect that the binary has been overwritten and will restart gracefully once it's done, meaning that the services will still be running during the new build. EDIT: added the codeintel command set for @efritz who's up to be our guinea pig on this ## Test plan <!-- All pull requests REQUIRE a test plan: https://docs.sourcegraph.com/dev/background-information/testing_principles --> Locally tested. --------- Co-authored-by: davejrt <davetry@gmail.com> Co-authored-by: William Bezuidenhout <william.bezuidenhout@sourcegraph.com>
2023-03-02 10:31:51 +00:00
bazelCommands:
oss-frontend:
target: //cmd/frontend
env:
CONFIGURATION_MODE: server
USE_ENHANCED_LANGUAGE_DETECTION: false
# frontend processes need this to be so that the paths to the assets are rendered correctly
WEBPACK_DEV_SERVER: 1
oss-worker:
target: //cmd/worker
oss-repo-updater:
target: //cmd/repo-updater
oss-symbols:
target: //cmd/symbols
env:
CTAGS_COMMAND: dev/universal-ctags-dev
CTAGS_PROCESSES: 2
oss-gitserver-0:
target: //cmd/gitserver
env:
<<: *gitserverenv
GITSERVER_EXTERNAL_ADDR: 127.0.0.1:3501
GITSERVER_ADDR: 127.0.0.1:3501
SRC_REPOS_DIR: $HOME/.sourcegraph/repos_1
SRC_PROF_HTTP: 127.0.0.1:3551
oss-gitserver-1:
target: //cmd/gitserver
env:
<<: *gitserverenv
GITSERVER_EXTERNAL_ADDR: 127.0.0.1:3501
GITSERVER_ADDR: 127.0.0.1:3501
SRC_REPOS_DIR: $HOME/.sourcegraph/repos_1
SRC_PROF_HTTP: 127.0.0.1:3551
searcher:
target: //cmd/searcher
github-proxy:
target: //cmd/github-proxy
frontend:
description: Enterprise frontend
target: //enterprise/cmd/frontend
precmd: |
export SOURCEGRAPH_LICENSE_GENERATION_KEY=$(cat ../dev-private/enterprise/dev/test-license-generation-key.pem)
# If EXTSVC_CONFIG_FILE is *unset*, set a default.
export EXTSVC_CONFIG_FILE=${EXTSVC_CONFIG_FILE-'../dev-private/enterprise/dev/external-services-config.json'}
env:
CONFIGURATION_MODE: server
USE_ENHANCED_LANGUAGE_DETECTION: false
ENTERPRISE: 1
SITE_CONFIG_FILE: '../dev-private/enterprise/dev/site-config.json'
SITE_CONFIG_ESCAPE_HATCH_PATH: '$HOME/.sourcegraph/site-config.json'
# frontend processes need this to be so that the paths to the assets are rendered correctly
WEBPACK_DEV_SERVER: 1
worker:
target: //enterprise/cmd/worker
precmd: |
export SOURCEGRAPH_LICENSE_GENERATION_KEY=$(cat ../dev-private/enterprise/dev/test-license-generation-key.pem)
repo-updater:
target: //enterprise/cmd/repo-updater
precmd: |
export SOURCEGRAPH_LICENSE_GENERATION_KEY=$(cat ../dev-private/enterprise/dev/test-license-generation-key.pem)
env:
ENTERPRISE: 1
symbols:
# TODO build ctags thing
target: //enterprise/cmd/symbols
checkBinary: .bin/symbols
env:
CTAGS_COMMAND: dev/universal-ctags-dev
CTAGS_PROCESSES: 2
USE_ROCKSKIP: 'false'
gitserver-template: &gitserver_bazel_template
target: //enterprise/cmd/gitserver
env: &gitserverenv
HOSTNAME: 127.0.0.1:3178
# This is only here to stay backwards-compatible with people's custom
# `sg.config.overwrite.yaml` files
gitserver:
<<: *gitserver_bazel_template
gitserver-0:
<<: *gitserver_bazel_template
env:
<<: *gitserverenv
GITSERVER_EXTERNAL_ADDR: 127.0.0.1:3501
GITSERVER_ADDR: 127.0.0.1:3501
SRC_REPOS_DIR: $HOME/.sourcegraph/repos_1
SRC_PROF_HTTP: 127.0.0.1:3551
gitserver-1:
<<: *gitserver_bazel_template
env:
<<: *gitserverenv
GITSERVER_EXTERNAL_ADDR: 127.0.0.1:3502
GITSERVER_ADDR: 127.0.0.1:3502
SRC_REPOS_DIR: $HOME/.sourcegraph/repos_2
SRC_PROF_HTTP: 127.0.0.1:3552
codeintel-worker:
precmd: |
export SOURCEGRAPH_LICENSE_GENERATION_KEY=$(cat ../dev-private/enterprise/dev/test-license-generation-key.pem)
target: //enterprise/cmd/precise-code-intel-worker
executor-template: &executor_template_bazel
target: //enterprise/cmd/executor
env:
EXECUTOR_QUEUE_NAME: TEMPLATE
TMPDIR: $HOME/.sourcegraph/executor-temp
Fix add host gateway behavior for app (#50020) We don't only need to add this when the executor frontend URL points to it, but also if the site config is set to host.docker.internal, hence I made this an executor config flag instead. I think this should fix it, but `sg start app` is acting up for me so would appreciate if someone could verify that. Closes https://github.com/sourcegraph/sourcegraph/issues/50011
2023-03-28 21:18:56 +00:00
# Required for frontend and executor to communicate
EXECUTOR_FRONTEND_URL: http://localhost:3080
# Must match the secret defined in the site config.
EXECUTOR_FRONTEND_PASSWORD: hunter2hunter2hunter2
# Disable firecracker inside executor in dev
EXECUTOR_USE_FIRECRACKER: false
bazel: add command sets that uses bazel under the hood (#48460) This PRs introduces `bazelCommand`(s) in `sg.config.yaml` to handle starting those services, plus command sets for both oss and enterprise. `go run ./dev/sg start enterprise-bazel` will start most of the services with bazel and will keep using normal commands for the others (caddy, etc ...). The code powering those is built on top of `startedCmd` which I don't want to touch until we have a bit more understanding how all of this behaves. Once we have that, we can start refactoring this so it's more straightforward. Running the command set with bazel is bit verbose, you'll see two new "services" being ran: - `[ bazel]` which is ran at the very beginning and the equivalent of the old `install` part of the commands, except it does all of them at once. Once it's complete, it will stop exit nicely. - `[ iBazel]` which handles rebuilding the services in the background. The only thing it will do is to watch the FS based on the bazel rules and the depencency tree to understand what exactly need to be rebuilt. Once it detects some changes, it will overwrite the existing binary that was previously built by the first step (`[ bazel]`). The bazel commands will detect that the binary has been overwritten and will restart gracefully once it's done, meaning that the services will still be running during the new build. EDIT: added the codeintel command set for @efritz who's up to be our guinea pig on this ## Test plan <!-- All pull requests REQUIRE a test plan: https://docs.sourcegraph.com/dev/background-information/testing_principles --> Locally tested. --------- Co-authored-by: davejrt <davetry@gmail.com> Co-authored-by: William Bezuidenhout <william.bezuidenhout@sourcegraph.com>
2023-03-02 10:31:51 +00:00
codeintel-executor:
<<: *executor_template_bazel
env:
EXECUTOR_QUEUE_NAME: codeintel
TMPDIR: $HOME/.sourcegraph/indexer-temp
#
# CommandSets ################################################################
#
sg: Fix capitalization of defaultCommandset (#25598)
2021-10-01 13:36:24 +00:00
defaultCommandset: enterprise
Move sg.config.yaml to repo root & allow private overwrites (#19487) This moves the sg.config.yaml to the repository root so that one can just use `sg start` and `sg run gitserver` etc. without having to specify the `-conf` flag. It also adds support for a local `.gitignore`d `sg.config.overwrites.yaml` file that users can add to the repository. If it exists, it will then get merged with the basic configuration when `sg` starts. What's the usecase? Custom command sets, custom env vars, etc. that you don't want to commit to the repository.
2021-03-29 08:14:53 +00:00
commandsets:
bazel: add command sets that uses bazel under the hood (#48460) This PRs introduces `bazelCommand`(s) in `sg.config.yaml` to handle starting those services, plus command sets for both oss and enterprise. `go run ./dev/sg start enterprise-bazel` will start most of the services with bazel and will keep using normal commands for the others (caddy, etc ...). The code powering those is built on top of `startedCmd` which I don't want to touch until we have a bit more understanding how all of this behaves. Once we have that, we can start refactoring this so it's more straightforward. Running the command set with bazel is bit verbose, you'll see two new "services" being ran: - `[ bazel]` which is ran at the very beginning and the equivalent of the old `install` part of the commands, except it does all of them at once. Once it's complete, it will stop exit nicely. - `[ iBazel]` which handles rebuilding the services in the background. The only thing it will do is to watch the FS based on the bazel rules and the depencency tree to understand what exactly need to be rebuilt. Once it detects some changes, it will overwrite the existing binary that was previously built by the first step (`[ bazel]`). The bazel commands will detect that the binary has been overwritten and will restart gracefully once it's done, meaning that the services will still be running during the new build. EDIT: added the codeintel command set for @efritz who's up to be our guinea pig on this ## Test plan <!-- All pull requests REQUIRE a test plan: https://docs.sourcegraph.com/dev/background-information/testing_principles --> Locally tested. --------- Co-authored-by: davejrt <davetry@gmail.com> Co-authored-by: William Bezuidenhout <william.bezuidenhout@sourcegraph.com>
2023-03-02 10:31:51 +00:00
oss-bazel:
# open-source version doesn't require the dev-private repository
requiresDevPrivate: false
checks:
- docker
- redis
- postgres
- git
bazel: fix checks in sg.config (#49491) `sg.config.yaml` was referring to a check that doesn't exist anymore - it should use the `bazelisk` check instead ## Test plan `sg start oss-bazel` started without the error `check 'bazel' not found` <!-- All pull requests REQUIRE a test plan: https://docs.sourcegraph.com/dev/background-information/testing_principles -->
2023-03-16 09:54:01 +00:00
- bazelisk
sg: bazel command sets block until bazel build is complete before anything else. (#48620) Early feedback we got was that when using the bazel commandsets, if the user never ran bazel build before, the logs would spam errors because normal services would come up way earlier and complained about not being able to reach other services. While is never happens if you have only a few things to build, the initial build command on a cold environment is really long. This fixes this. In essence, it behaves like the old install method for those services. ## Test plan <!-- All pull requests REQUIRE a test plan: https://docs.sourcegraph.com/dev/background-information/testing_principles --> Tested locally.
2023-03-03 13:58:50 +00:00
- ibazel
bazel: add command sets that uses bazel under the hood (#48460) This PRs introduces `bazelCommand`(s) in `sg.config.yaml` to handle starting those services, plus command sets for both oss and enterprise. `go run ./dev/sg start enterprise-bazel` will start most of the services with bazel and will keep using normal commands for the others (caddy, etc ...). The code powering those is built on top of `startedCmd` which I don't want to touch until we have a bit more understanding how all of this behaves. Once we have that, we can start refactoring this so it's more straightforward. Running the command set with bazel is bit verbose, you'll see two new "services" being ran: - `[ bazel]` which is ran at the very beginning and the equivalent of the old `install` part of the commands, except it does all of them at once. Once it's complete, it will stop exit nicely. - `[ iBazel]` which handles rebuilding the services in the background. The only thing it will do is to watch the FS based on the bazel rules and the depencency tree to understand what exactly need to be rebuilt. Once it detects some changes, it will overwrite the existing binary that was previously built by the first step (`[ bazel]`). The bazel commands will detect that the binary has been overwritten and will restart gracefully once it's done, meaning that the services will still be running during the new build. EDIT: added the codeintel command set for @efritz who's up to be our guinea pig on this ## Test plan <!-- All pull requests REQUIRE a test plan: https://docs.sourcegraph.com/dev/background-information/testing_principles --> Locally tested. --------- Co-authored-by: davejrt <davetry@gmail.com> Co-authored-by: William Bezuidenhout <william.bezuidenhout@sourcegraph.com>
2023-03-02 10:31:51 +00:00
bazelCommands:
- oss-frontend
- oss-worker
- oss-repo-updater
- oss-symbols
- oss-gitserver-0
- oss-gitserver-1
- github-proxy
- searcher
commands:
- oss-web
- caddy
- syntax-highlighter
- zoekt-index-0
- zoekt-index-1
- zoekt-web-0
- zoekt-web-1
- docsite
sg: Make `sg start` boot up enterprise environment (#22795) The people on Slack [have spoken](https://sourcegraph.slack.com/archives/C01N83PS4TU/p1626167506183100) and we all want the same thing: `sg start` should boot up the enterprise environment. `sg run-set oss` will boot up the open-source environment.
2021-07-13 16:13:18 +00:00
oss:
sg: check dev-private exists in 'sg start' (#25514) First attempt at https://github.com/sourcegraph/sourcegraph/issues/25501. I think this is fine for now and can avoid new colleagues running into this problem. It also got me thinking: I could've defined this as "check" in `sg.config.yaml`. But then I realised that in order to get a readable error message out of that I'd have to do some bash scripting. And that in turn reinforces my thought that the `checks:` should probably be Go functions. Thoughts?
2021-09-29 14:22:16 +00:00
# open-source version doesn't require the dev-private repository
requiresDevPrivate: false
sg: add ability to define `checks` per `commandset` (#23020) This allows us to check whether services are running before we boot up the environment. This should reduce the number of "why is service X failing?". At least that's the hope. The changes to the config file format are backwards compatible. See the `UnmarshalYAML` method on `Commandset`. It also updates the checks to make sure that they work for users of the docker-compose setup.
2021-07-21 15:37:24 +00:00
checks:
- docker
- redis
- postgres
sg: un-revert redesigned checks functionality after fixing wrong checks name and err handler (#31124) This restores #31055 after the revert in #31102. See a0c21370a635635cbd82df6f12fc9e5ee053969d for what the problem was: in the original code I managed to mess up an `if err != nil { return err }` and a checkname in `sg.config.yaml` was wrong.
2022-02-12 18:36:28 +00:00
- git
sg: add ability to define `checks` per `commandset` (#23020) This allows us to check whether services are running before we boot up the environment. This should reduce the number of "why is service X failing?". At least that's the hope. The changes to the config file format are backwards compatible. See the `UnmarshalYAML` method on `Commandset`. It also updates the checks to make sure that they work for users of the docker-compose setup.
2021-07-21 15:37:24 +00:00
commands:
sg: shorten prefix for commands name (#37437)
2022-06-20 17:57:54 +00:00
- oss-frontend
- oss-worker
- oss-repo-updater
- oss-symbols
authz: Move sub-repo perms to enterprise (#45659)
2022-12-28 08:44:07 +00:00
- oss-gitserver-0
- oss-gitserver-1
sg: add ability to define `checks` per `commandset` (#23020) This allows us to check whether services are running before we boot up the environment. This should reduce the number of "why is service X failing?". At least that's the hope. The changes to the config file format are backwards compatible. See the `UnmarshalYAML` method on `Commandset`. It also updates the checks to make sure that they work for users of the docker-compose setup.
2021-07-21 15:37:24 +00:00
- searcher
Blob keyboard navigation: Only load when code intelligence is available (#43458)
2022-10-26 15:01:44 +00:00
- oss-web
sg: add ability to define `checks` per `commandset` (#23020) This allows us to check whether services are running before we boot up the environment. This should reduce the number of "why is service X failing?". At least that's the hope. The changes to the config file format are backwards compatible. See the `UnmarshalYAML` method on `Commandset`. It also updates the checks to make sure that they work for users of the docker-compose setup.
2021-07-21 15:37:24 +00:00
- caddy
- docsite
sg: rename syntect_server to syntax-highlighter in dev (#25402)
2021-09-28 04:18:12 +00:00
- syntax-highlighter
sg: add ability to define `checks` per `commandset` (#23020) This allows us to check whether services are running before we boot up the environment. This should reduce the number of "why is service X failing?". At least that's the hope. The changes to the config file format are backwards compatible. See the `UnmarshalYAML` method on `Commandset`. It also updates the checks to make sure that they work for users of the docker-compose setup.
2021-07-21 15:37:24 +00:00
- github-proxy
sg: shorten prefix for commands name (#37437)
2022-06-20 17:57:54 +00:00
- zoekt-index-0
- zoekt-index-1
- zoekt-web-0
- zoekt-web-1
Move sg.config.yaml to repo root & allow private overwrites (#19487) This moves the sg.config.yaml to the repository root so that one can just use `sg start` and `sg run gitserver` etc. without having to specify the `-conf` flag. It also adds support for a local `.gitignore`d `sg.config.overwrites.yaml` file that users can add to the repository. If it exists, it will then get merged with the basic configuration when `sg` starts. What's the usecase? Custom command sets, custom env vars, etc. that you don't want to commit to the repository.
2021-03-29 08:14:53 +00:00
bazel: add command sets that uses bazel under the hood (#48460) This PRs introduces `bazelCommand`(s) in `sg.config.yaml` to handle starting those services, plus command sets for both oss and enterprise. `go run ./dev/sg start enterprise-bazel` will start most of the services with bazel and will keep using normal commands for the others (caddy, etc ...). The code powering those is built on top of `startedCmd` which I don't want to touch until we have a bit more understanding how all of this behaves. Once we have that, we can start refactoring this so it's more straightforward. Running the command set with bazel is bit verbose, you'll see two new "services" being ran: - `[ bazel]` which is ran at the very beginning and the equivalent of the old `install` part of the commands, except it does all of them at once. Once it's complete, it will stop exit nicely. - `[ iBazel]` which handles rebuilding the services in the background. The only thing it will do is to watch the FS based on the bazel rules and the depencency tree to understand what exactly need to be rebuilt. Once it detects some changes, it will overwrite the existing binary that was previously built by the first step (`[ bazel]`). The bazel commands will detect that the binary has been overwritten and will restart gracefully once it's done, meaning that the services will still be running during the new build. EDIT: added the codeintel command set for @efritz who's up to be our guinea pig on this ## Test plan <!-- All pull requests REQUIRE a test plan: https://docs.sourcegraph.com/dev/background-information/testing_principles --> Locally tested. --------- Co-authored-by: davejrt <davetry@gmail.com> Co-authored-by: William Bezuidenhout <william.bezuidenhout@sourcegraph.com>
2023-03-02 10:31:51 +00:00
enterprise-bazel: &enterprise_bazel_set
requiresDevPrivate: true
checks:
- docker
- redis
- postgres
- git
bazel: fix checks in sg.config (#49491) `sg.config.yaml` was referring to a check that doesn't exist anymore - it should use the `bazelisk` check instead ## Test plan `sg start oss-bazel` started without the error `check 'bazel' not found` <!-- All pull requests REQUIRE a test plan: https://docs.sourcegraph.com/dev/background-information/testing_principles -->
2023-03-16 09:54:01 +00:00
- bazelisk
sg: bazel command sets block until bazel build is complete before anything else. (#48620) Early feedback we got was that when using the bazel commandsets, if the user never ran bazel build before, the logs would spam errors because normal services would come up way earlier and complained about not being able to reach other services. While is never happens if you have only a few things to build, the initial build command on a cold environment is really long. This fixes this. In essence, it behaves like the old install method for those services. ## Test plan <!-- All pull requests REQUIRE a test plan: https://docs.sourcegraph.com/dev/background-information/testing_principles --> Tested locally.
2023-03-03 13:58:50 +00:00
- ibazel
bazel: add command sets that uses bazel under the hood (#48460) This PRs introduces `bazelCommand`(s) in `sg.config.yaml` to handle starting those services, plus command sets for both oss and enterprise. `go run ./dev/sg start enterprise-bazel` will start most of the services with bazel and will keep using normal commands for the others (caddy, etc ...). The code powering those is built on top of `startedCmd` which I don't want to touch until we have a bit more understanding how all of this behaves. Once we have that, we can start refactoring this so it's more straightforward. Running the command set with bazel is bit verbose, you'll see two new "services" being ran: - `[ bazel]` which is ran at the very beginning and the equivalent of the old `install` part of the commands, except it does all of them at once. Once it's complete, it will stop exit nicely. - `[ iBazel]` which handles rebuilding the services in the background. The only thing it will do is to watch the FS based on the bazel rules and the depencency tree to understand what exactly need to be rebuilt. Once it detects some changes, it will overwrite the existing binary that was previously built by the first step (`[ bazel]`). The bazel commands will detect that the binary has been overwritten and will restart gracefully once it's done, meaning that the services will still be running during the new build. EDIT: added the codeintel command set for @efritz who's up to be our guinea pig on this ## Test plan <!-- All pull requests REQUIRE a test plan: https://docs.sourcegraph.com/dev/background-information/testing_principles --> Locally tested. --------- Co-authored-by: davejrt <davetry@gmail.com> Co-authored-by: William Bezuidenhout <william.bezuidenhout@sourcegraph.com>
2023-03-02 10:31:51 +00:00
bazelCommands:
- frontend
- worker
- repo-updater
- gitserver-0
- gitserver-1
- searcher
- symbols
- github-proxy
commands:
- web
- blobstore
- docsite
- syntax-highlighter
- zoekt-index-0
- zoekt-index-1
- zoekt-web-0
- zoekt-web-1
- caddy
sg: Make `sg start` boot up enterprise environment (#22795) The people on Slack [have spoken](https://sourcegraph.slack.com/archives/C01N83PS4TU/p1626167506183100) and we all want the same thing: `sg start` should boot up the enterprise environment. `sg run-set oss` will boot up the open-source environment.
2021-07-13 16:13:18 +00:00
enterprise: &enterprise_set
sg: check dev-private exists in 'sg start' (#25514) First attempt at https://github.com/sourcegraph/sourcegraph/issues/25501. I think this is fine for now and can avoid new colleagues running into this problem. It also got me thinking: I could've defined this as "check" in `sg.config.yaml`. But then I realised that in order to get a readable error message out of that I'd have to do some bash scripting. And that in turn reinforces my thought that the `checks:` should probably be Go functions. Thoughts?
2021-09-29 14:22:16 +00:00
requiresDevPrivate: true
sg: add ability to define `checks` per `commandset` (#23020) This allows us to check whether services are running before we boot up the environment. This should reduce the number of "why is service X failing?". At least that's the hope. The changes to the config file format are backwards compatible. See the `UnmarshalYAML` method on `Commandset`. It also updates the checks to make sure that they work for users of the docker-compose setup.
2021-07-21 15:37:24 +00:00
checks:
- docker
- redis
- postgres
sg: un-revert redesigned checks functionality after fixing wrong checks name and err handler (#31124) This restores #31055 after the revert in #31102. See a0c21370a635635cbd82df6f12fc9e5ee053969d for what the problem was: in the original code I managed to mess up an `if err != nil { return err }` and a checkname in `sg.config.yaml` was wrong.
2022-02-12 18:36:28 +00:00
- git
sg: add ability to define `checks` per `commandset` (#23020) This allows us to check whether services are running before we boot up the environment. This should reduce the number of "why is service X failing?". At least that's the hope. The changes to the config file format are backwards compatible. See the `UnmarshalYAML` method on `Commandset`. It also updates the checks to make sure that they work for users of the docker-compose setup.
2021-07-21 15:37:24 +00:00
commands:
sg: shorten prefix for commands name (#37437)
2022-06-20 17:57:54 +00:00
- frontend
- worker
- repo-updater
- web
sg config: use two gitserver instances by default (#42749) We talked about this in repo-management & IAM and we think we should increase our dogfood consumption. This is one step in that direction: most of our customers run mulitple gitserver instances. This helps us make sure we dogfood that. ATTENTION: this will lead to some repositories being recloned/reindexed locally.
2022-10-10 13:26:58 +00:00
- gitserver-0
- gitserver-1
sg: add ability to define `checks` per `commandset` (#23020) This allows us to check whether services are running before we boot up the environment. This should reduce the number of "why is service X failing?". At least that's the hope. The changes to the config file format are backwards compatible. See the `UnmarshalYAML` method on `Commandset`. It also updates the checks to make sure that they work for users of the docker-compose setup.
2021-07-21 15:37:24 +00:00
- searcher
sg: shorten prefix for commands name (#37437)
2022-06-20 17:57:54 +00:00
- symbols
sg: add ability to define `checks` per `commandset` (#23020) This allows us to check whether services are running before we boot up the environment. This should reduce the number of "why is service X failing?". At least that's the hope. The changes to the config file format are backwards compatible. See the `UnmarshalYAML` method on `Commandset`. It also updates the checks to make sure that they work for users of the docker-compose setup.
2021-07-21 15:37:24 +00:00
- caddy
- docsite
sg: rename syntect_server to syntax-highlighter in dev (#25402)
2021-09-28 04:18:12 +00:00
- syntax-highlighter
sg: add ability to define `checks` per `commandset` (#23020) This allows us to check whether services are running before we boot up the environment. This should reduce the number of "why is service X failing?". At least that's the hope. The changes to the config file format are backwards compatible. See the `UnmarshalYAML` method on `Commandset`. It also updates the checks to make sure that they work for users of the docker-compose setup.
2021-07-21 15:37:24 +00:00
- github-proxy
sg: shorten prefix for commands name (#37437)
2022-06-20 17:57:54 +00:00
- zoekt-index-0
- zoekt-index-1
- zoekt-web-0
- zoekt-web-1
sg: start blobstore whenever worker is started (#46326)
2023-01-11 13:51:36 +00:00
- blobstore
Move sg.config.yaml to repo root & allow private overwrites (#19487) This moves the sg.config.yaml to the repository root so that one can just use `sg start` and `sg run gitserver` etc. without having to specify the `-conf` flag. It also adds support for a local `.gitignore`d `sg.config.overwrites.yaml` file that users can add to the repository. If it exists, it will then get merged with the basic configuration when `sg` starts. What's the usecase? Custom command sets, custom env vars, etc. that you don't want to commit to the repository.
2021-03-29 08:14:53 +00:00
sg: create a test command to run e2e tests locally (#34627)
2022-05-04 07:12:30 +00:00
enterprise-e2e:
<<: *enterprise_set
env:
dev/sg: unify ExternalSecrets with secrets store, minor fixes for frontend-e2e (#34942)
2022-05-05 16:17:18 +00:00
# EXTSVC_CONFIG_FILE being set prevents the e2e test suite to add
sg: create a test command to run e2e tests locally (#34627)
2022-05-04 07:12:30 +00:00
# additional connections.
web: introduce `INTEGRATION_TESTS` env variable (#38805)
2022-07-14 13:05:49 +00:00
EXTSVC_CONFIG_FILE: ''
sg: create a test command to run e2e tests locally (#34627)
2022-05-04 07:12:30 +00:00
Add dotcom commandset & allow setting env per commandset (#24688) This allows us to use `sg start dotcom` to run Sourcegraph in dot-com mode. It also allows us to get rid of custom env overwrites in `sg.config.overwrite.yaml`.
2021-09-08 08:41:06 +00:00
dotcom:
<<: *enterprise_set
env:
SOURCEGRAPHDOTCOM_MODE: true
bazel: add command sets that uses bazel under the hood (#48460) This PRs introduces `bazelCommand`(s) in `sg.config.yaml` to handle starting those services, plus command sets for both oss and enterprise. `go run ./dev/sg start enterprise-bazel` will start most of the services with bazel and will keep using normal commands for the others (caddy, etc ...). The code powering those is built on top of `startedCmd` which I don't want to touch until we have a bit more understanding how all of this behaves. Once we have that, we can start refactoring this so it's more straightforward. Running the command set with bazel is bit verbose, you'll see two new "services" being ran: - `[ bazel]` which is ran at the very beginning and the equivalent of the old `install` part of the commands, except it does all of them at once. Once it's complete, it will stop exit nicely. - `[ iBazel]` which handles rebuilding the services in the background. The only thing it will do is to watch the FS based on the bazel rules and the depencency tree to understand what exactly need to be rebuilt. Once it detects some changes, it will overwrite the existing binary that was previously built by the first step (`[ bazel]`). The bazel commands will detect that the binary has been overwritten and will restart gracefully once it's done, meaning that the services will still be running during the new build. EDIT: added the codeintel command set for @efritz who's up to be our guinea pig on this ## Test plan <!-- All pull requests REQUIRE a test plan: https://docs.sourcegraph.com/dev/background-information/testing_principles --> Locally tested. --------- Co-authored-by: davejrt <davetry@gmail.com> Co-authored-by: William Bezuidenhout <william.bezuidenhout@sourcegraph.com>
2023-03-02 10:31:51 +00:00
codeintel-bazel: &codeintel_bazel_set
requiresDevPrivate: true
checks:
- docker
- redis
- postgres
- git
bazel: fix checks in sg.config (#49491) `sg.config.yaml` was referring to a check that doesn't exist anymore - it should use the `bazelisk` check instead ## Test plan `sg start oss-bazel` started without the error `check 'bazel' not found` <!-- All pull requests REQUIRE a test plan: https://docs.sourcegraph.com/dev/background-information/testing_principles -->
2023-03-16 09:54:01 +00:00
- bazelisk
sg: bazel command sets block until bazel build is complete before anything else. (#48620) Early feedback we got was that when using the bazel commandsets, if the user never ran bazel build before, the logs would spam errors because normal services would come up way earlier and complained about not being able to reach other services. While is never happens if you have only a few things to build, the initial build command on a cold environment is really long. This fixes this. In essence, it behaves like the old install method for those services. ## Test plan <!-- All pull requests REQUIRE a test plan: https://docs.sourcegraph.com/dev/background-information/testing_principles --> Tested locally.
2023-03-03 13:58:50 +00:00
- ibazel
bazel: add command sets that uses bazel under the hood (#48460) This PRs introduces `bazelCommand`(s) in `sg.config.yaml` to handle starting those services, plus command sets for both oss and enterprise. `go run ./dev/sg start enterprise-bazel` will start most of the services with bazel and will keep using normal commands for the others (caddy, etc ...). The code powering those is built on top of `startedCmd` which I don't want to touch until we have a bit more understanding how all of this behaves. Once we have that, we can start refactoring this so it's more straightforward. Running the command set with bazel is bit verbose, you'll see two new "services" being ran: - `[ bazel]` which is ran at the very beginning and the equivalent of the old `install` part of the commands, except it does all of them at once. Once it's complete, it will stop exit nicely. - `[ iBazel]` which handles rebuilding the services in the background. The only thing it will do is to watch the FS based on the bazel rules and the depencency tree to understand what exactly need to be rebuilt. Once it detects some changes, it will overwrite the existing binary that was previously built by the first step (`[ bazel]`). The bazel commands will detect that the binary has been overwritten and will restart gracefully once it's done, meaning that the services will still be running during the new build. EDIT: added the codeintel command set for @efritz who's up to be our guinea pig on this ## Test plan <!-- All pull requests REQUIRE a test plan: https://docs.sourcegraph.com/dev/background-information/testing_principles --> Locally tested. --------- Co-authored-by: davejrt <davetry@gmail.com> Co-authored-by: William Bezuidenhout <william.bezuidenhout@sourcegraph.com>
2023-03-02 10:31:51 +00:00
bazelCommands:
- frontend
- worker
- repo-updater
- gitserver-0
- gitserver-1
- searcher
- symbols
- github-proxy
- codeintel-worker
- codeintel-executor
commands:
- web
- blobstore
- docsite
- syntax-highlighter
- zoekt-index-0
- zoekt-index-1
- zoekt-web-0
- zoekt-web-1
- caddy
- jaeger
- grafana
- prometheus
sg: add 'codeintel' commandset and alias enterprise-codeintel to it (#32511)
2022-03-11 15:06:49 +00:00
codeintel: &codeintel_set
sg: check dev-private exists in 'sg start' (#25514) First attempt at https://github.com/sourcegraph/sourcegraph/issues/25501. I think this is fine for now and can avoid new colleagues running into this problem. It also got me thinking: I could've defined this as "check" in `sg.config.yaml`. But then I realised that in order to get a readable error message out of that I'd have to do some bash scripting. And that in turn reinforces my thought that the `checks:` should probably be Go functions. Thoughts?
2021-09-29 14:22:16 +00:00
requiresDevPrivate: true
sg: add ability to define `checks` per `commandset` (#23020) This allows us to check whether services are running before we boot up the environment. This should reduce the number of "why is service X failing?". At least that's the hope. The changes to the config file format are backwards compatible. See the `UnmarshalYAML` method on `Commandset`. It also updates the checks to make sure that they work for users of the docker-compose setup.
2021-07-21 15:37:24 +00:00
checks:
- docker
- redis
- postgres
sg: un-revert redesigned checks functionality after fixing wrong checks name and err handler (#31124) This restores #31055 after the revert in #31102. See a0c21370a635635cbd82df6f12fc9e5ee053969d for what the problem was: in the original code I managed to mess up an `if err != nil { return err }` and a checkname in `sg.config.yaml` was wrong.
2022-02-12 18:36:28 +00:00
- git
sg: add ability to define `checks` per `commandset` (#23020) This allows us to check whether services are running before we boot up the environment. This should reduce the number of "why is service X failing?". At least that's the hope. The changes to the config file format are backwards compatible. See the `UnmarshalYAML` method on `Commandset`. It also updates the checks to make sure that they work for users of the docker-compose setup.
2021-07-21 15:37:24 +00:00
commands:
sg: shorten prefix for commands name (#37437)
2022-06-20 17:57:54 +00:00
- frontend
- worker
- repo-updater
- web
sg config: use two gitserver instances by default (#42749) We talked about this in repo-management & IAM and we think we should increase our dogfood consumption. This is one step in that direction: most of our customers run mulitple gitserver instances. This helps us make sure we dogfood that. ATTENTION: this will lead to some repositories being recloned/reindexed locally.
2022-10-10 13:26:58 +00:00
- gitserver-0
- gitserver-1
sg: add ability to define `checks` per `commandset` (#23020) This allows us to check whether services are running before we boot up the environment. This should reduce the number of "why is service X failing?". At least that's the hope. The changes to the config file format are backwards compatible. See the `UnmarshalYAML` method on `Commandset`. It also updates the checks to make sure that they work for users of the docker-compose setup.
2021-07-21 15:37:24 +00:00
- searcher
sg: shorten prefix for commands name (#37437)
2022-06-20 17:57:54 +00:00
- symbols
sg: add ability to define `checks` per `commandset` (#23020) This allows us to check whether services are running before we boot up the environment. This should reduce the number of "why is service X failing?". At least that's the hope. The changes to the config file format are backwards compatible. See the `UnmarshalYAML` method on `Commandset`. It also updates the checks to make sure that they work for users of the docker-compose setup.
2021-07-21 15:37:24 +00:00
- caddy
- docsite
sg: rename syntect_server to syntax-highlighter in dev (#25402)
2021-09-28 04:18:12 +00:00
- syntax-highlighter
sg: add ability to define `checks` per `commandset` (#23020) This allows us to check whether services are running before we boot up the environment. This should reduce the number of "why is service X failing?". At least that's the hope. The changes to the config file format are backwards compatible. See the `UnmarshalYAML` method on `Commandset`. It also updates the checks to make sure that they work for users of the docker-compose setup.
2021-07-21 15:37:24 +00:00
- github-proxy
sg: shorten prefix for commands name (#37437)
2022-06-20 17:57:54 +00:00
- zoekt-index-0
- zoekt-index-1
- zoekt-web-0
- zoekt-web-1
all: introduce sourcegraph/blobstore; add option to cmd/server (#44255) * all: add new sg blobstore backend to supersede minio * cmd/server: optionally run sg blobstore backend over minio * lsifuploadstore: add minio back to help text * sg generate * uploadstore: add DeleteObjects / NewListObjectsV2Paginator to s3API interface * uploadstore: regenerate mocks * uploadstore: add gcsAPI.Objects listing API * uploadstore: regenerate mocks * uploadstore: add Store.ExpireObjects stub implementation (thanks eric) * uploadstore: implement ExpireObjects in GCS/S3/Lazy interfaces * worker: integrate lsifuploadstore expirer * uploadstore: update test API calls * uploadstore: add TODO for myself * uploadstore: batch S3 object deletions * uploadstore: use operation logger instead of log15 * cmd/server: use sglog * cmd/server: fix image build Signed-off-by: Stephen Gutekanst <stephen@sourcegraph.com>
2022-11-30 23:04:48 +00:00
- blobstore
sg: shorten prefix for commands name (#37437)
2022-06-20 17:57:54 +00:00
- codeintel-worker
sg: add ability to define `checks` per `commandset` (#23020) This allows us to check whether services are running before we boot up the environment. This should reduce the number of "why is service X failing?". At least that's the hope. The changes to the config file format are backwards compatible. See the `UnmarshalYAML` method on `Commandset`. It also updates the checks to make sure that they work for users of the docker-compose setup.
2021-07-21 15:37:24 +00:00
- codeintel-executor
codeintel: Tag content type on precise code intel uploads (#45187)
2022-12-05 18:10:39 +00:00
# - otel-collector
symbols: Bring this baby into 2021 (#27986) Co-authored-by: Noah Santschi-Cooney <noah@santschi-cooney.ch>
2021-11-30 17:56:45 +00:00
- jaeger
- grafana
- prometheus
sg: hacking hour with lots of YAML (#21250) This PR is a collection of all the things we did in the `sg` hacking hour just now: - Clean up the README so that the TODOs are up to date - Add definitions for `minio`, `codeinsights-db`, `executor-queue`, `precise-code-intel-worker` and `codeintel-executor` - Make it possible to launch Sourcegraph with codeinsights!! See the `sg.config.yaml` for which env vars to set and then run `sg run-set enterprise-codeinsights`
2021-05-21 16:01:11 +00:00
sg: add 'codeintel' commandset and alias enterprise-codeintel to it (#32511)
2022-03-11 15:06:49 +00:00
enterprise-codeintel:
<<: *codeintel_set
bazel: add command sets that uses bazel under the hood (#48460) This PRs introduces `bazelCommand`(s) in `sg.config.yaml` to handle starting those services, plus command sets for both oss and enterprise. `go run ./dev/sg start enterprise-bazel` will start most of the services with bazel and will keep using normal commands for the others (caddy, etc ...). The code powering those is built on top of `startedCmd` which I don't want to touch until we have a bit more understanding how all of this behaves. Once we have that, we can start refactoring this so it's more straightforward. Running the command set with bazel is bit verbose, you'll see two new "services" being ran: - `[ bazel]` which is ran at the very beginning and the equivalent of the old `install` part of the commands, except it does all of them at once. Once it's complete, it will stop exit nicely. - `[ iBazel]` which handles rebuilding the services in the background. The only thing it will do is to watch the FS based on the bazel rules and the depencency tree to understand what exactly need to be rebuilt. Once it detects some changes, it will overwrite the existing binary that was previously built by the first step (`[ bazel]`). The bazel commands will detect that the binary has been overwritten and will restart gracefully once it's done, meaning that the services will still be running during the new build. EDIT: added the codeintel command set for @efritz who's up to be our guinea pig on this ## Test plan <!-- All pull requests REQUIRE a test plan: https://docs.sourcegraph.com/dev/background-information/testing_principles --> Locally tested. --------- Co-authored-by: davejrt <davetry@gmail.com> Co-authored-by: William Bezuidenhout <william.bezuidenhout@sourcegraph.com>
2023-03-02 10:31:51 +00:00
enterprise-codeintel-bazel:
<<: *codeintel_bazel_set
sg: add 'codeintel' commandset and alias enterprise-codeintel to it (#32511)
2022-03-11 15:06:49 +00:00
sg: hacking hour with lots of YAML (#21250) This PR is a collection of all the things we did in the `sg` hacking hour just now: - Clean up the README so that the TODOs are up to date - Add definitions for `minio`, `codeinsights-db`, `executor-queue`, `precise-code-intel-worker` and `codeintel-executor` - Make it possible to launch Sourcegraph with codeinsights!! See the `sg.config.yaml` for which env vars to set and then run `sg run-set enterprise-codeinsights`
2021-05-21 16:01:11 +00:00
enterprise-codeinsights:
sg: check dev-private exists in 'sg start' (#25514) First attempt at https://github.com/sourcegraph/sourcegraph/issues/25501. I think this is fine for now and can avoid new colleagues running into this problem. It also got me thinking: I could've defined this as "check" in `sg.config.yaml`. But then I realised that in order to get a readable error message out of that I'd have to do some bash scripting. And that in turn reinforces my thought that the `checks:` should probably be Go functions. Thoughts?
2021-09-29 14:22:16 +00:00
requiresDevPrivate: true
sg: add ability to define `checks` per `commandset` (#23020) This allows us to check whether services are running before we boot up the environment. This should reduce the number of "why is service X failing?". At least that's the hope. The changes to the config file format are backwards compatible. See the `UnmarshalYAML` method on `Commandset`. It also updates the checks to make sure that they work for users of the docker-compose setup.
2021-07-21 15:37:24 +00:00
checks:
- docker
- redis
- postgres
sg: un-revert redesigned checks functionality after fixing wrong checks name and err handler (#31124) This restores #31055 after the revert in #31102. See a0c21370a635635cbd82df6f12fc9e5ee053969d for what the problem was: in the original code I managed to mess up an `if err != nil { return err }` and a checkname in `sg.config.yaml` was wrong.
2022-02-12 18:36:28 +00:00
- git
sg: add ability to define `checks` per `commandset` (#23020) This allows us to check whether services are running before we boot up the environment. This should reduce the number of "why is service X failing?". At least that's the hope. The changes to the config file format are backwards compatible. See the `UnmarshalYAML` method on `Commandset`. It also updates the checks to make sure that they work for users of the docker-compose setup.
2021-07-21 15:37:24 +00:00
commands:
sg: shorten prefix for commands name (#37437)
2022-06-20 17:57:54 +00:00
- frontend
- worker
- repo-updater
- web
sg config: use two gitserver instances by default (#42749) We talked about this in repo-management & IAM and we think we should increase our dogfood consumption. This is one step in that direction: most of our customers run mulitple gitserver instances. This helps us make sure we dogfood that. ATTENTION: this will lead to some repositories being recloned/reindexed locally.
2022-10-10 13:26:58 +00:00
- gitserver-0
- gitserver-1
sg: add ability to define `checks` per `commandset` (#23020) This allows us to check whether services are running before we boot up the environment. This should reduce the number of "why is service X failing?". At least that's the hope. The changes to the config file format are backwards compatible. See the `UnmarshalYAML` method on `Commandset`. It also updates the checks to make sure that they work for users of the docker-compose setup.
2021-07-21 15:37:24 +00:00
- searcher
- symbols
- caddy
- docsite
sg: rename syntect_server to syntax-highlighter in dev (#25402)
2021-09-28 04:18:12 +00:00
- syntax-highlighter
sg: add ability to define `checks` per `commandset` (#23020) This allows us to check whether services are running before we boot up the environment. This should reduce the number of "why is service X failing?". At least that's the hope. The changes to the config file format are backwards compatible. See the `UnmarshalYAML` method on `Commandset`. It also updates the checks to make sure that they work for users of the docker-compose setup.
2021-07-21 15:37:24 +00:00
- github-proxy
sg: shorten prefix for commands name (#37437)
2022-06-20 17:57:54 +00:00
- zoekt-index-0
- zoekt-index-1
- zoekt-web-0
- zoekt-web-1
sg: start blobstore whenever worker is started (#46326)
2023-01-11 13:51:36 +00:00
- blobstore
Add dotcom commandset & allow setting env per commandset (#24688) This allows us to use `sg start dotcom` to run Sourcegraph in dot-com mode. It also allows us to get rid of custom env overwrites in `sg.config.overwrite.yaml`.
2021-09-08 08:41:06 +00:00
env:
DISABLE_CODE_INSIGHTS_HISTORICAL: false
DISABLE_CODE_INSIGHTS: false
sg: hacking hour with lots of YAML (#21250) This PR is a collection of all the things we did in the `sg` hacking hour just now: - Clean up the README so that the TODOs are up to date - Add definitions for `minio`, `codeinsights-db`, `executor-queue`, `precise-code-intel-worker` and `codeintel-executor` - Make it possible to launch Sourcegraph with codeinsights!! See the `sg.config.yaml` for which env vars to set and then run `sg run-set enterprise-codeinsights`
2021-05-21 16:01:11 +00:00
sg: Add api-only commandset (#21325)
2021-05-25 08:33:48 +00:00
api-only:
sg: check dev-private exists in 'sg start' (#25514) First attempt at https://github.com/sourcegraph/sourcegraph/issues/25501. I think this is fine for now and can avoid new colleagues running into this problem. It also got me thinking: I could've defined this as "check" in `sg.config.yaml`. But then I realised that in order to get a readable error message out of that I'd have to do some bash scripting. And that in turn reinforces my thought that the `checks:` should probably be Go functions. Thoughts?
2021-09-29 14:22:16 +00:00
requiresDevPrivate: true
sg: add ability to define `checks` per `commandset` (#23020) This allows us to check whether services are running before we boot up the environment. This should reduce the number of "why is service X failing?". At least that's the hope. The changes to the config file format are backwards compatible. See the `UnmarshalYAML` method on `Commandset`. It also updates the checks to make sure that they work for users of the docker-compose setup.
2021-07-21 15:37:24 +00:00
checks:
- docker
- redis
- postgres
sg: un-revert redesigned checks functionality after fixing wrong checks name and err handler (#31124) This restores #31055 after the revert in #31102. See a0c21370a635635cbd82df6f12fc9e5ee053969d for what the problem was: in the original code I managed to mess up an `if err != nil { return err }` and a checkname in `sg.config.yaml` was wrong.
2022-02-12 18:36:28 +00:00
- git
sg: add ability to define `checks` per `commandset` (#23020) This allows us to check whether services are running before we boot up the environment. This should reduce the number of "why is service X failing?". At least that's the hope. The changes to the config file format are backwards compatible. See the `UnmarshalYAML` method on `Commandset`. It also updates the checks to make sure that they work for users of the docker-compose setup.
2021-07-21 15:37:24 +00:00
commands:
sg: shorten prefix for commands name (#37437)
2022-06-20 17:57:54 +00:00
- frontend
- worker
- repo-updater
sg config: use two gitserver instances by default (#42749) We talked about this in repo-management & IAM and we think we should increase our dogfood consumption. This is one step in that direction: most of our customers run mulitple gitserver instances. This helps us make sure we dogfood that. ATTENTION: this will lead to some repositories being recloned/reindexed locally.
2022-10-10 13:26:58 +00:00
- gitserver-0
- gitserver-1
sg: add ability to define `checks` per `commandset` (#23020) This allows us to check whether services are running before we boot up the environment. This should reduce the number of "why is service X failing?". At least that's the hope. The changes to the config file format are backwards compatible. See the `UnmarshalYAML` method on `Commandset`. It also updates the checks to make sure that they work for users of the docker-compose setup.
2021-07-21 15:37:24 +00:00
- searcher
- symbols
- github-proxy
sg: shorten prefix for commands name (#37437)
2022-06-20 17:57:54 +00:00
- zoekt-index-0
- zoekt-index-1
- zoekt-web-0
- zoekt-web-1
sg: start blobstore whenever worker is started (#46326)
2023-01-11 13:51:36 +00:00
- blobstore
sg: Add api-only commandset (#21325)
2021-05-25 08:33:48 +00:00
Add batches executor to dev env (#22578) * Add batch spec execution DB entity * Fixup migration sequence number * Add batches executor to dev env Co-authored-by: Erik Seliger <erikseliger@me.com>
2021-07-05 11:06:52 +00:00
batches:
sg: check dev-private exists in 'sg start' (#25514) First attempt at https://github.com/sourcegraph/sourcegraph/issues/25501. I think this is fine for now and can avoid new colleagues running into this problem. It also got me thinking: I could've defined this as "check" in `sg.config.yaml`. But then I realised that in order to get a readable error message out of that I'd have to do some bash scripting. And that in turn reinforces my thought that the `checks:` should probably be Go functions. Thoughts?
2021-09-29 14:22:16 +00:00
requiresDevPrivate: true
sg: add ability to define `checks` per `commandset` (#23020) This allows us to check whether services are running before we boot up the environment. This should reduce the number of "why is service X failing?". At least that's the hope. The changes to the config file format are backwards compatible. See the `UnmarshalYAML` method on `Commandset`. It also updates the checks to make sure that they work for users of the docker-compose setup.
2021-07-21 15:37:24 +00:00
checks:
- docker
- redis
- postgres
sg: un-revert redesigned checks functionality after fixing wrong checks name and err handler (#31124) This restores #31055 after the revert in #31102. See a0c21370a635635cbd82df6f12fc9e5ee053969d for what the problem was: in the original code I managed to mess up an `if err != nil { return err }` and a checkname in `sg.config.yaml` was wrong.
2022-02-12 18:36:28 +00:00
- git
sg: add ability to define `checks` per `commandset` (#23020) This allows us to check whether services are running before we boot up the environment. This should reduce the number of "why is service X failing?". At least that's the hope. The changes to the config file format are backwards compatible. See the `UnmarshalYAML` method on `Commandset`. It also updates the checks to make sure that they work for users of the docker-compose setup.
2021-07-21 15:37:24 +00:00
commands:
sg: shorten prefix for commands name (#37437)
2022-06-20 17:57:54 +00:00
- frontend
- worker
- repo-updater
- web
sg config: use two gitserver instances by default (#42749) We talked about this in repo-management & IAM and we think we should increase our dogfood consumption. This is one step in that direction: most of our customers run mulitple gitserver instances. This helps us make sure we dogfood that. ATTENTION: this will lead to some repositories being recloned/reindexed locally.
2022-10-10 13:26:58 +00:00
- gitserver-0
- gitserver-1
sg: add ability to define `checks` per `commandset` (#23020) This allows us to check whether services are running before we boot up the environment. This should reduce the number of "why is service X failing?". At least that's the hope. The changes to the config file format are backwards compatible. See the `UnmarshalYAML` method on `Commandset`. It also updates the checks to make sure that they work for users of the docker-compose setup.
2021-07-21 15:37:24 +00:00
- searcher
sg: shorten prefix for commands name (#37437)
2022-06-20 17:57:54 +00:00
- symbols
sg: add ability to define `checks` per `commandset` (#23020) This allows us to check whether services are running before we boot up the environment. This should reduce the number of "why is service X failing?". At least that's the hope. The changes to the config file format are backwards compatible. See the `UnmarshalYAML` method on `Commandset`. It also updates the checks to make sure that they work for users of the docker-compose setup.
2021-07-21 15:37:24 +00:00
- caddy
- docsite
sg: rename syntect_server to syntax-highlighter in dev (#25402)
2021-09-28 04:18:12 +00:00
- syntax-highlighter
sg: add ability to define `checks` per `commandset` (#23020) This allows us to check whether services are running before we boot up the environment. This should reduce the number of "why is service X failing?". At least that's the hope. The changes to the config file format are backwards compatible. See the `UnmarshalYAML` method on `Commandset`. It also updates the checks to make sure that they work for users of the docker-compose setup.
2021-07-21 15:37:24 +00:00
- github-proxy
sg: shorten prefix for commands name (#37437)
2022-06-20 17:57:54 +00:00
- zoekt-index-0
- zoekt-index-1
- zoekt-web-0
- zoekt-web-1
sg: start blobstore whenever worker is started (#46326)
2023-01-11 13:51:36 +00:00
- blobstore
sg: add ability to define `checks` per `commandset` (#23020) This allows us to check whether services are running before we boot up the environment. This should reduce the number of "why is service X failing?". At least that's the hope. The changes to the config file format are backwards compatible. See the `UnmarshalYAML` method on `Commandset`. It also updates the checks to make sure that they work for users of the docker-compose setup.
2021-07-21 15:37:24 +00:00
- batches-executor
Experiment: Natively run SSBC in docker (#44034) This adds an experimental code path that I will use to test a docker-only execution mode for server-side batch changes. This code path is never executed for customers until we make the switch when we deem it ready. This will allow me to dogfood this while it's not available to customer instances yet. Ultimately, the goal of this is to make executors simply be "the job runner platform through a generic interface". Today, this depends on src-cli to do a good bunch of the work. This is a blocker for going full docker-based with executors, which will ultimately be a requirement on the road to k8s-based executors. As this removes the dependency on src-cli, nothing but the job interface and API endpoints tie executor and Sourcegraph instance together. Ultimately, this will allow us to support larger version spans between the two (pending executors going GA and being feature-complete). Known issues/limitations: Steps skipped in between steps that run don't work yet Skipping steps dynamically is inefficient as we cannot tell the executor to skip a step IF X, so we replace the script by exit 0 It is unclear if all variants of file mounts still work. Basic cases do work. Files used to be read-only in src-cli, they aren't now, but content is still reset in between steps. The assumption that everything operates in /work is broken here, because we need to use what executors give us to persist out-of-repo state in between containers (like the step result from the previous step) It is unclear if workspace mounts work Cache keys are not correctly computed if using workspace mounts - the metadataretriever is nil We still use log outputs to transfer the AfterStepResults to the Sourcegraph instance, this should finally become an artifact instead. Then, we don't have to rely on the execution_log_entires anymore and can theoretically prune those after some time. This column is currently growing indefinitely. It depends on tee being available in the docker images to capture the cmd.stdout/cmd.stderr properly for template variable rendering Env-vars are not rendered in their evaluated form post-execution File permissions are unclear and might be similarly broken to how they are now - or even worse Disclaimer: It's not feature complete today! But it is also not hitting any default code paths either. As development on this goes on, we can eventually remove the feature flag and run the new job format on all instances. This PR handles fallback of rendering old records correctly in the UI already.
2022-11-09 23:20:43 +00:00
- batcheshelper-builder
Add batches executor to dev env (#22578) * Add batch spec execution DB entity * Fixup migration sequence number * Add batches executor to dev env Co-authored-by: Erik Seliger <erikseliger@me.com>
2021-07-05 11:06:52 +00:00
embeddings: searcher and indexer (#48017) # High-level architecture overview <img width="2231" alt="Screenshot 2023-02-24 at 15 13 59" src="https://user-images.githubusercontent.com/6417322/221200130-53c1ff25-4c47-4532-885f-5c4f9dadb05e.png"> # Embeddings Really quickly: embeddings are a semantic representation of text. Embeddings are usually floating-point vectors with 256+ elements. The neat thing about embeddings is that they allow us to search over textual information using a semantic correlation between the query and the text, not just syntactic (matching keywords). In this PR, we implemented an embedding service that will allow us to do semantic code search over repositories in Sourcegraph. So, for example, you'll be able to ask, "how do access tokens work in Sourcegraph", and it will give you a list of the closest matching code files. Additionally, we build a context detection service powered by embeddings. In chat applications, it is important to know whether the user's message requires additional context. We have to differentiate between two cases: the user asks a general question about the codebase, or the user references something in the existing conversation. In the latter case, including the context would ruin the flow of the conversation, and the chatbot would most likely return a confusing answer. We determine whether a query _does not_ require additional context using two approaches: 1. We check if the query contains well-known phrases that would indicate the user is referencing the existing conversation (e.g., translate previous, change that) 1. We have a static dataset of messages that require context and a dataset of messages that do not. We embed both datasets, and then, using embedding similarity, we can check which set is more similar to the query. ## GraphQL API We add four new resolvers to the GraphQL API: ```graphql extend type Query { embeddingsSearch(repo: ID!, query: String!, codeResultsCount: Int!, textResultsCount: Int!): EmbeddingsSearchResults! isContextRequiredForQuery(query: String!): Boolean! } extend type Mutation { scheduleRepositoriesForEmbedding(repoNames: [String!]!): EmptyResponse! scheduleContextDetectionForEmbedding: EmptyResponse! } ``` - `embeddingsSearch` performs embeddings search over the repo embeddings and returns the specified number of results - `isContextRequiredForQuery` determines whether the given query requires additional context - `scheduleRepositoriesForEmbedding` schedules a repo embedding background job - `scheduleContextDetectionForEmbedding` schedules a context detection embedding background job that embeds a static dataset of messages. ## Repo embedding background job Embedding a repository is implemented as a background job. The background job handler receives the repository and the revision, which should be embedded. Handler then gathers a list of files from the gitserver and excludes files >1MB in size. The list of files is split into code and text files (.md, .txt), and we build a separate embedding index for both. We split them because in a combined index, the text files always tended to feature as top results and didn't leave any room for code files. Once we have the list of files, the procedure is as follows: - For each file - Get file contents from gitserver - Check if the file is embeddable (is not autogenerated, is large enough, does not have long lines) - Split the file into embeddable chunks - Embed the file chunks using an external embedding service (defined in site config) - Add embedded file chunks and metadata to the index - Metadata contains the file name, the start line, and the end line of the chunk - Once all files are processed, the index is marshaled into JSON and stored in Cloud storage (GCS, S3) ### Site config changes As mentioned, we use a configurable external embedding API that does the actual text -> vector embedding part. Ideally, this allows us to swap embedding providers in the future. ```json "embeddings": { "description": "Configuration for embeddings service.", "type": "object", "required": ["enabled", "dimensions", "model", "accessToken", "url"], "properties": { "enabled": { "description": "Toggles whether embedding service is enabled.", "type": "boolean", "default": false }, "dimensions": { "description": "The dimensionality of the embedding vectors.", "type": "integer", "minimum": 0 }, "model": { "description": "The model used for embedding.", "type": "string" }, "accessToken": { "description": "The access token used to authenticate with the external embedding API service.", "type": "string" }, "url": { "description": "The url to the external embedding API service.", "type": "string", "format": "uri" } } } ``` ## Repo embeddings search The repo embeddings search is implemented in its own service. When a user queries a repo using embeddings search, the following happens: - Download the repo embedding index from blob storage and cache it in memory - We cache up to 5 embedding indexes in memory - Embed the query and use the embedded query vector to find similar code and text file metadata in the embedding index - Query gitserver for the actual file contents - Return the results ## Interesting files - [Similarity search](https://github.com/sourcegraph/sourcegraph/pull/48017/files#diff-102cc83520004eb0e2795e49bc435c5142ca555189b1db3a52bbf1ffb82fa3c6) - [Repo embedding job handler](https://github.com/sourcegraph/sourcegraph/pull/48017/files#diff-c345f373f426398beb4b9cd5852ba862a2718687882db2a8b2d9c7fbb5f1dc52) - [External embedding api client](https://github.com/sourcegraph/sourcegraph/pull/48017/files#diff-ad1e7956f518e4bcaee17dd9e7ac04a5e090c00d970fcd273919e887e1d2cf8f) - [Embedding a repo](https://github.com/sourcegraph/sourcegraph/pull/48017/files#diff-1f35118727128095b7816791b6f0a2e0e060cddee43d25102859b8159465585c) - [Embeddings searcher service](https://github.com/sourcegraph/sourcegraph/pull/48017/files#diff-5b20f3e7ef87041daeeaef98b58ebf7388519cedcdfc359dc5e6d4e0b021472e) - [Embeddings search](https://github.com/sourcegraph/sourcegraph/pull/48017/files#diff-79f95b9cc3f1ef39c1a0b88015bd9cd6c19c30a8d4c147409f1b8e8cd9462ea1) - [Repo embedding index cache management](https://github.com/sourcegraph/sourcegraph/pull/48017/files#diff-8a41f7dec31054889dbf86e97c52223d5636b4d408c6b375bcfc09160a8b70f8) - [GraphQL resolvers](https://github.com/sourcegraph/sourcegraph/pull/48017/files#diff-9b30a0b5efcb63e2f4611b99ab137fbe09629a769a4f30d10a1b2da41a01d21f) ## Test plan - Start by filling out the `embeddings` object in the site config (let me know if you need an API key) - Start the embeddings service using `sg start embeddings` - Go to the `/api/console` page and schedule a repo embedding job and a context detection embedding job: ```graphql mutation { scheduleRepositoriesForEmbedding(repoNames: ["github.com/sourcegraph/handbook"]) { __typename } scheduleContextDetectionForEmbedding { __typename } } ``` - Once both are finished, you should be able to query the repo embedding index, and determine whether context is need for a given query: ```graphql query { isContextRequiredForQuery(query: "how do access tokens work") embeddingsSearch( repo: "UmVwb3NpdG9yeToy", # github.com/sourcegraph/handbook GQL ID query: "how do access tokens work", codeResultsCount: 5, textResultsCount: 5) { codeResults { fileName content } textResults { fileName content } } } ```
2023-03-01 09:50:12 +00:00
embeddings:
requiresDevPrivate: true
checks:
- docker
- redis
- postgres
- git
commands:
- embeddings
- frontend
- worker
- repo-updater
- web
- gitserver-0
- gitserver-1
- searcher
- symbols
- caddy
- docsite
- syntax-highlighter
- github-proxy
- zoekt-index-0
- zoekt-index-1
- zoekt-web-0
- zoekt-web-1
- blobstore
sg: rename "core-app" to "iam" (#36886)
2022-06-10 06:28:27 +00:00
iam:
sg: check dev-private exists in 'sg start' (#25514) First attempt at https://github.com/sourcegraph/sourcegraph/issues/25501. I think this is fine for now and can avoid new colleagues running into this problem. It also got me thinking: I could've defined this as "check" in `sg.config.yaml`. But then I realised that in order to get a readable error message out of that I'd have to do some bash scripting. And that in turn reinforces my thought that the `checks:` should probably be Go functions. Thoughts?
2021-09-29 14:22:16 +00:00
requiresDevPrivate: true
sg: add run-set for core app (#23156)
2021-07-23 07:59:43 +00:00
checks:
- docker
- redis
- postgres
sg: un-revert redesigned checks functionality after fixing wrong checks name and err handler (#31124) This restores #31055 after the revert in #31102. See a0c21370a635635cbd82df6f12fc9e5ee053969d for what the problem was: in the original code I managed to mess up an `if err != nil { return err }` and a checkname in `sg.config.yaml` was wrong.
2022-02-12 18:36:28 +00:00
- git
sg: add run-set for core app (#23156)
2021-07-23 07:59:43 +00:00
commands:
sg: shorten prefix for commands name (#37437)
2022-06-20 17:57:54 +00:00
- frontend
- repo-updater
- web
sg config: use two gitserver instances by default (#42749) We talked about this in repo-management & IAM and we think we should increase our dogfood consumption. This is one step in that direction: most of our customers run mulitple gitserver instances. This helps us make sure we dogfood that. ATTENTION: this will lead to some repositories being recloned/reindexed locally.
2022-10-10 13:26:58 +00:00
- gitserver-0
- gitserver-1
sg: add run-set for core app (#23156)
2021-07-23 07:59:43 +00:00
- caddy
- github-proxy
sg: migrate jaeger and grafana to sg.config.yaml (#23413) The other two are just placeholders and will be migrated later.
2021-08-02 08:25:39 +00:00
monitoring:
prometheus: upgrade alertmanager to v0.22.2, improve upgrade docs (#23714) Co-authored-by: ᴜɴᴋɴᴡᴏɴ <joe@sourcegraph.com>
2021-08-09 10:30:54 +00:00
checks:
- docker
sg: migrate jaeger and grafana to sg.config.yaml (#23413) The other two are just placeholders and will be migrated later.
2021-08-02 08:25:39 +00:00
commands:
- jaeger
dev: enable OpenTelemetry by default in dev, improve tracer logging (#40946) Since observability components are not run by default in sg start, we also amend the error logging of the exporter to suppress issues unless observability.tracing.debug is enabled.
2022-08-29 16:42:36 +00:00
- otel-collector
sg: migrate jaeger and grafana to sg.config.yaml (#23413) The other two are just placeholders and will be migrated later.
2021-08-02 08:25:39 +00:00
- prometheus
- grafana
- postgres_exporter
sg: support reloading+retrying of non-long-running commands (#23895) Co-authored-by: Robert Lin <robert@bobheadxi.dev>
2021-08-13 17:50:27 +00:00
- monitoring-generator
sg: migrate jaeger and grafana to sg.config.yaml (#23413) The other two are just placeholders and will be migrated later.
2021-08-02 08:25:39 +00:00
prometheus: upgrade alertmanager to v0.22.2, improve upgrade docs (#23714) Co-authored-by: ᴜɴᴋɴᴡᴏɴ <joe@sourcegraph.com>
2021-08-09 10:30:54 +00:00
monitoring-alerts:
checks:
- docker
- redis
- postgres
commands:
- prometheus
- grafana
# For generated alerts docs
- docsite
# For the alerting integration with frontend
sg: shorten prefix for commands name (#37437)
2022-06-20 17:57:54 +00:00
- frontend
- web
prometheus: upgrade alertmanager to v0.22.2, improve upgrade docs (#23714) Co-authored-by: ᴜɴᴋɴᴡᴏɴ <joe@sourcegraph.com>
2021-08-09 10:30:54 +00:00
- caddy
Standalone dev server: Use HTTPS (#25900)
2021-10-12 10:44:39 +00:00
web-standalone:
commands:
Rename web-standalone to web-standalone-http for http-only (#26495)
2021-10-22 10:30:29 +00:00
- web-standalone-http
Standalone dev server: Use HTTPS (#25900)
2021-10-12 10:44:39 +00:00
- caddy
env:
ENTERPRISE: 1
oss-web-standalone:
commands:
Rename web-standalone to web-standalone-http for http-only (#26495)
2021-10-22 10:30:29 +00:00
- web-standalone-http
Standalone dev server: Use HTTPS (#25900)
2021-10-12 10:44:39 +00:00
- caddy
web-standalone-prod:
commands:
Rename web-standalone to web-standalone-http for http-only (#26495)
2021-10-22 10:30:29 +00:00
- web-standalone-http-prod
Standalone dev server: Use HTTPS (#25900)
2021-10-12 10:44:39 +00:00
- caddy
env:
ENTERPRISE: 1
oss-web-standalone-prod:
commands:
Rename web-standalone to web-standalone-http for http-only (#26495)
2021-10-22 10:30:29 +00:00
- web-standalone-http-prod
Standalone dev server: Use HTTPS (#25900)
2021-10-12 10:44:39 +00:00
- caddy
sg: Add description field to sg commands (#39240)
2022-07-26 14:51:04 +00:00
# For testing our OpenTelemetry stack
internal/tracer: add OpenTelemetry tracer (#37984) We extend internal/tracer to support a new tracer type, "opentelemetry", that sends everything in OpenTelemetry format using the OpenTracing bridge. See https://github.com/sourcegraph/sourcegraph/pull/37984 for more details.
2022-07-05 18:28:15 +00:00
otel:
checks:
- docker
commands:
- otel-collector
- jaeger
Make `sg start app` work (#47068) * `sg start app` (renamed from `enterprise-single-program`) Also adds comments and a temporarily necessary env var. * do not register insights migrators if Code Insights is disabled Fixes a panic upon startup in backfillerv2Migrator when running App (the single binary) in local dev.
2023-01-30 17:43:20 +00:00
app:
Sourcegraph App (single-binary branch) (#46547) * internal: add service and singleprogram packages * sg.config.yaml: add single-binary build targets * internal/env: add a function for clearing environ cache * internal/{workerutil,metrics}: add a hack to allow running 2 executors in the same process * internal/conf: add single-program deploy type * internal/singleprogram: clarify security * cmd/sourcegraph-oss: add initial single-binary main (will not build yet) * enterprise/cmd/sourcegraph: initial enterprise single-binary * Add multi-platform builds for single-program * single-binary: correctly build JS artifacts into binary * license_finder licenses add github.com/xi2/xz "Public domain" * internal/service/svcmain: correctly initialize logger for DeprecatedSingleServiceMain * worker: refactor to new service pattern * cmd/github-proxy: refactor to use new service pattern * symbols: refactor to use new service pattern * gitserver: refactor to user new service pattern * searcher: refactor to use new service pattern * gitserver: refactor to use new service pattern * repo-updater: refactor to use new service pattern * frontend: refactor to use new service pattern * executor: refactor to use new service pattern * internal/symbols: use new LoadConfig pattern * precise-code-intel-worker: refactor to use new service pattern * internal/symbols: load config for tests * cmd/repo-updater: remove LoadConfig approach * cmd/symbols: workaround env var conflict with searcher * executor: internal: add workaround to allow running 2 instances in same process * executors: add EXECUTOR_QUEUE_DISABLE_ACCESS_TOKEN for single-binary and dev deployments only * single-binary: use EXECUTOR_QUEUE_DISABLE_ACCESS_TOKEN * extsvc/github: fix default value for single-program deploy type * single-binary: stop relying on a local ctags image * single-binary: use unix sockets for postgres * release App snapshots in CI when pushed to app/release-snapshot branch * internal/service/svcmain: update TODO comment * executor: correct DEPLOY_TYPE check * dev/check: allow single-binary to import dbconn * executor: remove accidental reliance on dbconn package * executor: improve error logging when running commands (#46546) * executor: improve error logging when running commands * executor: do not attempt std config validation running e.g. install cmd * executor: do not pull in the conf package / frontend reliance * ci: executors: correct site config for passwordless auth * server: fix bug where github-proxy would try to be a conf server * CI: executors: fix integration test passwordless auth * executors: allow passwordless auth in sourcegraph/server for testing * repo-updater: fix enterprise init (caused regression in repository syncing) Signed-off-by: Stephen Gutekanst <stephen@sourcegraph.com> Co-authored-by: Peter Guy <peter.guy@sourcegraph.com> Co-authored-by: Quinn Slack <quinn@slack.org>
2023-01-20 00:35:39 +00:00
requiresDevPrivate: true
checks:
- git
commands:
- sourcegraph
- docsite
- web
- caddy
App: Improve Code Insights UI for App (#49220)
2023-03-14 16:22:09 +00:00
env:
DISABLE_CODE_INSIGHTS: false
Sourcegraph App (single-binary branch) (#46547) * internal: add service and singleprogram packages * sg.config.yaml: add single-binary build targets * internal/env: add a function for clearing environ cache * internal/{workerutil,metrics}: add a hack to allow running 2 executors in the same process * internal/conf: add single-program deploy type * internal/singleprogram: clarify security * cmd/sourcegraph-oss: add initial single-binary main (will not build yet) * enterprise/cmd/sourcegraph: initial enterprise single-binary * Add multi-platform builds for single-program * single-binary: correctly build JS artifacts into binary * license_finder licenses add github.com/xi2/xz "Public domain" * internal/service/svcmain: correctly initialize logger for DeprecatedSingleServiceMain * worker: refactor to new service pattern * cmd/github-proxy: refactor to use new service pattern * symbols: refactor to use new service pattern * gitserver: refactor to user new service pattern * searcher: refactor to use new service pattern * gitserver: refactor to use new service pattern * repo-updater: refactor to use new service pattern * frontend: refactor to use new service pattern * executor: refactor to use new service pattern * internal/symbols: use new LoadConfig pattern * precise-code-intel-worker: refactor to use new service pattern * internal/symbols: load config for tests * cmd/repo-updater: remove LoadConfig approach * cmd/symbols: workaround env var conflict with searcher * executor: internal: add workaround to allow running 2 instances in same process * executors: add EXECUTOR_QUEUE_DISABLE_ACCESS_TOKEN for single-binary and dev deployments only * single-binary: use EXECUTOR_QUEUE_DISABLE_ACCESS_TOKEN * extsvc/github: fix default value for single-program deploy type * single-binary: stop relying on a local ctags image * single-binary: use unix sockets for postgres * release App snapshots in CI when pushed to app/release-snapshot branch * internal/service/svcmain: update TODO comment * executor: correct DEPLOY_TYPE check * dev/check: allow single-binary to import dbconn * executor: remove accidental reliance on dbconn package * executor: improve error logging when running commands (#46546) * executor: improve error logging when running commands * executor: do not attempt std config validation running e.g. install cmd * executor: do not pull in the conf package / frontend reliance * ci: executors: correct site config for passwordless auth * server: fix bug where github-proxy would try to be a conf server * CI: executors: fix integration test passwordless auth * executors: allow passwordless auth in sourcegraph/server for testing * repo-updater: fix enterprise init (caused regression in repository syncing) Signed-off-by: Stephen Gutekanst <stephen@sourcegraph.com> Co-authored-by: Peter Guy <peter.guy@sourcegraph.com> Co-authored-by: Quinn Slack <quinn@slack.org>
2023-01-20 00:35:39 +00:00
Move sg.config.yaml to repo root & allow private overwrites (#19487) This moves the sg.config.yaml to the repository root so that one can just use `sg start` and `sg run gitserver` etc. without having to specify the `-conf` flag. It also adds support for a local `.gitignore`d `sg.config.overwrites.yaml` file that users can add to the repository. If it exists, it will then get merged with the basic configuration when `sg` starts. What's the usecase? Custom command sets, custom env vars, etc. that you don't want to commit to the repository.
2021-03-29 08:14:53 +00:00
tests:
# These can be run with `sg test [name]`
backend:
sg: Add ability to overwrite argument to tests (#20747) This allows you to run `sg test backend` (without args) and it will use the `defaultArgs` as defined in the `tests.backend` object in `sg.config.yaml`. _If_ you pass additional args those will get passed to the cmd instead: `sg test backend ./internal/my-folder` will then run the `tests.backend.cmd` with `./internal/my-folder` as arg.
2021-05-10 08:28:45 +00:00
cmd: go test
defaultArgs: ./...
Move sg.config.yaml to repo root & allow private overwrites (#19487) This moves the sg.config.yaml to the repository root so that one can just use `sg start` and `sg run gitserver` etc. without having to specify the `-conf` flag. It also adds support for a local `.gitignore`d `sg.config.overwrites.yaml` file that users can add to the repository. If it exists, it will then get merged with the basic configuration when `sg` starts. What's the usecase? Custom command sets, custom env vars, etc. that you don't want to commit to the repository.
2021-03-29 08:14:53 +00:00
backend-integration:
cmd: cd dev/gqltest && go test -long -base-url $BASE_URL -email $EMAIL -username $USERNAME -password $PASSWORD ./gqltest
env:
# These are defaults. They can be overwritten by setting the env vars when
# running the command.
sg: Fix configuration for code intel stack (#21532) Updates `sg.config.yaml` to fix booting up the code intel stack.
2021-05-31 07:51:39 +00:00
BASE_URL: 'http://localhost:3080'
EMAIL: 'joe@sourcegraph.com'
PASSWORD: '12345'
sg: Add ability to overwrite argument to tests (#20747) This allows you to run `sg test backend` (without args) and it will use the `defaultArgs` as defined in the `tests.backend` object in `sg.config.yaml`. _If_ you pass additional args those will get passed to the cmd instead: `sg test backend ./internal/my-folder` will then run the `tests.backend.cmd` with `./internal/my-folder` as arg.
2021-05-10 08:28:45 +00:00
Add browser extension sg cli support (#27276) * Add 'sg test bext|bext-integration|bext-e2e' to run corresponding tests * Add 'sg run bext' to start bext in dev mode
2021-11-10 09:26:15 +00:00
bext:
web: migrate from `yarn` to `pnpm` (#46143) Bazel's rules_js rely on the pnpm package manager. To simplify the integration, we're migrating to pnpm from our current package manager — yarn. Another reason to migrate is that pnpm is cool and fast. 😉
2023-01-12 03:50:09 +00:00
cmd: pnpm --filter @sourcegraph/browser test
Add browser extension sg cli support (#27276) * Add 'sg test bext|bext-integration|bext-e2e' to run corresponding tests * Add 'sg run bext' to start bext in dev mode
2021-11-10 09:26:15 +00:00
Disable browser extension for private repositories when using cloud URL (#28089) * Add 'sg run bext-build' command * Update GitLab & GitHub private repository detection * Fix Gitlab integration test * Add changelog
2021-11-25 12:46:36 +00:00
bext-build:
web: migrate from `yarn` to `pnpm` (#46143) Bazel's rules_js rely on the pnpm package manager. To simplify the integration, we're migrating to pnpm from our current package manager — yarn. Another reason to migrate is that pnpm is cool and fast. 😉
2023-01-12 03:50:09 +00:00
cmd: EXTENSION_PERMISSIONS_ALL_URLS=true pnpm --filter @sourcegraph/browser build
Disable browser extension for private repositories when using cloud URL (#28089) * Add 'sg run bext-build' command * Update GitLab & GitHub private repository detection * Fix Gitlab integration test * Add changelog
2021-11-25 12:46:36 +00:00
Add browser extension sg cli support (#27276) * Add 'sg test bext|bext-integration|bext-e2e' to run corresponding tests * Add 'sg run bext' to start bext in dev mode
2021-11-10 09:26:15 +00:00
bext-integration:
web: migrate from `yarn` to `pnpm` (#46143) Bazel's rules_js rely on the pnpm package manager. To simplify the integration, we're migrating to pnpm from our current package manager — yarn. Another reason to migrate is that pnpm is cool and fast. 😉
2023-01-12 03:50:09 +00:00
cmd: pnpm --filter @sourcegraph/browser test-integration
Add browser extension sg cli support (#27276) * Add 'sg test bext|bext-integration|bext-e2e' to run corresponding tests * Add 'sg run bext' to start bext in dev mode
2021-11-10 09:26:15 +00:00
bext-e2e:
web: migrate from `yarn` to `pnpm` (#46143) Bazel's rules_js rely on the pnpm package manager. To simplify the integration, we're migrating to pnpm from our current package manager — yarn. Another reason to migrate is that pnpm is cool and fast. 😉
2023-01-12 03:50:09 +00:00
cmd: pnpm --filter @sourcegraph/browser mocha ./src/end-to-end/github.test.ts ./src/end-to-end/gitlab.test.ts
Add browser extension sg cli support (#27276) * Add 'sg test bext|bext-integration|bext-e2e' to run corresponding tests * Add 'sg run bext' to start bext in dev mode
2021-11-10 09:26:15 +00:00
env:
SOURCEGRAPH_BASE_URL: https://sourcegraph.com
docs: improve running e2e and regression tests docs (#40715)
2022-08-24 11:41:51 +00:00
client:
web: migrate from `yarn` to `pnpm` (#46143) Bazel's rules_js rely on the pnpm package manager. To simplify the integration, we're migrating to pnpm from our current package manager — yarn. Another reason to migrate is that pnpm is cool and fast. 😉
2023-01-12 03:50:09 +00:00
cmd: pnpm jest --testPathIgnorePatterns end-to-end regression integration storybook
sg: Add ability to overwrite argument to tests (#20747) This allows you to run `sg test backend` (without args) and it will use the `defaultArgs` as defined in the `tests.backend` object in `sg.config.yaml`. _If_ you pass additional args those will get passed to the cmd instead: `sg test backend ./internal/my-folder` will then run the `tests.backend.cmd` with `./internal/my-folder` as arg.
2021-05-10 08:28:45 +00:00
sg: improve `sg` web testsuites (#41488)
2022-09-09 09:36:10 +00:00
docsite:
cmd: .bin/docsite_${DOCSITE_VERSION} check ./doc
env:
chore: bump docsite version in `sg.config.yaml` (#50054)
2023-03-28 10:21:52 +00:00
DOCSITE_VERSION: v1.9.2 # Update DOCSITE_VERSION everywhere in all places (including outside this repo)
sg: improve `sg` web testsuites (#41488)
2022-09-09 09:36:10 +00:00
web-e2e:
sg: create a test command to run e2e tests locally (#34627)
2022-05-04 07:12:30 +00:00
preamble: |
dev/sg: fix enterprise-e2e set with env hackery (#38327)
2022-07-06 21:38:47 +00:00
A Sourcegraph isntance must be already running for these tests to work, most
docs: improve running e2e and regression tests docs (#40715)
2022-08-24 11:41:51 +00:00
commonly with: `sg start enterprise-e2e`
sg: create a test command to run e2e tests locally (#34627)
2022-05-04 07:12:30 +00:00
docs: improve running e2e and regression tests docs (#40715)
2022-08-24 11:41:51 +00:00
See more details: https://docs.sourcegraph.com/dev/how-to/testing#running-end-to-end-tests
web: migrate from `yarn` to `pnpm` (#46143) Bazel's rules_js rely on the pnpm package manager. To simplify the integration, we're migrating to pnpm from our current package manager — yarn. Another reason to migrate is that pnpm is cool and fast. 😉
2023-01-12 03:50:09 +00:00
cmd: pnpm test-e2e
sg: create a test command to run e2e tests locally (#34627)
2022-05-04 07:12:30 +00:00
env:
TEST_USER_EMAIL: test@sourcegraph.com
TEST_USER_PASSWORD: supersecurepassword
SOURCEGRAPH_BASE_URL: https://sourcegraph.test:3443
BROWSER: chrome
dev/sg: unify ExternalSecrets with secrets store, minor fixes for frontend-e2e (#34942)
2022-05-05 16:17:18 +00:00
external_secrets:
sg: create a test command to run e2e tests locally (#34627)
2022-05-04 07:12:30 +00:00
GH_TOKEN:
web: introduce `INTEGRATION_TESTS` env variable (#38805)
2022-07-14 13:05:49 +00:00
project: 'sourcegraph-ci'
name: 'BUILDKITE_GITHUBDOTCOM_TOKEN'
sg: conditionally install docsite, add docsite check (#23606)
2021-08-05 08:19:45 +00:00
sg: improve `sg` web testsuites (#41488)
2022-09-09 09:36:10 +00:00
web-regression:
docs: improve running e2e and regression tests docs (#40715)
2022-08-24 11:41:51 +00:00
preamble: |
A Sourcegraph instance must be already running for these tests to work, most
commonly with: `sg start enterprise-e2e`
See more details: https://docs.sourcegraph.com/dev/how-to/testing#running-regression-tests
web: migrate from `yarn` to `pnpm` (#46143) Bazel's rules_js rely on the pnpm package manager. To simplify the integration, we're migrating to pnpm from our current package manager — yarn. Another reason to migrate is that pnpm is cool and fast. 😉
2023-01-12 03:50:09 +00:00
cmd: pnpm test-regression
docs: improve running e2e and regression tests docs (#40715)
2022-08-24 11:41:51 +00:00
env:
SOURCEGRAPH_SUDO_USER: test
SOURCEGRAPH_BASE_URL: https://sourcegraph.test:3443
TEST_USER_PASSWORD: supersecurepassword
BROWSER: chrome
sg: improve `sg` web testsuites (#41488)
2022-09-09 09:36:10 +00:00
web-integration:
preamble: |
A web application should be built for these tests to work, most
[SG-22831] - Use `production` build for frontend integration tests (#46495)
2023-01-30 06:51:24 +00:00
commonly with: `sg run web-integration-build` or `sg run web-integration-build-prod` for production build.
sg: improve `sg` web testsuites (#41488)
2022-09-09 09:36:10 +00:00
See more details: https://docs.sourcegraph.com/dev/how-to/testing#running-integration-tests
web: migrate from `yarn` to `pnpm` (#46143) Bazel's rules_js rely on the pnpm package manager. To simplify the integration, we're migrating to pnpm from our current package manager — yarn. Another reason to migrate is that pnpm is cool and fast. 😉
2023-01-12 03:50:09 +00:00
cmd: pnpm test-integration
sg: improve `sg` web testsuites (#41488)
2022-09-09 09:36:10 +00:00
web-integration:debug:
preamble: |
A Sourcegraph instance must be already running for these tests to work, most
commonly with: `sg start web-standalone`
See more details: https://docs.sourcegraph.com/dev/how-to/testing#running-integration-tests
web: migrate from `yarn` to `pnpm` (#46143) Bazel's rules_js rely on the pnpm package manager. To simplify the integration, we're migrating to pnpm from our current package manager — yarn. Another reason to migrate is that pnpm is cool and fast. 😉
2023-01-12 03:50:09 +00:00
cmd: pnpm test-integration:debug
Reference in New Issue Copy Permalink