sourcegraph/dev/sg

sg - the Sourcegraph developer tool

          _____                    _____
         /\    \                  /\    \
        /::\    \                /::\    \
       /::::\    \              /::::\    \
      /::::::\    \            /::::::\    \
     /:::/\:::\    \          /:::/\:::\    \
    /:::/__\:::\    \        /:::/  \:::\    \
    \:::\   \:::\    \      /:::/    \:::\    \
  ___\:::\   \:::\    \    /:::/    / \:::\    \
 /\   \:::\   \:::\    \  /:::/    /   \:::\ ___\
/::\   \:::\   \:::\____\/:::/____/  ___\:::|    |
\:::\   \:::\   \::/    /\:::\    \ /\  /:::|____|
 \:::\   \:::\   \/____/  \:::\    /::\ \::/    /
  \:::\   \:::\    \       \:::\   \:::\ \/____/
   \:::\   \:::\____\       \:::\   \:::\____\
    \:::\  /:::/    /        \:::\  /:::/    /
     \:::\/:::/    /          \:::\/:::/    /
      \::::::/    /            \::::::/    /
       \::::/    /              \::::/    /
        \::/    /                \::/____/
         \/____/

sg is the CLI tool that Sourcegraph developers can use to develop Sourcegraph.

• Quickstart
• Usage
  • sg [start,run,run-set] - Start dev environment
  • sg test - Running test suites
  • sg doctor - Check health of dev environment
  • sg live - See currently deployed version
  • sg migration - Run or manipulate database migrations
  • sg rfc - List, open, or search Sourcegraph RFCs
• Configuration
• TODOs
• Hacking
• Principles
• Inspiration
• Ideas
  • Generators
  • Edit configuration files
  • Tail logs

Quickstart

Run the following to install sg from inside sourcegraph/sourcegraph:

./dev/sg/install.sh

Make sure that $HOME/go/bin is in your $PATH. (If you use $GOPATH then $GOPATH/bin needs to be in the $PATH)

Note for Linux users: A command called sg is already available at /usr/bin/sg. To use the Sourcegraph sg CLI, you need to make sure that its location comes first in PATH. For example, by prepending $GOPATH/bin:

export PATH=$GOPATH/bin:$PATH

Instead of the more conventional:

export PATH=$PATH:$GOPATH/bin

Or you may add an alias to your .bashrc:

alias sg=$HOME/go/bin/sg

Then, in the root of sourcegraph/sourcegraph, run:

sg start

This will boot the default commands in sg.config.yaml in the root of the repository.

Alternative install method (if you want to move the binary to a custom location):

In the root of sourcegraph/sourcegraph, run the following:

go build -o ~/my/path/sg ./dev/sg

Make sure that ~/my/path is in your $PATH then.

Usage

sg [start,run,run-set] - Start dev environment

# Run default environment (this starts the 'default' command set defined in `sg.config.yaml`):
sg start

# Run the enterprise environment:
sg run-set enterprise

# Run specific commands:
sg run gitserver
sg run frontend

# List available commands:
sg run -help

# List available command sets:
sg run-set -help

sg test - Running test suites

# Run different test suites:
sg test backend
sg test backend-integration
sg test frontend
sg test frontend-e2e

# List available test suites:
sg test -help

# Arguments are passed along to the command
sg test backend-integration -run TestSearch

sg doctor - Check health of dev environment

# Run the checks defined in sg.config.yaml
sg doctor

sg live - See currently deployed version

# See which version is deployed on a preset environment
sg live dot-com
sg live k8s

# See which version is deployed on a custom environment
sg live https://demo.sourcegraph.com

# List environments:
sg live -help

sg migration - Run or manipulate database migrations

# Migrate local default database up
sg migration up

# Migrate specific database down one migration
sg migration down --db codeintel -n 1

# Add new migration for specific database
sg migration add --db codeintel 'add missing index'

# Squash migrations for default database
sg migration squash

# Fixup your migrations comapred to main for databases
sg migration fixup

# To see what operations `sg migration fixup` will run, you can check with
sg migration fixup -run=false

# Or to run for only one database, you can use the -db flag, as in other operations.

sg rfc - List or open Sourcegraph RFCs

# List all RFCs
sg rfc list

# Search for an RFC
sg rfc search "search terms"

# Open a specific RFC
sg rfc open 420

Configuration

sg is configured through the sg.yaml file in the root of the sourcegraph/sourcegraph repository:

commands:
  gitserver:
    cmd: .bin/gitserver
    install: go install github.com/sourcegraph/sourcegraph/cmd/gitserver -o .bin/gitserver
    env:
      HOSTNAME: $SRC_GIT_SERVER_1

  enterprise-gitserver:
    cmd: .bin/gitserver
    install: go install github.com/sourcegraph/sourcegraph/enterprise/cmd/gitserver -o .bin/gitserver
    env:
      HOSTNAME: $SRC_GIT_SERVER_1

  query-runner:
    cmd: .bin/query-runner
    install: go install github.com/sourcegraph/sourcegraph/cmd/query-runner -o .bin/gitserver

  searcher:
    cmd: .bin/searcher
    install: go install github.com/sourcegraph/sourcegraph/cmd/searcher -o .bin/gitserver

  caddy:
    installDoc.darwin: 'use brew install'
    installDoc.linux: 'use apt install'

  web:
    cmd: ./node_modules/.bin/gulp --silent --color dev
    install: yarn install

  # [...]

commandsets:
  # This is the set that will be used when `sg start` is run:
  default:
    - gitserver
    - query-runner
    - repo-updater
    - searcher
    - symbols
    - github-proxy
    - frontend
    - watch
    - caddy
    - web
    - syntect_server
    - zoekt-indexserver-0
    - zoekt-indexserver-1
    - zoekt-webserver-0
    - zoekt-webserver-1
  # Another set that can be run with `sg run-set monitoring`:
  monitoring:
    - jaeger
    - docsite
    - prometheus
    - grafana
    - postgres_exporter
  enterprise:
    - enterprise-gitserver
    - enterprise-query-runner
    - enterprise-repo-updater
    - enterprise-frontend
    - searcher
    - symbols
    - github-proxy
    - watch
    - caddy
    - web
    - syntect_server
    - zoekt-indexserver-0
    - zoekt-indexserver-1
    - zoekt-webserver-0
    - zoekt-webserver-1

tests:
  # These can be run with `sg test [name]`
  backend:
    cmd: go test ./...
  backend-integration:
    cmd: go test -long -base-url $BASE_URL -email $EMAIL -username $USERNAME -password $PASSWORD
    env:
      # These are defaults. They can be overwritten by setting the env vars when
      # running the command.
      BASE_URL: 'http://localhost:3080'
      EMAIL: 'joe@sourcegraph.com'
      PASSWORD: '12345'
  frontend:
    cmd: yarn run jest --testPathIgnorePatterns end-to-end regression integration storybook
  frontend-e2e:
    cmd: yarn run mocha ./client/web/src/end-to-end/end-to-end.test.ts
    env:
      TS_NODE_PROJECT: client/web/src/end-to-end/tsconfig.json

TODOs

• Rename install in the config files to build because it's clearer
• Add the remaining processes from <root>/dev/Procfile to <root>/sg.config.yaml
• All of the ideas below
  • Rebuild and restart a command (if it has build defined, see Configuration): sg build gitserver
  • Implement the sg generate command
  • Implement sg edit site-config and sg edit external-services
  • Implement sg tail-log
• Add built-in support for "download binary" so that the caddy command, for example, would be 3 lines instead of 20. That would allow us to get rid of the bash code.
• Add a sg rfc create command

Hacking

When you want to hack on sg it's best to be in the dev/sg directory and run it from there:

cd dev/sg
go run . -config ../../sg.config.yaml start

The -config can be anything you want, of course.

Principles

• sg should be fun to use.
• If you think "it would be cool if sg could do X": add it! Let's go :)
• sg should make Sourcegraph developers productive and happy.
• sg is not and should not be a build system.
• sg is not and should not be a container orchestrator.
• Try to fix a lot of the problems in this RFC by encoding conventions in executable code.
• No bash. sg was built to get rid of all the bash scripts in ./dev/. If you have a chance to build something into sg to avoid another bash script: do it. Try to keep shell scripts to easy-to-understand one liners if you must. Replicating something in Go code that could be done in 4 lines of bash is probably a good idea.
• Duplicated data is fine as long as it's dumb data. Copying some lines in sg.config.yaml to get something working is often (but not always) better than trying to be clever.

You can also watch this video to get an overview of the original thinking that lead to sg.

Inspiration

• GitLab Developer Kit (GDK)
• Stripe's pay command, described here
• Stack Exchange Local Environment Setup command

Ideas

The following are ideas for what could/should be built into sg:

Generators

# Generate code, equivalent to current `./dev/generate.sh`
sg generate

# Generate only specific things
sg generate mocks ./internal/enterprise/batches

Edit configuration files

# Edit the site configuration
sg edit site-config # opens site-config in $EDITOR
# Edit external service configuration
sg edit external-services # opens external-services.json in $EDITOR

Tail logs

# Tail the SQL logs
sg tail-log sql
# Tail the http logs
sg tail-log http
# Tail all logs
sg tail-log all