artin/sourcegraph
1
0
Fork 0
mirror of https://github.com/sourcegraph/sourcegraph.git synced 2026-02-06 18:51:59 +00:00
Code Issues Packages Projects Releases Wiki Activity

Update development setup information and simplify some setup steps (#21403)

Browse Source 
This commit does a couple of changes to the development setup, to update
information and to simplify the setup process:

- Update documentation with latest requirements
- Fix formatting of the requirements list
- Introduce `dev/compose.yml` to simplify redis and postgresql docker
  setups via docker-compose
- Add default values for PG* environment variables to the `dev/start.sh`
script.
- Restructure docker instructions

Co-authored-by: Thorsten Ball <mrnugget@gmail.com>
This commit is contained in:
Felix Kling 2021-05-28 11:27:43 +02:00 committed by GitHub
parent 0d238869a3
commit 8efaa1ec5d
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
8 changed files with 193 additions and 78 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

24
dev/compose.yml Normal file
View File

@ -0,0 +1,24 @@
# This files provides an easy way to start Redis and PostgreSQL servers for
# development, via docker-compose.
# See doc/dev/getting-started/quickstart_6_start_server.md for how to
# use it.
services:
redis:
image: redis
ports:
- 6379:6379
volumes:
- ${REDIS_DATA_DIR:-redis_data}:/data
postgresql:
image: postgres
ports:
- 5432:5432
environment:
- POSTGRES_PASSWORD=${PGUSER:-sourcegraph}
- POSTGRES_USER=${PGPASSWORD:-sourcegraph}
- "POSTGRES_INITDB_ARGS= --encoding=UTF8 "
volumes:
- ${PGDATA_DIR:-postgres_data}:/var/lib/postgresql/data
volumes:
redis_data:
postgres_data:

6
dev/start.sh
View File

@ -33,6 +33,12 @@ if [ -f .env ]; then
fi
# Verify postgresql config.
export PGPORT=${PGPORT:-5432}
export PGHOST=${PGHOST:-localhost}
export PGUSER=${PGUSER:-sourcegraph}
export PGPASSWORD=${PGPASSWORD:-sourcegraph}
export PGDATABASE=${PGDATABASE:-sourcegraph}
export PGSSLMODE=${PGSSLMODE:-disable}
hash psql 2>/dev/null || {
# "brew install postgres" does not put psql on the $PATH by default;
# try to fix this automatically if we can.

87
doc/dev/getting-started/quickstart_1_install_dependencies.md
View File

@ -3,8 +3,9 @@
> NOTE: Please see install instructions for [macOS](#macos) and [Ubuntu](#ubuntu) in succeeding sections.
Sourcegraph has the following dependencies:
- [Git](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git) (v2.18 or higher)
- [Go](https://golang.org/doc/install) (v1.14 or higher)
- [Go](https://golang.org/doc/install) (see current version in [`.tool-versions`](https://github.com/sourcegraph/sourcegraph/blob/main/.tool-versions))
- [Node JS](https://nodejs.org/en/download/) (see current recommended version in [.nvmrc](https://github.com/sourcegraph/sourcegraph/blob/main/.nvmrc))
- [make](https://www.gnu.org/software/make/)
- [Docker](https://docs.docker.com/engine/installation/) (v18 or higher)
@ -17,6 +18,14 @@ Sourcegraph has the following dependencies:
- [Comby](https://github.com/comby-tools/comby/) (v0.11.3 or higher)
- [Watchman](https://facebook.github.io/watchman/)
You can choose to install Redis and/or PostgreSQL directly on your system, or you can run them as docker containers with [docker compose](https://docs.docker.com/compose/). The following instructions will describe both options.
Running within a container provides some advantages such as storing the data separately from the container, you do not need to run it as a system service and its easy to use different database versions or multiple databases.
Running as system services might yield better performance, especially on macOS.
No matter which option you choose, docker is required because the development server starts additional docker containers.
The following are two recommendations for installing these dependencies:
## macOS
@ -30,13 +39,28 @@ The following are two recommendations for installing these dependencies:
brew install --cask docker
```
3. Install Go, Node Version Manager, PostgreSQL, Redis, Git, golang-migrate, Comby, SQLite tools, and jq with the following command:
3. Install Go, Yarn, Git, golang-migrate, Comby, SQLite tools, and jq with the following command:
```
brew install go yarn redis postgresql git gnu-sed golang-migrate comby sqlite pcre FiloSottile/musl-cross/musl-cross jq watchman
brew install go yarn git gnu-sed golang-migrate comby sqlite pcre FiloSottile/musl-cross/musl-cross jq watchman
```
4. Install the Node Version Manager (`nvm`) using:
4. (without docker) Install PostgreSQL and Redis
If you want to run Redis and/or PostgreSQL directly on your system install them with the follwing command:
```
brew install postgresql
brew install redis
```
5. (with docker) Install Docker Compose
We provide a docker compose file at `dev/compose.yml` to make it easy to run Redis and PostgreSQL as docker containers. Fortunately `docker-compose` comes with Docker for Mac so no additional step is required.
See the official [docker compose documentation](https://docs.docker.com/compose/install/).
6. Install the Node Version Manager (`nvm`) using:
```
NVM_VERSION="$(curl https://api.github.com/repos/nvm-sh/nvm/releases/latest | jq -r .name)"
@ -48,7 +72,7 @@ The following are two recommendations for installing these dependencies:
`source ~/.zshrc`) or restart your terminal session to pick up the `nvm`
definitions. Re-running the install script will update the installation.
Note: `nvm` is implemented as a shell function, so it may not show up in
> NOTE: `nvm` is implemented as a shell function, so it may not show up in
the output of `which nvm`. Use `type nvm` to verify whether it is set up.
There is also a Homebrew package for `nvm`, but it is unsupported by the
`nvm` maintainers.
@ -62,7 +86,7 @@ The following are two recommendations for installing these dependencies:
* Then add the following to your `config.fish`:
```
```sh
function nvm
bass source ~/.nvm/nvm.sh --no-use ';' nvm $argv
end
@ -70,7 +94,7 @@ The following are two recommendations for installing these dependencies:
set -x NVM_DIR ~/.nvm
```
5. Install the current recommended version of Node JS by running the following
7. Install the current recommended version of Node JS by running the following
from the working directory of a sourcegraph repository clone:
```
@ -81,20 +105,22 @@ The following are two recommendations for installing these dependencies:
After doing this, `node -v` should show the same version mentioned in
`.nvmrc` at the root of the sourcegraph repository.
Note: Although there is a Homebrew package for Node, we advise using `nvm`
> NOTE: Although there is a Homebrew package for Node, we advise using `nvm`
instead, to ensure you get a Node version compatible with the current state
of the sourcegraph repository.
6. Configure PostgreSQL and Redis to start automatically
8. (optional) Configure PostgreSQL and Redis to start automatically
If you have installed PostgreSQL and Redis directly on your system, start them with the following commands:
```
brew services start postgresql
brew services start redis
```
(You can stop them later by calling `stop` instead of `start` above.)
You can stop them later by calling `stop` instead of `start` above.
7. Ensure `psql`, the PostgreSQL command line client, is on your `$PATH`.
9. Ensure `psql`, the PostgreSQL command line client, is on your `$PATH`.
Homebrew does not put it there by default. Homebrew gives you the command to run to insert `psql` in your path in the "Caveats" section of `brew info postgresql`. Alternatively, you can use the command below. It might need to be adjusted depending on your Homebrew prefix (`/usr/local` below) and shell (bash below).
```
@ -102,7 +128,7 @@ The following are two recommendations for installing these dependencies:
source ~/.bash_profile
```
8. Open a new Terminal window to ensure `psql` is now on your `$PATH`.
10. Open a new Terminal window to ensure `psql` is now on your `$PATH`.
## Ubuntu
@ -131,7 +157,11 @@ The following are two recommendations for installing these dependencies:
3. Install dependencies:
```
sudo apt install -y make git-all postgresql postgresql-contrib redis-server libpcre3-dev libsqlite3-dev pkg-config golang-go musl-tools docker-ce docker-ce-cli containerd.io yarn jq libnss3-tools
sudo apt install -y make git-all libpcre3-dev libsqlite3-dev pkg-config golang-go musl-tools docker-ce docker-ce-cli containerd.io yarn jq libnss3-tools
# (without docker) install PostgreSQL and/or Redis if you don't want to run them as docker containers
sudo apt install -y redis-server
sudo apt install -y postgresql postgresql-contrib
# install golang-migrate (you must rename the extracted binary to `golang-migrate` and move the binary into your $PATH)
curl -L https://github.com/golang-migrate/migrate/releases/download/v4.7.0/migrate.linux-amd64.tar.gz | tar xvz
@ -159,28 +189,23 @@ The following are two recommendations for installing these dependencies:
nvm install
```
4. Configure startup services
3. (with docker) Install Docker Compose
We provide a docker compose file at `dev/compose.yml` to make it easy to run Redis and PostgreSQL as docker containers.
> NOTE: Although Ubuntu provides a `docker-compose` package, we recommend to install the latest version via `pip` so that it is compatible with our compose file.
See the official [docker compose documentation](https://docs.docker.com/compose/install/) for more details on different installation options.
4. (without docker) Configure startup services
If you have installed PostgreSQL and Redis directly on your system, start them with the following commands:
```
sudo systemctl enable postgresql
sudo systemctl enable redis-server.service
sudo systemctl enable --now postgresql
sudo systemctl enable --now redis-server.service
```
5. (optional) You can also run Redis using Docker
In this case you should not enable the `redis-server.service` from the previous step.
```
dockerd # if docker isn't already running
docker run -p 6379:6379 -v $REDIS_DATA_DIR redis
# $REDIS_DATA_DIR should be an absolute path to a folder where you intend to store Redis data
```
You need to have Redis running when you start the dev server later on. If you have issues running Docker, try [adding your user to the docker group][dockerGroup], and/or [updating the socket file permissions][socketPermissions], or try running these commands under `sudo`.
[dockerGroup]: https://stackoverflow.com/a/48957722
[socketPermissions]: https://stackoverflow.com/a/51362528
## (optional) asdf
[asdf](https://github.com/asdf-vm/asdf) is a CLI tool that manages runtime versions for a number of different languages and tools. It can be likened to a language-agnostic version of [nvm](https://github.com/nvm-sh/nvm) or [pyenv](https://github.com/pyenv/pyenv).

50
doc/dev/getting-started/quickstart_2_initialize_database.md
View File

@ -34,7 +34,7 @@ You need a fresh Postgres database and a database user that has full ownership o
The Sourcegraph server reads PostgreSQL connection configuration from the [`PG*` environment variables](http://www.postgresql.org/docs/current/static/libpq-envars.html).
Add these, for example, in your `~/.bashrc`:
The startup script sets default values that work with the setup described here, but if you are using different values you can overwrite them, for example, in your `~/.bashrc`:
```
export PGPORT=5432
@ -53,49 +53,19 @@ You need a fresh Postgres database and a database user that has full ownership o
## With Docker
You may also want to run Postgres within a docker container instead of as a system service. Running within a container provides some advantages such as storing the data seperately from the container, you do not need to run it as a system service and its easy to use different database versions or multiple databases.
The Sourcegraph server reads PostgreSQL connection configuration from the [`PG*` environment variables](http://www.postgresql.org/docs/current/static/libpq-envars.html).
1. Create a directory to store and mount the database from for persistence:
The development server startup script as well as the docker compose file provide default settings, so it will work out of the box.
```
# Create a seperate dir to store the database
mkdir PGDATA_DIR
You can overwrite these settings e.g. via your `~/.bashrc`:
# Also add this to your '~/.bashrc'
export PGDATA_DIR=/path/to/PGDATA_DIR/
```
```
export PGUSER=sourcegraph
export PGPASSWORD=sourcegraph
export PGDATABASE=sourcegraph
```
2. Run the container:
```
docker run -d -p 5432:5432 -e POSTGRES_PASSWORD=sourcegraph \
-e POSTGRES_USER=sourcegraph -e POSTGRES_INITDB_ARGS=" --encoding=UTF8 " \
-v $PGDATA_DIR:/var/lib/postgresql/data postgres
```
3. Ensure you can connect to the database in your container using `docker exec -it <container-id> psql -U sourcegraph` and enter password `sourcegraph`. You can use the command `docker ps` to view the container-id.
4. Configure database settings in your environment:
The Sourcegraph server reads PostgreSQL connection configuration from the [`PG*` environment variables](http://www.postgresql.org/docs/current/static/libpq-envars.html).
Add these, for example, in your `~/.bashrc`:
```
export PGPORT=5432
export PGHOST=localhost
export PGUSER=sourcegraph
export PGPASSWORD=sourcegraph
export PGDATABASE=sourcegraph
export PGSSLMODE=disable
```
You can also use a tool like [`envdir`][envdir] or [a `.dotenv` file][dotenv] to source these env vars on demand when you start the server.
[envdir]: https://cr.yp.to/daemontools/envdir.html
[dotenv]: https://github.com/joho/godotenv
5. On restarting docker, you may need to start the container again. Find the image with `docker ps --all` and then `docker run <$containerID>` to start again.
You can also use the `PGDATA_DIR` environment variable to specify a local folder (instead of a volume) to store the database files. See the `dev/compose.yml` file for more details.
## More info

25
doc/dev/getting-started/quickstart_3_start_docker.md
View File

@ -1,10 +1,12 @@
# Quickstart step 3: (macOS) Start Docker
# Quickstart step 3: Start Docker
## Option A: Docker for Mac
## macOS
### Option A: Docker for Mac
This is the easy way - just launch Docker.app and wait for it to finish loading.
## Option B: docker-machine
### Option B: docker-machine
The Docker daemon should be running in the background, which you can test by
running `docker ps`. If you're on OS X and using `docker-machine` instead of
@ -15,4 +17,21 @@ docker-machine start default
eval $(docker-machine env)
```
## Ubuntu/Linux
The docker daemon might already be running, but if necessary you can use the following commands to start it:
```sh
# as a system service
sudo systemctl enable --now docker
# manually
dockerd
```
If you have issues running Docker, try [adding your user to the docker group][dockerGroup], and/or [updating the socket file permissions][socketPermissions], or try running these commands under `sudo`.
[dockerGroup]: https://stackoverflow.com/a/48957722
[socketPermissions]: https://stackoverflow.com/a/51362528
[< Previous](quickstart_2_initialize_database.md) | [Next >](quickstart_4_clone_repository.md)

37
doc/dev/getting-started/quickstart_6_start_server.md
View File

@ -1,14 +1,43 @@
# Quickstart step 6: Start the server
```bash
cd sourcegraph
./dev/start.sh
```
1. Navigate to the sourcegraph source code folder
```
cd sourcegraph
```
2. (with docker) Start the Redis and PostgreSQL containers in the background with:
```
docker-compose -f dev/compose.yml up -d
```
You can also start either Redis or PostgreSQL, if you are running the other one directly on your system:
```
docker-compose -f dev/compose.yml up -d redis
# or
docker-compose -f dev/compose.yml up -d postgresql
```
When you want to stop the containers, run:
```
docker-compose -f dev/compose.yml down
```
3. Start the server with
```
./dev/start.sh
```
This will continuously compile your code and live reload your locally running instance of Sourcegraph.
Navigate your browser to https://sourcegraph.test:3443 to see if everything worked.
If the script exits with errors or outputs errors, have a look at [Troubleshooting](../how-to/troubleshooting_local_development.md).
## Environment
Sourcegraph server is a collection of smaller binaries. The development server, [dev/start.sh](https://github.com/sourcegraph/sourcegraph/blob/main/dev/start.sh), initializes the environment and starts a process manager that runs all of the binaries. See the [Architecture doc](../background-information/architecture/index.md) for a full description of what each of these services does. The sections below describe the dependencies you need to run `dev/start.sh`.

41
doc/dev/how-to/troubleshooting_local_development.md
View File

@ -145,3 +145,44 @@ If you're running macOS 10.15.x (Catalina) reinstalling the Xcode Command Line T
2. Reinstall it with `xcode-select --install`
3. Go to `sourcegraph/sourcegraph`’s root directory and run `rm -rf node_modules`
3. Re-run the launch script (`./dev/start.sh`)
## Permission errors for Grafana and Prometheus containers
The Grafana and Prometheus containers need group read access for specific files.
Otherwise you will see errors such as
```
grafana | standard_init_linux.go:228: exec user process caused: permission denied
prometheus | standard_init_linux.go:228: exec user process caused: permission denied
```
or
```
prometheus | t=2021-05-26T10:05:26+0000 lvl=eror msg="command [/prometheus.sh --web.listen-address=0.0.0.0:9092] exited: fork/exec /prometheus.sh: permission denied" cmd=prom-wrapper
prometheus | t=2021-05-26T10:05:26+0000 lvl=eror msg="command [/alertmanager.sh --config.file=/sg_config_prometheus/alertmanag
```
If you don't need these containers for your development work, the simplest solution
is to start the development server without those containers:
```sh
./dev/start.sh --except grafana,prometheus
```
Otherwise, if files do not normally have group permissions in your environment
(e.g. if you set `umask 077`), then you need to
1. Set group read permissions for the Grafana and Prometheus docker images, with
```sh
chmod -R g=rX docker-images/{grafana,prometheus}
```
2. Run `dev/start.sh` so that new files are group readable. If you have `umask`
set to a different value you can change that value just for this script by
starting it in a subshell:
```sh
(umask 027; ./dev/start.sh)
```

1
doc/dev/index.md
View File

@ -44,6 +44,7 @@ Guides to help with troubleshooting, configuring test instances, debugging, and
- [Running out of disk space](how-to/troubleshooting_local_development.md#running-out-of-disk-space)
- [Certificate expiry](how-to/troubleshooting_local_development.md#certificate-expiry)
- [CPU/RAM/bandwidth/battery usage](how-to/troubleshooting_local_development.md#cpurambandwidthbattery-usage)
- [Permission errors for Grafana and Prometheus](how-to/troubleshooting_local_development.md#permission-errors-for-grafana-and-prometheus-containers)
### Implementing Sourcegraph