We don't only need to add this when the executor frontend URL points to
it, but also if the site config is set to host.docker.internal, hence I
made this an executor config flag instead. I think this should fix it,
but `sg start app` is acting up for me so would appreciate if someone
could verify that.
Closes https://github.com/sourcegraph/sourcegraph/issues/50011
It's really useful to know when debugging customer issues what version
they are running, and in the macOS app bundle we'll only have access to
logs (can't ask them to run `sourcegraph version` command once it
exists) so with this change we always print the version on startup:
```
[ sourcegraph] ✱ Sourcegraph App version: 0.0.0+dev
```
This is the first thing printed, to ensure it goes out before the binary
crashes (if such an issue is occurring.)
## Test plan
`sg start app` & confirm output manually
Signed-off-by: Stephen Gutekanst <stephen@sourcegraph.com>
Sourcegraph already runs best-effort if `src` is not on your path.
However, the errors are not very clear about the impact or how to
install it. This simply adds a better error message for it:
Fixes#46543
## Test plan
Manually tested by removing `src` from my path and running `sg start
app`
---------
Signed-off-by: Stephen Gutekanst <stephen@sourcegraph.com>
Co-authored-by: Joe Chen <joe@sourcegraph.com>
Prior to this change, if you ran Sourcegraph App without Docker
installed or running then when you ran a search request the backend
would make numerous requests to the `symbols` service, which would then
spew an _endless_ stream of errors to the console complaining that
`docker run` failed as the App `CTAGS_COMMAND` tried to run ctags via
Docker.
**I am actively working on bundling/building ctags into the Go binary**
so that it will always be present and this will not be an issue,
however, I need more time to fix that proper and right now this issue is
*severe* and blocks us from showing App to more people:
https://github.com/sourcegraph/sourcegraph/issues/48766
This change improves the current status quo substantially by adding a
few `if isSingleProgram` conditions to simply disable `symbols` (have it
return no results) if you are running App and ctags cannot be started
via Docker. An easy to read message is also shown to the user to tell
them Docker is missing and how it would make Sourcegraph better for
them:
<img width="1163" alt="image"
src="https://user-images.githubusercontent.com/3173176/223233097-f184c4cd-c7a6-47bc-a437-bb301ef946f8.png">
Fixes#48766
Helps #47992
## Test plan
* Ran `sg start app` both with and without Docker running on my system,
verified behavior of both.
* All changes are gated behind a `if isSingleProgram` conditional and so
very low risk to other deployments.
---------
Signed-off-by: Stephen Gutekanst <stephen@sourcegraph.com>
- Prefer `SEARCHER_CACHE_DIR` env var name to `CACHE_DIR`.
- Use the `{SEARCHER,SYMBOLS}_CACHE_DIR` env var names in App init (`singleprogram`) and the single Docker image (`server`).
- Fix typos/grammar in the changelog and docs.
Also fixes a panic when lauching App: `env var "CACHE_DIR" already
registered with a different description or value`.
## Test plan
Run `sg start app` and notice it does not panic upon startup.
Spotted this warning from a devs log:
WARNING: The requested image's platform (linux/amd64) does not match
the detected host platform (linux/arm64/v8) and no specific platform
was requested
I think this is the only place we do docker run for app.
Test Plan: yolo
We use env.Get at init() time for the repo updater URL. This doesn't
give enough time for app's setup to set the variable. So instead we
switch to deciding the value based on deploy type. This is the same
strategy used by gitserver, symbols, searcher and zoekt.
Test Plan: sg start app and CI
If the CTAGS_COMMAND environment variable is already defined and points to a file, don't create the temporary shell script that uses a ctags Docker image.
* singleprogram: use localhost for gitserver
Using the hostname isn't gaurenteed to work, it depends on how the
machine is configured. For example on my linux desktop this fails. We
only start the services on localhost and we use ipv4 localhost address
for all other services.
Additionally introduce GITSERVER_EXTERNAL_ADDR. Treating the hostname as
an address is incorrect. We instead use a more common pattern of having
an environment variable which specifies the reachable address for the
gitserver service.
Test Plan: sg start app and sg start work
Previously one needed to specify all of these env vars in order to run the `sourcegraph` App:
```
SRC_FRONTEND_INTERNAL=localhost:3090
EXECUTOR_FRONTEND_PASSWORD=asdf
```
After this, just running `sourcegraph` is enough.
* `EXECUTOR_FRONTEND_PASSWORD`, although it is ignored in the App deployment type,
must be set to _something_ otherwise a validation error will occur.
Signed-off-by: Stephen Gutekanst <stephen@sourcegraph.com>
* blobstore: url.QueryEscape object names to store on disk
* blobstore: clarify what the bucket lock gives us
* blobstore: clarify why we do not close the file
* blobstore: explain multipartUploadMu intentions
* blobstore: realize that multipartUploadMu does not need to be held when reading
* blobstore: simplify multipart upload mutex holding
* singleprogram: fix BLOBSTORE_DATA_DIR
* blobstore: improve filesystem atomics based on Keegan feedback
* blobstore: change tmp file suffix so we can cleanup later
* blobstore: refactor S3 HTTP route serving for cleanliness
* blobstore: fix service config / env var init order
This was preventing `sg start app` (in both dev/prod) from detecting the
right `BLOBSTORE_DATA_DIR` as it was read from the environment before it
could be set by the single binary.
Signed-off-by: Stephen Gutekanst <stephen@sourcegraph.com>
* blobstore: update to use new service pattern following App changes
* app: begin using new blobstore Go service
* blobstore: only log bucket creation if it is a new bucket
* blobstore: add a (failing) test which shows init fails if bucket exists
* blobstore: fix failing TestInit_BucketAlreadyExists
* blobstore: better test explanations
* blobstore: ensure errors are always clearly from blobstore
* add autogold v2 dependency
* blobstore: implement object uploading API
* blobstore: go test ./cmd/blobstore/... -update
* blobstore: make TestGetExists fail (getting objects is not implemented)
* blobstore: implement object fetching
* blobstore: simplify interface{} -> any
* blobstore: do not pass deeply nested S3 error types to autogold
* blobstore: add more S3 XML response types
* blobstore: make putObject return object metadata
* blobstore: do not rely on x-id query param (inconsistent)
* blobstore: implement Service.deleteObject
* blobstore: initial multipart upload backend logic
* blobstore: implement S3 HeadObject API
* blobstore: rely on Go http.ResponseWriter for content type
* blobstore: add S3 DELETE API stubs
* blobstore: implement S3 CreateMultipartUpload API
* blobstore: implement S3 CompleteMultipartUpload API
* blobstore: implement S3 UploadPartCopy API
* blobstore: add test for object composition
* blobstore: add object deletion test
* blobstore: implement S3 AbortMultipartUpload API
* blobstore: implement S3 DeleteObject API
* blobstore: add failing ExpireObjects tests
* blobstore: disable TestExpireObjects for now (listing objects not implemented yet)
* blobstore: remove banned ioutil usage
* blobstore: fix error introducing in refactoring
* go mod tidy
Signed-off-by: Stephen Gutekanst <stephen@sourcegraph.com>
This change makes us actually always use the in-memory password when the
deployment type is Sourcegraph App.
After #46721 the executor password was randomly generated, but the site
config had the randomly generated password written once, while the env
var would always use the latest in-memory password. Effectively, the one
saved to disk could diverge from the one in memory on the second run.
In this change I instead have us respect the in-memory password at all
locations that we use the configured password, iff the deployment type
is App.
Signed-off-by: Stephen Gutekanst <stephen@sourcegraph.com>
This PR aims to address the feedback on the initial #app land on main.
I've added a standalone version of the `executor run` command function body, so we don’t run an actual CLI in there and parse cmd line args as well, which would've meant running `sourcegraph test-vm` would actually attempt to spawn a test-vm and os.Exit after - probably not desired.
Then I could restore the original main function we initially had before the shared package refactor, because executor is unlike all our other services, not only a service but a CLI tool, and only one of the subcommands is a service. This is already shared and can be invoked from singlebinary now.
So that made the shared.Main superfluous, so I renamed the package to `singlebinary` and added a comment to never use that outside of it, to make the purpose clear.
..And finally added an in-memory secret to address the security concerns that I raised in slack before. It seems that the app stack doesn't yet fully work in local dev, so I cannot test it, but I think this is the right way to go about it and we can follow up with a fix if required once the general stack is working properly.
## Test plan
Verified executor works in `sg run batches` again without error, that the `executor` cli tool properly exists after a command finished (which caused problems in CI too with it hanging forever), and tried to see if the app doesn't log worse errors than before, which doesn't seem to be the case.
Also ran a main-dry-run, a regular build, and just to be extra sure an executor-patch-notest CI job as well. This seems to fix the red builds on main that we've seen this morning.
* internal: add service and singleprogram packages
* sg.config.yaml: add single-binary build targets
* internal/env: add a function for clearing environ cache
* internal/{workerutil,metrics}: add a hack to allow running 2 executors in the same process
* internal/conf: add single-program deploy type
* internal/singleprogram: clarify security
* cmd/sourcegraph-oss: add initial single-binary main (will not build yet)
* enterprise/cmd/sourcegraph: initial enterprise single-binary
* Add multi-platform builds for single-program
* single-binary: correctly build JS artifacts into binary
* license_finder licenses add github.com/xi2/xz "Public domain"
* internal/service/svcmain: correctly initialize logger for DeprecatedSingleServiceMain
* worker: refactor to new service pattern
* cmd/github-proxy: refactor to use new service pattern
* symbols: refactor to use new service pattern
* gitserver: refactor to user new service pattern
* searcher: refactor to use new service pattern
* gitserver: refactor to use new service pattern
* repo-updater: refactor to use new service pattern
* frontend: refactor to use new service pattern
* executor: refactor to use new service pattern
* internal/symbols: use new LoadConfig pattern
* precise-code-intel-worker: refactor to use new service pattern
* internal/symbols: load config for tests
* cmd/repo-updater: remove LoadConfig approach
* cmd/symbols: workaround env var conflict with searcher
* executor: internal: add workaround to allow running 2 instances in same process
* executors: add EXECUTOR_QUEUE_DISABLE_ACCESS_TOKEN for single-binary and dev deployments only
* single-binary: use EXECUTOR_QUEUE_DISABLE_ACCESS_TOKEN
* extsvc/github: fix default value for single-program deploy type
* single-binary: stop relying on a local ctags image
* single-binary: use unix sockets for postgres
* release App snapshots in CI when pushed to app/release-snapshot branch
* internal/service/svcmain: update TODO comment
* executor: correct DEPLOY_TYPE check
* dev/check: allow single-binary to import dbconn
* executor: remove accidental reliance on dbconn package
* executor: improve error logging when running commands (#46546)
* executor: improve error logging when running commands
* executor: do not attempt std config validation running e.g. install cmd
* executor: do not pull in the conf package / frontend reliance
* ci: executors: correct site config for passwordless auth
* server: fix bug where github-proxy would try to be a conf server
* CI: executors: fix integration test passwordless auth
* executors: allow passwordless auth in sourcegraph/server for testing
* repo-updater: fix enterprise init (caused regression in repository syncing)
Signed-off-by: Stephen Gutekanst <stephen@sourcegraph.com>
Co-authored-by: Peter Guy <peter.guy@sourcegraph.com>
Co-authored-by: Quinn Slack <quinn@slack.org>
