2023-06-19 17:44:01 +00:00
load ( " :defs.bzl " , " server_integration_test " )
server_integration_test (
Switch to OCI/Wolfi based image (#52693)
This PR ships our freshly rewritten container images built with
rules_oci and Wolfi, which for now will only be used on S2.
*What is this about*
This work is the conjunction of [hardening container
images](https://github.com/orgs/sourcegraph/projects/302?pane=issue&itemId=25019223)
and fully building our container images with Bazel.
* All base images are now distroless, based on Wolfi, meaning we fully
control every little package version and we won't be subject anymore to
Alpine maintainers dropping a postgres version for example.
* Container images are now built with `rules_oci`, meaning we don't have
Dockerfile anymore, but instead created through [Bazel
rules](https://sourcegraph.sourcegraph.com/github.com/sourcegraph/sourcegraph@bzl/oci_wolfi/-/blob/enterprise/cmd/gitserver/BUILD.bazel).
Don't be scared, while this will look a bit strange to you at first,
it's much saner and simpler to do than our Dockerfiles and their muddy
shell scripts calling themselves in cascade.
:spiral_note_pad: *Plan*:
*1/ (NOW) We merge our branch on `main` today, here is what it does
change for you :point_down::skin-tone-3::*
* On `main`:
* It will introduce a new job on `main` _Bazel Push_, which will push
those new images on our registries with all tags prefixed by `bazel-`.
* These new images will be picked up by S2 and S2 only.
* The existing jobs building docker images and pushing them will stay in
place until we have QA'ed them enough and are confident to roll them out
on Dotcom.
* Because we'll be building both images, there will be more jobs running
on `main`, but this should not affect the wall clock time.
* On all branches (so your PRs and `main`)
* The _Bazel Test_ job will now run: Backend Integration Tests, E2E
Tests and CodeIntel QA
* This will increase the duration of your test jobs in PRs, but as we
haven't removed yet the `sg lint` step, it should not affect too much
the wall clock time of your PRs.
* But it will also increase your confidence toward your changes, as the
coverage will vastly increased compared to before.
* If you have ongoing branches which are affecting the docker images
(like adding a new binary, like the recent `scip-tags`, reach us out on
#job-fair-bazel so we can help you to port your changes. It's much much
simpler than before, but it's going to be unfamiliar to you).
* If something goes awfully wrong, we'll rollback and update this
thread.
*2/ (EOW / Early next week) Once we're confident enough with what we saw
on S2, we'll roll the new images on Dotcom.*
* After the first successful deploy and a few sanity checks, we will
drop the old images building jobs.
* At this point, we'll reach out to all TLs asking for their help to
exercise all features of our product to ensure we catch any potential
breakage.
## Test plan
<!-- All pull requests REQUIRE a test plan:
https://docs.sourcegraph.com/dev/background-information/testing_principles
-->
* We tested our new images on `scale-testing` and it worked.
* The new container building rules comes with _container tests_ which
ensures that produced images are containing and configured with what
should be in there:
[example](https://sourcegraph.sourcegraph.com/github.com/sourcegraph/sourcegraph@bzl/oci_wolfi/-/blob/enterprise/cmd/gitserver/image_test.yaml)
.
---------
Co-authored-by: Dave Try <davetry@gmail.com>
Co-authored-by: Will Dollman <will.dollman@sourcegraph.com>
2023-06-02 10:12:52 +00:00
name = " e2e_test " ,
2023-11-24 09:52:17 +00:00
timeout = " long " ,
Switch to OCI/Wolfi based image (#52693)
This PR ships our freshly rewritten container images built with
rules_oci and Wolfi, which for now will only be used on S2.
*What is this about*
This work is the conjunction of [hardening container
images](https://github.com/orgs/sourcegraph/projects/302?pane=issue&itemId=25019223)
and fully building our container images with Bazel.
* All base images are now distroless, based on Wolfi, meaning we fully
control every little package version and we won't be subject anymore to
Alpine maintainers dropping a postgres version for example.
* Container images are now built with `rules_oci`, meaning we don't have
Dockerfile anymore, but instead created through [Bazel
rules](https://sourcegraph.sourcegraph.com/github.com/sourcegraph/sourcegraph@bzl/oci_wolfi/-/blob/enterprise/cmd/gitserver/BUILD.bazel).
Don't be scared, while this will look a bit strange to you at first,
it's much saner and simpler to do than our Dockerfiles and their muddy
shell scripts calling themselves in cascade.
:spiral_note_pad: *Plan*:
*1/ (NOW) We merge our branch on `main` today, here is what it does
change for you :point_down::skin-tone-3::*
* On `main`:
* It will introduce a new job on `main` _Bazel Push_, which will push
those new images on our registries with all tags prefixed by `bazel-`.
* These new images will be picked up by S2 and S2 only.
* The existing jobs building docker images and pushing them will stay in
place until we have QA'ed them enough and are confident to roll them out
on Dotcom.
* Because we'll be building both images, there will be more jobs running
on `main`, but this should not affect the wall clock time.
* On all branches (so your PRs and `main`)
* The _Bazel Test_ job will now run: Backend Integration Tests, E2E
Tests and CodeIntel QA
* This will increase the duration of your test jobs in PRs, but as we
haven't removed yet the `sg lint` step, it should not affect too much
the wall clock time of your PRs.
* But it will also increase your confidence toward your changes, as the
coverage will vastly increased compared to before.
* If you have ongoing branches which are affecting the docker images
(like adding a new binary, like the recent `scip-tags`, reach us out on
#job-fair-bazel so we can help you to port your changes. It's much much
simpler than before, but it's going to be unfamiliar to you).
* If something goes awfully wrong, we'll rollback and update this
thread.
*2/ (EOW / Early next week) Once we're confident enough with what we saw
on S2, we'll roll the new images on Dotcom.*
* After the first successful deploy and a few sanity checks, we will
drop the old images building jobs.
* At this point, we'll reach out to all TLs asking for their help to
exercise all features of our product to ensure we catch any potential
breakage.
## Test plan
<!-- All pull requests REQUIRE a test plan:
https://docs.sourcegraph.com/dev/background-information/testing_principles
-->
* We tested our new images on `scale-testing` and it worked.
* The new container building rules comes with _container tests_ which
ensures that produced images are containing and configured with what
should be in there:
[example](https://sourcegraph.sourcegraph.com/github.com/sourcegraph/sourcegraph@bzl/oci_wolfi/-/blob/enterprise/cmd/gitserver/image_test.yaml)
.
---------
Co-authored-by: Dave Try <davetry@gmail.com>
Co-authored-by: Will Dollman <will.dollman@sourcegraph.com>
2023-06-02 10:12:52 +00:00
args = [
2023-06-19 17:44:01 +00:00
" $(location //client/web/src/end-to-end:e2e) " ,
Switch to OCI/Wolfi based image (#52693)
This PR ships our freshly rewritten container images built with
rules_oci and Wolfi, which for now will only be used on S2.
*What is this about*
This work is the conjunction of [hardening container
images](https://github.com/orgs/sourcegraph/projects/302?pane=issue&itemId=25019223)
and fully building our container images with Bazel.
* All base images are now distroless, based on Wolfi, meaning we fully
control every little package version and we won't be subject anymore to
Alpine maintainers dropping a postgres version for example.
* Container images are now built with `rules_oci`, meaning we don't have
Dockerfile anymore, but instead created through [Bazel
rules](https://sourcegraph.sourcegraph.com/github.com/sourcegraph/sourcegraph@bzl/oci_wolfi/-/blob/enterprise/cmd/gitserver/BUILD.bazel).
Don't be scared, while this will look a bit strange to you at first,
it's much saner and simpler to do than our Dockerfiles and their muddy
shell scripts calling themselves in cascade.
:spiral_note_pad: *Plan*:
*1/ (NOW) We merge our branch on `main` today, here is what it does
change for you :point_down::skin-tone-3::*
* On `main`:
* It will introduce a new job on `main` _Bazel Push_, which will push
those new images on our registries with all tags prefixed by `bazel-`.
* These new images will be picked up by S2 and S2 only.
* The existing jobs building docker images and pushing them will stay in
place until we have QA'ed them enough and are confident to roll them out
on Dotcom.
* Because we'll be building both images, there will be more jobs running
on `main`, but this should not affect the wall clock time.
* On all branches (so your PRs and `main`)
* The _Bazel Test_ job will now run: Backend Integration Tests, E2E
Tests and CodeIntel QA
* This will increase the duration of your test jobs in PRs, but as we
haven't removed yet the `sg lint` step, it should not affect too much
the wall clock time of your PRs.
* But it will also increase your confidence toward your changes, as the
coverage will vastly increased compared to before.
* If you have ongoing branches which are affecting the docker images
(like adding a new binary, like the recent `scip-tags`, reach us out on
#job-fair-bazel so we can help you to port your changes. It's much much
simpler than before, but it's going to be unfamiliar to you).
* If something goes awfully wrong, we'll rollback and update this
thread.
*2/ (EOW / Early next week) Once we're confident enough with what we saw
on S2, we'll roll the new images on Dotcom.*
* After the first successful deploy and a few sanity checks, we will
drop the old images building jobs.
* At this point, we'll reach out to all TLs asking for their help to
exercise all features of our product to ensure we catch any potential
breakage.
## Test plan
<!-- All pull requests REQUIRE a test plan:
https://docs.sourcegraph.com/dev/background-information/testing_principles
-->
* We tested our new images on `scale-testing` and it worked.
* The new container building rules comes with _container tests_ which
ensures that produced images are containing and configured with what
should be in there:
[example](https://sourcegraph.sourcegraph.com/github.com/sourcegraph/sourcegraph@bzl/oci_wolfi/-/blob/enterprise/cmd/gitserver/image_test.yaml)
.
---------
Co-authored-by: Dave Try <davetry@gmail.com>
Co-authored-by: Will Dollman <will.dollman@sourcegraph.com>
2023-06-02 10:12:52 +00:00
" $(rootpath //:mocha_config) " ,
" ' $(rootpath //client/web/src/end-to-end:testing_e2e_bundle)/**/*.test.js ' " ,
] ,
data = [
" //:mocha_config " ,
" //client/web/src/end-to-end:e2e " ,
" //client/web/src/end-to-end:testing_e2e_bundle " ,
2023-09-11 12:39:27 +00:00
" //cmd/server:image_tarball " ,
Switch to OCI/Wolfi based image (#52693)
This PR ships our freshly rewritten container images built with
rules_oci and Wolfi, which for now will only be used on S2.
*What is this about*
This work is the conjunction of [hardening container
images](https://github.com/orgs/sourcegraph/projects/302?pane=issue&itemId=25019223)
and fully building our container images with Bazel.
* All base images are now distroless, based on Wolfi, meaning we fully
control every little package version and we won't be subject anymore to
Alpine maintainers dropping a postgres version for example.
* Container images are now built with `rules_oci`, meaning we don't have
Dockerfile anymore, but instead created through [Bazel
rules](https://sourcegraph.sourcegraph.com/github.com/sourcegraph/sourcegraph@bzl/oci_wolfi/-/blob/enterprise/cmd/gitserver/BUILD.bazel).
Don't be scared, while this will look a bit strange to you at first,
it's much saner and simpler to do than our Dockerfiles and their muddy
shell scripts calling themselves in cascade.
:spiral_note_pad: *Plan*:
*1/ (NOW) We merge our branch on `main` today, here is what it does
change for you :point_down::skin-tone-3::*
* On `main`:
* It will introduce a new job on `main` _Bazel Push_, which will push
those new images on our registries with all tags prefixed by `bazel-`.
* These new images will be picked up by S2 and S2 only.
* The existing jobs building docker images and pushing them will stay in
place until we have QA'ed them enough and are confident to roll them out
on Dotcom.
* Because we'll be building both images, there will be more jobs running
on `main`, but this should not affect the wall clock time.
* On all branches (so your PRs and `main`)
* The _Bazel Test_ job will now run: Backend Integration Tests, E2E
Tests and CodeIntel QA
* This will increase the duration of your test jobs in PRs, but as we
haven't removed yet the `sg lint` step, it should not affect too much
the wall clock time of your PRs.
* But it will also increase your confidence toward your changes, as the
coverage will vastly increased compared to before.
* If you have ongoing branches which are affecting the docker images
(like adding a new binary, like the recent `scip-tags`, reach us out on
#job-fair-bazel so we can help you to port your changes. It's much much
simpler than before, but it's going to be unfamiliar to you).
* If something goes awfully wrong, we'll rollback and update this
thread.
*2/ (EOW / Early next week) Once we're confident enough with what we saw
on S2, we'll roll the new images on Dotcom.*
* After the first successful deploy and a few sanity checks, we will
drop the old images building jobs.
* At this point, we'll reach out to all TLs asking for their help to
exercise all features of our product to ensure we catch any potential
breakage.
## Test plan
<!-- All pull requests REQUIRE a test plan:
https://docs.sourcegraph.com/dev/background-information/testing_principles
-->
* We tested our new images on `scale-testing` and it worked.
* The new container building rules comes with _container tests_ which
ensures that produced images are containing and configured with what
should be in there:
[example](https://sourcegraph.sourcegraph.com/github.com/sourcegraph/sourcegraph@bzl/oci_wolfi/-/blob/enterprise/cmd/gitserver/image_test.yaml)
.
---------
Co-authored-by: Dave Try <davetry@gmail.com>
Co-authored-by: Will Dollman <will.dollman@sourcegraph.com>
2023-06-02 10:12:52 +00:00
] ,
env = {
" TEST_USER_EMAIL " : " test@sourcegraph.com " ,
" TEST_USER_PASSWORD " : " supersecurepassword " ,
" SOURCEGRAPH_SUDO_TOKEN " : " fake-sg-token " ,
" NO_CLEANUP " : " false " ,
" KEEP_BROWSER " : " false " ,
" DEVTOOLS " : " false " ,
" BROWSER " : " chrome " ,
" WINDOW_WIDTH " : " 1280 " ,
" WINDOW_HEIGHT " : " 1024 " ,
" LOG_BROWSER_CONSOLE " : " false " ,
# Enable findDom on CodeMirror
" INTEGRATION_TESTS " : " true " ,
} ,
env_inherit = [
" DISPLAY " ,
2024-01-25 02:50:19 +00:00
" HEADLESS " ,
" SOURCEGRAPH_BASE_URL " ,
Switch to OCI/Wolfi based image (#52693)
This PR ships our freshly rewritten container images built with
rules_oci and Wolfi, which for now will only be used on S2.
*What is this about*
This work is the conjunction of [hardening container
images](https://github.com/orgs/sourcegraph/projects/302?pane=issue&itemId=25019223)
and fully building our container images with Bazel.
* All base images are now distroless, based on Wolfi, meaning we fully
control every little package version and we won't be subject anymore to
Alpine maintainers dropping a postgres version for example.
* Container images are now built with `rules_oci`, meaning we don't have
Dockerfile anymore, but instead created through [Bazel
rules](https://sourcegraph.sourcegraph.com/github.com/sourcegraph/sourcegraph@bzl/oci_wolfi/-/blob/enterprise/cmd/gitserver/BUILD.bazel).
Don't be scared, while this will look a bit strange to you at first,
it's much saner and simpler to do than our Dockerfiles and their muddy
shell scripts calling themselves in cascade.
:spiral_note_pad: *Plan*:
*1/ (NOW) We merge our branch on `main` today, here is what it does
change for you :point_down::skin-tone-3::*
* On `main`:
* It will introduce a new job on `main` _Bazel Push_, which will push
those new images on our registries with all tags prefixed by `bazel-`.
* These new images will be picked up by S2 and S2 only.
* The existing jobs building docker images and pushing them will stay in
place until we have QA'ed them enough and are confident to roll them out
on Dotcom.
* Because we'll be building both images, there will be more jobs running
on `main`, but this should not affect the wall clock time.
* On all branches (so your PRs and `main`)
* The _Bazel Test_ job will now run: Backend Integration Tests, E2E
Tests and CodeIntel QA
* This will increase the duration of your test jobs in PRs, but as we
haven't removed yet the `sg lint` step, it should not affect too much
the wall clock time of your PRs.
* But it will also increase your confidence toward your changes, as the
coverage will vastly increased compared to before.
* If you have ongoing branches which are affecting the docker images
(like adding a new binary, like the recent `scip-tags`, reach us out on
#job-fair-bazel so we can help you to port your changes. It's much much
simpler than before, but it's going to be unfamiliar to you).
* If something goes awfully wrong, we'll rollback and update this
thread.
*2/ (EOW / Early next week) Once we're confident enough with what we saw
on S2, we'll roll the new images on Dotcom.*
* After the first successful deploy and a few sanity checks, we will
drop the old images building jobs.
* At this point, we'll reach out to all TLs asking for their help to
exercise all features of our product to ensure we catch any potential
breakage.
## Test plan
<!-- All pull requests REQUIRE a test plan:
https://docs.sourcegraph.com/dev/background-information/testing_principles
-->
* We tested our new images on `scale-testing` and it worked.
* The new container building rules comes with _container tests_ which
ensures that produced images are containing and configured with what
should be in there:
[example](https://sourcegraph.sourcegraph.com/github.com/sourcegraph/sourcegraph@bzl/oci_wolfi/-/blob/enterprise/cmd/gitserver/image_test.yaml)
.
---------
Co-authored-by: Dave Try <davetry@gmail.com>
Co-authored-by: Will Dollman <will.dollman@sourcegraph.com>
2023-06-02 10:12:52 +00:00
" GHE_GITHUB_TOKEN " ,
" GH_TOKEN " ,
" SOURCEGRAPH_LICENSE_GENERATION_KEY " ,
" SOURCEGRAPH_LICENSE_KEY " ,
] ,
2023-06-19 17:44:01 +00:00
port = " 7080 " ,
runner_src = " :e2e_test.sh " ,
feat/bazel: `//cmd/{frontend,server}` targets that don't include client bundle for backend integration tests (#62877)
Currently, all backend integration tests transitively depend on the client bundle. This results in rebuilds of the closure, and a (likely) cache miss on the test, when modifying any client-side files.
Given that the client bundle isnt needed for these tests, we can create targets that don't include the client bundle in its transitive closure, which should in theory improve the cache hit for backend integration tests by not having client side changes invalidate it. This would also be beneficial in local env, to keep frontend rebuilds down
To do this, we still need to create a `web.manifest.json` file due to some unfortunate requirement on that file existing as part of initializing the sourcegraph instance. For this I just create an empty json file, `select` this instead of the client bundle target in client/web/dist/BUILD.bazel based on a `//:integration_testing_enabled` config setting, and creating a `go_binary`-wrapping bazel rule + macro that automatically applies a transition to set this to true `go_binary_nobundle`, and using that rule for the specific `//cmd/{server,frontend}:{server,frontend}_nobundle` binary rules (along with the relevant `oci_{image,tarball}` etc rules to consume it).
## Test plan
- Integration tests in CI still work
- `bazel cquery 'kind("js_library", deps(//cmd/frontend:image_nobundle))'`, `bazel cquery 'kind("js_library", deps(//cmd/server:image_nobundle))'`, ``bazel cquery 'kind("js_library", deps(//testing:backend_integration_test))'` etc all return empty set
- Release test with marker in web bundle to ensure released images contain the web bundle via `sg release run test --version 5.4.2` (commenting out other tests for brevity)
2024-05-28 13:32:48 +00:00
server_image = " //cmd/server:image_tarball " ,
Switch to OCI/Wolfi based image (#52693)
This PR ships our freshly rewritten container images built with
rules_oci and Wolfi, which for now will only be used on S2.
*What is this about*
This work is the conjunction of [hardening container
images](https://github.com/orgs/sourcegraph/projects/302?pane=issue&itemId=25019223)
and fully building our container images with Bazel.
* All base images are now distroless, based on Wolfi, meaning we fully
control every little package version and we won't be subject anymore to
Alpine maintainers dropping a postgres version for example.
* Container images are now built with `rules_oci`, meaning we don't have
Dockerfile anymore, but instead created through [Bazel
rules](https://sourcegraph.sourcegraph.com/github.com/sourcegraph/sourcegraph@bzl/oci_wolfi/-/blob/enterprise/cmd/gitserver/BUILD.bazel).
Don't be scared, while this will look a bit strange to you at first,
it's much saner and simpler to do than our Dockerfiles and their muddy
shell scripts calling themselves in cascade.
:spiral_note_pad: *Plan*:
*1/ (NOW) We merge our branch on `main` today, here is what it does
change for you :point_down::skin-tone-3::*
* On `main`:
* It will introduce a new job on `main` _Bazel Push_, which will push
those new images on our registries with all tags prefixed by `bazel-`.
* These new images will be picked up by S2 and S2 only.
* The existing jobs building docker images and pushing them will stay in
place until we have QA'ed them enough and are confident to roll them out
on Dotcom.
* Because we'll be building both images, there will be more jobs running
on `main`, but this should not affect the wall clock time.
* On all branches (so your PRs and `main`)
* The _Bazel Test_ job will now run: Backend Integration Tests, E2E
Tests and CodeIntel QA
* This will increase the duration of your test jobs in PRs, but as we
haven't removed yet the `sg lint` step, it should not affect too much
the wall clock time of your PRs.
* But it will also increase your confidence toward your changes, as the
coverage will vastly increased compared to before.
* If you have ongoing branches which are affecting the docker images
(like adding a new binary, like the recent `scip-tags`, reach us out on
#job-fair-bazel so we can help you to port your changes. It's much much
simpler than before, but it's going to be unfamiliar to you).
* If something goes awfully wrong, we'll rollback and update this
thread.
*2/ (EOW / Early next week) Once we're confident enough with what we saw
on S2, we'll roll the new images on Dotcom.*
* After the first successful deploy and a few sanity checks, we will
drop the old images building jobs.
* At this point, we'll reach out to all TLs asking for their help to
exercise all features of our product to ensure we catch any potential
breakage.
## Test plan
<!-- All pull requests REQUIRE a test plan:
https://docs.sourcegraph.com/dev/background-information/testing_principles
-->
* We tested our new images on `scale-testing` and it worked.
* The new container building rules comes with _container tests_ which
ensures that produced images are containing and configured with what
should be in there:
[example](https://sourcegraph.sourcegraph.com/github.com/sourcegraph/sourcegraph@bzl/oci_wolfi/-/blob/enterprise/cmd/gitserver/image_test.yaml)
.
---------
Co-authored-by: Dave Try <davetry@gmail.com>
Co-authored-by: Will Dollman <will.dollman@sourcegraph.com>
2023-06-02 10:12:52 +00:00
)
2024-01-17 14:45:31 +00:00
# Warning, this targets explicitly exclude p4 subrepo perms tests as they're very heavy and time
# consuming, due to having to add a repo, index it, etc ...
#
# See :backend_integration_test_p4 for then one, as opposed to this target, runs those tests
# exclusively.
2023-06-19 17:44:01 +00:00
server_integration_test (
Switch to OCI/Wolfi based image (#52693)
This PR ships our freshly rewritten container images built with
rules_oci and Wolfi, which for now will only be used on S2.
*What is this about*
This work is the conjunction of [hardening container
images](https://github.com/orgs/sourcegraph/projects/302?pane=issue&itemId=25019223)
and fully building our container images with Bazel.
* All base images are now distroless, based on Wolfi, meaning we fully
control every little package version and we won't be subject anymore to
Alpine maintainers dropping a postgres version for example.
* Container images are now built with `rules_oci`, meaning we don't have
Dockerfile anymore, but instead created through [Bazel
rules](https://sourcegraph.sourcegraph.com/github.com/sourcegraph/sourcegraph@bzl/oci_wolfi/-/blob/enterprise/cmd/gitserver/BUILD.bazel).
Don't be scared, while this will look a bit strange to you at first,
it's much saner and simpler to do than our Dockerfiles and their muddy
shell scripts calling themselves in cascade.
:spiral_note_pad: *Plan*:
*1/ (NOW) We merge our branch on `main` today, here is what it does
change for you :point_down::skin-tone-3::*
* On `main`:
* It will introduce a new job on `main` _Bazel Push_, which will push
those new images on our registries with all tags prefixed by `bazel-`.
* These new images will be picked up by S2 and S2 only.
* The existing jobs building docker images and pushing them will stay in
place until we have QA'ed them enough and are confident to roll them out
on Dotcom.
* Because we'll be building both images, there will be more jobs running
on `main`, but this should not affect the wall clock time.
* On all branches (so your PRs and `main`)
* The _Bazel Test_ job will now run: Backend Integration Tests, E2E
Tests and CodeIntel QA
* This will increase the duration of your test jobs in PRs, but as we
haven't removed yet the `sg lint` step, it should not affect too much
the wall clock time of your PRs.
* But it will also increase your confidence toward your changes, as the
coverage will vastly increased compared to before.
* If you have ongoing branches which are affecting the docker images
(like adding a new binary, like the recent `scip-tags`, reach us out on
#job-fair-bazel so we can help you to port your changes. It's much much
simpler than before, but it's going to be unfamiliar to you).
* If something goes awfully wrong, we'll rollback and update this
thread.
*2/ (EOW / Early next week) Once we're confident enough with what we saw
on S2, we'll roll the new images on Dotcom.*
* After the first successful deploy and a few sanity checks, we will
drop the old images building jobs.
* At this point, we'll reach out to all TLs asking for their help to
exercise all features of our product to ensure we catch any potential
breakage.
## Test plan
<!-- All pull requests REQUIRE a test plan:
https://docs.sourcegraph.com/dev/background-information/testing_principles
-->
* We tested our new images on `scale-testing` and it worked.
* The new container building rules comes with _container tests_ which
ensures that produced images are containing and configured with what
should be in there:
[example](https://sourcegraph.sourcegraph.com/github.com/sourcegraph/sourcegraph@bzl/oci_wolfi/-/blob/enterprise/cmd/gitserver/image_test.yaml)
.
---------
Co-authored-by: Dave Try <davetry@gmail.com>
Co-authored-by: Will Dollman <will.dollman@sourcegraph.com>
2023-06-02 10:12:52 +00:00
name = " backend_integration_test " ,
2023-10-24 05:05:22 +00:00
timeout = " long " ,
Switch to OCI/Wolfi based image (#52693)
This PR ships our freshly rewritten container images built with
rules_oci and Wolfi, which for now will only be used on S2.
*What is this about*
This work is the conjunction of [hardening container
images](https://github.com/orgs/sourcegraph/projects/302?pane=issue&itemId=25019223)
and fully building our container images with Bazel.
* All base images are now distroless, based on Wolfi, meaning we fully
control every little package version and we won't be subject anymore to
Alpine maintainers dropping a postgres version for example.
* Container images are now built with `rules_oci`, meaning we don't have
Dockerfile anymore, but instead created through [Bazel
rules](https://sourcegraph.sourcegraph.com/github.com/sourcegraph/sourcegraph@bzl/oci_wolfi/-/blob/enterprise/cmd/gitserver/BUILD.bazel).
Don't be scared, while this will look a bit strange to you at first,
it's much saner and simpler to do than our Dockerfiles and their muddy
shell scripts calling themselves in cascade.
:spiral_note_pad: *Plan*:
*1/ (NOW) We merge our branch on `main` today, here is what it does
change for you :point_down::skin-tone-3::*
* On `main`:
* It will introduce a new job on `main` _Bazel Push_, which will push
those new images on our registries with all tags prefixed by `bazel-`.
* These new images will be picked up by S2 and S2 only.
* The existing jobs building docker images and pushing them will stay in
place until we have QA'ed them enough and are confident to roll them out
on Dotcom.
* Because we'll be building both images, there will be more jobs running
on `main`, but this should not affect the wall clock time.
* On all branches (so your PRs and `main`)
* The _Bazel Test_ job will now run: Backend Integration Tests, E2E
Tests and CodeIntel QA
* This will increase the duration of your test jobs in PRs, but as we
haven't removed yet the `sg lint` step, it should not affect too much
the wall clock time of your PRs.
* But it will also increase your confidence toward your changes, as the
coverage will vastly increased compared to before.
* If you have ongoing branches which are affecting the docker images
(like adding a new binary, like the recent `scip-tags`, reach us out on
#job-fair-bazel so we can help you to port your changes. It's much much
simpler than before, but it's going to be unfamiliar to you).
* If something goes awfully wrong, we'll rollback and update this
thread.
*2/ (EOW / Early next week) Once we're confident enough with what we saw
on S2, we'll roll the new images on Dotcom.*
* After the first successful deploy and a few sanity checks, we will
drop the old images building jobs.
* At this point, we'll reach out to all TLs asking for their help to
exercise all features of our product to ensure we catch any potential
breakage.
## Test plan
<!-- All pull requests REQUIRE a test plan:
https://docs.sourcegraph.com/dev/background-information/testing_principles
-->
* We tested our new images on `scale-testing` and it worked.
* The new container building rules comes with _container tests_ which
ensures that produced images are containing and configured with what
should be in there:
[example](https://sourcegraph.sourcegraph.com/github.com/sourcegraph/sourcegraph@bzl/oci_wolfi/-/blob/enterprise/cmd/gitserver/image_test.yaml)
.
---------
Co-authored-by: Dave Try <davetry@gmail.com>
Co-authored-by: Will Dollman <will.dollman@sourcegraph.com>
2023-06-02 10:12:52 +00:00
args = [
" $(location //dev/gqltest:gqltest_test) " , # actual test
" $(location //dev/authtest:authtest_test) " , # actual test
] ,
data = [
feat/bazel: `//cmd/{frontend,server}` targets that don't include client bundle for backend integration tests (#62877)
Currently, all backend integration tests transitively depend on the client bundle. This results in rebuilds of the closure, and a (likely) cache miss on the test, when modifying any client-side files.
Given that the client bundle isnt needed for these tests, we can create targets that don't include the client bundle in its transitive closure, which should in theory improve the cache hit for backend integration tests by not having client side changes invalidate it. This would also be beneficial in local env, to keep frontend rebuilds down
To do this, we still need to create a `web.manifest.json` file due to some unfortunate requirement on that file existing as part of initializing the sourcegraph instance. For this I just create an empty json file, `select` this instead of the client bundle target in client/web/dist/BUILD.bazel based on a `//:integration_testing_enabled` config setting, and creating a `go_binary`-wrapping bazel rule + macro that automatically applies a transition to set this to true `go_binary_nobundle`, and using that rule for the specific `//cmd/{server,frontend}:{server,frontend}_nobundle` binary rules (along with the relevant `oci_{image,tarball}` etc rules to consume it).
## Test plan
- Integration tests in CI still work
- `bazel cquery 'kind("js_library", deps(//cmd/frontend:image_nobundle))'`, `bazel cquery 'kind("js_library", deps(//cmd/server:image_nobundle))'`, ``bazel cquery 'kind("js_library", deps(//testing:backend_integration_test))'` etc all return empty set
- Release test with marker in web bundle to ensure released images contain the web bundle via `sg release run test --version 5.4.2` (commenting out other tests for brevity)
2024-05-28 13:32:48 +00:00
" //cmd/server:no_client_bundle.image_tarball " ,
Switch to OCI/Wolfi based image (#52693)
This PR ships our freshly rewritten container images built with
rules_oci and Wolfi, which for now will only be used on S2.
*What is this about*
This work is the conjunction of [hardening container
images](https://github.com/orgs/sourcegraph/projects/302?pane=issue&itemId=25019223)
and fully building our container images with Bazel.
* All base images are now distroless, based on Wolfi, meaning we fully
control every little package version and we won't be subject anymore to
Alpine maintainers dropping a postgres version for example.
* Container images are now built with `rules_oci`, meaning we don't have
Dockerfile anymore, but instead created through [Bazel
rules](https://sourcegraph.sourcegraph.com/github.com/sourcegraph/sourcegraph@bzl/oci_wolfi/-/blob/enterprise/cmd/gitserver/BUILD.bazel).
Don't be scared, while this will look a bit strange to you at first,
it's much saner and simpler to do than our Dockerfiles and their muddy
shell scripts calling themselves in cascade.
:spiral_note_pad: *Plan*:
*1/ (NOW) We merge our branch on `main` today, here is what it does
change for you :point_down::skin-tone-3::*
* On `main`:
* It will introduce a new job on `main` _Bazel Push_, which will push
those new images on our registries with all tags prefixed by `bazel-`.
* These new images will be picked up by S2 and S2 only.
* The existing jobs building docker images and pushing them will stay in
place until we have QA'ed them enough and are confident to roll them out
on Dotcom.
* Because we'll be building both images, there will be more jobs running
on `main`, but this should not affect the wall clock time.
* On all branches (so your PRs and `main`)
* The _Bazel Test_ job will now run: Backend Integration Tests, E2E
Tests and CodeIntel QA
* This will increase the duration of your test jobs in PRs, but as we
haven't removed yet the `sg lint` step, it should not affect too much
the wall clock time of your PRs.
* But it will also increase your confidence toward your changes, as the
coverage will vastly increased compared to before.
* If you have ongoing branches which are affecting the docker images
(like adding a new binary, like the recent `scip-tags`, reach us out on
#job-fair-bazel so we can help you to port your changes. It's much much
simpler than before, but it's going to be unfamiliar to you).
* If something goes awfully wrong, we'll rollback and update this
thread.
*2/ (EOW / Early next week) Once we're confident enough with what we saw
on S2, we'll roll the new images on Dotcom.*
* After the first successful deploy and a few sanity checks, we will
drop the old images building jobs.
* At this point, we'll reach out to all TLs asking for their help to
exercise all features of our product to ensure we catch any potential
breakage.
## Test plan
<!-- All pull requests REQUIRE a test plan:
https://docs.sourcegraph.com/dev/background-information/testing_principles
-->
* We tested our new images on `scale-testing` and it worked.
* The new container building rules comes with _container tests_ which
ensures that produced images are containing and configured with what
should be in there:
[example](https://sourcegraph.sourcegraph.com/github.com/sourcegraph/sourcegraph@bzl/oci_wolfi/-/blob/enterprise/cmd/gitserver/image_test.yaml)
.
---------
Co-authored-by: Dave Try <davetry@gmail.com>
Co-authored-by: Will Dollman <will.dollman@sourcegraph.com>
2023-06-02 10:12:52 +00:00
" //dev/authtest:authtest_test " ,
" //dev/gqltest:gqltest_test " ,
] ,
env_inherit = [
" AWS_ACCESS_KEY_ID " ,
" AWS_CODE_COMMIT_PASSWORD " ,
" AWS_CODE_COMMIT_USERNAME " ,
" AWS_SECRET_ACCESS_KEY " ,
" AZURE_DEVOPS_TOKEN " ,
" AZURE_DEVOPS_USERNAME " ,
" BITBUCKET_SERVER_TOKEN " ,
" BITBUCKET_SERVER_URL " ,
" BITBUCKET_SERVER_USERNAME " ,
" GHE_GITHUB_TOKEN " ,
" PERFORCE_PASSWORD " ,
" PERFORCE_PORT " ,
" PERFORCE_USER " ,
" SOURCEGRAPH_LICENSE_GENERATION_KEY " ,
" SOURCEGRAPH_LICENSE_KEY " ,
] ,
2023-06-19 17:44:01 +00:00
port = " 7081 " ,
runner_src = " :backend_integration_test.sh " ,
feat/bazel: `//cmd/{frontend,server}` targets that don't include client bundle for backend integration tests (#62877)
Currently, all backend integration tests transitively depend on the client bundle. This results in rebuilds of the closure, and a (likely) cache miss on the test, when modifying any client-side files.
Given that the client bundle isnt needed for these tests, we can create targets that don't include the client bundle in its transitive closure, which should in theory improve the cache hit for backend integration tests by not having client side changes invalidate it. This would also be beneficial in local env, to keep frontend rebuilds down
To do this, we still need to create a `web.manifest.json` file due to some unfortunate requirement on that file existing as part of initializing the sourcegraph instance. For this I just create an empty json file, `select` this instead of the client bundle target in client/web/dist/BUILD.bazel based on a `//:integration_testing_enabled` config setting, and creating a `go_binary`-wrapping bazel rule + macro that automatically applies a transition to set this to true `go_binary_nobundle`, and using that rule for the specific `//cmd/{server,frontend}:{server,frontend}_nobundle` binary rules (along with the relevant `oci_{image,tarball}` etc rules to consume it).
## Test plan
- Integration tests in CI still work
- `bazel cquery 'kind("js_library", deps(//cmd/frontend:image_nobundle))'`, `bazel cquery 'kind("js_library", deps(//cmd/server:image_nobundle))'`, ``bazel cquery 'kind("js_library", deps(//testing:backend_integration_test))'` etc all return empty set
- Release test with marker in web bundle to ensure released images contain the web bundle via `sg release run test --version 5.4.2` (commenting out other tests for brevity)
2024-05-28 13:32:48 +00:00
server_image = " //cmd/server:no_client_bundle.image_tarball " ,
Switch to OCI/Wolfi based image (#52693)
This PR ships our freshly rewritten container images built with
rules_oci and Wolfi, which for now will only be used on S2.
*What is this about*
This work is the conjunction of [hardening container
images](https://github.com/orgs/sourcegraph/projects/302?pane=issue&itemId=25019223)
and fully building our container images with Bazel.
* All base images are now distroless, based on Wolfi, meaning we fully
control every little package version and we won't be subject anymore to
Alpine maintainers dropping a postgres version for example.
* Container images are now built with `rules_oci`, meaning we don't have
Dockerfile anymore, but instead created through [Bazel
rules](https://sourcegraph.sourcegraph.com/github.com/sourcegraph/sourcegraph@bzl/oci_wolfi/-/blob/enterprise/cmd/gitserver/BUILD.bazel).
Don't be scared, while this will look a bit strange to you at first,
it's much saner and simpler to do than our Dockerfiles and their muddy
shell scripts calling themselves in cascade.
:spiral_note_pad: *Plan*:
*1/ (NOW) We merge our branch on `main` today, here is what it does
change for you :point_down::skin-tone-3::*
* On `main`:
* It will introduce a new job on `main` _Bazel Push_, which will push
those new images on our registries with all tags prefixed by `bazel-`.
* These new images will be picked up by S2 and S2 only.
* The existing jobs building docker images and pushing them will stay in
place until we have QA'ed them enough and are confident to roll them out
on Dotcom.
* Because we'll be building both images, there will be more jobs running
on `main`, but this should not affect the wall clock time.
* On all branches (so your PRs and `main`)
* The _Bazel Test_ job will now run: Backend Integration Tests, E2E
Tests and CodeIntel QA
* This will increase the duration of your test jobs in PRs, but as we
haven't removed yet the `sg lint` step, it should not affect too much
the wall clock time of your PRs.
* But it will also increase your confidence toward your changes, as the
coverage will vastly increased compared to before.
* If you have ongoing branches which are affecting the docker images
(like adding a new binary, like the recent `scip-tags`, reach us out on
#job-fair-bazel so we can help you to port your changes. It's much much
simpler than before, but it's going to be unfamiliar to you).
* If something goes awfully wrong, we'll rollback and update this
thread.
*2/ (EOW / Early next week) Once we're confident enough with what we saw
on S2, we'll roll the new images on Dotcom.*
* After the first successful deploy and a few sanity checks, we will
drop the old images building jobs.
* At this point, we'll reach out to all TLs asking for their help to
exercise all features of our product to ensure we catch any potential
breakage.
## Test plan
<!-- All pull requests REQUIRE a test plan:
https://docs.sourcegraph.com/dev/background-information/testing_principles
-->
* We tested our new images on `scale-testing` and it worked.
* The new container building rules comes with _container tests_ which
ensures that produced images are containing and configured with what
should be in there:
[example](https://sourcegraph.sourcegraph.com/github.com/sourcegraph/sourcegraph@bzl/oci_wolfi/-/blob/enterprise/cmd/gitserver/image_test.yaml)
.
---------
Co-authored-by: Dave Try <davetry@gmail.com>
Co-authored-by: Will Dollman <will.dollman@sourcegraph.com>
2023-06-02 10:12:52 +00:00
)
2024-01-17 14:45:31 +00:00
# The perforce subrepo permission testing is really heayv, as it requires to add a new repo,
# clone it, index it, etc... Therefore a crude yet effective optimization here is to separate
# those tests from the rest of the backeng integration tests, and instead run them in isolation,
# with the same exact runner and setup.
#
# But because Bazel tests targets are executed concurrently, even if the duration is ultimately the# same, if not even more due to the additional overhead of spawning a Sourcegraph server, the wall
# clock time is ultimately much better, because they're executed concurrently.
server_integration_test (
name = " backend_integration_test_p4 " ,
timeout = " long " ,
args = [
" $(location //dev/gqltest:gqltest_test) " , # actual test
] ,
data = [
feat/bazel: `//cmd/{frontend,server}` targets that don't include client bundle for backend integration tests (#62877)
Currently, all backend integration tests transitively depend on the client bundle. This results in rebuilds of the closure, and a (likely) cache miss on the test, when modifying any client-side files.
Given that the client bundle isnt needed for these tests, we can create targets that don't include the client bundle in its transitive closure, which should in theory improve the cache hit for backend integration tests by not having client side changes invalidate it. This would also be beneficial in local env, to keep frontend rebuilds down
To do this, we still need to create a `web.manifest.json` file due to some unfortunate requirement on that file existing as part of initializing the sourcegraph instance. For this I just create an empty json file, `select` this instead of the client bundle target in client/web/dist/BUILD.bazel based on a `//:integration_testing_enabled` config setting, and creating a `go_binary`-wrapping bazel rule + macro that automatically applies a transition to set this to true `go_binary_nobundle`, and using that rule for the specific `//cmd/{server,frontend}:{server,frontend}_nobundle` binary rules (along with the relevant `oci_{image,tarball}` etc rules to consume it).
## Test plan
- Integration tests in CI still work
- `bazel cquery 'kind("js_library", deps(//cmd/frontend:image_nobundle))'`, `bazel cquery 'kind("js_library", deps(//cmd/server:image_nobundle))'`, ``bazel cquery 'kind("js_library", deps(//testing:backend_integration_test))'` etc all return empty set
- Release test with marker in web bundle to ensure released images contain the web bundle via `sg release run test --version 5.4.2` (commenting out other tests for brevity)
2024-05-28 13:32:48 +00:00
" //cmd/server:no_client_bundle.image_tarball " ,
2024-01-17 14:45:31 +00:00
" //dev/authtest:authtest_test " ,
" //dev/gqltest:gqltest_test " ,
] ,
env_inherit = [
" AWS_ACCESS_KEY_ID " ,
" AWS_CODE_COMMIT_PASSWORD " ,
" AWS_CODE_COMMIT_USERNAME " ,
" AWS_SECRET_ACCESS_KEY " ,
" AZURE_DEVOPS_TOKEN " ,
" AZURE_DEVOPS_USERNAME " ,
" BITBUCKET_SERVER_TOKEN " ,
" BITBUCKET_SERVER_URL " ,
" BITBUCKET_SERVER_USERNAME " ,
" GHE_GITHUB_TOKEN " ,
" PERFORCE_PASSWORD " ,
" PERFORCE_PORT " ,
" PERFORCE_USER " ,
" SOURCEGRAPH_LICENSE_GENERATION_KEY " ,
" SOURCEGRAPH_LICENSE_KEY " ,
] ,
port = " 7082 " ,
runner_src = " :backend_integration_test_p4.sh " ,
feat/bazel: `//cmd/{frontend,server}` targets that don't include client bundle for backend integration tests (#62877)
Currently, all backend integration tests transitively depend on the client bundle. This results in rebuilds of the closure, and a (likely) cache miss on the test, when modifying any client-side files.
Given that the client bundle isnt needed for these tests, we can create targets that don't include the client bundle in its transitive closure, which should in theory improve the cache hit for backend integration tests by not having client side changes invalidate it. This would also be beneficial in local env, to keep frontend rebuilds down
To do this, we still need to create a `web.manifest.json` file due to some unfortunate requirement on that file existing as part of initializing the sourcegraph instance. For this I just create an empty json file, `select` this instead of the client bundle target in client/web/dist/BUILD.bazel based on a `//:integration_testing_enabled` config setting, and creating a `go_binary`-wrapping bazel rule + macro that automatically applies a transition to set this to true `go_binary_nobundle`, and using that rule for the specific `//cmd/{server,frontend}:{server,frontend}_nobundle` binary rules (along with the relevant `oci_{image,tarball}` etc rules to consume it).
## Test plan
- Integration tests in CI still work
- `bazel cquery 'kind("js_library", deps(//cmd/frontend:image_nobundle))'`, `bazel cquery 'kind("js_library", deps(//cmd/server:image_nobundle))'`, ``bazel cquery 'kind("js_library", deps(//testing:backend_integration_test))'` etc all return empty set
- Release test with marker in web bundle to ensure released images contain the web bundle via `sg release run test --version 5.4.2` (commenting out other tests for brevity)
2024-05-28 13:32:48 +00:00
server_image = " //cmd/server:no_client_bundle.image_tarball " ,
2024-05-16 14:51:16 +00:00
tags = [ TAG_PLATFORM_SOURCE ] ,
2024-01-17 14:45:31 +00:00
)
2023-06-19 17:44:01 +00:00
server_integration_test (
Switch to OCI/Wolfi based image (#52693)
This PR ships our freshly rewritten container images built with
rules_oci and Wolfi, which for now will only be used on S2.
*What is this about*
This work is the conjunction of [hardening container
images](https://github.com/orgs/sourcegraph/projects/302?pane=issue&itemId=25019223)
and fully building our container images with Bazel.
* All base images are now distroless, based on Wolfi, meaning we fully
control every little package version and we won't be subject anymore to
Alpine maintainers dropping a postgres version for example.
* Container images are now built with `rules_oci`, meaning we don't have
Dockerfile anymore, but instead created through [Bazel
rules](https://sourcegraph.sourcegraph.com/github.com/sourcegraph/sourcegraph@bzl/oci_wolfi/-/blob/enterprise/cmd/gitserver/BUILD.bazel).
Don't be scared, while this will look a bit strange to you at first,
it's much saner and simpler to do than our Dockerfiles and their muddy
shell scripts calling themselves in cascade.
:spiral_note_pad: *Plan*:
*1/ (NOW) We merge our branch on `main` today, here is what it does
change for you :point_down::skin-tone-3::*
* On `main`:
* It will introduce a new job on `main` _Bazel Push_, which will push
those new images on our registries with all tags prefixed by `bazel-`.
* These new images will be picked up by S2 and S2 only.
* The existing jobs building docker images and pushing them will stay in
place until we have QA'ed them enough and are confident to roll them out
on Dotcom.
* Because we'll be building both images, there will be more jobs running
on `main`, but this should not affect the wall clock time.
* On all branches (so your PRs and `main`)
* The _Bazel Test_ job will now run: Backend Integration Tests, E2E
Tests and CodeIntel QA
* This will increase the duration of your test jobs in PRs, but as we
haven't removed yet the `sg lint` step, it should not affect too much
the wall clock time of your PRs.
* But it will also increase your confidence toward your changes, as the
coverage will vastly increased compared to before.
* If you have ongoing branches which are affecting the docker images
(like adding a new binary, like the recent `scip-tags`, reach us out on
#job-fair-bazel so we can help you to port your changes. It's much much
simpler than before, but it's going to be unfamiliar to you).
* If something goes awfully wrong, we'll rollback and update this
thread.
*2/ (EOW / Early next week) Once we're confident enough with what we saw
on S2, we'll roll the new images on Dotcom.*
* After the first successful deploy and a few sanity checks, we will
drop the old images building jobs.
* At this point, we'll reach out to all TLs asking for their help to
exercise all features of our product to ensure we catch any potential
breakage.
## Test plan
<!-- All pull requests REQUIRE a test plan:
https://docs.sourcegraph.com/dev/background-information/testing_principles
-->
* We tested our new images on `scale-testing` and it worked.
* The new container building rules comes with _container tests_ which
ensures that produced images are containing and configured with what
should be in there:
[example](https://sourcegraph.sourcegraph.com/github.com/sourcegraph/sourcegraph@bzl/oci_wolfi/-/blob/enterprise/cmd/gitserver/image_test.yaml)
.
---------
Co-authored-by: Dave Try <davetry@gmail.com>
Co-authored-by: Will Dollman <will.dollman@sourcegraph.com>
2023-06-02 10:12:52 +00:00
name = " codeintel_integration_test " ,
args = [
" $(location //internal/cmd/init-sg) " ,
2023-08-29 16:08:47 +00:00
" $(rlocationpath //dev/tools:src-cli) " ,
Switch to OCI/Wolfi based image (#52693)
This PR ships our freshly rewritten container images built with
rules_oci and Wolfi, which for now will only be used on S2.
*What is this about*
This work is the conjunction of [hardening container
images](https://github.com/orgs/sourcegraph/projects/302?pane=issue&itemId=25019223)
and fully building our container images with Bazel.
* All base images are now distroless, based on Wolfi, meaning we fully
control every little package version and we won't be subject anymore to
Alpine maintainers dropping a postgres version for example.
* Container images are now built with `rules_oci`, meaning we don't have
Dockerfile anymore, but instead created through [Bazel
rules](https://sourcegraph.sourcegraph.com/github.com/sourcegraph/sourcegraph@bzl/oci_wolfi/-/blob/enterprise/cmd/gitserver/BUILD.bazel).
Don't be scared, while this will look a bit strange to you at first,
it's much saner and simpler to do than our Dockerfiles and their muddy
shell scripts calling themselves in cascade.
:spiral_note_pad: *Plan*:
*1/ (NOW) We merge our branch on `main` today, here is what it does
change for you :point_down::skin-tone-3::*
* On `main`:
* It will introduce a new job on `main` _Bazel Push_, which will push
those new images on our registries with all tags prefixed by `bazel-`.
* These new images will be picked up by S2 and S2 only.
* The existing jobs building docker images and pushing them will stay in
place until we have QA'ed them enough and are confident to roll them out
on Dotcom.
* Because we'll be building both images, there will be more jobs running
on `main`, but this should not affect the wall clock time.
* On all branches (so your PRs and `main`)
* The _Bazel Test_ job will now run: Backend Integration Tests, E2E
Tests and CodeIntel QA
* This will increase the duration of your test jobs in PRs, but as we
haven't removed yet the `sg lint` step, it should not affect too much
the wall clock time of your PRs.
* But it will also increase your confidence toward your changes, as the
coverage will vastly increased compared to before.
* If you have ongoing branches which are affecting the docker images
(like adding a new binary, like the recent `scip-tags`, reach us out on
#job-fair-bazel so we can help you to port your changes. It's much much
simpler than before, but it's going to be unfamiliar to you).
* If something goes awfully wrong, we'll rollback and update this
thread.
*2/ (EOW / Early next week) Once we're confident enough with what we saw
on S2, we'll roll the new images on Dotcom.*
* After the first successful deploy and a few sanity checks, we will
drop the old images building jobs.
* At this point, we'll reach out to all TLs asking for their help to
exercise all features of our product to ensure we catch any potential
breakage.
## Test plan
<!-- All pull requests REQUIRE a test plan:
https://docs.sourcegraph.com/dev/background-information/testing_principles
-->
* We tested our new images on `scale-testing` and it worked.
* The new container building rules comes with _container tests_ which
ensures that produced images are containing and configured with what
should be in there:
[example](https://sourcegraph.sourcegraph.com/github.com/sourcegraph/sourcegraph@bzl/oci_wolfi/-/blob/enterprise/cmd/gitserver/image_test.yaml)
.
---------
Co-authored-by: Dave Try <davetry@gmail.com>
Co-authored-by: Will Dollman <will.dollman@sourcegraph.com>
2023-06-02 10:12:52 +00:00
" $(location //dev/codeintel-qa/cmd/download) " ,
" $(location //dev/codeintel-qa/cmd/clear) " ,
" $(location //dev/codeintel-qa/cmd/upload) " ,
" $(location //dev/codeintel-qa/cmd/query) " ,
2024-01-17 11:39:33 +00:00
" $(location //testing/code-intel:repos.json) " ,
Switch to OCI/Wolfi based image (#52693)
This PR ships our freshly rewritten container images built with
rules_oci and Wolfi, which for now will only be used on S2.
*What is this about*
This work is the conjunction of [hardening container
images](https://github.com/orgs/sourcegraph/projects/302?pane=issue&itemId=25019223)
and fully building our container images with Bazel.
* All base images are now distroless, based on Wolfi, meaning we fully
control every little package version and we won't be subject anymore to
Alpine maintainers dropping a postgres version for example.
* Container images are now built with `rules_oci`, meaning we don't have
Dockerfile anymore, but instead created through [Bazel
rules](https://sourcegraph.sourcegraph.com/github.com/sourcegraph/sourcegraph@bzl/oci_wolfi/-/blob/enterprise/cmd/gitserver/BUILD.bazel).
Don't be scared, while this will look a bit strange to you at first,
it's much saner and simpler to do than our Dockerfiles and their muddy
shell scripts calling themselves in cascade.
:spiral_note_pad: *Plan*:
*1/ (NOW) We merge our branch on `main` today, here is what it does
change for you :point_down::skin-tone-3::*
* On `main`:
* It will introduce a new job on `main` _Bazel Push_, which will push
those new images on our registries with all tags prefixed by `bazel-`.
* These new images will be picked up by S2 and S2 only.
* The existing jobs building docker images and pushing them will stay in
place until we have QA'ed them enough and are confident to roll them out
on Dotcom.
* Because we'll be building both images, there will be more jobs running
on `main`, but this should not affect the wall clock time.
* On all branches (so your PRs and `main`)
* The _Bazel Test_ job will now run: Backend Integration Tests, E2E
Tests and CodeIntel QA
* This will increase the duration of your test jobs in PRs, but as we
haven't removed yet the `sg lint` step, it should not affect too much
the wall clock time of your PRs.
* But it will also increase your confidence toward your changes, as the
coverage will vastly increased compared to before.
* If you have ongoing branches which are affecting the docker images
(like adding a new binary, like the recent `scip-tags`, reach us out on
#job-fair-bazel so we can help you to port your changes. It's much much
simpler than before, but it's going to be unfamiliar to you).
* If something goes awfully wrong, we'll rollback and update this
thread.
*2/ (EOW / Early next week) Once we're confident enough with what we saw
on S2, we'll roll the new images on Dotcom.*
* After the first successful deploy and a few sanity checks, we will
drop the old images building jobs.
* At this point, we'll reach out to all TLs asking for their help to
exercise all features of our product to ensure we catch any potential
breakage.
## Test plan
<!-- All pull requests REQUIRE a test plan:
https://docs.sourcegraph.com/dev/background-information/testing_principles
-->
* We tested our new images on `scale-testing` and it worked.
* The new container building rules comes with _container tests_ which
ensures that produced images are containing and configured with what
should be in there:
[example](https://sourcegraph.sourcegraph.com/github.com/sourcegraph/sourcegraph@bzl/oci_wolfi/-/blob/enterprise/cmd/gitserver/image_test.yaml)
.
---------
Co-authored-by: Dave Try <davetry@gmail.com>
Co-authored-by: Will Dollman <will.dollman@sourcegraph.com>
2023-06-02 10:12:52 +00:00
] ,
data = [
feat/bazel: `//cmd/{frontend,server}` targets that don't include client bundle for backend integration tests (#62877)
Currently, all backend integration tests transitively depend on the client bundle. This results in rebuilds of the closure, and a (likely) cache miss on the test, when modifying any client-side files.
Given that the client bundle isnt needed for these tests, we can create targets that don't include the client bundle in its transitive closure, which should in theory improve the cache hit for backend integration tests by not having client side changes invalidate it. This would also be beneficial in local env, to keep frontend rebuilds down
To do this, we still need to create a `web.manifest.json` file due to some unfortunate requirement on that file existing as part of initializing the sourcegraph instance. For this I just create an empty json file, `select` this instead of the client bundle target in client/web/dist/BUILD.bazel based on a `//:integration_testing_enabled` config setting, and creating a `go_binary`-wrapping bazel rule + macro that automatically applies a transition to set this to true `go_binary_nobundle`, and using that rule for the specific `//cmd/{server,frontend}:{server,frontend}_nobundle` binary rules (along with the relevant `oci_{image,tarball}` etc rules to consume it).
## Test plan
- Integration tests in CI still work
- `bazel cquery 'kind("js_library", deps(//cmd/frontend:image_nobundle))'`, `bazel cquery 'kind("js_library", deps(//cmd/server:image_nobundle))'`, ``bazel cquery 'kind("js_library", deps(//testing:backend_integration_test))'` etc all return empty set
- Release test with marker in web bundle to ensure released images contain the web bundle via `sg release run test --version 5.4.2` (commenting out other tests for brevity)
2024-05-28 13:32:48 +00:00
" //cmd/server:no_client_bundle.image_tarball " ,
Switch to OCI/Wolfi based image (#52693)
This PR ships our freshly rewritten container images built with
rules_oci and Wolfi, which for now will only be used on S2.
*What is this about*
This work is the conjunction of [hardening container
images](https://github.com/orgs/sourcegraph/projects/302?pane=issue&itemId=25019223)
and fully building our container images with Bazel.
* All base images are now distroless, based on Wolfi, meaning we fully
control every little package version and we won't be subject anymore to
Alpine maintainers dropping a postgres version for example.
* Container images are now built with `rules_oci`, meaning we don't have
Dockerfile anymore, but instead created through [Bazel
rules](https://sourcegraph.sourcegraph.com/github.com/sourcegraph/sourcegraph@bzl/oci_wolfi/-/blob/enterprise/cmd/gitserver/BUILD.bazel).
Don't be scared, while this will look a bit strange to you at first,
it's much saner and simpler to do than our Dockerfiles and their muddy
shell scripts calling themselves in cascade.
:spiral_note_pad: *Plan*:
*1/ (NOW) We merge our branch on `main` today, here is what it does
change for you :point_down::skin-tone-3::*
* On `main`:
* It will introduce a new job on `main` _Bazel Push_, which will push
those new images on our registries with all tags prefixed by `bazel-`.
* These new images will be picked up by S2 and S2 only.
* The existing jobs building docker images and pushing them will stay in
place until we have QA'ed them enough and are confident to roll them out
on Dotcom.
* Because we'll be building both images, there will be more jobs running
on `main`, but this should not affect the wall clock time.
* On all branches (so your PRs and `main`)
* The _Bazel Test_ job will now run: Backend Integration Tests, E2E
Tests and CodeIntel QA
* This will increase the duration of your test jobs in PRs, but as we
haven't removed yet the `sg lint` step, it should not affect too much
the wall clock time of your PRs.
* But it will also increase your confidence toward your changes, as the
coverage will vastly increased compared to before.
* If you have ongoing branches which are affecting the docker images
(like adding a new binary, like the recent `scip-tags`, reach us out on
#job-fair-bazel so we can help you to port your changes. It's much much
simpler than before, but it's going to be unfamiliar to you).
* If something goes awfully wrong, we'll rollback and update this
thread.
*2/ (EOW / Early next week) Once we're confident enough with what we saw
on S2, we'll roll the new images on Dotcom.*
* After the first successful deploy and a few sanity checks, we will
drop the old images building jobs.
* At this point, we'll reach out to all TLs asking for their help to
exercise all features of our product to ensure we catch any potential
breakage.
## Test plan
<!-- All pull requests REQUIRE a test plan:
https://docs.sourcegraph.com/dev/background-information/testing_principles
-->
* We tested our new images on `scale-testing` and it worked.
* The new container building rules comes with _container tests_ which
ensures that produced images are containing and configured with what
should be in there:
[example](https://sourcegraph.sourcegraph.com/github.com/sourcegraph/sourcegraph@bzl/oci_wolfi/-/blob/enterprise/cmd/gitserver/image_test.yaml)
.
---------
Co-authored-by: Dave Try <davetry@gmail.com>
Co-authored-by: Will Dollman <will.dollman@sourcegraph.com>
2023-06-02 10:12:52 +00:00
" //dev/codeintel-qa/cmd/clear " ,
" //dev/codeintel-qa/cmd/download " ,
" //dev/codeintel-qa/cmd/query " ,
" //dev/codeintel-qa/cmd/upload " ,
2023-08-29 16:08:47 +00:00
" //dev/tools:src-cli " ,
Switch to OCI/Wolfi based image (#52693)
This PR ships our freshly rewritten container images built with
rules_oci and Wolfi, which for now will only be used on S2.
*What is this about*
This work is the conjunction of [hardening container
images](https://github.com/orgs/sourcegraph/projects/302?pane=issue&itemId=25019223)
and fully building our container images with Bazel.
* All base images are now distroless, based on Wolfi, meaning we fully
control every little package version and we won't be subject anymore to
Alpine maintainers dropping a postgres version for example.
* Container images are now built with `rules_oci`, meaning we don't have
Dockerfile anymore, but instead created through [Bazel
rules](https://sourcegraph.sourcegraph.com/github.com/sourcegraph/sourcegraph@bzl/oci_wolfi/-/blob/enterprise/cmd/gitserver/BUILD.bazel).
Don't be scared, while this will look a bit strange to you at first,
it's much saner and simpler to do than our Dockerfiles and their muddy
shell scripts calling themselves in cascade.
:spiral_note_pad: *Plan*:
*1/ (NOW) We merge our branch on `main` today, here is what it does
change for you :point_down::skin-tone-3::*
* On `main`:
* It will introduce a new job on `main` _Bazel Push_, which will push
those new images on our registries with all tags prefixed by `bazel-`.
* These new images will be picked up by S2 and S2 only.
* The existing jobs building docker images and pushing them will stay in
place until we have QA'ed them enough and are confident to roll them out
on Dotcom.
* Because we'll be building both images, there will be more jobs running
on `main`, but this should not affect the wall clock time.
* On all branches (so your PRs and `main`)
* The _Bazel Test_ job will now run: Backend Integration Tests, E2E
Tests and CodeIntel QA
* This will increase the duration of your test jobs in PRs, but as we
haven't removed yet the `sg lint` step, it should not affect too much
the wall clock time of your PRs.
* But it will also increase your confidence toward your changes, as the
coverage will vastly increased compared to before.
* If you have ongoing branches which are affecting the docker images
(like adding a new binary, like the recent `scip-tags`, reach us out on
#job-fair-bazel so we can help you to port your changes. It's much much
simpler than before, but it's going to be unfamiliar to you).
* If something goes awfully wrong, we'll rollback and update this
thread.
*2/ (EOW / Early next week) Once we're confident enough with what we saw
on S2, we'll roll the new images on Dotcom.*
* After the first successful deploy and a few sanity checks, we will
drop the old images building jobs.
* At this point, we'll reach out to all TLs asking for their help to
exercise all features of our product to ensure we catch any potential
breakage.
## Test plan
<!-- All pull requests REQUIRE a test plan:
https://docs.sourcegraph.com/dev/background-information/testing_principles
-->
* We tested our new images on `scale-testing` and it worked.
* The new container building rules comes with _container tests_ which
ensures that produced images are containing and configured with what
should be in there:
[example](https://sourcegraph.sourcegraph.com/github.com/sourcegraph/sourcegraph@bzl/oci_wolfi/-/blob/enterprise/cmd/gitserver/image_test.yaml)
.
---------
Co-authored-by: Dave Try <davetry@gmail.com>
Co-authored-by: Will Dollman <will.dollman@sourcegraph.com>
2023-06-02 10:12:52 +00:00
" //internal/cmd/init-sg " ,
2024-01-17 11:39:33 +00:00
" //testing/code-intel:repos.json " ,
Switch to OCI/Wolfi based image (#52693)
This PR ships our freshly rewritten container images built with
rules_oci and Wolfi, which for now will only be used on S2.
*What is this about*
This work is the conjunction of [hardening container
images](https://github.com/orgs/sourcegraph/projects/302?pane=issue&itemId=25019223)
and fully building our container images with Bazel.
* All base images are now distroless, based on Wolfi, meaning we fully
control every little package version and we won't be subject anymore to
Alpine maintainers dropping a postgres version for example.
* Container images are now built with `rules_oci`, meaning we don't have
Dockerfile anymore, but instead created through [Bazel
rules](https://sourcegraph.sourcegraph.com/github.com/sourcegraph/sourcegraph@bzl/oci_wolfi/-/blob/enterprise/cmd/gitserver/BUILD.bazel).
Don't be scared, while this will look a bit strange to you at first,
it's much saner and simpler to do than our Dockerfiles and their muddy
shell scripts calling themselves in cascade.
:spiral_note_pad: *Plan*:
*1/ (NOW) We merge our branch on `main` today, here is what it does
change for you :point_down::skin-tone-3::*
* On `main`:
* It will introduce a new job on `main` _Bazel Push_, which will push
those new images on our registries with all tags prefixed by `bazel-`.
* These new images will be picked up by S2 and S2 only.
* The existing jobs building docker images and pushing them will stay in
place until we have QA'ed them enough and are confident to roll them out
on Dotcom.
* Because we'll be building both images, there will be more jobs running
on `main`, but this should not affect the wall clock time.
* On all branches (so your PRs and `main`)
* The _Bazel Test_ job will now run: Backend Integration Tests, E2E
Tests and CodeIntel QA
* This will increase the duration of your test jobs in PRs, but as we
haven't removed yet the `sg lint` step, it should not affect too much
the wall clock time of your PRs.
* But it will also increase your confidence toward your changes, as the
coverage will vastly increased compared to before.
* If you have ongoing branches which are affecting the docker images
(like adding a new binary, like the recent `scip-tags`, reach us out on
#job-fair-bazel so we can help you to port your changes. It's much much
simpler than before, but it's going to be unfamiliar to you).
* If something goes awfully wrong, we'll rollback and update this
thread.
*2/ (EOW / Early next week) Once we're confident enough with what we saw
on S2, we'll roll the new images on Dotcom.*
* After the first successful deploy and a few sanity checks, we will
drop the old images building jobs.
* At this point, we'll reach out to all TLs asking for their help to
exercise all features of our product to ensure we catch any potential
breakage.
## Test plan
<!-- All pull requests REQUIRE a test plan:
https://docs.sourcegraph.com/dev/background-information/testing_principles
-->
* We tested our new images on `scale-testing` and it worked.
* The new container building rules comes with _container tests_ which
ensures that produced images are containing and configured with what
should be in there:
[example](https://sourcegraph.sourcegraph.com/github.com/sourcegraph/sourcegraph@bzl/oci_wolfi/-/blob/enterprise/cmd/gitserver/image_test.yaml)
.
---------
Co-authored-by: Dave Try <davetry@gmail.com>
Co-authored-by: Will Dollman <will.dollman@sourcegraph.com>
2023-06-02 10:12:52 +00:00
] ,
env = {
" TEST_USER_EMAIL " : " test@sourcegraph.com " ,
" TEST_USER_PASSWORD " : " supersecurepassword " ,
" SOURCEGRAPH_SUDO_USER " : " admin " ,
2023-08-29 16:08:47 +00:00
" HOME " : " . " ,
" __LOCAL__ " : " false " ,
Switch to OCI/Wolfi based image (#52693)
This PR ships our freshly rewritten container images built with
rules_oci and Wolfi, which for now will only be used on S2.
*What is this about*
This work is the conjunction of [hardening container
images](https://github.com/orgs/sourcegraph/projects/302?pane=issue&itemId=25019223)
and fully building our container images with Bazel.
* All base images are now distroless, based on Wolfi, meaning we fully
control every little package version and we won't be subject anymore to
Alpine maintainers dropping a postgres version for example.
* Container images are now built with `rules_oci`, meaning we don't have
Dockerfile anymore, but instead created through [Bazel
rules](https://sourcegraph.sourcegraph.com/github.com/sourcegraph/sourcegraph@bzl/oci_wolfi/-/blob/enterprise/cmd/gitserver/BUILD.bazel).
Don't be scared, while this will look a bit strange to you at first,
it's much saner and simpler to do than our Dockerfiles and their muddy
shell scripts calling themselves in cascade.
:spiral_note_pad: *Plan*:
*1/ (NOW) We merge our branch on `main` today, here is what it does
change for you :point_down::skin-tone-3::*
* On `main`:
* It will introduce a new job on `main` _Bazel Push_, which will push
those new images on our registries with all tags prefixed by `bazel-`.
* These new images will be picked up by S2 and S2 only.
* The existing jobs building docker images and pushing them will stay in
place until we have QA'ed them enough and are confident to roll them out
on Dotcom.
* Because we'll be building both images, there will be more jobs running
on `main`, but this should not affect the wall clock time.
* On all branches (so your PRs and `main`)
* The _Bazel Test_ job will now run: Backend Integration Tests, E2E
Tests and CodeIntel QA
* This will increase the duration of your test jobs in PRs, but as we
haven't removed yet the `sg lint` step, it should not affect too much
the wall clock time of your PRs.
* But it will also increase your confidence toward your changes, as the
coverage will vastly increased compared to before.
* If you have ongoing branches which are affecting the docker images
(like adding a new binary, like the recent `scip-tags`, reach us out on
#job-fair-bazel so we can help you to port your changes. It's much much
simpler than before, but it's going to be unfamiliar to you).
* If something goes awfully wrong, we'll rollback and update this
thread.
*2/ (EOW / Early next week) Once we're confident enough with what we saw
on S2, we'll roll the new images on Dotcom.*
* After the first successful deploy and a few sanity checks, we will
drop the old images building jobs.
* At this point, we'll reach out to all TLs asking for their help to
exercise all features of our product to ensure we catch any potential
breakage.
## Test plan
<!-- All pull requests REQUIRE a test plan:
https://docs.sourcegraph.com/dev/background-information/testing_principles
-->
* We tested our new images on `scale-testing` and it worked.
* The new container building rules comes with _container tests_ which
ensures that produced images are containing and configured with what
should be in there:
[example](https://sourcegraph.sourcegraph.com/github.com/sourcegraph/sourcegraph@bzl/oci_wolfi/-/blob/enterprise/cmd/gitserver/image_test.yaml)
.
---------
Co-authored-by: Dave Try <davetry@gmail.com>
Co-authored-by: Will Dollman <will.dollman@sourcegraph.com>
2023-06-02 10:12:52 +00:00
} ,
env_inherit = [
" AWS_ACCESS_KEY_ID " ,
" AWS_CODE_COMMIT_PASSWORD " ,
" AWS_CODE_COMMIT_USERNAME " ,
" AWS_SECRET_ACCESS_KEY " ,
" AZURE_DEVOPS_TOKEN " ,
" AZURE_DEVOPS_USERNAME " ,
" BITBUCKET_SERVER_TOKEN " ,
" BITBUCKET_SERVER_URL " ,
" BITBUCKET_SERVER_USERNAME " ,
" GITHUB_TOKEN " ,
" PERFORCE_PASSWORD " ,
" PERFORCE_PORT " ,
" PERFORCE_USER " ,
" SOURCEGRAPH_LICENSE_GENERATION_KEY " ,
" SOURCEGRAPH_LICENSE_KEY " ,
] ,
2023-07-21 14:00:53 +00:00
port = " 7083 " ,
2023-06-19 17:44:01 +00:00
runner_src = " :codeintel_integration_test.sh " ,
feat/bazel: `//cmd/{frontend,server}` targets that don't include client bundle for backend integration tests (#62877)
Currently, all backend integration tests transitively depend on the client bundle. This results in rebuilds of the closure, and a (likely) cache miss on the test, when modifying any client-side files.
Given that the client bundle isnt needed for these tests, we can create targets that don't include the client bundle in its transitive closure, which should in theory improve the cache hit for backend integration tests by not having client side changes invalidate it. This would also be beneficial in local env, to keep frontend rebuilds down
To do this, we still need to create a `web.manifest.json` file due to some unfortunate requirement on that file existing as part of initializing the sourcegraph instance. For this I just create an empty json file, `select` this instead of the client bundle target in client/web/dist/BUILD.bazel based on a `//:integration_testing_enabled` config setting, and creating a `go_binary`-wrapping bazel rule + macro that automatically applies a transition to set this to true `go_binary_nobundle`, and using that rule for the specific `//cmd/{server,frontend}:{server,frontend}_nobundle` binary rules (along with the relevant `oci_{image,tarball}` etc rules to consume it).
## Test plan
- Integration tests in CI still work
- `bazel cquery 'kind("js_library", deps(//cmd/frontend:image_nobundle))'`, `bazel cquery 'kind("js_library", deps(//cmd/server:image_nobundle))'`, ``bazel cquery 'kind("js_library", deps(//testing:backend_integration_test))'` etc all return empty set
- Release test with marker in web bundle to ensure released images contain the web bundle via `sg release run test --version 5.4.2` (commenting out other tests for brevity)
2024-05-28 13:32:48 +00:00
server_image = " //cmd/server:no_client_bundle.image_tarball " ,
Switch to OCI/Wolfi based image (#52693)
This PR ships our freshly rewritten container images built with
rules_oci and Wolfi, which for now will only be used on S2.
*What is this about*
This work is the conjunction of [hardening container
images](https://github.com/orgs/sourcegraph/projects/302?pane=issue&itemId=25019223)
and fully building our container images with Bazel.
* All base images are now distroless, based on Wolfi, meaning we fully
control every little package version and we won't be subject anymore to
Alpine maintainers dropping a postgres version for example.
* Container images are now built with `rules_oci`, meaning we don't have
Dockerfile anymore, but instead created through [Bazel
rules](https://sourcegraph.sourcegraph.com/github.com/sourcegraph/sourcegraph@bzl/oci_wolfi/-/blob/enterprise/cmd/gitserver/BUILD.bazel).
Don't be scared, while this will look a bit strange to you at first,
it's much saner and simpler to do than our Dockerfiles and their muddy
shell scripts calling themselves in cascade.
:spiral_note_pad: *Plan*:
*1/ (NOW) We merge our branch on `main` today, here is what it does
change for you :point_down::skin-tone-3::*
* On `main`:
* It will introduce a new job on `main` _Bazel Push_, which will push
those new images on our registries with all tags prefixed by `bazel-`.
* These new images will be picked up by S2 and S2 only.
* The existing jobs building docker images and pushing them will stay in
place until we have QA'ed them enough and are confident to roll them out
on Dotcom.
* Because we'll be building both images, there will be more jobs running
on `main`, but this should not affect the wall clock time.
* On all branches (so your PRs and `main`)
* The _Bazel Test_ job will now run: Backend Integration Tests, E2E
Tests and CodeIntel QA
* This will increase the duration of your test jobs in PRs, but as we
haven't removed yet the `sg lint` step, it should not affect too much
the wall clock time of your PRs.
* But it will also increase your confidence toward your changes, as the
coverage will vastly increased compared to before.
* If you have ongoing branches which are affecting the docker images
(like adding a new binary, like the recent `scip-tags`, reach us out on
#job-fair-bazel so we can help you to port your changes. It's much much
simpler than before, but it's going to be unfamiliar to you).
* If something goes awfully wrong, we'll rollback and update this
thread.
*2/ (EOW / Early next week) Once we're confident enough with what we saw
on S2, we'll roll the new images on Dotcom.*
* After the first successful deploy and a few sanity checks, we will
drop the old images building jobs.
* At this point, we'll reach out to all TLs asking for their help to
exercise all features of our product to ensure we catch any potential
breakage.
## Test plan
<!-- All pull requests REQUIRE a test plan:
https://docs.sourcegraph.com/dev/background-information/testing_principles
-->
* We tested our new images on `scale-testing` and it worked.
* The new container building rules comes with _container tests_ which
ensures that produced images are containing and configured with what
should be in there:
[example](https://sourcegraph.sourcegraph.com/github.com/sourcegraph/sourcegraph@bzl/oci_wolfi/-/blob/enterprise/cmd/gitserver/image_test.yaml)
.
---------
Co-authored-by: Dave Try <davetry@gmail.com>
Co-authored-by: Will Dollman <will.dollman@sourcegraph.com>
2023-06-02 10:12:52 +00:00
tags = [
2023-11-23 13:23:50 +00:00
" exclusive " ,
Switch to OCI/Wolfi based image (#52693)
This PR ships our freshly rewritten container images built with
rules_oci and Wolfi, which for now will only be used on S2.
*What is this about*
This work is the conjunction of [hardening container
images](https://github.com/orgs/sourcegraph/projects/302?pane=issue&itemId=25019223)
and fully building our container images with Bazel.
* All base images are now distroless, based on Wolfi, meaning we fully
control every little package version and we won't be subject anymore to
Alpine maintainers dropping a postgres version for example.
* Container images are now built with `rules_oci`, meaning we don't have
Dockerfile anymore, but instead created through [Bazel
rules](https://sourcegraph.sourcegraph.com/github.com/sourcegraph/sourcegraph@bzl/oci_wolfi/-/blob/enterprise/cmd/gitserver/BUILD.bazel).
Don't be scared, while this will look a bit strange to you at first,
it's much saner and simpler to do than our Dockerfiles and their muddy
shell scripts calling themselves in cascade.
:spiral_note_pad: *Plan*:
*1/ (NOW) We merge our branch on `main` today, here is what it does
change for you :point_down::skin-tone-3::*
* On `main`:
* It will introduce a new job on `main` _Bazel Push_, which will push
those new images on our registries with all tags prefixed by `bazel-`.
* These new images will be picked up by S2 and S2 only.
* The existing jobs building docker images and pushing them will stay in
place until we have QA'ed them enough and are confident to roll them out
on Dotcom.
* Because we'll be building both images, there will be more jobs running
on `main`, but this should not affect the wall clock time.
* On all branches (so your PRs and `main`)
* The _Bazel Test_ job will now run: Backend Integration Tests, E2E
Tests and CodeIntel QA
* This will increase the duration of your test jobs in PRs, but as we
haven't removed yet the `sg lint` step, it should not affect too much
the wall clock time of your PRs.
* But it will also increase your confidence toward your changes, as the
coverage will vastly increased compared to before.
* If you have ongoing branches which are affecting the docker images
(like adding a new binary, like the recent `scip-tags`, reach us out on
#job-fair-bazel so we can help you to port your changes. It's much much
simpler than before, but it's going to be unfamiliar to you).
* If something goes awfully wrong, we'll rollback and update this
thread.
*2/ (EOW / Early next week) Once we're confident enough with what we saw
on S2, we'll roll the new images on Dotcom.*
* After the first successful deploy and a few sanity checks, we will
drop the old images building jobs.
* At this point, we'll reach out to all TLs asking for their help to
exercise all features of our product to ensure we catch any potential
breakage.
## Test plan
<!-- All pull requests REQUIRE a test plan:
https://docs.sourcegraph.com/dev/background-information/testing_principles
-->
* We tested our new images on `scale-testing` and it worked.
* The new container building rules comes with _container tests_ which
ensures that produced images are containing and configured with what
should be in there:
[example](https://sourcegraph.sourcegraph.com/github.com/sourcegraph/sourcegraph@bzl/oci_wolfi/-/blob/enterprise/cmd/gitserver/image_test.yaml)
.
---------
Co-authored-by: Dave Try <davetry@gmail.com>
Co-authored-by: Will Dollman <will.dollman@sourcegraph.com>
2023-06-02 10:12:52 +00:00
" manual " ,
2024-05-16 14:51:16 +00:00
TAG_PLATFORM_GRAPH ,
Switch to OCI/Wolfi based image (#52693)
This PR ships our freshly rewritten container images built with
rules_oci and Wolfi, which for now will only be used on S2.
*What is this about*
This work is the conjunction of [hardening container
images](https://github.com/orgs/sourcegraph/projects/302?pane=issue&itemId=25019223)
and fully building our container images with Bazel.
* All base images are now distroless, based on Wolfi, meaning we fully
control every little package version and we won't be subject anymore to
Alpine maintainers dropping a postgres version for example.
* Container images are now built with `rules_oci`, meaning we don't have
Dockerfile anymore, but instead created through [Bazel
rules](https://sourcegraph.sourcegraph.com/github.com/sourcegraph/sourcegraph@bzl/oci_wolfi/-/blob/enterprise/cmd/gitserver/BUILD.bazel).
Don't be scared, while this will look a bit strange to you at first,
it's much saner and simpler to do than our Dockerfiles and their muddy
shell scripts calling themselves in cascade.
:spiral_note_pad: *Plan*:
*1/ (NOW) We merge our branch on `main` today, here is what it does
change for you :point_down::skin-tone-3::*
* On `main`:
* It will introduce a new job on `main` _Bazel Push_, which will push
those new images on our registries with all tags prefixed by `bazel-`.
* These new images will be picked up by S2 and S2 only.
* The existing jobs building docker images and pushing them will stay in
place until we have QA'ed them enough and are confident to roll them out
on Dotcom.
* Because we'll be building both images, there will be more jobs running
on `main`, but this should not affect the wall clock time.
* On all branches (so your PRs and `main`)
* The _Bazel Test_ job will now run: Backend Integration Tests, E2E
Tests and CodeIntel QA
* This will increase the duration of your test jobs in PRs, but as we
haven't removed yet the `sg lint` step, it should not affect too much
the wall clock time of your PRs.
* But it will also increase your confidence toward your changes, as the
coverage will vastly increased compared to before.
* If you have ongoing branches which are affecting the docker images
(like adding a new binary, like the recent `scip-tags`, reach us out on
#job-fair-bazel so we can help you to port your changes. It's much much
simpler than before, but it's going to be unfamiliar to you).
* If something goes awfully wrong, we'll rollback and update this
thread.
*2/ (EOW / Early next week) Once we're confident enough with what we saw
on S2, we'll roll the new images on Dotcom.*
* After the first successful deploy and a few sanity checks, we will
drop the old images building jobs.
* At this point, we'll reach out to all TLs asking for their help to
exercise all features of our product to ensure we catch any potential
breakage.
## Test plan
<!-- All pull requests REQUIRE a test plan:
https://docs.sourcegraph.com/dev/background-information/testing_principles
-->
* We tested our new images on `scale-testing` and it worked.
* The new container building rules comes with _container tests_ which
ensures that produced images are containing and configured with what
should be in there:
[example](https://sourcegraph.sourcegraph.com/github.com/sourcegraph/sourcegraph@bzl/oci_wolfi/-/blob/enterprise/cmd/gitserver/image_test.yaml)
.
---------
Co-authored-by: Dave Try <davetry@gmail.com>
Co-authored-by: Will Dollman <will.dollman@sourcegraph.com>
2023-06-02 10:12:52 +00:00
] ,
)
2023-08-29 16:08:47 +00:00
# This one is bit perculiar, when running the codeintel-qa locally, we need to disable the sandbox
# so the runner can access the gcloud creds which are living in the users HOME, under
# .config/gcloud/application_default_credentials.json.
#
# It's more complicated to make the server_integration_test macro handle this automatically than it is to
# duplicate it, so we just duplicate and tweak it instead. It uses the same port, because those two tasks will
# never be run at the same time anyway. There is a safe guard in the original one that informs the user
# to use this variant instead.
server_integration_test (
name = " codeintel_integration_test_local " ,
2024-02-20 13:57:56 +00:00
args = [
" $(location //internal/cmd/init-sg) " ,
" $(rlocationpath //dev/tools:src-cli) " ,
" $(location //dev/codeintel-qa/cmd/download) " ,
" $(location //dev/codeintel-qa/cmd/clear) " ,
" $(location //dev/codeintel-qa/cmd/upload) " ,
" $(location //dev/codeintel-qa/cmd/query) " ,
" $(location //testing/code-intel:repos.json) " ,
] ,
data = [
feat/bazel: `//cmd/{frontend,server}` targets that don't include client bundle for backend integration tests (#62877)
Currently, all backend integration tests transitively depend on the client bundle. This results in rebuilds of the closure, and a (likely) cache miss on the test, when modifying any client-side files.
Given that the client bundle isnt needed for these tests, we can create targets that don't include the client bundle in its transitive closure, which should in theory improve the cache hit for backend integration tests by not having client side changes invalidate it. This would also be beneficial in local env, to keep frontend rebuilds down
To do this, we still need to create a `web.manifest.json` file due to some unfortunate requirement on that file existing as part of initializing the sourcegraph instance. For this I just create an empty json file, `select` this instead of the client bundle target in client/web/dist/BUILD.bazel based on a `//:integration_testing_enabled` config setting, and creating a `go_binary`-wrapping bazel rule + macro that automatically applies a transition to set this to true `go_binary_nobundle`, and using that rule for the specific `//cmd/{server,frontend}:{server,frontend}_nobundle` binary rules (along with the relevant `oci_{image,tarball}` etc rules to consume it).
## Test plan
- Integration tests in CI still work
- `bazel cquery 'kind("js_library", deps(//cmd/frontend:image_nobundle))'`, `bazel cquery 'kind("js_library", deps(//cmd/server:image_nobundle))'`, ``bazel cquery 'kind("js_library", deps(//testing:backend_integration_test))'` etc all return empty set
- Release test with marker in web bundle to ensure released images contain the web bundle via `sg release run test --version 5.4.2` (commenting out other tests for brevity)
2024-05-28 13:32:48 +00:00
" //cmd/server:no_client_bundle.image_tarball " ,
2024-02-20 13:57:56 +00:00
" //dev/codeintel-qa/cmd/clear " ,
" //dev/codeintel-qa/cmd/download " ,
" //dev/codeintel-qa/cmd/query " ,
" //dev/codeintel-qa/cmd/upload " ,
" //dev/tools:src-cli " ,
" //internal/cmd/init-sg " ,
" //testing/code-intel:repos.json " ,
] ,
2023-08-29 16:08:47 +00:00
env = {
" TEST_USER_EMAIL " : " test@sourcegraph.com " ,
" TEST_USER_PASSWORD " : " supersecurepassword " ,
" SOURCEGRAPH_SUDO_USER " : " admin " ,
" HOME " : " . " ,
" __LOCAL__ " : " true " ,
} ,
env_inherit = [
" AWS_ACCESS_KEY_ID " ,
" AWS_CODE_COMMIT_PASSWORD " ,
" AWS_CODE_COMMIT_USERNAME " ,
" AWS_SECRET_ACCESS_KEY " ,
" AZURE_DEVOPS_TOKEN " ,
" AZURE_DEVOPS_USERNAME " ,
" BITBUCKET_SERVER_TOKEN " ,
" BITBUCKET_SERVER_URL " ,
" BITBUCKET_SERVER_USERNAME " ,
" GITHUB_TOKEN " ,
" PERFORCE_PASSWORD " ,
" PERFORCE_PORT " ,
" PERFORCE_USER " ,
" SOURCEGRAPH_LICENSE_GENERATION_KEY " ,
" SOURCEGRAPH_LICENSE_KEY " ,
" GOOGLE_APPLICATION_CREDENTIALS " ,
] ,
port = " 7083 " ,
runner_src = " :codeintel_integration_test.sh " ,
feat/bazel: `//cmd/{frontend,server}` targets that don't include client bundle for backend integration tests (#62877)
Currently, all backend integration tests transitively depend on the client bundle. This results in rebuilds of the closure, and a (likely) cache miss on the test, when modifying any client-side files.
Given that the client bundle isnt needed for these tests, we can create targets that don't include the client bundle in its transitive closure, which should in theory improve the cache hit for backend integration tests by not having client side changes invalidate it. This would also be beneficial in local env, to keep frontend rebuilds down
To do this, we still need to create a `web.manifest.json` file due to some unfortunate requirement on that file existing as part of initializing the sourcegraph instance. For this I just create an empty json file, `select` this instead of the client bundle target in client/web/dist/BUILD.bazel based on a `//:integration_testing_enabled` config setting, and creating a `go_binary`-wrapping bazel rule + macro that automatically applies a transition to set this to true `go_binary_nobundle`, and using that rule for the specific `//cmd/{server,frontend}:{server,frontend}_nobundle` binary rules (along with the relevant `oci_{image,tarball}` etc rules to consume it).
## Test plan
- Integration tests in CI still work
- `bazel cquery 'kind("js_library", deps(//cmd/frontend:image_nobundle))'`, `bazel cquery 'kind("js_library", deps(//cmd/server:image_nobundle))'`, ``bazel cquery 'kind("js_library", deps(//testing:backend_integration_test))'` etc all return empty set
- Release test with marker in web bundle to ensure released images contain the web bundle via `sg release run test --version 5.4.2` (commenting out other tests for brevity)
2024-05-28 13:32:48 +00:00
server_image = " //cmd/server:no_client_bundle.image_tarball " ,
2023-08-29 16:08:47 +00:00
tags = [
" manual " ,
2023-10-30 13:45:15 +00:00
" no-sandbox " ,
2024-05-16 14:51:16 +00:00
TAG_PLATFORM_GRAPH ,
2023-08-29 16:08:47 +00:00
] ,
)
