Commit Graph

162 Commits

Author SHA1 Message Date
Varun Gandhi
8b5aeac553
cleanup: Delete code for index job hints (#55863)
The hints function is largely superseded by the generate function,
so let's just get rid of it. See prior discussion:
https://github.com/sourcegraph/sourcegraph/pull/55823#discussion_r1293242555

## Test plan

Covered by existing tests
2023-08-16 08:14:32 +08:00
Varun Gandhi
80a154c084
uploads: Simplify code for streaming indexes (#54567)
Previously, we had a bunch of channels and goroutines, which
aren't necessary, since the invoking goroutine anyways processes
the documents one-by-one and writes them into the database

## Test plan

Covered by existing tests
2023-07-24 05:51:32 +00:00
Jean-Hadrien Chabran
dd6a30389b
bzl: convert go generate + stringer to bazel (#54619)
Replaces an explicit `go:generate` call to `stringer` by its Bazel
counterpart.

This is a part of the effort to remove all `go:generate` statements and
to have them handled by Bazel. See tracked issue for more details.

## Test plan

<!-- All pull requests REQUIRE a test plan:
https://docs.sourcegraph.com/dev/background-information/testing_principles
-->

CI
2023-07-07 17:31:46 +02:00
Jean-Hadrien Chabran
a0bcbc4986
bzl: bazlify lsif/protocol stringer uses (#54517)
Creates a `go_stringer` macro and replace lsif code that used a
`go:generate` + stringer. Because this generates Go code, it's still
better to have the generated code versioned, as it doesn't require
everyone to configure `gopls` so it plays nicely with Bazel.

This also introduces a `//dev:write_all` target that calls all known
targets that write back to the source tree.

## Test plan

<!-- All pull requests REQUIRE a test plan:
https://docs.sourcegraph.com/dev/background-information/testing_principles
-->

CI
2023-07-03 15:01:17 +02:00
Auguste Rame
51b2260b61
scip-ctags: add Python, Java and C# (#52880)
- [x] Python
- [x] Java
- [x] C#
- [ ] Ruby (doing in #52957)

## Test plan

Snapshots.

---------

Co-authored-by: TJ DeVries <devries.timothyj@gmail.com>
2023-06-12 06:47:13 -04:00
Jean-Hadrien Chabran
3d36d34b3d
ci: re-enable race detection (#52776)
The previous approach to enable race detection was too radical and
accidently led to build our binaries with the race flage enabled, which
caused issues when building images down the line.

This happened because putting a `test --something` in bazelrc also sets
it on `build` which is absolutely not what we wanted. Usually folks get
this one working by having a `--stamp` config setting that fixes this
when releasing binaries, which we don't at this stage, as we're still
learning Bazel.

Luckily, this was caught swiftly. The current approach insteads takes a
more granular approach, which makes the `go_test` rule uses our own
variant, which injects the `race = "on"` attribute, but only on
`go_test`.


## Test plan

<!-- All pull requests REQUIRE a test plan:
https://docs.sourcegraph.com/dev/background-information/testing_principles
-->

CI, being a main-dry-run, this will cover the container building jobs,
which were the ones failing.

---------

Co-authored-by: Alex Ostrikov <alex.ostrikov@sourcegraph.com>
2023-06-05 20:41:47 +02:00
Auguste Rame
b40592f203
syntax-highlighter: initial ctags work (w/ musl scip-ctags) (#52643)
- Write Rust one-for-one protocol compatible replacement for
universal-ctags in JSON streaming mode (scip-ctags)
- Use tree-sitter to generate scip symbols, then emit those through
scip-ctags
- These symbols will be reused for Cody context 
- Ensure code is built with musl libc

## Test plan

Unit and snapshot tests in the Rust symbol generation code - verified
working in the symbols sidebar and symbol search for enabled languages.

---------

Co-authored-by: TJ DeVries <devries.timothyj@gmail.com>
Co-authored-by: William Bezuidenhout <william.bezuidenhout@sourcegraph.com>
Co-authored-by: Eric Fritz <eric@eric-fritz.com>
Co-authored-by: Eric Fritz <eric@sourcegraph.com>
Co-authored-by: Jean-Hadrien Chabran <jh@chabran.fr>
2023-05-30 17:19:39 -04:00
Eric Fritz
9554353f0b
Revert "syntax-highlighter: scip-ctags implementation (#50600)" (#52642)
This reverts commit 1ff6fb12d9.



## Test plan

<!-- All pull requests REQUIRE a test plan:
https://docs.sourcegraph.com/dev/background-information/testing_principles
-->
2023-05-30 18:29:57 +00:00
TJ DeVries
1ff6fb12d9
syntax-highlighter: scip-ctags implementation (#50600)
- Write Rust one-for-one protocol compatible replacement for
universal-ctags in JSON streaming mode (scip-ctags)
- Use tree-sitter to generate scip symbols, then emit those through
scip-ctags
- These symbols will be reused for Cody context 

Currently, only zig is enabled (so other languages should remain unaffected by this change).

We will add other languages throughout the next week as we're able to check them off.

## Test plan

Unit and snapshot tests in the Rust symbol generation code - verified
working in the symbols sidebar and symbol search for enabled languages.

---------

Co-authored-by: SuperAuguste <19855629+SuperAuguste@users.noreply.github.com>
Co-authored-by: William Bezuidenhout <william.bezuidenhout@sourcegraph.com>
Co-authored-by: Eric Fritz <eric@eric-fritz.com>
Co-authored-by: Eric Fritz <eric@sourcegraph.com>
2023-05-30 15:53:36 +00:00
Dave Try
2b8fa079f0
bazel: fix buf files (#49444)
fix protoc-gen-go version
2023-03-15 20:21:38 +00:00
Dave Try
293385d5dd
bazel: update timeouts to suppress warnings (#49399)
Updates all of the BUILD fields with timeouts to suppress warnings and
reduce log spam.


## Test plan

Green CI
2023-03-15 15:04:16 +02:00
Jean-Hadrien Chabran
7c55c2351e
bk: add soft-failing bazel jobs (#47601)
This PR introduces soft-failing bazel jobs which are running on all
builds, as way to gather experience and feedback on running Bazel in CI.

Changes in the go code are mostly about adding missing git configuration
bits that were failing in the sandbox.

## Test plan

<!-- All pull requests REQUIRE a test plan:
https://docs.sourcegraph.com/dev/background-information/testing_principles
-->

CI will show up bazel stuff, but ok to fail. 


![image](https://user-images.githubusercontent.com/10151/223689285-ba4f91f9-991e-42ba-b333-f8adfd503ee9.png)

---------

Co-authored-by: Greg Magolan <gmagolan@gmail.com>
2023-03-08 11:28:18 +00:00
Eric Fritz
91ea6973dc
codeintel: Unblock auto-indexing experience frontend (#48770) 2023-03-06 16:09:43 -06:00
Jean-Hadrien Chabran
bc6a791710
bazel: fix remaining backend tests (#47961)
This PR fixes the last round of backend tests that are getting in the
way.

While most fixes are pertaining to adapting the code to deal with the
sandbox, or adjusting targets in some cases, we had to ignore the tests
from the database stitch migration (see
123cb55005)
(cc @efritz) because they are making strong assumptions toward running
inside the Sourcegraph repository, which is kinda non trivial to adapt.

Another one, albeit non consequential is the disabling of some tests
peforming external requests, which I think should be deleted anyway
(c83d8ebba8)

Fix https://github.com/sourcegraph/sourcegraph/issues/46837
Fix https://github.com/sourcegraph/sourcegraph/issues/46856
Fix https://github.com/sourcegraph/sourcegraph/issues/46862
Fix https://github.com/sourcegraph/sourcegraph/issues/46864
Fix https://github.com/sourcegraph/sourcegraph/issues/46833
Fix https://github.com/sourcegraph/sourcegraph/issues/46835
Fix https://github.com/sourcegraph/sourcegraph/issues/46836
Fix https://github.com/sourcegraph/sourcegraph/issues/46847
Fix https://github.com/sourcegraph/sourcegraph/issues/46838
Fix https://github.com/sourcegraph/sourcegraph/issues/46843
Fix https://github.com/sourcegraph/sourcegraph/issues/46845
Fix https://github.com/sourcegraph/sourcegraph/issues/46849
Fix https://github.com/sourcegraph/sourcegraph/issues/46851
Fix https://github.com/sourcegraph/sourcegraph/issues/46855
Fix https://github.com/sourcegraph/sourcegraph/issues/46857
Fix https://github.com/sourcegraph/sourcegraph/issues/46858
Fix https://github.com/sourcegraph/sourcegraph/issues/46859
Fix https://github.com/sourcegraph/sourcegraph/issues/46861
Fix https://github.com/sourcegraph/sourcegraph/issues/46863
Fix https://github.com/sourcegraph/sourcegraph/issues/46865
Fix https://github.com/sourcegraph/sourcegraph/issues/46867
Fix https://github.com/sourcegraph/sourcegraph/issues/46853

## Test plan

<!-- All pull requests REQUIRE a test plan:
https://docs.sourcegraph.com/dev/background-information/testing_principles
-->

Ran locally with bazel.

---------

Co-authored-by: davejrt <davetry@gmail.com>
2023-03-01 17:03:01 +00:00
Jean-Hadrien Chabran
a115e0d08f
Migrate to autogold/v2 (needed by Bazel) (#47891)
Due to an issue with Autogold and Bazel, we have to migrate all
`autogold` tests to use the `v2`. This draft PR loses some annotations
describing the test cases, but I'll update that after once I have
validate that we can get those to work with Bazel.

Why is this a huge PR and not a couple of smaller ones: Because this PR
needed a few manual fixes, it was simpler to grind through the changes
after running comby locally. I have attached the config below so you can
review what I used instead.

@chwarwick the code insights tests were heavily relying on the attached
description on `autogold.Want` to provide context, I manually ported
them by adding a new `name string` field on the test cases.

Comby config I used: 

```
[update-imports]

match="\"github.com/hexops/autogold\""
rewrite="\"github.com/hexops/autogold/v2\""

[update-api-want]

match="autogold.Want(:[desc], :[v])"
rewrite="autogold.Expect(:[v])"

[update-api-equal]

match="autogold.Equal(:[v])"
rewrite="autogold.ExpectFile(:[v])"
```

```
$ comby -config ../autogold-comby.toml -matcher .go -exclude-dir vendor,node_modules -in-place
```

I then followed up with some manual fixes for the test cases that were
making the assumption that `want.Name()` exists.

## Test plan

<!-- All pull requests REQUIRE a test plan:
https://docs.sourcegraph.com/dev/background-information/testing_principles
-->

Green go tests.

---------

Co-authored-by: davejrt <davetry@gmail.com>
2023-02-21 10:37:13 +01:00
Dave Try
36e2831414
bazel: add filegroup for testdata (#47743)
Add filegroup group to glob all data  in `testdata` directory


## Test plan

`bazel build //...`
2023-02-16 15:51:03 +00:00
Jean-Hadrien Chabran
ffb0642497
bazel: update buildfiles (#47744)
Follow-up to https://github.com/sourcegraph/sourcegraph/pull/47540 which
updated the grpc gateway and required to update the patch.

It also adds the necessary `#keep` comments to avoid having gazelle
messing up the current buildfiles.

Release: this doesn't affect the app in anyway, so we're safe. 

## Test plan

<!-- All pull requests REQUIRE a test plan:
https://docs.sourcegraph.com/dev/background-information/testing_principles
-->

```
bazel build //internal/...
``` 

suceeded.
2023-02-16 16:32:59 +01:00
Jean-Hadrien Chabran
3fb9b64cfe
bzl: bazel test sourcegraph/lib (#47605)
Run all tests using bazel under `sourcegraph/lib`
2023-02-15 14:06:28 -05:00
David Veszelovszki
f23537a669
Housekeeping: Rename variables to avoid collisions with packages (#47179) 2023-01-31 16:28:43 +01:00
David Veszelovszki
a733d031f8
Housekeeping: Remove redundant stuff from Go code (#47104)
* Fix "Type can be omitted" warnings
* Fix "Redundant import alias" warnings
* Fix "Redundant type conversion" warnings
* Fix "Redundant parentheses" warnings
* Fix typo and case

Co-authored-by: Eric Fritz <eric@sourcegraph.com>
2023-01-30 20:39:40 +00:00
Eric Fritz
c69207f871
scip: Improve OOM conditions for LSIF migration (#47040) 2023-01-27 19:57:12 +00:00
Alex Ostrikov
6a3096bfbd
chore: remove some dead code found by linter. (#46940)
Test plan:
CI should pass.
2023-01-25 20:43:54 +04:00
Jean-Hadrien Chabran
bc5490c4bb
bazel: introduce build files for Go (#46770) 2023-01-23 14:00:01 +01:00
Eric Fritz
542e437ce9
codeintel: Fix reference moniker translation (#46391) 2023-01-12 13:57:50 -07:00
Eric Fritz
836b6711c5
codeintel: Fix too many definitions in LSIF translation (#46392) 2023-01-12 12:42:42 -06:00
Eric Fritz
caac3f3d20
codeintel: Fix LSIF -> SCIP range conversion (#46350) 2023-01-12 08:24:45 -06:00
Eric Fritz
04ab13c7e3
codeintel: Use project root when converting LSIF to SCIP (#46352) 2023-01-12 09:55:28 +00:00
Noah S-C
bfa074f6c6
codeintel: dont swallow errors in upload retry mechanism (#46281) 2023-01-10 14:53:49 +00:00
Cesar Jimenez
2d7912569c
[language-platform] infer name and version from scip index (#46174) 2023-01-06 14:57:37 +00:00
Eric Fritz
2d2c86bb8d
codeintel: Infer index name when converting LSIF to SCIP (#46091) 2023-01-05 11:57:02 -07:00
Noah S-C
8c996a77af
codeintel: first implementation of auto-indexing secrets (#45580) 2022-12-15 22:32:16 +00:00
Jean-Hadrien Chabran
e540a5a309
Clean linting errors from unparam (#45548)
* extsvc.gitlab: remove unused parameter and mock

* fixup

* Re-enable unparam

* sct: fix error handling + logs in bitbucket

* monitoring: remove unused param

* monitoring: remove unused param

* extsvc/gitlab: mute unused param (see #45336)

* sg: mute unused param

* sg/setup: fix additional checks not being called

* sg: fix unused params

* internal/user: remove unused error

* gitlab/oauth: remove nil+unused param

* webhooks: remove unused param DB

The return callback takes a DB, not the outer function.

* codeintel/autoindexing: mute unused parameter

* oobmigration: remove unused error

* database/event_logs: add missing error handling

* executors: remove unused context

* executors: remove unused err

* gitserver: remove unused param ShardId

* executors: remove another unused err

* fixup

* database/user_credentials: remove unused error

* codeintel/resolvers: remove unused ctx

* codeintel/uploads: removed unused param repositoryName

* codeintel/uploads: remove unused param repo

* fixup

* (sus) codeintel/uploads remove unused param default branch

* codenav/observability: add nolint pragma

* gitserver: remove unused return value from internal func

* add explanation to nolint

* codenav: add nolint pragma

* gitlab: remove unused context

* internal/repos: remove unused context

* cliutil: remove unused parameter

* cliutil: remove unused params/errors

* cliutil: drop another unused parameter

* repos/gitlab: remove unused parameter db

* sg/feedback: remove unused ctx

* migration: remove unused ctx parameter

* fixup cliutil ctx

* sg/src: remove unused out parameter

* codeintel/autoindexing: remove unused trace param

* rockskip: remove always nil return value

* insights: remove unused parameters/nil errs

* dotcom: remove unused logger param

* batches/httpapi: add nolint pragma for semantics

* extsvc: removed unused ctx + fix struct

* graphqlbackend: remove unused db parameter

* insights: remove unused logger

* executor: drop always nil err return val

* batches: remove always nil error

* oauth: remove unused db parameter

* batches: remove always nil error

* batches: remove always nil error

* insights: remove unused feature flags store

* insights(sus): remove unused parameter alert

* dotcom: remove unused logger

* dotcom: remove unused logger param

* insights: remove unused featureflagstore

* graphqlbackend/batches: removed unused ctx

* gitserver: simplify expression

* codeintel: remove always nil error

* batches: removed always nil error

* codeintel/lsif: remove another nil error
2022-12-13 11:18:25 +01:00
Cesar Jimenez
0f90de027d
[language-platform] adding route to check for SCIP availability (#45269) 2022-12-08 11:25:22 -07:00
Eric Fritz
e872ed6410
codeintel: Write SCIP data in uploads service (#45104) 2022-12-06 15:32:54 -06:00
Thorsten Ball
49c5aee840
Disable flaky TestConvertLSIF (#45252)
Disable flkay TestConvertLSIF
2022-12-06 12:18:17 +00:00
Eric Fritz
52fefe5292
codeintel: Add layer to translate raw/processed LSIF to SCIP (#45126) 2022-12-05 18:32:30 -06:00
Cesar Jimenez
f7970690c1
[language-platform] Code intelligence badge updates (#45015) 2022-12-02 13:28:42 -07:00
Eric Fritz
0a110a25c5
chore: Mirror Manager field in oobmigration package (#44978) 2022-11-30 18:01:09 -07:00
Eric Fritz
cdba72ae4d
codeintel: Add manager to monikers/package information (#44958) 2022-11-30 14:26:58 -06:00
Keegan Carruthers-Smith
27569d1fc7
all: run gofmt -s -w from 1.19 (#41629)
gofmt in go1.19 does a lot of reformating of godoc strings, mostly to
make them more consistent around lists.

Test Plan: CI
2022-09-13 07:44:06 +00:00
Noah S-C
30275c288b
codeintel: add dashboard for precide-code-intel-worker total in-progress upload size (#39690) 2022-08-11 21:43:14 +01:00
Noah S-C
abbe5e2cb4
codeintel: use io.SectionReader for codeintel uploading (#40198)
* codeintel: use io.SectionReader for codeintel uploading

* Update lib/codeintel/upload/upload.go
2022-08-10 13:33:31 +00:00
Cesar Jimenez
af80d54072
Moving transport layer functionality to the service layer (#39516) 2022-08-02 14:19:01 -04:00
Ólafur Páll Geirsson
2e5a366f52
Use api.RepoName and reposource.PackageName type aliases (#38730)
* Use `api.RepoName` and `reposource.PackageName` type aliases

Previously, we used strings for both types, which caused a regression
that got fixed in https://github.com/sourcegraph/sourcegraph/pull/38722.
This commit introduces type aliases to avoid regressions like that.

While implementing the refactoring I also found a bug in Python
packages, that's fixed by this commit.

* Revert functional difference that caused tests to fail

The versioned package syntax should include the version, but the golden
tests assume the version is not included and I want to minimize the
blast radius from the refactoring.
2022-07-14 05:47:30 +00:00
Robert Lin
95c6946b9d
lib/log: migrate to github.com/sourcegraph/log (#36834)
This change migrates lib/log to https://github.com/sourcegraph/log in the hopes of easing adoption for other services.
2022-06-08 15:02:38 -07:00
Jean-Hadrien Chabran
3ea3c84343
Update services to use the sentry sink (#36569)
This patches all services that were using the HTTP middleware to capture sentry errors to now use the sentry sink.
2022-06-08 11:17:13 -07:00
Robert Lin
b2779c2221
lib: remove all usages of log15 (#36259) 2022-05-30 17:53:05 -07:00
Ólafur Páll Geirsson
9dcdfb95d9
Remove unused LSIF Typed code (#35791)
* Add diff for a TypeScript file to trigger CI job

* Rename job

* Delete unnecessary TypeScript file after validating CI job

* Remove unused "LSIF Typed" code

This code got moved to a separate repo github.com/sourcegraph/scip.
2022-05-25 11:23:13 +00:00
Varun Gandhi
0facf872c1
Update docs and comments to refer to scip-typescript (#34097)
codeintel: Update docs + code for scip-typescript.

Co-authored-by: Ólafur Páll Geirsson <olafurpg@gmail.com>
2022-05-24 15:38:04 +00:00
Ólafur Páll Geirsson
862f0e87d8
Start using scip-typescript for auto-indexing (#35786)
* Start using scip-typescript for auto-indexing

Previously, we used lsif-typescript for auto-indexing.

* Port scip-typescript CI upload job

* Add diff for a TypeScript file to trigger CI job

* Rename job

* Delete unnecessary TypeScript file after validating CI job
2022-05-23 16:13:25 +02:00