Commit Graph

180 Commits

Author SHA1 Message Date
Matthew Manela
19ef2d973e
feat(search): Add Syntax Highlighting for Magik language (#62919)
Fixes GRAPH-621

 Add syntax highlighting for magik programming language
2024-06-06 16:49:07 -04:00
Varun Gandhi
d6842538b7
chore: Use new cmp package to simplify comparisons (#63029) 2024-06-03 19:44:48 +08:00
Varun Gandhi
e3c5a61e47
chore: Remove unused codeintel utils code (#63030) 2024-06-03 08:25:20 +02:00
Noah S-C
9b6ba7741e
bazel: transcribe test ownership to bazel tags (#62664) 2024-05-16 15:51:16 +01:00
Varun Gandhi
2dec00f228
chore: Use min/max builtins over math.Max/math.Min (#62233) 2024-04-29 14:27:28 -04:00
Matthew Manela
665c1475d4
Add syntax highlighting for Pkl files (#61478)
Added .pkl file syntax highlighting support
2024-04-04 12:28:35 -04:00
Varun Gandhi
ab9587a7d1
cleanup: Remove 'Dump' terminology from backend (#61131)
'Dump' really meant an uploads that is a 'completed' state,
and hence is eligible for various operations such as 'Find references'

This patch makes that explicit by removing the term 'Dump'
from the backend in favor of 'CompletedUpload'
2024-03-15 10:30:18 +08:00
Petri-Johan Last
0b5e7fd490
Replace all traditional for-loops (#60988) 2024-03-11 16:05:47 +02:00
Christoph Hegemann
e014f1f60a
Adds a little program to check potential space savings in symbol trie (#60703)
* adds a script to check potential space savings

* Simplify logic for trie merging (#60712)

* Simplify logic for trie merging

* Reformat

* Update dev/codeintel-merge-tries/README.md

Co-authored-by: Varun Gandhi <varun.gandhi@sourcegraph.com>

* Update dev/codeintel-merge-tries/README.md

Co-authored-by: Varun Gandhi <varun.gandhi@sourcegraph.com>

* moves merge tool

* splits out command to run to make copy easier

---------

Co-authored-by: Varun Gandhi <varun.gandhi@sourcegraph.com>
2024-02-23 11:16:17 +01:00
Varun Gandhi
900dfc3ffa
codeintel: Refine language detection for extensions (#59318)
This should reduce file fetching for binary files as well
as certain common extensions such as .md, .yaml and .ts
2024-01-04 23:52:54 +08:00
Varun Gandhi
b1f7ee0547
client: Obtain languages from server for file tree icons (#59234)
To determine the icon to show for a file, we need to know
the language. The code previously relied on the getFileInfo
API which did this using file extensions, leading to
incorrect results for header files with the '.h' extension,
which could have either C, C++ or Objective-C code.

Instead, we now get the language from the server,
where the go-enry library can optionally use the file
contents to return a more appropriate result.

Consequently, we can remove the getFileInfo API altogether.

Updates both the React and Svelte versions of the code.

Other changes in this patch:

- I've added icons for a bunch of languages which were missing.
- I've centralized the icon lists (and the bulk of the related logic) for
   both React and Svelte in one place.
2023-12-29 22:34:40 +05:30
Camden Cheek
b9a7e3b809
Cleanup: use new go 1.21 features (#58617)
Now that we're running go 1.21, we can take advantage of its new features. This cleans up a few utilites that have since been included in the standard library.
2023-11-28 13:49:38 -07:00
William Bezuidenhout
1ae6cc6bfd
logger: update log lib and remove use of description (#57690)
* log: remove use of description paramter in Scoped

* temporarily point to sglog branch

* bazel configure + gazelle

* remove additional use of description param

* use latest versions of zoekt,log,mountinfo

* go.mod
2023-10-18 17:29:08 +02:00
Eric Fritz
e04d225bf7
codenotify: dr. fritz is kil (#57313) 2023-10-03 16:19:55 +00:00
Auguste Rame
59ed6a1bd6
Let enry decide the filetype unconditionally for code highlighting (#56559)
* Let enry decide the filetype unconditionally for code highlighting

* Add Go tests

* Add Rust test, change lang detection

* Adjust tests, expose multilanguage detection logic

* Add CHANGELOG entry
2023-09-19 15:50:32 +00:00
Noah S-C
0a1f727c24
codeintel: allow configuring max concurrency in uploads v2 (#56277)
* codeintel: allow configuring max concurrency in uploads v2

* bazel configure
2023-08-29 15:06:14 -05:00
Jean-Hadrien Chabran
29f74f622b
Revert "codeintel: allow configuring max concurrency in uploads" (#56275)
Revert "codeintel: allow configuring max concurrency in uploads (#56117)"

This reverts commit de4d5471e3.
2023-08-29 12:41:45 -05:00
Noah S-C
de4d5471e3
codeintel: allow configuring max concurrency in uploads (#56117)
Our HTTP endpoints have a [hard-coded timelimit of 75s](https://sourcegraph.com/search?q=context:global+repo:%5Egithub%5C.com/sourcegraph/sourcegraph%24+75+*+lang:Go&patternType=standard&sm=1&groupBy=path). On slower connections with multipart uploads, it is possible for the bandwidth congestion of the competing upload parts to result in the upload failing due to any part taking longer than 75s to upload. 
This PR adds a new toggleable to configure the number of parts to be uploaded concurrently, so reduce the congestion/competing between parts.

## Test plan

Tested in combination with src-cli over at https://github.com/sourcegraph/src-cli/pull/1023
2023-08-28 21:56:00 +01:00
Varun Gandhi
8b5aeac553
cleanup: Delete code for index job hints (#55863)
The hints function is largely superseded by the generate function,
so let's just get rid of it. See prior discussion:
https://github.com/sourcegraph/sourcegraph/pull/55823#discussion_r1293242555

## Test plan

Covered by existing tests
2023-08-16 08:14:32 +08:00
Varun Gandhi
80a154c084
uploads: Simplify code for streaming indexes (#54567)
Previously, we had a bunch of channels and goroutines, which
aren't necessary, since the invoking goroutine anyways processes
the documents one-by-one and writes them into the database

## Test plan

Covered by existing tests
2023-07-24 05:51:32 +00:00
Jean-Hadrien Chabran
dd6a30389b
bzl: convert go generate + stringer to bazel (#54619)
Replaces an explicit `go:generate` call to `stringer` by its Bazel
counterpart.

This is a part of the effort to remove all `go:generate` statements and
to have them handled by Bazel. See tracked issue for more details.

## Test plan

<!-- All pull requests REQUIRE a test plan:
https://docs.sourcegraph.com/dev/background-information/testing_principles
-->

CI
2023-07-07 17:31:46 +02:00
Jean-Hadrien Chabran
a0bcbc4986
bzl: bazlify lsif/protocol stringer uses (#54517)
Creates a `go_stringer` macro and replace lsif code that used a
`go:generate` + stringer. Because this generates Go code, it's still
better to have the generated code versioned, as it doesn't require
everyone to configure `gopls` so it plays nicely with Bazel.

This also introduces a `//dev:write_all` target that calls all known
targets that write back to the source tree.

## Test plan

<!-- All pull requests REQUIRE a test plan:
https://docs.sourcegraph.com/dev/background-information/testing_principles
-->

CI
2023-07-03 15:01:17 +02:00
Auguste Rame
51b2260b61
scip-ctags: add Python, Java and C# (#52880)
- [x] Python
- [x] Java
- [x] C#
- [ ] Ruby (doing in #52957)

## Test plan

Snapshots.

---------

Co-authored-by: TJ DeVries <devries.timothyj@gmail.com>
2023-06-12 06:47:13 -04:00
Jean-Hadrien Chabran
3d36d34b3d
ci: re-enable race detection (#52776)
The previous approach to enable race detection was too radical and
accidently led to build our binaries with the race flage enabled, which
caused issues when building images down the line.

This happened because putting a `test --something` in bazelrc also sets
it on `build` which is absolutely not what we wanted. Usually folks get
this one working by having a `--stamp` config setting that fixes this
when releasing binaries, which we don't at this stage, as we're still
learning Bazel.

Luckily, this was caught swiftly. The current approach insteads takes a
more granular approach, which makes the `go_test` rule uses our own
variant, which injects the `race = "on"` attribute, but only on
`go_test`.


## Test plan

<!-- All pull requests REQUIRE a test plan:
https://docs.sourcegraph.com/dev/background-information/testing_principles
-->

CI, being a main-dry-run, this will cover the container building jobs,
which were the ones failing.

---------

Co-authored-by: Alex Ostrikov <alex.ostrikov@sourcegraph.com>
2023-06-05 20:41:47 +02:00
Auguste Rame
b40592f203
syntax-highlighter: initial ctags work (w/ musl scip-ctags) (#52643)
- Write Rust one-for-one protocol compatible replacement for
universal-ctags in JSON streaming mode (scip-ctags)
- Use tree-sitter to generate scip symbols, then emit those through
scip-ctags
- These symbols will be reused for Cody context 
- Ensure code is built with musl libc

## Test plan

Unit and snapshot tests in the Rust symbol generation code - verified
working in the symbols sidebar and symbol search for enabled languages.

---------

Co-authored-by: TJ DeVries <devries.timothyj@gmail.com>
Co-authored-by: William Bezuidenhout <william.bezuidenhout@sourcegraph.com>
Co-authored-by: Eric Fritz <eric@eric-fritz.com>
Co-authored-by: Eric Fritz <eric@sourcegraph.com>
Co-authored-by: Jean-Hadrien Chabran <jh@chabran.fr>
2023-05-30 17:19:39 -04:00
Eric Fritz
9554353f0b
Revert "syntax-highlighter: scip-ctags implementation (#50600)" (#52642)
This reverts commit 1ff6fb12d9.



## Test plan

<!-- All pull requests REQUIRE a test plan:
https://docs.sourcegraph.com/dev/background-information/testing_principles
-->
2023-05-30 18:29:57 +00:00
TJ DeVries
1ff6fb12d9
syntax-highlighter: scip-ctags implementation (#50600)
- Write Rust one-for-one protocol compatible replacement for
universal-ctags in JSON streaming mode (scip-ctags)
- Use tree-sitter to generate scip symbols, then emit those through
scip-ctags
- These symbols will be reused for Cody context 

Currently, only zig is enabled (so other languages should remain unaffected by this change).

We will add other languages throughout the next week as we're able to check them off.

## Test plan

Unit and snapshot tests in the Rust symbol generation code - verified
working in the symbols sidebar and symbol search for enabled languages.

---------

Co-authored-by: SuperAuguste <19855629+SuperAuguste@users.noreply.github.com>
Co-authored-by: William Bezuidenhout <william.bezuidenhout@sourcegraph.com>
Co-authored-by: Eric Fritz <eric@eric-fritz.com>
Co-authored-by: Eric Fritz <eric@sourcegraph.com>
2023-05-30 15:53:36 +00:00
Dave Try
2b8fa079f0
bazel: fix buf files (#49444)
fix protoc-gen-go version
2023-03-15 20:21:38 +00:00
Dave Try
293385d5dd
bazel: update timeouts to suppress warnings (#49399)
Updates all of the BUILD fields with timeouts to suppress warnings and
reduce log spam.


## Test plan

Green CI
2023-03-15 15:04:16 +02:00
Jean-Hadrien Chabran
7c55c2351e
bk: add soft-failing bazel jobs (#47601)
This PR introduces soft-failing bazel jobs which are running on all
builds, as way to gather experience and feedback on running Bazel in CI.

Changes in the go code are mostly about adding missing git configuration
bits that were failing in the sandbox.

## Test plan

<!-- All pull requests REQUIRE a test plan:
https://docs.sourcegraph.com/dev/background-information/testing_principles
-->

CI will show up bazel stuff, but ok to fail. 


![image](https://user-images.githubusercontent.com/10151/223689285-ba4f91f9-991e-42ba-b333-f8adfd503ee9.png)

---------

Co-authored-by: Greg Magolan <gmagolan@gmail.com>
2023-03-08 11:28:18 +00:00
Eric Fritz
91ea6973dc
codeintel: Unblock auto-indexing experience frontend (#48770) 2023-03-06 16:09:43 -06:00
Jean-Hadrien Chabran
bc6a791710
bazel: fix remaining backend tests (#47961)
This PR fixes the last round of backend tests that are getting in the
way.

While most fixes are pertaining to adapting the code to deal with the
sandbox, or adjusting targets in some cases, we had to ignore the tests
from the database stitch migration (see
123cb55005)
(cc @efritz) because they are making strong assumptions toward running
inside the Sourcegraph repository, which is kinda non trivial to adapt.

Another one, albeit non consequential is the disabling of some tests
peforming external requests, which I think should be deleted anyway
(c83d8ebba8)

Fix https://github.com/sourcegraph/sourcegraph/issues/46837
Fix https://github.com/sourcegraph/sourcegraph/issues/46856
Fix https://github.com/sourcegraph/sourcegraph/issues/46862
Fix https://github.com/sourcegraph/sourcegraph/issues/46864
Fix https://github.com/sourcegraph/sourcegraph/issues/46833
Fix https://github.com/sourcegraph/sourcegraph/issues/46835
Fix https://github.com/sourcegraph/sourcegraph/issues/46836
Fix https://github.com/sourcegraph/sourcegraph/issues/46847
Fix https://github.com/sourcegraph/sourcegraph/issues/46838
Fix https://github.com/sourcegraph/sourcegraph/issues/46843
Fix https://github.com/sourcegraph/sourcegraph/issues/46845
Fix https://github.com/sourcegraph/sourcegraph/issues/46849
Fix https://github.com/sourcegraph/sourcegraph/issues/46851
Fix https://github.com/sourcegraph/sourcegraph/issues/46855
Fix https://github.com/sourcegraph/sourcegraph/issues/46857
Fix https://github.com/sourcegraph/sourcegraph/issues/46858
Fix https://github.com/sourcegraph/sourcegraph/issues/46859
Fix https://github.com/sourcegraph/sourcegraph/issues/46861
Fix https://github.com/sourcegraph/sourcegraph/issues/46863
Fix https://github.com/sourcegraph/sourcegraph/issues/46865
Fix https://github.com/sourcegraph/sourcegraph/issues/46867
Fix https://github.com/sourcegraph/sourcegraph/issues/46853

## Test plan

<!-- All pull requests REQUIRE a test plan:
https://docs.sourcegraph.com/dev/background-information/testing_principles
-->

Ran locally with bazel.

---------

Co-authored-by: davejrt <davetry@gmail.com>
2023-03-01 17:03:01 +00:00
Jean-Hadrien Chabran
a115e0d08f
Migrate to autogold/v2 (needed by Bazel) (#47891)
Due to an issue with Autogold and Bazel, we have to migrate all
`autogold` tests to use the `v2`. This draft PR loses some annotations
describing the test cases, but I'll update that after once I have
validate that we can get those to work with Bazel.

Why is this a huge PR and not a couple of smaller ones: Because this PR
needed a few manual fixes, it was simpler to grind through the changes
after running comby locally. I have attached the config below so you can
review what I used instead.

@chwarwick the code insights tests were heavily relying on the attached
description on `autogold.Want` to provide context, I manually ported
them by adding a new `name string` field on the test cases.

Comby config I used: 

```
[update-imports]

match="\"github.com/hexops/autogold\""
rewrite="\"github.com/hexops/autogold/v2\""

[update-api-want]

match="autogold.Want(:[desc], :[v])"
rewrite="autogold.Expect(:[v])"

[update-api-equal]

match="autogold.Equal(:[v])"
rewrite="autogold.ExpectFile(:[v])"
```

```
$ comby -config ../autogold-comby.toml -matcher .go -exclude-dir vendor,node_modules -in-place
```

I then followed up with some manual fixes for the test cases that were
making the assumption that `want.Name()` exists.

## Test plan

<!-- All pull requests REQUIRE a test plan:
https://docs.sourcegraph.com/dev/background-information/testing_principles
-->

Green go tests.

---------

Co-authored-by: davejrt <davetry@gmail.com>
2023-02-21 10:37:13 +01:00
Dave Try
36e2831414
bazel: add filegroup for testdata (#47743)
Add filegroup group to glob all data  in `testdata` directory


## Test plan

`bazel build //...`
2023-02-16 15:51:03 +00:00
Jean-Hadrien Chabran
ffb0642497
bazel: update buildfiles (#47744)
Follow-up to https://github.com/sourcegraph/sourcegraph/pull/47540 which
updated the grpc gateway and required to update the patch.

It also adds the necessary `#keep` comments to avoid having gazelle
messing up the current buildfiles.

Release: this doesn't affect the app in anyway, so we're safe. 

## Test plan

<!-- All pull requests REQUIRE a test plan:
https://docs.sourcegraph.com/dev/background-information/testing_principles
-->

```
bazel build //internal/...
``` 

suceeded.
2023-02-16 16:32:59 +01:00
Jean-Hadrien Chabran
3fb9b64cfe
bzl: bazel test sourcegraph/lib (#47605)
Run all tests using bazel under `sourcegraph/lib`
2023-02-15 14:06:28 -05:00
David Veszelovszki
f23537a669
Housekeeping: Rename variables to avoid collisions with packages (#47179) 2023-01-31 16:28:43 +01:00
David Veszelovszki
a733d031f8
Housekeeping: Remove redundant stuff from Go code (#47104)
* Fix "Type can be omitted" warnings
* Fix "Redundant import alias" warnings
* Fix "Redundant type conversion" warnings
* Fix "Redundant parentheses" warnings
* Fix typo and case

Co-authored-by: Eric Fritz <eric@sourcegraph.com>
2023-01-30 20:39:40 +00:00
Eric Fritz
c69207f871
scip: Improve OOM conditions for LSIF migration (#47040) 2023-01-27 19:57:12 +00:00
Alex Ostrikov
6a3096bfbd
chore: remove some dead code found by linter. (#46940)
Test plan:
CI should pass.
2023-01-25 20:43:54 +04:00
Jean-Hadrien Chabran
bc5490c4bb
bazel: introduce build files for Go (#46770) 2023-01-23 14:00:01 +01:00
Eric Fritz
542e437ce9
codeintel: Fix reference moniker translation (#46391) 2023-01-12 13:57:50 -07:00
Eric Fritz
836b6711c5
codeintel: Fix too many definitions in LSIF translation (#46392) 2023-01-12 12:42:42 -06:00
Eric Fritz
caac3f3d20
codeintel: Fix LSIF -> SCIP range conversion (#46350) 2023-01-12 08:24:45 -06:00
Eric Fritz
04ab13c7e3
codeintel: Use project root when converting LSIF to SCIP (#46352) 2023-01-12 09:55:28 +00:00
Noah S-C
bfa074f6c6
codeintel: dont swallow errors in upload retry mechanism (#46281) 2023-01-10 14:53:49 +00:00
Cesar Jimenez
2d7912569c
[language-platform] infer name and version from scip index (#46174) 2023-01-06 14:57:37 +00:00
Eric Fritz
2d2c86bb8d
codeintel: Infer index name when converting LSIF to SCIP (#46091) 2023-01-05 11:57:02 -07:00
Noah S-C
8c996a77af
codeintel: first implementation of auto-indexing secrets (#45580) 2022-12-15 22:32:16 +00:00
Jean-Hadrien Chabran
e540a5a309
Clean linting errors from unparam (#45548)
* extsvc.gitlab: remove unused parameter and mock

* fixup

* Re-enable unparam

* sct: fix error handling + logs in bitbucket

* monitoring: remove unused param

* monitoring: remove unused param

* extsvc/gitlab: mute unused param (see #45336)

* sg: mute unused param

* sg/setup: fix additional checks not being called

* sg: fix unused params

* internal/user: remove unused error

* gitlab/oauth: remove nil+unused param

* webhooks: remove unused param DB

The return callback takes a DB, not the outer function.

* codeintel/autoindexing: mute unused parameter

* oobmigration: remove unused error

* database/event_logs: add missing error handling

* executors: remove unused context

* executors: remove unused err

* gitserver: remove unused param ShardId

* executors: remove another unused err

* fixup

* database/user_credentials: remove unused error

* codeintel/resolvers: remove unused ctx

* codeintel/uploads: removed unused param repositoryName

* codeintel/uploads: remove unused param repo

* fixup

* (sus) codeintel/uploads remove unused param default branch

* codenav/observability: add nolint pragma

* gitserver: remove unused return value from internal func

* add explanation to nolint

* codenav: add nolint pragma

* gitlab: remove unused context

* internal/repos: remove unused context

* cliutil: remove unused parameter

* cliutil: remove unused params/errors

* cliutil: drop another unused parameter

* repos/gitlab: remove unused parameter db

* sg/feedback: remove unused ctx

* migration: remove unused ctx parameter

* fixup cliutil ctx

* sg/src: remove unused out parameter

* codeintel/autoindexing: remove unused trace param

* rockskip: remove always nil return value

* insights: remove unused parameters/nil errs

* dotcom: remove unused logger param

* batches/httpapi: add nolint pragma for semantics

* extsvc: removed unused ctx + fix struct

* graphqlbackend: remove unused db parameter

* insights: remove unused logger

* executor: drop always nil err return val

* batches: remove always nil error

* oauth: remove unused db parameter

* batches: remove always nil error

* batches: remove always nil error

* insights: remove unused feature flags store

* insights(sus): remove unused parameter alert

* dotcom: remove unused logger

* dotcom: remove unused logger param

* insights: remove unused featureflagstore

* graphqlbackend/batches: removed unused ctx

* gitserver: simplify expression

* codeintel: remove always nil error

* batches: removed always nil error

* codeintel/lsif: remove another nil error
2022-12-13 11:18:25 +01:00