* Use `api.RepoName` and `reposource.PackageName` type aliases
Previously, we used strings for both types, which caused a regression
that got fixed in https://github.com/sourcegraph/sourcegraph/pull/38722.
This commit introduces type aliases to avoid regressions like that.
While implementing the refactoring I also found a bug in Python
packages, that's fixed by this commit.
* Revert functional difference that caused tests to fail
The versioned package syntax should include the version, but the golden
tests assume the version is not included and I want to minimize the
blast radius from the refactoring.
* Add diff for a TypeScript file to trigger CI job
* Rename job
* Delete unnecessary TypeScript file after validating CI job
* Remove unused "LSIF Typed" code
This code got moved to a separate repo github.com/sourcegraph/scip.
* Start using scip-typescript for auto-indexing
Previously, we used lsif-typescript for auto-indexing.
* Port scip-typescript CI upload job
* Add diff for a TypeScript file to trigger CI job
* Rename job
* Delete unnecessary TypeScript file after validating CI job
go1.18 has new warnings for Println usages which trigger go test to
fail:
codeintel/tools/lsif-repl/main.go:63:5: fmt.Println arg list ends with redundant newline
codeintel/tools/lsif-repl/main.go:145:4: fmt.Println arg list ends with redundant newline
I kept the same output, but made the newline explicit to avoid the
failure.
Test Plan: go test with go1.18 locally
* LSIF Typed: handle symbols with duplicate definitions
Fixes https://github.com/sourcegraph/lsif-typescript/issues/42
* Permit duplicate definitions with conflicting symbols
This feature is needed to reproduce real-world behavior
* Emit textDocument/references edge for definition ranges
Previously, we only added textDocument/definition edges to the
definition site meaning that "find references" did not show multiple
results for cases where the same symbol had multiple definitions.
* Use lsif-typescript instead of lsif-node for JS/TS indexing
This commit migrates from lsif-node to the new lsif-typescript indexer
that we have been working on for the past months. It's difficult to do
this migration incrementally so this PR touches on several different
components:
- the CI config for JS/TS indexing
- auto-inference for auto-indexing
- several bugfixes in LSIF Typed to enable cross-repo navigation
* Revert changes for Node logic
* Undo unwanted changes to yarn.lock
* Address review feedback
- Spelled as Npm or NPM depending on local case convention.
For local variables and in types, Npm should be preferred.
For fully uppercase spellings, NPM should be used.
- Default to spelling it as npm otherwise.
This matches the convention in the npm/cli code and the
recommendation in the npm/cli README.
This commit is large because it also adds a new DSL for testing LSIF behavior. Previously, it was difficult to write minimized examples that reproduce bugs related to code navigations. We typically implemented new LSIF features directly with a heavy-duty indexer like lsif-go or lsif-java. Now, we can implement new LSIF features with the testing DSL instead, which (hopefully) increases the development velocity at which we can iterate on new LSIF features.
grafana/regexp's speedup branch contains the stdlib regexp with multiple
patches applied to speed up regexp. The benchmark improvements are quite
impressive across the board. The series of patches are being considered
for inclusion in golang. I took a look over them, and they look legit
enough for us to start using it.
This commit replaces all uses of regexp with grafana/regexp. This
replacement was done via a simple string replace on
^\s*"regexp"
Followed by goimports -w across the repository. I then manually
inspected each result to ensure it was on an import.
* Filter paths passed to auto-index inference functions.
Passing only relevant indexing paths makes debugging easier,
and reduces the amount of work when we are auto-indexing projects
which do not contain all languages.
* Simplify implementation of inference.Patterns.
In the worst case, DefinitelyTyped/DefinitelyTyped has 8.4k tsconfig.json
files. If we tried to do auto-inference for it, we would end up with at
least a few hundred million path checks.
Let's not do that.
* Update LSIF spec based on offline discussions
- Move `Occurrence.Role` to a toplevel `SymbolRole` enum. This enum now
permits multiple roles to encode multiple bits of information such as
definitions, imports and write-access.
- Rename `Symbol` to `SymbolInformation` and replace `symbol_uri` with
`symbol`.
- Rename `Package` to `PackageInformation` and replace `package_uri`
with `package`.
- Rename `Highlight` to `SyntaxKind`.
- Use CamelCase for enum values and disable the linters that require
ALL_CAPS_SNAKE_CASE.
* Use `int32` field for `symbol_role` field.
* Use `Occurrence.override_documentation`
* Address review feedback
* Run the generator
* Remove `PackageInformation`
* Add test and generated to SymbolRole
This change is needed so that we no longer send raw git commands to
gitserver form outside of the vcs/git package.
We ended up just copying the implementation from
lib/codeintel/pathexistnce as the vcs/git package is not visible to that
package and we don't want out internal vcs/git package to rely on
packages in codeintel/lib.
* lsifstore: add ability to query API docs search index
Helps #21938
Signed-off-by: Stephen Gutekanst <stephen@sourcegraph.com>
Co-authored-by: Eric Fritz <eric@sourcegraph.com>
I noticed that `golangci-lint` only runs over the packages belonging to `github.com/sourcegraph/sourcegraph`. It doesn't run over `lib`, `dev/sg`, etc.
This PR changes the check script we use in CI to run golangci-lint over every project in our repository with a `go.mod` (which is what we do for `go test` in CI too).
That's the first commit. The second commit fixes all of the reported errors.
Previously, we inferred lsif-java indexing configuration for repos that
didn't have any Java or Scala source files. This caused indexing jobs to
fail when we shouldn't have indexed the repo in the first place. This
commit fixes that issue by requiring the repo to contain at least a
single Java or Scala file in order to infer the lsif-java indexing
configuration.
The refactoring for this commit also made it easy to fix another bug
where lsif-java would fail to index repos that contained both a toplevel
`pom.xml` file and `lsif-java.json` file (because it ambiguous which
build tool to use). Now, we use the `lsif-java index --build-tool=NAME` to
explicitly set the build tool to LSIF when there is an `lsif-java.json`
file.
I ran into issues setting up Sourcegraph on NixOS, because NixOS, like
some other distros, doesn't have a /bin/bash. We already use /usr/bin/env
in many of our scripts, so this improves consistency, too.
There was a race condition here: If the `src` channel of the queue
was closed and had no more data, but the buffer had not yet been flushed,
then the `dst` channel would be closed - dropping all `buf` values that
were still waiting to be written to the `dst` channel.
This seems to have been one of the issues leading to inconsistent usage
examples on API docs pages (you'd get no results in such instances.)
This file is clearly the most complex part of the API docs backend
implementation, and as such I am rethinking how it is structured and
how to test it. I'll send more PRs for this later.
Signed-off-by: Stephen Gutekanst <stephen@sourcegraph.com>
* API docs: LSIF: conversion: emit mapping of documentation result -> filepath
* migrations: add file_path column to lsif_data_documentation_mappings
* API docs: LSIF: write file_path mappings to DB
* API docs: LSIF: add documentation references/definitions to Tree type
* Fix issue where UI would show an error:
```
lsifStore.Ranges: ERROR: relation \"lsif_documentation_mappings\" does not exist (SQLSTATE 42P01)
```
Signed-off-by: Stephen Gutekanst <stephen@sourcegraph.com>
