e2e test: define base tst framework for easier scenario setup (#56774)

* e2e tst: add actions to create github org (#56775)

* add basic actions to create org

* e2e tst: add actions to create github users (#56776)

* add basic actions to create and get users

* fix adminUser

* add unique id to user email

* e2e tst: add actions to create github teams (#56777)

* add basic actions to manage teams

* e2e tst: add actions to create repos (#56778)

---------

Co-authored-by: Petri-Johan Last <petri.last@sourcegraph.com>
Co-authored-by: Jean-Hadrien Chabran <jean-hadrien.chabran@sourcegraph.com>
This commit is contained in:
William Bezuidenhout 2023-10-10 16:30:00 +02:00 committed by GitHub
parent 9d86a4f7d1
commit f533eadf59
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
15 changed files with 1540 additions and 0 deletions

View File

@ -0,0 +1,30 @@
load("//dev:go_defs.bzl", "go_test")
load("@io_bazel_rules_go//go:def.bzl", "go_library")
go_library(
name = "codehost_testing",
srcs = [
"github_client.go",
"org.go",
"repo.go",
"reporter.go",
"scenario.go",
"team.go",
"user.go",
"util.go",
],
importpath = "github.com/sourcegraph/sourcegraph/dev/codehost_testing",
visibility = ["//visibility:public"],
deps = [
"//dev/codehost_testing/config",
"//lib/errors",
"@com_github_google_go_github_v55//github",
"@com_github_google_uuid//:uuid",
],
)
go_test(
name = "codehost_testing_test",
srcs = ["scenario_test.go"],
embed = [":codehost_testing"],
)

View File

@ -0,0 +1,110 @@
# Codehost Testing Library
This library makes it easier to setup Codehost resources in a predicatable and reproducible manner. It accomplishes this by introducing the concept of a scenario.
A scenario describes the state a collection of related resources on a codehost must be in. A collection of resources can at it's most basic just be an Organisation or at it's most complex be an Organisation with various teams and repositories with varying permissions.
Supported Codehosts:
- GitHub
## Configuration
The following configuration is required by this library
```json
{
"github": {
"url": "https://path.codehost.org",
"adminUser": "dude",
"password": "whereismycar",
"token": "do_not_leak_me"
},
"sourcegraph": {
"url": "https://path.sourcegraph.org",
"user": "boi",
"password": "towers_of_hanoi",
"token": "do_not_leak_me_plz"
}
}
```
Additionally, the GitHub token is **required** to have the following scopes:
- admin:enterprise
- delete_repo
- repo
- site_admin
- user
- write:org
To load your configuration you can use `config.FromFile` as per the snippet below.
```golang
cfg, err := config.FromFile("config.json")
if err != nil {
t.Fatalf("error loading scenario config: %v\n", err)
}
```
## Usage
We first need to create a scenario. A scenario exposes various methods to add or alter the scenario. In the below snippet we create a scenario that describes the following:
- One Organization
- One normal user
- One Admin
- Two teams. One team with the normal user and the other team with the Admin
- Two repositories are forked. One repository is public while the other is private and only accessible by the private team.
```golang
scenario, err := scenario.NewGitHubScenario(t, *cfg)
if err != nil {
t.Fatalf("error creating scenario: %v\n", err)
}
org := scenario.CreateOrg("tst-org")
user := scenario.CreateUser("tst-user")
admin := scenario.GetAdmin()
org.AllowPrivateForks()
team := org.CreateTeam("team-1")
team.AddUser(user)
adminTeam := org.CreateTeam("team-admin")
adminTeam.AddUser(admin)
publicRepo := org.CreateRepoFork("sgtest/go-diff")
publicRepo.AddTeam(team)
privateRepo := org.CreateRepoFork("sgtest/private")
privateRepo.AddTeam(adminTeam)
```
Adding a resource to the scenario does not immediately create or alter it. Instead, when adding a resource to the scenario, what you are actually doing is telling the library that "I want this resource to exist with this particular make up". The scenario keeps track of how all these resources that should be created and altered as **Actions** to be applied sequentially. Thus when calling `scenario.CreateOrg` the Org isn't immediately created, instead, an action is added that \_will create it when the scenario is applied.
We can see what actions our scenario **plans** to apply by calling `Plan` as per the below snippet. `Plan` returns a string that prints out the action names that will be applied by the scenario.
```golang
fmt.Println(scenario.Plan())
```
As was mentioned before, since we haven't applied the scenario, nothing exists yet on the Codehost. To create the resources described by the scenario `Apply` has to be called. In the snippet below the verbosity of the scenario is increased so that we can see how and when the actions are applied.
```
scenario.SetVerbose()
if err := scenario.Apply(context.Background()); err != nil {
t.Fatalf("error applying scenario: %v", err)
}
```
When `Apply` is called on the scenario, the scenario will not only apply all the actions but will also register a corresponding cleanup method with `testing.T` to teardown all resources that will be created by this scenario. Thus, if anything fails, the resources that _have been_ created up and till that point, will be properly
cleaned up.
After the scenario has been successfully been applied, the corresponding Codehost resource can be retrieved by calling the `Get()` on the scenario resource. For example on the below snippet, the `github.Organization` is retrieved.
```golang
ghOrg, err := org.Get(ctx)
if err != nil {
t.Fatalf("failed to get Organzation: %v", err)
}
```
**IMPORTANT** Calling `Get()` on any scenario resource before the scenario has been applied will result in an error. **Get() will only return the Codehost resource if the scenario has been applied**.

View File

@ -0,0 +1,8 @@
load("@io_bazel_rules_go//go:def.bzl", "go_library")
go_library(
name = "config",
srcs = ["config.go"],
importpath = "github.com/sourcegraph/sourcegraph/dev/codehost_testing/config",
visibility = ["//visibility:public"],
)

View File

@ -0,0 +1,47 @@
package config
import (
"encoding/json"
"os"
)
// GitHub represents the GitHub client configuration to connect to a GitHub codehost
type GitHub struct {
URL string `json:"url"`
AdminUser string `json:"adminUser"`
Password string `json:"password"`
Token string `json:"token"`
}
type SourcegraphCfg struct {
URL string `json:"url"`
User string `json:"user"`
Password string `json:"password"`
Token string `json:"token"`
}
// Config represents the configuration which should be used when connecting to codehosts.
//
// Currently we only support connecting to GitHub.
type Config struct {
GitHub GitHub `json:"github"`
Sourcegraph SourcegraphCfg `json:"sourcegraph"`
}
// FromFile reads the configuration from the specified file. This method will return an error when we either fail
// to open a file or fail to decode the JSON into the Config struct.
func FromFile(filename string) (*Config, error) {
var c Config
fd, err := os.Open(filename)
if err != nil {
return nil, err
}
defer fd.Close()
if err := json.NewDecoder(fd).Decode(&c); err != nil {
return nil, err
}
return &c, nil
}

View File

@ -0,0 +1,10 @@
load("//dev:go_defs.bzl", "go_test")
go_test(
name = "example_test",
srcs = ["example_test.go"],
deps = [
"//dev/codehost_testing",
"//dev/codehost_testing/config",
],
)

View File

@ -0,0 +1,71 @@
package main
import (
"context"
"flag"
"fmt"
"os"
"testing"
scenario "github.com/sourcegraph/sourcegraph/dev/codehost_testing"
"github.com/sourcegraph/sourcegraph/dev/codehost_testing/config"
)
var runGithub bool
func TestGithubScenario(t *testing.T) {
if !runGithub {
t.Skip("`run.github` flag not provided - skipping github scenario as this is only an example test")
}
cfg, err := config.FromFile("config.json")
if err != nil {
t.Fatalf("error loading scenario config: %v\n", err)
}
scenario, err := scenario.NewGitHubScenario(t, *cfg)
if err != nil {
t.Fatalf("error creating scenario: %v\n", err)
}
org := scenario.CreateOrg("tst-org")
user := scenario.CreateUser("tst-user")
otherUser := scenario.CreateUser("other-user")
admin := scenario.GetAdmin()
org.AllowPrivateForks()
team := org.CreateTeam("team-1")
team.AddUser(user)
team.AddUser(otherUser)
adminTeam := org.CreateTeam("team-admin")
adminTeam.AddUser(admin)
publicRepo := org.CreateRepoFork("sgtest/go-diff")
publicRepo.AddTeam(team)
privateRepo := org.CreateRepoFork("sgtest/private")
privateRepo.AddTeam(adminTeam)
fmt.Println(scenario.Plan())
ctx := context.Background()
// Get the Organization WILL FAIL since the scenario has not been applied yet
_, err = org.Get(ctx)
if err != nil {
t.Logf("failed to get github.Organization since it hasn't been applied yet: %v", err)
}
scenario.SetVerbose()
if err := scenario.Apply(ctx); err != nil {
t.Fatalf("error applying scenario: %v", err)
}
// Get the Organization
ghOrg, err := org.Get(ctx)
if err != nil {
t.Fatalf("failed to get github.Organization: %v", err)
}
t.Logf("GitHub Organization: %s", ghOrg.GetLogin())
}
func TestMain(m *testing.M) {
flag.BoolVar(&runGithub, "run.github", false, "Run example github scenario setup")
flag.Parse()
os.Exit(m.Run())
}

View File

@ -0,0 +1,310 @@
package codehost_testing
import (
"context"
"fmt"
"io"
"net/http"
"testing"
"github.com/google/go-github/v55/github"
"github.com/sourcegraph/sourcegraph/dev/codehost_testing/config"
"github.com/sourcegraph/sourcegraph/lib/errors"
)
// GitHubClient provides methods for creating and retrieving resources with the configured GitHub codehost.
// The configured token is required to have the following scopes:
// - admin:enterprise
// - delete_repo
// - repo
// - site_admin
// - user
// - write:org
type GitHubClient struct {
t *testing.T
cfg *config.GitHub
c *github.Client
}
// GetOrg returns the GitHub organization with the given name.
func (gh *GitHubClient) GetOrg(ctx context.Context, name string) (*github.Organization, error) {
org, resp, err := gh.c.Organizations.Get(ctx, name)
if err != nil {
return nil, err
}
defer resp.Body.Close()
if resp.StatusCode >= 400 {
respErrMsg := formatResponseErrMsg(gh.t, resp)
return nil, errors.Newf("failed to find org: %s - %s", name, respErrMsg)
}
return org, err
}
// CreateOrg creates a new GitHub organization with the given name using the Admin GitHub API.
func (gh *GitHubClient) CreateOrg(ctx context.Context, name string) (*github.Organization, error) {
newOrg := github.Organization{
Login: &name,
}
org, resp, err := gh.c.Admin.CreateOrg(ctx, &newOrg, gh.cfg.AdminUser)
if err != nil {
return nil, err
}
defer resp.Body.Close()
if resp.StatusCode >= 400 {
respErrMsg := formatResponseErrMsg(gh.t, resp)
return nil, errors.Newf("failed to create org %q - %s", name, respErrMsg)
}
return org, err
}
// UpdateOrg updates an existing GitHub organization with the given Org values
func (gh *GitHubClient) UpdateOrg(ctx context.Context, org *github.Organization) (*github.Organization, error) {
_, resp, err := gh.c.Organizations.Edit(ctx, org.GetLogin(), org)
if err != nil {
return nil, err
}
defer resp.Body.Close()
if resp.StatusCode >= 400 {
respErrMsg := formatResponseErrMsg(gh.t, resp)
return nil, errors.Newf("failed to update actions permissions for org %q - %s", org.GetLogin(), respErrMsg)
}
return org, err
}
// CreateUser creates a new GitHub user with the given username using the GitHub Admin API
func (gh *GitHubClient) CreateUser(ctx context.Context, name, email string) (*github.User, error) {
user, resp, err := gh.c.Admin.CreateUser(ctx, name, email)
if err != nil {
return nil, err
}
defer resp.Body.Close()
if resp.StatusCode >= 400 {
respErrMsg := formatResponseErrMsg(gh.t, resp)
return nil, errors.Newf("failed to create user %q - %s", name, respErrMsg)
}
return user, nil
}
// GetUser returns the GitHub user with the given username.
func (gh *GitHubClient) GetUser(ctx context.Context, name string) (*github.User, error) {
user, resp, err := gh.c.Users.Get(ctx, name)
if err != nil {
return nil, err
}
defer resp.Body.Close()
if resp.StatusCode >= 400 {
respErrMsg := formatResponseErrMsg(gh.t, resp)
return nil, errors.Newf("failed to get user %q - %s", name, respErrMsg)
}
return user, nil
}
// DeleteUser deletes a GitHub user with the given username using the GitHub Admin API.
func (gh *GitHubClient) DeleteUser(ctx context.Context, username string) error {
resp, err := gh.c.Admin.DeleteUser(ctx, username)
if err != nil {
return err
}
defer resp.Body.Close()
if resp.StatusCode >= 400 {
respErrMsg := formatResponseErrMsg(gh.t, resp)
return errors.Newf("failed to delete user %q - %s", username, respErrMsg)
}
return nil
}
// GetTeam returns the GitHub team with the given name in the given Organization name.
func (gh *GitHubClient) GetTeam(ctx context.Context, org string, name string) (*github.Team, error) {
team, resp, err := gh.c.Teams.GetTeamBySlug(ctx, org, name)
if err != nil {
return nil, err
}
defer resp.Body.Close()
if resp.StatusCode >= 400 {
respErrMsg := formatResponseErrMsg(gh.t, resp)
return nil, errors.Newf("failed to get team %q - %s", name, respErrMsg)
}
return team, err
}
// CreateTeam creates a new GitHub team with the given name in the given Organization.
func (gh *GitHubClient) CreateTeam(ctx context.Context, org *github.Organization, name string) (*github.Team, error) {
newTeam := github.NewTeam{
Name: name,
Description: github.String("auto created team"),
Privacy: github.String("closed"),
}
team, resp, err := gh.c.Teams.CreateTeam(ctx, org.GetLogin(), newTeam)
if err != nil {
return nil, err
}
defer resp.Body.Close()
if resp.StatusCode >= 400 {
respErrMsg := formatResponseErrMsg(gh.t, resp)
return nil, errors.Newf("failed to create team %q - %s", name, respErrMsg)
}
return team, err
}
// DeleteTeam deletes a GitHub team with the given name in the given Organization.
func (gh *GitHubClient) DeleteTeam(ctx context.Context, org *github.Organization, name string) error {
resp, err := gh.c.Teams.DeleteTeamBySlug(ctx, org.GetLogin(), name)
if resp.StatusCode >= 400 {
respErrMsg := formatResponseErrMsg(gh.t, resp)
return errors.Newf("failed to delete team %q - %s", name, respErrMsg)
}
return err
}
// AssignTeamMembership adds team membership for a user in a team
func (gh *GitHubClient) AssignTeamMembership(ctx context.Context, org *github.Organization, team *github.Team, user *github.User) (*github.Team, error) {
_, resp, err := gh.c.Teams.AddTeamMembershipBySlug(ctx, org.GetLogin(), team.GetSlug(), user.GetLogin(), &github.TeamAddTeamMembershipOptions{
Role: "member",
})
if err != nil {
return nil, err
}
defer resp.Body.Close()
return team, nil
}
// GetRepo returns the GitHub repository with the given name in the given owner which should typically get the Organization name.
func (gh *GitHubClient) GetRepo(ctx context.Context, owner, repoName string) (*github.Repository, error) {
repo, resp, err := gh.c.Repositories.Get(ctx, owner, repoName)
if err != nil {
return nil, err
}
defer resp.Body.Close()
if resp.StatusCode >= 400 {
respErrMsg := formatResponseErrMsg(gh.t, resp)
return nil, errors.Newf("failed to get repo %q - %s", repoName, respErrMsg)
}
return repo, nil
}
// CreateRepo creates a new GitHub repository with the given name under the given org.
func (gh *GitHubClient) CreateRepo(ctx context.Context, org *github.Organization, repoName string, private bool) (*github.Repository, error) {
repo, resp, err := gh.c.Repositories.Create(ctx, org.GetLogin(), &github.Repository{
Name: &repoName,
Private: &private,
})
if err != nil {
return nil, err
}
defer resp.Body.Close()
if resp.StatusCode >= 400 {
respErrMsg := formatResponseErrMsg(gh.t, resp)
return nil, errors.Newf("failed to create repo %q - %s", repoName, respErrMsg)
}
return repo, err
}
// ForkRepo forks a repository into an Organization. The repostiry will have the same name but the owner will be the given
// organization. Note that only the default branch is forked.
func (gh *GitHubClient) ForkRepo(ctx context.Context, org *github.Organization, owner, repoName string) error {
_, resp, err := gh.c.Repositories.CreateFork(ctx, owner, repoName, &github.RepositoryCreateForkOptions{
Organization: org.GetLogin(),
Name: repoName,
DefaultBranchOnly: true,
})
if err != nil {
if resp.StatusCode == 202 {
// expected - forking schedules a job on github side and the repo isn't immediately available
return nil
}
return err
}
defer resp.Body.Close()
if resp.StatusCode >= 400 {
respErrMsg := formatResponseErrMsg(gh.t, resp)
return errors.Newf("failed to fork repo %q - %s", repoName, respErrMsg)
}
return nil
}
// UpdateRepo updates an existing GitHub repository with the given repo values
func (gh *GitHubClient) UpdateRepo(ctx context.Context, org *github.Organization, repo *github.Repository) (*github.Repository, error) {
result, resp, err := gh.c.Repositories.Edit(ctx, org.GetLogin(), repo.GetName(), repo)
if err != nil {
return nil, err
}
defer resp.Body.Close()
if resp.StatusCode >= 400 {
respErrMsg := formatResponseErrMsg(gh.t, resp)
return nil, errors.Newf("failed to edit repository %q - %s", repo.GetName(), respErrMsg)
}
return result, nil
}
// DeleteTeam deletes a GitHub repo with the given name in the given Organization.
func (gh *GitHubClient) DeleteRepo(ctx context.Context, org *github.Organization, repo *github.Repository) error {
resp, err := gh.c.Repositories.Delete(ctx, org.GetLogin(), repo.GetName())
if err != nil {
return err
}
defer resp.Body.Close()
if resp.StatusCode >= 400 {
respErrMsg := formatResponseErrMsg(gh.t, resp)
return errors.Newf("failed to edit repository %q - %s", repo.GetName(), respErrMsg)
}
if err != nil {
return err
}
return nil
}
// UpdateTeamRepoPermissions updates the permissions of the given team for the given repository in the provided Organization.
func (gh *GitHubClient) UpdateTeamRepoPermissions(ctx context.Context, org *github.Organization, team *github.Team, repo *github.Repository) error {
resp, err := gh.c.Teams.AddTeamRepoByID(ctx, org.GetID(), team.GetID(), org.GetLogin(), repo.GetName(), &github.TeamAddTeamRepoOptions{})
if err != nil {
return err
}
defer resp.Body.Close()
if resp.StatusCode != 204 {
respErrMsg := formatResponseErrMsg(gh.t, resp)
return errors.Newf("failed to update repo %q permissions for team %q: %v", repo.GetName(), team.GetSlug(), respErrMsg)
}
return nil
}
func formatResponseErrMsg(t *testing.T, resp *github.Response) string {
code := resp.StatusCode
raw, err := io.ReadAll(resp.Body)
if err != nil {
t.Logf("failed to read response body: %v", err)
return ""
}
return fmt.Sprintf("Status Code: %d\nBody: %s\n", code, string(raw))
}
// NewGitHubClient returns a new GitHub client from the given config. Note that the client sets InsecureSkipVerify to true
func NewGitHubClient(t *testing.T, cfg config.GitHub) (*GitHubClient, error) {
t.Helper()
httpClient := &http.Client{}
gh, err := github.NewClient(httpClient).WithAuthToken(cfg.Token).WithEnterpriseURLs(cfg.URL, cfg.URL)
if err != nil {
return nil, err
}
c := GitHubClient{
cfg: &cfg,
c: gh,
}
return &c, nil
}

205
dev/codehost_testing/org.go Normal file
View File

@ -0,0 +1,205 @@
package codehost_testing
import (
"context"
"fmt"
"strings"
"github.com/google/go-github/v55/github"
"github.com/sourcegraph/sourcegraph/lib/errors"
)
// Org represents a GitHub organization and provides actions that operate on the org.
//
// All methods except Get, create actions which are added to the GitHubScenario this
// org was was created from.
type Org struct {
// s is the GithubScenario instance this org was created from
s *GitHubScenario
// name is the name of the GitHub organization
name string
}
// Get returns the corresponding GitHub Organization object that was created by the `CreateOrg`
//
// This method will only return a Org if the Scenario that created it has been applied otherwise
// it will panic.
func (o *Org) Get(ctx context.Context) (*github.Organization, error) {
if o.s.IsApplied() {
return o.s.client.GetOrg(ctx, o.name)
}
return nil, errors.New("cannot retrieve org before scenario is applied")
}
// get retrieves the GitHub organization without panicking if not applied. It is meant as an
// internal helper method while actions are getting applied.
func (o *Org) get(ctx context.Context) (*github.Organization, error) {
return o.s.client.GetOrg(ctx, o.name)
}
// AllowPrivateForks adds an action to the scenario to enable private forks and repos for the org
func (o *Org) AllowPrivateForks() {
updateOrgPermissions := &Action{
Name: "org:permissions:update:" + o.name,
Apply: func(ctx context.Context) error {
org, err := o.get(ctx)
if err != nil {
return err
}
org.MembersCanCreatePrivateRepos = github.Bool(true)
org.MembersCanForkPrivateRepos = github.Bool(true)
_, err = o.s.client.UpdateOrg(ctx, org)
if err != nil {
return err
}
return nil
},
Teardown: nil,
}
o.s.Append(updateOrgPermissions)
}
// CreateTeam adds an action to the scenario to create a team with the given name for the org.
// The Scenario ID will be added as a suffix to the given name.
func (o *Org) CreateTeam(name string) *Team {
baseTeam := &Team{
s: o.s,
org: o,
name: name,
}
action := &Action{
Name: "org:team:create:" + name,
Apply: func(ctx context.Context) error {
name := fmt.Sprintf("team-%s-%s", name, o.s.id)
org, err := o.get(ctx)
if err != nil {
return err
}
team, err := o.s.client.CreateTeam(ctx, org, name)
if err != nil {
return err
}
baseTeam.name = team.GetName()
return nil
},
Teardown: func(ctx context.Context) error {
org, err := o.get(ctx)
if err != nil {
return err
}
return o.s.client.DeleteTeam(ctx, org, baseTeam.name)
},
}
o.s.Append(action)
return baseTeam
}
// CreateRepo adds an action to the scenario to create a repo with the given name and visibility for the org.
func (o *Org) CreateRepo(name string, public bool) *Repo {
baseRepo := &Repo{
s: o.s,
org: o,
name: name,
}
action := &Action{
Name: fmt.Sprintf("repo:create:%s", name),
Apply: func(ctx context.Context) error {
org, err := o.get(ctx)
if err != nil {
return err
}
var repoName string
parts := strings.Split(name, "/")
if len(parts) >= 2 {
repoName = parts[1]
} else {
return errors.Newf("incorrect repo format for %q - expecting {owner}/{name}")
}
repo, err := o.s.client.CreateRepo(ctx, org, repoName, public)
if err != nil {
return err
}
baseRepo.name = repo.GetFullName()
return nil
},
Teardown: func(ctx context.Context) error {
org, err := o.get(ctx)
if err != nil {
return err
}
repo, err := baseRepo.get(ctx)
if err != nil {
return err
}
return o.s.client.DeleteRepo(ctx, org, repo)
},
}
o.s.Append(action)
return baseRepo
}
// CreateRepoFork adds an action to the scenario to fork a target repo into the org.
//
// NOTE: This method actually adds two actions to the scenario. One which performs the Fork and a subsequent
// action which waits till the forked repo exists on GitHub.
func (o *Org) CreateRepoFork(target string) *Repo {
baseRepo := &Repo{
s: o.s,
org: o,
name: target,
}
action := &Action{
Name: fmt.Sprintf("repo:fork:%s", target),
Apply: func(ctx context.Context) error {
org, err := o.get(ctx)
if err != nil {
return err
}
var owner, repoName string
parts := strings.Split(target, "/")
if len(parts) >= 2 {
owner = parts[0]
repoName = parts[1]
} else {
return errors.Newf("incorrect repo format for %q - expecting {owner}/{name}")
}
err = o.s.client.ForkRepo(ctx, org, owner, repoName)
if err != nil {
return err
}
// Wait till fork has synced
baseRepo.name = repoName
return nil
},
Teardown: func(ctx context.Context) error {
org, err := o.get(ctx)
if err != nil {
return err
}
repo, err := baseRepo.get(ctx)
if err != nil {
return err
}
return o.s.client.DeleteRepo(ctx, org, repo)
},
}
o.s.Append(action)
baseRepo.WaitTillExists()
return baseRepo
}

View File

@ -0,0 +1,127 @@
package codehost_testing
import (
"context"
"fmt"
"time"
"github.com/google/go-github/v55/github"
"github.com/sourcegraph/sourcegraph/lib/errors"
)
// Repo represents a GitHub repository in the scenario
type Repo struct {
// s is the GithubScenario instance this repo is part of. Actions it creates will be added to this scenario
s *GitHubScenario
// team is the team this repo belongs to
team *Team
// org is the Org that owns this repo
org *Org
// name is the name of the repo
name string
}
// Get returns the corresponding GitHub Repository object that was created by the `CreateOrg`
//
// This method will only return a Repository if the Scenario that created it has been applied otherwise
// it will panic.
func (r *Repo) Get(ctx context.Context) (*github.Repository, error) {
if r.s.IsApplied() {
return r.get(ctx)
}
r.s.t.Fatal("cannot retrieve repo before scenario is applied")
return nil, nil
}
// get retrieves the GitHub repository without panicking if not applied. It is meant as an
// internal helper method while actions are getting applied.
func (r *Repo) get(ctx context.Context) (*github.Repository, error) {
return r.s.client.GetRepo(ctx, r.org.name, r.name)
}
// AddTeam creats an action that will update the repo permissions so that the given team
// has access to this repo.
func (r *Repo) AddTeam(team *Team) {
r.team = team
action := &Action{
Name: fmt.Sprintf("repo:team:%s:membership:%s", team.name, r.name),
Apply: func(ctx context.Context) error {
org, err := r.org.get(ctx)
if err != nil {
return err
}
repo, err := r.get(ctx)
if err != nil {
return err
}
team, err := r.team.get(ctx)
if err != nil {
return err
}
err = r.s.client.UpdateTeamRepoPermissions(ctx, org, team, repo)
if err != nil {
return err
}
return nil
},
Teardown: nil,
}
r.s.Append(action)
}
// SetPermissions adds an action that will set the permissions (public or private) for the repository
func (r *Repo) SetPermissions(private bool) {
permissionKey := "private"
if !private {
permissionKey = "public"
}
action := &Action{
Name: fmt.Sprintf("repo:permissions:%s:%s", r.name, permissionKey),
Apply: func(ctx context.Context) error {
repo, err := r.get(ctx)
if err != nil {
return err
}
repo.Private = &private
org, err := r.org.get(ctx)
if err != nil {
return err
}
_, err = r.s.client.UpdateRepo(ctx, org, repo)
if err != nil {
return err
}
return err
},
}
r.s.Append(action)
}
// WaitTillExists creates an action that waits for the repository to exist on GitHub. This action is especially
// useful for when a repo is forked since a forked repo doesn't immediately exist when requested on GitHub.
func (r *Repo) WaitTillExists() {
action := &Action{
Name: fmt.Sprintf("repo:exists:%s", r.name),
Apply: func(ctx context.Context) error {
var err error
for i := 0; i < 5; i++ {
time.Sleep(1 * time.Second)
_, err = r.get(ctx)
if err == nil {
return nil
}
}
return errors.Newf("repo %q did not exist after waiting: %v", r.name, err)
},
}
r.s.Append(action)
}

View File

@ -0,0 +1,35 @@
package codehost_testing
import "fmt"
// Reporter defines an interface for writing formatted output.
type Reporter interface {
Writef(format string, args ...any) (int, error)
Writeln(v string) (int, error)
}
// ConsoleReporter implements the Reporter interface for writing to stdout
type ConsoleReporter struct{}
// NoopReporter implements the Reporter interface by providing no-op operations
type NoopReporter struct{}
// Writef writes the args to the console according the specified format
func (r ConsoleReporter) Writef(format string, args ...any) (int, error) {
return fmt.Printf(format, args...)
}
// Writeln writes the args to the console according to the specified format with a newline
func (r ConsoleReporter) Writeln(v string) (int, error) {
return fmt.Println(v)
}
// Writef is a no-op for NoopReporter
func (r NoopReporter) Writef(format string, args ...any) (int, error) {
return 0, nil
}
// Writeln is a no-op for NoopReporter
func (r NoopReporter) Writeln(v string) (int, error) {
return 0, nil
}

View File

@ -0,0 +1,303 @@
package codehost_testing
import (
"context"
"crypto/md5"
"encoding/base64"
"encoding/hex"
"fmt"
"strings"
"testing"
"time"
"github.com/google/uuid"
"github.com/sourcegraph/sourcegraph/dev/codehost_testing/config"
"github.com/sourcegraph/sourcegraph/lib/errors"
)
type ActionFn func(context.Context) error
// Action represents a task that is performed that cosists of task which creates or alters some resource with Apply
// and a corresponding Teardown task that destroys the resource created with Apply or undos any alterations. Effectively,
// Teardown should do the inverse of Apply.
//
// A Nil Teardown means there is no teardown to be performed, whereas Apply should never be nil
type Action struct {
Name string
Apply ActionFn
Teardown ActionFn
}
// Scenario is an interface for executing a sequence of actions in a test scenario. Actions can be added
// by the relevant struct implementing the interface.
//
// The methods of the interface have the following intentions:
// Apply: should apply the actions that are part of the interface
// Teardown: should remove or undo the actions applied by Apply
// Plan: should return a human-readable string describing the actions that will be performed
type Scenario interface {
Append(a ...*Action)
Plan() string
Apply(ctx context.Context) error
Teardown(ctx context.Context) error
}
// GitHubScenario implements the Scenario interface for testing GitHub functionality. At its base GitHubScenario
// provides two top level methods to create GitHub resources namely:
// * create GitHub Organization, which returns a codehost_testing Org
// * create a GitHub User, which returns a codehost_testing User
//
// Further resources can be created by calling methods on the returned Org or User. For instance, since a repository
// is tied to an organization, one can call org.CreateRepo, which will add an action for a repo to be created in the
// organization.
//
// Calling any action creating method does not immediately create the resource in GitHub. Instead a action is added
// the list of actions contained in this scenario. Only once Apply() has been called on the Scenario itself will
// the resources be created on GitHub.
//
// Once Apply() is called, all the corresponding resources should be realized on GitHub. To fetch the corresponding
// GitHub resources once can call Get() on the resources.
type GitHubScenario struct {
id string
t *testing.T
client *GitHubClient
actions []*Action
reporter Reporter
nextActionIdx int
adminUser *User
}
var _ Scenario = (*GitHubScenario)(nil)
// NewGitHubScenario creates a new GitHubScenario instance. A base64 ID will be generated to identify this scenario.
// This ID will also be used to uniquely identify any resources created as part of this scenario.
//
// By default a GitHubScenario is created with a NoopReporter. To have more verbose output, call SetVerbose() on the scenario,
// and to reduce the output, call SetQuiet().
func NewGitHubScenario(t *testing.T, cfg config.Config) (*GitHubScenario, error) {
client, err := NewGitHubClient(t, cfg.GitHub)
if err != nil {
return nil, err
}
uid := []byte(uuid.NewString())
id := base64.RawStdEncoding.EncodeToString(uid[:])[:10]
scenario := &GitHubScenario{
id: id,
t: t,
client: client,
actions: make([]*Action, 0),
reporter: &NoopReporter{},
}
scenario.adminUser = &User{
s: scenario,
name: cfg.GitHub.AdminUser,
}
return scenario, err
}
// Verbose sets the reporter to ConsoleReporter to enable verbose output
func (s *GitHubScenario) SetVerbose() {
s.reporter = &ConsoleReporter{}
}
// Quiet sets the reporter to a no-op reporter to reduce output
func (s *GitHubScenario) SetQuiet() {
s.reporter = &NoopReporter{}
}
func (s *GitHubScenario) Append(actions ...*Action) {
s.actions = append(s.actions, actions...)
}
// Plan returns a string describing the actions that will be performed
func (s *GitHubScenario) Plan() string {
sb := &strings.Builder{}
fmt.Fprintf(sb, "Scenario %q\n", s.id)
sb.WriteString("== Setup ==\n")
for _, action := range s.actions {
fmt.Fprintf(sb, "- %s\n", action.Name)
}
sb.WriteString("== Teardown ==\n")
for _, action := range reverse(s.actions) {
if action.Teardown == nil {
continue
}
fmt.Fprintf(sb, "- %s\n", action.Name)
}
return sb.String()
}
// IsApplied returns whether Apply has already been called on this scenario. If more actions
// have been added since the last Apply(), it will return false.
func (s *GitHubScenario) IsApplied() bool {
return s.nextActionIdx >= len(s.actions)
}
// Apply performs all the actions that have been added to this scenario sequentially in the order they were added.
// Furthemore cleanup function is registered so Teardown is called even if Apply fails to make sure we cleanup any
// left over resources due to a half applied scenario.
//
// Note that calling Apply more than once and with no new actions added, will result in an error be returned. This
// is done since duplicate resources cannot be created.
//
// If a scenario is applied and fails midway and Apply is called again, it will continue where it left off. The only
// way to reset this behaviour is to call teardown.
//
// Finally, if any action fails, no further actions will be executed and this method will return with the error
func (s *GitHubScenario) Apply(ctx context.Context) error {
s.t.Helper()
s.t.Cleanup(func() { s.Teardown(ctx) })
var errs errors.MultiError
setup := s.actions
failFast := true
if s.nextActionIdx >= len(s.actions) {
return errors.New("all actions already applied")
}
start := time.Now()
for currActionIdx, action := range setup {
now := time.Now().UTC()
if s.nextActionIdx > currActionIdx {
s.reporter.Writef("(Setup) Skipping [%-50s]\n", action.Name)
continue
}
if action.Apply == nil {
return errors.Newf("action %q has nil Apply", action.Name)
}
s.reporter.Writef("(Setup) Applying [%-50s] ", action.Name)
err := action.Apply(ctx)
duration := time.Now().UTC().Sub(now)
if err != nil {
errs = errors.Append(errs, err)
s.reporter.Writef("FAILED (%s)\n", duration.String())
if failFast {
break
}
} else {
s.nextActionIdx++
s.reporter.Writef("SUCCESS (%s)\n", duration.String())
}
}
s.reporter.Writef("Setup complete in %s\n\n", time.Now().UTC().Sub(start))
return errs
}
// Teardown cleans up any resources created by Apply. This method is automatically registered with *testing.Cleanup to
// cleanup resources, so generally it would not have to be called explicitly.
//
// Teardown iterates through the scenario actions in reverse order, calling teardown on each action. If a action
// has a nil teardown function it will be skipped. Teardown does not stop iterating when an action returns with an error,
// instead the error is accumulated and the next teardown action is executed.
//
// Note that Teardown is not idempotent. Multiple calls will result in failures.
func (s *GitHubScenario) Teardown(ctx context.Context) error {
s.t.Helper()
var errs errors.MultiError
teardown := reverse(s.actions)
failFast := false
start := time.Now()
for _, action := range teardown {
// Nil means this action has no means of being torn down
if action.Teardown == nil {
continue
}
now := time.Now().UTC()
s.reporter.Writef("(Teardown) Applying [%-50s] ", action.Name)
err := action.Teardown(ctx)
duration := time.Now().UTC().Sub(now)
if err != nil {
s.reporter.Writef("FAILED (%s)\n", duration.String())
if failFast {
break
}
errs = errors.Append(errs, err)
} else {
s.reporter.Writef("SUCCESS (%s)\n", duration.String())
}
}
// Actions create new resources, therefore we can safely set the nextActionIdx to 0 here
// so that new resources be created
s.nextActionIdx = 0
s.reporter.Writef("Teardown complete in %s\n", time.Now().UTC().Sub(start))
return errs
}
func (s *GitHubScenario) CreateOrg(name string) *Org {
baseOrg := &Org{
s: s,
name: name,
}
createOrg := &Action{
Name: "org:create:" + name,
Apply: func(ctx context.Context) error {
orgName := fmt.Sprintf("org-%s-%s", name, s.id)
org, err := s.client.CreateOrg(ctx, orgName)
if err != nil {
return err
}
baseOrg.name = org.GetLogin()
return nil
},
Teardown: func(context.Context) error {
host := baseOrg.s.client.cfg.URL
deleteURL := fmt.Sprintf("%s/organizations/%s/settings/profile", host, baseOrg.name)
fmt.Printf("Visit %q to delete the org\n", deleteURL)
return nil
},
}
s.Append(createOrg)
return baseOrg
}
// CreateUser adds an action to the scenario that will create a GitHub user with the given name. The username of the
// user will have the following format `user-{name}-{scenario id}` and email `test-user-e2e@sourcegraph.com`.
func (s *GitHubScenario) CreateUser(name string) *User {
baseUser := &User{
s: s,
name: name,
}
createUser := &Action{
Name: "user:create:" + name,
Apply: func(ctx context.Context) error {
name := fmt.Sprintf("user-%s-%s", name, s.id)
emailID := md5.Sum([]byte(s.id + time.Now().String()))
email := fmt.Sprintf("test-user-e2e-%s@sourcegraph.com", hex.EncodeToString(emailID[:]))
user, err := s.client.CreateUser(ctx, name, email)
if err != nil {
return err
}
baseUser.name = user.GetLogin()
return nil
},
Teardown: func(ctx context.Context) error {
return s.client.DeleteUser(ctx, baseUser.name)
},
}
s.Append(createUser)
return baseUser
}
// GetAdmin returns a User representing the GitHub admin user configured in the client.
//
// NOTE: this method does not actually add an explicit action to the scenario, but will still
// require that the scenario has been applied before the admin user can be retrieved - even though
// it is not strictly required as the Admin already exists.
func (s *GitHubScenario) GetAdmin() *User {
return s.adminUser
}

View File

@ -0,0 +1,177 @@
package codehost_testing
import (
"context"
"errors"
"testing"
)
func testAction(name string) *Action {
return &Action{
Name: name,
Apply: func(ctx context.Context) error {
return nil
},
Teardown: func(ctx context.Context) error {
return nil
},
}
}
func TestScenarioApplyAndTeardown(t *testing.T) {
t.Run("Applied actions updates nextActionIdx", func(t *testing.T) {
scenario := &GitHubScenario{
id: "testing-id",
t: t,
client: nil,
actions: []*Action{},
reporter: NoopReporter{},
nextActionIdx: 0,
}
scenario.Append(testAction("t1"), testAction("t2"))
if len(scenario.actions) != 2 {
t.Errorf("actions not appended - got %d wanted %d", len(scenario.actions), 2)
}
err := scenario.Apply(context.TODO())
if err != nil {
t.Fatalf("failed to apply test scenario with mock actions: %v", err)
}
if scenario.nextActionIdx != 2 {
t.Errorf("actions applied count mismatch - got %d wanted %d", scenario.nextActionIdx, 2)
}
if !scenario.IsApplied() {
t.Error("all actions have been applied thus IsApplied should be true")
}
})
t.Run("Next Action Idx at 1 if remaining action errors", func(t *testing.T) {
scenario := &GitHubScenario{
id: "testing-id",
t: t,
client: nil,
actions: []*Action{},
reporter: NoopReporter{},
nextActionIdx: 0,
}
errAction := testAction("err1")
fakeErr := errors.New("fake error")
errAction.Apply = func(ctx context.Context) error {
return fakeErr
}
scenario.Append(testAction("t1"), errAction)
if len(scenario.actions) != 2 {
t.Errorf("actions not appended - got %d wanted %d", len(scenario.actions), 2)
}
err := scenario.Apply(context.TODO())
if err != nil && !errors.Is(err, fakeErr) {
t.Fatalf("failed to apply test scenario with mock actions: %v", err)
}
if scenario.nextActionIdx != 1 {
t.Errorf("actions applied count mismatch - got %d wanted %d", scenario.nextActionIdx, 1)
}
if scenario.IsApplied() {
t.Error("not all actions have been applied thus IsApplied should be false")
}
err = scenario.Teardown(context.TODO())
if err != nil {
t.Fatalf("teardown not expected to fail here: %v", err)
}
if scenario.IsApplied() {
t.Error("after teardown, IsApplied should be false")
}
if scenario.nextActionIdx != 0 {
t.Errorf("after teardown nextActionIdx should be 0 - got %d", scenario.nextActionIdx)
}
})
t.Run("3 Actions with 1 skipped teardown", func(t *testing.T) {
scenario := &GitHubScenario{
id: "testing-id",
t: t,
client: nil,
actions: []*Action{},
reporter: NoopReporter{},
nextActionIdx: 0,
}
skipAction := testAction("s2")
skipAction.Teardown = nil
scenario.Append(testAction("t1"), skipAction, testAction("t3"))
if len(scenario.actions) != 3 {
t.Errorf("actions not appended - got %d wanted %d", len(scenario.actions), 2)
}
scenario.Apply(context.TODO())
if scenario.nextActionIdx != 3 {
t.Errorf("actions applied count mismatch - got %d wanted %d", scenario.nextActionIdx, 1)
}
if !scenario.IsApplied() {
t.Error("all actions should be applied")
}
err := scenario.Teardown(context.TODO())
if err != nil {
t.Fatalf("teardown not expected to fail here: %v", err)
}
if scenario.IsApplied() {
t.Error("after teardown, scenario should not be Applied")
}
if scenario.nextActionIdx != 0 {
t.Errorf("after teardown nextActionIdx should be 0 - got %d", scenario.nextActionIdx)
}
})
t.Run("4 Actions with 2 failed teardown", func(t *testing.T) {
scenario := &GitHubScenario{
id: "testing-id",
t: t,
client: nil,
actions: []*Action{},
reporter: NoopReporter{},
nextActionIdx: 0,
}
errTeardown := func(_ context.Context) error { return errors.New("fake") }
errAction := testAction("e2")
errAction.Teardown = errTeardown
scenario.Append(testAction("t1"), errAction, testAction("t3"))
errTeardown = func(_ context.Context) error { return errors.New("fake") }
errAction = testAction("e4")
errAction.Teardown = errTeardown
scenario.Append(errAction)
if len(scenario.actions) != 4 {
t.Errorf("actions not appended - got %d wanted %d", len(scenario.actions), 2)
}
scenario.Apply(context.TODO())
if scenario.nextActionIdx != 4 {
t.Errorf("actions applied count mismatch - got %d wanted %d", scenario.nextActionIdx, 4)
}
if !scenario.IsApplied() {
t.Error("all actions should be applied")
}
scenario.Teardown(context.TODO())
if scenario.IsApplied() {
t.Error("after teardown, scenario should not be Applied")
}
if scenario.nextActionIdx != 0 {
t.Errorf("after teardown nextActionIdx should be 0 - got %d", scenario.nextActionIdx)
}
})
}

View File

@ -0,0 +1,66 @@
package codehost_testing
import (
"context"
"fmt"
"github.com/google/go-github/v55/github"
"github.com/sourcegraph/sourcegraph/lib/errors"
)
// Team represents a GitHub team and provdes actions that operate on a GitHub team.
//
// All methods except Get, create actions which are added to the parent GitHubScenario this
// team belongs to.
type Team struct {
// s is the GithubScenario instance this team was created from
s *GitHubScenario
// org is the Org this team belongs to, and is ultimately the one who created this team
org *Org
// name is the name of the team
name string
}
// Get returns the corresponding GitHub Team object that was created by the `CreateTeam`
//
// This method will only return a Team if the Scenario that created it has been applied otherwise
// it will panic.
func (team *Team) Get(ctx context.Context) (*github.Team, error) {
if team.s.IsApplied() {
return team.get(ctx)
}
return nil, errors.New("cannot retrieve org before scenario is applied")
}
// get retrieves the GitHub team without panicking if not applied. It is meant as an
// internal helper method while actions are getting applied.
func (team *Team) get(ctx context.Context) (*github.Team, error) {
return team.s.client.GetTeam(ctx, team.org.name, team.name)
}
// AddUser adds an action that will add the given user to this team
func (tm *Team) AddUser(u *User) {
assignTeamMembership := &Action{
Name: fmt.Sprintf("team:membership:%s:%s", tm.name, u.name),
Apply: func(ctx context.Context) error {
org, err := tm.org.get(ctx)
if err != nil {
return err
}
team, err := tm.get(ctx)
if err != nil {
return err
}
user, err := u.get(ctx)
if err != nil {
return err
}
_, err = tm.s.client.AssignTeamMembership(ctx, org, team, user)
return err
},
Teardown: nil,
}
tm.s.Append(assignTeamMembership)
}

View File

@ -0,0 +1,32 @@
package codehost_testing
import (
"context"
"github.com/google/go-github/v55/github"
"github.com/sourcegraph/sourcegraph/lib/errors"
)
// User represents a GitHub user in the scenario.
type User struct {
s *GitHubScenario
name string
}
// Get returns the corresponding GitHub user object that was created by the `CreateUser`
//
// This method will only return a User if the Scenario that created it has been applied otherwise
// it will panic.
func (u *User) Get(ctx context.Context) (*github.User, error) {
if u.s.IsApplied() {
return u.get(ctx)
}
return nil, errors.New("cannot retrieve user before scenario is applied")
}
// get retrieves the GitHub user without panicking if not applied. It is meant as an
// internal helper method while actions are getting applied.
func (u *User) get(ctx context.Context) (*github.User, error) {
return u.s.client.GetUser(ctx, u.name)
}

View File

@ -0,0 +1,9 @@
package codehost_testing
func reverse[T any](src []T) []T {
reversed := make([]T, 0, len(src))
for i := len(src) - 1; i >= 0; i-- {
reversed = append(reversed, src[i])
}
return reversed
}