flip/fsc-utils
1
0
Fork 0
mirror of https://github.com/FlipsideCrypto/fsc-utils.git synced 2026-02-06 10:56:49 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity

Merge pull request #21 from FlipsideCrypto/GHA-readme-update

Browse Source 
read me updates
This commit is contained in:
Austin 2023-11-06 14:26:42 -05:00 committed by GitHub
commit 11cb982fe9
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 55 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

55
README.md
View File

@ -142,6 +142,61 @@ config:
- INTERNAL_DEV
```
## Snowflake Tasks for GitHub Actions
A set of macros and UDFs have been created to help with the creation of Snowflake tasks to manage runs in GitHub Actions.
### Basic Setup ###
1. Make sure `fsc-utils` package referenced in the project is version `v1.11.0` or greater. Re-run `dbt deps` if revision was changed.
2. Make sure LiveQuery has been deployed to the project. See [LiveQuery Functions](#livequery-functions) for more information.
> Note - If you are using tags to run your workflows, it is highly recommend to add the project name to the tag. For example, `"ethereum_models,tag:core"` instead of `tag:core`. This will ensure that the correct workflows are being ran within your project.
3. Install the GitHub LiveQuery Functions
```
dbt run -s livequery_models.deploy.marketplace.github_actions --vars '{UPDATE_UDFS_AND_SPS: true}'
```
Use `-t prod` when running in production
4. Add `github_actions__workflows.csv` to the data folder in your project. This file will contain the list of workflows to be created. See [Polygon](https://github.com/FlipsideCrypto/polygon-models/blob/main/data/github_actions__workflows.csv) for sample format.
Seed the file into dbt
```
dbt seed -s github_actions__workflows
```
Add file to `sources.yml`
```
- name: github_actions
database: {{prod_db}}
schema: github_actions
tables:
- name: workflows
```
If you would like to test in dev, you will need to seed your file to prod with a separate PR.
5. Add the `github_actions` folder to your project's `models` folder. This folder contains the models that will be used to create and monitor the workflows. See [Polygon](https://github.com/FlipsideCrypto/polygon-models/tree/main/models/github_actions)
Build the GitHub Actions View
```
dbt run -m models/github_actions --full-refresh
```
Add `--vars '{UPDATE_UDFS_AND_SPS: true}'` if you have not already created UDFs on version `v1.11.0` or greater.
6. Add the template workflows `dbt_alter_gha_tasks.yml` and `dbt_test_tasks.yml`
> The [alter workflow](https://github.com/FlipsideCrypto/arbitrum-models/blob/main/.github/workflows/dbt_alter_gha_task.yml) is used to `SUSPEND` or `RESUME` tasks, which you will need to do if you want to pause a workflow while merging a big PR, for example. This is intended to be ran on an ad-hoc basis.
> The [test workflow](https://github.com/FlipsideCrypto/arbitrum-models/blob/main/.github/workflows/dbt_test_tasks.yml) is used to test the workflows. It ensures that workflows are running according to the schedule and that the tasks are completing successfully. You will want to include this workflow within `github_actions__workflows.csv`. You can change the `.yml` included in the `models/github_actions` folder to better suite your testing needs, if necessary.
7. Add the `START_GHA_TASKS` variable to `dbt_project.yml`
```
START_GHA_TASKS: False
```
8. Create the Tasks
```
dbt run-operation fsc_utils.create_gha_tasks --var '{"START_GHA_TASKS":True}'
```
> This will create the tasks in Snowflake and the workflows in GitHub Actions. The tasks will only be started if `START_GHA_TASKS` is set to `True` and the target is the production database for your project.
9. Add a Data Dog CI Pipeline Alert on the logs of `dbt_test_tasks` to ensure that the test is checking the workflows successfully. See `Polygon Task Alert` in Data Dog for sample alert.
## Resources
* Learn more about dbt [in the docs](https://docs.getdbt.com/docs/introduction)