fsc-utils/README.md
2025-12-10 13:34:41 -05:00

587 lines
24 KiB
Markdown

# Flipside Utility Functions
Dbt repo for managing the Flipside Utility Functions (FSC_UTILS) dbt package.
## Variables
Control the creation of `UDF` or `SP` macros with dbt run:
- `UPDATE_UDFS_AND_SPS` -
When `True`, executes all macros included in the on-run-start hooks within dbt_project.yml on model run as normal
When False, none of the on-run-start macros are executed on model run
Default values is `False`
Usage:
```sh
dbt run --var 'UPDATE_UDFS_AND_SPS": True' -m ...
```
Dropping and creating udfs can also be done without running a model:
```sh
dbt run-operation create_udfs --var 'UPDATE_UDFS_AND_SPS": True' --args 'drop_:false'
dbt run-operation create_udfs --var 'UPDATE_UDFS_AND_SPS": True' --args 'drop_:true'
```
## Adding Release Versions
1. Make the necessary changes to your code in your dbt package repository (e.g., fsc-utils).
2. Commit your changes with `git add .` and `git commit -m "Your commit message"`.
3. Tag your commit with a version number using `git tag -a v1.1.0 -m "version 1.1.0"`.
4. Push your commits to the remote repository with `git push origin ...`.
5. Push your tags to the remote repository with `git push origin --tags`.
6. In the `packages.yml` file of your other dbt project, specify the new version of the package with:
```
packages:
- git: "https://github.com/FlipsideCrypto/fsc-utils.git"
revision: "v1.1.0"
```
---
**NOTE** Steps `2-5` above can also be automated using `make tag` directive:
### Tag Makefile Directives
#### `tag`
The `tag` directive is used to tag the current commit with a version number.
**Usage**:
```sh
make tag version=<version_number>
```
Replace <version_number> with the version number you want to use.
What it does:
Adds all changes to the staging area with `git add .`
Commits the changes with a commit message of `Bump version to <version_number>`.
Creates a new `git tag` with the name `v<version_number>` and a message of `version <version_number>`.
Pushes the new tag to the origin remote.
#### get_latest_tags
The get_latest_tags directive is used to display the latest git tags. By default, it displays the latest tag. You can change the number of tags displayed by setting the MAX_COUNT variable.
Usage:
```sh
make get_latest_tags MAX_COUNT=<count>
```
Replace <count> with the number of latest tags you want to display. If you don't specify a count, it defaults to `1`.
What it does:
Displays the latest `<count> git tags` in green text.
---
7. Run dbt deps in the other dbt project to pull the specific version of the package or follow the steps on `adding the dbt package` below.
Regarding Semantic Versioning;
1. Semantic versioning is a versioning scheme for software that aims to convey meaning about the underlying changes with each new release.
2. It's typically formatted as MAJOR.MINOR.PATCH (e.g. v1.2.3), where:
- MAJOR version (first number) should increment when there are potential breaking or incompatible changes.
- MINOR version (second number) should increment when functionality or features are added in a backwards-compatible manner.
- PATCH version (third number) should increment when bug fixes are made without adding new features.
3. Semantic versioning helps package users understand the degree of changes in a new release, and decide when to adopt new versions. With dbt packages, when you tag a release with a semantic version, users can specify the exact version they want to use in their projects.
## Adding the `fsc_utils` dbt package
The `fsc_utils` dbt package is a centralized repository consisting of various dbt macros and snowflake functions that can be utilized across other repos.
1. Navigate to the `create_udfs.sql` macro in your respective repo where you want to install the package.
2. Add the following:
```
{% set name %}
{{- fsc_utils.create_udfs() -}}
{% endset %}
{% do run_query(sql) %}
```
3. Note: fsc*utils.create_udfs() takes two parameters (drop*=False, schema="utils"). Set `drop_` to `True` to drop existing functions or define `schema` for the functions (default set to `utils`). Params not required.
4. Navigate to `packages.yml` in your respective repo.
5. Add the following:
```
- git: https://github.com/FlipsideCrypto/fsc-utils.git
```
6. Run `dbt deps` to install the package
7. Run the macro `dbt run-operation create_udfs --var '{"UPDATE_UDFS_AND_SPS":True}'`
### **Overview of Available Functions**
#### **UTILS Functions**
- `utils.udf_hex_to_int`: Use this UDF to transform any hex string to integer
```
ex: Curve Swaps
SELECT
regexp_substr_all(SUBSTR(DATA, 3, len(DATA)), '.{64}') AS segmented_data,
utils.hex_to_int(segmented_data [1] :: STRING) :: INTEGER AS tokens_sold
FROM
optimism.core.fact_event_logs
WHERE
topics [0] :: STRING IN (
'0x8b3e96f2b889fa771c53c981b40daf005f63f637f1869f707052d15a3dd97140',
'0xd013ca23e77a65003c2c659c5442c00c805371b7fc1ebd4c206c41d1536bd90b'
)
```
- `utils.udf_hex_to_string`: Use this UDF to transform any hexadecimal string to a regular string, removing any non-printable or control characters from the resulting string.
```
ex: Token Names
WITH base AS (
SELECT
'0x0000000000000000000000000000000000000000000000000000000000000020000000000000000000000000000000000000000000000000000000000000005452617265202d204368616e74616c20486167656c202d20576f6d656e2773204575726f2032303232202d2032303232205371756164202d20576f6d656e2773204e6174696f6e616c205465616d202d2032303232000000000000000000000000' AS input_token_name
)
SELECT
utils.udf_hex_to_string(SUBSTR(input_token_name,(64*2+3),LEN(input_token_name))) AS output_token_name
FROM base;
NOTE: The expression 64 * 2 + 3 in the query navigates to the 131st character of the hexadecimal string returned by an EVM blockchain contract's function, skipping metadata and adjusting for Snowflake's 1-based indexing. Keep in mind that the exact start of relevant data may vary between different contracts and functions.
```
- `utils.udf_encode_contract_call`: Encodes EVM contract function calls into ABI-encoded calldata format for eth_call RPC requests. Handles all Solidity types including tuples and arrays.
```
-- Simple function with no inputs
SELECT utils.udf_encode_contract_call(
PARSE_JSON('{"name": "totalSupply", "inputs": []}'),
ARRAY_CONSTRUCT()
);
-- Returns: 0x18160ddd
-- Function with single address parameter
SELECT utils.udf_encode_contract_call(
PARSE_JSON('{
"name": "balanceOf",
"inputs": [{"name": "account", "type": "address"}]
}'),
ARRAY_CONSTRUCT('0xa0b86991c6218b36c1d19d4a2e9eb0ce3606eb48')
);
-- Returns: 0x70a08231000000000000000000000000a0b86991c6218b36c1d19d4a2e9eb0ce3606eb48
-- Function with multiple parameters
SELECT utils.udf_encode_contract_call(
PARSE_JSON('{
"name": "transfer",
"inputs": [
{"name": "to", "type": "address"},
{"name": "amount", "type": "uint256"}
]
}'),
ARRAY_CONSTRUCT('0x1234567890123456789012345678901234567890', 1000000)
);
-- Complex function with nested tuples
SELECT utils.udf_encode_contract_call(
PARSE_JSON('{
"name": "swap",
"inputs": [{
"name": "params",
"type": "tuple",
"components": [
{"name": "tokenIn", "type": "address"},
{"name": "tokenOut", "type": "address"},
{"name": "amountIn", "type": "uint256"}
]
}]
}'),
ARRAY_CONSTRUCT(
ARRAY_CONSTRUCT(
'0xa0b86991c6218b36c1d19d4a2e9eb0ce3606eb48',
'0xc02aaa39b223fe8d0a0e5c4f27ead9083c756cc2',
1000000
)
)
);
```
- `utils.udf_create_eth_call`: Creates an eth_call JSON-RPC request object from contract address and encoded calldata. Supports block parameter as string or number (auto-converts numbers to hex).
```
-- Using default 'latest' block
SELECT utils.udf_create_eth_call(
'0xa0b86991c6218b36c1d19d4a2e9eb0ce3606eb48',
'0x70a08231000000000000000000000000a0b86991c6218b36c1d19d4a2e9eb0ce3606eb48'
);
-- Using specific block number (auto-converted to hex)
SELECT utils.udf_create_eth_call(
'0xa0b86991c6218b36c1d19d4a2e9eb0ce3606eb48',
'0x70a08231000000000000000000000000a0b86991c6218b36c1d19d4a2e9eb0ce3606eb48',
18500000
);
```
- `utils.udf_create_eth_call_from_abi`: Convenience function that combines contract call encoding and JSON-RPC request creation in a single call. Recommended for most use cases.
```
-- Simple balanceOf call with default 'latest' block
SELECT utils.udf_create_eth_call_from_abi(
'0xa0b86991c6218b36c1d19d4a2e9eb0ce3606eb48',
PARSE_JSON('{
"name": "balanceOf",
"inputs": [{"name": "account", "type": "address"}]
}'),
ARRAY_CONSTRUCT('0xbcca60bb61934080951369a648fb03df4f96263c')
);
-- Same call but at a specific block number
SELECT utils.udf_create_eth_call_from_abi(
'0xa0b86991c6218b36c1d19d4a2e9eb0ce3606eb48',
PARSE_JSON('{
"name": "balanceOf",
"inputs": [{"name": "account", "type": "address"}]
}'),
ARRAY_CONSTRUCT('0xbcca60bb61934080951369a648fb03df4f96263c'),
18500000
);
-- Using ABI from a table
WITH abi_data AS (
SELECT
abi,
'0xa0b86991c6218b36c1d19d4a2e9eb0ce3606eb48' as contract_address,
'0xbcca60bb61934080951369a648fb03df4f96263c' as user_address
FROM ethereum.silver.flat_function_abis
WHERE contract_address = LOWER('0x43506849d7c04f9138d1a2050bbf3a0c054402dd')
AND function_name = 'balanceOf'
)
SELECT
utils.udf_create_eth_call_from_abi(
contract_address,
abi,
ARRAY_CONSTRUCT(user_address)
) as rpc_call
FROM abi_data;
```
## **Streamline V 2.0 Functions**
The `Streamline V 2.0` functions are a set of macros and UDFs that are designed to be used with `Streamline V 2.0` deployments.
### Available macros:
- [if_data_call_function_v2](/macros/streamline/utils.sql#L86): This macro is used to call a udf in the `Streamline V 2.0` deployment. It is defined in the dbt model config block and accepts the `udf name` and the `udf` parameters. For legibility the `udf` parameters are passed as a `JSON object`.
**NOTE**: Ensure your project has registered the `udf` being invoked here prior to using this macro.
**`Parameters`**:
- `func` - The name of the udf to be called.
- `target` - The target table for the udf to be called on, interpolated in the [if_data_call_function_v2 macro](/macros/streamline/utils.sql#L101).
- `params` - The parameters to be passed to the udf, a `JSON object` that contains the minimum parameters required by the udf all Streamline 2.0 udfs.
```sql
-- Example usage in a dbt model config block
{{ config (
materialized = "view",
post_hook = fsc_utils.if_data_call_function_v2(
func = 'streamline.udf_bulk_rest_api_v2',
target = "{{this.schema}}.{{this.identifier}}",
params = {
"external_table": "external_table",
"sql_limit": "10",
"producer_batch_size": "10",
"worker_batch_size": "10",
"sql_source": "{{this.identifier}}",
"exploded_key": tojson(["result.transactions"])
}
),
tags = ['model_tags']
) }}
```
When a dbt model with this config block is run we will see the following in the logs:
```sh
# Example dbt run logs
21:59:44 Found 244 models, 15 seeds, 7 operations, 5 analyses, 875 tests, 282 sources, 0 exposures, 0 metrics, 1024 macros, 0 groups, 0 semantic models
21:59:44
21:59:49
21:59:49 Running 6 on-run-start hooks
...
21:59:50
21:59:51 Concurrency: 12 threads (target='dev')
21:59:51
21:59:51 1 of 1 START sql view model streamline.coingecko_realtime_ohlc ................. [RUN]
21:59:51 Running macro `if_data_call_function`: Calling udf udf_bulk_rest_api_v2 with params:
{
"external_table": "ASSET_OHLC_API/COINGECKO",
"producer_batch_size": "10",
"sql_limit": "10",
"sql_source": "{{this.identifier}}",
"worker_batch_size": "10",
"exploded_key": tojson(["result.transactions"])
}
on {{this.schema}}.{{this.identifier}}
22:00:03 1 of 1 OK created sql view model streamline.coingecko_realtime_ohlc ............ [SUCCESS 1 in 12.75s]
22:00:03
```
```yml
# Setup variables in dbt_project.yml
API_INTEGRATION: '{{ var("config")[target.name]["API_INTEGRATION"] }}'
EXTERNAL_FUNCTION_URI: '{{ var("config")[target.name]["EXTERNAL_FUNCTION_URI"] }}'
ROLES: '{{ var("config")[target.name]["ROLES"] }}'
config:
# The keys correspond to dbt profiles and are case sensitive
dev:
API_INTEGRATION: AWS_CROSSCHAIN_API_STG
EXTERNAL_FUNCTION_URI: q0bnjqvs9a.execute-api.us-east-1.amazonaws.com/stg/
ROLES:
- AWS_LAMBDA_CROSSCHAIN_API
- INTERNAL_DEV
prod:
API_INTEGRATION: AWS_CROSSCHAIN_API_PROD
EXTERNAL_FUNCTION_URI: 35hm1qhag9.execute-api.us-east-1.amazonaws.com/prod/
ROLES:
- AWS_LAMBDA_CROSSCHAIN_API
- INTERNAL_DEV
- DBT_CLOUD_CROSSCHAIN
```
- [create_udf_bulk_rest_api_v2](/macros/streamline/udfs.sql#L1): This macro is used to create a `udf` named `udf_bulk_rest_api_v2` in the `streamline` schema of the database this is invoked in. This function returns a `variant` type and uses an API integration. The API integration and the external function URI are determined based on the target environment (`prod`, `dev`, or `sbx`).
The [macro interpolates](/macros/streamline/udfs.sql#L9) the `API_INTEGRATION` and `EXTERNAL_FUNCTION_URI` vars from the `dbt_project.yml` file. This is available starting with `v1.27.0`.
**NOTE**: To be congruent with how `EXTERNAL_FUNCTION_URI` is being used by other macros and maintain consistency, starting from `v1.21.7` we need to append a trailing `/` to the `EXTERNAL_FUNCTION_URI` in the `dbt_project.yml` file.
- [create_udf_bulk_decode_logs](/macros/streamline/udfs.sql#L25): This macro is used to create a `udf` name `udf_bulk_decode_logs_v2 ` in the `streamline` schema of the databae this is invoked in. This function returns a `variant` type and uses an API integration. The API integration and the external function URI are determined based on the target environment (`prod`, `dev`, or `sbx`).
The [macro interpolates](/macros/streamline/udfs.sql#L32) the `API_INTEGRATION` and `EXTERNAL_FUNCTION_URI` vars from the `dbt_project.yml` file.
- [create_streamline_udfs](macros/create_streamline_udfs.sql#L1). This macro runs [create_udf_bulk_rest_api_v2](/macros/streamline/udfs.sql#L1) when ran with `--vars '{UPDATE_UDFS_AND_SPS: true}'`.
- [create_evm_streamline_udfs](macros/create_streamline_udfs.sql#L8). This macro runs [create_udf_bulk_rest_api_v2](/macros/streamline/udfs.sql#L1), [create_udf_bulk_decode_logs](/macros/streamline/udfs.sql#L25), and [create_udf_bulk_decode_traces](/macros/streamline/udfs.sql#L49) when ran with `--vars '{UPDATE_UDFS_AND_SPS: true}'`. This is designed to be used on the EVM chains due to the inclusion of `create_udf_bulk_decode_logs` and `create_udf_bulk_decode_traces`.
## **LiveQuery Functions**
LiveQuery is now available to be deployed into individual projects. For base functionality, you will need to deploy the core functions using `dbt run` in your project and reference the path to the LiveQuery schema or by tag.
### Basic Setup
1. Make sure `fsc-utils` package referenced in the project is version `v1.33.2` or greater. Re-run `dbt deps` if revision was changed.
`livequery_models deploy core` uses ephemeral models, therefore it is recommended to specify the materialization for `livequery_models` in your project's `dbt_project.yml` to `ephemeral` to avoid any conflicts.
```yml
# dbt_project.yml
---
models:
livequery_models:
deploy:
core:
materialized: ephemeral
```
2. Deploy the core LiveQuery functions by schema or tag
By Schema
```
dbt run -s livequery_models.deploy.core --vars '{UPDATE_UDFS_AND_SPS: true}'
```
By Tag
```
dbt run -s "livequery_models,tag:core" --vars '{UPDATE_UDFS_AND_SPS: true}'
```
3. Deploy any additional functions
For example, deploy quicknode solana nft function + any dependencies (in this case the quicknode utils function)
```
dbt run -s livequery_models.deploy.quicknode.quicknode_utils__quicknode_utils livequery_models.deploy.quicknode.quicknode_solana_nfts__quicknode_utils --vars '{UPDATE_UDFS_AND_SPS: true}'
```
4. Override default LiveQuery configuration values by adding the below lines in the `vars` section of your project's `dbt_project.yml`
```
API_INTEGRATION: '{{ var("config")[target.name]["API_INTEGRATION"] if var("config")[target.name] else var("config")["dev"]["API_INTEGRATION"] }}'
EXTERNAL_FUNCTION_URI: '{{ var("config")[target.name]["EXTERNAL_FUNCTION_URI"] if var("config")[target.name] else var("config")["dev"]["EXTERNAL_FUNCTION_URI"] }}'
ROLES: |
["INTERNAL_DEV"]
```
### Configuring LiveQuery API endpoints
Individual projects have the option to point to a different LiveQuery API endpoint. To do so, modify your project's `dbt_projects.yml` to include the additional configurations within the project `vars`. If no configurations are specified, the default endpoints defined in the `livequery_models` package are used.
Below is a sample configuration. The `API_INTEGRATION` and `EXTERNAL_FUNCTION_URI` should point to the specific resources deployed for your project. The `ROLES` property is a list of Snowflake role names that are granted usage to the LiveQuery functions on deployment. You can also add the optional `MAX_BATCH_ROWS` variable to limit the number of rows processed in a single batch to the `udf_api_batched` function (available starting with `v1.8.0`).
```
config:
# The keys correspond to dbt profiles and are case sensitive
dev:
API_INTEGRATION: AWS_MY_PROJECT_LIVE_QUERY
EXTERNAL_FUNCTION_URI: myproject.api.livequery.com/path-to-endpoint/
ROLES:
- INTERNAL_DEV
MAX_BATCH_ROWS: 10
```
## Snowflake Tasks for GitHub Actions
A set of macros and UDFs have been created to help with the creation of Snowflake tasks to manage runs in GitHub Actions.
### Basic Setup
1. Make sure `fsc-utils` package referenced in the project is version `v1.11.0` or greater. Re-run `dbt deps` if revision was changed.
2. Make sure LiveQuery has been deployed to the project. See [LiveQuery Functions](#livequery-functions) for more information.
> If you are using tags to run your workflows, it is highly recommend to add the project name to the tag. For example, `"ethereum_models,tag:core"` instead of `tag:core`. This will ensure that the correct workflows are being ran within your project.
3. Install the GitHub LiveQuery Functions
```
dbt run -s livequery_models.deploy.marketplace.github --vars '{UPDATE_UDFS_AND_SPS: true}'
```
Use `-t prod` when running in production
GitHub secrets have been registered to the Snowflake System account, which is the user that will execute tasks. If you wish to use a different user to interact with the GitHub API, you will need to register the secrets to that user using [Ephit](https://science.flipsidecrypto.xyz/ephit).
4. Deploy UDFs from `fsc-utils` package
```
dbt run-operation fsc_utils.create_udfs --vars '{UPDATE_UDFS_AND_SPS: true}'
```
Use `-t prod` when running in production
Alternatively, you can add `{{- fsc_utils.create_udfs() -}}` to the `create_udfs` macro in your project to deploy the UDFs from `fsc-utils` on model start and when `UPDATE_UDFS_AND_SPS` is set to `True`.
5. Add `github_actions__workflows.csv` to the data folder in your project. This file will contain the list of workflows to be created. The workflow name should be the same as the name of the `.yml` file in your project. It is recommended that the file name be the same as the workflow and run name. See [Polygon](https://github.com/FlipsideCrypto/polygon-models/blob/main/data/github_actions__workflows.csv) for sample format.
Seed the file into dbt
```
dbt seed -s github_actions__workflows
```
Add file to `sources.yml`
```
- name: github_actions
database: {{prod_db}}
schema: github_actions
tables:
- name: workflows
```
If you would like to test in dev, you will need to seed your file to prod with a separate PR.
6. Add the `github_actions` folder to your project's `models` folder. This folder contains the models that will be used to create and monitor the workflows. See [Polygon](https://github.com/FlipsideCrypto/polygon-models/tree/main/models/github_actions)
Build the GitHub Actions View
```
dbt run -m models/github_actions --full-refresh
```
Add `--vars '{UPDATE_UDFS_AND_SPS: true}'` if you have not already created UDFs on version `v1.11.0` or greater.
7. Add the template workflows `dbt_alter_gha_tasks.yml` and `dbt_test_tasks.yml`
> The [alter workflow](https://github.com/FlipsideCrypto/arbitrum-models/blob/main/.github/workflows/dbt_alter_gha_task.yml) is used to `SUSPEND` or `RESUME` tasks, which you will need to do if you want to pause a workflow while merging a big PR, for example. This is intended to be ran on an ad-hoc basis.
> The [test workflow](https://github.com/FlipsideCrypto/arbitrum-models/blob/main/.github/workflows/dbt_test_tasks.yml) is used to test the workflows. It ensures that workflows are running according to the schedule and that the tasks are completing successfully. You will want to include this workflow within `github_actions__workflows.csv`. You can change the `.yml` included in the `models/github_actions` folder to better suite your testing needs, if necessary.
8. Remove the cron schedule from any workflow `.yml` files that have been added to `github_actions__workflows.csv`, replace with workflow_dispatch:
```
on:
workflow_dispatch:
branches:
- "main"
```
9. Add the `START_GHA_TASKS` variable to `dbt_project.yml`
```
START_GHA_TASKS: False
```
10. Create the Tasks
```
dbt run-operation fsc_utils.create_gha_tasks --vars '{"START_GHA_TASKS":True}'
```
> This will create the tasks in Snowflake and the workflows in GitHub Actions. The tasks will only be started if `START_GHA_TASKS` is set to `True` and the target is the production database for your project.
11. Add a Data Dog CI Pipeline Alert on the logs of `dbt_test_tasks` to ensure that the test is checking the workflows successfully. See `Polygon Task Alert` in Data Dog for sample alert.
## Dynamic Merge Predicate
A set of macros to help with generating dynamic merge predicate statements for models in chain projects. Specifically this will output a concatenanted set of BETWEEN statements of contiguous ranges.
### Setup and Usage
The macro only supports generating predicates for column types of DATE and INTEGER
1. Make sure fsc-utils package referenced in the project is version `v1.16.1` or greater. Re-run dbt deps if revision was changed.
#### Inline Usage
{% set between_stmts = fsc_utils.dynamic_range_predicate("silver.my_temp_table", "block_timestamp::date") %}
...
SELECT
*
FROM
some_other_table
WHERE
{{ between_stmts }}
#### DBT Snowflake incremental_predicate Usage
1. Requires overriding behavior of `get_merge_sql` macro
2. Create a file in `macros/dbt/` ex: `macros/dbt/get_merge.sql`
3. Copy this to the new file
```
{% macro get_merge_sql(target, source, unique_key, dest_columns, incremental_predicates) -%}
{% set merge_sql = fsc_utils.get_merge_sql(target, source, unique_key, dest_columns, incremental_predicates) %}
{{ return(merge_sql) }}
{% endmacro %}
```
**NOTE**: This is backwards compatible with the default dbt merge behavior, however it does override the default macro. If additional customization is needed, the above macro should be modified.
4. Example usage to create predicates using block_id
```
{{ config(
...
incremental_predicates = ["dynamic_range_predicate", "block_id"],
...
) }}
```
Example Output: `(DBT_INTERNAL_DEST.block_id between 100 and 200 OR DBT_INTERNAL_DEST.block_id between 100000 and 150000)`
## Resources
- Learn more about dbt [in the docs](https://docs.getdbt.com/docs/introduction)
- Check out [Discourse](https://discourse.getdbt.com/) for commonly asked questions and answers
- Join the [chat](https://community.getdbt.com/) on Slack for live discussions and support
- Find [dbt events](https://events.getdbt.com) near you
- Check out [the blog](https://blog.getdbt.com/) for the latest news on dbt's development and best practices