flip/flow-models
1
0
Fork 0
mirror of https://github.com/FlipsideCrypto/flow-models.git synced 2026-02-06 11:47:00 +00:00
Code Issues Actions 29 Packages Projects Releases Wiki Activity

AN-6231/modernize incremental predicates (#442)
Some checks failed
docs_update / run_dbt_jobs (push) Has been cancelled
Details
dbt_run_streamline_evm_daily_realtime / run_dbt_jobs (push) Has been cancelled
Details
dbt_run_moments_metadata / run_dbt_jobs (push) Has been cancelled
Details
dbt_observability_models / run_dbt_jobs (push) Has been cancelled
Details
dbt_run_streamline_evm_daily_silver / run_dbt_jobs (push) Has been cancelled
Details
dbt_run_evm_decoded_logs / run_dbt_jobs (push) Has been cancelled
Details
dbt_run_streamline_transactions / run_dbt_jobs (push) Has been cancelled
Details
dbt_run_streamline_transaction_results / run_dbt_jobs (push) Has been cancelled
Details
dbt_run_streamline_external_realtime / run_dbt_jobs (push) Has been cancelled
Details
dbt_run_streamline_external_points_balances_realtime / run_dbt_jobs (push) Has been cancelled
Details
dbt_run_streamline_evm_realtime / run_dbt_jobs (push) Has been cancelled
Details
dbt_run_scheduled / run_dbt_jobs (push) Has been cancelled
Details
dbt_run_streamline_blocks / run_dbt_jobs (push) Has been cancelled
Details
dbt_run_evm / run_dbt_jobs (push) Has been cancelled
Details
dbt_run_scheduled_non_core / run_dbt_jobs (push) Has been cancelled
Details
dbt_run_streamline_collections / run_dbt_jobs (push) Has been cancelled
Details
docs_update / notify-failure (push) Has been cancelled
Details
dbt_run_streamline_evm_daily_realtime / notify-failure (push) Has been cancelled
Details
dbt_run_moments_metadata / notify-failure (push) Has been cancelled
Details
dbt_observability_models / notify-failure (push) Has been cancelled
Details
dbt_run_streamline_evm_daily_silver / notify-failure (push) Has been cancelled
Details
dbt_run_evm_decoded_logs / notify-failure (push) Has been cancelled
Details
dbt_run_streamline_transactions / notify-failure (push) Has been cancelled
Details
dbt_run_streamline_transaction_results / notify-failure (push) Has been cancelled
Details
dbt_run_streamline_external_realtime / notify-failure (push) Has been cancelled
Details
dbt_run_streamline_external_points_balances_realtime / notify-failure (push) Has been cancelled
Details
dbt_run_streamline_evm_realtime / notify-failure (push) Has been cancelled
Details
dbt_run_scheduled / notify-failure (push) Has been cancelled
Details
dbt_run_streamline_blocks / notify-failure (push) Has been cancelled
Details
dbt_run_evm / notify-failure (push) Has been cancelled
Details
dbt_run_scheduled_non_core / notify-failure (push) Has been cancelled
Details
dbt_run_streamline_collections / notify-failure (push) Has been cancelled
Details
dbt_run_streamline_decoded_logs_history / run_dbt_jobs (push) Has been cancelled
Details
dbt_run_streamline_decoded_logs_history / notify-failure (push) Has been cancelled
Details

Browse Source 
* upd incr predicate on silver core models

* upd rest of the models

* rm predicate from d+i model
This commit is contained in:
Jack Forgash 2025-08-06 12:40:17 -04:00 committed by GitHub
parent 5194f813dd
commit 45a68debed
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
20 changed files with 25 additions and 8 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
macros/dbt/get_merge.sql Normal file
View File

@ -0,0 +1,4 @@
{% macro get_merge_sql(target, source, unique_key, dest_columns, incremental_predicates) -%}
{% set merge_sql = fsc_utils.get_merge_sql(target, source, unique_key, dest_columns, incremental_predicates) %}
{{ return(merge_sql) }}
{% endmacro %}

1
models/evm/silver/core/silver_evm__created_contracts.sql
View File

@ -1,5 +1,6 @@
{{ config (
materialized = "incremental",
incremental_predicates = ["dynamic_range_predicate", "block_timestamp::DATE"],
unique_key = "created_contract_address",
merge_exclude_columns = ["inserted_timestamp"],
post_hook = "ALTER TABLE {{ this }} ADD SEARCH OPTIMIZATION ON EQUALITY(block_timestamp, tx_hash, created_contract_address, creator_address), SUBSTRING(created_contract_address, creator_address)",

1
models/evm/streamline/core/complete/streamline__complete_get_evm_blocks.sql
View File

@ -2,6 +2,7 @@
-- depends_on: {{ ref('bronze_evm__FR_blocks') }}
{{ config (
materialized = "incremental",
incremental_predicates = ["dynamic_range_predicate", "partition_key"],
unique_key = "block_number",
cluster_by = "ROUND(block_number, -3)",
merge_exclude_columns = ["inserted_timestamp"],

1
models/evm/streamline/core/complete/streamline__complete_get_evm_receipts.sql
View File

@ -2,6 +2,7 @@
-- depends_on: {{ ref('bronze_evm__FR_receipts') }}
{{ config (
materialized = "incremental",
incremental_predicates = ["dynamic_range_predicate", "partition_key"],
unique_key = "block_number",
cluster_by = "ROUND(block_number, -3)",
merge_exclude_columns = ["inserted_timestamp"],

1
models/evm/streamline/core/complete/streamline__complete_get_evm_traces.sql
View File

@ -2,6 +2,7 @@
-- depends_on: {{ ref('bronze_evm__FR_traces') }}
{{ config (
materialized = "incremental",
incremental_predicates = ["dynamic_range_predicate", "partition_key"],
unique_key = "block_number",
cluster_by = "ROUND(block_number, -3)",
merge_exclude_columns = ["inserted_timestamp"],

1
models/gold/core/core__dim_address_mapping.sql
View File

@ -1,5 +1,6 @@
{{ config (
materialized = 'incremental',
incremental_predicates = ["dynamic_range_predicate", "block_timestamp_associated::DATE"],
incremental_strategy = 'merge',
merge_exclude_columns = ['inserted_timestamp'],
unique_key = 'dim_address_mapping_id',

2
models/gold/core/core__ez_token_transfers.sql
View File

@ -2,7 +2,7 @@
materialized = 'incremental',
incremental_strategy = 'merge',
merge_exclude_columns = ['inserted_timestamp'],
incremental_predicates = ["COALESCE(DBT_INTERNAL_DEST.block_timestamp::DATE,'2099-12-31') >= (select min(block_timestamp::DATE) from " ~ generate_tmp_view_name(this) ~ ")"],
incremental_predicates = ["dynamic_range_predicate", "block_timestamp::DATE"],
cluster_by = ['block_timestamp::date', 'modified_timestamp::date'],
unique_key = "ez_token_transfers_id",
post_hook = "ALTER TABLE {{ this }} ADD SEARCH OPTIMIZATION ON EQUALITY(tx_id,sender,recipient,token_contract);",

2
models/gold/core/core__ez_transaction_actors.sql
View File

@ -3,7 +3,7 @@
unique_key = 'ez_transaction_actors_id',
incremental_strategy = 'merge',
merge_exclude_columns = ['inserted_timestamp'],
incremental_predicates = ["COALESCE(DBT_INTERNAL_DEST.block_timestamp::DATE,'2099-12-31') >= (select min(block_timestamp::DATE) from " ~ generate_tmp_view_name(this) ~ ")"],
incremental_predicates = ["dynamic_range_predicate", "block_timestamp::DATE"],
cluster_by = 'block_timestamp::date',
post_hook = 'ALTER TABLE {{ this }} ADD SEARCH OPTIMIZATION ON EQUALITY(tx_id,actors);',
tags = ['scheduled_non_core']

1
models/silver/core/silver__streamline_blocks.sql
View File

@ -5,6 +5,7 @@
materialized = 'incremental',
unique_key = "block_number",
incremental_strategy = 'merge',
incremental_predicates = ["dynamic_range_predicate", "_partition_by_block_id"],
merge_exclude_columns = ["inserted_timestamp"],
cluster_by = "block_timestamp::date",
tags = ['streamline_load', 'core', 'scheduled_core']

1
models/silver/core/silver__streamline_collections.sql
View File

@ -3,6 +3,7 @@
materialized = 'incremental',
unique_key = "collection_id",
incremental_strategy = 'merge',
incremental_predicates = ["dynamic_range_predicate", "block_number"],
merge_exclude_columns = ["inserted_timestamp"],
cluster_by = ['_inserted_timestamp :: DATE', 'block_number'],
tags = ['streamline_load', 'core', 'scheduled_core']

1
models/silver/core/silver__streamline_events.sql
View File

@ -2,6 +2,7 @@
materialized = 'incremental',
unique_key = 'event_id',
incremental_strategy = 'merge',
incremental_predicates = ["dynamic_range_predicate", "block_timestamp::date"],
merge_exclude_columns = ["inserted_timestamp"],
cluster_by = "block_timestamp::date",
post_hook = "ALTER TABLE {{ this }} ADD SEARCH OPTIMIZATION ON EQUALITY(tx_id,event_id,event_contract,event_type);",

2
models/silver/core/silver__streamline_transaction_results.sql
View File

@ -1,7 +1,7 @@
-- depends_on: {{ ref('bronze__streamline_transaction_results') }}
{{ config(
materialized = 'incremental',
incremental_predicates = ['DBT_INTERNAL_DEST.block_number >= (select min(block_number) from ' ~ generate_tmp_view_name(this) ~ ')'],
incremental_predicates = ["dynamic_range_predicate", "_partition_by_block_id"],
unique_key = "tx_id",
incremental_strategy = 'merge',
merge_exclude_columns = ["inserted_timestamp"],

1
models/silver/core/silver__streamline_transactions.sql
View File

@ -3,6 +3,7 @@
materialized = 'incremental',
unique_key = "tx_id",
incremental_strategy = 'merge',
incremental_predicates = ["dynamic_range_predicate", "_partition_by_block_id"],
merge_exclude_columns = ["inserted_timestamp"],
cluster_by = "_inserted_timestamp::date",
tags = ['streamline_load', 'core', 'scheduled_core']

1
models/silver/core/silver__streamline_transactions_final.sql
View File

@ -3,6 +3,7 @@
materialized = 'incremental',
unique_key = "tx_id",
incremental_strategy = 'merge',
incremental_predicates = ["dynamic_range_predicate", "_partition_by_block_id"],
merge_exclude_columns = ["inserted_timestamp"],
cluster_by = "block_timestamp::date",
post_hook = "ALTER TABLE {{ this }} ADD SEARCH OPTIMIZATION ON EQUALITY(tx_id,proposer,payer,authorizers);",

1
models/silver/curated/silver__flow_evm_address_map.sql
View File

@ -1,5 +1,6 @@
{{ config(
materialized = 'incremental',
incremental_predicates = ["dynamic_range_predicate", "block_timestamp::DATE"],
unique_key = 'flow_evm_address_map_id',
incremental_strategy = 'merge',
merge_exclude_columns = ['inserted_timestamp'],

1
models/silver/defi/silver__swaps_aggregator.sql
View File

@ -1,5 +1,6 @@
{{ config(
materialized = 'incremental',
incremental_predicates = ["dynamic_range_predicate", "block_timestamp::DATE"],
incremental_strategy = 'merge',
merge_exclude_columns = ["inserted_timestamp"],
cluster_by = ['inserted_timestamp::DATE'],

1
models/silver/transfers/silver__bridge_complete.sql
View File

@ -1,5 +1,6 @@
{{ config(
materialized = 'incremental',
incremental_predicates = ["dynamic_range_predicate", "block_timestamp::DATE"],
incremental_strategy = 'merge',
merge_exclude_columns = ["inserted_timestamp"],
cluster_by = ['block_timestamp::date'],

2
models/silver/transfers/silver__token_transfers.sql
View File

@ -2,7 +2,7 @@
materialized = 'incremental',
incremental_strategy = 'merge',
merge_exclude_columns = ['inserted_timestamp'],
incremental_predicates = ["COALESCE(DBT_INTERNAL_DEST.block_timestamp::DATE,'2099-12-31') >= (select min(block_timestamp::DATE) from " ~ generate_tmp_view_name(this) ~ ")"],
incremental_predicates = ["dynamic_range_predicate", "block_timestamp::DATE"],
cluster_by = ['block_timestamp::date', 'modified_timestamp::date'],
unique_key = "token_transfers_id",
tags = ['scheduled_non_core']

6
package-lock.yml
View File

@ -6,11 +6,11 @@ packages:
- package: dbt-labs/dbt_utils
version: 1.0.0
- git: https://github.com/FlipsideCrypto/fsc-utils.git
revision: d3cf679e079f0cf06142de9386f215e55fe26b3b
revision: 87e00eb90acddcc7a34aa8e67e3b3bac86b262e6
- package: get-select/dbt_snowflake_query_tags
version: 2.5.0
- package: calogica/dbt_date
version: 0.7.2
- git: https://github.com/FlipsideCrypto/livequery-models.git
revision: b024188be4e9c6bc00ed77797ebdc92d351d620e
sha1_hash: 3fb8d6ca492a03f5aef6f281508aaa0b34c989d4
revision: 2651a45b7e123f7bd421bcc0e7e2a7bcbaf7652f
sha1_hash: a1cc3545d7ef13fcf5b3908a9e888b4421018792

2
packages.yml
View File

@ -6,6 +6,6 @@ packages:
- package: dbt-labs/dbt_utils
version: 1.0.0
- git: https://github.com/FlipsideCrypto/fsc-utils.git
revision: v1.32.0
revision: v1.35.1
- package: get-select/dbt_snowflake_query_tags
version: [">=2.0.0", "<3.0.0"]