diff --git a/README.md b/README.md index 6a46cf4..b0b9cb3 100644 --- a/README.md +++ b/README.md @@ -35,36 +35,43 @@ blast: query_tag: ``` -### Variables +## Variables To control the creation of UDF or SP macros with dbt run: * UPDATE_UDFS_AND_SPS -When True, executes all macros included in the on-run-start hooks within dbt_project.yml on model run as normal -When False, none of the on-run-start macros are executed on model run + * Default values are False + * When True, executes all macros included in the on-run-start hooks within dbt_project.yml on model run as normal + * When False, none of the on-run-start macros are executed on model run -Default values are False + * Usage: `dbt run --vars '{"UPDATE_UDFS_AND_SPS":True}' -m ...` -* Usage: -dbt run --vars '{"UPDATE_UDFS_AND_SPS":True}' -m ... +Use a variable to heal a model incrementally: +* HEAL_MODEL + * Default is FALSE (Boolean) + * When FALSE, logic will be negated + * When TRUE, heal logic will apply + * Include `heal` in model tags within the config block for inclusion in the `dbt_run_heal_models` workflow, e.g. `tags = 'heal'` -To reload records in a curated complete table without a full-refresh, such as `silver_bridge.complete_bridge_activity`: -* HEAL_CURATED_MODEL -Default is an empty array [] -When item is included in var array [], incremental logic will be skipped for that CTE / code block -When item is not included in var array [] or does not match specified item in model, incremental logic will apply -Example set up: `{% if is_incremental() and 'axelar' not in var('HEAL_CURATED_MODEL') %}` + * Usage: `dbt run --vars '{"HEAL_MODEL":True}' -m ...` -* Usage: -Single CTE: dbt run --vars '{"HEAL_CURATED_MODEL":"axelar"}' -m ... -Multiple CTEs: dbt run --vars '{"HEAL_CURATED_MODEL":["axelar","across","celer_cbridge"]}' -m ... +Use a variable to negate incremental logic: +* Example use case: reload records in a curated complete table without a full-refresh, such as `silver_bridge.complete_bridge_activity`: +* HEAL_MODELS + * Default is an empty array [] + * When item is included in var array [], incremental logic will be skipped for that CTE / code block + * When item is not included in var array [] or does not match specified item in model, incremental logic will apply + * Example set up: `{% if is_incremental() and 'axelar' not in var('HEAL_MODELS') %}` + * Usage: + * Single CTE: `dbt run --vars '{"HEAL_MODELS":"axelar"}' -m ...` + * Multiple CTEs: `dbt run --vars '{"HEAL_MODELS":["axelar","across","celer_cbridge"]}' -m ...` -### Resources: -- Learn more about dbt [in the docs](https://docs.getdbt.com/docs/introduction) -- Check out [Discourse](https://discourse.getdbt.com/) for commonly asked questions and answers -- Join the [chat](https://community.getdbt.com/) on Slack for live discussions and support -- Find [dbt events](https://events.getdbt.com) near you -- Check out [the blog](https://blog.getdbt.com/) for the latest news on dbt's development and best practices +Use a variable to extend the incremental lookback period: +* LOOKBACK + * Default is a string representing the specified time interval e.g. '12 hours', '7 days' etc. + * Example set up: `SELECT MAX(_inserted_timestamp) - INTERVAL '{{ var("LOOKBACK", "4 hours") }}'` + + * Usage: `dbt run --vars '{"LOOKBACK":"36 hours"}' -m ...` ## Applying Model Tags @@ -106,4 +113,11 @@ dbt run --vars '{"UPDATE_SNOWFLAKE_TAGS":False}' -s models/core/core__fact_block ``` select * from table(blast.information_schema.tag_references('blast.core.fact_blocks', 'table')); -``` \ No newline at end of file +``` + +### Resources: +- Learn more about dbt [in the docs](https://docs.getdbt.com/docs/introduction) +- Check out [Discourse](https://discourse.getdbt.com/) for commonly asked questions and answers +- Join the [chat](https://community.getdbt.com/) on Slack for live discussions and support +- Find [dbt events](https://events.getdbt.com) near you +- Check out [the blog](https://blog.getdbt.com/) for the latest news on dbt's development and best practices \ No newline at end of file diff --git a/dbt_project.yml b/dbt_project.yml index 2ae45fc..1701a9b 100644 --- a/dbt_project.yml +++ b/dbt_project.yml @@ -65,7 +65,7 @@ vars: OBSERV_FULL_TEST: False WAIT: 0 HEAL_MODEL: False - HEAL_CURATED_MODEL: [] + HEAL_MODELS: [] START_GHA_TASKS: False #### STREAMLINE 2.0 BEGIN #### diff --git a/models/silver/core/silver__transfers.sql b/models/silver/core/silver__transfers.sql index 7db266f..0592226 100644 --- a/models/silver/core/silver__transfers.sql +++ b/models/silver/core/silver__transfers.sql @@ -35,7 +35,7 @@ AND _inserted_timestamp >= ( SELECT MAX( _inserted_timestamp - ) - INTERVAL '36 hours' + ) - INTERVAL '{{ var("LOOKBACK", "4 hours") }}' FROM {{ this }} ) @@ -169,7 +169,7 @@ heal_model AS ( SELECT MAX( _inserted_timestamp - ) - INTERVAL '36 hours' + ) - INTERVAL '{{ var("LOOKBACK", "4 hours") }}' FROM {{ this }} ) @@ -195,7 +195,7 @@ heal_model AS ( SELECT MAX( _inserted_timestamp - ) - INTERVAL '36 hours' + ) - INTERVAL '{{ var("LOOKBACK", "4 hours") }}' FROM {{ this }} )