diff --git a/models/descriptions/__overview__.md b/models/descriptions/__overview__.md index f8b4700..818a2f8 100644 --- a/models/descriptions/__overview__.md +++ b/models/descriptions/__overview__.md @@ -24,6 +24,7 @@ There is more information on how to use dbt docs in the last section of this doc - [fact_transactions](#!/model/model.movement_models.core__fact_transactions) - [fact_transactions_block_metadata](#!/model/model.movement_models.core__fact_transactions_block_metadata) - [fact_transactions_state_checkpoint](#!/model/model.movement_models.core__fact_transactions_state_checkpoint) +- [fact_transfers](#!/model/model.movement_models.core__fact_transfers) ### Price Tables (`movement`.`PRICE`.``) @@ -73,8 +74,7 @@ Note that you can also right-click on models to interactively filter and explore ### **More information** - [Flipside](https://flipsidecrypto.xyz/) -- [Velocity](https://app.flipsidecrypto.com/velocity?nav=Discover) -- [Tutorials](https://docs.flipsidecrypto.com/our-data/tutorials) +- [Community](https://docs.flipsidecrypto.xyz/welcome-to-flipside/flipside-community-overview) - [Github](https://github.com/FlipsideCrypto/movement-models) - [What is dbt?](https://docs.getdbt.com/docs/introduction) diff --git a/models/descriptions/block_number_first.md b/models/descriptions/block_number_first.md new file mode 100644 index 0000000..6b9a0a9 --- /dev/null +++ b/models/descriptions/block_number_first.md @@ -0,0 +1,5 @@ +{% docs block_number_first %} + +Block number of the first time the store metadata was posted to the chain + +{% enddocs %} \ No newline at end of file diff --git a/models/descriptions/block_timestamp_first.md b/models/descriptions/block_timestamp_first.md new file mode 100644 index 0000000..339cd2d --- /dev/null +++ b/models/descriptions/block_timestamp_first.md @@ -0,0 +1,5 @@ +{% docs block_timestamp_first %} + +Block timestamp of the first time the store metadata was posted to the chain + +{% enddocs %} \ No newline at end of file diff --git a/models/descriptions/from_address.md b/models/descriptions/from_address.md new file mode 100644 index 0000000..a452c37 --- /dev/null +++ b/models/descriptions/from_address.md @@ -0,0 +1,5 @@ +{% docs from_address %} + +The account address that sent the transfer. + +{% enddocs %} \ No newline at end of file diff --git a/models/descriptions/is_fungible.md b/models/descriptions/is_fungible.md new file mode 100644 index 0000000..ab90cb9 --- /dev/null +++ b/models/descriptions/is_fungible.md @@ -0,0 +1,5 @@ +{% docs is_fungible %} + +Boolean indicating whether the transfer was conducted using the legacy coin transfer mechanism (a simpler, original method for transferring coins) or the fungible_asset module (a newer, more flexible system for managing fungible assets). + +{% enddocs %} \ No newline at end of file diff --git a/models/descriptions/metadata_address.md b/models/descriptions/metadata_address.md new file mode 100644 index 0000000..bbf92dc --- /dev/null +++ b/models/descriptions/metadata_address.md @@ -0,0 +1,5 @@ +{% docs metadata_address %} + +Metadata address of the token in the store + +{% enddocs %} \ No newline at end of file diff --git a/models/descriptions/owner_address.md b/models/descriptions/owner_address.md new file mode 100644 index 0000000..cd44f3e --- /dev/null +++ b/models/descriptions/owner_address.md @@ -0,0 +1,5 @@ +{% docs owner_address %} + +Address of the store owner + +{% enddocs %} \ No newline at end of file diff --git a/models/descriptions/store_address.md b/models/descriptions/store_address.md new file mode 100644 index 0000000..dfa4a99 --- /dev/null +++ b/models/descriptions/store_address.md @@ -0,0 +1,5 @@ +{% docs store_address %} + +Address of the fungible store + +{% enddocs %} \ No newline at end of file diff --git a/models/descriptions/tables/core__ez_transfers.md b/models/descriptions/tables/core__ez_transfers.md new file mode 100644 index 0000000..f7093d0 --- /dev/null +++ b/models/descriptions/tables/core__ez_transfers.md @@ -0,0 +1,5 @@ +{% docs core__ez_transfers %} + +This table contains a flattened easy version of the native transfers. This table uses the fact_transfers table as a base and filters down to only Movement (Aptos) tokens. The logic used to derive this table requires the withdrawal event to occur in the previous event to the deposit event and also requires the withdrawal event to be the same amount as the deposit event. The only exception to that rule is when a "CoinRegisterEvent" occurs in between the withdraw and deposit events. Any transfers that do not meet this criteria are not included in this table. + +{% enddocs %} diff --git a/models/descriptions/tables/core__fact_transfers.md b/models/descriptions/tables/core__fact_transfers.md new file mode 100644 index 0000000..d1453f8 --- /dev/null +++ b/models/descriptions/tables/core__fact_transfers.md @@ -0,0 +1,5 @@ +{% docs core__fact_transfers %} + +This table contains Deposit and Withdraw events on the Movement blockchain. Note: transfers with a 0 amount are excluded. + +{% enddocs %} diff --git a/models/descriptions/to_address.md b/models/descriptions/to_address.md new file mode 100644 index 0000000..9d9d64d --- /dev/null +++ b/models/descriptions/to_address.md @@ -0,0 +1,5 @@ +{% docs to_address %} + +The account address that received the transfer. + +{% enddocs %} \ No newline at end of file diff --git a/models/descriptions/version.md b/models/descriptions/version.md index ac9836e..39e6f83 100644 --- a/models/descriptions/version.md +++ b/models/descriptions/version.md @@ -1,5 +1,5 @@ {% docs version %} -The version also know as the height of the transactions that have been executed on the Aptos blockchain. The first transaction in the blockchain has a version of 0. The version number is incremented by 1 for each transaction that is executed on the blockchain. +The version also know as the height of the transactions that have been executed on the Movement blockchain. The first transaction in the blockchain has a version of 0. The version number is incremented by 1 for each transaction that is executed on the blockchain. {% enddocs %} \ No newline at end of file diff --git a/models/gold/core/core__ez_transfers.sql b/models/gold/core/core__ez_transfers.sql new file mode 100644 index 0000000..6a86c09 --- /dev/null +++ b/models/gold/core/core__ez_transfers.sql @@ -0,0 +1,43 @@ +{{ config( + materialized = 'incremental', + unique_key = ['tx_hash','to_address','from_address','block_timestamp::DATE'], + incremental_strategy = 'merge', + incremental_predicates = ["dynamic_range_predicate", "block_timestamp::DATE"], + merge_exclude_columns = ["inserted_timestamp"], + cluster_by = ['block_timestamp::DATE'], + post_hook = "ALTER TABLE {{ this }} ADD SEARCH OPTIMIZATION ON EQUALITY(tx_hash, version, from_address, to_address);", + tags = ['core'], + enabled = false +) }} + +SELECT + block_number, + block_timestamp, + tx_hash, + version, + success, + transfer_event, + account_address, + amount, + is_fungible, + token_address, + store_address, + {{ dbt_utils.generate_surrogate_key( + ['tx_hash','to_address','from_address','block_timestamp::DATE'] + ) }} AS ez_transfers_id, + SYSDATE() AS inserted_timestamp, + SYSDATE() AS modified_timestamp, + '{{ invocation_id }}' AS _invocation_id +FROM + {{ ref( + 'core__fact_transfers' + ) }} + +{% if is_incremental() %} +WHERE modified_timestamp >= ( + SELECT + MAX(modified_timestamp) + FROM + {{ this }} +) +{% endif %} \ No newline at end of file diff --git a/models/gold/core/core__fact_transfers.sql b/models/gold/core/core__fact_transfers.sql new file mode 100644 index 0000000..40c7ba0 --- /dev/null +++ b/models/gold/core/core__fact_transfers.sql @@ -0,0 +1,78 @@ +{{ config( + materialized = 'incremental', + unique_key = ['tx_hash','event_index','version','block_timestamp::DATE'], + incremental_strategy = 'merge', + incremental_predicates = ["dynamic_range_predicate", "block_timestamp::DATE"], + merge_exclude_columns = ["inserted_timestamp"], + cluster_by = ['block_timestamp::DATE','modified_timestamp::DATE'], + post_hook = "ALTER TABLE {{ this }} ADD SEARCH OPTIMIZATION ON EQUALITY(tx_hash, version, account_address,token_address);", + tags = ['core'] +) }} + +SELECT + block_number, + block_timestamp, + tx_hash, + version, + success, + event_index, + creation_number, + transfer_event, + account_address, + amount, + token_address, + FALSE AS is_fungible, + NULL :: STRING AS store_address, + {{ dbt_utils.generate_surrogate_key( + ['tx_hash','event_index','version'] + ) }} AS fact_transfers_id, + SYSDATE() AS inserted_timestamp, + SYSDATE() AS modified_timestamp +FROM + {{ ref( + 'silver__transfers' + ) }} +WHERE + amount <> 0 + +{% if is_incremental() %} +AND modified_timestamp >= ( + SELECT + MAX(modified_timestamp) + FROM + {{ this }} +) +{% endif %} + +UNION ALL + +SELECT + block_number, + block_timestamp, + tx_hash, + version, + success, + event_index, + NULL AS creation_number, + transfer_event, + owner_address AS account_address, + amount, + metadata_address AS token_address, + TRUE AS is_fungible, + store_address, + transfers_fungible_id AS fact_transfers_id, + SYSDATE() AS inserted_timestamp, + SYSDATE() AS modified_timestamp, +FROM + {{ ref('silver__transfers_fungible') }} +WHERE + amount <> 0 + +{% if is_incremental() %} +AND modified_timestamp >= ( + SELECT + MAX(modified_timestamp) + FROM + {{ this }} +) +{% endif %} \ No newline at end of file diff --git a/models/gold/core/gold_core.yml b/models/gold/core/gold_core.yml index 79f6206..ad13abf 100644 --- a/models/gold/core/gold_core.yml +++ b/models/gold/core/gold_core.yml @@ -1,5 +1,44 @@ version: 2 models: + - name: core__ez_transfers + description: '{{ doc("core__ez_transfers") }}' + tests: + - dbt_utils.recency: + datepart: hour + field: MODIFIED_TIMESTAMP + interval: 3 + severity: error + tags: ['test_recency'] + columns: + - name: block_number + description: '{{ doc("block_number") }}' + - name: block_timestamp + description: '{{ doc("block_timestamp") }}' + - name: tx_hash + description: '{{ doc("tx_hash") }}' + - name: version + description: '{{ doc("version") }}' + - name: success + description: '{{ doc("success") }}' + - name: transfer_event + description: '{{ doc("transfer_event") }}' + - name: account_address + description: '{{ doc("account_address") }}' + - name: amount + description: '{{ doc("amount") }}' + - name: is_fungible + description: '{{ doc("is_fungible") }}' + - name: token_address + description: '{{ doc("token_address") }}' + - name: store_address + description: '{{ doc("store_address") }}' + - name: ez_transfers_id + description: '{{ doc("pk") }}' + - name: inserted_timestamp + description: '{{ doc("inserted_timestamp") }}' + - name: modified_timestamp + description: '{{ doc("modified_timestamp") }}' + - name: core__fact_blocks description: '{{ doc("core__fact_blocks") }}' tests: @@ -285,3 +324,46 @@ models: description: '{{ doc("inserted_timestamp") }}' - name: MODIFIED_TIMESTAMP description: '{{ doc("modified_timestamp") }}' + + - name: core__fact_transfers + description: '{{ doc("core__fact_transfers") }}' + tests: + - dbt_utils.recency: + datepart: hour + field: MODIFIED_TIMESTAMP + interval: 3 + severity: error + tags: ['test_recency'] + columns: + - name: BLOCK_NUMBER + description: '{{ doc("block_number") }}' + - name: BLOCK_TIMESTAMP + description: '{{ doc("block_timestamp") }}' + - name: TX_HASH + description: '{{ doc("tx_hash") }}' + - name: VERSION + description: '{{ doc("version") }}' + - name: SUCCESS + description: '{{ doc("success") }}' + - name: EVENT_INDEX + description: '{{ doc("event_index") }}' + - name: CREATION_NUMBER + description: '{{ doc("creation_number") }}' + - name: TRANSFER_EVENT + description: '{{ doc("transfer_event") }}' + - name: ACCOUNT_ADDRESS + description: '{{ doc("address_event") }}' + - name: AMOUNT + description: '{{ doc("amount") }}' + - name: TOKEN_ADDRESS + description: '{{ doc("token_address") }}' + - name: IS_FUNGIBLE + description: '{{ doc("is_fungible") }}' + - name: STORE_ADDRESS + description: '{{ doc("store_address") }}' + - name: FACT_TRANSFERS_ID + description: '{{ doc("pk") }}' + - name: INSERTED_TIMESTAMP + description: '{{ doc("inserted_timestamp") }}' + - name: MODIFIED_TIMESTAMP + description: '{{ doc("modified_timestamp") }}' diff --git a/models/silver/core/silver__fungiblestore_metadata.sql b/models/silver/core/silver__fungiblestore_metadata.sql new file mode 100644 index 0000000..9d72b00 --- /dev/null +++ b/models/silver/core/silver__fungiblestore_metadata.sql @@ -0,0 +1,42 @@ +{{ config( + materialized = 'incremental', + unique_key = ['store_address'], + incremental_strategy = 'merge', + merge_exclude_columns = ["inserted_timestamp","block_timestamp_first","block_number_first"], + post_hook = "ALTER TABLE {{ this }} ADD SEARCH OPTIMIZATION ON EQUALITY(store_address);", + tags = ['core'] +) }} + +-- depends_on: {{ ref('core__fact_changes') }} + +SELECT + block_timestamp AS block_timestamp_first, + block_number AS block_number_first, + address AS store_address, + change_data :metadata :inner :: STRING AS metadata_address, + {{ dbt_utils.generate_surrogate_key( + ['store_address'] + ) }} AS fungiblestore_metadata_id, + SYSDATE() AS inserted_timestamp, + SYSDATE() AS modified_timestamp, + '{{ invocation_id }}' AS _invocation_id +FROM + {{ ref('core__fact_changes') }} +WHERE + change_module = 'fungible_asset' + AND change_resource = 'FungibleStore' + +{% if is_incremental() %} +AND modified_timestamp >= ( + SELECT + MAX(modified_timestamp) + FROM + {{ this }} +) +{% endif %} + +qualify ROW_NUMBER() over ( + PARTITION BY address + ORDER BY + block_number +) = 1 diff --git a/models/silver/core/silver__fungiblestore_owners.sql b/models/silver/core/silver__fungiblestore_owners.sql new file mode 100644 index 0000000..ba38ee5 --- /dev/null +++ b/models/silver/core/silver__fungiblestore_owners.sql @@ -0,0 +1,40 @@ +{{ config( + materialized = 'incremental', + unique_key = ['tx_hash','change_index'], + incremental_strategy = 'merge', + cluster_by = ['block_timestamp::DATE'], + post_hook = "ALTER TABLE {{ this }} ADD SEARCH OPTIMIZATION ON EQUALITY(store_address);", + tags = ['core'] +) }} + +-- depends_on: {{ ref('core__fact_changes') }} + +SELECT + block_timestamp, + block_number, + version, + tx_hash, + change_index, + address AS store_address, + change_data :owner :: STRING AS owner_address, + {{ dbt_utils.generate_surrogate_key( + ['tx_hash','change_index'] + ) }} AS fungiblestore_owners_id, + SYSDATE() AS inserted_timestamp, + SYSDATE() AS modified_timestamp, + '{{ invocation_id }}' AS _invocation_id +FROM + {{ ref('core__fact_changes') }} +WHERE + change_address = '0x1' + AND change_module = 'object' + AND change_resource = 'ObjectCore' + +{% if is_incremental() %} +AND modified_timestamp >= ( + SELECT + MAX(modified_timestamp) + FROM + {{ this }} +) +{% endif %} diff --git a/models/silver/core/silver__transfers.sql b/models/silver/core/silver__transfers.sql new file mode 100644 index 0000000..a688a22 --- /dev/null +++ b/models/silver/core/silver__transfers.sql @@ -0,0 +1,148 @@ +{{ config( + materialized = 'incremental', + unique_key = ['tx_hash','event_index','block_timestamp::DATE'], + incremental_strategy = 'merge', + merge_exclude_columns = ["inserted_timestamp"], + cluster_by = ['block_timestamp::DATE','modified_timestamp::DATE'], + tags = ['core'] +) }} + +WITH events AS ( + + SELECT + block_number, + version, + success, + block_timestamp, + block_timestamp :: DATE AS block_date, + tx_hash, + event_index, + event_resource, + event_data :amount :: bigint AS amount, + account_address, + creation_number, + modified_timestamp + FROM + {{ ref( + 'core__fact_events' + ) }} + WHERE + event_module = 'coin' + AND event_resource IN ( + 'WithdrawEvent', + 'DepositEvent' + ) + +{% if is_incremental() %} +AND modified_timestamp >= ( + SELECT + MAX(modified_timestamp) + FROM + {{ this }} +) +{% endif %} +), +changes AS ( + SELECT + block_timestamp :: DATE AS block_date, + tx_hash, + change_index, + change_data, + change_data :deposit_events :guid :id :creation_num :: INT AS creation_number_deposit, + change_data :withdraw_events :guid :id :creation_num :: INT AS creation_number_withdraw, + address, + change_resource AS token_address + FROM + {{ ref( + 'core__fact_changes' + ) }} + WHERE + change_module = 'coin' + AND ( + creation_number_deposit IS NOT NULL + OR creation_number_withdraw IS NOT NULL + ) + +{% if is_incremental() %} +AND modified_timestamp >= ( + SELECT + MAX(modified_timestamp) + FROM + {{ this }} +) +{% endif %} +), +changes_dep AS ( + SELECT + block_date, + tx_hash, + address, + creation_number_deposit AS creation_number, + token_address + FROM + changes + WHERE + creation_number_deposit IS NOT NULL qualify(ROW_NUMBER() over(PARTITION BY tx_hash, creation_number_deposit, address + ORDER BY + change_index DESC) = 1) +), +changes_wth AS ( + SELECT + block_date, + tx_hash, + address, + creation_number_withdraw AS creation_number, + token_address + FROM + changes + WHERE + creation_number_withdraw IS NOT NULL qualify(ROW_NUMBER() over(PARTITION BY tx_hash, creation_number_withdraw, address + ORDER BY + change_index DESC) = 1) +) +SELECT + e.block_number, + e.block_timestamp, + e.tx_hash, + e.version, + e.success, + e.event_index, + e.creation_number, + e.event_resource AS transfer_event, + e.account_address, + e.amount, + REPLACE( + REPLACE( + COALESCE( + dep.token_address, + wth.token_address + ), + 'CoinStore<' + ), + '>' + ) AS token_address, + {{ dbt_utils.generate_surrogate_key( + ['e.tx_hash','e.event_index'] + ) }} AS transfers_id, + SYSDATE() AS inserted_timestamp, + SYSDATE() AS modified_timestamp, + '{{ invocation_id }}' AS _invocation_id +FROM + events e + LEFT JOIN changes_dep dep + ON e.block_date = dep.block_date + AND e.tx_hash = dep.tx_hash + AND e.creation_number = dep.creation_number + AND e.account_address = dep.address + AND e.event_resource = 'DepositEvent' + LEFT JOIN changes_wth wth + ON e.block_date = wth.block_date + AND e.tx_hash = wth.tx_hash + AND e.creation_number = wth.creation_number + AND e.account_address = wth.address + AND e.event_resource = 'WithdrawEvent' +WHERE + COALESCE( + dep.token_address, + wth.token_address + ) IS NOT NULL diff --git a/models/silver/core/silver__transfers_fungible.sql b/models/silver/core/silver__transfers_fungible.sql new file mode 100644 index 0000000..d9d484d --- /dev/null +++ b/models/silver/core/silver__transfers_fungible.sql @@ -0,0 +1,173 @@ +{{ config( + materialized = 'incremental', + unique_key = ['tx_hash','event_index','block_timestamp::DATE'], + incremental_strategy = 'merge', + merge_exclude_columns = ["inserted_timestamp"], + cluster_by = ['block_timestamp::DATE','modified_timestamp::DATE'], + tags = ['core'] +) }} + +-- depends_on: {{ ref('core__fact_events') }} +-- depends_on: {{ ref('silver__fungiblestore_owners') }} +-- depends_on: {{ ref('silver__fungiblestore_metadata') }} + +{% if execute %} + {% set base_query %} + CREATE + OR REPLACE temporary TABLE silver.transfers_fungible__intermediate_tmp AS + + SELECT + block_number, + version, + success, + block_timestamp, + tx_hash, + event_index, + event_resource, + event_data :amount :: bigint AS amount, + event_data :store :: STRING AS store_address + FROM + {{ ref('core__fact_events') }} + WHERE + event_module = 'fungible_asset' + AND event_resource IN ( + 'WithdrawEvent', + 'DepositEvent', + 'Withdraw', + 'Deposit' + ) + +{% if is_incremental() %} +AND modified_timestamp >= ( + SELECT + MAX(modified_timestamp) + FROM + {{ this }} +) +{% endif %} + +{% endset %} +{% do run_query(base_query) %} +{% set owner_query %} +CREATE +OR REPLACE temporary TABLE silver.transfers_fungible_store_owners__intermediate_tmp AS +SELECT + version, + block_timestamp, + store_address, + owner_address, + COUNT( + DISTINCT owner_address + ) over( + PARTITION BY store_address + ) owner_cnt +FROM + {{ ref('silver__fungiblestore_owners') }} +WHERE + store_address IN ( + SELECT + store_address + FROM + silver.transfers_fungible__intermediate_tmp + ) +ORDER BY + block_timestamp {% endset %} + {% do run_query(owner_query) %} + {% set owner_query_single %} + CREATE + OR REPLACE temporary TABLE silver.transfers_fungible_store_owners_single__intermediate_tmp AS +SELECT + DISTINCT store_address, + owner_address +FROM + silver.transfers_fungible_store_owners__intermediate_tmp +WHERE + owner_cnt = 1 {% endset %} + {% do run_query(owner_query_single) %} + {% set owner_query_many %} + CREATE + OR REPLACE temporary TABLE silver.transfers_fungible_store_owners_many__intermediate_tmp AS WITH base AS ( + SELECT + store_address, + owner_address, + block_timestamp, + conditional_change_event(owner_address) over ( + PARTITION BY store_address + ORDER BY + block_timestamp + ) AS change_event, + ROW_NUMBER() over ( + PARTITION BY store_address + ORDER BY + block_timestamp + ) AS rn + FROM + silver.transfers_fungible_store_owners__intermediate_tmp + WHERE + owner_cnt > 1 + ) +SELECT + store_address, + owner_address, + block_timestamp +FROM + base qualify ROW_NUMBER() over( + PARTITION BY store_address, + change_event + ORDER BY + rn + ) = 1; +{% endset %} + {% do run_query(owner_query_many) %} +{% endif %} + +WITH md AS ( + SELECT + store_address, + metadata_address + FROM + {{ ref('silver__fungiblestore_metadata') }} +) +SELECT + e.block_number, + e.block_timestamp, + e.tx_hash, + e.version, + e.success, + e.event_index, + CASE + WHEN event_resource IN ( + 'WithdrawEvent', + 'Withdraw' + ) THEN 'WithdrawEvent' + WHEN event_resource IN ( + 'DepositEvent', + 'Deposit' + ) THEN 'DepositEvent' + END AS transfer_event, + e.store_address, + COALESCE( + os.owner_address, + om.owner_address + ) AS owner_address, + m.metadata_address, + e.amount, + os.owner_address os, + om.owner_address om, + om.block_timestamp om_block_timestamp, + {{ dbt_utils.generate_surrogate_key( + ['e.tx_hash','e.event_index'] + ) }} AS transfers_fungible_id, + SYSDATE() AS inserted_timestamp, + SYSDATE() AS modified_timestamp, + '{{ invocation_id }}' AS _invocation_id +FROM + silver.transfers_fungible__intermediate_tmp e asof + JOIN silver.transfers_fungible_store_owners_many__intermediate_tmp om match_condition( + e.block_timestamp >= om.block_timestamp + ) + ON e.store_address = om.store_address + LEFT JOIN silver.transfers_fungible_store_owners_single__intermediate_tmp os + ON e.store_address = os.store_address + LEFT JOIN md m + ON e.store_address = m.store_address diff --git a/models/silver/core/silver__transfers_native.sql b/models/silver/core/silver__transfers_native.sql new file mode 100644 index 0000000..bbdc5af --- /dev/null +++ b/models/silver/core/silver__transfers_native.sql @@ -0,0 +1,111 @@ +{{ config( + materialized = 'incremental', + unique_key = ['tx_hash','_transfer_key','block_timestamp::DATE'], + incremental_strategy = 'merge', + merge_exclude_columns = ["inserted_timestamp"], + cluster_by = ['block_timestamp::DATE'], + post_hook = "ALTER TABLE {{ this }} ADD SEARCH OPTIMIZATION ON EQUALITY(tx_hash, version, from_address, to_address);", + tags = ['core'], + enabled = false +) }} + +-- depends_on: {{ ref('silver__transfers') }} +-- depends_on: {{ ref('core__fact_events') }} + +WITH xfer AS ( + + SELECT + block_number, + block_timestamp, + tx_hash, + version, + success, + event_index, + transfer_event, + account_address, + amount, + token_address + FROM + {{ ref('silver__transfers') }} + WHERE + amount > 0 + AND token_address = '0x1::aptos_coin::AptosCoin' + +{% if is_incremental() %} +AND modified_timestamp >= ( + SELECT + MAX(modified_timestamp) + FROM + {{ this }} +) +{% endif %} +), +wth AS ( + SELECT + * + FROM + xfer + WHERE + transfer_event = 'WithdrawEvent' +), +dep AS ( + SELECT + * + FROM + xfer + WHERE + transfer_event = 'DepositEvent' +), +reg AS ( + SELECT + block_number, + block_timestamp, + tx_hash, + version, + success, + event_index + FROM + {{ ref('core__fact_events') }} + WHERE + event_type = '0x1::account::CoinRegisterEvent' + +{% if is_incremental() %} +AND modified_timestamp >= ( + SELECT + MAX(modified_timestamp) + FROM + {{ this }} +) +{% endif %} +) +SELECT + wth.block_number, + wth.block_timestamp, + wth.tx_hash, + wth.version, + wth.success, + wth.account_address AS from_address, + dep.account_address AS to_address, + wth.amount, + wth.token_address, + wth.event_index || ':' || dep.event_index AS _transfer_key, + {{ dbt_utils.generate_surrogate_key( + ['wth.tx_hash','wth.event_index','dep.event_index'] + ) }} AS transfers_native_id, + SYSDATE() AS inserted_timestamp, + SYSDATE() AS modified_timestamp, + '{{ invocation_id }}' AS _invocation_id +FROM + wth + LEFT JOIN reg + ON wth.tx_hash = reg.tx_hash + AND wth.event_index + 1 = reg.event_index + JOIN dep + ON wth.tx_hash = dep.tx_hash + AND wth.amount = dep.amount +WHERE + wth.account_address <> dep.account_address + AND ( + wth.event_index + 2 = dep.event_index + OR (reg.tx_hash IS NOT NULL AND reg.event_index + 1 = dep.event_index) + ) \ No newline at end of file diff --git a/models/silver/core/silver_core.yml b/models/silver/core/silver_core.yml index f897e0e..593f547 100644 --- a/models/silver/core/silver_core.yml +++ b/models/silver/core/silver_core.yml @@ -230,6 +230,62 @@ models: - name: _invocation_id data_type: VARCHAR + - name: silver__fungiblestore_metadata + config: + contract: + enforced: true + columns: + - name: block_timestamp_first + data_type: TIMESTAMP_NTZ + - name: block_number_first + data_type: NUMBER + - name: store_address + data_type: VARCHAR + - name: metadata_address + data_type: VARCHAR + tests: + - not_null: + tags: ['test_quality'] + - name: fungiblestore_metadata_id + data_type: VARCHAR + - name: inserted_timestamp + data_type: TIMESTAMP_NTZ + - name: modified_timestamp + data_type: TIMESTAMP_NTZ + - name: _invocation_id + data_type: VARCHAR + + - name: silver__fungiblestore_owners + config: + contract: + enforced: true + columns: + - name: block_timestamp + data_type: TIMESTAMP_NTZ + - name: block_number + data_type: NUMBER + - name: version + data_type: NUMBER + - name: tx_hash + data_type: VARCHAR + - name: change_index + data_type: NUMBER + - name: store_address + data_type: VARCHAR + - name: owner_address + data_type: VARCHAR + tests: + - not_null: + tags: ['test_quality'] + - name: fungiblestore_owners_id + data_type: VARCHAR + - name: inserted_timestamp + data_type: TIMESTAMP_NTZ + - name: modified_timestamp + data_type: TIMESTAMP_NTZ + - name: _invocation_id + data_type: VARCHAR + - name: silver__transactions config: contract: @@ -330,5 +386,138 @@ models: data_type: TIMESTAMP_NTZ - name: modified_timestamp data_type: TIMESTAMP_NTZ + - name: _invocation_id + data_type: VARCHAR + + - name: silver__transfers + config: + contract: + enforced: true + columns: + - name: block_number + data_type: NUMBER + - name: block_timestamp + data_type: TIMESTAMP_NTZ + - name: tx_hash + data_type: VARCHAR + - name: version + data_type: NUMBER + - name: success + data_type: BOOLEAN + - name: event_index + data_type: NUMBER + tests: + - not_null: + tags: ['test_quality'] + - name: creation_number + data_type: NUMBER + tests: + - not_null: + tags: ['test_quality'] + - name: transfer_event + data_type: VARCHAR + tests: + - not_null: + tags: ['test_quality'] + - name: account_address + data_type: VARCHAR + tests: + - not_null: + tags: ['test_quality'] + - name: amount + data_type: NUMBER + tests: + - not_null: + tags: ['test_quality'] + - dbt_utils.expression_is_true: + expression: ">= 0" + tags: ['test_quality'] + - name: token_address + data_type: VARCHAR + tests: + - not_null: + tags: ['test_quality'] + - name: transfers_id + data_type: VARCHAR + - name: inserted_timestamp + data_type: TIMESTAMP_NTZ + - name: modified_timestamp + data_type: TIMESTAMP_NTZ + - name: _invocation_id + data_type: VARCHAR + + - name: silver__transfers_fungible + config: + contract: + enforced: true + columns: + - name: block_number + data_type: NUMBER + - name: block_timestamp + data_type: TIMESTAMP_NTZ + - name: tx_hash + data_type: VARCHAR + - name: version + data_type: NUMBER + - name: success + data_type: BOOLEAN + - name: event_index + data_type: NUMBER + - name: transfer_event + data_type: VARCHAR + - name: store_address + data_type: VARCHAR + - name: owner_address + data_type: VARCHAR + - name: metadata_address + data_type: VARCHAR + - name: amount + data_type: NUMBER + - name: os + data_type: VARCHAR + - name: om + data_type: VARCHAR + - name: om_block_timestamp + data_type: TIMESTAMP_NTZ + - name: transfers_fungible_id + data_type: VARCHAR + - name: inserted_timestamp + data_type: TIMESTAMP_NTZ + - name: modified_timestamp + data_type: TIMESTAMP_NTZ + - name: _invocation_id + data_type: VARCHAR + + - name: silver__transfers_native + config: + contract: + enforced: true + columns: + - name: block_number + data_type: NUMBER + - name: block_timestamp + data_type: TIMESTAMP_NTZ + - name: tx_hash + data_type: VARCHAR + - name: version + data_type: NUMBER + - name: success + data_type: BOOLEAN + - name: from_address + data_type: VARCHAR + - name: to_address + data_type: VARCHAR + - name: amount + data_type: NUMBER + - name: token_address + data_type: VARCHAR + - name: _transfer_key + data_type: VARCHAR + - name: transfers_native_id + data_type: VARCHAR + - name: inserted_timestamp + data_type: TIMESTAMP_NTZ + - name: modified_timestamp + data_type: TIMESTAMP_NTZ - name: _invocation_id data_type: VARCHAR \ No newline at end of file