added transfers models

2026-02-06 11:06:56 +00:00 · 2025-07-29 12:55:57 -06:00 · 2025-07-29 12:55:57 -06:00 · a4fe0a74c0
commit a4fe0a74c0
parent 3e7da19f47
7 changed files with 364 additions and 54 deletions
--- a/models/descriptions/columns.md
+++ b/models/descriptions/columns.md
@ -94,6 +94,18 @@ Type/category of object state modification or event. Enum: created, modified, de
 Sui address (32-byte hex) representing the transaction or event sender. Used for authorization, security analysis, and user activity tracking. Example: '0xabc123...'.
 {% enddocs %}

+{% docs receiver %}
+Sui address (32-byte hex) representing the transaction or event receiver. Used for tracking destination addresses, transfer flows, and recipient analytics. In transfer contexts, this is the address receiving tokens or assets. Example: '0xdef456...'.
+{% enddocs %}
+
+{% docs ez_transfers_id %}
+Surrogate key for the enhanced transfers table. Generated unique identifier by combining transaction digest and balance change index, ensuring each transfer event enriched with token metadata is uniquely addressable. Used as the primary key for user-friendly transfer analytics, dashboard queries, and cross-model joins. In Sui, this supports transfer analysis with normalized amounts and token symbols, enabling easy identification and comparison of token movements.
+{% enddocs %}
+
+{% docs fact_transfers_id %}
+Surrogate key for the core fact transfers table. Generated unique identifier by combining transaction digest and balance change index, ensuring each transfer event is uniquely addressable. Used as the primary key for transfer tracking, analytics workflows, and cross-model joins. In Sui, this supports precise transfer analysis, portfolio tracking, and compliance reporting by enabling unique identification of each token movement between addresses.
+{% enddocs %}
+
 {% docs digest %}
 32-byte cryptographic hash (hex) of object contents, using SHA-256. Used for content verification, integrity checking, and unauthorized modification detection. Example: 'a1b2c3...'.
 {% enddocs %}
@ -284,4 +296,8 @@ Variant data structure indicating this object has shared ownership, meaning it's

 {% docs modules %}
 Comma-separated list of Move module names contained within the package. Modules define the package's functionality and can be called by transactions to execute smart contract logic. Each module has a unique name within its package and contains functions, structs, and resources. Used for analyzing package composition, tracking module usage patterns, and understanding smart contract functionality. Example: 'coin,transfer,governance'.
+{% enddocs %}
+
+{% docs amount_normalized %}
+Decimal-adjusted token amount calculated by dividing the raw amount by 10^decimals. Provides human-readable token quantities that can be directly compared across different token types. Essential for financial analysis, balance calculations, and user-facing applications where raw blockchain amounts need to be converted to meaningful values. Example: if amount is 1000000000 and decimals is 9, amount_normalized would be 1.0.
 {% enddocs %}
--- a/models/descriptions/tables.md
+++ b/models/descriptions/tables.md
@ -2,6 +2,14 @@
 Dimension table providing authoritative metadata for all fungible and non-fungible tokens on the Sui blockchain. Includes decimals, symbols, names, descriptions, and icon URLs sourced from on-chain metadata and Move package definitions. Serves as the canonical reference for token identification, decimal normalization, and UI display across analytics workflows. Data is sourced from bronze_api__coin_metadata and cross-referenced with on-chain Move modules, covering both native SUI and custom tokens. Essential for accurate balance calculations, token flow analysis, and user-facing applications. Supports lineage tracing from raw on-chain metadata to analytics-ready token attributes.
 {% enddocs %}

+{% docs core__fact_transfers %}
+Fact table capturing all token and coin transfers on the Sui blockchain at the finest granularity. Each row represents a single transfer event between a sender and receiver, including the raw amount, coin type, and transaction context. Serves as the foundational transfer table that filters out null amounts from the silver layer and provides clean transfer data for downstream analytics. Includes checkpoint metadata, transaction success status, and balance change indexing to support precise tracking of asset movements across the network. Enables reconstruction of transfer flows, portfolio analysis, and compliance reporting by providing complete visibility into all token movements between addresses. Data is derived from transaction execution effects and object state transitions, following Sui's explicit ownership model where transfers represent actual balance changes between distinct owners.
+{% enddocs %}
+
+{% docs core__ez_transfers %}
+Enhanced fact table providing user-friendly transfer analytics by joining core transfer data with token metadata. Each row represents a single transfer event enriched with normalized amounts (decimal-adjusted), token symbols, and human-readable identifiers. Serves as the primary table for transfer analysis, portfolio tracking, and user-facing applications by converting raw blockchain amounts into meaningful values. The amount_normalized field automatically applies decimal precision adjustments based on token metadata, enabling direct comparison across different token types and simplifying balance calculations. Includes all transfer metadata from the base fact table while adding token symbols for easy identification and normalized amounts for accurate financial analysis. Essential for dashboards, reporting tools, and analytics workflows that require human-readable token information and precise decimal calculations.
+{% enddocs %}
+
 {% docs core__fact_balance_changes %}
 Fact table recording every token and coin balance change event on the Sui blockchain at the finest granularity. Each row represents a single balance delta (positive or negative) for a specific owner, coin type, and transaction, capturing the full flow of assets across wallets and contracts. Includes object IDs, transaction context, and ownership metadata, supporting precise tracking of token movements, portfolio changes, and treasury operations. Enables reconstruction of wallet balances, detection of large transfers, and analysis of token velocity. Data is derived from transaction execution effects and object state transitions, following Sui's explicit ownership and versioning model.
 {% enddocs %}
--- a/models/gold/core/core__ez_transfers.sql
+++ b/models/gold/core/core__ez_transfers.sql
@ -0,0 +1,41 @@
+{{ config(
+    materialized = 'incremental',
+    unique_key = ['tx_digest','balance_change_index'],
+    incremental_strategy = 'merge',
+    incremental_predicates = ["dynamic_range_predicate", "block_timestamp::DATE"],
+    merge_exclude_columns = ["inserted_timestamp"],
+    cluster_by = ['block_timestamp::DATE','modified_timestamp::DATE'],
+    post_hook = "ALTER TABLE {{ this }} ADD SEARCH OPTIMIZATION ON EQUALITY(tx_digest, sender, receiver, coin_type, symbol);",
+    tags = ['core']
+) }}
+
+SELECT
+    checkpoint_number,
+    block_timestamp,
+    tx_digest,
+    balance_change_index,
+    tx_succeeded,
+    sender,
+    receiver,
+    ft.coin_type,
+    symbol,
+    amount,
+    ROUND(NULLIFZERO(DIV0NULL(amount, POWER(10, dt.decimals))), 3) as amount_normalized,
+    {{ dbt_utils.generate_surrogate_key(
+        ['tx_digest','balance_change_index']
+    ) }} AS ez_transfers_id,
+    SYSDATE() AS inserted_timestamp,
+    SYSDATE() AS modified_timestamp
+FROM
+    {{ ref('core__fact_transfers') }} ft
+LEFT JOIN
+    {{ ref('core__dim_tokens') }} dt 
+    ON ft.coin_type = dt.coin_type
+{% if is_incremental() %}
+WHERE ft.modified_timestamp >= (
+    SELECT
+        MAX(modified_timestamp)
+    FROM
+        {{ this }}
+)
+{% endif %}
--- a/models/gold/core/core__fact_transfers.sql
+++ b/models/gold/core/core__fact_transfers.sql
@ -0,0 +1,41 @@
+{{ config(
+    materialized = 'incremental',
+    unique_key = ['tx_digest','balance_change_index'],
+    incremental_strategy = 'merge',
+    incremental_predicates = ["dynamic_range_predicate", "block_timestamp::DATE"],
+    merge_exclude_columns = ["inserted_timestamp"],
+    cluster_by = ['block_timestamp::DATE','modified_timestamp::DATE'],
+    post_hook = "ALTER TABLE {{ this }} ADD SEARCH OPTIMIZATION ON EQUALITY(tx_digest, sender, receiver, coin_type);",
+    tags = ['core']
+) }}
+
+SELECT
+    checkpoint_number,
+    block_timestamp,
+    tx_digest,
+    tx_succeeded,
+    sender,
+    receiver,
+    balance_change_index,
+    coin_type,
+    amount,
+    {{ dbt_utils.generate_surrogate_key(
+        ['tx_digest','balance_change_index']
+    ) }} AS fact_transfers_id,
+    SYSDATE() AS inserted_timestamp,
+    SYSDATE() AS modified_timestamp
+FROM
+    {{ ref(
+        'silver__transfers'
+    ) }}
+WHERE
+    amount is not null
+
+{% if is_incremental() %}
+AND modified_timestamp >= (
+    SELECT
+        MAX(modified_timestamp)
+    FROM
+        {{ this }}
+)
+{% endif %}
--- a/models/gold/core/gold_core.yml
+++ b/models/gold/core/gold_core.yml
@ -661,7 +661,6 @@ models:
          severity: error
          tags: ['test_recency']

-
  - name: core__dim_labels
    description: "The labels table is a store of one-to-one address identifiers, or an address name. Labels are broken out into a \"type\" (such as cex, dex, dapp, games, etc.) and a \"subtype\" (ex: contract_deployer, hot_wallet, token_contract, etc.) in order to help classify each address name into similar groups. Our labels are sourced from many different places, but can primarily be grouped into two categories: automatic and manual. Automatic labels are continuously labeled based on certain criteria, such as a known contract deploying another contract, behavior based algorithms for finding deposit wallets, and consistent data pulls of custom protocol APIs. Manual labels are done periodically to find addresses that cannot be found programmatically such as finding new protocol addresses, centralized exchange hot wallets, or trending addresses. Labels can also be added by our community by using our add-a-label tool (https://science.flipsidecrypto.xyz/add-a-label/).A label can be removed by our labels team if it is found to be incorrect or no longer relevant; this generally will only happen for mislabeled deposit wallets."

@ -689,4 +688,160 @@ models:
      - name: INSERTED_TIMESTAMP
        description: "Timestamp when this record was inserted."
      - name: MODIFIED_TIMESTAMP
-        description: "Timestamp when this record was last modified."
+        description: "Timestamp when this record was last modified."
+
+  - name: core__fact_transfers
+    description: "{{ doc('core__fact_transfers') }}"
+    config:
+      contract:
+        enforced: true
+    columns:
+      - name: CHECKPOINT_NUMBER
+        description: "{{ doc('checkpoint_number') }}"
+        data_type: NUMBER
+        tests:
+          - not_null
+      - name: BLOCK_TIMESTAMP
+        description: "{{ doc('block_timestamp') }}"
+        data_type: TIMESTAMP_NTZ
+        tests:
+          - not_null
+      - name: TX_DIGEST
+        description: "{{ doc('tx_digest') }}"
+        data_type: VARCHAR
+        tests:
+          - not_null
+      - name: TX_SUCCEEDED
+        description: "{{ doc('tx_succeeded') }}"
+        data_type: BOOLEAN
+        tests:
+          - not_null
+      - name: SENDER
+        description: "{{ doc('sender') }}"
+        data_type: VARCHAR
+        tests:
+          - not_null
+      - name: RECEIVER
+        description: "{{ doc('receiver') }}"
+        data_type: VARCHAR
+        tests:
+          - not_null
+      - name: BALANCE_CHANGE_INDEX
+        description: "{{ doc('balance_change_index') }}"
+        data_type: NUMBER
+        tests:
+          - not_null
+      - name: COIN_TYPE
+        description: "{{ doc('coin_type') }}"
+        data_type: VARCHAR
+        tests:
+          - not_null
+      - name: AMOUNT
+        description: "{{ doc('amount') }}"
+        data_type: NUMBER
+        tests:
+          - not_null
+      - name: FACT_TRANSFERS_ID
+        description: "{{ doc('fact_transfers_id') }}"
+        data_type: VARCHAR
+        tests:
+          - not_null
+          - unique
+      - name: INSERTED_TIMESTAMP
+        description: "{{ doc('inserted_timestamp') }}"
+        data_type: TIMESTAMP_NTZ
+        tests:
+          - not_null
+      - name: MODIFIED_TIMESTAMP
+        description: "{{ doc('modified_timestamp') }}"
+        data_type: TIMESTAMP_NTZ
+        tests:
+          - not_null
+    tests:
+      - dbt_utils.recency:
+          datepart: hour
+          field: block_timestamp
+          interval: 12
+          severity: error
+          tags: ['test_recency']
+
+  - name: core__ez_transfers
+    description: "{{ doc('core__ez_transfers') }}"
+    config:
+      contract:
+        enforced: true
+    columns:
+      - name: CHECKPOINT_NUMBER
+        description: "{{ doc('checkpoint_number') }}"
+        data_type: NUMBER
+        tests:
+          - not_null
+      - name: BLOCK_TIMESTAMP
+        description: "{{ doc('block_timestamp') }}"
+        data_type: TIMESTAMP_NTZ
+        tests:
+          - not_null
+      - name: TX_DIGEST
+        description: "{{ doc('tx_digest') }}"
+        data_type: VARCHAR
+        tests:
+          - not_null
+      - name: BALANCE_CHANGE_INDEX
+        description: "{{ doc('balance_change_index') }}"
+        data_type: NUMBER
+        tests:
+          - not_null
+      - name: TX_SUCCEEDED
+        description: "{{ doc('tx_succeeded') }}"
+        data_type: BOOLEAN
+        tests:
+          - not_null
+      - name: SENDER
+        description: "{{ doc('sender') }}"
+        data_type: VARCHAR
+        tests:
+          - not_null
+      - name: RECEIVER
+        description: "{{ doc('receiver') }}"
+        data_type: VARCHAR
+        tests:
+          - not_null
+      - name: COIN_TYPE
+        description: "{{ doc('coin_type') }}"
+        data_type: VARCHAR
+        tests:
+          - not_null
+      - name: SYMBOL
+        description: "{{ doc('symbol') }}"
+        data_type: VARCHAR
+      - name: AMOUNT
+        description: "{{ doc('amount') }}"
+        data_type: NUMBER
+        tests:
+          - not_null
+      - name: AMOUNT_NORMALIZED
+        description: "{{ doc('amount_normalized') }}"
+        data_type: FLOAT
+      - name: EZ_TRANSFERS_ID
+        description: "{{ doc('ez_transfers_id') }}"
+        data_type: VARCHAR
+        tests:
+          - not_null
+          - unique
+      - name: INSERTED_TIMESTAMP
+        description: "{{ doc('inserted_timestamp') }}"
+        data_type: TIMESTAMP_NTZ
+        tests:
+          - not_null
+      - name: MODIFIED_TIMESTAMP
+        description: "{{ doc('modified_timestamp') }}"
+        data_type: TIMESTAMP_NTZ
+        tests:
+          - not_null
+    tests:
+      - dbt_utils.recency:
+          datepart: hour
+          field: block_timestamp
+          interval: 12
+          severity: error
+          tags: ['test_recency']
--- a/models/silver/core/silver__transfers.sql
+++ b/models/silver/core/silver__transfers.sql
@ -14,63 +14,50 @@ allowed_tx AS (
        tx_digest
    FROM 
        {{ ref('core__fact_transactions') }}
-    {% if is_incremental() %}
-    WHERE 
-        modified_timestamp >= (SELECT COALESCE(MAX(modified_timestamp),'1970-01-01') FROM {{ this }})
-    {% endif %}
-    GROUP BY 
-        tx_digest
-    HAVING SUM(
-        CASE
-            WHEN payload_type IN ('TransferObjects','SplitCoins','MergeCoins') THEN 0
-            WHEN payload_type = 'MoveCall' AND payload_details :package
-                = '0x0000000000000000000000000000000000000000000000000000000000000002'
-            THEN 0
-            ELSE 1
-        END
-    ) = 0
-),
-
-coin_only_tx AS (
-    SELECT
-        tx_digest
-    FROM 
-        {{ ref('core__fact_changes') }}
-    WHERE 
-        tx_digest IN (SELECT tx_digest FROM allowed_tx)
+    WHERE
+        (payload_type IN ('TransferObjects','SplitCoins','MergeCoins'))
+        OR 
+        (payload_type = 'MoveCall' AND payload_details :package = '0x0000000000000000000000000000000000000000000000000000000000000002')
    {% if is_incremental() %}
        AND modified_timestamp >= (SELECT COALESCE(MAX(modified_timestamp),'1970-01-01') FROM {{ this }})
    {% endif %}
-    GROUP BY 
-        tx_digest
-    HAVING MAX(
-        CASE WHEN object_type ILIKE '0x2::coin::Coin%' THEN 0 ELSE 1 END
-    ) = 0
+),
+filtered as (
+    SELECT 
+        fbc.checkpoint_number,
+        fbc.block_timestamp,
+        fbc.tx_digest,
+        fbc.tx_succeeded,
+        fbc.tx_sender as sender,
+        fbc.owner as receiver,
+        fbc.balance_change_index,
+        fbc.coin_type,
+        fbc.amount
+    FROM 
+        {{ ref('core__fact_balance_changes') }} fbc
+    JOIN
+        allowed_tx at 
+        ON fbc.tx_digest = at.tx_digest
+    WHERE 
+        fbc.tx_sender != fbc.owner
+        AND NOT (balance_change_index = 0 AND amount < 0) -- remove mints, self-splits, proofs, flash loans
+    {% if is_incremental() %}
+        AND fbc.modified_timestamp >= (SELECT COALESCE(MAX(modified_timestamp),'1970-01-01') FROM {{ this }})
+    {% endif %}
 )
-SELECT 
-    fbc.checkpoint_number,
-    fbc.block_timestamp,
-    fbc.tx_digest,
-    fbc.tx_succeeded,
-    fbc.tx_sender as sender,
-    fbc.owner as receiver,
-    fbc.balance_change_index,
-    dt.symbol,
-    dt.decimals,
-    fbc.amount as amount_raw,
-    fbc.amount / POWER(10, dt.decimals) as amount_normalized,
+SELECT DISTINCT
+    checkpoint_number,
+    block_timestamp,
+    tx_digest,
+    tx_succeeded,
+    sender,
+    receiver,
+    balance_change_index,
+    coin_type,
+    amount,
    {{ dbt_utils.generate_surrogate_key(['tx_digest','balance_change_index']) }} AS transfers_id,
    SYSDATE() AS inserted_timestamp,
    SYSDATE() AS modified_timestamp,
    '{{ invocation_id }}' AS _invocation_id
-FROM 
-    {{ ref('core__fact_balance_changes') }} fbc
-JOIN 
-    {{ ref('core__dim_tokens') }} dt ON fbc.coin_type = dt.coin_type
-WHERE 
-    fbc.tx_digest IN (SELECT tx_digest FROM coin_only_tx)
-    AND fbc.tx_sender != fbc.owner
-    {% if is_incremental() %}
-        AND fbc.modified_timestamp >= (SELECT COALESCE(MAX(modified_timestamp),'1970-01-01') FROM {{ this }})
-        AND dt.modified_timestamp >= (SELECT COALESCE(MAX(modified_timestamp),'1970-01-01') FROM {{ this }})
-    {% endif %}
+FROM
+    filtered
--- a/models/silver/core/silver_core.yml
+++ b/models/silver/core/silver_core.yml
@ -65,3 +65,65 @@ models:
        data_type: TIMESTAMP_NTZ
      - name: _invocation_id
        data_type: VARCHAR
+
+  - name: silver__transfers
+    config:
+      contract:
+        enforced: true
+    columns:
+      - name: CHECKPOINT_NUMBER
+        description: "{{ doc('checkpoint_number') }}"
+        data_type: NUMBER
+        tests:
+          - not_null
+      - name: BLOCK_TIMESTAMP
+        description: "{{ doc('block_timestamp') }}"
+        data_type: TIMESTAMP_NTZ
+        tests:
+          - not_null
+      - name: TX_DIGEST
+        description: "{{ doc('tx_digest') }}"
+        data_type: VARCHAR
+        tests:
+          - not_null
+      - name: TX_SUCCEEDED
+        description: "{{ doc('tx_succeeded') }}"
+        data_type: BOOLEAN
+        tests:
+          - not_null
+      - name: SENDER
+        description: "{{ doc('sender') }}"
+        data_type: VARCHAR
+        tests:
+          - not_null
+      - name: RECEIVER
+        description: "{{ doc('receiver') }}"
+        data_type: VARCHAR
+        tests:
+          - not_null
+      - name: BALANCE_CHANGE_INDEX
+        description: "{{ doc('balance_change_index') }}"
+        data_type: NUMBER
+        tests:
+          - not_null
+      - name: COIN_TYPE
+        description: "{{ doc('coin_type') }}"
+        data_type: VARCHAR
+        tests:
+          - not_null
+      - name: AMOUNT
+        description: "{{ doc('amount') }}"
+        data_type: NUMBER
+        tests:
+          - not_null
+      - name: TRANSFERS_ID
+        data_type: VARCHAR
+      - name: INSERTED_TIMESTAMP
+        description: "{{ doc('inserted_timestamp') }}"
+        data_type: TIMESTAMP_NTZ
+      - name: MODIFIED_TIMESTAMP
+        description: "{{ doc('modified_timestamp') }}"
+        data_type: TIMESTAMP_NTZ
+      - name: _INVOCATION_ID
+        data_type: VARCHAR
+