Merge branch 'main' into AN-5791-observe

models/bronze/core/bronze__trust_lines.sql

@@ -0,0 +1,10 @@
+{{ config (
+    materialized = 'view'
+) }}
+{{ streamline_external_table_query_v2(
+    model = "trust_lines",
+    partition_function = "TRY_TO_DATE(left(split_part(split_part(file_name, '=', -1), '/', -1),8), 'YYYYMMDD')",
+    partition_name = "partition_gte_id",
+    unique_key = "metadata",
+    other_cols = "partition_id"
+) }}

models/bronze/core/bronze__trust_lines_FR.sql

@@ -0,0 +1,10 @@
+{{ config (
+    materialized = 'view'
+) }}
+{{ streamline_external_table_FR_query_v2(
+    model = "trust_lines",
+    partition_function = "TRY_TO_DATE(left(split_part(split_part(file_name, '=', -1), '/', -1),8), 'YYYYMMDD')",
+    partition_name = "partition_gte_id",
+    unique_key = "metadata",
+    other_cols = "partition_id"
+) }}

models/descriptions/__overview__.md

@@ -28,6 +28,7 @@ There is more information on how to use dbt docs in the last section of this doc
 - [core.fact_ledgers](https://flipsidecrypto.github.io/stellar-models/#!/model/model.stellar_models.core__fact_ledgers)
 - [core.fact_operations](https://flipsidecrypto.github.io/stellar-models/#!/model/model.stellar_models.core__fact_operations)
 - [core.fact_transactions](https://flipsidecrypto.github.io/stellar-models/#!/model/model.stellar_models.core__fact_transactions)
+- [core.fact_trust_lines](https://flipsidecrypto.github.io/stellar-models/#!/model/model.stellar_models.core__fact_trust_lines)
 
 **Convenience Views:**
 - [core.ez_operations](https://flipsidecrypto.github.io/stellar-models/#!/model/model.stellar_models.core__ez_operations)

models/descriptions/ledger_key.md

@@ -0,0 +1,3 @@
+{% docs ledger_key %}
+The unique ledger key when the trust line state last changed
+{% enddocs %}

models/descriptions/tables/dim_assets.md

@@ -0,0 +1,7 @@
+{% docs core__dim_assets %}
+
+Table reports which assets are used during the batch interval, which can help identify periods of time of large activity for an asset. The table does not have a primary key and assets are duplicated in the table as they are used during different periods of time. To get a distinct count of assets on the network, please refer to the history_assets table. Table not widely used.
+
+Learn more about Stellar assets: https://developers.stellar.org/docs/learn/fundamentals/stellar-data-structures/assets
+
+{% enddocs %}

models/descriptions/tables/ez_operations.md

@@ -0,0 +1,7 @@
+{% docs core__ez_operations %}
+
+An aggregated convenience table that joins `core.fact_operations`, `core.fact_transactions`, and `core.fact_ledgers`.
+
+Learn more about Stellar data structures: https://developers.stellar.org/docs/learn/fundamentals/stellar-data-structures
+
+{% enddocs %}

models/descriptions/tables/fact_accounts.md

@@ -0,0 +1,7 @@
+{% docs core__fact_accounts %}
+
+The accounts table stores detailed information for a given account, including current account status, preconditions for transaction authorization, security settings and account balance. The balance reported in the accounts table reflects the account’s XLM balance only. All other asset balances are reported in the 'trust_lines' table. Any changes to the account, whether it is an account settings change, balance increase/decrease or sponsorship change will result in an increase to the account sequence_number and last_modified_ledger. The sequence_number is incremented with each operation applied to an account so that order is preserved during account mutation.
+
+Learn more about Stellar accounts: https://developers.stellar.org/docs/learn/fundamentals/stellar-data-structures/accounts
+
+{% enddocs %}

models/descriptions/tables/fact_contract_events.md

@@ -0,0 +1,7 @@
+{% docs core__fact_contract_events %}
+
+This table contains all contract events, diagnostic events, and system events. Events cover a wide variety of information including token value movement, core metrics, and logging/debugging information.
+
+Learn more about Stellar smart contracts: https://developers.stellar.org/docs/learn/fundamentals/stellar-data-structures/contracts
+
+{% enddocs %}

models/descriptions/tables/fact_ledgers.md

@@ -0,0 +1,7 @@
+{% docs core__fact_ledgers %}
+
+Each ledger stores the state of the network at a given point in time and contains all changes applied to the network - transactions, operations, etc. This table summarizes the actions taken within a single ledger and is relevant in determining high level network conditions.
+
+Learn more about Stellar ledgers: https://developers.stellar.org/docs/learn/fundamentals/stellar-data-structures/ledgers
+
+{% enddocs %}

models/descriptions/tables/fact_liquidity_pools.md

@@ -0,0 +1,8 @@
+{% docs defi__fact_liquidity_pools %}
+
+Stellar rolled out Automated Market Makers to its network in Nov 2021, which improves liquidity between asset pairs while incentivizing users to stake money in pools. Liquidity pools provide a simple, non-interactive way to trade large amounts of capital and enable high volumes of trading. Liquidity pools can be created between asset pairs and algorithmically controls the supply of the assets to give the best exchange rate while not requiring an orderbook in order to execute the trade. For each trade executed through a liquidity pool, the users with staked liquidity in the pool earn fees, which are distributed automatically to their accounts. Users can deposit and withdraw money in the pools; trades only execute via path payment operation.
+
+Learn more about Stellar data concepts: https://developers.stellar.org/docs/learn/fundamentals/stellar-data-structures
+
+{% enddocs %}
+

models/descriptions/tables/fact_operations.md

@@ -0,0 +1,7 @@
+{% docs core__fact_operations %}
+
+This table contains the lowest granularity of data avaiable to the network. It contains all details regarding operations that were executed as part of a transaction set. Be careful - the table contains both failed and successful operations. The details record will return varying information about an operation according to the operation type.
+
+Learn more about Stellar operations: https://developers.stellar.org/docs/learn/fundamentals/transactions/operations-and-transactions
+
+{% enddocs %}

models/descriptions/tables/fact_trades.md

@@ -0,0 +1,8 @@
+{% docs defi__fact_trades %}
+
+This table reports trading activity that occurs in both the decentralized exchange and automated money markets. Trades fulfill one of four operations: manage buy offers, manage sell offers and path payments (strict send and receive). Trades can be executed either against the orderbook or a liquidity pool that contains both bid and ask asset pair. Trades can be either path payments or offers that are fully or partially fulfilled, which means that the trade can be split into segments. A full trade is compromised of all "order" numbers for a given history_operation_id.
+
+Learn more about Stellar data concepts: https://developers.stellar.org/docs/learn/fundamentals/stellar-data-structures
+
+{% enddocs %}
+

models/descriptions/tables/fact_transactions.md

@@ -0,0 +1,7 @@
+{% docs core__fact_transactions %}
+
+Transactions are commands that modify the ledger state and consist of one or more operations. In terms of taxonomy, a ledger contains a transaction set of multiple transactions, and a transaction contains an operation set of one to many operations. Transactions that are sent to the Stellar Network either succeed completely or fail completely. There is no partial transaction execution. The table is an event log of all transactions submitted and committed to a transaction set within a ledger.
+
+Learn more about Stellar transactions: https://developers.stellar.org/docs/learn/fundamentals/transactions/operations-and-transactions
+
+{% enddocs %}

models/descriptions/tables/fact_trust_lines.md

@@ -0,0 +1,7 @@
+{% docs core__fact_trust_lines %}
+
+Trustlines track authorized and deleted lines of trust between an account and assets. This table can be viewed as a subentry to an account because all trustlines must be associated with a single account. The trust line also tracks the balance of the asset held by the account and any buying or selling liabilities on the orderbook for a given account and asset. You do not have to hold a balance of an asset to trust the asset.
+
+Learn more about Stellar trust lines: https://developers.stellar.org/docs/data/analytics/hubble/data-catalog/data-dictionary/trustlines
+
+{% enddocs %}

models/descriptions/trust_line_limit.md

@@ -0,0 +1,3 @@
+{% docs trust_line_limit %}
+The maximum amount of this asset that this account is willing to accept. The limit is specified when opening a trust line
+{% enddocs %}

models/gold/core/core__dim_assets.yml

@@ -2,7 +2,7 @@ version: 2
 
 models:
   - name: core__dim_assets
-    description: Dimension table containing information about assets on the network.
+    description: "{{ doc('core__dim_assets') }}"
     tests:
       - dbt_utils.unique_combination_of_columns:
           combination_of_columns:

models/gold/core/core__fact_accounts.yml

@@ -2,7 +2,7 @@ version: 2
 
 models:
   - name: core__fact_accounts
-    description: Core fact table containing all Stellar account information and their current states
+    description: "{{ doc('core__fact_accounts') }}"
     tests:
       - dbt_utils.unique_combination_of_columns:
           combination_of_columns:

models/gold/core/core__fact_contract_events.yml

@@ -2,7 +2,7 @@ version: 2
 
 models:
   - name: core__fact_contract_events
-    description: Fact table containing contract event details from the Stellar network.
+    description: "{{ doc('core__fact_contract_events') }}"
     columns:
 
       - name: TRANSACTION_HASH

models/gold/core/core__fact_ledgers.yml

@@ -2,7 +2,7 @@ version: 2
 
 models:
   - name: core__fact_ledgers
-    description: Fact table containing information about ledgers on the Stellar network.
+    description: "{{ doc('core__fact_ledgers') }}"
     columns:
       - name: SEQUENCE
         description: '{{ doc("sequence") }}'

models/gold/core/core__fact_operations.yml

@@ -2,7 +2,7 @@ version: 2
 
 models:
   - name: core__fact_operations
-    description: A comprehensive view of all operations executed on the Stellar network, including transaction details, operation types, and their outcomes.
+    description: "{{ doc('core__fact_operations') }}"
     tests:
       - dbt_utils.unique_combination_of_columns:
           combination_of_columns:

models/gold/core/core__fact_transactions.yml

@@ -2,7 +2,7 @@ version: 2
 
 models:
   - name: core__fact_transactions
-    description: Fact table containing transaction details from the Stellar network.
+    description: "{{ doc('core__fact_transactions') }}"
     columns:
 
       - name: ID

models/gold/core/core__fact_trust_lines.sql

@@ -0,0 +1,49 @@
+-- depends_on: {{ ref('silver__transactions') }}
+{{ config(
+    materialized = 'incremental',
+    unique_key = ["fact_trust_lines_id"],
+    incremental_predicates = ["dynamic_range_predicate", "block_timestamp::date"],
+    merge_exclude_columns = ["inserted_timestamp"],
+    cluster_by = ['block_timestamp::DATE','closed_at::DATE'],
+    post_hook = "ALTER TABLE {{ this }} ADD SEARCH OPTIMIZATION ON EQUALITY(account_id,asset_id,asset_code);",
+    tags = ['scheduled_core']
+) }}
+
+SELECT
+    account_id,
+    closed_at,
+    closed_at AS block_timestamp,
+    asset_id,
+    asset_type,
+    asset_issuer,
+    asset_code,
+    liquidity_pool_id,
+    balance,
+    trust_line_limit,
+    buying_liabilities,
+    selling_liabilities,
+    flags,
+    last_modified_ledger,
+    ledger_entry_change,
+    deleted,
+    ledger_sequence,
+    ledger_key,
+    sponsor,
+    batch_id,
+    batch_run_date,
+    batch_insert_ts,
+    trust_lines_id AS fact_trust_lines_id,
+    SYSDATE() AS inserted_timestamp,
+    SYSDATE() AS modified_timestamp
+FROM
+    {{ ref('silver__trust_lines') }}
+
+{% if is_incremental() %}
+WHERE
+    modified_timestamp >= (
+        SELECT
+            MAX(modified_timestamp)
+        FROM
+            {{ this }}
+    )
+{% endif %}

models/gold/core/core__fact_trust_lines.yml

@@ -0,0 +1,64 @@
+version: 2
+
+models:
+  - name: core__fact_trust_lines
+    description: "{{ doc('core__fact_trust_lines') }}"
+    columns:
+
+
+      - name: account_id
+        description: "{{ doc('account_id') }}"
+      - name: closed_at
+        description: "{{ doc('closed_at') }}"
+      - name: block_timestamp
+        description: "{{ doc('block_timestamp') }}"
+      - name: asset_id
+        description: "{{ doc('asset_id') }}"
+      - name: asset_type
+        description: "{{ doc('asset_type') }}"
+      - name: asset_issuer
+        description: "{{ doc('asset_issuer') }}"
+      - name: asset_code
+        description: "{{ doc('asset_code') }}"
+      - name: liquidity_pool_id
+        description: "{{ doc('liquidity_pool_id') }}"
+      - name: balance
+        description: "{{ doc('balance') }}"
+      - name: trust_line_limit
+        description: "{{ doc('trust_line_limit') }}"
+      - name: buying_liabilities
+        description: "{{ doc('buying_liabilities') }}"
+      - name: selling_liabilities
+        description: "{{ doc('selling_liabilities') }}"
+      - name: flags
+        description: "{{ doc('flags') }}"
+      - name: last_modified_ledger
+        description: "{{ doc('last_modified_ledger') }}"
+      - name: ledger_entry_change
+        description: "{{ doc('ledger_entry_change') }}"
+      - name: deleted
+        description: "{{ doc('deleted') }}"
+      - name: ledger_sequence
+        description: "{{ doc('ledger_sequence') }}"
+      - name: ledger_key
+        description: "{{ doc('ledger_key') }}"
+      - name: sponsor
+        description: "{{ doc('sponsor') }}"
+      - name: batch_id
+        description: "{{ doc('batch_id') }}"
+      - name: batch_run_date
+        description: "{{ doc('batch_run_date') }}"
+      - name: batch_insert_ts
+        description: "{{ doc('batch_insert_ts') }}"
+      - name: fact_trust_lines_id
+        description: "{{ doc('pk') }}"
+        tests:
+          - not_null:
+              where: modified_timestamp > current_date - {{ var('test_days_threshold', 3) }}
+          - unique:
+              where: modified_timestamp > current_date - {{ var('test_days_threshold', 3) }}
+      - name: inserted_timestamp
+        description: "{{ doc('inserted_timestamp') }}"
+      - name: modified_timestamp
+        description: "{{ doc('modified_timestamp') }}"
+

models/gold/defi/defi__fact_liquidity_pools.yml

@@ -2,7 +2,7 @@ version: 2
 
 models:
   - name: defi__fact_liquidity_pools
-    description: Fact table containing liquidity pool data including pool details, asset pairs, and metrics.
+    description: "{{ doc('defi__fact_liquidity_pools') }}"
     
     columns:
 

models/gold/defi/defi__fact_trades.yml

@@ -2,7 +2,7 @@ version: 2
 
 models:
   - name: defi__fact_trades
-    description: A fact table containing trade execution details from both the decentralized exchange and liquidity pools.
+    description: "{{ doc('defi__fact_trades') }}"
     columns:
       - name: history_operation_id
         description: '{{ doc("history_operation_id") }}'

models/gold/operations/core__ez_operations.yml

@@ -2,7 +2,7 @@ version: 2
 
 models:
   - name: core__ez_operations
-    description: A comprehensive view combining operations with their associated transaction and ledger data.
+    description: "{{ doc('core__ez_operations') }}"
     columns:
 
       - name: OP_ID

models/silver/core/silver__trust_lines.sql

@@ -0,0 +1,122 @@
+-- depends_on: {{ ref('bronze__trust_lines') }}
+{{ config(
+    materialized = 'incremental',
+    unique_key = "trust_lines_id",
+    incremental_predicates = ["dynamic_range_predicate", "partition_id::date"],
+    merge_exclude_columns = ["inserted_timestamp"],
+    cluster_by = ['closed_at::DATE','partition_id','modified_timestamp::DATE'],
+    tags = ['scheduled_core'],
+) }}
+
+{% if execute %}
+
+{% if is_incremental() %}
+{% set max_is_query %}
+
+SELECT
+    MAX(_inserted_timestamp) AS _inserted_timestamp,
+    MAX(partition_gte_id) AS partition__gte_id
+FROM
+    {{ this }}
+
+    {% endset %}
+    {% set result = run_query(max_is_query) %}
+    {% set max_is = result [0] [0] %}
+    {% set max_part = result [0] [1] %}
+{% endif %}
+{% endif %}
+
+WITH pre_final AS (
+    SELECT
+        partition_id,
+        partition_gte_id,
+        VALUE :account_id :: STRING AS account_id,
+        VALUE :asset_id :: bigint AS asset_id,
+        VALUE :asset_type :: STRING AS asset_type,
+        VALUE :asset_issuer :: STRING AS asset_issuer,
+        VALUE :asset_code :: STRING AS asset_code,
+        VALUE :liquidity_pool_id :: STRING AS liquidity_pool_id,
+        VALUE :balance :: FLOAT AS balance,
+        VALUE :trust_line_limit :: FLOAT AS trust_line_limit,
+        VALUE :buying_liabilities :: FLOAT AS buying_liabilities,
+        VALUE :selling_liabilities :: FLOAT AS selling_liabilities,
+        VALUE :flags :: STRING AS flags,
+        VALUE :last_modified_ledger :: bigint AS last_modified_ledger,
+        VALUE :ledger_entry_change :: INT AS ledger_entry_change,
+        VALUE :deleted :: BOOLEAN AS deleted,
+        TO_TIMESTAMP(
+            VALUE :closed_at :: INT,
+            6
+        ) AS closed_at,
+        VALUE :ledger_sequence :: bigint AS ledger_sequence,
+        VALUE :ledger_key :: STRING AS ledger_key,
+        VALUE :sponsor :: STRING AS sponsor,
+        VALUE :batch_id :: STRING AS batch_id,
+        TO_TIMESTAMP(
+            VALUE :batch_run_date :: INT,
+            6
+        ) AS batch_run_date,
+        TO_TIMESTAMP(
+            VALUE :batch_insert_ts :: INT,
+            6
+        ) AS batch_insert_ts,
+        _inserted_timestamp
+    FROM
+
+{% if is_incremental() %}
+{{ ref('bronze__trust_lines') }}
+{% else %}
+    {{ ref('bronze__trust_lines_FR') }}
+{% endif %}
+
+{% if is_incremental() %}
+WHERE
+    partition_gte_id >= '{{ max_part }}'
+    AND _inserted_timestamp > '{{ max_is }}'
+{% endif %}
+
+qualify ROW_NUMBER() over (
+    PARTITION BY account_id,
+    asset_type,
+    asset_issuer,
+    asset_code,
+    liquidity_pool_id,
+    closed_at
+    ORDER BY
+        batch_insert_ts DESC,
+        _inserted_timestamp DESC
+) = 1
+)
+SELECT
+    partition_id,
+    partition_gte_id,
+    account_id,
+    asset_id,
+    asset_type,
+    asset_issuer,
+    asset_code,
+    liquidity_pool_id,
+    balance,
+    trust_line_limit,
+    buying_liabilities,
+    selling_liabilities,
+    flags,
+    last_modified_ledger,
+    ledger_entry_change,
+    deleted,
+    closed_at,
+    ledger_sequence,
+    ledger_key,
+    sponsor,
+    batch_id,
+    batch_run_date,
+    batch_insert_ts,
+    _inserted_timestamp,
+    {{ dbt_utils.generate_surrogate_key(
+        ['account_id', 'asset_type', 'asset_issuer', 'asset_code', 'liquidity_pool_id', 'closed_at']
+    ) }} AS trust_lines_id,
+    SYSDATE() AS inserted_timestamp,
+    SYSDATE() AS modified_timestamp,
+    '{{ invocation_id }}' AS _invocation_id
+FROM
+    pre_final

models/silver/core/silver__trust_lines.yml

@@ -0,0 +1,11 @@
+version: 2
+models:
+  - name: silver__trust_lines
+    columns:
+      - name: trust_lines_id
+        description: "{{ doc('id') }}"
+        tests:
+          - not_null:
+              where: modified_timestamp > current_date - {{ var('test_days_threshold', 3) }}
+          - unique:
+              where: modified_timestamp > current_date - {{ var('test_days_threshold', 3) }}

models/sources.yml

@@ -14,8 +14,8 @@ sources:
       - name: history_trades
       - name: history_transactions
       - name: liquidity_pools
-
       - name: streamline_ledgers
+      - name: trust_lines
   - name: crosschain
     database: "{{ 'crosschain' if target.database == 'STELLAR' else 'crosschain_dev' }}"
     schema: core