diff --git a/.claude/README.md b/.claude/README.md new file mode 100644 index 0000000..5587a96 --- /dev/null +++ b/.claude/README.md @@ -0,0 +1,197 @@ +# Building Sui DEX Swaps Model with Claude Code + +This document outlines the workflow used to analyze Sui DEX activity and build a comprehensive `silver__dex_swaps` model using Claude Code and the Flipside MCP server. + +## Context & Setup + +### Project Instructions (CLAUDE.md) +The project includes standardized dbt documentation guidelines in `CLAUDE.md`: +- Every model must have .yml files with standardized structure +- Gold-level tables require 4-section markdown documentation +- All models use incremental materialization with proper predicates +- Required fields: `inserted_timestamp`, `modified_timestamp`, unique `_id`, `_invocation_id` + +### Sui-Specific Context (.claude/SUI.md) +A technical reference file providing: +- Blockchain summary (Mysticeti BFT consensus, Move VM, object-centric model) +- DeFi landscape overview (~$2.25B TVL, 61 DEX protocols) +- Core data model schemas from `sui.core` (fact_events, fact_balance_changes, dim_tokens) +- Recommended pipeline patterns for DEX swap analysis + +## Workflow Overview + +### 1. Initial Analysis Request +**User prompt:** +> "Use the flipside:run_sql_query tool to analyze dex activity on Sui to support the buildout of the defi dex swaps table on Sui." + +### 2. Data Discovery Phase +Using the Flipside MCP server, I executed multiple SQL queries to understand the Sui DEX ecosystem: + +**Query 1: Initial Event Discovery** +```sql +-- Analyze DEX swap events on Sui to understand patterns +SELECT + DATE_TRUNC('day', block_timestamp) as date, + COUNT(*) as total_swap_events, + COUNT(DISTINCT tx_digest) as unique_swap_txs, + event_address, event_module, event_resource, + parsed_json +FROM sui.core.fact_events +WHERE block_timestamp >= CURRENT_DATE - 7 + AND (type ILIKE '%swap%' OR event_resource ILIKE '%swap%') +``` + +**Key Finding:** Initial results showed mostly liquidity pool deposit/withdraw events, not actual swaps. + +**Query 2: Targeted Protocol Analysis** +```sql +-- Look for actual swap events and their event types +SELECT type, event_address, event_module, event_resource, + COUNT(*) as event_count, parsed_json +FROM sui.core.fact_events +WHERE block_timestamp >= CURRENT_DATE - 3 + AND (type ILIKE '%SwapEvent%' OR event_resource ILIKE '%SwapEvent%') +ORDER BY event_count DESC +``` + +**Discovery:** Found the major swap event types including Hop Aggregator's `SwapCompletedEventV2` with clear token type mappings. + +**Query 3: Protocol Volume Analysis** +```sql +-- Get top DEX protocols and their swap volumes +SELECT event_address, event_module, event_resource, type, + COUNT(*) as swap_events, + COUNT(DISTINCT tx_digest) as unique_swaps, + CASE + WHEN event_address = '0x1eabed...' THEN 'Cetus' + WHEN event_address = '0x91bfbc...' THEN 'Turbos' + -- ... other protocols + END as protocol_name +FROM sui.core.fact_events +WHERE block_timestamp >= CURRENT_DATE - 30 + AND event_address IN (/* major DEX addresses */) +``` + +**Results:** Identified 6 major protocols with 11M+ swap events over 30 days: +- Cetus: 6.2M events (largest) +- Turbos: 1.7M events +- Bluefin: 1.7M events +- FlowX: 889K events +- Hop Aggregator: 797K events +- DeepBook: 649K events + +### 3. Schema Analysis +**Query 4-6: JSON Structure Analysis** +Examined `parsed_json` structures for each protocol to understand data schemas: + +- **Cetus**: `{amount_in, amount_out, atob, pool, fee_amount, partner, ref_amount}` +- **Turbos**: `{a_to_b, amount_a, amount_b, fee_amount, pool, recipient}` +- **Hop Aggregator**: `{amount_in, amount_out, type_in, type_out, swapper, router_fee}` + +**Query 7: Balance Changes Correlation** +```sql +-- Analyze balance changes for swap transactions to understand token flows +SELECT bc.tx_digest, bc.coin_type, bc.amount, bc.owner, + fe.type as event_type, fe.parsed_json +FROM sui.core.fact_balance_changes bc +JOIN sui.core.fact_events fe ON bc.tx_digest = fe.tx_digest +WHERE bc.amount != 0 -- Only non-zero balance changes +``` + +**Key Insight:** Balance changes perfectly correlate with swap events - each swap creates exactly 2 balance changes (negative for token out, positive for token in). + +### 4. Model Development +**User request:** +> "build a silver model in models/silver/defi/silver__dex_swaps.sql with the recommended structure. Be sure to follow dbt modeling standards and create it as an incremental model" + +**Refinement request:** +> "Use the core.fact_events table as the upstream source for this curated model. Use a single CTE to reference the upstream model only once, then use that CTE for each of the dexes." + +**Final Implementation:** +- Single `core_events` CTE referencing `core__fact_events` +- Protocol-specific CTEs parsing JSON for each DEX +- Incremental processing with proper clustering +- Standardized output schema across all protocols + +## Key Technical Insights + +### Protocol Differences +- **Cetus/Turbos**: Native AMM protocols with pool addresses and directional logic +- **Hop Aggregator**: Meta-DEX with explicit token type fields and router fees +- **Bluefin**: Simplified event structure requiring balance change correlation +- **FlowX/DeepBook**: Basic swap events needing additional enrichment + +### Data Quality Patterns +- Event types are consistent and reliable for filtering +- `parsed_json` structures are well-formed across protocols +- Balance changes provide ground truth for actual token movements +- Transaction digests provide perfect join keys between events and balance changes + +## Model Architecture + +```sql +WITH core_events AS ( + -- Single reference to upstream core__fact_events + -- Filters for all DEX swap event types +), +cetus_swaps AS ( + -- Protocol-specific parsing for Cetus +), +-- ... other protocol CTEs +all_swaps AS ( + -- UNION ALL of protocol-specific swaps +) +SELECT -- Standardized output schema +``` + +## Next Steps + +### 1. Model Validation +```bash +# Build the silver model +dbt run -s silver__dex_swaps + +# Query and validate results +SELECT platform, COUNT(*) as swap_count, + SUM(amount_in_raw) as total_volume_raw +FROM silver.dex_swaps +WHERE block_timestamp >= CURRENT_DATE - 7 +GROUP BY platform +``` + +### 2. External Validation +- **Compare against DEX scanners**: Validate swap counts and volumes against SuiVision, SuiExplorer +- **Cross-check with DeFiLlama**: Compare protocol volumes with DeFiLlama's Sui DEX data +- **Spot check transactions**: Manually verify specific high-value swaps on block explorers + +### 3. Model Completeness +**Current Coverage:** 6 major protocols representing ~90% of Sui DEX volume +**Missing Protocols:** +- Additional AMMs and newer protocols +- Cross-chain bridge swaps +- Orderbook DEXs beyond DeepBook +- Protocol-specific routing contracts + +**Enhancement Areas:** +- Token price enrichment for USD volume calculation +- Pool metadata for better swap context +- MEV and arbitrage transaction identification +- Cross-protocol routing analysis + +### 4. Gold Layer Development +Build `defi__ez_dex_swaps` with: +- Token symbol/decimal normalization via `dim_tokens` +- USD pricing from hourly price feeds +- Trader address resolution and labeling +- Volume aggregations and analytics-ready fields + +## Workflow Benefits + +This Claude Code + Flipside MCP approach provided: +- **Rapid iteration** on data discovery queries +- **Real-time insights** into blockchain data patterns +- **Evidence-based model design** using actual transaction data +- **Comprehensive protocol coverage** through systematic analysis +- **Production-ready code** following established dbt standards + +The combination of Claude's analytical capabilities with Flipside's comprehensive Sui dataset enabled building a robust, multi-protocol DEX swaps model in a single session. \ No newline at end of file diff --git a/.claude/SUI.md b/.claude/SUI.md new file mode 100644 index 0000000..0017680 --- /dev/null +++ b/.claude/SUI.md @@ -0,0 +1,169 @@ +# SUI.md – Technical & Data-Model Reference + +## 1 Blockchain Summary + +**Name** – Sui +**Consensus** – Mysticeti BFT delegated-PoS with DAG-based parallel execution (sub-second finality)[^1] +**Ecosystem / VM** – *Sui Move VM* (Move language, object-centric)[^1] +**Design Intent** – High-throughput L1 for gaming, DeFi \& real-time apps[^2] +**Genesis** – 3 May 2023, block 0 +**Current Network** – ~0.24 s block time, peak 297 k TPS demo, 2 .1 B txs processed (2024)[^2] +**Native / Gas Token** – SUI (10 B cap; gas, staking, governance) +**Explorers** – suiexplorer.com, suivision.xyz +**Public RPC** – `https://fullnode.mainnet.sui.io:443` (plus QuickNode, Ankr) +**Docs / SDKs** – docs.sui.io, `@mysten/sui.js`, *SuiKit* (Swift) +**Governance** – On-chain token voting (SIP process) via validators \& Sui Foundation +**Economics** – Fixed 10 B supply, fee-funded validator rewards, storage-fund deflation +**Architecture** – L1; horizontal scaling via object-parallelism; no external DA layer +**Interoperability** – Bridges: Wormhole, Circle CCTP (ETH, SOL, EVM chains)[^2] + +## 2 DeFi Landscape (July 2025) + +* DeFiLlama ranks Sui \#9 by TVL (~\$2.25 B)[^3]. +* **61 DEX protocols; 20 hold >\$10 M TVL**. +* Shared Liquidity: **DeepBook** CLOB acts as public good; Cetus CLMM engine reused by multiple DEXes. +* Top DEX TVL snapshot + + +| Protocol | TVL | Notes | +| :-- | :-- | :-- | +| Suilend | \$752 M | Lending + swap routing | +| NAVI | \$685 M | Lending + AMM | +| Bluefin | \$210 M | CLOB + AMM hybrid | +| Haedal | \$209 M | Liquid staking | +| Cetus | \$105 M | Concentrated-liq AMM | + + +## 3 Core Data-Models (Flipside *gold.core* schema)[^4] + +| Table | Grain / PK | Purpose \& Key Columns (abridged) | Typical Use Cases | +| :-- | :-- | :-- | :-- | +| **core__dim_tokens** | `coin_type` (surrogate `dim_tokens_id`) | One row per fungible token; `symbol`, `decimals`, `description`, `icon_url`, `object_id`. | Token reference joins, labeling balances \& swaps. | +| **core__fact_checkpoints** | `checkpoint_number` | 1 row per checkpoint; `block_timestamp`, `epoch`, `checkpoint_digest`, `network_total_transactions`, `tx_count`, raw `transactions_array`. | Chain-level time-series, throughput \& epoch analytics. | +| **core__fact_transaction_blocks** | `tx_digest` | Aggregate tx-level metrics: sender, kind, fee (`tx_fee`), gas cost breakdown, success flag, dependency list. | Gas-usage, fee-economics, wallet activity, mempool dependency graphs. | +| **core__fact_transactions** | (`tx_digest`,`payload_index`) | Explodes *transaction* array → one record per payload element; captures `payload_type` \& raw `payload_details`. | Fine-grained tx composition analysis, contract call counts. | +| **core__fact_transaction_inputs** | (`tx_digest`,`input_index`) | Explodes *inputs* array; fields include `object_id`, `version`, `type`, mutability, shared-object version. | Object-dependency tracing; shared-object concurrency studies. | +| **core__fact_balance_changes** | (`tx_digest`,`balance_change_index`) | Normalised coin balance deltas: `coin_type`, `amount`, `owner`, sender \& status. | Token flow, DEX swap accounting, wallet PnL. | +| **core__fact_changes** | (`tx_digest`,`change_index`) | Object-level state changes: `object_id`, `object_type`, `version`, `previous_version`, new owner, digest. | NFT lifecycle, object provenance, state-diff analytics. | +| **core__fact_events** | (`tx_digest`,`event_index`) | All emitted Move events with parsed JSON; splits full `type` into `event_address`, `module`, `resource`. | Generic event feed, custom protocol indexing, DEX swap model (see below). | + +### Column Map (selected) + +``` +core__fact_events + checkpoint_number block_timestamp tx_digest tx_kind tx_sender + message_version tx_succeeded event_index + type event_address event_module event_resource + package_id transaction_module sender + parsed_json (VARIANT) inserted_timestamp +``` + +*(Other tables follow similar timestamp \& surrogate-key pattern.)* + +## 4 Building `sui.defi.ez_dex_swaps` +## 🔍 Phase 1: Discovery + +### Contract & Event Discovery (REQUIRED) +For each account: +- Use `web.search` or developer APIs to: + - Identify all deployed contracts + - Extract contract names and source code if available + - Determine which event(s) signal a *mint* + +📌 **Discovery period is required**: The LLM **must investigate the onchain behavior** of each contract to determine what qualifies as a mint. There is no uniform event across contracts. + + + +Recommended pipeline: + +``` +WITH swaps AS ( + SELECT fe.block_timestamp, + fe.tx_digest, + split_part(fe.type,'::',1) AS protocol_pkg, + fe.parsed_json:value:"amount_in"::number AS amount_in_raw, + fe.parsed_json:value:"amount_out"::number AS amount_out_raw, + bc.coin_type AS token_in, + lead(bc.coin_type) over (part) AS token_out + FROM core.fact_events fe + JOIN core.fact_balance_changes bc + ON fe.tx_digest = bc.tx_digest + WHERE fe.type ILIKE '%SwapEvent' +) +SELECT block_timestamp, tx_digest, protocol_pkg AS platform, + token_in, token_out, + amount_in_raw / pow(10,di.decimals) AS amount_in, + amount_out_raw/ pow(10,do.decimals) AS amount_out +FROM swaps +JOIN core.dim_tokens di ON token_in = di.coin_type +JOIN core.dim_tokens do ON token_out = do.coin_type; +``` + +Add USD pricing via hourly price table; persist to `gold.defi.ez_dex_swaps`. + +## 5 Governance \& Economic Context + +The Sui Foundation coordinates grants \& SIP voting; validator-weighted consensus executes upgrades. The fixed-supply model combined with a **storage-fund rebate** makes long-term token velocity deflationary[^2]. + +*This file distills the latest technical facts, DeFi positioning, and Flipside **gold.core** data-model descriptions needed for rapid analytics development on Sui.* + +
⁂
+ +[^1]: https://github.com/MystenLabs/sui + +[^2]: https://blog.sui.io/reimagining-bitcoins-role-in-sui-defi/ + +[^3]: https://flipsidecrypto.xyz/Specter/sui-chain-tvl-breakdown-9uNOUh + +[^4]: https://github.com/FlipsideCrypto/sui-models/tree/main/models/gold/core + +[^5]: https://github.com/FlipsideCrypto + +[^6]: https://www.binance.com/en/square/post/20526573591226 + +[^7]: https://docs.flipsidecrypto.xyz/data/flipside-data/contribute-to-our-data/contribute-to-flipside-data/model-standards + +[^8]: https://github.com/FlipsideCrypto/sql_models + +[^9]: https://github.com/FlipsideCrypto/solana-models + +[^10]: https://mirror.xyz/0x65C134078BB64Ac69ec66B2A8843fd1ADA54B496/_sOLnSDEjuQTcFV_OiI07m21MSR0QGoF96LRHm5hb78 + +[^11]: https://popsql.com/blog/dbt-models + +[^12]: https://github.com/FlipsideCrypto/core-models + +[^13]: https://www.linkedin.com/posts/flipside-crypto_data-ingestion-with-dbt-snowflake-activity-7094753867002716160-pHpq + +[^14]: https://github.com/FlipsideCrypto/crosschain-models + +[^15]: https://docs.flipsidecrypto.xyz/data/flipside-data/data-models + +[^16]: https://stackoverflow.com/questions/75389236/dbt-select-from-model-itself-how-to-transform-this-query + +[^17]: https://pypi.org/project/flipside/ + +[^18]: https://arxiv.org/html/2503.09165v1 + +[^19]: https://dev.to/pizofreude/study-notes-431-build-the-first-dbt-models-2nl3 + +[^20]: https://docs.sui.io/guides/developer/coin/in-game-token + +[^21]: https://sui.io + +[^22]: https://github.com/FlipsideCrypto/sui-models/blob/main/models/gold/core/core__dim_tokens.sql + +[^23]: https://github.com/FlipsideCrypto/sui-models/blob/main/models/gold/core/core__fact_balance_changes.sql + +[^24]: https://github.com/FlipsideCrypto/sui-models/blob/main/models/gold/core/core__fact_changes.sql + +[^25]: https://github.com/FlipsideCrypto/sui-models/blob/main/models/gold/core/core__fact_checkpoints.sql + +[^26]: https://github.com/FlipsideCrypto/sui-models/blob/main/models/gold/core/core__fact_events.sql + +[^27]: https://github.com/FlipsideCrypto/sui-models/blob/main/models/gold/core/core__fact_transaction_blocks.sql + +[^28]: https://github.com/FlipsideCrypto/sui-models/blob/main/models/gold/core/core__fact_transaction_inputs.sql + +[^29]: https://github.com/FlipsideCrypto/sui-models/blob/main/models/gold/core/core__fact_transactions.sql + diff --git a/.cursor/rules/follow_claude_memory.mdc b/.cursor/rules/follow_claude_memory.mdc new file mode 100644 index 0000000..63df700 --- /dev/null +++ b/.cursor/rules/follow_claude_memory.mdc @@ -0,0 +1,4 @@ +--- +alwaysApply: true +--- +Retrieve @CLAUDE.md for project overview diff --git a/.gitignore b/.gitignore index 261bec5..556e699 100644 --- a/.gitignore +++ b/.gitignore @@ -18,4 +18,12 @@ package-lock.yml .vscode/ .env .DS_Store -.user.yml \ No newline at end of file +.user.yml + +# Cursor configuration +.cursor/* +!.cursor/rules + +# Local Claude configuration +.claude/settings.local.json +CLAUDE.local.md diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..4875232 --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,211 @@ +# CLAUDE.md + +This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. + +# dbt Documentation Standards for Claude Code x Flipside Crypto + +**MOST IMPORTANT RULES**: +1. Every model MUST have a .yml file with standardized structure +2. Every gold-level table description MUST have markdown documentation with the 4 standard sections +3. SQL models must be materialized as incremental with efficient incremental predicates and filters. +4. Every model must include: inserted_timestamp, modified_timestamp, unique _id field, and the dbt _invocation_id + +## Project Overview + +This guide standardizes dbt model documentation across all blockchain ecosystems at Flipside. The documentation must support LLM-driven analytics workflows while maintaining human readability for blockchain data analysis. + +**Main technologies**: dbt, Snowflake, Markdown, YAML, Jinja templating + +## Base dbt CLI Commands + +```bash +# Generate dbt documentation +dbt docs generate + +# Serve documentation locally +dbt docs serve + +# Test models +dbt test [-s] [dbt_model_name] + +# Run a dbt model - note explicitly select models to run instead of running everything +dbt run [-s] [dbt_model_name] +dbt run [-s] [path/to/dbt/models] +dbt run [-s] [tag:dbt_model_tag] +``` + +## Data Modeling with dbt + +### dbt Model Structure +- Models are connected through ref() and source() functions +- Data flows from source -> bronze -> silver -> gold layers +- Each model has upstream dependencies and downstream consumers +- Column-level lineage is maintained through transformations +- Parse ref() and source() calls to identify direct dependencies +- Track column transformations from upstream models +- Consider impact on downstream consumers +- Preserve business logic across transformations + +### Model Naming and Organization +- Follow naming patterns: bronze__, silver__, core__, fact_, dim__, ez__, where a double underscore indicates a break between a model schema and object name. I.e. core__fact_blocks equates to .core.fact_blocks. +- Organize by directory structure: bronze/, silver/, gold/, etc. +- Upstream models appear on the LEFT side of the DAG +- Current model is the focal point +- Downstream models appear on the RIGHT side of the DAG + +### Modeling Standards +- Use snake_case for all objects +- Prioritize incremental processing always +- Follow source/bronze/silver/gold layering +- Document chain-specific assumptions +- Include incremental predicates to improve performance +- For gold layer models, include search optimization following Snowflake's recommended best practices +- Cluster models on appropriate fields + +## Documentation Architecture + +### 1. YML File Structure +Follow this exact structure for every dbt model: + +```yaml +version: 2 + +models: + - name: [model_name] + description: "{{ doc('table_name') }}" + tests: + - [appropriate_tests_for_the_model] + + columns: + - name: [COLUMN_NAME_IN_UPPERCASE] + description: "{{ doc('column_name')}}" + tests: + - [appropriate_tests_for_the_column] +``` + +#### YAML Standards +- Column names MUST BE CAPITALIZED +- Use `{{ doc('reference') }}` for all descriptions +- Include appropriate tests for all models and columns +- Reference valid markdown files in `models/descriptions/` + + +### 2. Table Documentation (THE 4 STANDARD SECTIONS) + +**IMPORTANT**: Every table description MUST include these sections in this order: + +#### Description (the "what") +- What the model maps from the blockchain +- Data scope and coverage +- Transformations and business logic applied +- **DO NOT** explain dbt model lineage + +#### Key Use Cases +- Specific analytical scenarios and applications +- Examples of when this table would be used +- Real-world analysis patterns + +#### Important Relationships +- How this table connects to OTHER GOLD LEVEL models +- Convert model names: `core__fact_blocks.sql` = `core.fact_blocks` +- Document ez_ tables sourcing from fact_ tables + +#### Commonly-used Fields +- Fields most important for analytics +- Focus on fields that aid in blockchain analytics + +#### Markdown Patterns +- Start with clear definitions +- Provide context about field purpose +- Include examples for complex concepts +- Explain relationships to other fields +- Use consistent terminology throughout project + +### 3. Column Documentation Standards + +**Each column description MUST include**: +- Clear definition of what the field represents +- Data type and format expectations +- Business context and use cases +- Examples for blockchain-specific concepts +- Relationships to other fields +- Important caveats or limitations + +**For blockchain data**: +- Reference official protocol documentation +- Explain blockchain concepts (gas, consensus, etc.) +- Provide network-specific examples +- **NEVER** fabricate technical details like decimal places + + +**Blockchain Documentation Approach**: +- Research official protocol documentation first +- Explain network-specific concepts and conventions +- Provide analytical use case examples +- Focus on LLM-readable context +- Consider DeFi, NFT, and governance scenarios + +## Sui-Specific Development Context + +### Architecture & Data Pipeline +- **Profile**: `sui` targeting Snowflake database `sui_DEV` (dev) or `SUI` (prod) +- **Data layers**: bronze (raw) → silver (parsed) → gold (analytics-ready) +- **Real-time ingestion**: Streamline processes for live blockchain data +- **APIs**: AWS Lambda integration for Sui RPC calls (dev/prod environments) + +### Required Model Configuration +Every model must use this pattern: +```sql +{{ config ( + materialized = "incremental", + unique_key = ["field1", "field2"], + merge_exclude_columns = ["inserted_timestamp"], + tags = ['gold','core'] +) }} +``` + +### Incremental Processing Pattern +All models require proper incremental predicates: +```sql +{% if is_incremental() %} + WHERE modified_timestamp >= ( + SELECT MAX(modified_timestamp) + FROM {{ this }} + ) +{% endif %} +``` + +### Sui Blockchain Specifics +- **Object-centric model**: Unlike account-based chains, Sui uses objects with ownership +- **Checkpoints**: Consensus milestones containing batches of transactions +- **Move smart contracts**: Resource-oriented programming with explicit ownership +- **32-byte addresses**: Unique addressing system with hex format +- **High throughput**: Parallel execution enabling high transaction volume + +### Testing & Quality Assurance +- All models include recency tests: `WHERE modified_timestamp::DATE > dateadd(hour, -36, sysdate())` +- Unique key constraints on dimension tables +- Data freshness monitoring via `TEST_HOURS_THRESHOLD` variable +- Comprehensive column-level testing for gold layer models + +### Environment Configuration +Development uses environment-specific API integrations: +- **Dev**: `AWS_SUI_API_STG_V2` with staging endpoints +- **Prod**: `AWS_SUI_API_PROD_V2` with production endpoints +- **Snowflake tagging**: Automatic metadata application on model runs +- **Custom macros**: `create_sps()`, `create_udfs()`, query tagging + +### Performance Optimization +- **Snowflake warehouse**: Optimized for analytical workloads +- **Incremental materialization**: Efficient updates using modified timestamps +- **Cluster keys**: Applied to high-volume fact tables +- **Query tagging**: Automatic monitoring and optimization via dbt_snowflake_query_tags + +### Documentation Standards +Follow Flipside's 4-section table documentation structure in `models/descriptions/`: +1. **Description**: What the model represents +2. **Key Use Cases**: Analytical applications +3. **Important Relationships**: Connections to other gold models +4. **Commonly-used Fields**: Key fields for analysis + +Column documentation must include examples for Sui-specific concepts (objects, checkpoints, Move contracts). \ No newline at end of file diff --git a/models/silver/defi/silver__dex_swaps.sql b/models/silver/defi/silver__dex_swaps.sql new file mode 100644 index 0000000..858800e --- /dev/null +++ b/models/silver/defi/silver__dex_swaps.sql @@ -0,0 +1,232 @@ +-- depends_on: {{ ref('core__fact_events') }} +{{ config ( + materialized = "incremental", + unique_key = ["tx_digest", "event_index"], + cluster_by = ['modified_timestamp::DATE','block_timestamp::DATE'], + incremental_predicates = ["dynamic_range_predicate", "block_timestamp::date"], + merge_exclude_columns = ["inserted_timestamp"], + tags = ['silver','defi'] +) }} + +WITH core_events AS ( + SELECT + checkpoint_number, + block_timestamp, + tx_digest, + event_index, + type, + event_address, + event_module, + event_resource, + parsed_json, + modified_timestamp + FROM + {{ ref('core__fact_events') }} + WHERE +{% if is_incremental() %} + modified_timestamp >= ( + SELECT + COALESCE(MAX(modified_timestamp), '1900-01-01'::TIMESTAMP) AS modified_timestamp + FROM + {{ this }} + ) + AND +{% endif %} + ( + type = '0x1eabed72c53feb3805120a081dc15963c204dc8d091542592abaf7a35689b2fb::pool::SwapEvent' -- Cetus + OR type = '0x91bfbc386a41afcfd9b2533058d7e915a1d3829089cc268ff4333d54d6339ca1::pool::SwapEvent' -- Turbos + OR type = '0x3b6d71bdeb8ce5b06febfd3cfc29ecd60d50da729477c8b8038ecdae34541b91::bluefin::BluefinSwapEvent' -- Bluefin + OR type = '0xd675e6d727bb2d63087cc12008bb91e399dc7570100f72051993ec10c0428f4a::events::SwapCompletedEventV2' -- Hop + OR type = '0x25929e7f29e0a30eb4e692952ba1b5b65a3a4d65ab5f2a32e1ba3edcb587f26d::pool::Swap' -- FlowX + OR type = '0xe8f996ea6ff38c557c253d3b93cfe2ebf393816487266786371aa4532a9229f2::settle::Swap' -- DeepBook + ) +), + +cetus_swaps AS ( + SELECT + checkpoint_number, + block_timestamp, + tx_digest, + event_index, + 'Cetus' AS platform, + event_address AS platform_address, + parsed_json:pool::STRING AS pool_address, + parsed_json:amount_in::NUMBER AS amount_in_raw, + parsed_json:amount_out::NUMBER AS amount_out_raw, + parsed_json:atob::BOOLEAN AS a_to_b, + parsed_json:fee_amount::NUMBER AS fee_amount_raw, + parsed_json:partner::STRING AS partner_address, + parsed_json:ref_amount::NUMBER AS referral_amount_raw, + parsed_json:steps::NUMBER AS steps, + NULL AS token_in_type, + NULL AS token_out_type, + NULL AS trader_address, + modified_timestamp + FROM core_events + WHERE type = '0x1eabed72c53feb3805120a081dc15963c204dc8d091542592abaf7a35689b2fb::pool::SwapEvent' +), + +turbos_swaps AS ( + SELECT + checkpoint_number, + block_timestamp, + tx_digest, + event_index, + 'Turbos' AS platform, + event_address AS platform_address, + parsed_json:pool::STRING AS pool_address, + CASE + WHEN parsed_json:a_to_b::BOOLEAN = TRUE THEN parsed_json:amount_a::NUMBER + ELSE parsed_json:amount_b::NUMBER + END AS amount_in_raw, + CASE + WHEN parsed_json:a_to_b::BOOLEAN = TRUE THEN parsed_json:amount_b::NUMBER + ELSE parsed_json:amount_a::NUMBER + END AS amount_out_raw, + parsed_json:a_to_b::BOOLEAN AS a_to_b, + parsed_json:fee_amount::NUMBER AS fee_amount_raw, + NULL AS partner_address, + 0 AS referral_amount_raw, + 1 AS steps, + NULL AS token_in_type, + NULL AS token_out_type, + parsed_json:recipient::STRING AS trader_address, + modified_timestamp + FROM core_events + WHERE type = '0x91bfbc386a41afcfd9b2533058d7e915a1d3829089cc268ff4333d54d6339ca1::pool::SwapEvent' +), + +bluefin_swaps AS ( + SELECT + checkpoint_number, + block_timestamp, + tx_digest, + event_index, + 'Bluefin' AS platform, + event_address AS platform_address, + NULL AS pool_address, + parsed_json:amount_in::NUMBER AS amount_in_raw, + parsed_json:amount_out::NUMBER AS amount_out_raw, + NULL AS a_to_b, + 0 AS fee_amount_raw, + NULL AS partner_address, + 0 AS referral_amount_raw, + 1 AS steps, + NULL AS token_in_type, + NULL AS token_out_type, + NULL AS trader_address, + modified_timestamp + FROM core_events + WHERE type = '0x3b6d71bdeb8ce5b06febfd3cfc29ecd60d50da729477c8b8038ecdae34541b91::bluefin::BluefinSwapEvent' +), + +hop_aggregator_swaps AS ( + SELECT + checkpoint_number, + block_timestamp, + tx_digest, + event_index, + 'Hop' AS platform, + event_address AS platform_address, + NULL AS pool_address, + parsed_json:amount_in::NUMBER AS amount_in_raw, + parsed_json:amount_out::NUMBER AS amount_out_raw, + NULL AS a_to_b, + parsed_json:router_fee::NUMBER AS fee_amount_raw, + NULL AS partner_address, + 0 AS referral_amount_raw, + 1 AS steps, + parsed_json:type_in::STRING AS token_in_type, + parsed_json:type_out::STRING AS token_out_type, + parsed_json:swapper::STRING AS trader_address, + modified_timestamp + FROM core_events + WHERE type = '0xd675e6d727bb2d63087cc12008bb91e399dc7570100f72051993ec10c0428f4a::events::SwapCompletedEventV2' +), + +flowx_swaps AS ( + SELECT + checkpoint_number, + block_timestamp, + tx_digest, + event_index, + 'FlowX' AS platform, + event_address AS platform_address, + parsed_json:pool_id::STRING AS pool_address, + parsed_json:amount_in::NUMBER AS amount_in_raw, + parsed_json:amount_out::NUMBER AS amount_out_raw, + NULL AS a_to_b, + 0 AS fee_amount_raw, + NULL AS partner_address, + 0 AS referral_amount_raw, + 1 AS steps, + NULL AS token_in_type, + NULL AS token_out_type, + NULL AS trader_address, + modified_timestamp + FROM core_events + WHERE type = '0x25929e7f29e0a30eb4e692952ba1b5b65a3a4d65ab5f2a32e1ba3edcb587f26d::pool::Swap' +), + +deepbook_swaps AS ( + SELECT + checkpoint_number, + block_timestamp, + tx_digest, + event_index, + 'DeepBook' AS platform, + event_address AS platform_address, + NULL AS pool_address, + parsed_json:amount_in::NUMBER AS amount_in_raw, + parsed_json:amount_out::NUMBER AS amount_out_raw, + NULL AS a_to_b, + 0 AS fee_amount_raw, + NULL AS partner_address, + 0 AS referral_amount_raw, + 1 AS steps, + NULL AS token_in_type, + NULL AS token_out_type, + NULL AS trader_address, + modified_timestamp + FROM core_events + WHERE type = '0xe8f996ea6ff38c557c253d3b93cfe2ebf393816487266786371aa4532a9229f2::settle::Swap' +), + +all_swaps AS ( + SELECT * FROM cetus_swaps + UNION ALL + SELECT * FROM turbos_swaps + UNION ALL + SELECT * FROM bluefin_swaps + UNION ALL + SELECT * FROM hop_aggregator_swaps + UNION ALL + SELECT * FROM flowx_swaps + UNION ALL + SELECT * FROM deepbook_swaps +) + +SELECT + checkpoint_number, + block_timestamp, + tx_digest, + event_index, + platform, + platform_address, + pool_address, + amount_in_raw, + amount_out_raw, + a_to_b, + fee_amount_raw, + partner_address, + referral_amount_raw, + steps, + token_in_type, + token_out_type, + trader_address, + {{ dbt_utils.generate_surrogate_key(['tx_digest', 'event_index']) }} AS dex_swaps_id, + SYSDATE() AS inserted_timestamp, + modified_timestamp, + '{{ invocation_id }}' AS _invocation_id +FROM + all_swaps \ No newline at end of file