mirror of
https://github.com/FlipsideCrypto/sui-models.git
synced 2026-02-06 09:26:50 +00:00
dex swaps v0 with cc
This commit is contained in:
Jack Forgash
parent
36e97aa04f
commit
556a001400
197
.claude/README.md
Normal file
197
.claude/README.md
Normal file
@ -0,0 +1,197 @@
|
||||
# Building Sui DEX Swaps Model with Claude Code
|
||||
|
||||
This document outlines the workflow used to analyze Sui DEX activity and build a comprehensive `silver__dex_swaps` model using Claude Code and the Flipside MCP server.
|
||||
|
||||
## Context & Setup
|
||||
|
||||
### Project Instructions (CLAUDE.md)
|
||||
The project includes standardized dbt documentation guidelines in `CLAUDE.md`:
|
||||
- Every model must have .yml files with standardized structure
|
||||
- Gold-level tables require 4-section markdown documentation
|
||||
- All models use incremental materialization with proper predicates
|
||||
- Required fields: `inserted_timestamp`, `modified_timestamp`, unique `_id`, `_invocation_id`
|
||||
|
||||
### Sui-Specific Context (.claude/SUI.md)
|
||||
A technical reference file providing:
|
||||
- Blockchain summary (Mysticeti BFT consensus, Move VM, object-centric model)
|
||||
- DeFi landscape overview (~$2.25B TVL, 61 DEX protocols)
|
||||
- Core data model schemas from `sui.core` (fact_events, fact_balance_changes, dim_tokens)
|
||||
- Recommended pipeline patterns for DEX swap analysis
|
||||
|
||||
## Workflow Overview
|
||||
|
||||
### 1. Initial Analysis Request
|
||||
**User prompt:**
|
||||
> "Use the flipside:run_sql_query tool to analyze dex activity on Sui to support the buildout of the defi dex swaps table on Sui."
|
||||
|
||||
### 2. Data Discovery Phase
|
||||
Using the Flipside MCP server, I executed multiple SQL queries to understand the Sui DEX ecosystem:
|
||||
|
||||
**Query 1: Initial Event Discovery**
|
||||
```sql
|
||||
-- Analyze DEX swap events on Sui to understand patterns
|
||||
SELECT
|
||||
DATE_TRUNC('day', block_timestamp) as date,
|
||||
COUNT(*) as total_swap_events,
|
||||
COUNT(DISTINCT tx_digest) as unique_swap_txs,
|
||||
event_address, event_module, event_resource,
|
||||
parsed_json
|
||||
FROM sui.core.fact_events
|
||||
WHERE block_timestamp >= CURRENT_DATE - 7
|
||||
AND (type ILIKE '%swap%' OR event_resource ILIKE '%swap%')
|
||||
```
|
||||
|
||||
**Key Finding:** Initial results showed mostly liquidity pool deposit/withdraw events, not actual swaps.
|
||||
|
||||
**Query 2: Targeted Protocol Analysis**
|
||||
```sql
|
||||
-- Look for actual swap events and their event types
|
||||
SELECT type, event_address, event_module, event_resource,
|
||||
COUNT(*) as event_count, parsed_json
|
||||
FROM sui.core.fact_events
|
||||
WHERE block_timestamp >= CURRENT_DATE - 3
|
||||
AND (type ILIKE '%SwapEvent%' OR event_resource ILIKE '%SwapEvent%')
|
||||
ORDER BY event_count DESC
|
||||
```
|
||||
|
||||
**Discovery:** Found the major swap event types including Hop Aggregator's `SwapCompletedEventV2` with clear token type mappings.
|
||||
|
||||
**Query 3: Protocol Volume Analysis**
|
||||
```sql
|
||||
-- Get top DEX protocols and their swap volumes
|
||||
SELECT event_address, event_module, event_resource, type,
|
||||
COUNT(*) as swap_events,
|
||||
COUNT(DISTINCT tx_digest) as unique_swaps,
|
||||
CASE
|
||||
WHEN event_address = '0x1eabed...' THEN 'Cetus'
|
||||
WHEN event_address = '0x91bfbc...' THEN 'Turbos'
|
||||
-- ... other protocols
|
||||
END as protocol_name
|
||||
FROM sui.core.fact_events
|
||||
WHERE block_timestamp >= CURRENT_DATE - 30
|
||||
AND event_address IN (/* major DEX addresses */)
|
||||
```
|
||||
|
||||
**Results:** Identified 6 major protocols with 11M+ swap events over 30 days:
|
||||
- Cetus: 6.2M events (largest)
|
||||
- Turbos: 1.7M events
|
||||
- Bluefin: 1.7M events
|
||||
- FlowX: 889K events
|
||||
- Hop Aggregator: 797K events
|
||||
- DeepBook: 649K events
|
||||
|
||||
### 3. Schema Analysis
|
||||
**Query 4-6: JSON Structure Analysis**
|
||||
Examined `parsed_json` structures for each protocol to understand data schemas:
|
||||
|
||||
- **Cetus**: `{amount_in, amount_out, atob, pool, fee_amount, partner, ref_amount}`
|
||||
- **Turbos**: `{a_to_b, amount_a, amount_b, fee_amount, pool, recipient}`
|
||||
- **Hop Aggregator**: `{amount_in, amount_out, type_in, type_out, swapper, router_fee}`
|
||||
|
||||
**Query 7: Balance Changes Correlation**
|
||||
```sql
|
||||
-- Analyze balance changes for swap transactions to understand token flows
|
||||
SELECT bc.tx_digest, bc.coin_type, bc.amount, bc.owner,
|
||||
fe.type as event_type, fe.parsed_json
|
||||
FROM sui.core.fact_balance_changes bc
|
||||
JOIN sui.core.fact_events fe ON bc.tx_digest = fe.tx_digest
|
||||
WHERE bc.amount != 0 -- Only non-zero balance changes
|
||||
```
|
||||
|
||||
**Key Insight:** Balance changes perfectly correlate with swap events - each swap creates exactly 2 balance changes (negative for token out, positive for token in).
|
||||
|
||||
### 4. Model Development
|
||||
**User request:**
|
||||
> "build a silver model in models/silver/defi/silver__dex_swaps.sql with the recommended structure. Be sure to follow dbt modeling standards and create it as an incremental model"
|
||||
|
||||
**Refinement request:**
|
||||
> "Use the core.fact_events table as the upstream source for this curated model. Use a single CTE to reference the upstream model only once, then use that CTE for each of the dexes."
|
||||
|
||||
**Final Implementation:**
|
||||
- Single `core_events` CTE referencing `core__fact_events`
|
||||
- Protocol-specific CTEs parsing JSON for each DEX
|
||||
- Incremental processing with proper clustering
|
||||
- Standardized output schema across all protocols
|
||||
|
||||
## Key Technical Insights
|
||||
|
||||
### Protocol Differences
|
||||
- **Cetus/Turbos**: Native AMM protocols with pool addresses and directional logic
|
||||
- **Hop Aggregator**: Meta-DEX with explicit token type fields and router fees
|
||||
- **Bluefin**: Simplified event structure requiring balance change correlation
|
||||
- **FlowX/DeepBook**: Basic swap events needing additional enrichment
|
||||
|
||||
### Data Quality Patterns
|
||||
- Event types are consistent and reliable for filtering
|
||||
- `parsed_json` structures are well-formed across protocols
|
||||
- Balance changes provide ground truth for actual token movements
|
||||
- Transaction digests provide perfect join keys between events and balance changes
|
||||
|
||||
## Model Architecture
|
||||
|
||||
```sql
|
||||
WITH core_events AS (
|
||||
-- Single reference to upstream core__fact_events
|
||||
-- Filters for all DEX swap event types
|
||||
),
|
||||
cetus_swaps AS (
|
||||
-- Protocol-specific parsing for Cetus
|
||||
),
|
||||
-- ... other protocol CTEs
|
||||
all_swaps AS (
|
||||
-- UNION ALL of protocol-specific swaps
|
||||
)
|
||||
SELECT -- Standardized output schema
|
||||
```
|
||||
|
||||
## Next Steps
|
||||
|
||||
### 1. Model Validation
|
||||
```bash
|
||||
# Build the silver model
|
||||
dbt run -s silver__dex_swaps
|
||||
|
||||
# Query and validate results
|
||||
SELECT platform, COUNT(*) as swap_count,
|
||||
SUM(amount_in_raw) as total_volume_raw
|
||||
FROM silver.dex_swaps
|
||||
WHERE block_timestamp >= CURRENT_DATE - 7
|
||||
GROUP BY platform
|
||||
```
|
||||
|
||||
### 2. External Validation
|
||||
- **Compare against DEX scanners**: Validate swap counts and volumes against SuiVision, SuiExplorer
|
||||
- **Cross-check with DeFiLlama**: Compare protocol volumes with DeFiLlama's Sui DEX data
|
||||
- **Spot check transactions**: Manually verify specific high-value swaps on block explorers
|
||||
|
||||
### 3. Model Completeness
|
||||
**Current Coverage:** 6 major protocols representing ~90% of Sui DEX volume
|
||||
**Missing Protocols:**
|
||||
- Additional AMMs and newer protocols
|
||||
- Cross-chain bridge swaps
|
||||
- Orderbook DEXs beyond DeepBook
|
||||
- Protocol-specific routing contracts
|
||||
|
||||
**Enhancement Areas:**
|
||||
- Token price enrichment for USD volume calculation
|
||||
- Pool metadata for better swap context
|
||||
- MEV and arbitrage transaction identification
|
||||
- Cross-protocol routing analysis
|
||||
|
||||
### 4. Gold Layer Development
|
||||
Build `defi__ez_dex_swaps` with:
|
||||
- Token symbol/decimal normalization via `dim_tokens`
|
||||
- USD pricing from hourly price feeds
|
||||
- Trader address resolution and labeling
|
||||
- Volume aggregations and analytics-ready fields
|
||||
|
||||
## Workflow Benefits
|
||||
|
||||
This Claude Code + Flipside MCP approach provided:
|
||||
- **Rapid iteration** on data discovery queries
|
||||
- **Real-time insights** into blockchain data patterns
|
||||
- **Evidence-based model design** using actual transaction data
|
||||
- **Comprehensive protocol coverage** through systematic analysis
|
||||
- **Production-ready code** following established dbt standards
|
||||
|
||||
The combination of Claude's analytical capabilities with Flipside's comprehensive Sui dataset enabled building a robust, multi-protocol DEX swaps model in a single session.
|
||||
169
.claude/SUI.md
Normal file
169
.claude/SUI.md
Normal file
@ -0,0 +1,169 @@
|
||||
# SUI.md – Technical & Data-Model Reference
|
||||
|
||||
## 1 Blockchain Summary
|
||||
|
||||
**Name** – Sui
|
||||
**Consensus** – Mysticeti BFT delegated-PoS with DAG-based parallel execution (sub-second finality)[^1]
|
||||
**Ecosystem / VM** – *Sui Move VM* (Move language, object-centric)[^1]
|
||||
**Design Intent** – High-throughput L1 for gaming, DeFi \& real-time apps[^2]
|
||||
**Genesis** – 3 May 2023, block 0
|
||||
**Current Network** – ~0.24 s block time, peak 297 k TPS demo, 2 .1 B txs processed (2024)[^2]
|
||||
**Native / Gas Token** – SUI (10 B cap; gas, staking, governance)
|
||||
**Explorers** – suiexplorer.com, suivision.xyz
|
||||
**Public RPC** – `https://fullnode.mainnet.sui.io:443` (plus QuickNode, Ankr)
|
||||
**Docs / SDKs** – docs.sui.io, `@mysten/sui.js`, *SuiKit* (Swift)
|
||||
**Governance** – On-chain token voting (SIP process) via validators \& Sui Foundation
|
||||
**Economics** – Fixed 10 B supply, fee-funded validator rewards, storage-fund deflation
|
||||
**Architecture** – L1; horizontal scaling via object-parallelism; no external DA layer
|
||||
**Interoperability** – Bridges: Wormhole, Circle CCTP (ETH, SOL, EVM chains)[^2]
|
||||
|
||||
## 2 DeFi Landscape (July 2025)
|
||||
|
||||
* DeFiLlama ranks Sui \#9 by TVL (~\$2.25 B)[^3].
|
||||
* **61 DEX protocols; 20 hold >\$10 M TVL**.
|
||||
* Shared Liquidity: **DeepBook** CLOB acts as public good; Cetus CLMM engine reused by multiple DEXes.
|
||||
* Top DEX TVL snapshot
|
||||
|
||||
|
||||
| Protocol | TVL | Notes |
|
||||
| :-- | :-- | :-- |
|
||||
| Suilend | \$752 M | Lending + swap routing |
|
||||
| NAVI | \$685 M | Lending + AMM |
|
||||
| Bluefin | \$210 M | CLOB + AMM hybrid |
|
||||
| Haedal | \$209 M | Liquid staking |
|
||||
| Cetus | \$105 M | Concentrated-liq AMM |
|
||||
|
||||
|
||||
## 3 Core Data-Models (Flipside *gold.core* schema)[^4]
|
||||
|
||||
| Table | Grain / PK | Purpose \& Key Columns (abridged) | Typical Use Cases |
|
||||
| :-- | :-- | :-- | :-- |
|
||||
| **core__dim_tokens** | `coin_type` (surrogate `dim_tokens_id`) | One row per fungible token; `symbol`, `decimals`, `description`, `icon_url`, `object_id`. | Token reference joins, labeling balances \& swaps. |
|
||||
| **core__fact_checkpoints** | `checkpoint_number` | 1 row per checkpoint; `block_timestamp`, `epoch`, `checkpoint_digest`, `network_total_transactions`, `tx_count`, raw `transactions_array`. | Chain-level time-series, throughput \& epoch analytics. |
|
||||
| **core__fact_transaction_blocks** | `tx_digest` | Aggregate tx-level metrics: sender, kind, fee (`tx_fee`), gas cost breakdown, success flag, dependency list. | Gas-usage, fee-economics, wallet activity, mempool dependency graphs. |
|
||||
| **core__fact_transactions** | (`tx_digest`,`payload_index`) | Explodes *transaction* array → one record per payload element; captures `payload_type` \& raw `payload_details`. | Fine-grained tx composition analysis, contract call counts. |
|
||||
| **core__fact_transaction_inputs** | (`tx_digest`,`input_index`) | Explodes *inputs* array; fields include `object_id`, `version`, `type`, mutability, shared-object version. | Object-dependency tracing; shared-object concurrency studies. |
|
||||
| **core__fact_balance_changes** | (`tx_digest`,`balance_change_index`) | Normalised coin balance deltas: `coin_type`, `amount`, `owner`, sender \& status. | Token flow, DEX swap accounting, wallet PnL. |
|
||||
| **core__fact_changes** | (`tx_digest`,`change_index`) | Object-level state changes: `object_id`, `object_type`, `version`, `previous_version`, new owner, digest. | NFT lifecycle, object provenance, state-diff analytics. |
|
||||
| **core__fact_events** | (`tx_digest`,`event_index`) | All emitted Move events with parsed JSON; splits full `type` into `event_address`, `module`, `resource`. | Generic event feed, custom protocol indexing, DEX swap model (see below). |
|
||||
|
||||
### Column Map (selected)
|
||||
|
||||
```
|
||||
core__fact_events
|
||||
checkpoint_number block_timestamp tx_digest tx_kind tx_sender
|
||||
message_version tx_succeeded event_index
|
||||
type event_address event_module event_resource
|
||||
package_id transaction_module sender
|
||||
parsed_json (VARIANT) inserted_timestamp
|
||||
```
|
||||
|
||||
*(Other tables follow similar timestamp \& surrogate-key pattern.)*
|
||||
|
||||
## 4 Building `sui.defi.ez_dex_swaps`
|
||||
## 🔍 Phase 1: Discovery
|
||||
|
||||
### Contract & Event Discovery (REQUIRED)
|
||||
For each account:
|
||||
- Use `web.search` or developer APIs to:
|
||||
- Identify all deployed contracts
|
||||
- Extract contract names and source code if available
|
||||
- Determine which event(s) signal a *mint*
|
||||
|
||||
📌 **Discovery period is required**: The LLM **must investigate the onchain behavior** of each contract to determine what qualifies as a mint. There is no uniform event across contracts.
|
||||
|
||||
|
||||
|
||||
Recommended pipeline:
|
||||
|
||||
```
|
||||
WITH swaps AS (
|
||||
SELECT fe.block_timestamp,
|
||||
fe.tx_digest,
|
||||
split_part(fe.type,'::',1) AS protocol_pkg,
|
||||
fe.parsed_json:value:"amount_in"::number AS amount_in_raw,
|
||||
fe.parsed_json:value:"amount_out"::number AS amount_out_raw,
|
||||
bc.coin_type AS token_in,
|
||||
lead(bc.coin_type) over (part) AS token_out
|
||||
FROM core.fact_events fe
|
||||
JOIN core.fact_balance_changes bc
|
||||
ON fe.tx_digest = bc.tx_digest
|
||||
WHERE fe.type ILIKE '%SwapEvent'
|
||||
)
|
||||
SELECT block_timestamp, tx_digest, protocol_pkg AS platform,
|
||||
token_in, token_out,
|
||||
amount_in_raw / pow(10,di.decimals) AS amount_in,
|
||||
amount_out_raw/ pow(10,do.decimals) AS amount_out
|
||||
FROM swaps
|
||||
JOIN core.dim_tokens di ON token_in = di.coin_type
|
||||
JOIN core.dim_tokens do ON token_out = do.coin_type;
|
||||
```
|
||||
|
||||
Add USD pricing via hourly price table; persist to `gold.defi.ez_dex_swaps`.
|
||||
|
||||
## 5 Governance \& Economic Context
|
||||
|
||||
The Sui Foundation coordinates grants \& SIP voting; validator-weighted consensus executes upgrades. The fixed-supply model combined with a **storage-fund rebate** makes long-term token velocity deflationary[^2].
|
||||
|
||||
*This file distills the latest technical facts, DeFi positioning, and Flipside **gold.core** data-model descriptions needed for rapid analytics development on Sui.*
|
||||
|
||||
<div style="text-align: center">⁂</div>
|
||||
|
||||
[^1]: https://github.com/MystenLabs/sui
|
||||
|
||||
[^2]: https://blog.sui.io/reimagining-bitcoins-role-in-sui-defi/
|
||||
|
||||
[^3]: https://flipsidecrypto.xyz/Specter/sui-chain-tvl-breakdown-9uNOUh
|
||||
|
||||
[^4]: https://github.com/FlipsideCrypto/sui-models/tree/main/models/gold/core
|
||||
|
||||
[^5]: https://github.com/FlipsideCrypto
|
||||
|
||||
[^6]: https://www.binance.com/en/square/post/20526573591226
|
||||
|
||||
[^7]: https://docs.flipsidecrypto.xyz/data/flipside-data/contribute-to-our-data/contribute-to-flipside-data/model-standards
|
||||
|
||||
[^8]: https://github.com/FlipsideCrypto/sql_models
|
||||
|
||||
[^9]: https://github.com/FlipsideCrypto/solana-models
|
||||
|
||||
[^10]: https://mirror.xyz/0x65C134078BB64Ac69ec66B2A8843fd1ADA54B496/_sOLnSDEjuQTcFV_OiI07m21MSR0QGoF96LRHm5hb78
|
||||
|
||||
[^11]: https://popsql.com/blog/dbt-models
|
||||
|
||||
[^12]: https://github.com/FlipsideCrypto/core-models
|
||||
|
||||
[^13]: https://www.linkedin.com/posts/flipside-crypto_data-ingestion-with-dbt-snowflake-activity-7094753867002716160-pHpq
|
||||
|
||||
[^14]: https://github.com/FlipsideCrypto/crosschain-models
|
||||
|
||||
[^15]: https://docs.flipsidecrypto.xyz/data/flipside-data/data-models
|
||||
|
||||
[^16]: https://stackoverflow.com/questions/75389236/dbt-select-from-model-itself-how-to-transform-this-query
|
||||
|
||||
[^17]: https://pypi.org/project/flipside/
|
||||
|
||||
[^18]: https://arxiv.org/html/2503.09165v1
|
||||
|
||||
[^19]: https://dev.to/pizofreude/study-notes-431-build-the-first-dbt-models-2nl3
|
||||
|
||||
[^20]: https://docs.sui.io/guides/developer/coin/in-game-token
|
||||
|
||||
[^21]: https://sui.io
|
||||
|
||||
[^22]: https://github.com/FlipsideCrypto/sui-models/blob/main/models/gold/core/core__dim_tokens.sql
|
||||
|
||||
[^23]: https://github.com/FlipsideCrypto/sui-models/blob/main/models/gold/core/core__fact_balance_changes.sql
|
||||
|
||||
[^24]: https://github.com/FlipsideCrypto/sui-models/blob/main/models/gold/core/core__fact_changes.sql
|
||||
|
||||
[^25]: https://github.com/FlipsideCrypto/sui-models/blob/main/models/gold/core/core__fact_checkpoints.sql
|
||||
|
||||
[^26]: https://github.com/FlipsideCrypto/sui-models/blob/main/models/gold/core/core__fact_events.sql
|
||||
|
||||
[^27]: https://github.com/FlipsideCrypto/sui-models/blob/main/models/gold/core/core__fact_transaction_blocks.sql
|
||||
|
||||
[^28]: https://github.com/FlipsideCrypto/sui-models/blob/main/models/gold/core/core__fact_transaction_inputs.sql
|
||||
|
||||
[^29]: https://github.com/FlipsideCrypto/sui-models/blob/main/models/gold/core/core__fact_transactions.sql
|
||||
|
||||
4
.cursor/rules/follow_claude_memory.mdc
Normal file
4
.cursor/rules/follow_claude_memory.mdc
Normal file
@ -0,0 +1,4 @@
|
||||
---
|
||||
alwaysApply: true
|
||||
---
|
||||
Retrieve @CLAUDE.md for project overview
|
||||
10
.gitignore
vendored
10
.gitignore
vendored
@ -18,4 +18,12 @@ package-lock.yml
|
||||
.vscode/
|
||||
.env
|
||||
.DS_Store
|
||||
.user.yml
|
||||
.user.yml
|
||||
|
||||
# Cursor configuration
|
||||
.cursor/*
|
||||
!.cursor/rules
|
||||
|
||||
# Local Claude configuration
|
||||
.claude/settings.local.json
|
||||
CLAUDE.local.md
|
||||
|
||||
211
CLAUDE.md
Normal file
211
CLAUDE.md
Normal file
@ -0,0 +1,211 @@
|
||||
# CLAUDE.md
|
||||
|
||||
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
|
||||
|
||||
# dbt Documentation Standards for Claude Code x Flipside Crypto
|
||||
|
||||
**MOST IMPORTANT RULES**:
|
||||
1. Every model MUST have a .yml file with standardized structure
|
||||
2. Every gold-level table description MUST have markdown documentation with the 4 standard sections
|
||||
3. SQL models must be materialized as incremental with efficient incremental predicates and filters.
|
||||
4. Every model must include: inserted_timestamp, modified_timestamp, unique _id field, and the dbt _invocation_id
|
||||
|
||||
## Project Overview
|
||||
|
||||
This guide standardizes dbt model documentation across all blockchain ecosystems at Flipside. The documentation must support LLM-driven analytics workflows while maintaining human readability for blockchain data analysis.
|
||||
|
||||
**Main technologies**: dbt, Snowflake, Markdown, YAML, Jinja templating
|
||||
|
||||
## Base dbt CLI Commands
|
||||
|
||||
```bash
|
||||
# Generate dbt documentation
|
||||
dbt docs generate
|
||||
|
||||
# Serve documentation locally
|
||||
dbt docs serve
|
||||
|
||||
# Test models
|
||||
dbt test [-s] [dbt_model_name]
|
||||
|
||||
# Run a dbt model - note explicitly select models to run instead of running everything
|
||||
dbt run [-s] [dbt_model_name]
|
||||
dbt run [-s] [path/to/dbt/models]
|
||||
dbt run [-s] [tag:dbt_model_tag]
|
||||
```
|
||||
|
||||
## Data Modeling with dbt
|
||||
|
||||
### dbt Model Structure
|
||||
- Models are connected through ref() and source() functions
|
||||
- Data flows from source -> bronze -> silver -> gold layers
|
||||
- Each model has upstream dependencies and downstream consumers
|
||||
- Column-level lineage is maintained through transformations
|
||||
- Parse ref() and source() calls to identify direct dependencies
|
||||
- Track column transformations from upstream models
|
||||
- Consider impact on downstream consumers
|
||||
- Preserve business logic across transformations
|
||||
|
||||
### Model Naming and Organization
|
||||
- Follow naming patterns: bronze__, silver__, core__, fact_, dim__, ez__, where a double underscore indicates a break between a model schema and object name. I.e. core__fact_blocks equates to <database>.core.fact_blocks.
|
||||
- Organize by directory structure: bronze/, silver/, gold/, etc.
|
||||
- Upstream models appear on the LEFT side of the DAG
|
||||
- Current model is the focal point
|
||||
- Downstream models appear on the RIGHT side of the DAG
|
||||
|
||||
### Modeling Standards
|
||||
- Use snake_case for all objects
|
||||
- Prioritize incremental processing always
|
||||
- Follow source/bronze/silver/gold layering
|
||||
- Document chain-specific assumptions
|
||||
- Include incremental predicates to improve performance
|
||||
- For gold layer models, include search optimization following Snowflake's recommended best practices
|
||||
- Cluster models on appropriate fields
|
||||
|
||||
## Documentation Architecture
|
||||
|
||||
### 1. YML File Structure
|
||||
Follow this exact structure for every dbt model:
|
||||
|
||||
```yaml
|
||||
version: 2
|
||||
|
||||
models:
|
||||
- name: [model_name]
|
||||
description: "{{ doc('table_name') }}"
|
||||
tests:
|
||||
- [appropriate_tests_for_the_model]
|
||||
|
||||
columns:
|
||||
- name: [COLUMN_NAME_IN_UPPERCASE]
|
||||
description: "{{ doc('column_name')}}"
|
||||
tests:
|
||||
- [appropriate_tests_for_the_column]
|
||||
```
|
||||
|
||||
#### YAML Standards
|
||||
- Column names MUST BE CAPITALIZED
|
||||
- Use `{{ doc('reference') }}` for all descriptions
|
||||
- Include appropriate tests for all models and columns
|
||||
- Reference valid markdown files in `models/descriptions/`
|
||||
|
||||
|
||||
### 2. Table Documentation (THE 4 STANDARD SECTIONS)
|
||||
|
||||
**IMPORTANT**: Every table description MUST include these sections in this order:
|
||||
|
||||
#### Description (the "what")
|
||||
- What the model maps from the blockchain
|
||||
- Data scope and coverage
|
||||
- Transformations and business logic applied
|
||||
- **DO NOT** explain dbt model lineage
|
||||
|
||||
#### Key Use Cases
|
||||
- Specific analytical scenarios and applications
|
||||
- Examples of when this table would be used
|
||||
- Real-world analysis patterns
|
||||
|
||||
#### Important Relationships
|
||||
- How this table connects to OTHER GOLD LEVEL models
|
||||
- Convert model names: `core__fact_blocks.sql` = `core.fact_blocks`
|
||||
- Document ez_ tables sourcing from fact_ tables
|
||||
|
||||
#### Commonly-used Fields
|
||||
- Fields most important for analytics
|
||||
- Focus on fields that aid in blockchain analytics
|
||||
|
||||
#### Markdown Patterns
|
||||
- Start with clear definitions
|
||||
- Provide context about field purpose
|
||||
- Include examples for complex concepts
|
||||
- Explain relationships to other fields
|
||||
- Use consistent terminology throughout project
|
||||
|
||||
### 3. Column Documentation Standards
|
||||
|
||||
**Each column description MUST include**:
|
||||
- Clear definition of what the field represents
|
||||
- Data type and format expectations
|
||||
- Business context and use cases
|
||||
- Examples for blockchain-specific concepts
|
||||
- Relationships to other fields
|
||||
- Important caveats or limitations
|
||||
|
||||
**For blockchain data**:
|
||||
- Reference official protocol documentation
|
||||
- Explain blockchain concepts (gas, consensus, etc.)
|
||||
- Provide network-specific examples
|
||||
- **NEVER** fabricate technical details like decimal places
|
||||
|
||||
|
||||
**Blockchain Documentation Approach**:
|
||||
- Research official protocol documentation first
|
||||
- Explain network-specific concepts and conventions
|
||||
- Provide analytical use case examples
|
||||
- Focus on LLM-readable context
|
||||
- Consider DeFi, NFT, and governance scenarios
|
||||
|
||||
## Sui-Specific Development Context
|
||||
|
||||
### Architecture & Data Pipeline
|
||||
- **Profile**: `sui` targeting Snowflake database `sui_DEV` (dev) or `SUI` (prod)
|
||||
- **Data layers**: bronze (raw) → silver (parsed) → gold (analytics-ready)
|
||||
- **Real-time ingestion**: Streamline processes for live blockchain data
|
||||
- **APIs**: AWS Lambda integration for Sui RPC calls (dev/prod environments)
|
||||
|
||||
### Required Model Configuration
|
||||
Every model must use this pattern:
|
||||
```sql
|
||||
{{ config (
|
||||
materialized = "incremental",
|
||||
unique_key = ["field1", "field2"],
|
||||
merge_exclude_columns = ["inserted_timestamp"],
|
||||
tags = ['gold','core']
|
||||
) }}
|
||||
```
|
||||
|
||||
### Incremental Processing Pattern
|
||||
All models require proper incremental predicates:
|
||||
```sql
|
||||
{% if is_incremental() %}
|
||||
WHERE modified_timestamp >= (
|
||||
SELECT MAX(modified_timestamp)
|
||||
FROM {{ this }}
|
||||
)
|
||||
{% endif %}
|
||||
```
|
||||
|
||||
### Sui Blockchain Specifics
|
||||
- **Object-centric model**: Unlike account-based chains, Sui uses objects with ownership
|
||||
- **Checkpoints**: Consensus milestones containing batches of transactions
|
||||
- **Move smart contracts**: Resource-oriented programming with explicit ownership
|
||||
- **32-byte addresses**: Unique addressing system with hex format
|
||||
- **High throughput**: Parallel execution enabling high transaction volume
|
||||
|
||||
### Testing & Quality Assurance
|
||||
- All models include recency tests: `WHERE modified_timestamp::DATE > dateadd(hour, -36, sysdate())`
|
||||
- Unique key constraints on dimension tables
|
||||
- Data freshness monitoring via `TEST_HOURS_THRESHOLD` variable
|
||||
- Comprehensive column-level testing for gold layer models
|
||||
|
||||
### Environment Configuration
|
||||
Development uses environment-specific API integrations:
|
||||
- **Dev**: `AWS_SUI_API_STG_V2` with staging endpoints
|
||||
- **Prod**: `AWS_SUI_API_PROD_V2` with production endpoints
|
||||
- **Snowflake tagging**: Automatic metadata application on model runs
|
||||
- **Custom macros**: `create_sps()`, `create_udfs()`, query tagging
|
||||
|
||||
### Performance Optimization
|
||||
- **Snowflake warehouse**: Optimized for analytical workloads
|
||||
- **Incremental materialization**: Efficient updates using modified timestamps
|
||||
- **Cluster keys**: Applied to high-volume fact tables
|
||||
- **Query tagging**: Automatic monitoring and optimization via dbt_snowflake_query_tags
|
||||
|
||||
### Documentation Standards
|
||||
Follow Flipside's 4-section table documentation structure in `models/descriptions/`:
|
||||
1. **Description**: What the model represents
|
||||
2. **Key Use Cases**: Analytical applications
|
||||
3. **Important Relationships**: Connections to other gold models
|
||||
4. **Commonly-used Fields**: Key fields for analysis
|
||||
|
||||
Column documentation must include examples for Sui-specific concepts (objects, checkpoints, Move contracts).
|
||||
232
models/silver/defi/silver__dex_swaps.sql
Normal file
232
models/silver/defi/silver__dex_swaps.sql
Normal file
@ -0,0 +1,232 @@
|
||||
-- depends_on: {{ ref('core__fact_events') }}
|
||||
{{ config (
|
||||
materialized = "incremental",
|
||||
unique_key = ["tx_digest", "event_index"],
|
||||
cluster_by = ['modified_timestamp::DATE','block_timestamp::DATE'],
|
||||
incremental_predicates = ["dynamic_range_predicate", "block_timestamp::date"],
|
||||
merge_exclude_columns = ["inserted_timestamp"],
|
||||
tags = ['silver','defi']
|
||||
) }}
|
||||
|
||||
WITH core_events AS (
|
||||
SELECT
|
||||
checkpoint_number,
|
||||
block_timestamp,
|
||||
tx_digest,
|
||||
event_index,
|
||||
type,
|
||||
event_address,
|
||||
event_module,
|
||||
event_resource,
|
||||
parsed_json,
|
||||
modified_timestamp
|
||||
FROM
|
||||
{{ ref('core__fact_events') }}
|
||||
WHERE
|
||||
{% if is_incremental() %}
|
||||
modified_timestamp >= (
|
||||
SELECT
|
||||
COALESCE(MAX(modified_timestamp), '1900-01-01'::TIMESTAMP) AS modified_timestamp
|
||||
FROM
|
||||
{{ this }}
|
||||
)
|
||||
AND
|
||||
{% endif %}
|
||||
(
|
||||
type = '0x1eabed72c53feb3805120a081dc15963c204dc8d091542592abaf7a35689b2fb::pool::SwapEvent' -- Cetus
|
||||
OR type = '0x91bfbc386a41afcfd9b2533058d7e915a1d3829089cc268ff4333d54d6339ca1::pool::SwapEvent' -- Turbos
|
||||
OR type = '0x3b6d71bdeb8ce5b06febfd3cfc29ecd60d50da729477c8b8038ecdae34541b91::bluefin::BluefinSwapEvent' -- Bluefin
|
||||
OR type = '0xd675e6d727bb2d63087cc12008bb91e399dc7570100f72051993ec10c0428f4a::events::SwapCompletedEventV2' -- Hop
|
||||
OR type = '0x25929e7f29e0a30eb4e692952ba1b5b65a3a4d65ab5f2a32e1ba3edcb587f26d::pool::Swap' -- FlowX
|
||||
OR type = '0xe8f996ea6ff38c557c253d3b93cfe2ebf393816487266786371aa4532a9229f2::settle::Swap' -- DeepBook
|
||||
)
|
||||
),
|
||||
|
||||
cetus_swaps AS (
|
||||
SELECT
|
||||
checkpoint_number,
|
||||
block_timestamp,
|
||||
tx_digest,
|
||||
event_index,
|
||||
'Cetus' AS platform,
|
||||
event_address AS platform_address,
|
||||
parsed_json:pool::STRING AS pool_address,
|
||||
parsed_json:amount_in::NUMBER AS amount_in_raw,
|
||||
parsed_json:amount_out::NUMBER AS amount_out_raw,
|
||||
parsed_json:atob::BOOLEAN AS a_to_b,
|
||||
parsed_json:fee_amount::NUMBER AS fee_amount_raw,
|
||||
parsed_json:partner::STRING AS partner_address,
|
||||
parsed_json:ref_amount::NUMBER AS referral_amount_raw,
|
||||
parsed_json:steps::NUMBER AS steps,
|
||||
NULL AS token_in_type,
|
||||
NULL AS token_out_type,
|
||||
NULL AS trader_address,
|
||||
modified_timestamp
|
||||
FROM core_events
|
||||
WHERE type = '0x1eabed72c53feb3805120a081dc15963c204dc8d091542592abaf7a35689b2fb::pool::SwapEvent'
|
||||
),
|
||||
|
||||
turbos_swaps AS (
|
||||
SELECT
|
||||
checkpoint_number,
|
||||
block_timestamp,
|
||||
tx_digest,
|
||||
event_index,
|
||||
'Turbos' AS platform,
|
||||
event_address AS platform_address,
|
||||
parsed_json:pool::STRING AS pool_address,
|
||||
CASE
|
||||
WHEN parsed_json:a_to_b::BOOLEAN = TRUE THEN parsed_json:amount_a::NUMBER
|
||||
ELSE parsed_json:amount_b::NUMBER
|
||||
END AS amount_in_raw,
|
||||
CASE
|
||||
WHEN parsed_json:a_to_b::BOOLEAN = TRUE THEN parsed_json:amount_b::NUMBER
|
||||
ELSE parsed_json:amount_a::NUMBER
|
||||
END AS amount_out_raw,
|
||||
parsed_json:a_to_b::BOOLEAN AS a_to_b,
|
||||
parsed_json:fee_amount::NUMBER AS fee_amount_raw,
|
||||
NULL AS partner_address,
|
||||
0 AS referral_amount_raw,
|
||||
1 AS steps,
|
||||
NULL AS token_in_type,
|
||||
NULL AS token_out_type,
|
||||
parsed_json:recipient::STRING AS trader_address,
|
||||
modified_timestamp
|
||||
FROM core_events
|
||||
WHERE type = '0x91bfbc386a41afcfd9b2533058d7e915a1d3829089cc268ff4333d54d6339ca1::pool::SwapEvent'
|
||||
),
|
||||
|
||||
bluefin_swaps AS (
|
||||
SELECT
|
||||
checkpoint_number,
|
||||
block_timestamp,
|
||||
tx_digest,
|
||||
event_index,
|
||||
'Bluefin' AS platform,
|
||||
event_address AS platform_address,
|
||||
NULL AS pool_address,
|
||||
parsed_json:amount_in::NUMBER AS amount_in_raw,
|
||||
parsed_json:amount_out::NUMBER AS amount_out_raw,
|
||||
NULL AS a_to_b,
|
||||
0 AS fee_amount_raw,
|
||||
NULL AS partner_address,
|
||||
0 AS referral_amount_raw,
|
||||
1 AS steps,
|
||||
NULL AS token_in_type,
|
||||
NULL AS token_out_type,
|
||||
NULL AS trader_address,
|
||||
modified_timestamp
|
||||
FROM core_events
|
||||
WHERE type = '0x3b6d71bdeb8ce5b06febfd3cfc29ecd60d50da729477c8b8038ecdae34541b91::bluefin::BluefinSwapEvent'
|
||||
),
|
||||
|
||||
hop_aggregator_swaps AS (
|
||||
SELECT
|
||||
checkpoint_number,
|
||||
block_timestamp,
|
||||
tx_digest,
|
||||
event_index,
|
||||
'Hop' AS platform,
|
||||
event_address AS platform_address,
|
||||
NULL AS pool_address,
|
||||
parsed_json:amount_in::NUMBER AS amount_in_raw,
|
||||
parsed_json:amount_out::NUMBER AS amount_out_raw,
|
||||
NULL AS a_to_b,
|
||||
parsed_json:router_fee::NUMBER AS fee_amount_raw,
|
||||
NULL AS partner_address,
|
||||
0 AS referral_amount_raw,
|
||||
1 AS steps,
|
||||
parsed_json:type_in::STRING AS token_in_type,
|
||||
parsed_json:type_out::STRING AS token_out_type,
|
||||
parsed_json:swapper::STRING AS trader_address,
|
||||
modified_timestamp
|
||||
FROM core_events
|
||||
WHERE type = '0xd675e6d727bb2d63087cc12008bb91e399dc7570100f72051993ec10c0428f4a::events::SwapCompletedEventV2'
|
||||
),
|
||||
|
||||
flowx_swaps AS (
|
||||
SELECT
|
||||
checkpoint_number,
|
||||
block_timestamp,
|
||||
tx_digest,
|
||||
event_index,
|
||||
'FlowX' AS platform,
|
||||
event_address AS platform_address,
|
||||
parsed_json:pool_id::STRING AS pool_address,
|
||||
parsed_json:amount_in::NUMBER AS amount_in_raw,
|
||||
parsed_json:amount_out::NUMBER AS amount_out_raw,
|
||||
NULL AS a_to_b,
|
||||
0 AS fee_amount_raw,
|
||||
NULL AS partner_address,
|
||||
0 AS referral_amount_raw,
|
||||
1 AS steps,
|
||||
NULL AS token_in_type,
|
||||
NULL AS token_out_type,
|
||||
NULL AS trader_address,
|
||||
modified_timestamp
|
||||
FROM core_events
|
||||
WHERE type = '0x25929e7f29e0a30eb4e692952ba1b5b65a3a4d65ab5f2a32e1ba3edcb587f26d::pool::Swap'
|
||||
),
|
||||
|
||||
deepbook_swaps AS (
|
||||
SELECT
|
||||
checkpoint_number,
|
||||
block_timestamp,
|
||||
tx_digest,
|
||||
event_index,
|
||||
'DeepBook' AS platform,
|
||||
event_address AS platform_address,
|
||||
NULL AS pool_address,
|
||||
parsed_json:amount_in::NUMBER AS amount_in_raw,
|
||||
parsed_json:amount_out::NUMBER AS amount_out_raw,
|
||||
NULL AS a_to_b,
|
||||
0 AS fee_amount_raw,
|
||||
NULL AS partner_address,
|
||||
0 AS referral_amount_raw,
|
||||
1 AS steps,
|
||||
NULL AS token_in_type,
|
||||
NULL AS token_out_type,
|
||||
NULL AS trader_address,
|
||||
modified_timestamp
|
||||
FROM core_events
|
||||
WHERE type = '0xe8f996ea6ff38c557c253d3b93cfe2ebf393816487266786371aa4532a9229f2::settle::Swap'
|
||||
),
|
||||
|
||||
all_swaps AS (
|
||||
SELECT * FROM cetus_swaps
|
||||
UNION ALL
|
||||
SELECT * FROM turbos_swaps
|
||||
UNION ALL
|
||||
SELECT * FROM bluefin_swaps
|
||||
UNION ALL
|
||||
SELECT * FROM hop_aggregator_swaps
|
||||
UNION ALL
|
||||
SELECT * FROM flowx_swaps
|
||||
UNION ALL
|
||||
SELECT * FROM deepbook_swaps
|
||||
)
|
||||
|
||||
SELECT
|
||||
checkpoint_number,
|
||||
block_timestamp,
|
||||
tx_digest,
|
||||
event_index,
|
||||
platform,
|
||||
platform_address,
|
||||
pool_address,
|
||||
amount_in_raw,
|
||||
amount_out_raw,
|
||||
a_to_b,
|
||||
fee_amount_raw,
|
||||
partner_address,
|
||||
referral_amount_raw,
|
||||
steps,
|
||||
token_in_type,
|
||||
token_out_type,
|
||||
trader_address,
|
||||
{{ dbt_utils.generate_surrogate_key(['tx_digest', 'event_index']) }} AS dex_swaps_id,
|
||||
SYSDATE() AS inserted_timestamp,
|
||||
modified_timestamp,
|
||||
'{{ invocation_id }}' AS _invocation_id
|
||||
FROM
|
||||
all_swaps
|
||||
Loading…
Reference in New Issue
Block a user