This change introduces a separate backfill data pipeline to ingest historical
Sui blockchain data from an alternative source while preserving the existing
real-time ingestion pipeline.
Architecture:
- Bronze layer: View-based models pointing to streamline.sui backfill tables
- Silver layer: Table-materialized transformations with backfill-specific JSON parsing
- Gold layer: UNION ALL queries combining real-time and backfill data streams
The backfill pipeline handles different JSON structures from the alternative
data source, particularly for transaction and checkpoint parsing. Gold models
now merge both data streams without modifying the incremental processing logic
for the primary real-time pipeline.
Current scope: Epoch 628, checkpoints < 96605300
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
- Add transaction_module filters for Cetus (cetus) and Turbos (turbos)
- Extract Universal Router swaps into separate model for routed swap analysis
- Remove Universal Router from main dex_swaps model to eliminate multi-token parsing issues
- Add comprehensive router analysis documentation
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
new file: models/bronze/prices/bronze__complete_token_asset_metadata.sql
new file: models/bronze/prices/bronze__complete_token_prices.sql
modified: models/descriptions/columns.md
new file: models/descriptions/prices.md
new file: models/gold/prices/price__ez_asset_metadata.sql
new file: models/gold/prices/price__ez_asset_metadata.yml
new file: models/gold/prices/price__ez_prices_hourly.sql
new file: models/gold/prices/price__ez_prices_hourly.yml
new file: models/silver/prices/silver__complete_token_asset_metadata.sql
new file: models/silver/prices/silver__complete_token_asset_metadata.yml
new file: models/silver/prices/silver__complete_token_prices.sql
new file: models/silver/prices/silver__complete_token_prices.yml
modified: models/sources.yml
