Eric Laurello
fdffc38cdf
rm temp filters
2025-10-10 16:39:33 -04:00
Eric Laurello
89fcf827e6
Replace hard-coded streamline database references with dbt sources
...
Update bronze backfill models to use source() function instead of direct
table references. This improves maintainability and allows for environment-
specific configuration through the existing bronze_streamline source definition.
Changes:
- Add checkpoints_backfill, transactions_backfill, events_backfill to sources.yml
- Update all bronze backfill models to use {{ source('bronze_streamline', '...') }}
🤖 Generated with [Claude Code](https://claude.com/claude-code )
Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-10 16:27:30 -04:00
Eric Laurello
8f7e87b71a
Implement parallel backfill pipeline for historical Sui data
...
This change introduces a separate backfill data pipeline to ingest historical
Sui blockchain data from an alternative source while preserving the existing
real-time ingestion pipeline.
Architecture:
- Bronze layer: View-based models pointing to streamline.sui backfill tables
- Silver layer: Table-materialized transformations with backfill-specific JSON parsing
- Gold layer: UNION ALL queries combining real-time and backfill data streams
The backfill pipeline handles different JSON structures from the alternative
data source, particularly for transaction and checkpoint parsing. Gold models
now merge both data streams without modifying the incremental processing logic
for the primary real-time pipeline.
Current scope: Epoch 628, checkpoints < 96605300
🤖 Generated with [Claude Code](https://claude.com/claude-code )
Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-10 16:19:13 -04:00
Eric Laurello
5dcec2b197
prep data for gold, no combined silver table
2025-10-10 11:27:14 -04:00
tarikceric
a57fdd80d3
temp backfill tables
2025-10-08 15:33:08 -07:00
Jack Forgash
2b219238ca
Merge pull request #16 from FlipsideCrypto/quickfix-surrogate-key
...
docs_update / run_dbt_jobs (push) Has been cancelled
docs_update / notify-failure (push) Has been cancelled
upd surrogate key
2025-08-25 13:10:53 -04:00
Jack Forgash
43ab009c87
simplify surrogate key
2025-08-25 10:17:36 -06:00
Jack Forgash
0fba387c66
upd surrogate key
2025-08-25 08:51:11 -06:00
Jack Forgash
b9193f7470
Merge pull request #8 from FlipsideCrypto/AN-6430-DEFI-forg
...
AN-6430/dex swaps
2025-08-22 17:21:35 -04:00
Jack Forgash
2339db84bf
fixes
2025-08-22 17:21:27 -04:00
Jack Forgash
665ff0cdb8
rm swaps with type in txs
2025-08-20 14:21:28 -04:00
Mike Stepanovic
9a9e8228e5
Merge pull request #15 from FlipsideCrypto/ez-bridge
...
fix amount_usd calculation
2025-08-19 09:11:13 -06:00
Mike Stepanovic
c6320ff382
fix amount_usd calculation
2025-08-19 09:01:32 -06:00
Jack Forgash
0eef0d7220
upd incr logic, temp excl token type from tx
2025-08-14 13:41:22 -04:00
Jack Forgash
29940a9608
merge native Sui price joins
2025-08-14 10:29:06 -04:00
Jack Forgash
832c98622a
ez_dex_swaps join price on token type
2025-08-14 10:21:25 -04:00
Jack Forgash
6cbe9555f6
fix price join, rm dev 30d restriction
2025-08-14 10:15:09 -04:00
Jack Forgash
8db8378dfd
upd silver docs
2025-08-12 17:22:34 -04:00
Jack Forgash
5fe9b46c6f
excl limit orders
2025-08-12 17:04:29 -04:00
Jack Forgash
ca3befce2b
simplify aftermath model
2025-08-12 16:36:18 -04:00
Jack Forgash
efd3f3b718
split out aftermath
2025-08-12 16:00:57 -04:00
Jack Forgash
c1cd794e16
upd swap index groupings
2025-08-12 15:17:32 -04:00
Jack Forgash
349ff9a663
better handle redundant SwapEvent modules, use price tbls
2025-08-12 14:49:41 -04:00
Eric Laurello
03bc9f29fd
daily
2025-08-11 16:27:07 -04:00
Eric Laurello
5ee320ebc3
Merge branch 'main' into AN-6430-DEFI-forg
2025-08-08 11:24:29 -04:00
Eric Laurello
e244988c41
.
2025-08-08 10:10:38 -04:00
Eric Laurello
a600de5540
update chains
2025-08-08 10:09:45 -04:00
Jack Forgash
18275170d5
Haedal event handling
2025-08-07 16:33:50 -04:00
Jack Forgash
f0e9be1dc7
big refactor - identify swaps by event_resource
2025-08-07 16:18:03 -04:00
eric-laurello
c1db808a7d
Merge pull request #14 from FlipsideCrypto/ez-bridge
...
Bridging to gold
2025-08-07 13:52:10 -04:00
Mike Stepanovic
895e245f09
fixed a typo and enhanced table description for MCP
2025-08-07 11:36:26 -06:00
Eric Laurello
8eb1a498a1
better split
2025-08-07 13:36:21 -04:00
Mike Stepanovic
87f5232ec8
fixed typo
2025-08-07 11:33:20 -06:00
Eric Laurello
6fab7af044
updates
2025-08-07 12:43:04 -04:00
Eric Laurello
eaafed1b0f
rerun for nulls
docs_update / run_dbt_jobs (push) Has been cancelled
docs_update / notify-failure (push) Has been cancelled
dbt_run_incremental_core / run_dbt_jobs (push) Has been cancelled
dbt_run_incremental_non_core / run_dbt_jobs (push) Has been cancelled
dbt_run_streamline_realtime_tx / run_dbt_jobs (push) Has been cancelled
dbt_run_coin_info_backfill / run_dbt_jobs (push) Has been cancelled
dbt_run_streamline_realtime / run_dbt_jobs (push) Has been cancelled
dbt_run_incremental_core / notify-failure (push) Has been cancelled
dbt_run_incremental_non_core / notify-failure (push) Has been cancelled
2025-08-05 16:13:05 -04:00
eric-laurello
f7005e6326
Merge pull request #13 from FlipsideCrypto/EZ-transfers
...
docs_update / run_dbt_jobs (push) Has been cancelled
docs_update / notify-failure (push) Has been cancelled
EZ transfers
2025-07-31 15:04:28 -04:00
Eric Laurello
0cfc3393a2
EZ transfers
2025-07-31 12:19:50 -04:00
Eric Laurello
c17a12f5f4
add overview
2025-07-31 09:55:15 -04:00
eric-laurello
1d941b001a
Merge pull request #7 from FlipsideCrypto/prices
...
Add prices
2025-07-31 09:36:55 -04:00
eric-laurello
8a594ffd2c
Merge pull request #12 from FlipsideCrypto/birdge
...
docs_update / run_dbt_jobs (push) Waiting to run
docs_update / notify-failure (push) Blocked by required conditions
Bridging
2025-07-30 13:28:43 -04:00
Eric Laurello
1941088268
min_bd
2025-07-30 13:22:01 -04:00
Eric Laurello
65a55a951c
add non core wf
2025-07-30 13:15:54 -04:00
Eric Laurello
a8131dea42
rm lower, add case for native
2025-07-30 10:41:09 -04:00
Jack Forgash
d70143106d
direct dex swaps
2025-07-29 17:28:52 -06:00
Jack Forgash
a29f048bd7
expand direct dex coverage
2025-07-29 17:21:34 -06:00
Jack Forgash
09a466e26d
add event_resource filter
2025-07-29 17:08:55 -06:00
Jack Forgash
976741233e
direct dex model + gold
2025-07-29 17:06:43 -06:00
Jack Forgash
9e907a1b1f
silver model -> direct volume. split out router vol. upd token join in gold
2025-07-29 15:56:21 -06:00
Eric Laurello
d5873beb8a
clean
2025-07-29 16:35:03 -04:00
Eric Laurello
83e00c62a6
cols
2025-07-29 16:34:20 -04:00
