tarikceric
1a0e0c5049
remove pyth workflow ( #873 )
2025-09-12 11:13:58 -07:00
tarikceric
818fd59939
An 6361/cte logic txs and idls ( #851 )
...
* update tx's logic
* set active col in idls
* same run and tag for idl tables
* fix tag name
2025-06-23 11:30:33 -07:00
tarikceric
13d0344b3c
update partition filter ( #843 )
2025-05-20 10:52:47 -07:00
desmond-hui
979e42cb98
tmp add partition date
2025-02-13 15:55:58 -08:00
desmond-hui
165e50bb5c
hotfix auth
2025-02-13 14:24:46 -08:00
desmond-hui
3c229e32ee
fill in error txs ( #802 )
2025-02-12 08:33:42 -08:00
desmond-hui
e0772708bf
refactor using events_inner, static program list ( #799 )
2025-02-12 07:08:56 -08:00
desmond-hui
9a2112f7ae
Refactor decoded instructions realtime 3 ( #796 )
...
* use events_inner, formatting
* get IDLs as static list, remove JOIN
2025-02-10 09:45:42 -08:00
desmond-hui
e16b72085a
change partition key to 15min intervals instead of daily ( #778 )
2025-01-23 09:37:53 -08:00
desmond-hui
6dd339c802
Backfill solana tx index pipeline ( #773 )
...
* wip
* wip
* add bronze tables and streamline source
* add silver table to store backfilled data
* adjust streamline request sizes
* update cluster keys
* make sure bronze data is not null
* update backfill workflow
* fix incremental
* update job frequency
2025-01-21 09:10:01 -08:00
desmond-hui
a4b27bb6a4
add solscan discrepancy retries ( #745 )
2025-01-02 07:40:24 -08:00
desmond-hui
cfcb06c56f
switch to ankr as provider for tx counts ( #736 )
2024-12-11 16:46:48 -08:00
desmond-hui
af81bfb6d9
update partition start min ( #734 )
2024-12-11 16:18:55 -08:00
desmond-hui
767a92a49b
Deprecate legacy decode logs pipeline - attempt #2 ( #731 )
...
* call streamline 2.0 endpoints
* update request query
* update to use streamline 2.0 data after cutoff
* formatting
* add larger lookback for first run, revert after
* fix comments
* get fields from new data structure, add coalesce for backwards compat
* update cutoff time
2024-12-11 14:38:46 -08:00
desmond-hui
4102bf2b73
revert decoded logs deploy
2024-12-09 10:07:48 -08:00
desmond-hui
420128b2a1
Deprecate legacy decode logs pipeline ( #726 )
...
* call streamline 2.0 endpoints
* update request query
* update to use streamline 2.0 data after cutoff
* formatting
* use new complete tables for backfill macros
* add larger lookback for first run, revert after
* fix comments
2024-12-09 09:13:47 -08:00
desmond-hui
8ea30e9285
Deprecate legacy decode instructions pipeline ( #723 )
...
* update requests model
* only call streamline 2.0 endpoint
* use streamline 2.0 data after cutoff, formatting
* fix id column reference
* remove FR and add comment
* update helper to use new completed table
* update cutoff date, extend lookback
2024-12-09 08:05:34 -08:00
desmond-hui
3ce9fe139b
hotfix, try optimize block txs 2 workflow
2024-12-09 07:49:05 -08:00
desmond-hui
ef07bd37f5
Add backup block txs count pipeline ( #729 )
...
* add backup pipeline to get tx counts from helius
* use helius if solscan not available
* temporarily replace solscan call w/ helius call
2024-12-06 09:29:07 -08:00
desmond-hui
71c1d09787
update cutoff blocks for block txs 2 pipeline
2024-12-06 09:28:48 -08:00
desmond-hui
f69c126030
Migrate block txs to streamline 2.0 ( #727 )
...
* add views for block txs streamline 2.0 data
* streamline 2.0 requests for block txs
* fix view macro to allow for no other cols
* add top level block timestamp, adjust batch size
* add workflow for pipeline
* add block tx 2 to gha schedule, reorder records
2024-12-06 07:53:50 -08:00
desmond-hui
7c4d296081
hotfix table name
2024-12-02 07:55:53 -08:00
desmond-hui
935ca3c728
Migrate solana decoder to streamline 2 ( #711 )
...
* udf to call decoder on streamline 2.0
* bronze instruction decoder models
* add more v2 apis
* add bronze models for streamline 2.0 decoded logs
* add streamline models for 2.0 decoded instructions & logs
* run streamline 2.0 real time decoding pipelines in parallel
2024-11-25 08:37:24 -08:00
desmond-hui
469db641b1
Migrate solscan token metadata to solscan v2 ( #717 )
...
* get solscan token list using v2 api
* bronze models to retrieve token list data
* get token list once per week
* fix workflow name
2024-11-25 08:20:52 -08:00
desmond-hui
8bbc0801fb
Migrate solscan blocks to solscan v2 ( #713 )
...
* use solscan v2.0 api
* delete unused solscan legacy models (#712 )
* reget failed requests
2024-11-20 14:24:51 -08:00
desmond-hui
55fef95602
if data is null, don't mark it as complete ( #708 )
2024-11-14 09:05:43 -08:00
desmond-hui
9d6cd7ec59
Fix table not found error for streamline block txs ( #709 )
...
* materialize the test output as a dbt modeled table
* reference the model instead of test output table
2024-11-14 06:51:52 -08:00
desmond-hui
b74ff15fd6
Migrate get solscan blocks to 2.0 ( #685 )
...
* streamline requests for solscan blocks
* bronze views for solscan streamline data
* workflow schedule for solscan streamline requests
* add todo comment
* update uri
* update the silver table before requesting the next set of blocks
* swap to using 2.0 data
* update TODO
* change vault path
2024-11-05 09:53:04 -08:00
desmond-hui
3e084698ed
Migrate validator list to streamline 2.0 ( #691 )
...
* streamliine 2.0 request
* add bronze tables and workflow
* fix workflow name for program accounts
* stagger slightly
2024-11-04 09:57:11 -08:00
desmond-hui
e44589038b
Migrate validator rank metadata ( #692 )
...
* add streamline 2.0 request for validator metadata
* add bronze models
* add workflow
2024-11-04 09:43:03 -08:00
desmond-hui
180f5a28df
Migrate validator vote program accounts ( #681 )
...
* streamline requests
* bronze models
* add workflow
2024-10-31 07:11:55 -07:00
desmond-hui
d3688507b3
Migrate validator vote accounts to streamline 2.0 ( #680 )
...
* streamline vote accounts requests
* bronze models
* add workflow
2024-10-31 07:09:41 -07:00
desmond-hui
eb42fa2814
Migrate blocks to streamline 2.0 ( #675 )
...
* bronze streamline blocks
* complete blocks and streamline requests
* add workflow
* revert
* update cutoff
2024-10-16 09:38:58 -07:00
desmond-hui
bbbae51728
partition by hour ( #677 )
...
* partition by hour
* update partition function
2024-10-16 08:01:18 -07:00
desmond-hui
122fa61bc9
An 5047/helius cnft metadata ( #669 )
...
* streamline bronze data models
* wip complete and requests nft metadata models
* add source
* formatting
* update columns and explode key
* fix partition key
* update requests logic
* rename to cnft
* run cnft metadata every 10 minutes
* add SO
* formatting
2024-10-15 09:21:52 -07:00
desmond-hui
0d32c421e1
deprecate rewards models built on streamline 1.0 raw data ( #668 )
...
* deprecate rewards models built on streamline 1.0 raw data
* remove streamline 1.0 block rewards udf and tasks
* disable streamline 1.0 block rewards request views
* disable complete block rewards, change ref to source
2024-10-08 09:43:16 -07:00
desmond-hui
e76e28e9ff
hotfix order by
2024-10-03 07:35:39 -07:00
desmond-hui
a4c3eaab4f
update cutover values
2024-09-27 09:37:30 -07:00
desmond-hui
0f59a077ac
Rewards streamline 2 ( #660 )
...
* wip
* wip
* tmp change to get it to work
* fix service var, change producer values
* wip
* add source and bronze models
* wip rewards
* wip
* wip
* wip
* adjust partition key logic for block rewards
* add dbt depends_on
* compare to completed
* add workflow for calling streamline 2.0 block rewards
* make and use v2 streamline external table macro
* rename and adjust starting block/partition
* update workflow
* match existing schedule
2024-09-27 09:31:33 -07:00
tarikceric
9b9b408745
add jupv4 to realtime logs ( #651 )
2024-09-10 14:51:05 -07:00
desmond-hui
a276c93be1
Stake program accounts streamline v2 ( #652 )
...
* rest api v2 external function
* bronze streamline models
* streamline request
* change bronze partition key
* update other cols
* remove explode key
* remove comment
* change name to match pipeline
* add to workflow
2024-09-10 08:16:32 -07:00
desmond-hui
e05dd1baa9
add layerzero to realtime log decoding: ( #646 )
2024-08-27 09:41:24 -07:00
desmond-hui
fb2fdf2a51
add dca program ( #640 )
2024-08-22 06:43:26 -07:00
desmond-hui
f2ba9fb086
Auto heal block rewards and test ( #639 )
...
* add test for missing streamline rewards
* self re-request missing streamline rewards
2024-08-20 07:21:37 -07:00
tarikceric
cbf4458f8f
add pumpfun to realtime ( #633 )
2024-08-13 12:18:27 -07:00
desmond-hui
9562fd5ace
Stake account states ( #627 )
...
* add streamline stake accounts states
* add job to run snapshot
2024-08-12 09:29:29 -07:00
desmond-hui
5fed2a6d55
exclude pumpfun logs from decode instruction request ( #629 )
2024-08-05 12:47:21 -07:00
desmond-hui
4dcc84d281
Decode phoenix logs ( #603 )
...
* remove phoenix logs from instruction decode requests
* add phoenix logs to realtime logs decoder requests
2024-07-23 09:23:27 -07:00
desmond-hui
4ae84eda4f
add metadata to decode log requests ( #607 )
2024-07-16 08:50:49 -07:00
desmond-hui
8272ba8601
request log decode for jup6 swaps called by a diff program than jup6 ( #589 )
...
* request log decode for jup6 swaps called by a diff program than jup6
* formatting
2024-06-20 06:28:19 -07:00
