tarikceric
dd87a5a4d3
move stake accts snapshot to gha workflow file ( #826 )
...
* move stake accts snapshot to gha workflow file
* cleanup
2025-04-10 13:41:32 -07:00
tarikceric
b03cce6a4c
extra core schedule to fill gap ( #817 )
...
* extra core schedule to fill gap
* run name
2025-03-07 13:01:52 -08:00
tarikceric
75a16253d9
hourly schedule for ez dex swaps and upstream ( #815 )
...
* hourly schedule for ez dex swaps and upstream
* hourly schedule for ez dex swaps and upstream
* cleanup
2025-03-06 08:59:36 -08:00
tarikceric
94b5f2942c
Update/core noncore schedule ( #805 )
...
* update core and noncore runs
* run votes with core
2025-02-14 09:27:12 -08:00
desmond-hui
ca55076476
stage block txs data for ingestion to silver ( #781 )
2025-01-28 09:45:21 -08:00
tarikceric
ccd085163c
remove cnft backfill ( #749 )
...
* remove cnft backfill
* add comment
2025-01-07 09:37:43 -08:00
tarikceric
40c3f1688e
remove tx units backfill job ( #742 )
2024-12-20 10:49:39 -08:00
desmond-hui
f69c126030
Migrate block txs to streamline 2.0 ( #727 )
...
* add views for block txs streamline 2.0 data
* streamline 2.0 requests for block txs
* fix view macro to allow for no other cols
* add top level block timestamp, adjust batch size
* add workflow for pipeline
* add block tx 2 to gha schedule, reorder records
2024-12-06 07:53:50 -08:00
tarikceric
7b139acfba
increase batch size; run interval ( #702 )
2024-11-08 09:45:09 -08:00
tarikceric
5688bce773
Update/cnft backfill helper ( #697 )
...
* update cnft backfill helper
* add cnft backfill
* revert index, add filter
* run 4x hour
2024-11-05 12:19:29 -08:00
desmond-hui
b74ff15fd6
Migrate get solscan blocks to 2.0 ( #685 )
...
* streamline requests for solscan blocks
* bronze views for solscan streamline data
* workflow schedule for solscan streamline requests
* add todo comment
* update uri
* update the silver table before requesting the next set of blocks
* swap to using 2.0 data
* update TODO
* change vault path
2024-11-05 09:53:04 -08:00
tarikceric
a4da0f567e
model and workflow ( #686 )
...
* model and workflow
* name, filter large partitions, max partition
* formatting
2024-11-01 09:32:02 -07:00
desmond-hui
eb42fa2814
Migrate blocks to streamline 2.0 ( #675 )
...
* bronze streamline blocks
* complete blocks and streamline requests
* add workflow
* revert
* update cutoff
2024-10-16 09:38:58 -07:00
desmond-hui
122fa61bc9
An 5047/helius cnft metadata ( #669 )
...
* streamline bronze data models
* wip complete and requests nft metadata models
* add source
* formatting
* update columns and explode key
* fix partition key
* update requests logic
* rename to cnft
* run cnft metadata every 10 minutes
* add SO
* formatting
2024-10-15 09:21:52 -07:00
desmond-hui
0f59a077ac
Rewards streamline 2 ( #660 )
...
* wip
* wip
* tmp change to get it to work
* fix service var, change producer values
* wip
* add source and bronze models
* wip rewards
* wip
* wip
* wip
* adjust partition key logic for block rewards
* add dbt depends_on
* compare to completed
* add workflow for calling streamline 2.0 block rewards
* make and use v2 streamline external table macro
* rename and adjust starting block/partition
* update workflow
* match existing schedule
2024-09-27 09:31:33 -07:00
tarikceric
ba915cf20b
temp exclude events_inner test; remove workflow ( #661 )
...
* temp exclude events_inner test; remove workflow
* reduce unique test to 7 days
2024-09-26 12:35:32 -07:00
tarikceric
b22caeab39
An 4528/events inner ( #655 )
...
* initial silver models
* create gold tables
* backfill
* incremental backfill runs
* update run interval
* update tests
* removed filter
* updated per pr comments
* update backfill and cluster/so
* update gold inc filter
* backfill updates
* clean up
* disable fr and update backfill date
* update backfill logic
* correct filter date
2024-09-19 10:53:14 -07:00
desmond-hui
401c31ab26
remove transactions backfill from github workflow
2024-08-06 12:43:35 -07:00
desmond-hui
8b33b55737
Backfill txs with vote txs with non vote instructions ( #619 )
...
* add model to identify txs to be backfilled
* add workflow to run backfill
* add comment
2024-07-25 06:34:35 -07:00
desmond-hui
dda387f040
realtime scheduling for decoding logs ( #579 )
2024-06-12 09:04:31 -07:00
Desmond Hui
319b76b669
remove extra double quote
2024-04-29 07:10:08 -07:00
desmond-hui
9d30ea2e3f
remove the cnft backfill task ( #546 )
2024-04-29 07:04:43 -07:00
tarikceric
1cef5247b9
An 4769/me solsniper cnft sales ( #540 )
...
* wip
* wip
* get compressed mint
* add schedule, pr comments update
* temp backfill logic
* backfill only
2024-04-25 10:41:45 -07:00
desmond-hui
7eda8c5cd2
remove cnft backfill from schedule, it is completed ( #532 )
2024-04-12 09:32:38 -07:00
Desmond Hui
9886ee4654
update cnft backfill scheduler
2024-02-06 06:39:20 -08:00
desmond-hui
92c57a4e86
An 4435/compressed mints backfill helpers ( #475 )
...
* wip
* make the DMLs ACID
* add backfill job, rename existing workflow
* check there is data to backfill before continuing
2024-02-05 07:16:02 -08:00
desmond-hui
bc9637a79b
use orchestrator to trigger decoded realtime and backfill ( #466 )
2024-01-26 07:51:28 -08:00
desmond-hui
68174b45cc
test async workflow orchestration ( #461 )
...
* test async workflow orchestration
* update
2024-01-25 08:17:11 -08:00
desmond-hui
bd23649b00
An 4429/setup pyth backfill ( #460 )
...
* setup pyth backfill
* wip
* remove dev file
* adjust backfill scheduler, add more runs of complete decoded
* adjust backfill rane
2024-01-24 07:49:59 -08:00
Desmond Hui
cc14acf494
add decoder backfill gha workflow
2024-01-04 09:21:57 -08:00
desmond-hui
73a2d32f29
An 4323/initial streamline 2.0 setup ( #430 )
...
* wip
* change udf name, fix surrogate key for inner instruct calls
* (wip) decoded instructions backfill helpers
* complete backfill helpers, clean up views with 0 requests
* rename
* revert
* temp change will revert
* rename
* use streamline 2.0 verify idl api endpoint
* this is placeholder model, will replace existing when ready
* reorg files
* add workflow for decode instructions real time
* use pip cache
* update with prod endpoints
* update sql limit to real value, should be union all
* use ref for model names
* remove placeholder, replace existing model logic
* add new decoder flow to gha
* temp disable incremental
* add inner index to core tables
* match existing behavior until we properly incorporate inner_instruction txs
* temp disable
2024-01-03 11:40:57 -08:00
desmond-hui
8bf643dcd2
make core run as part of non-core ( #407 )
2023-11-17 07:40:05 -08:00
desmond-hui
f8e0ca0e51
make it 15mins instead of every 10 ( #403 )
2023-11-15 11:27:56 -08:00
desmond-hui
d32599e851
An 4090/force paralell execution ( #402 )
...
* force paralell execution for each udf request
* increase number of commands per workflow, decrease schedule frequency
2023-11-15 09:51:03 -08:00
desmond-hui
c567d20d7b
add compressed nft backfill job to SF ( #400 )
2023-11-14 10:16:18 -08:00
tarikceric
95e392d5b1
Update/schedule core tags ( #396 )
...
* wip
* add schedule tags
* use snowflake tasks
---------
Co-authored-by: Eric Laurello <eric.laurello@flipsidecrypto.com>
2023-11-10 15:59:21 -05:00
desmond-hui
c5e10794c9
this should be the filename ( #389 )
2023-11-07 08:38:38 -08:00
desmond-hui
2734d36d7c
Dispatch gha via snowflake ( #388 )
...
* update package
* setup GHA dispatches via Snowflake tasks
* remove unused workflows
* add model qualifier to only run models in solana
* include task status monitoring workflow
2023-11-07 08:17:36 -08:00
