tarikceric
00f87a344e
Update/tx size udf optimization ( #863 )
...
dbt_run_macro_get_block_production / run_dbt_jobs (push) Has been cancelled
dbt_run_idls_history / run_dbt_jobs (push) Has been cancelled
dbt_run_scheduled_daily / run_dbt_jobs (push) Has been cancelled
dbt_test_intraday / run_dbt_jobs (push) Has been cancelled
dbt_run_streamline_validator_vote_program_accounts_snapshot / run_dbt_jobs (push) Has been cancelled
dbt_run_streamline_validator_vote_accounts_snapshot / run_dbt_jobs (push) Has been cancelled
dbt_run_streamline_validator_metadata_snapshot / run_dbt_jobs (push) Has been cancelled
dbt_run_streamline_validators_list_snapshot / run_dbt_jobs (push) Has been cancelled
dbt_test_scheduled_hourly / run_dbt_jobs (push) Has been cancelled
dbt_run_macro_get_compressed_nft / run_dbt_jobs (push) Has been cancelled
dbt_run_helius_metadata / run_dbt_jobs (push) Has been cancelled
dbt_run_batch_backfill / run_dbt_jobs (push) Has been cancelled
docs_update / notify-failure (push) Has been cancelled
dbt_run_macro_get_block_production / notify-failure (push) Has been cancelled
dbt_run_idls_history / notify-failure (push) Has been cancelled
dbt_run_scheduled_daily / notify-failure (push) Has been cancelled
dbt_test_intraday / notify-failure (push) Has been cancelled
dbt_run_streamline_validator_vote_program_accounts_snapshot / notify-failure (push) Has been cancelled
dbt_run_streamline_validator_vote_accounts_snapshot / notify-failure (push) Has been cancelled
dbt_run_streamline_validator_metadata_snapshot / notify-failure (push) Has been cancelled
dbt_run_streamline_validators_list_snapshot / notify-failure (push) Has been cancelled
dbt_test_scheduled_hourly / notify-failure (push) Has been cancelled
dbt_run_macro_get_compressed_nft / notify-failure (push) Has been cancelled
dbt_run_helius_metadata / notify-failure (push) Has been cancelled
dbt_test_scheduled_weekly / run_dbt_jobs (push) Has been cancelled
dbt_run_streamline_solscan_token_list / run_dbt_jobs (push) Has been cancelled
dbt_run_full_observability / run_dbt_jobs (push) Has been cancelled
dbt_test_scheduled_weekly / notify-failure (push) Has been cancelled
dbt_run_streamline_solscan_token_list / notify-failure (push) Has been cancelled
dbt_run_full_observability / notify-failure (push) Has been cancelled
* update tx_size and compute unit udf; update version
* revert temp source
* fix udf param
2025-08-06 11:01:03 -07:00
tarikceric
3e2f9438d3
add ref count test ( #846 )
...
* add ref count test
* add coalesce and full outer join
* add recency and unique test
2025-05-27 10:05:58 -07:00
tarikceric
cfba275882
An 5767/deprecate old decoder and optimizations ( #821 )
...
* deprecate old decoded_instructions_data
* use events_inner and set max ts
* wip
* use events_inner
2025-04-01 10:07:41 -07:00
desmond-hui
31f532a129
only resend if not exist or error in silver for single day retries ( #780 )
2025-01-28 07:10:51 -08:00
desmond-hui
bd52d9032b
Add decoder backfill helper for single dates ( #772 )
...
* backfill macro for all verified IDLs for specified date
* fix checking for completed after single date retry
* add block_id predicate for completed subset
* add retry macro for single date
* make macro names consistent
2025-01-15 14:22:09 -08:00
desmond-hui
63dc6591e4
Migrate compressed mint calls to streamline 2.0 ( #739 )
...
* fix batch size for udf
* update usage to v2
2024-12-17 07:23:12 -08:00
desmond-hui
a3b8cf8458
Deprecate legacy block txs pipeline ( #735 )
...
* move to backfill folder
* remove legacy udfs and calls
* use streamline 2.0 data after cutover block & partition
* update start block to do checks, add streamline 2.0 data
* fix ambiguous reference
* this test isnt used, it is currently ignored in the test workflow
* get partition from streamline 2.0 upstream when available
* add TODOs
* update partition and block cutoffs
2024-12-12 15:40:01 -08:00
desmond-hui
420128b2a1
Deprecate legacy decode logs pipeline ( #726 )
...
* call streamline 2.0 endpoints
* update request query
* update to use streamline 2.0 data after cutoff
* formatting
* use new complete tables for backfill macros
* add larger lookback for first run, revert after
* fix comments
2024-12-09 09:13:47 -08:00
desmond-hui
8ea30e9285
Deprecate legacy decode instructions pipeline ( #723 )
...
* update requests model
* only call streamline 2.0 endpoint
* use streamline 2.0 data after cutoff, formatting
* fix id column reference
* remove FR and add comment
* update helper to use new completed table
* update cutoff date, extend lookback
2024-12-09 08:05:34 -08:00
desmond-hui
f69c126030
Migrate block txs to streamline 2.0 ( #727 )
...
* add views for block txs streamline 2.0 data
* streamline 2.0 requests for block txs
* fix view macro to allow for no other cols
* add top level block timestamp, adjust batch size
* add workflow for pipeline
* add block tx 2 to gha schedule, reorder records
2024-12-06 07:53:50 -08:00
desmond-hui
0caeaca6ae
Migrate solscan token list call to solcan v2 ( #720 )
...
* remove legacy v1 call
* use solscan v2 data moving forward
* cleanup descriptions, columns, tests and formatting
* remove this is never used, remake it if we need in future
2024-11-26 08:59:23 -08:00
desmond-hui
935ca3c728
Migrate solana decoder to streamline 2 ( #711 )
...
* udf to call decoder on streamline 2.0
* bronze instruction decoder models
* add more v2 apis
* add bronze models for streamline 2.0 decoded logs
* add streamline models for 2.0 decoded instructions & logs
* run streamline 2.0 real time decoding pipelines in parallel
2024-11-25 08:37:24 -08:00
tarikceric
7b139acfba
increase batch size; run interval ( #702 )
2024-11-08 09:45:09 -08:00
desmond-hui
5f7d79fc3b
refactor using quicknode and latest liverquery function ( #705 )
...
* refactor using quicknode and latest liverquery function
* allow manual workflow execution
2024-11-08 09:35:29 -08:00
tarikceric
5688bce773
Update/cnft backfill helper ( #697 )
...
* update cnft backfill helper
* add cnft backfill
* revert index, add filter
* run 4x hour
2024-11-05 12:19:29 -08:00
desmond-hui
d12f1cd9fa
Deprecate legacy validator metadata ( #698 )
...
* remove legacy assets
* remove silver model
* use bronze directly in model
2024-11-05 10:31:39 -08:00
desmond-hui
0b5863cbf2
Deprecate legacy validators app list api ( #696 )
...
* remove legacy assets
* use streamline 2.0 data in silver model moving forward
2024-11-05 09:07:42 -08:00
desmond-hui
b962553b98
turn off legacy snowflake tasks that populate bronze.blocks2 ( #690 )
2024-11-04 08:56:05 -08:00
desmond-hui
16224abec9
Deprecate legacy streamline validator vote program accounts ( #688 )
...
* use streamline 2.0 data after cutoff
* remove legacy streamline assets
2024-11-01 08:21:13 -07:00
desmond-hui
fb8b97d0af
Deprecate legacy streamline validator vote accounts ( #687 )
...
* update model with 2.0 data
* remove legacy pipeline assets
* remove sp create call
* remove bronze refresh
2024-11-01 08:04:19 -07:00
desmond-hui
0d32c421e1
deprecate rewards models built on streamline 1.0 raw data ( #668 )
...
* deprecate rewards models built on streamline 1.0 raw data
* remove streamline 1.0 block rewards udf and tasks
* disable streamline 1.0 block rewards request views
* disable complete block rewards, change ref to source
2024-10-08 09:43:16 -07:00
desmond-hui
7a25e4a021
include program logs that are base64 strings ( #666 )
2024-10-07 09:23:29 -07:00
desmond-hui
0f59a077ac
Rewards streamline 2 ( #660 )
...
* wip
* wip
* tmp change to get it to work
* fix service var, change producer values
* wip
* add source and bronze models
* wip rewards
* wip
* wip
* wip
* adjust partition key logic for block rewards
* add dbt depends_on
* compare to completed
* add workflow for calling streamline 2.0 block rewards
* make and use v2 streamline external table macro
* rename and adjust starting block/partition
* update workflow
* match existing schedule
2024-09-27 09:31:33 -07:00
tarikceric
b22caeab39
An 4528/events inner ( #655 )
...
* initial silver models
* create gold tables
* backfill
* incremental backfill runs
* update run interval
* update tests
* removed filter
* updated per pr comments
* update backfill and cluster/so
* update gold inc filter
* backfill updates
* clean up
* disable fr and update backfill date
* update backfill logic
* correct filter date
2024-09-19 10:53:14 -07:00
desmond-hui
a276c93be1
Stake program accounts streamline v2 ( #652 )
...
* rest api v2 external function
* bronze streamline models
* streamline request
* change bronze partition key
* update other cols
* remove explode key
* remove comment
* change name to match pipeline
* add to workflow
2024-09-10 08:16:32 -07:00
tarikceric
059ebdca7e
An 5146/gold tests ( #650 )
...
* add gold missing tx test
* update gold tests
2024-09-06 09:18:18 -07:00
desmond-hui
d37703693b
use pytz timezones in decode backfill macro ( #643 )
2024-08-23 07:13:57 -07:00
tarikceric
595e30ac09
handle null log_messages ( #614 )
2024-07-22 13:28:29 -07:00
desmond-hui
4ae84eda4f
add metadata to decode log requests ( #607 )
2024-07-16 08:50:49 -07:00
desmond-hui
ed8db95341
spread work across more workers ( #602 )
2024-07-12 08:28:13 -07:00
desmond-hui
5d439b06a6
reset child index when a new top level program is invoked ( #597 )
2024-07-05 07:13:20 -07:00
desmond-hui
8272ba8601
request log decode for jup6 swaps called by a diff program than jup6 ( #589 )
...
* request log decode for jup6 swaps called by a diff program than jup6
* formatting
2024-06-20 06:28:19 -07:00
desmond-hui
3258b155cd
Streamline decoded logs backfill ( #580 )
...
* add completed backfill table
* add helper macros for decoded log backfills
* add decoded log backfills to scheduler
2024-06-13 06:52:18 -07:00
desmond-hui
fce38d5057
logs hotfix
2024-06-12 07:19:39 -07:00
desmond-hui
182b6f29ab
Add b58 decoding to logs extraction udf ( #576 )
...
* add _udf_error col to schema
* handle logs that end in success but is not end of program log
* handle logs truncated as last or not last msg
* add dependency descriptor
* handle legacy BPF log format
* filter out txs with null log_messages
* handle legacy upgrade logs
* ignore single length logs
* add expect _udf_error to be null test
2024-06-12 07:02:56 -07:00
desmond-hui
9a927a2d44
Encoded program data logs table ( #570 )
...
* udf to grab program data logs from log messages
* output table with encoded program data in tx logs
* add comments
* add event_type, index, inner_index and refactor existing
* handle additional nested calls
* add columns, add to SO
* use backfill macro
* handle non-contiguous program log messaging
2024-06-10 07:26:24 -07:00
desmond-hui
4860722d32
Token account owners v2 ( #569 )
...
* new SCD logic for token owners
* test workflow
* test workflow
* test workflow
* fix account ownership recording change event when no change event
* update logic to run off intermediate upstream, modify merge strategy
* reprocess out of order events, get last known state of first reprocessed record
* temp look at prod table hardcoded
* add end_block_id to cluster key
* add predicate
* try with diff cluster key and simple min value predicate
* delete test workflow
* remove hardcoded values, add check for if incremental has 0 rows
* add schema and tests
* cleanup and formatting, remove hardcoded value
2024-06-04 07:05:55 -07:00
desmond-hui
2865420293
reduce size per execution ( #562 )
2024-05-15 08:26:02 -07:00
Desmond Hui
d514c10a89
revert SO macro if execute change
2024-04-24 07:31:37 -07:00
Desmond Hui
fe092eea99
hotfix macro to compile only on execution
2024-04-24 07:28:14 -07:00
desmond-hui
6ab9324ea9
add lpad zeros to view name, fix set formatting ( #527 )
...
* add lpad zeros to view name, fix set formatting
* allow setting of custom priority value
2024-04-11 07:15:03 -07:00
desmond-hui
cd9c4ba9b0
An 4546/allow parameterized retries for decoder errs ( #493 )
...
* add retry helper macro for decoder backfill reqs
* update comment
2024-02-20 06:48:36 -08:00
desmond-hui
ed66de34ec
drop dev SO for transfers and jupv4 ( #490 )
2024-02-15 10:05:15 -08:00
desmond-hui
02bacfa997
An 4508/integrate fsc utils dynamic predicate macro ( #482 )
...
* wip
* refactor dynamic predicates macro
* wip
* bump fsc-utils to 1.16.1
* update base silver models to use dynamic predicate
* remove
2024-02-09 07:00:54 -08:00
desmond-hui
869f4c8ec6
An 4501/generalize dynamic predicate ( #480 )
...
* wip
* refactor dynamic predicates macro
2024-02-07 08:33:53 -08:00
desmond-hui
46600fa16b
match batch size of realtime job ( #478 )
2024-02-05 09:40:50 -08:00
desmond-hui
e9500c7d61
revert patch fix, force udf max batch rows, decrease batch size ( #477 )
...
* revert patch fix, force udf max batch rows, decrease batch size
* remove dev ref
2024-02-05 08:59:31 -08:00
desmond-hui
92c57a4e86
An 4435/compressed mints backfill helpers ( #475 )
...
* wip
* make the DMLs ACID
* add backfill job, rename existing workflow
* check there is data to backfill before continuing
2024-02-05 07:16:02 -08:00
eric-laurello
137fa2eb2a
testing second model for decoded instruc ( #472 )
...
* initial
* block range macro and model work
* fix incr
* drop _2 update workflow
* delete the right file
---------
Co-authored-by: Desmond Hui <desmond@flipsidecrypto.com>
2024-01-31 10:13:52 -05:00
desmond-hui
673fff3a1b
An 4384/new compressed nft endpoint ( #471 )
...
* add streamline udf for compressed mints
* new table to hold idls for compression parsing workflow
* adjust number of records to be parsed per run
2024-01-31 06:51:36 -08:00
