Commit Graph

35 Commits

Author SHA1 Message Date
desmond-hui
31f532a129
only resend if not exist or error in silver for single day retries (#780) 2025-01-28 07:10:51 -08:00
desmond-hui
bd52d9032b
Add decoder backfill helper for single dates (#772)
* backfill macro for all verified IDLs for specified date

* fix checking for completed after single date retry

* add block_id predicate for completed subset

* add retry macro for single date

* make macro names consistent
2025-01-15 14:22:09 -08:00
desmond-hui
63dc6591e4
Migrate compressed mint calls to streamline 2.0 (#739)
* fix batch size for udf

* update usage to v2
2024-12-17 07:23:12 -08:00
desmond-hui
420128b2a1
Deprecate legacy decode logs pipeline (#726)
* call streamline 2.0 endpoints

* update request query

* update to use streamline 2.0 data after cutoff

* formatting

* use new complete tables for backfill macros

* add larger lookback for first run, revert after

* fix comments
2024-12-09 09:13:47 -08:00
desmond-hui
8ea30e9285
Deprecate legacy decode instructions pipeline (#723)
* update requests model

* only call streamline 2.0 endpoint

* use streamline 2.0 data after cutoff, formatting

* fix id column reference

* remove FR and add comment

* update helper to use new completed table

* update cutoff date, extend lookback
2024-12-09 08:05:34 -08:00
tarikceric
7b139acfba
increase batch size; run interval (#702) 2024-11-08 09:45:09 -08:00
tarikceric
5688bce773
Update/cnft backfill helper (#697)
* update cnft backfill helper

* add cnft backfill

* revert index, add filter

* run 4x hour
2024-11-05 12:19:29 -08:00
tarikceric
b22caeab39
An 4528/events inner (#655)
* initial silver models

* create gold tables

* backfill

* incremental backfill runs

* update run interval

* update tests

* removed filter

* updated per pr comments

* update backfill and cluster/so

* update gold inc filter

* backfill updates

* clean up

* disable fr and update backfill date

* update backfill logic

* correct filter date
2024-09-19 10:53:14 -07:00
desmond-hui
d37703693b
use pytz timezones in decode backfill macro (#643) 2024-08-23 07:13:57 -07:00
desmond-hui
4ae84eda4f
add metadata to decode log requests (#607) 2024-07-16 08:50:49 -07:00
desmond-hui
ed8db95341
spread work across more workers (#602) 2024-07-12 08:28:13 -07:00
desmond-hui
8272ba8601
request log decode for jup6 swaps called by a diff program than jup6 (#589)
* request log decode for jup6 swaps called by a diff program than jup6

* formatting
2024-06-20 06:28:19 -07:00
desmond-hui
3258b155cd
Streamline decoded logs backfill (#580)
* add completed backfill table

* add helper macros for decoded log backfills

* add decoded log backfills to scheduler
2024-06-13 06:52:18 -07:00
desmond-hui
2865420293
reduce size per execution (#562) 2024-05-15 08:26:02 -07:00
desmond-hui
6ab9324ea9
add lpad zeros to view name, fix set formatting (#527)
* add lpad zeros to view name, fix set formatting

* allow setting of custom priority value
2024-04-11 07:15:03 -07:00
desmond-hui
cd9c4ba9b0
An 4546/allow parameterized retries for decoder errs (#493)
* add retry helper macro for decoder backfill reqs

* update comment
2024-02-20 06:48:36 -08:00
desmond-hui
02bacfa997
An 4508/integrate fsc utils dynamic predicate macro (#482)
* wip

* refactor dynamic predicates macro

* wip

* bump fsc-utils to 1.16.1

* update base silver models to use dynamic predicate

* remove
2024-02-09 07:00:54 -08:00
desmond-hui
869f4c8ec6
An 4501/generalize dynamic predicate (#480)
* wip

* refactor dynamic predicates macro
2024-02-07 08:33:53 -08:00
desmond-hui
46600fa16b
match batch size of realtime job (#478) 2024-02-05 09:40:50 -08:00
desmond-hui
92c57a4e86
An 4435/compressed mints backfill helpers (#475)
* wip

* make the DMLs ACID

* add backfill job, rename existing workflow

* check there is data to backfill before continuing
2024-02-05 07:16:02 -08:00
eric-laurello
137fa2eb2a
testing second model for decoded instruc (#472)
* initial

* block range macro and model work

* fix incr

* drop _2 update workflow

* delete the right file

---------

Co-authored-by: Desmond Hui <desmond@flipsidecrypto.com>
2024-01-31 10:13:52 -05:00
Desmond Hui
96886c3686 fix missing quote 2024-01-27 06:41:15 -08:00
desmond-hui
bc9637a79b
use orchestrator to trigger decoded realtime and backfill (#466) 2024-01-26 07:51:28 -08:00
desmond-hui
bd23649b00
An 4429/setup pyth backfill (#460)
* setup pyth backfill

* wip

* remove dev file

* adjust backfill scheduler, add more runs of complete decoded

* adjust backfill rane
2024-01-24 07:49:59 -08:00
desmond-hui
9dafcdd99e
remove macro and pre-hooks (#449) 2024-01-17 08:45:19 -08:00
desmond-hui
39338f6576
An 4419/force file registration (#448)
* bronze decoded table force unregistered file check for >= cur_hour-1

* force file registration as prehook
2024-01-16 11:17:19 -08:00
desmond-hui
ffe9120f46
An 4390/retry views for failed old decoder txs (#441)
* generate retry view from prev decoder output

* add default config vals for livequery
2024-01-11 09:37:23 -08:00
desmond-hui
2259be5002
use union all instead of union, fix inner_instruct backfill cte (#436) 2024-01-05 10:56:50 -08:00
desmond-hui
73a2d32f29
An 4323/initial streamline 2.0 setup (#430)
* wip

* change udf name, fix surrogate key for inner instruct calls

* (wip) decoded instructions backfill helpers

* complete backfill helpers, clean up views with 0 requests

* rename

* revert

* temp change will revert

* rename

* use streamline 2.0 verify idl api endpoint

* this is placeholder model, will replace existing when ready

* reorg files

* add workflow for decode instructions real time

* use pip cache

* update with prod endpoints

* update sql limit to real value, should be union all

* use ref for model names

* remove placeholder, replace existing model logic

* add new decoder flow to gha

* temp disable incremental

* add inner index to core tables

* match existing behavior until we properly incorporate inner_instruction txs

* temp disable
2024-01-03 11:40:57 -08:00
desmond-hui
5edc2fcf22
An 4323/initial streamline 2.0 setup (#428)
* wip

* change udf name, fix surrogate key for inner instruct calls

* (wip) decoded instructions backfill helpers

* complete backfill helpers, clean up views with 0 requests

* rename

* revert

* temp change will revert

* rename

* use streamline 2.0 verify idl api endpoint

* this is placeholder model, will replace existing when ready

* reorg files

* add workflow for decode instructions real time

* use pip cache

* update with prod endpoints

* update sql limit to real value, should be union all
2024-01-03 09:24:43 -08:00
Jessica Huhnke
4ca6fef32d
An 1904/user interactions (#254)
* wip

* seperating out tokens / nfts still wip

* wip - token burns

* move nft / token burns into same model

* daily nft balance change

* signers add in NFT balances

* wip

* wip - ready to test

* signers

* signers change

* make the incremental / FR work

* added in reference to account ownership change table

* wip - ready to test

* wip

* remove test address

* add in token account column into the mints table

* merge main & fixes to incremental

* remove null signers

* add batch load macro that adds alias

* incremental update

* simplify model & remove column from signers

* fix it

---------

Co-authored-by: Desmond Hui <desmond@flipsidecrypto.com>
2023-02-21 13:33:04 -06:00
Jessica Huhnke
342c9b27ab
Revert "An 1904/user interactions (#247)" (#251)
This reverts commit a1d9df0acc.
2023-02-16 16:32:06 -06:00
Jessica Huhnke
a1d9df0acc
An 1904/user interactions (#247)
* wip

* seperating out tokens / nfts still wip

* wip - token burns

* move nft / token burns into same model

* daily nft balance change

* signers add in NFT balances

* wip

* wip - ready to test

* signers

* signers change

* make the incremental / FR work

* added in reference to account ownership change table

* wip - ready to test

* wip

* remove test address

* add in token account column into the mints table

* merge main & fixes to incremental

* remove null signers

* add batch load macro that adds alias

* incremental update

---------

Co-authored-by: Desmond Hui <desmond@flipsidecrypto.com>
2023-02-16 14:46:16 -06:00
desmond-hui
d16d79cfab
fix syntax (#245) 2023-02-10 11:34:04 -08:00
desmond-hui
12442761fc
An 2149/account owners model (#237)
* wip

* wip

* wip

* change dev owner to use new access role

* wip

* update names

* wip

* use macro for dynamic incremental logic

* change names, split stakes acct and token acct into own tables

* workflow to backfill

* disable new models from incremental while it backfills

* include mint of token account on creation

* inner index should not have no null tests, add mint to yml

* make inner_index -1 so that it can be included in unique key merge

* pr fixes

* add dupe tests

* change to full refresh only, works faster for backfills at least

* fix typo

* fix mint null test, only for create events

* account initialization also has mint value

* add ordering based on events to fix same block event issues

* simplify temporary exclude
2023-02-09 14:37:48 -08:00