Commit Graph

104 Commits

Author SHA1 Message Date
tarikceric
86bfac1046
separate stake accts workflow (#836)
* separate stake accts workflow

* run on hourly job
2025-04-29 12:14:59 -07:00
desmond-hui
4da6ac01b3
Refactor orca swaps whirlpool (#819)
* WIP

* fix join to transfers, handle twoHopSwap

* add schema and tests

* add SO, fix bad tests
2025-04-22 14:06:39 -07:00
tarikceric
51a18161c0 hot fix update stake accts to weekly schedule 2025-04-21 13:57:57 -07:00
tarikceric
dd87a5a4d3
move stake accts snapshot to gha workflow file (#826)
* move stake accts snapshot to gha workflow file

* cleanup
2025-04-10 13:41:32 -07:00
tarikceric
b03cce6a4c
extra core schedule to fill gap (#817)
* extra core schedule to fill gap

* run name
2025-03-07 13:01:52 -08:00
tarikceric
75a16253d9
hourly schedule for ez dex swaps and upstream (#815)
* hourly schedule for ez dex swaps and upstream

* hourly schedule for ez dex swaps and upstream

* cleanup
2025-03-06 08:59:36 -08:00
tarikceric
94b5f2942c
Update/core noncore schedule (#805)
* update core and noncore runs

* run votes with core
2025-02-14 09:27:12 -08:00
desmond-hui
ca55076476
stage block txs data for ingestion to silver (#781) 2025-01-28 09:45:21 -08:00
desmond-hui
c923480751
Marinade native stakers remap withdraw authority (#746)
* (wip) map withdraw authority using recursion

* WIP v2 staking upstream models

* fix issue with vote accounts and formattiing

* fix inner index expected values

* add docs for common columns

* initial schema

* add native 1 acct, fix dupes, add usd

* initial schema and tests

* add withdraw amount, recency test

* convert lamports to decimals

* add flag for whether still using marinade

* PR feedback

* PR fix

* tmp update to load next batch

* revert tmp change

* revert

* add to scheduler
2025-01-08 08:30:48 -08:00
tarikceric
ccd085163c
remove cnft backfill (#749)
* remove cnft backfill

* add comment
2025-01-07 09:37:43 -08:00
tarikceric
40c3f1688e
remove tx units backfill job (#742) 2024-12-20 10:49:39 -08:00
desmond-hui
f69c126030
Migrate block txs to streamline 2.0 (#727)
* add views for block txs streamline 2.0 data

* streamline 2.0 requests for block txs

* fix view macro to allow for no other cols

* add top level block timestamp, adjust batch size

* add workflow for pipeline

* add block tx 2 to gha schedule, reorder records
2024-12-06 07:53:50 -08:00
tarikceric
7b139acfba
increase batch size; run interval (#702) 2024-11-08 09:45:09 -08:00
tarikceric
5688bce773
Update/cnft backfill helper (#697)
* update cnft backfill helper

* add cnft backfill

* revert index, add filter

* run 4x hour
2024-11-05 12:19:29 -08:00
desmond-hui
b74ff15fd6
Migrate get solscan blocks to 2.0 (#685)
* streamline requests for solscan blocks

* bronze views for solscan streamline data

* workflow schedule for solscan streamline requests

* add todo comment

* update uri

* update the silver table before requesting the next set of blocks

* swap to using 2.0 data

* update TODO

* change vault path
2024-11-05 09:53:04 -08:00
tarikceric
a4da0f567e
model and workflow (#686)
* model and workflow

* name, filter large partitions, max partition

* formatting
2024-11-01 09:32:02 -07:00
tarikceric
890a2adab3
An 5156/jupv4 refactor (#672)
* wip

* clean

* add extra join logic

* fix colon

* add to core models

* core models wip

* revert

* cleanup

* revamp logic; pr comment changes

* update logic

* relationship test; logic update

* id truncated logs

* updated truncated logic
2024-10-30 06:58:12 -07:00
tarikceric
0f8c825794
raydium cpmm swaps (#676)
* raydium cpmm swaps

* remove filter
2024-10-17 08:22:51 -07:00
desmond-hui
eb42fa2814
Migrate blocks to streamline 2.0 (#675)
* bronze streamline blocks

* complete blocks and streamline requests

* add workflow

* revert

* update cutoff
2024-10-16 09:38:58 -07:00
desmond-hui
122fa61bc9
An 5047/helius cnft metadata (#669)
* streamline bronze data models

* wip complete and requests nft metadata models

* add source

* formatting

* update columns and explode key

* fix partition key

* update requests logic

* rename to cnft

* run cnft metadata every 10 minutes

* add SO

* formatting
2024-10-15 09:21:52 -07:00
tarikceric
c6785fa3e2
get all distinct txs; null amt/mint bug (#667)
* get all distinct txs; null amt/mint bug

* formatting
2024-10-08 10:24:14 -07:00
desmond-hui
0f59a077ac
Rewards streamline 2 (#660)
* wip

* wip

* tmp change to get it to work

* fix service var, change producer values

* wip

* add source and bronze models

* wip rewards

* wip

* wip

* wip

* adjust partition key logic for block rewards

* add dbt depends_on

* compare to completed

* add workflow for calling streamline 2.0 block rewards

* make and use v2 streamline external table macro

* rename and adjust starting block/partition

* update workflow

* match existing schedule
2024-09-27 09:31:33 -07:00
tarikceric
ba915cf20b
temp exclude events_inner test; remove workflow (#661)
* temp exclude events_inner test; remove workflow

* reduce unique test to 7 days
2024-09-26 12:35:32 -07:00
tarikceric
24a660de1c
An 5156/jupv4 v5 swaps logs (#657)
* wip

* finalize jupv5, update v4

* add filter

* tests and cleanup

* remove v4 and add tests

* cleanup and add missing log_index to yml
2024-09-23 15:37:00 -07:00
tarikceric
b22caeab39
An 4528/events inner (#655)
* initial silver models

* create gold tables

* backfill

* incremental backfill runs

* update run interval

* update tests

* removed filter

* updated per pr comments

* update backfill and cluster/so

* update gold inc filter

* backfill updates

* clean up

* disable fr and update backfill date

* update backfill logic

* correct filter date
2024-09-19 10:53:14 -07:00
tarikceric
0faab509a8
An 5080/pumpfun swaps (#637)
* pumpfun models

* silver update and add to gold

* wip

* clean up

* add relationship test

* formatting, remove hardcoded refs

* remove backfill date predicate for pumpfun

---------

Co-authored-by: desmond-hui <desmond@flipsidecrypto.com>
2024-08-19 08:30:01 -07:00
desmond-hui
401c31ab26 remove transactions backfill from github workflow 2024-08-06 12:43:35 -07:00
desmond-hui
8b33b55737
Backfill txs with vote txs with non vote instructions (#619)
* add model to identify txs to be backfilled

* add workflow to run backfill

* add comment
2024-07-25 06:34:35 -07:00
tarikceric
698b9469eb
An 5001/update amm sell (#616)
* wip

* add relationship test

* update ref

* add tests

* revert

* revert

* new model; convert old one to view; update gold

* add events

* update ME signers filter

* update relationship test
2024-07-24 11:27:12 -07:00
tarikceric
d91214791f
update meteora swaps (#599) 2024-07-09 15:25:45 -05:00
desmond-hui
780e4a5ef1
Tensorswap models using logs (#600)
* udf to grab program data logs from log messages

* output table with encoded program data in tx logs

* add comments

* setup views and source

* update tag

* wip realtime view

* wip

* fix bad join

* wip

* decoded logs

* wip

* add event_type, index, inner_index and refactor existing

* handle additional nested calls

* add columns, add to SO

* use backfill macro

* wip

* handle non-contiguous program log messaging

* tmp upperbound

* minor changes

* reset child index when a new top level program is invoked

* rebuild transaction logs program data

* remove backfill

* still need to use min price as backup

* add post PR TODO

* revert tmp full-refresh changes

* add comment

* add tests

* update columns

* add docs and tests, fix col name

* update col description
2024-07-09 09:27:57 -07:00
desmond-hui
44416d7744
Jupiter v6 refactor (#595)
* reorganize swaps models folder structure

* wip jupv6 inner swaps

* wip

* wip

* update logic

* refactored

* remove comments

* docs and tests

* cleanup and add metadata cols

* pick the larger of the 2 dates so that it gets picked up downstream

* fix bad swap index in test seed

* initial model and docs

* add test cases

* adjust input/output swap join logic

* remove upper bound

* modify incremental logic to work with window function cols

* fix seed file

* remove no longer valid test cases

* put schedule tag last

* add relationship test
2024-07-02 06:50:37 -07:00
desmond-hui
79b061e755
Jupiter inner swaps (#588)
* reorganize swaps models folder structure

* wip jupv6 inner swaps

* wip

* wip

* update logic

* refactored

* remove comments

* docs and tests

* cleanup and add metadata cols

* pick the larger of the 2 dates so that it gets picked up downstream

* fix bad swap index in test seed

* adjust swap index to relate to each top level jupiter instruction

* add test cases

* put tags last

* remove _inserted_timestamp greatest, causing unexpected incremental behavior

* handle multiple top level swaps inner instructions each using >1 route

* remove incremental upper bound

* add test case
2024-06-25 19:39:44 -07:00
desmond-hui
dda387f040
realtime scheduling for decoding logs (#579) 2024-06-12 09:04:31 -07:00
tarikceric
d807559755
An 4867/add new raydium swaps (#575)
* replace old raydium with new models

* add new raydium models and deprecate existing

* add source for deprecated raydium

* remove unnecessary view+source
2024-06-11 12:44:36 -07:00
tarikceric
3cf4b17de5
An 4825/raydium swaps (#565)
* wip

* wip

* clean up

* update date filter

* update per pr comments

* remove unnecessary model
2024-05-30 13:53:00 -07:00
tarikceric
aec96d8069
meteora lp multi token pools (#557) 2024-05-13 12:43:15 -07:00
tarikceric
db5a1ebf21
Meteora DLMM LP actions (#555)
* model and tests

* update col name + col tests in yml

* clean up tests
2024-05-09 10:58:28 -07:00
desmond-hui
71c9a9f5c0
An 4787/liquidity pool actions meteora (#552)
* (wip) initial model

* create base ctes, fix syntax

* move combining specific event types downstream

* revert temp full refresh range change

* add tests and schema

* add relationship test

* fix predicate

* remove unnecessary predicates

* filter out add balances where no balance is added

* add coalesce in cases where this arg is not provided

* include metadata fields in schema, fix wrong col name
2024-05-09 08:39:23 -07:00
desmond-hui
c0d866b4fd
An 4800/tensorswap refactor (#553)
* (wip) initial model

* remove testing lines, add depends on comment

* handle more than  one sale per tx

* separate sol escrow from margin account

* use min_price as sale price for bid sales

* fix column name

* add model docs and tests

* add caveat for tensorswap sales amount description

* add business logic tests
2024-05-08 09:01:42 -07:00
Desmond Hui
319b76b669 remove extra double quote 2024-04-29 07:10:08 -07:00
desmond-hui
9d30ea2e3f
remove the cnft backfill task (#546) 2024-04-29 07:04:43 -07:00
tarikceric
1cef5247b9
An 4769/me solsniper cnft sales (#540)
* wip

* wip

* get compressed mint

* add schedule, pr comments update

* temp backfill logic

* backfill only
2024-04-25 10:41:45 -07:00
tarikceric
284e0c93ba
fix incremental, logic, fix from/to cols (#539)
* fix incremental, logic, fix from/to cols

* update desc

* update silver desc
2024-04-23 09:54:11 -07:00
tarikceric
ac4d0d8c9e
An 4638/cnft sales (#536)
* tensor swap models

* update code style

* fix

* update tensorswap cnft sales

* update per pr comments

* update logic for mint and transfers, add relationship test

* add relationship test
2024-04-18 07:46:01 -07:00
desmond-hui
7eda8c5cd2
remove cnft backfill from schedule, it is completed (#532) 2024-04-12 09:32:38 -07:00
desmond-hui
38c5d96397
Fix jupv6 test failures (#531)
* fix seed file column order

* add sanctum burn logic

* fix issue with wrong decimals for certain burns

* add and fix tests
2024-04-12 08:58:52 -07:00
tarikceric
d54b300de1
add tests for solsniper view (#522) 2024-03-20 09:20:25 -07:00
desmond-hui
1e9e3abe94
refactor model logic to include missing swaps, add tests (#520)
* refactor model logic to include missing swaps, add tests

* fix hardcoded name
2024-03-20 08:11:06 -07:00
tarikceric
1335883dd1
add solsniper to nft sales (#505)
* add solsniper

* include additional events

* clean up v1 views

* update incremental logic

* left join to transfers

* remove null test

* update code style

* add solsniper relation test

* updates per comments

* select view in core
2024-03-19 10:55:20 -07:00