Commit Graph

89 Commits

Author SHA1 Message Date
tarikceric
a4da0f567e
model and workflow (#686)
* model and workflow

* name, filter large partitions, max partition

* formatting
2024-11-01 09:32:02 -07:00
tarikceric
890a2adab3
An 5156/jupv4 refactor (#672)
* wip

* clean

* add extra join logic

* fix colon

* add to core models

* core models wip

* revert

* cleanup

* revamp logic; pr comment changes

* update logic

* relationship test; logic update

* id truncated logs

* updated truncated logic
2024-10-30 06:58:12 -07:00
tarikceric
0f8c825794
raydium cpmm swaps (#676)
* raydium cpmm swaps

* remove filter
2024-10-17 08:22:51 -07:00
desmond-hui
eb42fa2814
Migrate blocks to streamline 2.0 (#675)
* bronze streamline blocks

* complete blocks and streamline requests

* add workflow

* revert

* update cutoff
2024-10-16 09:38:58 -07:00
desmond-hui
122fa61bc9
An 5047/helius cnft metadata (#669)
* streamline bronze data models

* wip complete and requests nft metadata models

* add source

* formatting

* update columns and explode key

* fix partition key

* update requests logic

* rename to cnft

* run cnft metadata every 10 minutes

* add SO

* formatting
2024-10-15 09:21:52 -07:00
tarikceric
c6785fa3e2
get all distinct txs; null amt/mint bug (#667)
* get all distinct txs; null amt/mint bug

* formatting
2024-10-08 10:24:14 -07:00
desmond-hui
0f59a077ac
Rewards streamline 2 (#660)
* wip

* wip

* tmp change to get it to work

* fix service var, change producer values

* wip

* add source and bronze models

* wip rewards

* wip

* wip

* wip

* adjust partition key logic for block rewards

* add dbt depends_on

* compare to completed

* add workflow for calling streamline 2.0 block rewards

* make and use v2 streamline external table macro

* rename and adjust starting block/partition

* update workflow

* match existing schedule
2024-09-27 09:31:33 -07:00
tarikceric
ba915cf20b
temp exclude events_inner test; remove workflow (#661)
* temp exclude events_inner test; remove workflow

* reduce unique test to 7 days
2024-09-26 12:35:32 -07:00
tarikceric
24a660de1c
An 5156/jupv4 v5 swaps logs (#657)
* wip

* finalize jupv5, update v4

* add filter

* tests and cleanup

* remove v4 and add tests

* cleanup and add missing log_index to yml
2024-09-23 15:37:00 -07:00
tarikceric
b22caeab39
An 4528/events inner (#655)
* initial silver models

* create gold tables

* backfill

* incremental backfill runs

* update run interval

* update tests

* removed filter

* updated per pr comments

* update backfill and cluster/so

* update gold inc filter

* backfill updates

* clean up

* disable fr and update backfill date

* update backfill logic

* correct filter date
2024-09-19 10:53:14 -07:00
tarikceric
0faab509a8
An 5080/pumpfun swaps (#637)
* pumpfun models

* silver update and add to gold

* wip

* clean up

* add relationship test

* formatting, remove hardcoded refs

* remove backfill date predicate for pumpfun

---------

Co-authored-by: desmond-hui <desmond@flipsidecrypto.com>
2024-08-19 08:30:01 -07:00
desmond-hui
401c31ab26 remove transactions backfill from github workflow 2024-08-06 12:43:35 -07:00
desmond-hui
8b33b55737
Backfill txs with vote txs with non vote instructions (#619)
* add model to identify txs to be backfilled

* add workflow to run backfill

* add comment
2024-07-25 06:34:35 -07:00
tarikceric
698b9469eb
An 5001/update amm sell (#616)
* wip

* add relationship test

* update ref

* add tests

* revert

* revert

* new model; convert old one to view; update gold

* add events

* update ME signers filter

* update relationship test
2024-07-24 11:27:12 -07:00
tarikceric
d91214791f
update meteora swaps (#599) 2024-07-09 15:25:45 -05:00
desmond-hui
780e4a5ef1
Tensorswap models using logs (#600)
* udf to grab program data logs from log messages

* output table with encoded program data in tx logs

* add comments

* setup views and source

* update tag

* wip realtime view

* wip

* fix bad join

* wip

* decoded logs

* wip

* add event_type, index, inner_index and refactor existing

* handle additional nested calls

* add columns, add to SO

* use backfill macro

* wip

* handle non-contiguous program log messaging

* tmp upperbound

* minor changes

* reset child index when a new top level program is invoked

* rebuild transaction logs program data

* remove backfill

* still need to use min price as backup

* add post PR TODO

* revert tmp full-refresh changes

* add comment

* add tests

* update columns

* add docs and tests, fix col name

* update col description
2024-07-09 09:27:57 -07:00
desmond-hui
44416d7744
Jupiter v6 refactor (#595)
* reorganize swaps models folder structure

* wip jupv6 inner swaps

* wip

* wip

* update logic

* refactored

* remove comments

* docs and tests

* cleanup and add metadata cols

* pick the larger of the 2 dates so that it gets picked up downstream

* fix bad swap index in test seed

* initial model and docs

* add test cases

* adjust input/output swap join logic

* remove upper bound

* modify incremental logic to work with window function cols

* fix seed file

* remove no longer valid test cases

* put schedule tag last

* add relationship test
2024-07-02 06:50:37 -07:00
desmond-hui
79b061e755
Jupiter inner swaps (#588)
* reorganize swaps models folder structure

* wip jupv6 inner swaps

* wip

* wip

* update logic

* refactored

* remove comments

* docs and tests

* cleanup and add metadata cols

* pick the larger of the 2 dates so that it gets picked up downstream

* fix bad swap index in test seed

* adjust swap index to relate to each top level jupiter instruction

* add test cases

* put tags last

* remove _inserted_timestamp greatest, causing unexpected incremental behavior

* handle multiple top level swaps inner instructions each using >1 route

* remove incremental upper bound

* add test case
2024-06-25 19:39:44 -07:00
desmond-hui
dda387f040
realtime scheduling for decoding logs (#579) 2024-06-12 09:04:31 -07:00
tarikceric
d807559755
An 4867/add new raydium swaps (#575)
* replace old raydium with new models

* add new raydium models and deprecate existing

* add source for deprecated raydium

* remove unnecessary view+source
2024-06-11 12:44:36 -07:00
tarikceric
3cf4b17de5
An 4825/raydium swaps (#565)
* wip

* wip

* clean up

* update date filter

* update per pr comments

* remove unnecessary model
2024-05-30 13:53:00 -07:00
tarikceric
aec96d8069
meteora lp multi token pools (#557) 2024-05-13 12:43:15 -07:00
tarikceric
db5a1ebf21
Meteora DLMM LP actions (#555)
* model and tests

* update col name + col tests in yml

* clean up tests
2024-05-09 10:58:28 -07:00
desmond-hui
71c9a9f5c0
An 4787/liquidity pool actions meteora (#552)
* (wip) initial model

* create base ctes, fix syntax

* move combining specific event types downstream

* revert temp full refresh range change

* add tests and schema

* add relationship test

* fix predicate

* remove unnecessary predicates

* filter out add balances where no balance is added

* add coalesce in cases where this arg is not provided

* include metadata fields in schema, fix wrong col name
2024-05-09 08:39:23 -07:00
desmond-hui
c0d866b4fd
An 4800/tensorswap refactor (#553)
* (wip) initial model

* remove testing lines, add depends on comment

* handle more than  one sale per tx

* separate sol escrow from margin account

* use min_price as sale price for bid sales

* fix column name

* add model docs and tests

* add caveat for tensorswap sales amount description

* add business logic tests
2024-05-08 09:01:42 -07:00
Desmond Hui
319b76b669 remove extra double quote 2024-04-29 07:10:08 -07:00
desmond-hui
9d30ea2e3f
remove the cnft backfill task (#546) 2024-04-29 07:04:43 -07:00
tarikceric
1cef5247b9
An 4769/me solsniper cnft sales (#540)
* wip

* wip

* get compressed mint

* add schedule, pr comments update

* temp backfill logic

* backfill only
2024-04-25 10:41:45 -07:00
tarikceric
284e0c93ba
fix incremental, logic, fix from/to cols (#539)
* fix incremental, logic, fix from/to cols

* update desc

* update silver desc
2024-04-23 09:54:11 -07:00
tarikceric
ac4d0d8c9e
An 4638/cnft sales (#536)
* tensor swap models

* update code style

* fix

* update tensorswap cnft sales

* update per pr comments

* update logic for mint and transfers, add relationship test

* add relationship test
2024-04-18 07:46:01 -07:00
desmond-hui
7eda8c5cd2
remove cnft backfill from schedule, it is completed (#532) 2024-04-12 09:32:38 -07:00
desmond-hui
38c5d96397
Fix jupv6 test failures (#531)
* fix seed file column order

* add sanctum burn logic

* fix issue with wrong decimals for certain burns

* add and fix tests
2024-04-12 08:58:52 -07:00
tarikceric
d54b300de1
add tests for solsniper view (#522) 2024-03-20 09:20:25 -07:00
desmond-hui
1e9e3abe94
refactor model logic to include missing swaps, add tests (#520)
* refactor model logic to include missing swaps, add tests

* fix hardcoded name
2024-03-20 08:11:06 -07:00
tarikceric
1335883dd1
add solsniper to nft sales (#505)
* add solsniper

* include additional events

* clean up v1 views

* update incremental logic

* left join to transfers

* remove null test

* update code style

* add solsniper relation test

* updates per comments

* select view in core
2024-03-19 10:55:20 -07:00
tarikceric
a73f9134c1
An 4450/add swap protocols (#494)
* add bonk, meteora, phoenix swaps

* update desc

* add meteora

* add test case

* remove solana reference

* add dooar swaps

* remove phoenix; remove solana prefix

* add phoenix

* phoenix fix

* update backfill date

* logic for multi swaps w incorrect join

* add tests, update logic for multi swap txs

* remove db prefix
2024-03-01 11:26:22 -08:00
desmond-hui
263ca35e27
Fix/support new magic eden seller pattern (#497)
* direct transfer from system program to seller

* add test, fix issue w/ full refresh logic followed by incremental
2024-02-29 08:39:05 -08:00
tarikceric
41f0417890
An 4449/staking tables update (#487)
* update jito staking

* update token
2024-02-19 09:13:11 -08:00
Desmond Hui
9886ee4654 update cnft backfill scheduler 2024-02-06 06:39:20 -08:00
desmond-hui
92c57a4e86
An 4435/compressed mints backfill helpers (#475)
* wip

* make the DMLs ACID

* add backfill job, rename existing workflow

* check there is data to backfill before continuing
2024-02-05 07:16:02 -08:00
desmond-hui
bc9637a79b
use orchestrator to trigger decoded realtime and backfill (#466) 2024-01-26 07:51:28 -08:00
desmond-hui
68174b45cc
test async workflow orchestration (#461)
* test async workflow orchestration

* update
2024-01-25 08:17:11 -08:00
tarikceric
4e1da2c97b
An 4280/bridges v2 (#450)
* wip

* fix naming

* revamp bridges

* split models

* updates

* update tests and get inner events

* fix pk

* further exclude mayan

* exclude fee amounts

* add debridge tests

---------

Co-authored-by: Desmond Hui <desmond@flipsidecrypto.com>
2024-01-24 10:00:04 -08:00
desmond-hui
bd23649b00
An 4429/setup pyth backfill (#460)
* setup pyth backfill

* wip

* remove dev file

* adjust backfill scheduler, add more runs of complete decoded

* adjust backfill rane
2024-01-24 07:49:59 -08:00
tarikceric
771e3422ad
add missing tokens (#458) 2024-01-23 12:26:13 -08:00
desmond-hui
034e88e344
An 3943/get compressed nft addresses (#439)
* external funcs (wip)

* call compressed nft mint parser with external func

* ignore bronze_api models

* move to match standard location

* fix missing mints due to inconsistent ordering

* add a tag to bronze_api model, exclude accordingly

* update silver table to use new bronze api data

* wip

* handle non-mint events and multi mints per tx_id

* add regression test
2024-01-15 07:46:46 -08:00
tarikceric
19ea1ac5e7
update to use local lq (#432)
* update to use local lq

* create request in udf_api call

* add missing tokens
2024-01-04 14:56:28 -08:00
Desmond Hui
cc14acf494 add decoder backfill gha workflow 2024-01-04 09:21:57 -08:00
desmond-hui
73a2d32f29
An 4323/initial streamline 2.0 setup (#430)
* wip

* change udf name, fix surrogate key for inner instruct calls

* (wip) decoded instructions backfill helpers

* complete backfill helpers, clean up views with 0 requests

* rename

* revert

* temp change will revert

* rename

* use streamline 2.0 verify idl api endpoint

* this is placeholder model, will replace existing when ready

* reorg files

* add workflow for decode instructions real time

* use pip cache

* update with prod endpoints

* update sql limit to real value, should be union all

* use ref for model names

* remove placeholder, replace existing model logic

* add new decoder flow to gha

* temp disable incremental

* add inner index to core tables

* match existing behavior until we properly incorporate inner_instruction txs

* temp disable
2024-01-03 11:40:57 -08:00
Desmond Hui
36ff2314a1 hotfix broken models part 1 2023-12-28 08:19:03 -08:00