Commit Graph

133 Commits

Author SHA1 Message Date
desmond-hui
5da67c478c
An 4565/missing block txs and votes (#496)
* wip

* rename

* remove inserted timestamp filter

* revert

* remove semi colon

* add silver tx data quality checks, test hourly

* re-ingest any discrepancies found in solcan+silver check

* limit amount we can retry at one time
2024-02-27 08:07:55 -08:00
Desmond Hui
c631ec1be2 revert to regular schedule 2024-02-24 12:03:50 -08:00
Desmond Hui
e20e48a408 tmp schedule change 2024-02-24 08:50:53 -08:00
Desmond Hui
f7dd22c517 tmp increase realtime resources and change schedule 2024-02-24 08:25:27 -08:00
Desmond Hui
cd12dc2855 increase solscan calls, make requests resend missing blocks 2024-02-24 07:56:43 -08:00
Desmond Hui
5099a73163 slowing down backfill, seeing alot of repeats 2024-02-09 08:03:15 -08:00
desmond-hui
9bc47cac30
add hourly test to check missing completed (#481) 2024-02-08 07:40:39 -08:00
desmond-hui
92c57a4e86
An 4435/compressed mints backfill helpers (#475)
* wip

* make the DMLs ACID

* add backfill job, rename existing workflow

* check there is data to backfill before continuing
2024-02-05 07:16:02 -08:00
eric-laurello
10e4c4f4bc
ez_decoded---updates (#469)
* simplify

* spelling

* add in parsed columns to ez

* combined model, updates to gold

* id col

* del temp workflow

* add date to deprecation
2024-02-01 20:48:30 -05:00
eric-laurello
137fa2eb2a
testing second model for decoded instruc (#472)
* initial

* block range macro and model work

* fix incr

* drop _2 update workflow

* delete the right file

---------

Co-authored-by: Desmond Hui <desmond@flipsidecrypto.com>
2024-01-31 10:13:52 -05:00
Desmond Hui
f6cd545f68 hotfix udf deploy, old pseudo model disabled 2024-01-31 06:59:33 -08:00
eric-laurello
162825513c
decoded_instructions_temp (#470)
* temp table to test incremetla updates

* add date filter to transaction join

* c comes after b
2024-01-29 14:50:02 -05:00
desmond-hui
964aa4c555
add workflow for pyth realtime (#467) 2024-01-26 08:36:51 -08:00
desmond-hui
bc9637a79b
use orchestrator to trigger decoded realtime and backfill (#466) 2024-01-26 07:51:28 -08:00
desmond-hui
85e21d4101
An 4429/decode instructions orchestrator (#464)
* test async workflow orchestration

* update

* update syntax

* log values for debugging

* fix wrong commit

* change log statement

* try using cur timestamp
2024-01-25 09:35:08 -08:00
desmond-hui
0652176735
An 4429/decode instructions orchestrator (#463)
* test async workflow orchestration

* update

* update syntax

* log values for debugging

* fix wrong commit

* change log statement
2024-01-25 09:16:03 -08:00
desmond-hui
c31c312374
An 4429/decode instructions orchestrator (#462)
* test async workflow orchestration

* update

* update syntax
2024-01-25 08:51:47 -08:00
desmond-hui
68174b45cc
test async workflow orchestration (#461)
* test async workflow orchestration

* update
2024-01-25 08:17:11 -08:00
desmond-hui
bd23649b00
An 4429/setup pyth backfill (#460)
* setup pyth backfill

* wip

* remove dev file

* adjust backfill scheduler, add more runs of complete decoded

* adjust backfill rane
2024-01-24 07:49:59 -08:00
desmond-hui
7c0c71241e
auto re-deploy seed files on change (#447)
* auto re-deploy seed files on change

* remove hardcoded py version
2024-01-16 08:36:09 -08:00
desmond-hui
034e88e344
An 3943/get compressed nft addresses (#439)
* external funcs (wip)

* call compressed nft mint parser with external func

* ignore bronze_api models

* move to match standard location

* fix missing mints due to inconsistent ordering

* add a tag to bronze_api model, exclude accordingly

* update silver table to use new bronze api data

* wip

* handle non-mint events and multi mints per tx_id

* add regression test
2024-01-15 07:46:46 -08:00
desmond-hui
a1cd0a3547
move files to orig loc, gha doesnt supp. subfolders (#440) 2024-01-10 07:11:32 -08:00
desmond-hui
169299e865
upgrade to dbt1.7, re-org workflow folder struct (#433)
* upgrade to dbt1.7, re-org workflow folder struct

* allow 1.4 until everything is fully upgraded

* fix wrong template refs
2024-01-10 07:03:45 -08:00
Desmond Hui
cc14acf494 add decoder backfill gha workflow 2024-01-04 09:21:57 -08:00
desmond-hui
f2f28ee9e4
An 4323/initial streamline 2.0 setup (#431)
* wip

* change udf name, fix surrogate key for inner instruct calls

* (wip) decoded instructions backfill helpers

* complete backfill helpers, clean up views with 0 requests

* rename

* revert

* temp change will revert

* rename

* use streamline 2.0 verify idl api endpoint

* this is placeholder model, will replace existing when ready

* reorg files

* add workflow for decode instructions real time

* use pip cache

* update with prod endpoints

* update sql limit to real value, should be union all

* use ref for model names

* remove placeholder, replace existing model logic

* add new decoder flow to gha

* temp disable incremental

* add inner index to core tables

* match existing behavior until we properly incorporate inner_instruction txs

* temp disable

* re-enable models

* remove unused comment

* properly handle inner_index null instances in unique key
2024-01-03 12:39:56 -08:00
desmond-hui
5edc2fcf22
An 4323/initial streamline 2.0 setup (#428)
* wip

* change udf name, fix surrogate key for inner instruct calls

* (wip) decoded instructions backfill helpers

* complete backfill helpers, clean up views with 0 requests

* rename

* revert

* temp change will revert

* rename

* use streamline 2.0 verify idl api endpoint

* this is placeholder model, will replace existing when ready

* reorg files

* add workflow for decode instructions real time

* use pip cache

* update with prod endpoints

* update sql limit to real value, should be union all
2024-01-03 09:24:43 -08:00
Desmond Hui
95345e351a hotfix name 2024-01-02 08:23:37 -08:00
desmond-hui
0bccb491d9
An 4332/auto deploy udf changes (#427)
* update udfs/sps when changes are made to macros in  main branch

* fix template
2024-01-02 08:13:16 -08:00
desmond-hui
f877309b76
revert 2023-12-29 07:14:42 -08:00
desmond-hui
6cf47b85a5
Update dbt_run_helius_metadata.yml 2023-12-28 09:10:51 -08:00
tarikceric
9ab6b17123
An 4313/clean up deprecation (#425)
* depreacte models

* no new records

* test and run intervals
2023-12-18 10:43:18 -08:00
desmond-hui
feba0f7752
Change parse compressed nft backfill to realtime (#415)
* use temp table to parse next 10k compressed mints

* reduce runs to 5
2023-11-30 09:07:56 -08:00
tarikceric
eb0898d11c
An 4136/nft collection names (#409)
* wip

* move logic to silver

* gold table and job

* update core table

* include new and old cols in gold

* incremental logic

* adjust number of calls

* add cluster

* update surrogate key name + remove edition

* cleanup, update surrogate name

* update udf_api, add solscan_collection col

* add pk description

* remove loop and case for pk, fix udf_api prefix

* revert

* pass collection to get name

* add qualify

* add loop

* update loop size

---------

Co-authored-by: Desmond Hui <desmond@flipsidecrypto.com>
2023-11-29 16:55:44 -08:00
desmond-hui
8bf643dcd2
make core run as part of non-core (#407) 2023-11-17 07:40:05 -08:00
tarikceric
7c3a4cd745
Hot fix/rewards backfill (#404)
* wip

* update backfill

* Revert "wip"

This reverts commit 77a46b0a75.

* add var

* temp logic comment

* adjust partition size

* remove null test

* update epoch conditional
2023-11-16 10:43:25 -08:00
desmond-hui
d32599e851
An 4090/force paralell execution (#402)
* force paralell execution for each udf request

* increase number of commands per workflow, decrease schedule frequency
2023-11-15 09:51:03 -08:00
desmond-hui
c567d20d7b
add compressed nft backfill job to SF (#400) 2023-11-14 10:16:18 -08:00
desmond-hui
50d39fa835
update tags for the two main incremental jobs (#397) 2023-11-13 10:19:24 -08:00
tarikceric
891100d4b6
An 4089/block rewards (#395)
* wip

* seperate silver models and create core

* update tag and desc

* update per pr comment

* use prod db source

* adjust fr partition range

* add batc load var

* add gold col descriptions
2023-11-10 15:03:43 -08:00
tarikceric
95e392d5b1
Update/schedule core tags (#396)
* wip

* add schedule tags

* use snowflake tasks

---------

Co-authored-by: Eric Laurello <eric.laurello@flipsidecrypto.com>
2023-11-10 15:59:21 -05:00
desmond-hui
2734d36d7c
Dispatch gha via snowflake (#388)
* update package

* setup GHA dispatches via Snowflake tasks

* remove unused workflows

* add model qualifier to only run models in solana

* include task status monitoring workflow
2023-11-07 08:17:36 -08:00
desmond-hui
4d965d709e
Tmp change history for jupv5 backfill (#383)
* temp change to pull all jupv5 txs

* allow manual dispatch for history parser workflow
2023-11-03 13:25:23 -07:00
desmond-hui
85876493df
An 4000/obeservability improvements (#380)
* wip

* wip udf

* bronze and silver models for txcounts (wip)

* optimize for chainhead

* check for blocks with 0 txs

* change tx count call to solscan blocks

* only call 1000 at a time to solscan due to rate limits

* update rewards realtime task (#378)

* run solscan blocks multiple times per workflow dispatch

* fix name

* disable full refresh on bronze api table
2023-11-02 07:09:19 -07:00
tarikceric
a1e73a1c91
update frequency and add test (#369)
* update frequency and add test

* adjust schedule
2023-10-19 09:54:47 -07:00
desmond-hui
ef65497970
Update dbt_run_parser_program_historical_build_queue.yml 2023-10-19 07:37:26 -07:00
desmond-hui
0ee5603884
Update dbt_run_incremental.yml 2023-10-16 07:57:58 -07:00
desmond-hui
1047c6304b
break up the model to only run 50 requests per invocation (#364) 2023-10-12 09:14:04 -07:00
desmond-hui
561a9b7f6a
An 3943/get compressed nft addresses (#358)
* external funcs (wip)

* call compressed nft mint parser with external func

* ignore bronze_api models

* move to match standard location

* fix missing mints due to inconsistent ordering

* add a tag to bronze_api model, exclude accordingly
2023-10-04 08:41:01 -07:00
tarikceric
6464730baf
update helius job (#359) 2023-10-03 13:51:23 -07:00
tarikceric
82c0005222
An 3761/helius metadata (#353)
* helius metadata grabs

* update

* exclude null metadata

* adjust increment

* update jobs

* yml and dups

* tags

* use livequery endpoint

* update schema, add error handling

* remove unnecessary ORDER BY

* fix timestamp for backfill start

* update per pr comments

* update status cols
2023-10-03 12:29:36 -07:00