Commit Graph

161 Commits

Author SHA1 Message Date
WHYTEWYLL
1cfb107b46 fix: dedeuplicates 2023-11-29 13:41:41 -03:00
WHYTEWYLL
3e5e0a4bdd
Update silver__nft_allday_metadata_s.sql (#236) 2023-11-29 08:29:39 -08:00
WHYTEWYLL
6d6f478df6
alldaynft metadata with LQ (#229)
* base

* clean up; add: silver model

* add: new standards

* fix: refactor

* delete: folders

* refactor simplying

* remove no longer needed

* fix: refactor to silver layer

* ready to merge

---------

Co-authored-by: Desmond Hui <desmond@flipsidecrypto.com>
2023-11-27 07:55:44 -08:00
Jack Forgash
b33dc89a39
swap set description col (#232)
* swap set description col

* cast str
2023-11-20 16:39:47 -07:00
Jack Forgash
c82c04ad5d
del invocation id from gold (#231) 2023-11-20 16:30:38 -07:00
Jack Forgash
121c3e6a2c
AN-4203/ufc strike metadata (#230)
* ufc strike metadata, plus organized nft folder better

* upd db tag on gold

* upd col name to serial number

* del listing id

* add back listing id
2023-11-20 16:26:39 -07:00
Jack Forgash
821214294c
AN-4096/DEX Swaps Bug (#228)
* add event counts col to easily ignore infra events

* excl txs where to/from is null

* add tests
2023-11-15 17:15:56 -07:00
Jack Forgash
cff19a7bb6
upd test window (#227) 2023-11-14 17:42:25 -07:00
Jack Forgash
80c2462ac4
core v non-core tags and gha workflow (#225) 2023-11-10 14:30:57 -07:00
Jack Forgash
5b096dd83f
upd labels source (#223) 2023-11-07 11:48:11 -07:00
WHYTEWYLL
d17d5f06d7
deprecation (#222)
* deprecation

* del old udf creation

---------

Co-authored-by: forgash_ <jmfxyz@pm.me>
2023-11-07 10:40:24 -07:00
Jack Forgash
cc6f797d71
alter test on bitquery job (#217) 2023-10-31 11:48:27 -06:00
Jack Forgash
eb40b6d91d
Backfill Model - Incorporate all external tables (#211)
* upd sources, convert blocks to loop model, fix collections history

* multiple external tbl query macro and blocks update

* del network version.yml output

* add back rt blocks

* lower blocks

* revert str blocks and collections and move change to history view

* bronze history views and move into folders

* history workflow - blocks

* del history step from streamline blocks

* create complete get history models and upd get history streamline views - BLOCKS

* upd remaining method get history views to use history complete tables

* upd txs history dependency note

* upd history gha and del parameterized models

* fix collections external table config

* collections template fix

* upd txs templates

* upd history views

* fix to tx results template / views

* set history schedule to hourly

* del commented out backfill from streamline RT GHAs

* move blocks filter up to cte
2023-10-26 11:31:19 -06:00
desmond-hui
63ab57f844
An 4065/incremental load tuning (#215)
* modify how dbt performs merges by default

* require at least dbt 1.4

* adjust cluster keys, add incremental pred

* ensure query is being pruned on timestamp

* add cluster key
2023-10-26 07:02:53 -07:00
Jack Forgash
45a07a1da8
An 4050/txs final test (#214)
* test on str txs final for pending response

* add events final gap test

* upd test to check gaps
2023-10-24 11:52:01 -06:00
Jack Forgash
856ac1629f
AN-3935/backfill views (#208)
* archive parameterized model

* create views for each network ver

* network version views

* newline

* switch tx results realtime back to us-east-1

* del blocks history mainnet23

* add backfill step to streamline GHA

* newline

* rm dup WHERE

* upd name and del dup where from template
2023-10-19 13:12:13 -06:00
Jack Forgash
19bbac48a2
Quickfix - Update retry_tx_ids in txs final (#209)
* adj greatest to handle null val

* del if null from retry_tx_ids cte

* add not null test to ins timestamp
2023-10-18 13:23:11 -06:00
Jack Forgash
a18123405e
add pending res to txs final lookback (#207) 2023-10-17 16:55:45 -06:00
Jack Forgash
ea46f3ee02
Adj Test on Beta and Upd Staking Actions (#206) 2023-10-16 18:17:45 -06:00
Jack Forgash
75901993ad
del tests from deprecating core models (#205) 2023-10-16 13:13:28 -06:00
Jack Forgash
bab9e01077
upd collections cte w lookback (#204)
* upd collections cte w lookback

* move lookback to cte
2023-10-13 16:00:16 -06:00
Jack Forgash
753c91d4fa
An 4030/obs model migration (#203)
* upd to streamline models

* add start block var
2023-10-13 13:29:24 -06:00
Jack Forgash
be73c9ba99
Upd workflows for streamline models. Disables CW models. (#202)
* upd workflows for streamline models. Disables CW models.

* repo var

* upd tx_count null test

* ch to 6 hrs
2023-10-12 17:35:59 -06:00
Jack Forgash
bd08ee7f8f
An 4015/streamline cutover (#200)
* core views

* defi schema

* bridge test upd

* gov schema

* nft schema

* price schema

* move tx count to blocks model

* add collection_count_agg to blocks

* upd core views to UNION ALL

* upd collections cluster_by

* upd dim moment metadata to qualify

* upd dim moment metadata test

* upd core dim contract labels to qualify and silver (cw) to incr

* upd str blocks lookback`
2023-10-12 16:23:20 -06:00
Jack Forgash
964f14ab16
upd expiring model to incl usdc (#201) 2023-10-12 09:35:00 -06:00
Jack Forgash
af9a6912ab
del+ins (#199) 2023-10-10 15:56:28 -06:00
Jack Forgash
17bb26999c
Upd tags on workflow to include silver__streamline_events (#198)
* upd tags on scheduled run

* cron to 30 mins
2023-10-10 15:35:43 -06:00
Jack Forgash
50b61ee6d8
add col (#197) 2023-10-10 13:35:09 -06:00
Jack Forgash
792f11599f
AN-3903/Migration Reorg (#196)
* txs final and upd streamline tests w 1 hr inserted timestamp buffer

* copy silver models into silver_cw dir

* txs final, events model, curated model _s copies

* add pending col to txs final

* test config for pending_result_response

* add coalesce to tx id in txs final

* rem tests on inserted timestamp

* uppercase gap test vars

* add retry for null block timestamp and del mainnet23 filter from events

* address adj append 0x, add network ver to blocks, etc

* upd coalesce on events data

* saving event migration test, disabled

* req changes

* upd txs final ingested logic
2023-10-10 13:24:43 -06:00
Jack Forgash
d28d09f53e
AN-3939/add fix gap logic and obs models tests (#195)
* add fix gap logic and obs models tests

* block height recency test on bitquery table

* add tag

* upd workflow

* explicit cast as int
2023-09-15 13:45:28 -06:00
Shah Newaz Khan
c663e7523e
lq & udf_api overload deploy (#186)
* lq overload deploy success

* fixed make directive line continuation

---------

Co-authored-by: shah <info@shahnewazkhan.ca>
2023-09-13 10:26:37 -07:00
Jack Forgash
9386c2af39
Streamline rt - flip blocks priority to desc (#191)
* add desc to RT model to prioritize CH

* flip to desc
2023-09-11 17:09:00 -06:00
Jack Forgash
ad70599d38
upd rt tx results to ohio (#190) 2023-09-08 20:03:46 -06:00
Shah Newaz Khan
bfa514ed7e
added multi region udfs (#189)
Co-authored-by: shah <info@shahnewazkhan.ca>
2023-09-08 18:50:28 -07:00
Jack Forgash
fe7c2b724b
AN-3890/migrate udf (#188)
* upd to live

* migrate to new udf_api deployment

* adjust metadata schedule to daily
2023-09-08 12:52:40 -06:00
Jack Forgash
b1f0925a90
AN-3712/Metadata livequery (#146)
* topshot metadata pull v1

* before swap from lambda

* ready for stress test / batch sizing tmrw

* define unique key in helper tbl

* typo

* add todo deloy udf_api

* slight format tweak

* named

* migrate to livequery schema

* add parameter yml

* add livequery model yml, need tests

* alter metadata job run

* del old streamline macros

* silver upd - in flight

* clean up py model

* limit comment

* revert silver topshot metadata

* new silver model

* union new lq data into nft metadata table(s)

* del block_timestamp from metadata needed view

* untag silver nft allday metadata

* set limit to 3500

* addback topshot tag

* update schedule to hourly
2023-09-07 10:13:02 -06:00
Jack Forgash
f3f362ef54
Add 3 Mainnets 19-21 to Blocks (#182) 2023-09-05 16:44:07 -06:00
Jack Forgash
aca459a005
AN-3898/fix qualify in silver load models (#181)
* fix qualify

* del txs alias
2023-09-05 13:42:49 -06:00
Jack Forgash
92089f7b84
incr limit to 1m (#178) 2023-09-01 18:27:07 -06:00
Jack Forgash
7608c1c611
add last_3_days filter, rem tbl (#177)
* add last_3_days filter, rem tbl

* max

* rem max
2023-09-01 15:03:15 -06:00
Jack Forgash
eca7813bae
Tx Result Status + Run RT Only, at increased interval (#176)
* add tx status filter commented and comment out blocks filter

* incr frequency of RT models

* typo

* format block
2023-08-31 16:26:04 -06:00
Jack Forgash
feded0669f
disregard unsealed current mainnet blocks (#175)
* disregard unsealed current mainnet blocks

* root height from seeds

* move alias
2023-08-31 15:45:06 -06:00
Jack Forgash
caf9f4804c
An 3780/schema reorg (#153)
* folder structures

* new views

* update docs overview

* upd sources

* upd sources and move silver from base to core

* del flipside prod db prices v2

* add scheduled tag to new views
2023-08-31 10:38:17 -06:00
Jack Forgash
b55f82f5ad uncomment RT limit 2023-08-30 18:01:21 -06:00
Jack Forgash
3f3c841115
add limit and col alias to blocks rt (#172)
* add limit and col alias to blocks

* comment out limit
2023-08-30 15:17:28 -06:00
Jack Forgash
38b4e4abc9
upd start height of realtime to mainnet23 root height and add STREAMLINE_RUN_HISTORY to GHA (#171)
* upd start height of realtime to mainnet23 root height and add "STREAMLINE_RUN_HISTORY": True

* add select
2023-08-30 11:06:30 -06:00
WHYTEWYLL
6ce17a7219
Streamline Realtime and Backfill Models (#156)
* add: model to backfill

* fix: node's string format

* add tags

* upd collection model

* mainnet21 hardcoded collections history model

* del tag

* mainnet 21 getblock

* tag for 21

* realtime models

* alias num as height

* realtime tags

* add missing tag and newlines

* backfiller

* backfiller

* move script to folder (renamed) python, upd test accordingly w dir name ch

* upd script to accept model input, update jobs per method call

* error w use_dev arg

* add: silver mdoels

* limit backfill job in python script

* rename silver dbt models to streamline_ and move into silver/core

* explicit casting to silver streamline models

* add documentation to silver streamline models

* run only current mainnet and history mainnet 22 first

* activate schedule for gha

* del hardcoded mainnet models

* move history modes out of subdirs into history dir

* fix GHA vars

* del upstream 1+ from history step

* del tag

---------

Co-authored-by: Jack Forgash <jmfxyz@pm.me>
2023-08-30 08:38:01 -06:00
Jack Forgash
66ba1c1d59
add fr false to obs (#168) 2023-08-28 16:09:25 -06:00
Jack Forgash
d484b0e164
add fr false (#167) 2023-08-28 16:00:47 -06:00
Jack Forgash
6b94cb4234
remove underscore (#166) 2023-08-28 15:59:11 -06:00