Commit Graph

235 Commits

Author SHA1 Message Date
WHYTEWYLL
1cfb107b46 fix: dedeuplicates 2023-11-29 13:41:41 -03:00
WHYTEWYLL
3e5e0a4bdd
Update silver__nft_allday_metadata_s.sql (#236) 2023-11-29 08:29:39 -08:00
WHYTEWYLL
d3b3c9667f
Update dbt_run_moments_metadata.yml (#235) 2023-11-29 06:44:14 -08:00
Jack Forgash
df7b62123c
Streamline Backfill: upd workflow activate txs tx results (#233)
* upd workflow activate txs tx results

* upd schedule to 30 mins
2023-11-27 12:22:55 -07:00
desmond-hui
b0c422cfdf
reduce risk of running livequery_models accidentally (#234) 2023-11-27 08:09:06 -08:00
WHYTEWYLL
6d6f478df6
alldaynft metadata with LQ (#229)
* base

* clean up; add: silver model

* add: new standards

* fix: refactor

* delete: folders

* refactor simplying

* remove no longer needed

* fix: refactor to silver layer

* ready to merge

---------

Co-authored-by: Desmond Hui <desmond@flipsidecrypto.com>
2023-11-27 07:55:44 -08:00
Jack Forgash
b33dc89a39
swap set description col (#232)
* swap set description col

* cast str
2023-11-20 16:39:47 -07:00
Jack Forgash
c82c04ad5d
del invocation id from gold (#231) 2023-11-20 16:30:38 -07:00
Jack Forgash
121c3e6a2c
AN-4203/ufc strike metadata (#230)
* ufc strike metadata, plus organized nft folder better

* upd db tag on gold

* upd col name to serial number

* del listing id

* add back listing id
2023-11-20 16:26:39 -07:00
Jack Forgash
821214294c
AN-4096/DEX Swaps Bug (#228)
* add event counts col to easily ignore infra events

* excl txs where to/from is null

* add tests
2023-11-15 17:15:56 -07:00
Jack Forgash
cff19a7bb6
upd test window (#227) 2023-11-14 17:42:25 -07:00
Jack Forgash
f7fa714ddc
Update dbt_run_history.yml (#226) 2023-11-10 16:29:25 -07:00
Jack Forgash
80c2462ac4
core v non-core tags and gha workflow (#225) 2023-11-10 14:30:57 -07:00
Jack Forgash
fc83740b50
upd nv csv (#224) 2023-11-08 09:37:24 -07:00
Jack Forgash
5b096dd83f
upd labels source (#223) 2023-11-07 11:48:11 -07:00
WHYTEWYLL
d17d5f06d7
deprecation (#222)
* deprecation

* del old udf creation

---------

Co-authored-by: forgash_ <jmfxyz@pm.me>
2023-11-07 10:40:24 -07:00
Jack Forgash
366f25b1d3
fix to 1 line (#220) 2023-10-31 22:06:42 -06:00
Jack Forgash
7f5f9c79b5
upd history to blocks and collections, 5x/hr (#219) 2023-10-31 21:49:01 -06:00
Jack Forgash
59097b6a1f
upd history run to every half hour (#218) 2023-10-31 15:51:24 -06:00
Jack Forgash
cc6f797d71
alter test on bitquery job (#217) 2023-10-31 11:48:27 -06:00
Jack Forgash
cd39800d32
add streamline vars as repo vars and upd history workflow (#216) 2023-10-31 11:24:07 -06:00
Jack Forgash
eb40b6d91d
Backfill Model - Incorporate all external tables (#211)
* upd sources, convert blocks to loop model, fix collections history

* multiple external tbl query macro and blocks update

* del network version.yml output

* add back rt blocks

* lower blocks

* revert str blocks and collections and move change to history view

* bronze history views and move into folders

* history workflow - blocks

* del history step from streamline blocks

* create complete get history models and upd get history streamline views - BLOCKS

* upd remaining method get history views to use history complete tables

* upd txs history dependency note

* upd history gha and del parameterized models

* fix collections external table config

* collections template fix

* upd txs templates

* upd history views

* fix to tx results template / views

* set history schedule to hourly

* del commented out backfill from streamline RT GHAs

* move blocks filter up to cte
2023-10-26 11:31:19 -06:00
desmond-hui
63ab57f844
An 4065/incremental load tuning (#215)
* modify how dbt performs merges by default

* require at least dbt 1.4

* adjust cluster keys, add incremental pred

* ensure query is being pruned on timestamp

* add cluster key
2023-10-26 07:02:53 -07:00
Jack Forgash
45a07a1da8
An 4050/txs final test (#214)
* test on str txs final for pending response

* add events final gap test

* upd test to check gaps
2023-10-24 11:52:01 -06:00
desmond-hui
8ff2e1aa07
modify how dbt performs merges by default (#213) 2023-10-23 10:04:29 -07:00
eric-laurello
fc5045a44a
schedules (#212) 2023-10-20 15:19:04 -04:00
Jack Forgash
62274b6b64
pause backfill (#210) 2023-10-19 16:16:24 -06:00
Jack Forgash
856ac1629f
AN-3935/backfill views (#208)
* archive parameterized model

* create views for each network ver

* network version views

* newline

* switch tx results realtime back to us-east-1

* del blocks history mainnet23

* add backfill step to streamline GHA

* newline

* rm dup WHERE

* upd name and del dup where from template
2023-10-19 13:12:13 -06:00
Jack Forgash
19bbac48a2
Quickfix - Update retry_tx_ids in txs final (#209)
* adj greatest to handle null val

* del if null from retry_tx_ids cte

* add not null test to ins timestamp
2023-10-18 13:23:11 -06:00
Jack Forgash
a18123405e
add pending res to txs final lookback (#207) 2023-10-17 16:55:45 -06:00
Jack Forgash
ea46f3ee02
Adj Test on Beta and Upd Staking Actions (#206) 2023-10-16 18:17:45 -06:00
Jack Forgash
75901993ad
del tests from deprecating core models (#205) 2023-10-16 13:13:28 -06:00
Jack Forgash
bab9e01077
upd collections cte w lookback (#204)
* upd collections cte w lookback

* move lookback to cte
2023-10-13 16:00:16 -06:00
Jack Forgash
753c91d4fa
An 4030/obs model migration (#203)
* upd to streamline models

* add start block var
2023-10-13 13:29:24 -06:00
Jack Forgash
be73c9ba99
Upd workflows for streamline models. Disables CW models. (#202)
* upd workflows for streamline models. Disables CW models.

* repo var

* upd tx_count null test

* ch to 6 hrs
2023-10-12 17:35:59 -06:00
Jack Forgash
bd08ee7f8f
An 4015/streamline cutover (#200)
* core views

* defi schema

* bridge test upd

* gov schema

* nft schema

* price schema

* move tx count to blocks model

* add collection_count_agg to blocks

* upd core views to UNION ALL

* upd collections cluster_by

* upd dim moment metadata to qualify

* upd dim moment metadata test

* upd core dim contract labels to qualify and silver (cw) to incr

* upd str blocks lookback`
2023-10-12 16:23:20 -06:00
Jack Forgash
964f14ab16
upd expiring model to incl usdc (#201) 2023-10-12 09:35:00 -06:00
Jack Forgash
af9a6912ab
del+ins (#199) 2023-10-10 15:56:28 -06:00
Jack Forgash
17bb26999c
Upd tags on workflow to include silver__streamline_events (#198)
* upd tags on scheduled run

* cron to 30 mins
2023-10-10 15:35:43 -06:00
Jack Forgash
50b61ee6d8
add col (#197) 2023-10-10 13:35:09 -06:00
Jack Forgash
792f11599f
AN-3903/Migration Reorg (#196)
* txs final and upd streamline tests w 1 hr inserted timestamp buffer

* copy silver models into silver_cw dir

* txs final, events model, curated model _s copies

* add pending col to txs final

* test config for pending_result_response

* add coalesce to tx id in txs final

* rem tests on inserted timestamp

* uppercase gap test vars

* add retry for null block timestamp and del mainnet23 filter from events

* address adj append 0x, add network ver to blocks, etc

* upd coalesce on events data

* saving event migration test, disabled

* req changes

* upd txs final ingested logic
2023-10-10 13:24:43 -06:00
Jack Forgash
d28d09f53e
AN-3939/add fix gap logic and obs models tests (#195)
* add fix gap logic and obs models tests

* block height recency test on bitquery table

* add tag

* upd workflow

* explicit cast as int
2023-09-15 13:45:28 -06:00
Shah Newaz Khan
aaacbcf7e7
add UPDATE_LQ_UDFS project var (#194)
* add UPDATE_LQ_UDFS project var

* rename UPDATE_LQ_UDFS to UPDATE_EPHEMERAL_UDFS

---------

Co-authored-by: shah <info@shahnewazkhan.ca>
2023-09-13 10:52:15 -07:00
Shah Newaz Khan
c663e7523e
lq & udf_api overload deploy (#186)
* lq overload deploy success

* fixed make directive line continuation

---------

Co-authored-by: shah <info@shahnewazkhan.ca>
2023-09-13 10:26:37 -07:00
Jack Forgash
80cee1f37b
incr obs model to 4x/day (#193) 2023-09-12 17:54:15 -06:00
Jack Forgash
e487cd7e60
upd blocks backfill schedule, del 20 and 21 (#192) 2023-09-12 10:17:52 -06:00
Jack Forgash
9386c2af39
Streamline rt - flip blocks priority to desc (#191)
* add desc to RT model to prioritize CH

* flip to desc
2023-09-11 17:09:00 -06:00
Jack Forgash
ad70599d38
upd rt tx results to ohio (#190) 2023-09-08 20:03:46 -06:00
Shah Newaz Khan
bfa514ed7e
added multi region udfs (#189)
Co-authored-by: shah <info@shahnewazkhan.ca>
2023-09-08 18:50:28 -07:00
Jack Forgash
fe7c2b724b
AN-3890/migrate udf (#188)
* upd to live

* migrate to new udf_api deployment

* adjust metadata schedule to daily
2023-09-08 12:52:40 -06:00