Commit Graph

143 Commits

Author SHA1 Message Date
Jack Forgash
ea46f3ee02
Adj Test on Beta and Upd Staking Actions (#206) 2023-10-16 18:17:45 -06:00
Jack Forgash
75901993ad
del tests from deprecating core models (#205) 2023-10-16 13:13:28 -06:00
Jack Forgash
bab9e01077
upd collections cte w lookback (#204)
* upd collections cte w lookback

* move lookback to cte
2023-10-13 16:00:16 -06:00
Jack Forgash
753c91d4fa
An 4030/obs model migration (#203)
* upd to streamline models

* add start block var
2023-10-13 13:29:24 -06:00
Jack Forgash
be73c9ba99
Upd workflows for streamline models. Disables CW models. (#202)
* upd workflows for streamline models. Disables CW models.

* repo var

* upd tx_count null test

* ch to 6 hrs
2023-10-12 17:35:59 -06:00
Jack Forgash
bd08ee7f8f
An 4015/streamline cutover (#200)
* core views

* defi schema

* bridge test upd

* gov schema

* nft schema

* price schema

* move tx count to blocks model

* add collection_count_agg to blocks

* upd core views to UNION ALL

* upd collections cluster_by

* upd dim moment metadata to qualify

* upd dim moment metadata test

* upd core dim contract labels to qualify and silver (cw) to incr

* upd str blocks lookback`
2023-10-12 16:23:20 -06:00
Jack Forgash
964f14ab16
upd expiring model to incl usdc (#201) 2023-10-12 09:35:00 -06:00
Jack Forgash
af9a6912ab
del+ins (#199) 2023-10-10 15:56:28 -06:00
Jack Forgash
17bb26999c
Upd tags on workflow to include silver__streamline_events (#198)
* upd tags on scheduled run

* cron to 30 mins
2023-10-10 15:35:43 -06:00
Jack Forgash
50b61ee6d8
add col (#197) 2023-10-10 13:35:09 -06:00
Jack Forgash
792f11599f
AN-3903/Migration Reorg (#196)
* txs final and upd streamline tests w 1 hr inserted timestamp buffer

* copy silver models into silver_cw dir

* txs final, events model, curated model _s copies

* add pending col to txs final

* test config for pending_result_response

* add coalesce to tx id in txs final

* rem tests on inserted timestamp

* uppercase gap test vars

* add retry for null block timestamp and del mainnet23 filter from events

* address adj append 0x, add network ver to blocks, etc

* upd coalesce on events data

* saving event migration test, disabled

* req changes

* upd txs final ingested logic
2023-10-10 13:24:43 -06:00
Jack Forgash
d28d09f53e
AN-3939/add fix gap logic and obs models tests (#195)
* add fix gap logic and obs models tests

* block height recency test on bitquery table

* add tag

* upd workflow

* explicit cast as int
2023-09-15 13:45:28 -06:00
Shah Newaz Khan
c663e7523e
lq & udf_api overload deploy (#186)
* lq overload deploy success

* fixed make directive line continuation

---------

Co-authored-by: shah <info@shahnewazkhan.ca>
2023-09-13 10:26:37 -07:00
Jack Forgash
9386c2af39
Streamline rt - flip blocks priority to desc (#191)
* add desc to RT model to prioritize CH

* flip to desc
2023-09-11 17:09:00 -06:00
Jack Forgash
ad70599d38
upd rt tx results to ohio (#190) 2023-09-08 20:03:46 -06:00
Shah Newaz Khan
bfa514ed7e
added multi region udfs (#189)
Co-authored-by: shah <info@shahnewazkhan.ca>
2023-09-08 18:50:28 -07:00
Jack Forgash
fe7c2b724b
AN-3890/migrate udf (#188)
* upd to live

* migrate to new udf_api deployment

* adjust metadata schedule to daily
2023-09-08 12:52:40 -06:00
Jack Forgash
b1f0925a90
AN-3712/Metadata livequery (#146)
* topshot metadata pull v1

* before swap from lambda

* ready for stress test / batch sizing tmrw

* define unique key in helper tbl

* typo

* add todo deloy udf_api

* slight format tweak

* named

* migrate to livequery schema

* add parameter yml

* add livequery model yml, need tests

* alter metadata job run

* del old streamline macros

* silver upd - in flight

* clean up py model

* limit comment

* revert silver topshot metadata

* new silver model

* union new lq data into nft metadata table(s)

* del block_timestamp from metadata needed view

* untag silver nft allday metadata

* set limit to 3500

* addback topshot tag

* update schedule to hourly
2023-09-07 10:13:02 -06:00
Jack Forgash
f3f362ef54
Add 3 Mainnets 19-21 to Blocks (#182) 2023-09-05 16:44:07 -06:00
Jack Forgash
aca459a005
AN-3898/fix qualify in silver load models (#181)
* fix qualify

* del txs alias
2023-09-05 13:42:49 -06:00
Jack Forgash
92089f7b84
incr limit to 1m (#178) 2023-09-01 18:27:07 -06:00
Jack Forgash
7608c1c611
add last_3_days filter, rem tbl (#177)
* add last_3_days filter, rem tbl

* max

* rem max
2023-09-01 15:03:15 -06:00
Jack Forgash
eca7813bae
Tx Result Status + Run RT Only, at increased interval (#176)
* add tx status filter commented and comment out blocks filter

* incr frequency of RT models

* typo

* format block
2023-08-31 16:26:04 -06:00
Jack Forgash
feded0669f
disregard unsealed current mainnet blocks (#175)
* disregard unsealed current mainnet blocks

* root height from seeds

* move alias
2023-08-31 15:45:06 -06:00
Jack Forgash
caf9f4804c
An 3780/schema reorg (#153)
* folder structures

* new views

* update docs overview

* upd sources

* upd sources and move silver from base to core

* del flipside prod db prices v2

* add scheduled tag to new views
2023-08-31 10:38:17 -06:00
Jack Forgash
b55f82f5ad uncomment RT limit 2023-08-30 18:01:21 -06:00
Jack Forgash
3f3c841115
add limit and col alias to blocks rt (#172)
* add limit and col alias to blocks

* comment out limit
2023-08-30 15:17:28 -06:00
Jack Forgash
38b4e4abc9
upd start height of realtime to mainnet23 root height and add STREAMLINE_RUN_HISTORY to GHA (#171)
* upd start height of realtime to mainnet23 root height and add "STREAMLINE_RUN_HISTORY": True

* add select
2023-08-30 11:06:30 -06:00
WHYTEWYLL
6ce17a7219
Streamline Realtime and Backfill Models (#156)
* add: model to backfill

* fix: node's string format

* add tags

* upd collection model

* mainnet21 hardcoded collections history model

* del tag

* mainnet 21 getblock

* tag for 21

* realtime models

* alias num as height

* realtime tags

* add missing tag and newlines

* backfiller

* backfiller

* move script to folder (renamed) python, upd test accordingly w dir name ch

* upd script to accept model input, update jobs per method call

* error w use_dev arg

* add: silver mdoels

* limit backfill job in python script

* rename silver dbt models to streamline_ and move into silver/core

* explicit casting to silver streamline models

* add documentation to silver streamline models

* run only current mainnet and history mainnet 22 first

* activate schedule for gha

* del hardcoded mainnet models

* move history modes out of subdirs into history dir

* fix GHA vars

* del upstream 1+ from history step

* del tag

---------

Co-authored-by: Jack Forgash <jmfxyz@pm.me>
2023-08-30 08:38:01 -06:00
Jack Forgash
66ba1c1d59
add fr false to obs (#168) 2023-08-28 16:09:25 -06:00
Jack Forgash
d484b0e164
add fr false (#167) 2023-08-28 16:00:47 -06:00
Jack Forgash
6b94cb4234
remove underscore (#166) 2023-08-28 15:59:11 -06:00
Jack Forgash
51951df051
An 3859/obs model on (#163)
* obs models on

* rename workflow
2023-08-28 10:02:13 -06:00
Jack Forgash
45f72e71d8
An 3660/observability models (#155)
* blocks completeness

* block tx count external api model

* txs completeness model

* add col alias

* test result agg cte and temp test range

* curr range test

* revert source change - other pr

* xchain silver

* nuke api key & move to secrets, create workflow

* add scheduled tag

* update workflow w cache

* add obs tag

* rem testing range from txs complete

* comment out schedule and del double comman

* add default null for key var
2023-08-23 13:37:16 -06:00
Shah Newaz Khan
7e98bd6029
bug fix for non block spine coll/tx ingestion (#157)
Co-authored-by: shah <info@shahnewazkhan.ca>
2023-08-23 12:13:40 -07:00
Shah Newaz Khan
f3a5b863e1
Stream 392 fix prod (#154)
* node url join optimized | warehouse privilege grants

* added udf api

* revert udtf alias

* fixed incremental complete runs

* node url join optimized | warehouse privilege grants

* added udf api

* reverted case insentive filename changes

---------

Co-authored-by: shah <info@shahnewazkhan.ca>
2023-08-22 12:01:55 -07:00
Shah Newaz Khan
1beb5f8ad6
Stream 392 lq udf (#151)
* node url join optimized | warehouse privilege grants

* added udf api

* revert udtf alias

---------

Co-authored-by: shah <info@shahnewazkhan.ca>
2023-08-22 10:29:42 -07:00
Shah Newaz Khan
2413473512
GHA & Backfill Docs (#149)
* node url join optimized | warehouse privilege grants

* added string interpolation for env based target.database

* revert to candidate node 7 root height

* collections history success

* removed end height from network_version

* removed EOF

* added prod udf

* added tx & tx_result backfill models

* adds FR models

* add prod api integraton

* switched dev flag off

* added gha & docs

* updated serialization png

* added tx specific gha model selection

---------

Co-authored-by: shah <info@shahnewazkhan.ca>
2023-08-16 15:57:04 -07:00
Shah Newaz Khan
fe7f688cc2
Stream 98 collection backfill (#145)
* node url join optimized | warehouse privilege grants

* added string interpolation for env based target.database

* revert to candidate node 7 root height

* collections history success

* removed end height from network_version

* removed EOF

* added prod udf

---------

Co-authored-by: shah <info@shahnewazkhan.ca>
2023-08-10 14:23:29 -07:00
Shah Newaz Khan
d14843bd0c
Stream 98 fix nodeurl (#141)
* fixed node urls

* limited streamline_blocks to candidate node 7 root height

---------

Co-authored-by: shah <info@shahnewazkhan.ca>
2023-08-03 12:30:13 -07:00
Jack Forgash
67790931a4
Quickfix 080323/bad cast (#142)
* update cast to str

* add second str cast

* int
2023-08-03 11:07:39 -06:00
Shah Newaz Khan
b344373700
Stream 98 node url seed (#138)
* added clone purge fix make directive

* fixed json parsing for get blocks history

* added aws_lambda_role envar to undo_clone directive | added database grants to lambda role

* get_node_url parsing error

* fixed varchar return type for get_node_url

* node mapping window function success

* removed get_node_url macro | added node_url seed

* removed debug logs | removed create node_url macro

* added filter to node_mapping CTE

* Update models/silver/streamline/core/streamline__blocks.sql

* fixed merge conflict

* Reduce columns for get_blocks_history

---------

Co-authored-by: shah <info@shahnewazkhan.ca>
2023-08-01 14:56:15 -07:00
Jack Forgash
12e951471d
upd slack msg (#139)
* upd slack msg

* Update dbt_test_alert.py

* !

* alter tests
2023-08-01 10:29:14 -06:00
Shah Newaz Khan
4a1596b0d0
Stream 97 blocks assembly (#134)
* added clone purge fix make directive

* fixed json parsing for get blocks history

* added aws_lambda_role envar to undo_clone directive | added database grants to lambda role

* removed debug logs

---------

Co-authored-by: shah <info@shahnewazkhan.ca>
2023-07-31 12:54:18 -07:00
Jack Forgash
7552fb7dfe
seed and test adj (#133)
* seed and test adj

* rm_typo
2023-07-25 11:00:32 -06:00
Shah Newaz Khan
e39845bc3d
Stream 97 streamline blocks (#129)
* udf_bulk_grpc success

* removed test analysis sql

---------

Co-authored-by: shah <info@shahnewazkhan.ca>
2023-07-24 09:24:52 -07:00
Shah Newaz Khan
cecced9db1
streamline flow chainhead udf (#128)
* added makefile udf/api integration directives | added api integration / udf macros

* added reamdme comments

---------

Co-authored-by: shah <info@shahnewazkhan.ca>
2023-07-18 13:04:21 -07:00
WHYTEWYLL
de09e6e00f
add: 248 celer id (#127) 2023-06-27 12:07:39 -07:00
Jack Forgash
d25b02df55
Prices from Swaps update (#126)
* workflow_prod and core->gold

* update prices from swaps
2023-06-07 16:42:53 -06:00
Jack Forgash
dce8ba2ec9
move allday error filter to silver (#124) 2023-06-07 13:45:07 -04:00