Commit Graph

72 Commits

Author SHA1 Message Date
Jack Forgash
e0818966ad
AN-4428/add user agent to topshot and upd metadata job (#258)
* add user agent to topshot and upd metadata job

* upd agent

* upd limit to 600
2024-01-18 09:19:18 -07:00
Jack Forgash
ed9f306241
upd schedule to hourly (#254)
* upd schedule to hourly

* revert allday metadata
2024-01-03 16:33:56 -07:00
Jack Forgash
6e666d29a7
upd history cron - daily (#250) 2023-12-19 16:09:14 -05:00
Jack Forgash
9690c3102c
chg schedule to 3x/hour (#244)
* chg schedule to 3x/hour

* blocks

* dbt schd
2023-12-13 19:32:13 -07:00
Jack Forgash
cf26afc365
chg schedule to 4x/hour (#243) 2023-12-13 08:33:54 -07:00
Jack Forgash
c51d0c38b3
incr to 3x/hr (#242) 2023-12-11 16:52:05 -07:00
WHYTEWYLL
d3b3c9667f
Update dbt_run_moments_metadata.yml (#235) 2023-11-29 06:44:14 -08:00
Jack Forgash
df7b62123c
Streamline Backfill: upd workflow activate txs tx results (#233)
* upd workflow activate txs tx results

* upd schedule to 30 mins
2023-11-27 12:22:55 -07:00
desmond-hui
b0c422cfdf
reduce risk of running livequery_models accidentally (#234) 2023-11-27 08:09:06 -08:00
WHYTEWYLL
6d6f478df6
alldaynft metadata with LQ (#229)
* base

* clean up; add: silver model

* add: new standards

* fix: refactor

* delete: folders

* refactor simplying

* remove no longer needed

* fix: refactor to silver layer

* ready to merge

---------

Co-authored-by: Desmond Hui <desmond@flipsidecrypto.com>
2023-11-27 07:55:44 -08:00
Jack Forgash
f7fa714ddc
Update dbt_run_history.yml (#226) 2023-11-10 16:29:25 -07:00
Jack Forgash
80c2462ac4
core v non-core tags and gha workflow (#225) 2023-11-10 14:30:57 -07:00
Jack Forgash
366f25b1d3
fix to 1 line (#220) 2023-10-31 22:06:42 -06:00
Jack Forgash
7f5f9c79b5
upd history to blocks and collections, 5x/hr (#219) 2023-10-31 21:49:01 -06:00
Jack Forgash
59097b6a1f
upd history run to every half hour (#218) 2023-10-31 15:51:24 -06:00
Jack Forgash
cd39800d32
add streamline vars as repo vars and upd history workflow (#216) 2023-10-31 11:24:07 -06:00
Jack Forgash
eb40b6d91d
Backfill Model - Incorporate all external tables (#211)
* upd sources, convert blocks to loop model, fix collections history

* multiple external tbl query macro and blocks update

* del network version.yml output

* add back rt blocks

* lower blocks

* revert str blocks and collections and move change to history view

* bronze history views and move into folders

* history workflow - blocks

* del history step from streamline blocks

* create complete get history models and upd get history streamline views - BLOCKS

* upd remaining method get history views to use history complete tables

* upd txs history dependency note

* upd history gha and del parameterized models

* fix collections external table config

* collections template fix

* upd txs templates

* upd history views

* fix to tx results template / views

* set history schedule to hourly

* del commented out backfill from streamline RT GHAs

* move blocks filter up to cte
2023-10-26 11:31:19 -06:00
Jack Forgash
45a07a1da8
An 4050/txs final test (#214)
* test on str txs final for pending response

* add events final gap test

* upd test to check gaps
2023-10-24 11:52:01 -06:00
eric-laurello
fc5045a44a
schedules (#212) 2023-10-20 15:19:04 -04:00
Jack Forgash
62274b6b64
pause backfill (#210) 2023-10-19 16:16:24 -06:00
Jack Forgash
856ac1629f
AN-3935/backfill views (#208)
* archive parameterized model

* create views for each network ver

* network version views

* newline

* switch tx results realtime back to us-east-1

* del blocks history mainnet23

* add backfill step to streamline GHA

* newline

* rm dup WHERE

* upd name and del dup where from template
2023-10-19 13:12:13 -06:00
Jack Forgash
be73c9ba99
Upd workflows for streamline models. Disables CW models. (#202)
* upd workflows for streamline models. Disables CW models.

* repo var

* upd tx_count null test

* ch to 6 hrs
2023-10-12 17:35:59 -06:00
Jack Forgash
17bb26999c
Upd tags on workflow to include silver__streamline_events (#198)
* upd tags on scheduled run

* cron to 30 mins
2023-10-10 15:35:43 -06:00
Jack Forgash
d28d09f53e
AN-3939/add fix gap logic and obs models tests (#195)
* add fix gap logic and obs models tests

* block height recency test on bitquery table

* add tag

* upd workflow

* explicit cast as int
2023-09-15 13:45:28 -06:00
Jack Forgash
80cee1f37b
incr obs model to 4x/day (#193) 2023-09-12 17:54:15 -06:00
Jack Forgash
e487cd7e60
upd blocks backfill schedule, del 20 and 21 (#192) 2023-09-12 10:17:52 -06:00
Jack Forgash
fe7c2b724b
AN-3890/migrate udf (#188)
* upd to live

* migrate to new udf_api deployment

* adjust metadata schedule to daily
2023-09-08 12:52:40 -06:00
Jack Forgash
ed62a557c1
update tx results schedule to 10 mins (#184) 2023-09-07 14:12:12 -06:00
Jack Forgash
b1f0925a90
AN-3712/Metadata livequery (#146)
* topshot metadata pull v1

* before swap from lambda

* ready for stress test / batch sizing tmrw

* define unique key in helper tbl

* typo

* add todo deloy udf_api

* slight format tweak

* named

* migrate to livequery schema

* add parameter yml

* add livequery model yml, need tests

* alter metadata job run

* del old streamline macros

* silver upd - in flight

* clean up py model

* limit comment

* revert silver topshot metadata

* new silver model

* union new lq data into nft metadata table(s)

* del block_timestamp from metadata needed view

* untag silver nft allday metadata

* set limit to 3500

* addback topshot tag

* update schedule to hourly
2023-09-07 10:13:02 -06:00
Jack Forgash
5357fa70cb
Increase RT tx results batching (#183)
* incr tx results producer and worker batch, still 10 concurrent

* adj to 30 workers
2023-09-06 14:30:40 -06:00
Jack Forgash
f3f362ef54
Add 3 Mainnets 19-21 to Blocks (#182) 2023-09-05 16:44:07 -06:00
Jack Forgash
5cedbce02a
add blocks mainnet22 (#179) 2023-09-05 12:47:39 -06:00
Jack Forgash
eca7813bae
Tx Result Status + Run RT Only, at increased interval (#176)
* add tx status filter commented and comment out blocks filter

* incr frequency of RT models

* typo

* format block
2023-08-31 16:26:04 -06:00
Jack Forgash
d821ec8b78
del steps, only call RT and mainnet22 blocks (#174) 2023-08-30 18:02:36 -06:00
Jack Forgash
cb9c6a6b3e
add mainnet21 to streamline history jobs (#173)
* add mainnet21

* newline
2023-08-30 16:23:10 -06:00
Jack Forgash
38b4e4abc9
upd start height of realtime to mainnet23 root height and add STREAMLINE_RUN_HISTORY to GHA (#171)
* upd start height of realtime to mainnet23 root height and add "STREAMLINE_RUN_HISTORY": True

* add select
2023-08-30 11:06:30 -06:00
Jack Forgash
492c3cb352
change to TEST_WAREHOUSE var (#170) 2023-08-30 09:58:57 -06:00
Jack Forgash
f64be0d142
rename dbt streamline hourly workflow (#169) 2023-08-30 08:43:49 -06:00
WHYTEWYLL
6ce17a7219
Streamline Realtime and Backfill Models (#156)
* add: model to backfill

* fix: node's string format

* add tags

* upd collection model

* mainnet21 hardcoded collections history model

* del tag

* mainnet 21 getblock

* tag for 21

* realtime models

* alias num as height

* realtime tags

* add missing tag and newlines

* backfiller

* backfiller

* move script to folder (renamed) python, upd test accordingly w dir name ch

* upd script to accept model input, update jobs per method call

* error w use_dev arg

* add: silver mdoels

* limit backfill job in python script

* rename silver dbt models to streamline_ and move into silver/core

* explicit casting to silver streamline models

* add documentation to silver streamline models

* run only current mainnet and history mainnet 22 first

* activate schedule for gha

* del hardcoded mainnet models

* move history modes out of subdirs into history dir

* fix GHA vars

* del upstream 1+ from history step

* del tag

---------

Co-authored-by: Jack Forgash <jmfxyz@pm.me>
2023-08-30 08:38:01 -06:00
Jack Forgash
241bae5a08
rename yml file (#164) 2023-08-28 10:05:17 -06:00
Jack Forgash
51951df051
An 3859/obs model on (#163)
* obs models on

* rename workflow
2023-08-28 10:02:13 -06:00
WHYTEWYLL
0b0b33b8a0
adjusting job (#162)
* adjusting job

* delete: old job

* reduce to 1 call

---------

Co-authored-by: Jack Forgash <jmfxyz@pm.me>
2023-08-28 09:52:07 -06:00
Jack Forgash
fd6358c9fd
incr to 6 per run and 5 mins (#160) 2023-08-23 15:30:54 -06:00
Jack Forgash
1072c103ce
every minute cron (#159) 2023-08-23 13:44:49 -06:00
Jack Forgash
45f72e71d8
An 3660/observability models (#155)
* blocks completeness

* block tx count external api model

* txs completeness model

* add col alias

* test result agg cte and temp test range

* curr range test

* revert source change - other pr

* xchain silver

* nuke api key & move to secrets, create workflow

* add scheduled tag

* update workflow w cache

* add obs tag

* rem testing range from txs complete

* comment out schedule and del double comman

* add default null for key var
2023-08-23 13:37:16 -06:00
Shah Newaz Khan
2413473512
GHA & Backfill Docs (#149)
* node url join optimized | warehouse privilege grants

* added string interpolation for env based target.database

* revert to candidate node 7 root height

* collections history success

* removed end height from network_version

* removed EOF

* added prod udf

* added tx & tx_result backfill models

* adds FR models

* add prod api integraton

* switched dev flag off

* added gha & docs

* updated serialization png

* added tx specific gha model selection

---------

Co-authored-by: shah <info@shahnewazkhan.ca>
2023-08-16 15:57:04 -07:00
Jack Forgash
f8ee61adf2
upd ad hoc run dbt jobs cmd (#140) 2023-08-02 09:45:20 -06:00
Jack Forgash
76d5ef0dd9
add requirements file (#137) 2023-07-28 09:05:45 -06:00
Jack Forgash
1a4f78ad75
Ad hoc workflow (#136)
* adhoc workflow

* adhoc workflow w logs
2023-07-28 08:39:35 -06:00
Jack Forgash
b6a52d3aae
adj select on scheduled runs (#132) 2023-07-24 11:31:57 -06:00