Commit Graph

146 Commits

Author SHA1 Message Date
Jack Forgash
fe7c2b724b
AN-3890/migrate udf (#188)
* upd to live

* migrate to new udf_api deployment

* adjust metadata schedule to daily
2023-09-08 12:52:40 -06:00
Jack Forgash
ed62a557c1
update tx results schedule to 10 mins (#184) 2023-09-07 14:12:12 -06:00
Jack Forgash
b1f0925a90
AN-3712/Metadata livequery (#146)
* topshot metadata pull v1

* before swap from lambda

* ready for stress test / batch sizing tmrw

* define unique key in helper tbl

* typo

* add todo deloy udf_api

* slight format tweak

* named

* migrate to livequery schema

* add parameter yml

* add livequery model yml, need tests

* alter metadata job run

* del old streamline macros

* silver upd - in flight

* clean up py model

* limit comment

* revert silver topshot metadata

* new silver model

* union new lq data into nft metadata table(s)

* del block_timestamp from metadata needed view

* untag silver nft allday metadata

* set limit to 3500

* addback topshot tag

* update schedule to hourly
2023-09-07 10:13:02 -06:00
Jack Forgash
5357fa70cb
Increase RT tx results batching (#183)
* incr tx results producer and worker batch, still 10 concurrent

* adj to 30 workers
2023-09-06 14:30:40 -06:00
Jack Forgash
f3f362ef54
Add 3 Mainnets 19-21 to Blocks (#182) 2023-09-05 16:44:07 -06:00
Jack Forgash
5cedbce02a
add blocks mainnet22 (#179) 2023-09-05 12:47:39 -06:00
Jack Forgash
eca7813bae
Tx Result Status + Run RT Only, at increased interval (#176)
* add tx status filter commented and comment out blocks filter

* incr frequency of RT models

* typo

* format block
2023-08-31 16:26:04 -06:00
Jack Forgash
d821ec8b78
del steps, only call RT and mainnet22 blocks (#174) 2023-08-30 18:02:36 -06:00
Jack Forgash
cb9c6a6b3e
add mainnet21 to streamline history jobs (#173)
* add mainnet21

* newline
2023-08-30 16:23:10 -06:00
Jack Forgash
38b4e4abc9
upd start height of realtime to mainnet23 root height and add STREAMLINE_RUN_HISTORY to GHA (#171)
* upd start height of realtime to mainnet23 root height and add "STREAMLINE_RUN_HISTORY": True

* add select
2023-08-30 11:06:30 -06:00
Jack Forgash
492c3cb352
change to TEST_WAREHOUSE var (#170) 2023-08-30 09:58:57 -06:00
Jack Forgash
f64be0d142
rename dbt streamline hourly workflow (#169) 2023-08-30 08:43:49 -06:00
WHYTEWYLL
6ce17a7219
Streamline Realtime and Backfill Models (#156)
* add: model to backfill

* fix: node's string format

* add tags

* upd collection model

* mainnet21 hardcoded collections history model

* del tag

* mainnet 21 getblock

* tag for 21

* realtime models

* alias num as height

* realtime tags

* add missing tag and newlines

* backfiller

* backfiller

* move script to folder (renamed) python, upd test accordingly w dir name ch

* upd script to accept model input, update jobs per method call

* error w use_dev arg

* add: silver mdoels

* limit backfill job in python script

* rename silver dbt models to streamline_ and move into silver/core

* explicit casting to silver streamline models

* add documentation to silver streamline models

* run only current mainnet and history mainnet 22 first

* activate schedule for gha

* del hardcoded mainnet models

* move history modes out of subdirs into history dir

* fix GHA vars

* del upstream 1+ from history step

* del tag

---------

Co-authored-by: Jack Forgash <jmfxyz@pm.me>
2023-08-30 08:38:01 -06:00
Jack Forgash
241bae5a08
rename yml file (#164) 2023-08-28 10:05:17 -06:00
Jack Forgash
51951df051
An 3859/obs model on (#163)
* obs models on

* rename workflow
2023-08-28 10:02:13 -06:00
WHYTEWYLL
0b0b33b8a0
adjusting job (#162)
* adjusting job

* delete: old job

* reduce to 1 call

---------

Co-authored-by: Jack Forgash <jmfxyz@pm.me>
2023-08-28 09:52:07 -06:00
Jack Forgash
fd6358c9fd
incr to 6 per run and 5 mins (#160) 2023-08-23 15:30:54 -06:00
Jack Forgash
1072c103ce
every minute cron (#159) 2023-08-23 13:44:49 -06:00
Jack Forgash
45f72e71d8
An 3660/observability models (#155)
* blocks completeness

* block tx count external api model

* txs completeness model

* add col alias

* test result agg cte and temp test range

* curr range test

* revert source change - other pr

* xchain silver

* nuke api key & move to secrets, create workflow

* add scheduled tag

* update workflow w cache

* add obs tag

* rem testing range from txs complete

* comment out schedule and del double comman

* add default null for key var
2023-08-23 13:37:16 -06:00
Shah Newaz Khan
2413473512
GHA & Backfill Docs (#149)
* node url join optimized | warehouse privilege grants

* added string interpolation for env based target.database

* revert to candidate node 7 root height

* collections history success

* removed end height from network_version

* removed EOF

* added prod udf

* added tx & tx_result backfill models

* adds FR models

* add prod api integraton

* switched dev flag off

* added gha & docs

* updated serialization png

* added tx specific gha model selection

---------

Co-authored-by: shah <info@shahnewazkhan.ca>
2023-08-16 15:57:04 -07:00
Jack Forgash
f8ee61adf2
upd ad hoc run dbt jobs cmd (#140) 2023-08-02 09:45:20 -06:00
Jack Forgash
76d5ef0dd9
add requirements file (#137) 2023-07-28 09:05:45 -06:00
Jack Forgash
1a4f78ad75
Ad hoc workflow (#136)
* adhoc workflow

* adhoc workflow w logs
2023-07-28 08:39:35 -06:00
Jack Forgash
b6a52d3aae
adj select on scheduled runs (#132) 2023-07-24 11:31:57 -06:00
Jack Forgash
659f9c8cf8
workflow_prod and core->gold (#125) 2023-06-07 11:57:19 -06:00
Jack Forgash
59d72bfea8
add utils package (#122)
* add utils package

* add create udfs

* rename core dir to gold

* vars->inputs
2023-06-06 10:42:38 -06:00
Jack Forgash
aca4c95ec5
update workflows (#116) 2023-04-10 14:15:23 -06:00
Jack Forgash
61171cbeb0
point to other workflow (#115) 2023-04-10 11:37:02 -06:00
Jack Forgash
cca4990bcb del workflow 2 env section 2023-04-07 15:43:27 -06:00
Jack Forgash
a6676efb84 workflow 2023-04-07 15:19:49 -06:00
Jack Forgash
cfe6e21933 workflow 2023-04-07 15:19:22 -06:00
Jack Forgash
57e1908daf update_workflow 2023-04-07 15:16:43 -06:00
Jack Forgash
eb11727ae5 update workflow 2023-04-07 15:12:45 -06:00
Jack Forgash
dd4d7ffef6 add test workflow 2023-04-07 14:58:16 -06:00
Jack Forgash
1c4c93ccd2 del test file 2023-04-07 13:31:02 -06:00
Jack Forgash
a6faca83d7 alter workflow var in test 2023-04-07 13:28:05 -06:00
Jack Forgash
465137c3bf
test (#114) 2023-04-07 13:24:50 -06:00
Jack Forgash
98982fc910
update workflow template (#111)
* update workflow template

* str literal
2023-02-16 16:16:48 -07:00
Jack Forgash
3068287973
escape chars (#106) 2023-01-26 10:31:29 -07:00
Jack Forgash
09aa518880
An 2719/actions labels tweaks (#105)
* use template for GHA

* add custom notification

* move nft models to folder

* put silver models in folder

* swap labels to crosschain

* move script

* move script

* update failed test list if none
2023-01-26 09:24:49 -07:00
Jack Forgash
1196f4f0c5
dirs (#102)
* dirs

* ignore dbt user
2023-01-05 12:29:49 -07:00
Jack Forgash
c1b2099dd8
dbt profile absolute path (#101)
* absolute path

* path
2023-01-05 11:59:13 -07:00
Jack Forgash
9c5faa8dbf
workflow config (#100)
* workflow config

* github actions workflow

* small tweak

* env update for ad hoc

* daily job
2023-01-05 11:47:16 -07:00
Jack Forgash
0307aa4615
target profile (#40) 2022-07-06 16:14:29 -06:00
Jack Forgash
352f4f25f9
Auto-gen docs to separate branch (#22)
* first pass

* test 2

* test 3

* test 4

* test 5

* test 6

* test 6

* test 7

* test 8

* test 9

* test 10

* test 11

* repo token test

* token test 2

* token test 3

* token test 4

* token test 5

* Auto-update docs

* removed with on checkout test

* testing file move

* mkdir command

* no spaces

* Auto-update docs

* update gitignore for target again

* Auto-update docs

* clean target

* Auto-update docs

* will it delete user.yml

* Auto-update docs

* test profile as env

* echo echo

* space

* secretS

* nope, only encrypted

* Auto-update docs

* clean up test profile

* Auto-update docs

* pythonless test

* Auto-update docs

* del blank line

* jk keep python

* Auto-update docs

* docs branch

* docs branch

* docs branch

* manual -b

* checkout main first

* upstream

* set upstream later

* ok now if theres a change

* or

* git

* fetch

* new branch success, what about existing branch

* ff only

* conditional git

* del and create new docs branch

* true

* true elsewhere

* del docs dir

Co-authored-by: github-actions <abc@xyz>
2022-05-25 09:25:04 -06:00
Jack Forgash
f7fe080eef
Automate Docs Generate (#21)
* first pass

* test 2

* test 3

* test 4

* test 5

* test 6

* test 6

* test 7

* test 8

* test 9

* test 10

* test 11

* repo token test

* token test 2

* token test 3

* token test 4

* token test 5

* Auto-update docs

* removed with on checkout test

* testing file move

* mkdir command

* no spaces

* Auto-update docs

* update gitignore for target again

* Auto-update docs

* clean target

* Auto-update docs

* will it delete user.yml

* Auto-update docs

* test profile as env

* echo echo

* space

* secretS

* nope, only encrypted

* Auto-update docs

* clean up test profile

* Auto-update docs

* pythonless test

* Auto-update docs

* del blank line

* jk keep python

Co-authored-by: github-actions <abc@xyz>
2022-05-24 18:34:03 -06:00