Commit Graph

97 Commits

Author SHA1 Message Date
Jack Forgash
3bf75dfd18
fix typo (#340) 2024-06-25 15:01:38 -06:00
Jack Forgash
9127e6de86
AN-4852/Testnet Integration (#326)
* source, bronze, complete, rt views

* complete testnet_ dependency

* upd sources.yml

* upd models and bronze

* upd models.sql

* upd model in bronze views

* adding chainhead_testnet

* add jobs

* fix: hour

* upd api integration

* various fixes. useast2, gha workflows, bad refs, etc

* update udtf_get_base_table

---------

Co-authored-by: WHYTEWYLL <juanmiblanco1@gmail.com>
2024-06-25 12:12:51 -06:00
Jack Forgash
e41b0411f0
adj schedule and rename non core workflow group (#336) 2024-06-20 14:02:35 -06:00
Jack Forgash
64a996b2c7
new workflow/adhoc with alert (#334)
* new workflow

* print run status

* add workflow dispatch for testing

* add temp default for inputs

* add temp default for inputs in steps

* modify echo status after run

* add outcome

* print data

* pull echo into separate step

* define dbt_run_status

* run again

* try intentional failed dbt command

* add emoji to status

* fix single quote

* pass data directly

* del testing defaults

* del unused imports
2024-06-20 10:40:48 -06:00
Jack Forgash
12bf4e2e86
AN-4799/Backfill Migration (#329)
* move models to bronze

* blocks backfill logic ez ez ez

* streamline_load models for all 4 methods

* add back a set of views for the complete models to use

* del dbt run history gha workflow

* migrate txs final to modified ts

* upd events to modified

* set default val for LOAD_BACKFILL_VERSION in project yml

* update curated model incr logic
2024-06-12 12:10:27 -06:00
Jack Forgash
cd3e24bb92
rm 14 and 15 (#327) 2024-05-24 09:35:44 -06:00
Jack Forgash
cc51f4bcca
add 14-16 back for last few (#323) 2024-05-23 11:30:34 -06:00
Jack Forgash
40ef79f4fc
flip to non-batch method (#322) 2024-05-23 11:17:51 -06:00
Jack Forgash
6e288d5661
del 14-16 (#320) 2024-05-06 13:15:08 -06:00
Jack Forgash
c2353ab0f3
rm transactions history - complete (#316) 2024-04-24 09:55:20 -06:00
Jack Forgash
797cb9a430
hourly (#311) 2024-03-26 09:22:32 -06:00
Jack Forgash
a531ee5c98
upd txs away from batch (#299) 2024-03-18 11:28:23 -06:00
WHYTEWYLL
cdc0d9e668
2h (#298) 2024-03-18 10:50:27 -06:00
Jack Forgash
dd2b42d326
fix indent dbt test gha (#290) 2024-03-15 10:34:18 -06:00
WHYTEWYLL
e90fb88156
Fix job (#293)
* fix job

* key

* space

* format

* one line
2024-03-14 17:11:05 -06:00
WHYTEWYLL
605b43092a
fix job (#292)
* fix job

* key

* space
2024-03-14 16:52:02 -06:00
WHYTEWYLL
0dee1c27e9
adding batch methods for chain and cronjob (#284)
Finetune
2024-03-14 15:38:49 -06:00
Jack Forgash
db39b35ceb
upd workflows to use requirements.txt (#289) 2024-03-13 13:54:26 -06:00
Jack Forgash
490b0cdea3
add deps back (#287) 2024-03-13 13:19:07 -06:00
Jack Forgash
2f356f9403
del dbt deps (#285) 2024-03-13 13:12:22 -06:00
WHYTEWYLL
6b65655da8
Finetune (#277)
* test a from the base line on env var

* quotes

* fine tune trasnactions and results

* format

* error

* increasing producer_batch_size

* add: transactions mainnet

* mainet 18

* reducing job

* var del

* increasing producer batch size

* increasing batch size and worker

* workers limit

* reducing producer

* add fin

* fix doubled key values
2024-02-20 12:07:10 -03:00
WHYTEWYLL
1043c83d4f
Finetune (#272)
* test a from the base line on env var

* quotes

* fine tune trasnactions and results

* format

* error

* increasing producer_batch_size

* add: transactions mainnet

* mainet 18

* reducing job

* var del
2024-02-15 17:08:45 -03:00
Jack Forgash
3261a09b4a
fix model selection in history (#269) 2024-02-13 19:06:27 -07:00
Jack Forgash
668b22097b
expliticly call putstanding NV and methods (#268) 2024-02-13 14:50:37 -07:00
WHYTEWYLL
1a6a72f5f2
An 4442 flow dbt1.7 (#261)
* pkg lock and dbt docs

* update: 1.7

* profile

* update docs

---------

Co-authored-by: forgash_ <jmfxyz@pm.me>
2024-01-30 15:44:19 -03:00
Jack Forgash
e0818966ad
AN-4428/add user agent to topshot and upd metadata job (#258)
* add user agent to topshot and upd metadata job

* upd agent

* upd limit to 600
2024-01-18 09:19:18 -07:00
Jack Forgash
ed9f306241
upd schedule to hourly (#254)
* upd schedule to hourly

* revert allday metadata
2024-01-03 16:33:56 -07:00
Jack Forgash
6e666d29a7
upd history cron - daily (#250) 2023-12-19 16:09:14 -05:00
Jack Forgash
9690c3102c
chg schedule to 3x/hour (#244)
* chg schedule to 3x/hour

* blocks

* dbt schd
2023-12-13 19:32:13 -07:00
Jack Forgash
cf26afc365
chg schedule to 4x/hour (#243) 2023-12-13 08:33:54 -07:00
Jack Forgash
c51d0c38b3
incr to 3x/hr (#242) 2023-12-11 16:52:05 -07:00
WHYTEWYLL
d3b3c9667f
Update dbt_run_moments_metadata.yml (#235) 2023-11-29 06:44:14 -08:00
Jack Forgash
df7b62123c
Streamline Backfill: upd workflow activate txs tx results (#233)
* upd workflow activate txs tx results

* upd schedule to 30 mins
2023-11-27 12:22:55 -07:00
desmond-hui
b0c422cfdf
reduce risk of running livequery_models accidentally (#234) 2023-11-27 08:09:06 -08:00
WHYTEWYLL
6d6f478df6
alldaynft metadata with LQ (#229)
* base

* clean up; add: silver model

* add: new standards

* fix: refactor

* delete: folders

* refactor simplying

* remove no longer needed

* fix: refactor to silver layer

* ready to merge

---------

Co-authored-by: Desmond Hui <desmond@flipsidecrypto.com>
2023-11-27 07:55:44 -08:00
Jack Forgash
f7fa714ddc
Update dbt_run_history.yml (#226) 2023-11-10 16:29:25 -07:00
Jack Forgash
80c2462ac4
core v non-core tags and gha workflow (#225) 2023-11-10 14:30:57 -07:00
Jack Forgash
366f25b1d3
fix to 1 line (#220) 2023-10-31 22:06:42 -06:00
Jack Forgash
7f5f9c79b5
upd history to blocks and collections, 5x/hr (#219) 2023-10-31 21:49:01 -06:00
Jack Forgash
59097b6a1f
upd history run to every half hour (#218) 2023-10-31 15:51:24 -06:00
Jack Forgash
cd39800d32
add streamline vars as repo vars and upd history workflow (#216) 2023-10-31 11:24:07 -06:00
Jack Forgash
eb40b6d91d
Backfill Model - Incorporate all external tables (#211)
* upd sources, convert blocks to loop model, fix collections history

* multiple external tbl query macro and blocks update

* del network version.yml output

* add back rt blocks

* lower blocks

* revert str blocks and collections and move change to history view

* bronze history views and move into folders

* history workflow - blocks

* del history step from streamline blocks

* create complete get history models and upd get history streamline views - BLOCKS

* upd remaining method get history views to use history complete tables

* upd txs history dependency note

* upd history gha and del parameterized models

* fix collections external table config

* collections template fix

* upd txs templates

* upd history views

* fix to tx results template / views

* set history schedule to hourly

* del commented out backfill from streamline RT GHAs

* move blocks filter up to cte
2023-10-26 11:31:19 -06:00
Jack Forgash
45a07a1da8
An 4050/txs final test (#214)
* test on str txs final for pending response

* add events final gap test

* upd test to check gaps
2023-10-24 11:52:01 -06:00
eric-laurello
fc5045a44a
schedules (#212) 2023-10-20 15:19:04 -04:00
Jack Forgash
62274b6b64
pause backfill (#210) 2023-10-19 16:16:24 -06:00
Jack Forgash
856ac1629f
AN-3935/backfill views (#208)
* archive parameterized model

* create views for each network ver

* network version views

* newline

* switch tx results realtime back to us-east-1

* del blocks history mainnet23

* add backfill step to streamline GHA

* newline

* rm dup WHERE

* upd name and del dup where from template
2023-10-19 13:12:13 -06:00
Jack Forgash
be73c9ba99
Upd workflows for streamline models. Disables CW models. (#202)
* upd workflows for streamline models. Disables CW models.

* repo var

* upd tx_count null test

* ch to 6 hrs
2023-10-12 17:35:59 -06:00
Jack Forgash
17bb26999c
Upd tags on workflow to include silver__streamline_events (#198)
* upd tags on scheduled run

* cron to 30 mins
2023-10-10 15:35:43 -06:00
Jack Forgash
d28d09f53e
AN-3939/add fix gap logic and obs models tests (#195)
* add fix gap logic and obs models tests

* block height recency test on bitquery table

* add tag

* upd workflow

* explicit cast as int
2023-09-15 13:45:28 -06:00
Jack Forgash
80cee1f37b
incr obs model to 4x/day (#193) 2023-09-12 17:54:15 -06:00