Commit Graph

31 Commits

Author SHA1 Message Date
eric-laurello
8699abd562
tags (#380) 2024-11-21 15:12:49 -05:00
Jack Forgash
70d0393cfa
AN-5196 coreevm final (#356)
* an-5198-upd-block-gap-test

* evm block gap test

* evm gaps tests

* blocks and txs tests

* logs tests

* add evm gap test to daily test gha

* run evm job in separate gha workflow
2024-09-09 12:02:03 -06:00
Jack Forgash
02a587c10d
uncomment query-comment (#351) 2024-08-21 16:22:29 -06:00
Jack Forgash
30ce62f6a0
AN-4936/Backfill Migration Final (#335)
* del cw models

* core views

* defi views

* gov view

* nft_views

* rm ref to old cw models

* del internal cols

* nv 13-15

* nv 13-15

* nv 17-18

* nv 20

* del temp logic

* upd tests on allday metadata

* upd to warn
2024-07-01 13:27:42 -06:00
Jack Forgash
965d037877
upd aws_flow_api_dev_2 (#338)
* upd aws_flow_api_dev_2

* update everywhere
2024-06-25 12:45:19 -06:00
Jack Forgash
12bf4e2e86
AN-4799/Backfill Migration (#329)
* move models to bronze

* blocks backfill logic ez ez ez

* streamline_load models for all 4 methods

* add back a set of views for the complete models to use

* del dbt run history gha workflow

* migrate txs final to modified ts

* upd events to modified

* set default val for LOAD_BACKFILL_VERSION in project yml

* update curated model incr logic
2024-06-12 12:10:27 -06:00
WHYTEWYLL
e7615a0cc6
Update labels (#262)
* a

* lavels source updated

* add core view

* unique id

* delete coalesces
2024-01-31 13:44:59 -03:00
WHYTEWYLL
1a6a72f5f2
An 4442 flow dbt1.7 (#261)
* pkg lock and dbt docs

* update: 1.7

* profile

* update docs

---------

Co-authored-by: forgash_ <jmfxyz@pm.me>
2024-01-30 15:44:19 -03:00
WHYTEWYLL
6d6f478df6
alldaynft metadata with LQ (#229)
* base

* clean up; add: silver model

* add: new standards

* fix: refactor

* delete: folders

* refactor simplying

* remove no longer needed

* fix: refactor to silver layer

* ready to merge

---------

Co-authored-by: Desmond Hui <desmond@flipsidecrypto.com>
2023-11-27 07:55:44 -08:00
desmond-hui
63ab57f844
An 4065/incremental load tuning (#215)
* modify how dbt performs merges by default

* require at least dbt 1.4

* adjust cluster keys, add incremental pred

* ensure query is being pruned on timestamp

* add cluster key
2023-10-26 07:02:53 -07:00
Jack Forgash
bd08ee7f8f
An 4015/streamline cutover (#200)
* core views

* defi schema

* bridge test upd

* gov schema

* nft schema

* price schema

* move tx count to blocks model

* add collection_count_agg to blocks

* upd core views to UNION ALL

* upd collections cluster_by

* upd dim moment metadata to qualify

* upd dim moment metadata test

* upd core dim contract labels to qualify and silver (cw) to incr

* upd str blocks lookback`
2023-10-12 16:23:20 -06:00
Shah Newaz Khan
aaacbcf7e7
add UPDATE_LQ_UDFS project var (#194)
* add UPDATE_LQ_UDFS project var

* rename UPDATE_LQ_UDFS to UPDATE_EPHEMERAL_UDFS

---------

Co-authored-by: shah <info@shahnewazkhan.ca>
2023-09-13 10:52:15 -07:00
Shah Newaz Khan
c663e7523e
lq & udf_api overload deploy (#186)
* lq overload deploy success

* fixed make directive line continuation

---------

Co-authored-by: shah <info@shahnewazkhan.ca>
2023-09-13 10:26:37 -07:00
Shah Newaz Khan
bfa514ed7e
added multi region udfs (#189)
Co-authored-by: shah <info@shahnewazkhan.ca>
2023-09-08 18:50:28 -07:00
Jack Forgash
4877011c99
add select tags (#185) 2023-09-08 08:33:59 -04:00
WHYTEWYLL
6ce17a7219
Streamline Realtime and Backfill Models (#156)
* add: model to backfill

* fix: node's string format

* add tags

* upd collection model

* mainnet21 hardcoded collections history model

* del tag

* mainnet 21 getblock

* tag for 21

* realtime models

* alias num as height

* realtime tags

* add missing tag and newlines

* backfiller

* backfiller

* move script to folder (renamed) python, upd test accordingly w dir name ch

* upd script to accept model input, update jobs per method call

* error w use_dev arg

* add: silver mdoels

* limit backfill job in python script

* rename silver dbt models to streamline_ and move into silver/core

* explicit casting to silver streamline models

* add documentation to silver streamline models

* run only current mainnet and history mainnet 22 first

* activate schedule for gha

* del hardcoded mainnet models

* move history modes out of subdirs into history dir

* fix GHA vars

* del upstream 1+ from history step

* del tag

---------

Co-authored-by: Jack Forgash <jmfxyz@pm.me>
2023-08-30 08:38:01 -06:00
Jack Forgash
45f72e71d8
An 3660/observability models (#155)
* blocks completeness

* block tx count external api model

* txs completeness model

* add col alias

* test result agg cte and temp test range

* curr range test

* revert source change - other pr

* xchain silver

* nuke api key & move to secrets, create workflow

* add scheduled tag

* update workflow w cache

* add obs tag

* rem testing range from txs complete

* comment out schedule and del double comman

* add default null for key var
2023-08-23 13:37:16 -06:00
Shah Newaz Khan
2d4753ffda
added STREAMLINE_INVOKE_STREAMS (#158)
Co-authored-by: shah <info@shahnewazkhan.ca>
2023-08-23 12:22:01 -07:00
Shah Newaz Khan
2413473512
GHA & Backfill Docs (#149)
* node url join optimized | warehouse privilege grants

* added string interpolation for env based target.database

* revert to candidate node 7 root height

* collections history success

* removed end height from network_version

* removed EOF

* added prod udf

* added tx & tx_result backfill models

* adds FR models

* add prod api integraton

* switched dev flag off

* added gha & docs

* updated serialization png

* added tx specific gha model selection

---------

Co-authored-by: shah <info@shahnewazkhan.ca>
2023-08-16 15:57:04 -07:00
Jack Forgash
659f9c8cf8
workflow_prod and core->gold (#125) 2023-06-07 11:57:19 -06:00
Jack Forgash
eb20a6c924
fix create udfs (#123)
* fix create udfs

* move_block
2023-06-06 11:33:54 -06:00
Jack Forgash
6e6fbc3bdd
tags (#104) 2023-01-23 10:59:46 -07:00
Jack Forgash
861e667d4b
AN-1870/allday ids (#51)
* allday nft ids

* silver topshot view for compat

* limit allday to 3000

* str

* add nfl all day metadata integration and refactor streamline process

* no longer needed

Co-authored-by: Desmond Hui <desmond@flipsidecrypto.com>
2022-08-26 11:07:40 -07:00
Jack Forgash
edb39921fa
AN-1807/store tests (#53)
* store tests

* dbt 1.2

* test names and off for ingested at
2022-08-19 14:13:02 -06:00
Jack Forgash
f0e071cc6a
AN-1714/prices from swaps (#37)
* init silver model

* cluster by block_timestamp

* column fixes

* prices from swaps, need to update core to add symbol & contract

* token labels seed and updated core view

* docs
2022-07-06 16:36:41 -04:00
desmond-hui
ab3c73e994
An 1709/make dev refresh dbt cloud op (#38)
* dev refresh will now run on dbt cloud

* remove unintended file commit
2022-06-30 18:07:25 -07:00
Jack Forgash
6854a94ed1
AN-1638/inserted_timestamp (#35)
* bronze

* del netlify

* append column

* cluster

* incremental logic

* bridge tables

* docs
2022-06-30 13:12:08 -06:00
desmond-hui
a1bdcbb529
An 1625/integrate moments minted api (#34)
* clone stages to dev

* add api integration for topshot metadata
2022-06-28 09:24:09 -07:00
Jack Forgash
7d1b08adfa
An 1340/tests (#20)
* severity

* docs

* del docs files
2022-05-25 15:39:35 -06:00
Jack Forgash
c3a240d429
AN-1121 Initial push of Silver Transactions (#3)
* tx structure

* yml, qualify, ingested_at

* tests

* block_height

* tests, format, rename error column
2022-05-02 16:01:51 -06:00
Desmond Hui
fdc4e6cf12 initial setup files 2022-04-14 11:34:12 -07:00