Commit Graph

37 Commits

Author SHA1 Message Date
shah
96c18610c7 fix udf 2 api integration 2024-02-13 08:05:28 -08:00
WHYTEWYLL
2fe9333eea
add: user agent to not get ban (#257) 2024-01-10 14:29:37 -03:00
Jack Forgash
971d555dff
update prod clone macro (#240) 2023-12-06 13:12:46 -07:00
WHYTEWYLL
9dae9f240f
Pipeline metadata (#238)
* add: if condition

* removed row_list

* add:  gold view, fix macro, and silver models

* gold view extension

* add: test bronce

* g

* adding test

* add: test

* add: working test

* delete: tags

* delete: hardcode ref
2023-12-04 07:16:32 -08:00
WHYTEWYLL
6d6f478df6
alldaynft metadata with LQ (#229)
* base

* clean up; add: silver model

* add: new standards

* fix: refactor

* delete: folders

* refactor simplying

* remove no longer needed

* fix: refactor to silver layer

* ready to merge

---------

Co-authored-by: Desmond Hui <desmond@flipsidecrypto.com>
2023-11-27 07:55:44 -08:00
WHYTEWYLL
d17d5f06d7
deprecation (#222)
* deprecation

* del old udf creation

---------

Co-authored-by: forgash_ <jmfxyz@pm.me>
2023-11-07 10:40:24 -07:00
Jack Forgash
eb40b6d91d
Backfill Model - Incorporate all external tables (#211)
* upd sources, convert blocks to loop model, fix collections history

* multiple external tbl query macro and blocks update

* del network version.yml output

* add back rt blocks

* lower blocks

* revert str blocks and collections and move change to history view

* bronze history views and move into folders

* history workflow - blocks

* del history step from streamline blocks

* create complete get history models and upd get history streamline views - BLOCKS

* upd remaining method get history views to use history complete tables

* upd txs history dependency note

* upd history gha and del parameterized models

* fix collections external table config

* collections template fix

* upd txs templates

* upd history views

* fix to tx results template / views

* set history schedule to hourly

* del commented out backfill from streamline RT GHAs

* move blocks filter up to cte
2023-10-26 11:31:19 -06:00
desmond-hui
8ff2e1aa07
modify how dbt performs merges by default (#213) 2023-10-23 10:04:29 -07:00
Jack Forgash
792f11599f
AN-3903/Migration Reorg (#196)
* txs final and upd streamline tests w 1 hr inserted timestamp buffer

* copy silver models into silver_cw dir

* txs final, events model, curated model _s copies

* add pending col to txs final

* test config for pending_result_response

* add coalesce to tx id in txs final

* rem tests on inserted timestamp

* uppercase gap test vars

* add retry for null block timestamp and del mainnet23 filter from events

* address adj append 0x, add network ver to blocks, etc

* upd coalesce on events data

* saving event migration test, disabled

* req changes

* upd txs final ingested logic
2023-10-10 13:24:43 -06:00
Shah Newaz Khan
aaacbcf7e7
add UPDATE_LQ_UDFS project var (#194)
* add UPDATE_LQ_UDFS project var

* rename UPDATE_LQ_UDFS to UPDATE_EPHEMERAL_UDFS

---------

Co-authored-by: shah <info@shahnewazkhan.ca>
2023-09-13 10:52:15 -07:00
Shah Newaz Khan
c663e7523e
lq & udf_api overload deploy (#186)
* lq overload deploy success

* fixed make directive line continuation

---------

Co-authored-by: shah <info@shahnewazkhan.ca>
2023-09-13 10:26:37 -07:00
Shah Newaz Khan
bfa514ed7e
added multi region udfs (#189)
Co-authored-by: shah <info@shahnewazkhan.ca>
2023-09-08 18:50:28 -07:00
Jack Forgash
4877011c99
add select tags (#185) 2023-09-08 08:33:59 -04:00
Jack Forgash
b1f0925a90
AN-3712/Metadata livequery (#146)
* topshot metadata pull v1

* before swap from lambda

* ready for stress test / batch sizing tmrw

* define unique key in helper tbl

* typo

* add todo deloy udf_api

* slight format tweak

* named

* migrate to livequery schema

* add parameter yml

* add livequery model yml, need tests

* alter metadata job run

* del old streamline macros

* silver upd - in flight

* clean up py model

* limit comment

* revert silver topshot metadata

* new silver model

* union new lq data into nft metadata table(s)

* del block_timestamp from metadata needed view

* untag silver nft allday metadata

* set limit to 3500

* addback topshot tag

* update schedule to hourly
2023-09-07 10:13:02 -06:00
Jack Forgash
d8c023ba46
An 3897/streamline tests (#180) 2023-09-05 16:44:38 -06:00
Shah Newaz Khan
920d811c90
update dev udf urls (#165)
Co-authored-by: shah <info@shahnewazkhan.ca>
2023-08-28 15:58:53 -07:00
Shah Newaz Khan
f3a5b863e1
Stream 392 fix prod (#154)
* node url join optimized | warehouse privilege grants

* added udf api

* revert udtf alias

* fixed incremental complete runs

* node url join optimized | warehouse privilege grants

* added udf api

* reverted case insentive filename changes

---------

Co-authored-by: shah <info@shahnewazkhan.ca>
2023-08-22 12:01:55 -07:00
Shah Newaz Khan
1beb5f8ad6
Stream 392 lq udf (#151)
* node url join optimized | warehouse privilege grants

* added udf api

* revert udtf alias

---------

Co-authored-by: shah <info@shahnewazkhan.ca>
2023-08-22 10:29:42 -07:00
Shah Newaz Khan
2413473512
GHA & Backfill Docs (#149)
* node url join optimized | warehouse privilege grants

* added string interpolation for env based target.database

* revert to candidate node 7 root height

* collections history success

* removed end height from network_version

* removed EOF

* added prod udf

* added tx & tx_result backfill models

* adds FR models

* add prod api integraton

* switched dev flag off

* added gha & docs

* updated serialization png

* added tx specific gha model selection

---------

Co-authored-by: shah <info@shahnewazkhan.ca>
2023-08-16 15:57:04 -07:00
Shah Newaz Khan
fe7f688cc2
Stream 98 collection backfill (#145)
* node url join optimized | warehouse privilege grants

* added string interpolation for env based target.database

* revert to candidate node 7 root height

* collections history success

* removed end height from network_version

* removed EOF

* added prod udf

---------

Co-authored-by: shah <info@shahnewazkhan.ca>
2023-08-10 14:23:29 -07:00
Shah Newaz Khan
b344373700
Stream 98 node url seed (#138)
* added clone purge fix make directive

* fixed json parsing for get blocks history

* added aws_lambda_role envar to undo_clone directive | added database grants to lambda role

* get_node_url parsing error

* fixed varchar return type for get_node_url

* node mapping window function success

* removed get_node_url macro | added node_url seed

* removed debug logs | removed create node_url macro

* added filter to node_mapping CTE

* Update models/silver/streamline/core/streamline__blocks.sql

* fixed merge conflict

* Reduce columns for get_blocks_history

---------

Co-authored-by: shah <info@shahnewazkhan.ca>
2023-08-01 14:56:15 -07:00
Shah Newaz Khan
4a1596b0d0
Stream 97 blocks assembly (#134)
* added clone purge fix make directive

* fixed json parsing for get blocks history

* added aws_lambda_role envar to undo_clone directive | added database grants to lambda role

* removed debug logs

---------

Co-authored-by: shah <info@shahnewazkhan.ca>
2023-07-31 12:54:18 -07:00
Shah Newaz Khan
e39845bc3d
Stream 97 streamline blocks (#129)
* udf_bulk_grpc success

* removed test analysis sql

---------

Co-authored-by: shah <info@shahnewazkhan.ca>
2023-07-24 09:24:52 -07:00
Shah Newaz Khan
cecced9db1
streamline flow chainhead udf (#128)
* added makefile udf/api integration directives | added api integration / udf macros

* added reamdme comments

---------

Co-authored-by: shah <info@shahnewazkhan.ca>
2023-07-18 13:04:21 -07:00
Jack Forgash
eb20a6c924
fix create udfs (#123)
* fix create udfs

* move_block
2023-06-06 11:33:54 -06:00
Jack Forgash
59d72bfea8
add utils package (#122)
* add utils package

* add create udfs

* rename core dir to gold

* vars->inputs
2023-06-06 10:42:38 -06:00
Jack Forgash
ea19cd3857
tags fix (#119) 2023-05-04 12:39:55 -06:00
Jack Forgash
6e6fbc3bdd
tags (#104) 2023-01-23 10:59:46 -07:00
eric-laurello
c5dc787f21
add custom query tags (#97) 2022-12-13 16:22:42 -08:00
desmond-hui
ed39c69c7a
add resources for community curation (#75) 2022-10-17 16:32:28 -07:00
Jack Forgash
861e667d4b
AN-1870/allday ids (#51)
* allday nft ids

* silver topshot view for compat

* limit allday to 3000

* str

* add nfl all day metadata integration and refactor streamline process

* no longer needed

Co-authored-by: Desmond Hui <desmond@flipsidecrypto.com>
2022-08-26 11:07:40 -07:00
Jack Forgash
edb39921fa
AN-1807/store tests (#53)
* store tests

* dbt 1.2

* test names and off for ingested at
2022-08-19 14:13:02 -06:00
desmond-hui
ab3c73e994
An 1709/make dev refresh dbt cloud op (#38)
* dev refresh will now run on dbt cloud

* remove unintended file commit
2022-06-30 18:07:25 -07:00
desmond-hui
a1bdcbb529
An 1625/integrate moments minted api (#34)
* clone stages to dev

* add api integration for topshot metadata
2022-06-28 09:24:09 -07:00
desmond-hui
8279eaac27
clone stages to dev (#30) 2022-06-21 09:20:06 -07:00
Jack Forgash
a956618336
AN-1201/tests (#18)
* tests

* docs

* date filter

* and

* events
2022-05-19 17:53:07 -06:00
Desmond Hui
fdc4e6cf12 initial setup files 2022-04-14 11:34:12 -07:00