flip/flow-models
1
0
Fork 0
mirror of https://github.com/FlipsideCrypto/flow-models.git synced 2026-02-06 14:22:06 +00:00
Code Issues Actions 29 Packages Projects Releases Wiki Activity
479 Commits 103 Branches 0 Tags 26 MiB
main
Commit Graph

479 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Jack Forgash
aca459a005
AN-3898/fix qualify in silver load models (#181) 
* fix qualify

* del txs alias
 2023-09-05 13:42:49 -06:00
Jack Forgash
5cedbce02a
add blocks mainnet22 (#179) 2023-09-05 12:47:39 -06:00
Jack Forgash
92089f7b84
incr limit to 1m (#178) 2023-09-01 18:27:07 -06:00
Jack Forgash
7608c1c611
add last_3_days filter, rem tbl (#177) 
* add last_3_days filter, rem tbl

* max

* rem max
 2023-09-01 15:03:15 -06:00
Jack Forgash
eca7813bae
Tx Result Status + Run RT Only, at increased interval (#176) 
* add tx status filter commented and comment out blocks filter

* incr frequency of RT models

* typo

* format block
 2023-08-31 16:26:04 -06:00
Jack Forgash
feded0669f
disregard unsealed current mainnet blocks (#175) 
* disregard unsealed current mainnet blocks

* root height from seeds

* move alias
 2023-08-31 15:45:06 -06:00
Jack Forgash
caf9f4804c
An 3780/schema reorg (#153) 
* folder structures

* new views

* update docs overview

* upd sources

* upd sources and move silver from base to core

* del flipside prod db prices v2

* add scheduled tag to new views
 2023-08-31 10:38:17 -06:00
Jack Forgash
d821ec8b78
del steps, only call RT and mainnet22 blocks (#174) 2023-08-30 18:02:36 -06:00
Jack Forgash
b55f82f5ad uncomment RT limit 2023-08-30 18:01:21 -06:00
Jack Forgash
cb9c6a6b3e
add mainnet21 to streamline history jobs (#173) 
* add mainnet21

* newline
 2023-08-30 16:23:10 -06:00
Jack Forgash
3f3c841115
add limit and col alias to blocks rt (#172) 
* add limit and col alias to blocks

* comment out limit
 2023-08-30 15:17:28 -06:00
Jack Forgash
38b4e4abc9
upd start height of realtime to mainnet23 root height and add STREAMLINE_RUN_HISTORY to GHA (#171) 
* upd start height of realtime to mainnet23 root height and add "STREAMLINE_RUN_HISTORY": True

* add select
 2023-08-30 11:06:30 -06:00
Jack Forgash
492c3cb352
change to TEST_WAREHOUSE var (#170) 2023-08-30 09:58:57 -06:00
Jack Forgash
f64be0d142
rename dbt streamline hourly workflow (#169) 2023-08-30 08:43:49 -06:00
WHYTEWYLL
6ce17a7219
Streamline Realtime and Backfill Models (#156) 
* add: model to backfill

* fix: node's string format

* add tags

* upd collection model

* mainnet21 hardcoded collections history model

* del tag

* mainnet 21 getblock

* tag for 21

* realtime models

* alias num as height

* realtime tags

* add missing tag and newlines

* backfiller

* backfiller

* move script to folder (renamed) python, upd test accordingly w dir name ch

* upd script to accept model input, update jobs per method call

* error w use_dev arg

* add: silver mdoels

* limit backfill job in python script

* rename silver dbt models to streamline_ and move into silver/core

* explicit casting to silver streamline models

* add documentation to silver streamline models

* run only current mainnet and history mainnet 22 first

* activate schedule for gha

* del hardcoded mainnet models

* move history modes out of subdirs into history dir

* fix GHA vars

* del upstream 1+ from history step

* del tag

---------

Co-authored-by: Jack Forgash <jmfxyz@pm.me>
 2023-08-30 08:38:01 -06:00
Shah Newaz Khan
920d811c90
update dev udf urls (#165) 
Co-authored-by: shah <info@shahnewazkhan.ca>
 2023-08-28 15:58:53 -07:00
Jack Forgash
66ba1c1d59
add fr false to obs (#168) 2023-08-28 16:09:25 -06:00
Jack Forgash
d484b0e164
add fr false (#167) 2023-08-28 16:00:47 -06:00
Jack Forgash
6b94cb4234
remove underscore (#166) 2023-08-28 15:59:11 -06:00
Jack Forgash
241bae5a08
rename yml file (#164) 2023-08-28 10:05:17 -06:00
Jack Forgash
51951df051
An 3859/obs model on (#163) 
* obs models on

* rename workflow
 2023-08-28 10:02:13 -06:00
WHYTEWYLL
0b0b33b8a0
adjusting job (#162) 
* adjusting job

* delete: old job

* reduce to 1 call

---------

Co-authored-by: Jack Forgash <jmfxyz@pm.me>
 2023-08-28 09:52:07 -06:00
Jack Forgash
fd6358c9fd
incr to 6 per run and 5 mins (#160) 2023-08-23 15:30:54 -06:00
Jack Forgash
1072c103ce
every minute cron (#159) 2023-08-23 13:44:49 -06:00
Jack Forgash
45f72e71d8
An 3660/observability models (#155) 
* blocks completeness

* block tx count external api model

* txs completeness model

* add col alias

* test result agg cte and temp test range

* curr range test

* revert source change - other pr

* xchain silver

* nuke api key & move to secrets, create workflow

* add scheduled tag

* update workflow w cache

* add obs tag

* rem testing range from txs complete

* comment out schedule and del double comman

* add default null for key var
 2023-08-23 13:37:16 -06:00
Shah Newaz Khan
2d4753ffda
added STREAMLINE_INVOKE_STREAMS (#158) 
Co-authored-by: shah <info@shahnewazkhan.ca>
 2023-08-23 12:22:01 -07:00
Shah Newaz Khan
7e98bd6029
bug fix for non block spine coll/tx ingestion (#157) 
Co-authored-by: shah <info@shahnewazkhan.ca>
 2023-08-23 12:13:40 -07:00
Shah Newaz Khan
f3a5b863e1
Stream 392 fix prod (#154) 
* node url join optimized | warehouse privilege grants

* added udf api

* revert udtf alias

* fixed incremental complete runs

* node url join optimized | warehouse privilege grants

* added udf api

* reverted case insentive filename changes

---------

Co-authored-by: shah <info@shahnewazkhan.ca>
 2023-08-22 12:01:55 -07:00
Shah Newaz Khan
1beb5f8ad6
Stream 392 lq udf (#151) 
* node url join optimized | warehouse privilege grants

* added udf api

* revert udtf alias

---------

Co-authored-by: shah <info@shahnewazkhan.ca>
 2023-08-22 10:29:42 -07:00
Shah Newaz Khan
2413473512
GHA & Backfill Docs (#149) 
* node url join optimized | warehouse privilege grants

* added string interpolation for env based target.database

* revert to candidate node 7 root height

* collections history success

* removed end height from network_version

* removed EOF

* added prod udf

* added tx & tx_result backfill models

* adds FR models

* add prod api integraton

* switched dev flag off

* added gha & docs

* updated serialization png

* added tx specific gha model selection

---------

Co-authored-by: shah <info@shahnewazkhan.ca>
 2023-08-16 15:57:04 -07:00
Shah Newaz Khan
fe7f688cc2
Stream 98 collection backfill (#145) 
* node url join optimized | warehouse privilege grants

* added string interpolation for env based target.database

* revert to candidate node 7 root height

* collections history success

* removed end height from network_version

* removed EOF

* added prod udf

---------

Co-authored-by: shah <info@shahnewazkhan.ca>
 2023-08-10 14:23:29 -07:00
Shah Newaz Khan
d14843bd0c
Stream 98 fix nodeurl (#141) 
* fixed node urls

* limited streamline_blocks to candidate node 7 root height

---------

Co-authored-by: shah <info@shahnewazkhan.ca>
 2023-08-03 12:30:13 -07:00
Jack Forgash
67790931a4
Quickfix 080323/bad cast (#142) 
* update cast to str

* add second str cast

* int
 2023-08-03 11:07:39 -06:00
Jack Forgash
f8ee61adf2
upd ad hoc run dbt jobs cmd (#140) 2023-08-02 09:45:20 -06:00
Shah Newaz Khan
b344373700
Stream 98 node url seed (#138) 
* added clone purge fix make directive

* fixed json parsing for get blocks history

* added aws_lambda_role envar to undo_clone directive | added database grants to lambda role

* get_node_url parsing error

* fixed varchar return type for get_node_url

* node mapping window function success

* removed get_node_url macro | added node_url seed

* removed debug logs | removed create node_url macro

* added filter to node_mapping CTE

* Update models/silver/streamline/core/streamline__blocks.sql

* fixed merge conflict

* Reduce columns for get_blocks_history

---------

Co-authored-by: shah <info@shahnewazkhan.ca>
 2023-08-01 14:56:15 -07:00
Jack Forgash
12e951471d
upd slack msg (#139) 
* upd slack msg

* Update dbt_test_alert.py

* !

* alter tests
 2023-08-01 10:29:14 -06:00
Shah Newaz Khan
4a1596b0d0
Stream 97 blocks assembly (#134) 
* added clone purge fix make directive

* fixed json parsing for get blocks history

* added aws_lambda_role envar to undo_clone directive | added database grants to lambda role

* removed debug logs

---------

Co-authored-by: shah <info@shahnewazkhan.ca>
 2023-07-31 12:54:18 -07:00
Jack Forgash
76d5ef0dd9
add requirements file (#137) 2023-07-28 09:05:45 -06:00
Jack Forgash
1a4f78ad75
Ad hoc workflow (#136) 
* adhoc workflow

* adhoc workflow w logs
 2023-07-28 08:39:35 -06:00
Jack Forgash
7552fb7dfe
seed and test adj (#133) 
* seed and test adj

* rm_typo
 2023-07-25 11:00:32 -06:00
Jack Forgash
b6a52d3aae
adj select on scheduled runs (#132) 2023-07-24 11:31:57 -06:00
Shah Newaz Khan
e39845bc3d
Stream 97 streamline blocks (#129) 
* udf_bulk_grpc success

* removed test analysis sql

---------

Co-authored-by: shah <info@shahnewazkhan.ca>
 2023-07-24 09:24:52 -07:00
Jack Forgash
a06fac90ba
update version seed (#131) 
* update version sewed

* newline
 2023-07-23 09:28:22 -06:00
Shah Newaz Khan
cecced9db1
streamline flow chainhead udf (#128) 
* added makefile udf/api integration directives | added api integration / udf macros

* added reamdme comments

---------

Co-authored-by: shah <info@shahnewazkhan.ca>
 2023-07-18 13:04:21 -07:00
WHYTEWYLL
de09e6e00f
add: 248 celer id (#127) 2023-06-27 12:07:39 -07:00
Jack Forgash
d25b02df55
Prices from Swaps update (#126) 
* workflow_prod and core->gold

* update prices from swaps
 2023-06-07 16:42:53 -06:00
Jack Forgash
659f9c8cf8
workflow_prod and core->gold (#125) 2023-06-07 11:57:19 -06:00
Jack Forgash
dce8ba2ec9
move allday error filter to silver (#124) 2023-06-07 13:45:07 -04:00
Jack Forgash
eb20a6c924
fix create udfs (#123) 
* fix create udfs

* move_block
 2023-06-06 11:33:54 -06:00
Jack Forgash
59d72bfea8
add utils package (#122) 
* add utils package

* add create udfs

* rename core dir to gold

* vars->inputs
 2023-06-06 10:42:38 -06:00