Commit Graph

6 Commits

Author SHA1 Message Date
drethereum
d4147c16f6
Add/quantum-prices-sl (#318)
* quantum for cg metadata

* test query

* Update revision for fsc-utils package to "STREAM-844-RC1"

* Update revision for fsc-utils package to "foobar"

* quantum structures

* spacing

* merge conflicts
\

* v1.22

* remove limit

---------

Co-authored-by: Julius Remigio <14811322+juls858@users.noreply.github.com>
2024-04-15 12:46:08 -06:00
drethereum
59dec0dd02
AN-4377/prices-update (#267)
* test tables

* draft

* unique id

* streamline reorg and silver join

* folder

* asset metadata

* folders

* legacy models

* tags

* comment

* json

* streamline models 2

* asset metadata streamline tests

* complete model

* comments

* asset metadata incremental

* realtime params and filter

* incremental logic on assets

* spacing

* inserted timestamp in complete

* workers

* configured realtime, history, complete and bronze

* inserted timestamp from info schema

* history vars, calls, silver impute logic

* comments

* asset metadata source in v1

* cast

* workflow changes

* typo

* stats in workflow

* integrate changes for fsc-utils

* ref

* dbt_project config

* dbt project

* v1.20.1

* history sched

* bump fsc-utils to 1.20.2

* limits

---------

Co-authored-by: shah <info@shahnewazkhan.ca>
2024-03-05 13:45:02 -07:00
Shah Newaz Khan
f6dcb6ab37
Stream 559 udf rest api (#256)
* added realtime OHLC generic rest api model

* added api_integration macro

* fixed url param delimter

* add v2 suffix

---------

Co-authored-by: shah <info@shahnewazkhan.ca>
2024-02-08 00:09:32 -08:00
Shah Newaz Khan
c27f535a69
Stream 384 fan out backfill (#248)
* remove qualify clause | add udf args to fan out to 9 lambdas

* remove qualify clause

* remove qualify clause from complete model

* set key distribution arg params | success 7 concurrent lambdas

* reduce udf invocation to 15 mins frequency

* finalize key distribution udf args

---------

Co-authored-by: shah <info@shahnewazkhan.ca>
2024-01-28 08:54:19 -08:00
Shah Newaz Khan
f34b2c1083
added backfill complete model (#232)
* added backfill complete model

* added inserted_at logic

* tuned udf params

* added make ohlc dev make directive | update get cg prices udf params to 30 min

---------

Co-authored-by: shah <info@shahnewazkhan.ca>
2024-01-05 17:25:46 -08:00
Shah Newaz Khan
6b30fbc606
Stream 346 coingecko backfill (#224)
* prices udf success

* removed unused make directives

* removed unussed ohlc udf

* added CTE to generate tokens per chain

* generating time spine from dim_dates

* add dim_date ref()

* added uid filter to runtime cte

* added generic get_prices make directive

* added uid as unique key to complete_get_prices

* removed platform specific get prices model

* removed platform filter for get_prices_history

* fixed udf ponter for get prices

* added backfill gha

* fixed gha dbt command

* add  flag

* add role echo

* revert adhoc run roles echo

* added input type to adhoc run gha

* set use external dev tables

* cache deps

* revert cache

* text STREAMLINE_USE_DEV_FOR_EXTERNAL_TABLES envar

* fix envar capitalization

* echo vars

* revert var echo

* set update udfs to false for backfill gha

* moved UPDATE_UDFS_AND_SPS to dbt_project.yml

* update to use UID

* partition by uid

* added crossjoin uid to runtimes cte

* complete / runtimes

* use bronze view

* added udf params | added erro exclusion

* SO and cluster

* added prices backfill tag | renamed complete model

* pre-pended streamline to tags

* renamed concise tags

* renamed sl prices_history model | added appropriate tags to gha

---------

Co-authored-by: shah <info@shahnewazkhan.ca>
Co-authored-by: Austin <austin@flipsidecrypto.com>
2023-12-21 08:16:01 -08:00