Commit Graph

51 Commits

Author SHA1 Message Date
eric-laurello
b49b884663
protocol level stats (#500)
* protocol level

* use new stats

* typo
2025-07-23 15:57:41 -04:00
drethereum
02554bd0d5
versions (#463) 2025-06-02 15:38:58 -06:00
tarikceric
e255edcc5f
unifai sl ingestion bronze and silver (#441)
* unifai sl ingestion bronze and silver

* use call_function_v2 and add exploded_key

* use exploded key

* adjust float values

* use data col

* use secret

* add chain to qualify and pk

* update worker
2025-05-06 11:25:35 -07:00
drethereum
69fb49dd0f
macro comment (#440) 2025-04-25 13:19:49 -06:00
drethereum
6af82555d1
macro (#439) 2025-04-25 15:17:33 -04:00
drethereum
398df8f163
updates for fsc-evm migrations (#437)
* updates for fsc-evm migrations

* update workflows for slack notify and sources
2025-04-23 13:18:37 -06:00
drethereum
70338b810a
fix/prod-clone-owner (#360)
* revert

* change ref

* dispatch
2024-06-18 12:59:40 -06:00
drethereum
5d7ae42287
AN-4897/add-price-test (#355)
* custom test

* remove
2024-06-13 13:14:24 -06:00
WHYTEWYLL
bb14976e2f
Fix vaults token key name and adding tests (#344)
* adjust vault dr

* test and token rename

* fix_test

* right path

* rename test
2024-05-08 08:28:29 -07:00
WHYTEWYLL
3411b12a72
adjust vault dr (#343) 2024-05-07 13:00:57 -07:00
drethereum
fa258ad3ea
AN-4628/native-prices (#314)
* initial models

* updates for symbol

* complete and views

* updated native seed

* unique key

* native prices

* native assets unioned into gold views

* native

* is_native

* docs

* decimals

* 18

* docs
2024-04-05 14:58:51 -06:00
drethereum
e6d4c774e1
update/priority-prices2-inc (#308)
* heal for priority prices

* lookback period

* heal logic and test update

* complete models for performance

* provider column
2024-04-01 13:58:38 -06:00
drethereum
82332272c8
AN-4377/rebuild silver prices (#298)
* imputing period

* gap test

* >0

* cmc imputing

* all providers draft

* filters

* edits to asset metadata and last timestamp

* comments

* source

* token address adj

* comments

* rebuild all providers asset metadata

* rebuild all providers prices

* gap test filter

* typo

* osmo 2

* osmo2 yml

* lookback

* is deprecated

* tag for new models

* remove tag

* https filter

* http filter

* inner join solana cmc

* http

* final adj

* more filters

* trim quote

* denom extracted
2024-03-19 09:16:28 -06:00
drethereum
0f30a4b0a2
add tasks (#296) 2024-03-14 16:20:31 -06:00
drethereum
59dec0dd02
AN-4377/prices-update (#267)
* test tables

* draft

* unique id

* streamline reorg and silver join

* folder

* asset metadata

* folders

* legacy models

* tags

* comment

* json

* streamline models 2

* asset metadata streamline tests

* complete model

* comments

* asset metadata incremental

* realtime params and filter

* incremental logic on assets

* spacing

* inserted timestamp in complete

* workers

* configured realtime, history, complete and bronze

* inserted timestamp from info schema

* history vars, calls, silver impute logic

* comments

* asset metadata source in v1

* cast

* workflow changes

* typo

* stats in workflow

* integrate changes for fsc-utils

* ref

* dbt_project config

* dbt project

* v1.20.1

* history sched

* bump fsc-utils to 1.20.2

* limits

---------

Co-authored-by: shah <info@shahnewazkhan.ca>
2024-03-05 13:45:02 -07:00
Shah Newaz Khan
37bbc3d157
Stream 556 udf rest api macro (#264)
* add prod api_integration

* add udf param legibility macro

* fix doc string for api integration

* add param defaults

* configured prod v2 udf

* added stg/dev api integration

* added if_data_v2

---------

Co-authored-by: shah <info@shahnewazkhan.ca>
2024-02-16 11:28:20 -08:00
Shah Newaz Khan
f6dcb6ab37
Stream 559 udf rest api (#256)
* added realtime OHLC generic rest api model

* added api_integration macro

* fixed url param delimter

* add v2 suffix

---------

Co-authored-by: shah <info@shahnewazkhan.ca>
2024-02-08 00:09:32 -08:00
Shah Newaz Khan
6b30fbc606
Stream 346 coingecko backfill (#224)
* prices udf success

* removed unused make directives

* removed unussed ohlc udf

* added CTE to generate tokens per chain

* generating time spine from dim_dates

* add dim_date ref()

* added uid filter to runtime cte

* added generic get_prices make directive

* added uid as unique key to complete_get_prices

* removed platform specific get prices model

* removed platform filter for get_prices_history

* fixed udf ponter for get prices

* added backfill gha

* fixed gha dbt command

* add  flag

* add role echo

* revert adhoc run roles echo

* added input type to adhoc run gha

* set use external dev tables

* cache deps

* revert cache

* text STREAMLINE_USE_DEV_FOR_EXTERNAL_TABLES envar

* fix envar capitalization

* echo vars

* revert var echo

* set update udfs to false for backfill gha

* moved UPDATE_UDFS_AND_SPS to dbt_project.yml

* update to use UID

* partition by uid

* added crossjoin uid to runtimes cte

* complete / runtimes

* use bronze view

* added udf params | added erro exclusion

* SO and cluster

* added prices backfill tag | renamed complete model

* pre-pended streamline to tags

* renamed concise tags

* renamed sl prices_history model | added appropriate tags to gha

---------

Co-authored-by: shah <info@shahnewazkhan.ca>
Co-authored-by: Austin <austin@flipsidecrypto.com>
2023-12-21 08:16:01 -08:00
desmond-hui
07d4b399ba
Rework near GitHub retry logic (#223)
* update sp macro

* reducing lambda

* second debug

* wait fix

* fix batch

* try just with 1

* simplify batch results and retry logic

* make response_data table temporary

* remove unused var

* fix syntax

* remove union, limit 1000 endpoints per run

* run every hour

* add comment

* do not execute if 0 rows or remaining calls less than needed for a batch

---------

Co-authored-by: WHYTEWYLL <juanmiblanco1@gmail.com>
2023-11-20 14:56:14 -08:00
WHYTEWYLL
482497db52
GitHub procedure (#219)
* update sp macro

* adding target
2023-11-10 10:49:09 -08:00
WHYTEWYLL
7af7cfa212
update sp macro (#212) 2023-11-06 07:35:26 -08:00
WHYTEWYLL
6ff1c026cb
Retry logic for 202 (#195)
* adding full refresh

* handling 202 retry

* adding colum

* rename

* core

* deleting coma

* 1 call

* 5 and system wait

* adding frequency

* working version

* refactored

* adjusting limit

* working version

* contributors job

* refactor

* fix: format

* add: missing repos

* job adjust

* csv updated

* livequery update

* jobs run
2023-11-03 06:27:23 -07:00
WHYTEWYLL
60d12f7f58
Batch size (#192)
* batch sizes

* ommiting contributors

* adding general endpoint, and commenting contributos

* format

* reducing batch size

* format

* resizing

* cap
2023-10-03 10:52:32 -07:00
WHYTEWYLL
b97704ac77
Batch size (#190)
* batch sizes

* ommiting contributors

* adding general endpoint, and commenting contributos

* format

* reducing batch size

* format

* resizing
2023-10-03 10:37:08 -07:00
WHYTEWYLL
6f5c3bbbd1
Fix cred (#189)
* fix: current date

* sysdate

* delete cred

* fix current

* macro current_date
2023-10-02 17:57:23 -07:00
WHYTEWYLL
6b59b76432
fix: current date (#188)
* fix: current date

* sysdate

* delete cred

* fix current
2023-10-02 17:49:47 -07:00
WHYTEWYLL
e593eb27cb
add: ephit (#187) 2023-09-29 10:45:56 -07:00
WHYTEWYLL
c2fc3c51a7
GitHub job (#186)
* no template

* space

* run silver

* add: project name

* project name to silver
2023-09-28 08:27:07 -07:00
WHYTEWYLL
19f2c08452
addin dev activity (#179)
* add: base github

* fix: arrays

* fix: list

* fix: SW script to javascript

* fix: execite

* working version

* env_variable

* add: frecuency

* rentry logic

* stable code'

* delete extra

* refactor no SYSTEMWAIT

* limit

* Working version

* jobs and working version

* deleting: type column

* data unpacking

* silver model and test

* fix: limit; delete: not used functions;

* working version

* add: date

* secret name

* add: secret string
2023-09-27 12:51:55 -07:00
drethereum
0634e70350
AN-2917/upgrade-prices (#117)
* coin gecko and cmc prices tables for all blockchains

* all_providers prices draft

* standardized platform column

* asset metadata, priority prices, and core dimensional/fact/ez tables for prices, metadata and contracts

* yml changes

* resolved issues with compiling

* prioritized symbol in prices

* unique address on ez prices

* updates to asset_metadata for efficiency

* asset_metadata updates for blockchain and unique_key

* prioritized coingecko symbol and decimals

* removed test

* removed eth pow and eth classic

* SO on priority prices model

* moved symbol and decimal joins to view layer

* moved dim_contracts join to asset_metadata view

* moved asset_metadata contracts join to view layer

* priority model for ez_asset_metadata and docs

* qualify statements modified
2023-04-06 10:32:05 -06:00
desmond-hui
c3da15254a
change streamline tasks to github actions (#109) 2023-02-21 16:11:41 -08:00
desmond-hui
13cc647626
Migrate to new streamline endpoints (#108)
* wip

* update with prod endpoint

* backend modified to use SQS, only need to call once
2023-02-20 14:42:28 -08:00
desmond-hui
f1c819a5d6
call this more often (#104) 2023-02-13 13:47:47 -08:00
desmond-hui
1ee61411a6
Adjust for role changes (#103)
* execute as caller

* change dev db ownership
2023-02-13 10:34:12 -08:00
Austin
0082d40941
marco for prod user abi table (#97)
* marco for prod user abi table

* rename schema
2023-01-24 18:12:58 -05:00
desmond-hui
f9687cec08
these resources no longer exist (#91) 2023-01-18 09:14:13 -08:00
desmond-hui
54dd848a90
remove legacy and unused resources (#89) 2023-01-17 10:08:20 -08:00
eric-laurello
7d2a3443fd
custom qery tags (#77) 2022-12-13 17:51:23 -08:00
desmond-hui
e2200f37e7
switch task to use xsmall warehouse (#74) 2022-11-29 14:41:41 -08:00
desmond-hui
e99051e298
turn off job for legacy cg prices: (#69) 2022-11-17 16:37:44 -05:00
Graham Onders
dbadd6cac0
Ens table (#65)
* added ENS table codes

* added ENS and udfs
2022-11-15 13:57:16 -05:00
desmond-hui
e6fe9cd707
change model references to new streamline sources location (#44) 2022-09-20 11:48:59 -07:00
desmond-hui
fab34ab39e
add sp for nightly copy of coin gecko data (#31)
* add sp for nightly copy of coin gecko data

* make diff call for dev
2022-09-07 10:49:30 -07:00
desmond-hui
79f9e33a29
split bronze asset metadata tables into one for each api (#21)
* split bronze asset metadata tables into one for each api

* clone tasks for dev
2022-08-25 10:18:32 -07:00
desmond-hui
eeed833eef
An 1871/switch cmc ohlc job to tasks (#20)
* udf endpoint condition on database name

* schedule cmc ohlc job with sf task

* also refresh asset table in case of new/deprecated coins
2022-08-24 14:31:40 -07:00
desmond-hui
06f2f37fd6
integrate with coingecko prices endpoints: (#19) 2022-08-24 13:27:31 -07:00
desmond-hui
08e30661ca
An 1871/add streamline model for historical cmc ohlc (#10)
* add streamline model to load cmc ohlc historical data

* reduce frequency of coingecko tick prices due to rate limits
2022-08-09 12:18:11 -07:00
desmond-hui
8c48ff41e5
grant ownership on snowflake tasks on dev refresh (#6) 2022-08-01 12:16:57 -07:00
Desmond Hui
d3ef98369e fix wrong schema ref 2022-07-22 09:55:58 -07:00
desmond-hui
c79ec93d98
An 1740/new crosschain prices (#4)
* integrate coin market cap asset ohlc api

* revert, accidentally moved this file

* add sample profile
2022-07-22 08:58:02 -07:00