Commit Graph

28 Commits

Author SHA1 Message Date
stanz
27e3414619
AN-6514 Streamline 2.0 Upgrade (#465)
Some checks failed
docs_update / run_dbt_jobs (push) Has been cancelled
docs_update / notify-failure (push) Has been cancelled
dbt_run_streamline_decoded_logs_history / run_dbt_jobs (push) Has been cancelled
dbt_run_streamline_decoded_logs_history / notify-failure (push) Has been cancelled
* rm topshot

* upd yml

* deploy udf

* upd blocks_realtime

* sl func

* bronze upd

* v1 namespace

* bronze v2

* prod endpoint

* define api integrations for v2

---------

Co-authored-by: Jack Forgash <58153492+forgxyz@users.noreply.github.com>
2025-10-24 01:15:13 +07:00
SAI
3d13e03a45
Topshot data (#440)
* livequery UDF model

* streamline models

* Add streamline model for TopShot metadata real-time processing

* buyback table

* add

* topshot and all day

* added streamline changes

* realtime

* update

* updated

* SL work - proposed changes (#441)

* updates

* .

* rev

* view to table

* fixed a few bugs

* sql limit

---------

Co-authored-by: eric-laurello <102970824+eric-laurello@users.noreply.github.com>
Co-authored-by: Eric Laurello <eric.laurello@flipsidecrypto.com>
2025-05-28 15:40:14 -05:00
Jack Forgash
e37db5668f
d+i (#352) 2024-09-04 18:45:33 -06:00
Jack Forgash
3e471404e5
AN-5069/expand onchain metadata mapping (#346) 2024-07-31 08:24:25 -06:00
Jack Forgash
51607b2222
add tag back (#344) 2024-07-26 13:49:04 -06:00
eric-laurello
f6b98de014
so & clsutering (#341) 2024-07-15 09:28:47 -06:00
Jack Forgash
30ce62f6a0
AN-4936/Backfill Migration Final (#335)
* del cw models

* core views

* defi views

* gov view

* nft_views

* rm ref to old cw models

* del internal cols

* nv 13-15

* nv 13-15

* nv 17-18

* nv 20

* del temp logic

* upd tests on allday metadata

* upd to warn
2024-07-01 13:27:42 -06:00
Jack Forgash
12bf4e2e86
AN-4799/Backfill Migration (#329)
* move models to bronze

* blocks backfill logic ez ez ez

* streamline_load models for all 4 methods

* add back a set of views for the complete models to use

* del dbt run history gha workflow

* migrate txs final to modified ts

* upd events to modified

* set default val for LOAD_BACKFILL_VERSION in project yml

* update curated model incr logic
2024-06-12 12:10:27 -06:00
WHYTEWYLL
2d85fe3e92
reducing batch size (#282) 2024-05-01 10:58:23 -07:00
WHYTEWYLL
759836fc8d
An 4437 add or account for all day burned moments (#263)
* a

* filter previus attempts

* fix empty pack

* del filt

* severity adjust
2024-02-05 15:07:51 -03:00
Jack Forgash
e0818966ad
AN-4428/add user agent to topshot and upd metadata job (#258)
* add user agent to topshot and upd metadata job

* upd agent

* upd limit to 600
2024-01-18 09:19:18 -07:00
WHYTEWYLL
e8851383c9
Flow - add pk/inserted & modified dates/_invocation id (#239)
* add: keys core

* add: silver labels jk

* add: silver metadata jk

* add silver prices pk

* add: silver staking jk

* add: silver swaps and transfers jk

* fix: doc

* gold core pk

* gold: defi keys

* price and half nfts

* add missing comma in silver__streamline_blocks

* add: nfts

* fix: models

* list col names in swaps_events_s

* add underscore _invocation

* fix: gold layer

* delete: description

* fix: order and names

* delete: unecesarry qualify

* fix: errors

* fix: name

* dix: name

* rename id

* add: pk to topshot

* add: dim rename

* fix dim metadata views

---------

Co-authored-by: Jack Forgash <58153492+forgxyz@users.noreply.github.com>
2023-12-12 11:49:18 -08:00
WHYTEWYLL
9dae9f240f
Pipeline metadata (#238)
* add: if condition

* removed row_list

* add:  gold view, fix macro, and silver models

* gold view extension

* add: test bronce

* g

* adding test

* add: test

* add: working test

* delete: tags

* delete: hardcode ref
2023-12-04 07:16:32 -08:00
WHYTEWYLL
150fbea1ab
fix: dedeuplicates (#237) 2023-11-29 08:44:07 -08:00
WHYTEWYLL
3e5e0a4bdd
Update silver__nft_allday_metadata_s.sql (#236) 2023-11-29 08:29:39 -08:00
WHYTEWYLL
6d6f478df6
alldaynft metadata with LQ (#229)
* base

* clean up; add: silver model

* add: new standards

* fix: refactor

* delete: folders

* refactor simplying

* remove no longer needed

* fix: refactor to silver layer

* ready to merge

---------

Co-authored-by: Desmond Hui <desmond@flipsidecrypto.com>
2023-11-27 07:55:44 -08:00
Jack Forgash
b33dc89a39
swap set description col (#232)
* swap set description col

* cast str
2023-11-20 16:39:47 -07:00
Jack Forgash
121c3e6a2c
AN-4203/ufc strike metadata (#230)
* ufc strike metadata, plus organized nft folder better

* upd db tag on gold

* upd col name to serial number

* del listing id

* add back listing id
2023-11-20 16:26:39 -07:00
Jack Forgash
80c2462ac4
core v non-core tags and gha workflow (#225) 2023-11-10 14:30:57 -07:00
Jack Forgash
bd08ee7f8f
An 4015/streamline cutover (#200)
* core views

* defi schema

* bridge test upd

* gov schema

* nft schema

* price schema

* move tx count to blocks model

* add collection_count_agg to blocks

* upd core views to UNION ALL

* upd collections cluster_by

* upd dim moment metadata to qualify

* upd dim moment metadata test

* upd core dim contract labels to qualify and silver (cw) to incr

* upd str blocks lookback`
2023-10-12 16:23:20 -06:00
Jack Forgash
792f11599f
AN-3903/Migration Reorg (#196)
* txs final and upd streamline tests w 1 hr inserted timestamp buffer

* copy silver models into silver_cw dir

* txs final, events model, curated model _s copies

* add pending col to txs final

* test config for pending_result_response

* add coalesce to tx id in txs final

* rem tests on inserted timestamp

* uppercase gap test vars

* add retry for null block timestamp and del mainnet23 filter from events

* address adj append 0x, add network ver to blocks, etc

* upd coalesce on events data

* saving event migration test, disabled

* req changes

* upd txs final ingested logic
2023-10-10 13:24:43 -06:00
Jack Forgash
fe7c2b724b
AN-3890/migrate udf (#188)
* upd to live

* migrate to new udf_api deployment

* adjust metadata schedule to daily
2023-09-08 12:52:40 -06:00
Jack Forgash
b1f0925a90
AN-3712/Metadata livequery (#146)
* topshot metadata pull v1

* before swap from lambda

* ready for stress test / batch sizing tmrw

* define unique key in helper tbl

* typo

* add todo deloy udf_api

* slight format tweak

* named

* migrate to livequery schema

* add parameter yml

* add livequery model yml, need tests

* alter metadata job run

* del old streamline macros

* silver upd - in flight

* clean up py model

* limit comment

* revert silver topshot metadata

* new silver model

* union new lq data into nft metadata table(s)

* del block_timestamp from metadata needed view

* untag silver nft allday metadata

* set limit to 3500

* addback topshot tag

* update schedule to hourly
2023-09-07 10:13:02 -06:00
Jack Forgash
45f72e71d8
An 3660/observability models (#155)
* blocks completeness

* block tx count external api model

* txs completeness model

* add col alias

* test result agg cte and temp test range

* curr range test

* revert source change - other pr

* xchain silver

* nuke api key & move to secrets, create workflow

* add scheduled tag

* update workflow w cache

* add obs tag

* rem testing range from txs complete

* comment out schedule and del double comman

* add default null for key var
2023-08-23 13:37:16 -06:00
Jack Forgash
67790931a4
Quickfix 080323/bad cast (#142)
* update cast to str

* add second str cast

* int
2023-08-03 11:07:39 -06:00
WHYTEWYLL
de09e6e00f
add: 248 celer id (#127) 2023-06-27 12:07:39 -07:00
Jack Forgash
dce8ba2ec9
move allday error filter to silver (#124) 2023-06-07 13:45:07 -04:00
Jack Forgash
09aa518880
An 2719/actions labels tweaks (#105)
* use template for GHA

* add custom notification

* move nft models to folder

* put silver models in folder

* swap labels to crosschain

* move script

* move script

* update failed test list if none
2023-01-26 09:24:49 -07:00