Commit Graph

129 Commits

Author SHA1 Message Date
github-actions
5239cac9e3 Auto-update docs 2025-09-02 15:55:56 +00:00
Jensen Yap
f6e472d80c
Update GitHub Actions workflow for dbt integration tests (#131)
Some checks failed
docs_update / docs_update (push) Has been cancelled
dbt_run_dev_refresh / dev_refresh (push) Has been cancelled
* Update GitHub Actions workflow for dbt integration tests

- Renamed the workflow to 'dbt_run_integration_test' for clarity.
- Removed the scheduled cron job to simplify the workflow.
- Updated the job to use a template from FlipsideCrypto for better maintainability.
- Adjusted the command to run integration tests using a selector.
- Enhanced environment handling based on the branch context.

* Update GitHub Actions workflow for dbt integration tests

- Renamed the workflow to 'integration test' for clarity.
- Added support for manual dispatch with environment input.
- Implemented scheduled runs for both hosted and production environments.
- Enhanced job configurations to streamline environment handling and command execution.
- Updated commands to run specific integration tests for better accuracy.

* Update dbt integration test workflow to enhance environment handling

- Modified the warehouse selection logic to include 'workflow_prod' as a valid condition for using 'DBT_CLOUD'.
- Updated the environment assignment to correctly map 'workflow_prod' to 'prod' and 'workflow_dev' to 'dev', improving clarity and functionality in the workflow.

* Update dbt integration test workflow to refine environment conditions

- Adjusted the warehouse selection logic to include 'workflow_dev' alongside 'workflow_prod' for using 'DBT_CLOUD'.
- Enhanced the environment assignment to ensure accurate mapping for 'workflow_dev' and 'workflow_prod', improving the workflow's clarity and functionality.
2025-09-03 00:54:23 +09:00
Jensen Yap
430bc25db1
Refactor external access integration and clean up SQL macros (#129)
- Removed the unused external access SQL file to streamline the project structure.
- Updated the S3 express external access integration macro by eliminating unnecessary schema usage.
- Adjusted dependencies in the utils SQL file to reflect the removal of the external access integration.
2025-08-16 02:26:46 +09:00
Jensen Yap
bafc04a856
[STREAM-1324] Enhance UDF compatibility checks and refactor UDF definitions (#128) 2025-08-13 02:00:11 +09:00
Jensen Yap
8138951db2
[STREAM-1324] Add Slack, Github Action Logs & Slack AI Alerts (#127)
* Update GitHub Actions workflow to reduce thread count and add extensive API integration documentation

- Changed thread count from 24 to 5 in GitHub Actions workflows for improved performance.
- Added comprehensive README files for various API integrations including Alchemy, NBA All Day, API Layer, Binance, and more.
- Introduced new UDFs and UDTFs for Groq and Slack API integrations, enhancing functionality and usability.
- Implemented tests for new UDFs and UDTFs to ensure reliability and correctness.
- Updated existing UDF definitions and added new tests for enhanced coverage and robustness.

* Refactor Slack UDFs to use webhook secret names and improve error handling

- Updated UDF definitions to replace WEBHOOK_URL and BOT_TOKEN with WEBHOOK_SECRET_NAME for enhanced security.
- Improved error messages for required parameters in the SQL logic.
- Standardized comments for clarity and consistency across UDFs.
- Ensured proper handling of user context for accessing secrets in the vault.

* update test

* fix test

* update slack

* remove test

* fix tests

* fix

* fix test args

* fix

* add .gitignore

* Add Slack Notification Macros and Enhance UDFs

- Introduced a new dbt macro for sending Slack notifications from GitHub Actions with AI-powered failure analysis.
- Added comprehensive README documentation for the new macro, detailing setup, configuration options, and usage examples.
- Implemented a SQL macro to handle Slack message formatting and sending, including support for AI analysis and threading.
- Updated existing UDFs to utilize webhook secret names for improved security and added detailed comments for clarity.
- Enhanced error handling and logging within the macros to ensure robust operation and easier debugging.

* update slack alerts

* update

* remove groq

* Enhance Slack Alert Macros with AI Analysis Features

- Updated README documentation to clarify AI provider options and added new parameters for model selection and custom prompts.
- Modified SQL macros to support the new `model_name` and `ai_prompt` parameters for improved AI analysis capabilities.
- Adjusted UDF signatures and comments to reflect the changes in AI provider functionality and requirements.
- Improved test cases to validate the new features and ensure robust performance of the Slack alert macros.

* update slack_alert

* change secret path to data_platform

* add backward compatibility for udf_api_v2

* revert to Object return type

* update type
2025-08-08 23:42:24 +09:00
Jensen Yap
a3b004d0cc
[STREAM-1155] Enhance UDF definitions and add new UDF for S3 presigned URL retrieval (#125)
Some checks failed
docs_update / docs_update (push) Has been cancelled
dbt_run_dev_refresh / dev_refresh (push) Has been cancelled
integration test / ${{ matrix.environment }} (hosted, XSMALL) (push) Has been cancelled
integration test / ${{ matrix.environment }} (prod, DBT_CLOUD) (push) Has been cancelled
2025-07-30 01:26:54 +09:00
Shah Newaz Khan
b3d6329d32
bump python runtime -> 3.10 (#123) 2025-06-11 10:53:52 -07:00
Shah Newaz Khan
b8dc1b09a4
add udf_api_batched (#122)
* add udf_api_batched

* add makefile & max_batch_rows to dbt vars

* update dev endpoint | add udf_api_batched to _live

* add udf_api_batched to live

* add update inner macros to account for max_batch_rows

* fix pointer to livequery_dev

* add udf_api_batched tests
2025-05-21 23:01:11 -07:00
Jensen Yap
da46dc4a8e
Add additional UDF tests for live API endpoints in live.yml (#121)
- Introduced multiple test cases for GET and POST methods, including batch JSON-RPC requests for Solana and Ethereum.
- Ensured assertions validate response status codes and data structure integrity.
- Enhanced coverage for API interactions to improve reliability and robustness of the deployment core.
2025-04-29 10:42:24 +09:00
Shah Newaz Khan
2651a45b7e
Stream 1175/reinstate deploy core (#120)
* add deploy/core & updated test_udf

* namespace scope macros

* remove package namespace

* add render.sql

* namespace scole apply_grants_by_schema

* add depnends_on clause for deploy/martketplace models
2025-04-16 09:18:28 -07:00
Jensen Yap
9f98d2b9b2
update helius rpc url to mainnet (#118) 2025-03-13 09:29:31 +09:00
Jensen Yap
2c5d0cca67
Fix: failing tests in livequery marketplace (#117)
* fix tests
2025-03-12 15:35:15 +09:00
Shah Newaz Khan
04bc1253f7
remove models/live__allow_list (#115)
* remove models/live__allow_list

* add livequery-base

* remove live__verified_api_endpoints.csv

* Re-trigger deployment checks
2025-03-10 10:12:00 -07:00
Jensen Yap
d01e8d47f8 revert changes in join logic 2025-03-05 15:42:00 -08:00
Jensen Yap
97a2811e8c fix bug in duplicate rows 2025-03-05 15:42:00 -08:00
Jensen Yap
90198898d0 add batch trace, eth balance & token balance 2025-03-05 15:42:00 -08:00
Jensen Yap
bc7b78e592 fix missing alias 2025-03-05 15:42:00 -08:00
Jensen Yap
10414a122e add batch call to debug trace call 2025-03-05 15:42:00 -08:00
Julius Remigio
fc4a6752c1 refactor evm_live_views.sql to improve batch processing and remove unused sql_live_rpc_batch_call macro 2025-03-05 15:42:00 -08:00
Julius Remigio
1eecc8f048 refactor evm_live_view_bronze_blocks macro to include blockchain and network parameters; add sql_live_rpc_batch_call helper macro for batch JSON RPC calls 2025-03-05 15:42:00 -08:00
Julius Remigio
2bdf24eaf3 update evm_live_view_target_blocks macro to accept batch_size parameter 2025-03-05 15:42:00 -08:00
Jensen Yap
606ec83c25 add batch calls in bronze blocks and logs 2025-03-05 15:42:00 -08:00
shah
c8dce1d376 fix typo in readme.md 2025-03-05 15:42:00 -08:00
Julius Remigio
6a9f4c343a Fix typo in README.md and update macro in evm_live_views.sql to fix to_latest 2025-03-05 15:42:00 -08:00
Jensen Yap
673a55c1b7 revert requirements 2025-03-05 15:42:00 -08:00
Jensen Yap
5d6db27740 [STREAM-1070] add fact decode traces livequery 2025-03-05 15:42:00 -08:00
Jensen Yap
318fd70418 [STREAM-1069] Add fact eth balance livequery 2025-03-05 15:42:00 -08:00
Jensen Yap
d5f6103f4d [STREAM-1068] update logic on balance 2025-03-05 15:42:00 -08:00
Jensen Yap
01bd6dd51e [STREAM-1068] Add fact token balances 2025-03-05 15:42:00 -08:00
Jensen Yap
29be791268 remove wave apps folder 2025-03-05 15:42:00 -08:00
Jensen Yap
476eee68ec [STREAM-1065] Add ez decoded event logs 2025-03-05 15:42:00 -08:00
Jensen Yap
5525d9f6c1 rename fact logs to fact event logs 2025-03-05 15:42:00 -08:00
shah
1e8cf27f67 STREAM-1051 Refactor ez_native_transfers to use reusable CTE macros 2025-03-05 15:42:00 -08:00
shah
4066365114 STREAM-1062 Refactor fact_traces UDTF SQL | Add evm_live_view_bronze_traces CTE macro 2025-03-05 15:42:00 -08:00
shah
76661504d2 STREAM-1049 Pin view columns for mapping_tables 2025-03-05 15:42:00 -08:00
Jensen Yap
1a40d0705f remove ez_token_transfers_id from signature 2025-03-05 15:42:00 -08:00
shah
4a5e83e913 STREAM-1053 Fix refactored reusable CTE column references 2025-03-05 15:42:00 -08:00
shah
3594bdbda6 STREAM-1053 Refactor fact_transactions to use reusable LiveView CTE's 2025-03-05 15:42:00 -08:00
Jensen Yap
1da9b610fb remove contract_name 2025-03-05 15:42:00 -08:00
Jensen Yap
39c656ca77 fix cols 2025-03-05 15:42:00 -08:00
Jensen Yap
6d1407bdff add discrepent columns between gold and livequery 2025-03-05 15:42:00 -08:00
shah
1fc41519ef STREAM-1049 Added secrets interpolation to fact_traces, fact_transactions & ez_native_transfers 2025-03-05 15:42:00 -08:00
shah
3d563d8710 STREAM-1049 Add secrets interpoation for fact_logs & fact_decoded_event_logs 2025-03-05 15:42:00 -08:00
shah
80590d8f9d STREAM-1049 Secrets interpolate fact_blocks macros 2025-03-05 15:42:00 -08:00
shah
0bc02b03a5 STREAM-1053 Add fact_transactions UDTF 2025-03-05 15:42:00 -08:00
shah
14cdcb769f STREAM-1062 fix tf_fact_traces signature to match return column aliases 2025-03-05 15:42:00 -08:00
shah
5e26c60d25 STREAM-1062 add fact_traces UDTF 2025-03-05 15:42:00 -08:00
shah
e72eec1eae STREAM-1051 Rename UDTF args 2025-03-05 15:42:00 -08:00
shah
213acecd40 STREAM-1051 add ez_native_transfers sql 2025-03-05 15:42:00 -08:00
Jensen Yap
a844411ef2 [STREAM-1061] Add EVM Decoded Logs udtf 2025-03-05 15:42:00 -08:00