mirror of
https://github.com/FlipsideCrypto/axelar-models.git
synced 2026-02-06 11:01:45 +00:00
ez models
This commit is contained in:
parent
11b86966d3
commit
35d20fb5ef
158
.cursor/rules/dbt-documentation-standards.mdc
Normal file
158
.cursor/rules/dbt-documentation-standards.mdc
Normal file
@ -0,0 +1,158 @@
|
||||
---
|
||||
description:
|
||||
globs: models/descriptions/*,*.yml,models/gold/**/*.sql
|
||||
alwaysApply: false
|
||||
---
|
||||
# dbt Documentation Standards
|
||||
When working with dbt projects, ensure comprehensive documentation that supports LLM-driven analytics workflows. This includes rich table and column descriptions that provide complete context for understanding blockchain data.
|
||||
|
||||
## Table Documentation Standards
|
||||
Every dbt Model must have an accompanying yml file that provides model documentation.
|
||||
|
||||
### Basic YML File Format
|
||||
Every dbt model yml file must follow this basic structure:
|
||||
|
||||
```yaml
|
||||
version: 2
|
||||
|
||||
models:
|
||||
- name: [model_name]
|
||||
description: "{{ doc('table_name') }}"
|
||||
tests:
|
||||
- [appropriate_tests_for_the_model]
|
||||
|
||||
columns:
|
||||
- name: [COLUMN_NAME]
|
||||
description: "{{ doc('column_name')}}"
|
||||
tests:
|
||||
- [appropriate_tests_for_the_column]
|
||||
```
|
||||
|
||||
#### Required Elements:
|
||||
- **version: 2** - Must be the first line
|
||||
- **models:** - Top-level key containing the model definitions
|
||||
- **name:** - The exact name of the dbt model (without .sql extension)
|
||||
- **description:** - Reference to markdown documentation using `{{ doc('table_name') }}`
|
||||
- **columns:** - List of all columns in the model with their documentation
|
||||
|
||||
#### Column Documentation Format:
|
||||
```yaml
|
||||
- name: [COLUMN_NAME_IN_UPPERCASE]
|
||||
description: "{{ doc('column_name')}}"
|
||||
tests:
|
||||
- [test_name]:
|
||||
[test_parameters]
|
||||
```
|
||||
|
||||
### Table Descriptions
|
||||
Table documentation must include 4 standard elements, formatted in markdown. As the base documentation file is YML, the table description must be written in a dbt documentation markdown file in the `models/descriptions/` directory. The Table YML can then use the jinja doc block to reference it.
|
||||
|
||||
The 4 standard categories (designed for LLM client consumption to aid in data model discovery and selection):
|
||||
|
||||
1. **Description** (the "what"): What the model is mapping from the blockchain, data scope and coverage, transformations and business logic applied. DO NOT EXPLAIN THE DBT MODEL LINEAGE. This is not important for the use case of LLM-driven blockchain analytics.
|
||||
2. **Key Use Cases**: Examples of when this table might be used and for what analysis, specific analytical scenarios and applications
|
||||
3. **Important Relationships**: How this table might be used alongside OTHER GOLD LEVEL models, dependencies and connections to other key gold models. For example, a table like logs, events, or receipts may contain information from a larget transactions. To build this description, you SHOULD review the dbt model lineage to understand genuine model relationships. You must convert the model name to a database object, for example `core__fact_blocks.sql` = `core.fact_blocks` = `<schema>__<table_name>.sql`
|
||||
4. **Commonly-used Fields**: Fields most important to condicting analytics. Determining these requires an understanding of the data model, what the columns are (via their descriptions) and how those fields aid in analytics. One way an understanding can be inferred is by analyzing curated models (anything that is not a core model in the gold/core/ directory is curated. Core models clean and map basic data objects of the blockchain like blocks, transactions, events, logs, etc. Curated models map specialized areas of activity like defi, nfts, governance, etc.). Blockchain data is often logged to a data table as a json object with a rich mapping of event-based details.
|
||||
|
||||
### Lineage Analysis
|
||||
Before writing table descriptions:
|
||||
- Read the dbt model SQL to understand the logic
|
||||
- Follow upstream dependencies to understand data flow
|
||||
- Review source models and transformations
|
||||
- Understand the business context and use cases
|
||||
- Review the column descriptions to estabish an understanding of the model, as a whole.
|
||||
- At the gold level, an ez_ table typically sources data from a fact_ table and this relationship should be documented. Ez_ tables add business logic such as (but not limited to) labels and USD price information
|
||||
|
||||
## Column Documentation Standards
|
||||
|
||||
### Rich Descriptions
|
||||
Each column description must include:
|
||||
- Clear definition of what the field represents
|
||||
- Data type and format expectations
|
||||
- Business context and use cases
|
||||
- Examples where helpful (especially for blockchain-specific concepts)
|
||||
- Relationships to other fields when relevant
|
||||
- Any important caveats or limitations
|
||||
|
||||
### Blockchain-Specific Context
|
||||
For blockchain data:
|
||||
- Reference official protocol documentation for technical accuracy. Use web search to find official developer documentation for the subject blockchain.
|
||||
- Explain blockchain-specific concepts (gas, consensus, etc.)
|
||||
- Provide examples using the specific blockchain's conventions
|
||||
- Clarify differences from other blockchains when relevant
|
||||
|
||||
### YAML Requirements
|
||||
- Column names MUST BE CAPITALIZED in YAML files
|
||||
- Use `{{ doc('column_name') }}` references for consistent desciption across models. The doc block must refer to a valid description in `models/descriptions`
|
||||
- Include appropriate tests for data quality
|
||||
|
||||
## Examples of Good Documentation
|
||||
|
||||
### Table Documentation Example
|
||||
```markdown
|
||||
{% docs table_transfers %}
|
||||
## Description
|
||||
This table tracks all token transfers on the <name> blockchain, capturing movements of native tokens and fungible tokens between accounts. The data includes both successful and failed transfers, with complete transaction context and token metadata.
|
||||
|
||||
## Key Use Cases
|
||||
- Token flow analysis and wallet tracking
|
||||
- DeFi protocol volume measurements
|
||||
- Cross-chain bridge monitoring
|
||||
- Whale movement detection and alerts
|
||||
- Token distribution and holder analysis
|
||||
|
||||
## Important Relationships
|
||||
- Subset of `gold.transactions`
|
||||
- Maps events emitted in `gold.events` or `gold.logs`
|
||||
- Utilizes token price data from `gold.prices` to compute USD columns
|
||||
|
||||
## Commonly-used Fields
|
||||
- `tx_hash`: Essential for linking to transaction details and verification
|
||||
- `sender_id` and `receiver_id`: Core fields for flow analysis and network mapping
|
||||
- `amount_raw` and `amount_usd`: Critical for value calculations and financial analysis
|
||||
- `token_address`: Key for filtering by specific tokens and DeFi analysis
|
||||
- `block_timestamp`: Primary field for time-series analysis and trend detection
|
||||
{% enddocs %}
|
||||
```
|
||||
|
||||
### Column Documentation Example
|
||||
```markdown
|
||||
{% docs amount_raw %}
|
||||
Unadjusted amount of tokens as it appears on-chain (not decimal adjusted). This is the raw token amount before any decimal precision adjustments are applied. For example, if transferring 1 native token, the amount_raw would be 1000000000000000000000000 (1e24) since <this blockchain's native token> has 24 decimal places. This field preserves the exact on-chain representation of the token amount for precise calculations and verification.
|
||||
{% enddocs %}
|
||||
```
|
||||
Important consideration: be sure to research and confirm figures such as decimal places. If unknown DO NOT MAKE UP A NUMBER.
|
||||
|
||||
## Common Patterns to Follow
|
||||
- Start with a clear definition
|
||||
- Provide context about why the field exists
|
||||
- Include examples for complex concepts
|
||||
- Explain relationships to other fields
|
||||
- Mention any important limitations or considerations
|
||||
- Use consistent terminology throughout the project
|
||||
|
||||
## Quality Standards
|
||||
|
||||
### Completeness
|
||||
- Every column must have a clear, detailed description
|
||||
- Table descriptions must explain the model's purpose and scope
|
||||
- Documentation must be self-contained without requiring external context
|
||||
- All business logic and transformations must be explained
|
||||
|
||||
### Accuracy
|
||||
- Technical details must match official blockchain documentation
|
||||
- Data types and formats must be correctly described
|
||||
- Examples must use appropriate blockchain conventions
|
||||
- Relationships between fields must be accurately described
|
||||
|
||||
### Clarity
|
||||
- Descriptions must be clear and easy to understand
|
||||
- Complex concepts must be explained with examples
|
||||
- Terminology must be consistent throughout the project
|
||||
- Language must support LLM understanding
|
||||
|
||||
### Consistency
|
||||
- Use consistent terminology across all models
|
||||
- Follow established documentation patterns
|
||||
- Maintain consistent formatting and structure
|
||||
- Ensure similar fields have similar descriptions
|
||||
186
.cursor/rules/dbt-overview-standard.mdc
Normal file
186
.cursor/rules/dbt-overview-standard.mdc
Normal file
@ -0,0 +1,186 @@
|
||||
---
|
||||
description:
|
||||
globs: __overview__.md,models/descriptions/*,models/gold/**/*.sql
|
||||
alwaysApply: false
|
||||
---
|
||||
# dbt Overview Standards
|
||||
When working with dbt projects, ensure comprehensive documentation exists about the project and a base reference to the gold models exists in the `__overview__.md` file for both human and LLM consumption.
|
||||
|
||||
## Project __overview__
|
||||
The file `models/descriptions/__overview__.md` is the entry point to the model description and documentation and must contain a rich description of what the project is.
|
||||
The file MUST START AND END WITH DBT JINJA DOCS TAGS `{% docs __overview__ %}` and `{% ENDDOCS %}`
|
||||
|
||||
## Quick Links Section Requirements
|
||||
The `__overview__.md` file MUST contain a "Quick Links to Table Documentation" section that provides direct navigation to all gold model documentation. This section must include a simple list, organized by gold schema, with the models and a hyperlink to the model documentation. If there is an existing section like "using dbt docs" that instructs the user on how to navigate dbt docs or a list of links to flipside and dbt, remove it! These are outdated.
|
||||
|
||||
### Required Elements:
|
||||
**Hyperlinks to Gold Model Documentation** - A comprehensive list of all gold models organized by schema. The schema can be inferred from the model name as the slug before the double underscore. For example, `core__fact_blocks` is a model named `fact_blocks` in the schema `core`.
|
||||
|
||||
### Gold Model Links Structure:
|
||||
The quicklinks section must be organized by schema and use the relative link to load the page generated by dbt documentation. The relative link structure is `#!/model/dbt_uniqueId` where dbt's `uniqueId` format is `node_type.project_name.model_name`. All of these `node_types` are `model`. `project_name` is the name of the dbt models project established in `dbt_project.yml` by the `name` variable. `model_name` is finally the name of the model as determed by the name of the sql file OR the value of `name` in the model's associated `.yml` file. For example, a uniqueId for the blockchain's `fact_blocks` model would be `model.<blockchain_name>_models.core__fact_blocks` making the relative URL `#!/model/model.<blockchain_name>_models.core__fact_blocks`.
|
||||
|
||||
```markdown
|
||||
## **Quick Links to Table Documentation**
|
||||
|
||||
**Click on the links below to jump to the documentation for each schema.**
|
||||
|
||||
### [Schema Name] Tables
|
||||
|
||||
**[Model Type Tables:]**
|
||||
|
||||
- model_1
|
||||
- model_2
|
||||
|
||||
### CORE Tables
|
||||
**Dimension Tables:**
|
||||
- [core__fact_blocks](relative/path/to/model)
|
||||
|
||||
**Fact Tables:**
|
||||
- [model_1](relative/path/to/model)
|
||||
|
||||
```
|
||||
|
||||
### Schema Organization Rules:
|
||||
1. **Group by Schema**: Organize models by their schema (core, defi, nft, price, social, governance, etc.)
|
||||
2. **Use Exact Schema Names**: Use the exact schema names as they appear in the database (e.g., `<blockchain_database>.CORE`, `<blockchain_database>.DEFI`, `<blockchain_database>.NFT`)
|
||||
3. **Model Type Subgrouping**: Within each schema, subgroup by model type:
|
||||
- **Dimension Tables:** (dim_* models)
|
||||
- **Fact Tables:** (fact_* models)
|
||||
- **Easy Views:** (ez_* models)
|
||||
4. **Link Format**: Use the exact dbt docs link format: `#!/model/model.[project_name].[schema]__[model]`
|
||||
5. **Model Naming**: Use the exact model name as it appears in the file system (without .sql extension)
|
||||
|
||||
### Implementation Guidelines for Coding Agents:
|
||||
1. **Scan Directory Structure**: Read `models/gold/` directory to identify all schema subdirectories
|
||||
2. **Extract Model Names**: For each schema directory, list all `.sql` files and remove the `.sql` extension
|
||||
3. **Determine Schema Mapping**: Map model names to database schema names:
|
||||
dbt models in this project utilize a double underscore in the model name to denote schema vs table <schema>__<table_name>.sql:
|
||||
- `core__fact_blocks` → `<blockchain_database>.CORE.FACT_BLOCKS`
|
||||
- `defi__ez_dex_swaps` → `<blockchain_database>.DEFI.EZ_DEX_SWAPS`
|
||||
4. **Categorize Models**: Group models by prefix:
|
||||
- `dim_*` → Dimension Tables
|
||||
- `fact_*` → Fact Tables
|
||||
- `ez_*` → Easy Views
|
||||
- `udf_*`, `udtf_*` → Custom Functions
|
||||
5. **Generate Links**: Create markdown links using the proper format
|
||||
6. **Maintain Order**: Keep models in alphabetical order within each category
|
||||
|
||||
### Validation Requirements:
|
||||
- Every gold model must have a corresponding link
|
||||
- Links must use the correct dbt docs format
|
||||
- Schema names must match the actual database schema
|
||||
- Model names must match the actual file names (without .sql extension)
|
||||
- Links must be organized by schema and model type
|
||||
- All links must be functional and point to valid dbt documentation
|
||||
- Do NOT add a hyperlink to the category headers. Only hyperlink individual models
|
||||
|
||||
## XML Tag Requirements
|
||||
Every `__overview__.md` file MUST include structured `<llm>` XML tags for easy interpretation by an LLM.
|
||||
```xml
|
||||
<llm>
|
||||
<blockchain>[Protocol Name]</blockchain>
|
||||
<aliases>[Common Aliases]</aliases>
|
||||
<ecosystem>[Execution Environment or Layer Type, for example EVM, SVM, IBC, Layer 1, Layer 2]</ecosystem>
|
||||
<description>[Rich 3-5 sentence description of the blockchain, its consensus mechanism, key features, and developer/user benefits including if the blockchain was built for a specific usecase.]</description>
|
||||
<external_resources>
|
||||
<block_scanner>[Link to the primary block scanner for the blockchain]</block_scanner>
|
||||
<developer_documenation>[Link to the primary developer documentation, maintained by the blockchain devs]</developer_documentation>
|
||||
</external_resources>
|
||||
<expert>
|
||||
<constraints>
|
||||
<table_availability>
|
||||
<!-- Specify which tables/schemas are available for this blockchain -->
|
||||
<!-- Example: "Ensure that your queries use only available tables for [BLOCKCHAIN]" -->
|
||||
</table_availability>
|
||||
|
||||
<schema_structure>
|
||||
<!-- Explain how the database is organized (dimensions, facts, naming conventions) -->
|
||||
<!-- Example: "Understand that dimensions and facts combine to make ez_ tables" -->
|
||||
</schema_structure>
|
||||
</constraints>
|
||||
|
||||
<optimization>
|
||||
<performance_filters>
|
||||
<!-- Define key filtering strategies for query performance -->
|
||||
<!-- Example: "use filters like block_timestamp over the last N days to improve speed" -->
|
||||
</performance_filters>
|
||||
|
||||
<query_structure>
|
||||
<!-- Specify preferred SQL patterns and structures -->
|
||||
<!-- Example: "Use CTEs, not subqueries, as readability is important" -->
|
||||
</query_structure>
|
||||
|
||||
<implementation_guidance>
|
||||
<!-- Provide guidelines for advanced SQL features -->
|
||||
<!-- Example: "Be smart with aggregations, window functions, etc." -->
|
||||
</implementation_guidance>
|
||||
</optimization>
|
||||
|
||||
<domain_mapping>
|
||||
<token_operations>
|
||||
<!-- Map token-related queries to specific tables -->
|
||||
<!-- Example: "For token transfers, use ez_token_transfers table" -->
|
||||
</token_operations>
|
||||
|
||||
<defi_analysis>
|
||||
<!-- Specify DeFi-related tables and their use cases -->
|
||||
<!-- Example: "For DeFi analysis, use ez_bridge_activity, ez_dex_swaps, ez_lending" -->
|
||||
</defi_analysis>
|
||||
|
||||
<nft_analysis>
|
||||
<!-- Define NFT-specific tables and functionality -->
|
||||
<!-- Example: "For NFT queries, utilize ez_nft_sales table in nft schema" -->
|
||||
</nft_analysis>
|
||||
|
||||
<specialized_features>
|
||||
<!-- Cover blockchain-specific features or complex data structures -->
|
||||
<!-- Example: "The XYZ data is complex, so ensure you ask clarifying questions" -->
|
||||
</specialized_features>
|
||||
</domain_mapping>
|
||||
|
||||
<interaction_modes>
|
||||
<direct_user>
|
||||
<!-- Define behavior for direct user interactions -->
|
||||
<!-- Example: "Ask clarifying questions when dealing with complex data" -->
|
||||
</direct_user>
|
||||
|
||||
<agent_invocation>
|
||||
<!-- Specify response format when invoked by other AI agents -->
|
||||
<!-- Example: "When invoked by another AI agent, respond with relevant query text" -->
|
||||
</agent_invocation>
|
||||
</interaction_modes>
|
||||
|
||||
<engagement>
|
||||
<exploration_tone>
|
||||
<!-- Set the overall tone and encouragement for data exploration -->
|
||||
<!-- Example: "Have fun exploring the [BLOCKCHAIN] ecosystem through data!" -->
|
||||
</exploration_tone>
|
||||
</engagement>
|
||||
</expert>
|
||||
</llm>
|
||||
```
|
||||
Place these XML tags at the end of the documentation (BUT STILL BEFORE THE JINJA ENDDOCS TAG).
|
||||
|
||||
## Update Process for Coding Agents:
|
||||
To update the overview, the coding agent MUST:
|
||||
|
||||
1. **Scan Current Gold Models**:
|
||||
- Read the entire `models/gold/` directory structure
|
||||
- Identify all `.sql` files across all schema subdirectories
|
||||
- Extract model names (remove `.sql` extension)
|
||||
|
||||
2. **Generate Updated Quicklinks Section**:
|
||||
- Follow these implementation guidelines
|
||||
- Create a complete new quicklinks section with all current gold models
|
||||
- Maintain proper schema organization and model type grouping
|
||||
|
||||
3. **Update __overview__.md**:
|
||||
- Replace the entire "Quick Links to Table Documentation" section with the newly generated content
|
||||
- Ensure proper markdown formatting and link structure
|
||||
- Create or update the XML tag block
|
||||
|
||||
4. **Validation Check**:
|
||||
- Verify all gold models have corresponding links
|
||||
- Confirm links use correct dbt docs format
|
||||
- Check that schema names and model names are accurate
|
||||
- Ensure alphabetical ordering within categories
|
||||
76
.cursor/rules/general-coding-standards.mdc
Normal file
76
.cursor/rules/general-coding-standards.mdc
Normal file
@ -0,0 +1,76 @@
|
||||
---
|
||||
description:
|
||||
globs:
|
||||
alwaysApply: true
|
||||
---
|
||||
# dbt Model Standards for Flipside Crypto
|
||||
|
||||
## General Rules
|
||||
- Follow the existing code style and patterns in the codebase
|
||||
- Write clear, concise, and well-documented code
|
||||
- Use meaningful variable, function, column and model names
|
||||
- Handle errors gracefully and provide helpful error messages
|
||||
- Test your code thoroughly before submitting
|
||||
- Follow the existing project structure and conventions
|
||||
|
||||
## Code Quality
|
||||
- Write self-documenting code with clear and consistent names
|
||||
- Use consistent formatting and indentation
|
||||
- Implement proper error handling and logging
|
||||
- Follow DRY (Don't Repeat Yourself) principles
|
||||
- Use meaningful commit messages
|
||||
- Use snake_case for all objects (tables, columns, models)
|
||||
- Maintain column naming consistency through the pipeline
|
||||
|
||||
## dbt Model Structure
|
||||
- Models are connected through ref() and source() functions
|
||||
- Data flows from source -> bronze -> silver -> gold layers
|
||||
- Each model has upstream dependencies and downstream consumers
|
||||
- Column-level lineage is maintained through transformations
|
||||
- Parse ref() and source() calls to identify direct dependencies
|
||||
- Track column transformations from upstream models
|
||||
- Consider impact on downstream consumers
|
||||
- Preserve business logic across transformations
|
||||
|
||||
## Model Naming and Organization
|
||||
- Follow naming patterns: bronze__, silver__, core__, fact_, dim__, ez__, where a double underscore indicates a break between a model schema and object name. I.e. core__fact_blocks equates to <database>.core.fact_blocks.
|
||||
- Organize by directory structure: bronze/, silver/, gold/, etc.
|
||||
- Upstream models appear on the LEFT side of the DAG
|
||||
- Current model is the focal point
|
||||
- Downstream models appear on the RIGHT side of the DAG
|
||||
|
||||
## Modeling Standards
|
||||
- Use snake_case for all objects
|
||||
- Prioritize incremental processing always
|
||||
- Follow source/bronze/silver/gold layering
|
||||
- Document chain-specific assumptions
|
||||
- Include incremental predicates to improve performance
|
||||
- For gold layer models, include search optimization following Snowflake's recommended best practices
|
||||
- Cluster models on appropriate fields
|
||||
|
||||
## Testing Requirements
|
||||
- Ensure proper token decimal handling
|
||||
- Implement unique tests for primary keys
|
||||
- Implement recency tests for tables that are expected to have frequent data updates
|
||||
- Add not_null tests for required columns
|
||||
- Use relationships tests for foreign keys
|
||||
|
||||
## Performance Guidelines
|
||||
- Optimize for high TPS (blockchain data)
|
||||
- Handle large state change volumes efficiently
|
||||
- Index frequently queried dimensions
|
||||
- Consider partition pruning strategies
|
||||
- Implement appropriate clustering keys
|
||||
- Optimize database queries for large datasets
|
||||
- Use appropriate indexing strategies
|
||||
- Monitor resource usage and optimize accordingly
|
||||
- Consider the impact of changes on existing systems
|
||||
|
||||
## Documentation
|
||||
- Document data sources
|
||||
- Map entity relationships
|
||||
- Include model descriptions in yml files per expanded rule [dbt-documentation-standards.mdc](mdc:.cursor/rules/dbt-documentation-standards.mdc)
|
||||
- Document column descriptions and business logic
|
||||
- Explain incremental logic and predicates
|
||||
- Note any data quality considerations
|
||||
|
||||
220
.cursor/rules/review-dbt-documentation.mdc
Normal file
220
.cursor/rules/review-dbt-documentation.mdc
Normal file
@ -0,0 +1,220 @@
|
||||
---
|
||||
description:
|
||||
globs:
|
||||
alwaysApply: false
|
||||
---
|
||||
# Review dbt Documentation Process
|
||||
|
||||
## Overview
|
||||
This document outlines the comprehensive process for reviewing and improving column and table descriptions for gold models in dbt projects. The goal is to provide robust, rich details that improve context for LLM-driven analytics workflows, ensuring that documentation is complete, accurate, and self-contained without requiring external expert files.
|
||||
|
||||
## Objectives
|
||||
- Create clear, detailed documentation that supports LLM understanding of blockchain data
|
||||
- Ensure technical accuracy by referencing official protocol documentation
|
||||
- Provide rich context for each table and column to enable effective analytics
|
||||
- Maintain consistency across all models and schemas
|
||||
- Support automated analytics workflows without requiring expert context files
|
||||
|
||||
## Pre-Review Requirements
|
||||
|
||||
### 1. Research Phase
|
||||
**Blockchain Protocol Documentation**
|
||||
- Search and read official developer documentation for the target blockchain. Utilize web search to find authentic and accurate developer documentation
|
||||
- Review technical specifications, whitepapers, and API documentation
|
||||
- Understand the blockchain's consensus mechanism, data structures, and conventions
|
||||
- Research common use cases and analytics patterns specific to the blockchain
|
||||
- Identify key technical concepts that need explanation (e.g., gas mechanics, consensus, token standards)
|
||||
|
||||
**External Resources to Consult**
|
||||
- Official blockchain documentation
|
||||
- Developer guides and tutorials
|
||||
- Technical specifications and whitepapers
|
||||
- Community documentation and forums
|
||||
- Block explorers and API documentation
|
||||
|
||||
### 2. Project Context Analysis
|
||||
- Review the `__overview__.md` file and rewrite it per the @dbt-overview-standard rule to create a summary of what this particular blockchain is, unique characteristics, and any other general information about the chain itself
|
||||
- Review existing documentation patterns and terminology
|
||||
- Understand the data flow and model lineage structure
|
||||
|
||||
## Review Process
|
||||
|
||||
### Step 1: Model Analysis
|
||||
**SQL Logic Review**
|
||||
- Read the dbt model SQL file to understand the transformations and business logic
|
||||
- Follow upstream dependencies to understand data flow from source to gold layer
|
||||
- Review bronze source models, silver staging models, and intermediate transformations
|
||||
- Identify any complex joins, aggregations, or business logic that needs explanation
|
||||
- Understand the incremental logic and any filtering conditions
|
||||
|
||||
**Lineage Analysis**
|
||||
- Map the complete data lineage from source to gold model
|
||||
- Identify key transformations and their purposes
|
||||
- Understand relationships between related models for the sole purpose of generating a robust description
|
||||
- Do not include data lineage analysis in the table description
|
||||
|
||||
### Step 2: Column Description Review
|
||||
**Individual Column Analysis**
|
||||
For each column in the model:
|
||||
|
||||
1. **Technical Understanding**
|
||||
- Read the SQL to understand how the column is derived
|
||||
- Check upstream models if the column comes from a transformation
|
||||
- Understand the data type and format expectations
|
||||
- Identify any business logic applied to the column
|
||||
|
||||
2. **Blockchain Context**
|
||||
- Research the blockchain-specific meaning of the column
|
||||
- Reference official documentation for technical accuracy
|
||||
- Understand how this field relates to blockchain concepts
|
||||
- Identify any blockchain-specific conventions or requirements
|
||||
|
||||
3. **Documentation Assessment**
|
||||
- Review existing column description
|
||||
- Evaluate completeness and clarity
|
||||
- Check for missing context or examples
|
||||
- Ensure the description supports LLM understanding
|
||||
|
||||
**Required Elements for Column Descriptions**
|
||||
- Clear definition of what the field represents
|
||||
- Data type and format expectations
|
||||
- Business context and use cases
|
||||
- Examples where helpful (especially for blockchain-specific concepts)
|
||||
- Relationships to other fields when relevant
|
||||
- Any important caveats or limitations
|
||||
- Blockchain-specific context and conventions
|
||||
|
||||
### Step 3: Table Description Review
|
||||
**Current State Assessment**
|
||||
- Review the updated column descriptions
|
||||
- Review existing table description in the YAML file
|
||||
- Evaluate completeness and clarity
|
||||
- Identify missing context or unclear explanations
|
||||
|
||||
**Required Elements for Table Descriptions**
|
||||
Table documentation must include 4 standard elements, formatted in markdown. As the base documentation file is YML, the table description must be written in a dbt documentation markdown file in the `models/descriptions/` directory. The Table YML can then use the jinja doc block to reference it.
|
||||
|
||||
The 4 standard categories are fully defined in the @dbt-documentation-standards rule. They are:
|
||||
|
||||
1. **Description**
|
||||
2. **Key Use Cases**
|
||||
3. **Important Relationships**
|
||||
4. **Commonly-used Fields**
|
||||
|
||||
### Step 4: Documentation File Review
|
||||
**Individual Documentation Files**
|
||||
- Check if each column has a corresponding `.md` file in `models/descriptions/`
|
||||
- Review existing documentation for completeness and accuracy
|
||||
- Update or create documentation files as needed
|
||||
|
||||
**Documentation File Format**
|
||||
```markdown
|
||||
{% docs column_name %}
|
||||
[Rich, detailed description including:
|
||||
- Clear definition
|
||||
- Data format and examples
|
||||
- Business context
|
||||
- Blockchain-specific details
|
||||
- Relationships to other fields
|
||||
- Important considerations]
|
||||
{% enddocs %}
|
||||
```
|
||||
|
||||
### Step 5: YAML File Review
|
||||
**YAML Structure Validation**
|
||||
- Ensure column names are CAPITALIZED in YAML files
|
||||
- Verify all columns reference documentation using `{{ doc('column_name') }}`
|
||||
- Check that appropriate tests are included
|
||||
- Validate the overall YAML structure
|
||||
|
||||
**YAML File Format**
|
||||
```yaml
|
||||
version: 2
|
||||
|
||||
models:
|
||||
- name: [model_name]
|
||||
description: |-
|
||||
[Clear, direct table description]
|
||||
|
||||
columns:
|
||||
- name: [COLUMN_NAME_IN_UPPERCASE]
|
||||
description: "{{ doc('column_name') }}"
|
||||
tests:
|
||||
- [appropriate_tests]
|
||||
```
|
||||
|
||||
## Review Checklist
|
||||
|
||||
### Table Level
|
||||
- [ ] Table documentation is in markdown file in `models/descriptions/` directory
|
||||
- [ ] Table YAML references documentation using jinja doc block
|
||||
- [ ] **Description** section explains what blockchain data is being modeled
|
||||
- [ ] **Key Use Cases** section provides specific analytical scenarios
|
||||
- [ ] **Important Relationships** section explains connections to other GOLD models
|
||||
- [ ] **Commonly-used Fields** section identifies critical columns and their importance
|
||||
- [ ] Documentation is optimized for LLM client consumption
|
||||
|
||||
### Column Level
|
||||
- [ ] Each column has a comprehensive description
|
||||
- [ ] Data types and formats are clearly specified
|
||||
- [ ] Business context and use cases are explained
|
||||
- [ ] Examples are provided for complex concepts
|
||||
- [ ] Relationships to other fields are documented
|
||||
- [ ] Important limitations or caveats are noted
|
||||
- [ ] Blockchain-specific context is included
|
||||
|
||||
### Documentation Files
|
||||
- [ ] All columns have corresponding `.md` files
|
||||
- [ ] Documentation files contain rich, detailed descriptions
|
||||
- [ ] Examples use appropriate blockchain conventions
|
||||
- [ ] Technical accuracy is verified against official documentation
|
||||
|
||||
### YAML Files
|
||||
- [ ] Column names are CAPITALIZED
|
||||
- [ ] All columns reference documentation using `{{ doc('column_name') }}`
|
||||
- [ ] Appropriate tests are included
|
||||
- [ ] YAML structure is valid
|
||||
|
||||
## Implementation Guidelines
|
||||
|
||||
### Documentation Writing Tips
|
||||
- Start with a clear definition of what the field represents
|
||||
- Provide context about why the field exists and its importance
|
||||
- Include examples for complex concepts, especially blockchain-specific ones
|
||||
- Explain relationships to other fields when relevant
|
||||
- Mention any important limitations or considerations
|
||||
- Use consistent terminology throughout the project
|
||||
|
||||
### Blockchain-Specific Considerations
|
||||
- Reference official protocol documentation for technical concepts
|
||||
- Explain blockchain-specific concepts (gas, consensus, etc.)
|
||||
- Provide examples using the specific blockchain's conventions
|
||||
- Clarify differences from other blockchains when relevant
|
||||
- Include information about data freshness and update mechanisms
|
||||
|
||||
### LLM Optimization
|
||||
- Write descriptions that are complete and self-contained
|
||||
- Use clear, structured language that supports automated understanding
|
||||
- Include context that helps LLMs understand the data's purpose
|
||||
- Provide examples that illustrate common use cases
|
||||
- Ensure descriptions support common analytics workflows
|
||||
|
||||
## Post-Review Actions
|
||||
|
||||
### Validation
|
||||
- Verify all documentation is technically accurate
|
||||
- Check that descriptions are complete and self-contained
|
||||
- Ensure consistency across related models
|
||||
- Validate that documentation supports common analytics use cases
|
||||
|
||||
### Testing
|
||||
- Test documentation by having an LLM attempt to understand the data
|
||||
- Verify that descriptions enable effective query generation
|
||||
- Check that examples are clear and helpful
|
||||
- Ensure documentation supports the intended analytics workflows
|
||||
|
||||
### Maintenance
|
||||
- Update documentation when models change
|
||||
- Review and refresh documentation periodically
|
||||
- Maintain consistency as new models are added
|
||||
- Keep documentation aligned with blockchain protocol updates
|
||||
24
.gitignore
vendored
24
.gitignore
vendored
@ -5,4 +5,26 @@ logs/
|
||||
.user.yml
|
||||
.DS_Store
|
||||
.vscode
|
||||
dbt-env/
|
||||
dbt-env/
|
||||
|
||||
# Visual Studio Code files
|
||||
*/.vscode
|
||||
*.code-workspace
|
||||
.history/
|
||||
**/.DS_Store
|
||||
.vscode/
|
||||
.env
|
||||
.DS_Store
|
||||
.user.yml
|
||||
|
||||
# Cursor configuration
|
||||
.cursor/*
|
||||
!.cursor/rules
|
||||
|
||||
# Local Claude configuration
|
||||
.claude/settings.local.json
|
||||
.claude/log
|
||||
CLAUDE.local.md
|
||||
CLAUDE.md
|
||||
|
||||
__pycache__
|
||||
74
models/bronze/axelscan/LQ/bronze__axelscan_tokens_meta.sql
Normal file
74
models/bronze/axelscan/LQ/bronze__axelscan_tokens_meta.sql
Normal file
@ -0,0 +1,74 @@
|
||||
{{ config(
|
||||
materialized = 'incremental',
|
||||
tags = ['daily']
|
||||
) }}
|
||||
|
||||
WITH tokens AS (
|
||||
|
||||
SELECT
|
||||
token_symbol,
|
||||
COUNT(1) AS xc
|
||||
FROM
|
||||
axelar.axelscan.ez_bridge_activity -- hard code to avoid ref cycle
|
||||
WHERE
|
||||
token_symbol IS NOT NULL
|
||||
GROUP BY
|
||||
token_symbol
|
||||
),
|
||||
WORK AS (
|
||||
SELECT
|
||||
A.token_symbol
|
||||
FROM
|
||||
tokens A
|
||||
|
||||
{% if is_incremental() %}
|
||||
LEFT JOIN {{ this }}
|
||||
b
|
||||
ON A.token_symbol = b.token_symbol
|
||||
WHERE
|
||||
b.token_symbol IS NULL
|
||||
OR b._inserted_timestamp <= DATEADD(DAY, -3, SYSDATE())
|
||||
{% endif %}
|
||||
ORDER BY
|
||||
xc DESC
|
||||
LIMIT
|
||||
100), api_calls AS (
|
||||
SELECT
|
||||
token_symbol,
|
||||
live.udf_api(
|
||||
'POST',
|
||||
'https://api.axelarscan.io/api/getTokensPrice',
|
||||
OBJECT_CONSTRUCT(
|
||||
'accept',
|
||||
'application/json',
|
||||
'content-type',
|
||||
'application/json'
|
||||
),
|
||||
OBJECT_CONSTRUCT(
|
||||
'symbol',
|
||||
token_symbol
|
||||
)
|
||||
) AS response,
|
||||
SYSDATE() AS _inserted_timestamp
|
||||
FROM
|
||||
WORK
|
||||
)
|
||||
SELECT
|
||||
token_symbol,
|
||||
response,
|
||||
response :status_code :: INT AS status_code,
|
||||
response :data AS price_data,
|
||||
-- Parse the price data for the token
|
||||
-- API returns data with token symbol as key
|
||||
response :data [token_symbol] :price :: FLOAT AS price,
|
||||
response :data [token_symbol] :coingecko_id :: STRING AS coingecko_id,
|
||||
response :data [token_symbol] :decimals :: INT AS decimals,
|
||||
response :data [token_symbol] :symbol :: STRING AS api_symbol,
|
||||
response :data [token_symbol] :name :: STRING AS token_name,
|
||||
response :data [token_symbol] :native_chain :: STRING AS native_chain,
|
||||
response :data [token_symbol] :type :: STRING AS token_type,
|
||||
response :data [token_symbol] :chains AS chains_data,
|
||||
-- Time tracking
|
||||
_inserted_timestamp
|
||||
FROM
|
||||
api_calls
|
||||
105
models/gold/axelscan/axelscan__ez_bridge_activity.sql
Normal file
105
models/gold/axelscan/axelscan__ez_bridge_activity.sql
Normal file
@ -0,0 +1,105 @@
|
||||
{{ config(
|
||||
materialized = 'view',
|
||||
tags = ['daily']
|
||||
) }}
|
||||
|
||||
-- Union of GMP activity and Transfer activity for a unified bridge activity view
|
||||
SELECT
|
||||
'gmp' AS activity_type,
|
||||
gmp_id AS transaction_id,
|
||||
created_at,
|
||||
status,
|
||||
simplified_status,
|
||||
|
||||
-- Chain and address info
|
||||
source_chain,
|
||||
destination_chain,
|
||||
sender_address,
|
||||
destination_contract_address,
|
||||
recipient_address,
|
||||
|
||||
-- Token and amount info
|
||||
token_symbol,
|
||||
token_amount AS amount,
|
||||
token_amount_usd AS amount_value_usd,
|
||||
token_price_usd,
|
||||
|
||||
-- Transaction details
|
||||
source_tx_hash,
|
||||
source_event AS event_type,
|
||||
source_block_number,
|
||||
|
||||
-- Transfer-specific fields (NULL for GMP)
|
||||
NULL AS transfer_type,
|
||||
NULL AS amount_received,
|
||||
NULL AS insufficient_fee,
|
||||
NULL AS confirm_tx_hash,
|
||||
NULL AS deposit_address,
|
||||
NULL AS fee,
|
||||
NULL AS fee_value_usd,
|
||||
|
||||
-- GMP-specific fields
|
||||
callback_chain,
|
||||
callback_destination_address,
|
||||
parent_message_id,
|
||||
token_contract_address,
|
||||
data_source,
|
||||
price_source,
|
||||
|
||||
-- Metadata
|
||||
ez_gmp_activity_id AS ez_bridge_activity_id,
|
||||
inserted_timestamp,
|
||||
modified_timestamp
|
||||
|
||||
FROM {{ ref('axelscan__ez_gmp_activity') }}
|
||||
|
||||
UNION ALL
|
||||
|
||||
SELECT
|
||||
'transfer' AS activity_type,
|
||||
transfer_id AS transaction_id,
|
||||
created_at,
|
||||
status,
|
||||
simplified_status,
|
||||
|
||||
-- Chain and address info
|
||||
source_chain,
|
||||
destination_chain,
|
||||
sender_address,
|
||||
NULL AS destination_contract_address,
|
||||
recipient_address,
|
||||
|
||||
-- Token and amount info
|
||||
token_denom AS token_symbol,
|
||||
amount,
|
||||
amount_value_usd,
|
||||
NULL AS token_price_usd,
|
||||
|
||||
-- Transaction details
|
||||
source_tx_hash,
|
||||
transfer_type AS event_type,
|
||||
source_height AS source_block_number,
|
||||
|
||||
-- Transfer-specific fields
|
||||
transfer_type,
|
||||
amount_received,
|
||||
insufficient_fee,
|
||||
confirm_tx_hash,
|
||||
deposit_address,
|
||||
fee,
|
||||
fee_value_usd,
|
||||
|
||||
-- GMP-specific fields (NULL for transfers)
|
||||
NULL AS callback_chain,
|
||||
NULL AS callback_destination_address,
|
||||
NULL AS parent_message_id,
|
||||
NULL AS token_contract_address,
|
||||
NULL AS data_source,
|
||||
NULL AS price_source,
|
||||
|
||||
-- Metadata
|
||||
ez_transfer_activity_id AS ez_bridge_activity_id,
|
||||
inserted_timestamp,
|
||||
modified_timestamp
|
||||
|
||||
FROM {{ ref('axelscan__ez_transfer_activity') }}
|
||||
77
models/gold/axelscan/axelscan__ez_bridge_activity.yml
Normal file
77
models/gold/axelscan/axelscan__ez_bridge_activity.yml
Normal file
@ -0,0 +1,77 @@
|
||||
version: 2
|
||||
models:
|
||||
- name: axelscan__ez_bridge_activity
|
||||
description: Unified view of all Axelar cross-chain bridge activity, combining both GMP (General Message Protocol) transactions and direct token transfers. Use the activity_type field to distinguish between 'gmp' and 'transfer' records. This view harmonizes the schemas of both activity types, focusing on common bridging fields (chains, addresses, tokens, amounts, fees). For GMP-specific fields (payload, command_id, gas_status, etc.), query axelscan__ez_gmp_activity directly.
|
||||
columns:
|
||||
- name: ACTIVITY_TYPE
|
||||
description: Type of bridge activity - 'gmp' for GMP transactions or 'transfer' for direct transfers
|
||||
tests:
|
||||
- not_null
|
||||
- accepted_values:
|
||||
values: ['gmp', 'transfer']
|
||||
- name: TRANSACTION_ID
|
||||
description: The unique identifier for the transaction (gmp_id or transfer_id depending on type)
|
||||
tests:
|
||||
- not_null
|
||||
- name: CREATED_AT
|
||||
description: The timestamp when the transaction was created
|
||||
tests:
|
||||
- not_null
|
||||
- name: STATUS
|
||||
description: The detailed status of the transaction
|
||||
- name: SIMPLIFIED_STATUS
|
||||
description: Simplified transaction status (e.g., received, failed, approved, sent)
|
||||
- name: SOURCE_CHAIN
|
||||
description: The name of the source blockchain (lowercase, normalized)
|
||||
- name: DESTINATION_CHAIN
|
||||
description: The name of the destination blockchain (lowercase, normalized)
|
||||
- name: SENDER_ADDRESS
|
||||
description: The wallet address of the sender (lowercase)
|
||||
- name: CONTRACT_ADDRESS
|
||||
description: The contract address involved in the transaction. Only populated for GMP records (Axelar gateway or source contract).
|
||||
- name: RECIPIENT_ADDRESS
|
||||
description: The recipient address. For GMP this is the destination contract, for transfers this is the recipient wallet.
|
||||
- name: TOKEN_SYMBOL
|
||||
description: The token symbol (e.g., axlUSDC, USDC, XRP). For GMP uses COALESCE(call:returnValues:symbol, DATA:symbol). For transfers uses token_denom.
|
||||
- name: AMOUNT_RAW
|
||||
description: The raw token amount as a string. Only populated for GMP records.
|
||||
- name: AMOUNT
|
||||
description: The token amount as a numeric value
|
||||
- name: DENOM
|
||||
description: The token denomination
|
||||
- name: FEE_TOKEN
|
||||
description: The token used to pay fees. Only populated for GMP records.
|
||||
- name: FEE_AMOUNT
|
||||
description: The fee amount (base_fee for GMP, fee for transfers)
|
||||
- name: FEE_VALUE_USD
|
||||
description: The fee value in USD. Only populated for transfer records.
|
||||
- name: SOURCE_TX_HASH
|
||||
description: The transaction hash on the source chain
|
||||
- name: EVENT_TYPE
|
||||
description: The type of event (e.g., ContractCall, ContractCallWithToken for GMP; link_transfer, axelar_transfer for transfers)
|
||||
- name: BLOCK_NUMBER
|
||||
description: The block number on the source chain
|
||||
- name: BLOCK_TIMESTAMP
|
||||
description: The block timestamp. Only populated for GMP records.
|
||||
- name: TRANSFER_TYPE
|
||||
description: The type of transfer (e.g., link_transfer, axelar_transfer, ibc_transfer). Transfer only.
|
||||
- name: AMOUNT_RECEIVED
|
||||
description: The amount received on the destination chain. Transfer only.
|
||||
- name: AMOUNT_VALUE_USD
|
||||
description: The transfer amount value in USD. Transfer only.
|
||||
- name: INSUFFICIENT_FEE
|
||||
description: Flag indicating whether the fee was insufficient. Transfer only.
|
||||
- name: CONFIRM_TX_HASH
|
||||
description: The confirmation transaction hash. Transfer only.
|
||||
- name: DEPOSIT_ADDRESS
|
||||
description: The deposit address used for the transfer. Transfer only.
|
||||
- name: EZ_BRIDGE_ACTIVITY_ID
|
||||
description: The unique identifier for this record (maps to ez_gmp_activity_id or ez_transfer_activity_id)
|
||||
tests:
|
||||
- not_null
|
||||
- name: SOURCE_FACT_ID
|
||||
description: Foreign key reference to source fact table (fact_gmp_id or fact_transfers_id)
|
||||
- name: INSERTED_TIMESTAMP
|
||||
description: '{{ doc("inserted_timestamp") }}'
|
||||
- name: MODIFIED_TIMESTAMP
|
||||
description: '{{ doc("modified_timestamp") }}'
|
||||
@ -6,60 +6,45 @@
|
||||
) }}
|
||||
|
||||
WITH gmp_stats AS (
|
||||
|
||||
SELECT
|
||||
date_day,
|
||||
b.value :key :: STRING AS source_chain,
|
||||
C.value :key :: STRING AS destination_chain,
|
||||
C.value :num_txs :: INT AS path_txs,
|
||||
C.value :volume :: DECIMAL(
|
||||
18,
|
||||
2
|
||||
) AS path_volume
|
||||
source_chain,
|
||||
destination_chain,
|
||||
num_txs AS path_txs,
|
||||
volume_usd AS path_volume
|
||||
FROM
|
||||
{{ ref('bronze__axelscan_gmp_stats_by_chains') }},
|
||||
LATERAL FLATTEN(
|
||||
resp :data :source_chains
|
||||
) b,
|
||||
LATERAL FLATTEN(
|
||||
b.value :destination_chains
|
||||
) C
|
||||
{{ ref('silver__axelscan_gmp_stats_by_chains') }}
|
||||
|
||||
{% if is_incremental() %}
|
||||
WHERE
|
||||
date_day >= (
|
||||
SELECT
|
||||
COALESCE(MAX(day_utc), '1970-01-01' :: DATE) - 3
|
||||
FROM
|
||||
{{ this }})
|
||||
{% endif %}
|
||||
),
|
||||
transfer_stats AS (
|
||||
SELECT
|
||||
date_day,
|
||||
b.value :source_chain :: STRING AS source_chain,
|
||||
b.value :destination_chain :: STRING AS destination_chain,
|
||||
SUM(
|
||||
b.value :num_txs :: INT
|
||||
) AS path_txs,
|
||||
SUM(b.value :volume :: DECIMAL(18, 2)) AS path_volume
|
||||
FROM
|
||||
{{ ref('bronze__axelscan_transfer_stats_by_chains') }},
|
||||
LATERAL FLATTEN(
|
||||
resp :data :data
|
||||
) b
|
||||
WHERE
|
||||
date_day >= (
|
||||
SELECT
|
||||
COALESCE(MAX(day_utc), '1970-01-01' :: DATE) - 7
|
||||
FROM
|
||||
{{ this }}
|
||||
)
|
||||
{% endif %}
|
||||
),
|
||||
transfer_stats AS (
|
||||
SELECT
|
||||
date_day,
|
||||
source_chain,
|
||||
destination_chain,
|
||||
num_txs AS path_txs,
|
||||
volume_usd AS path_volume
|
||||
FROM
|
||||
{{ ref('silver__axelscan_transfer_stats_by_chains') }}
|
||||
|
||||
{% if is_incremental() %}
|
||||
WHERE
|
||||
date_day >= (
|
||||
SELECT
|
||||
COALESCE(MAX(day_utc), '1970-01-01' :: DATE) - 3
|
||||
FROM
|
||||
{{ this }})
|
||||
{% endif %}
|
||||
GROUP BY
|
||||
ALL
|
||||
),
|
||||
WHERE
|
||||
date_day >= (
|
||||
SELECT
|
||||
COALESCE(MAX(day_utc), '1970-01-01' :: DATE) - 7
|
||||
FROM
|
||||
{{ this }}
|
||||
)
|
||||
{% endif %}
|
||||
),
|
||||
combined AS (
|
||||
SELECT
|
||||
COALESCE(
|
||||
@ -98,8 +83,8 @@ WHERE
|
||||
AND g.destination_chain = t.destination_chain
|
||||
)
|
||||
SELECT
|
||||
source_blockchain,
|
||||
destination_blockchain,
|
||||
REPLACE(source_blockchain, 'axelarnet', 'axelar') AS source_blockchain,
|
||||
REPLACE(destination_blockchain, 'axelarnet', 'axelar') AS destination_blockchain,
|
||||
day_utc,
|
||||
COALESCE(
|
||||
gmp_num_txs + transfers_num_txs,
|
||||
|
||||
@ -15,7 +15,7 @@ models:
|
||||
tests:
|
||||
- dbt_expectations.expect_row_values_to_have_recent_data:
|
||||
datepart: hour
|
||||
interval: 12
|
||||
interval: 48 # Increased to 48 hours to account for processing delays and weekends
|
||||
- name: num_txs
|
||||
description: "Total number of transactions (GMP + Transfers)"
|
||||
- name: volume_usd
|
||||
|
||||
684
models/gold/axelscan/axelscan__ez_gmp_activity.sql
Normal file
684
models/gold/axelscan/axelscan__ez_gmp_activity.sql
Normal file
@ -0,0 +1,684 @@
|
||||
{{ config(
|
||||
materialized = 'incremental',
|
||||
unique_key = ['ez_gmp_activity_id'],
|
||||
incremental_strategy = 'delete+insert',
|
||||
cluster_by = ['created_at::DATE'],
|
||||
tags = ['daily']
|
||||
) }}
|
||||
|
||||
-- Enhanced Combined EZ GMP model that merges THREE complementary data sources:
|
||||
-- 1. interchain_transfers ARRAY data - multiple transfers in one GMP (with embedded prices)
|
||||
-- 2. interchain_transfer OBJECT data - single transfer in one GMP (with price lookups added)
|
||||
-- 3. Standard GMP activity data - GMPs without interchain transfer details (with price lookups)
|
||||
-- This is the DEFINITIVE model for all GMP transfer analysis
|
||||
|
||||
WITH
|
||||
-- Token metadata with coingecko_id mapping
|
||||
token_meta AS (
|
||||
SELECT
|
||||
token_symbol,
|
||||
coingecko_id
|
||||
FROM
|
||||
{{ ref('silver__axelscan_token_meta') }}
|
||||
WHERE
|
||||
coingecko_id IS NOT NULL
|
||||
),
|
||||
|
||||
coingecko_prices AS (
|
||||
SELECT
|
||||
HOUR,
|
||||
asset_id AS coingecko_id,
|
||||
CLOSE AS price
|
||||
FROM
|
||||
{{ source('crosschain_price','fact_prices_ohlc_hourly')}}
|
||||
WHERE
|
||||
provider = 'coingecko'
|
||||
qualify ROW_NUMBER() over (
|
||||
PARTITION BY HOUR,
|
||||
asset_id
|
||||
ORDER BY
|
||||
modified_timestamp DESC
|
||||
) = 1
|
||||
),
|
||||
|
||||
-- Get best price for each token/blockchain/hour combination (THIRD priority - fallback)
|
||||
prices AS (
|
||||
SELECT
|
||||
HOUR,
|
||||
blockchain,
|
||||
symbol,
|
||||
price,
|
||||
is_verified,
|
||||
is_imputed,
|
||||
-- Flag to track if this is a fallback price
|
||||
CASE
|
||||
WHEN blockchain = 'cosmos' THEN 'cosmos_fallback'
|
||||
WHEN blockchain = 'ethereum' THEN 'ethereum_stablecoin_fallback'
|
||||
ELSE 'direct_match'
|
||||
END AS price_source_type
|
||||
FROM
|
||||
{{ source('crosschain_price','ez_prices_hourly')}}
|
||||
qualify ROW_NUMBER() over (
|
||||
PARTITION BY HOUR,
|
||||
blockchain,
|
||||
symbol
|
||||
ORDER BY
|
||||
is_verified DESC,
|
||||
-- Prioritize verified prices
|
||||
is_imputed ASC,
|
||||
-- Prefer non-imputed
|
||||
inserted_timestamp DESC
|
||||
) = 1
|
||||
),
|
||||
|
||||
-- GMPs with interchain_transfers ARRAY - use flattened transfer data with embedded prices
|
||||
gmp_with_interchain_transfers_array AS (
|
||||
SELECT
|
||||
f.id AS gmp_id,
|
||||
f.created_at,
|
||||
f.status,
|
||||
f.simplified_status,
|
||||
|
||||
-- Source chain from the parent GMP call
|
||||
LOWER(f.call:chain::STRING) AS source_chain,
|
||||
|
||||
-- Destination chain from the interchain_transfer object
|
||||
LOWER(it.value:destinationChain::STRING) AS destination_chain,
|
||||
|
||||
-- Callback information
|
||||
LOWER(f.data:callback:chain::STRING) AS callback_chain,
|
||||
LOWER(f.data:callback:destinationAddress::STRING) AS callback_destination_address,
|
||||
|
||||
-- Parent-child relationship
|
||||
f.call:parentMessageID::STRING AS parent_message_id,
|
||||
|
||||
-- Token information
|
||||
it.value:symbol::STRING AS token_symbol,
|
||||
it.value:amount AS token_amount,
|
||||
it.value:value AS token_amount_usd,
|
||||
it.value:price AS token_price_usd,
|
||||
'interchain_transfers_embedded' AS price_source,
|
||||
|
||||
-- Address information
|
||||
it.value:sourceAddress::STRING AS sender_address,
|
||||
it.value:destinationAddress::STRING AS destination_contract_address,
|
||||
it.value:recipient::STRING AS recipient_address,
|
||||
|
||||
-- Transaction identifiers
|
||||
f.call:transactionHash::STRING AS source_tx_hash,
|
||||
f.call:event::STRING AS source_event,
|
||||
f.call:blockNumber::NUMBER AS source_block_number,
|
||||
|
||||
-- Additional metadata
|
||||
it.value:contract_address::STRING AS token_contract_address,
|
||||
|
||||
-- Source type flag
|
||||
'interchain_transfers' AS data_source,
|
||||
|
||||
-- Generate unique ID for each flattened transfer
|
||||
{{ dbt_utils.generate_surrogate_key([
|
||||
'f.id',
|
||||
'it.index'
|
||||
]) }} AS ez_gmp_activity_id,
|
||||
|
||||
f.inserted_timestamp,
|
||||
SYSDATE() AS modified_timestamp
|
||||
|
||||
FROM {{ ref('axelscan__fact_gmp') }} f,
|
||||
LATERAL FLATTEN(f.data:interchain_transfers) it
|
||||
WHERE f.data:interchain_transfers IS NOT NULL
|
||||
AND ARRAY_SIZE(f.data:interchain_transfers) > 0
|
||||
|
||||
{% if is_incremental() %}
|
||||
AND f.modified_timestamp >= (
|
||||
SELECT MAX(modified_timestamp) - INTERVAL '3 DAYS'
|
||||
FROM {{ this }}
|
||||
)
|
||||
{% endif %}
|
||||
),
|
||||
|
||||
-- GMPs with interchain_transfer SINGLE OBJECT - needs price lookups
|
||||
gmp_with_interchain_transfer_single_base AS (
|
||||
SELECT
|
||||
f.id AS gmp_id,
|
||||
f.created_at,
|
||||
f.status,
|
||||
f.simplified_status,
|
||||
|
||||
-- Source chain from the parent GMP call
|
||||
LOWER(f.call:chain::STRING) AS source_chain,
|
||||
|
||||
-- Destination chain from the interchain_transfer object (singular)
|
||||
LOWER(f.data:interchain_transfer:destinationChain::STRING) AS destination_chain,
|
||||
|
||||
-- Callback information
|
||||
LOWER(f.data:callback:chain::STRING) AS callback_chain,
|
||||
LOWER(f.data:callback:destinationAddress::STRING) AS callback_destination_address,
|
||||
|
||||
-- Parent-child relationship
|
||||
f.call:parentMessageID::STRING AS parent_message_id,
|
||||
|
||||
-- Token information
|
||||
f.data:interchain_transfer:symbol::STRING AS token_symbol,
|
||||
f.data:interchain_transfer:amount AS token_amount,
|
||||
|
||||
-- Address information
|
||||
f.data:interchain_transfer:sourceAddress::STRING AS sender_address,
|
||||
f.data:interchain_transfer:destinationAddress::STRING AS destination_contract_address,
|
||||
f.data:interchain_transfer:rawDestinationAddress::STRING AS recipient_address,
|
||||
|
||||
-- Transaction identifiers
|
||||
f.call:transactionHash::STRING AS source_tx_hash,
|
||||
f.call:event::STRING AS source_event,
|
||||
f.call:blockNumber::NUMBER AS source_block_number,
|
||||
|
||||
-- Additional metadata
|
||||
f.data:interchain_transfer:contract_address::STRING AS token_contract_address,
|
||||
|
||||
-- Source type flag
|
||||
'interchain_transfer_single' AS data_source,
|
||||
|
||||
-- Generate unique ID
|
||||
{{ dbt_utils.generate_surrogate_key([
|
||||
'f.id',
|
||||
"'single'"
|
||||
]) }} AS ez_gmp_activity_id,
|
||||
|
||||
f.inserted_timestamp,
|
||||
SYSDATE() AS modified_timestamp
|
||||
|
||||
FROM {{ ref('axelscan__fact_gmp') }} f
|
||||
WHERE f.data:interchain_transfer IS NOT NULL
|
||||
AND f.data:interchain_transfer != '{}'
|
||||
|
||||
{% if is_incremental() %}
|
||||
AND f.modified_timestamp >= (
|
||||
SELECT MAX(modified_timestamp) - INTERVAL '3 DAYS'
|
||||
FROM {{ this }}
|
||||
)
|
||||
{% endif %}
|
||||
),
|
||||
|
||||
-- Apply price lookups to singular interchain_transfers
|
||||
gmp_with_interchain_transfer_single AS (
|
||||
SELECT
|
||||
g.gmp_id,
|
||||
g.created_at,
|
||||
g.status,
|
||||
g.simplified_status,
|
||||
g.source_chain,
|
||||
g.destination_chain,
|
||||
g.callback_chain,
|
||||
g.callback_destination_address,
|
||||
g.parent_message_id,
|
||||
g.token_symbol,
|
||||
g.token_amount,
|
||||
|
||||
-- Calculate USD amount using price waterfall
|
||||
CASE
|
||||
WHEN g.token_amount IS NOT NULL
|
||||
AND COALESCE(
|
||||
p_coingecko.price,
|
||||
p_direct.price,
|
||||
p_xrp.price,
|
||||
p_native.price,
|
||||
p_wrapped_eth.price,
|
||||
p_cosmos.price,
|
||||
p_stablecoin.price
|
||||
) IS NOT NULL THEN g.token_amount * COALESCE(
|
||||
p_coingecko.price,
|
||||
p_direct.price,
|
||||
p_xrp.price,
|
||||
p_native.price,
|
||||
p_wrapped_eth.price,
|
||||
p_cosmos.price,
|
||||
p_stablecoin.price
|
||||
)
|
||||
ELSE NULL
|
||||
END AS token_amount_usd,
|
||||
|
||||
-- Get the price
|
||||
COALESCE(
|
||||
p_coingecko.price,
|
||||
p_direct.price,
|
||||
p_xrp.price,
|
||||
p_native.price,
|
||||
p_wrapped_eth.price,
|
||||
p_cosmos.price,
|
||||
p_stablecoin.price
|
||||
) AS token_price_usd,
|
||||
|
||||
-- Track price source
|
||||
CASE
|
||||
WHEN p_coingecko.price IS NOT NULL THEN 'coingecko_lookup'
|
||||
WHEN p_direct.price IS NOT NULL THEN 'direct'
|
||||
WHEN p_xrp.price IS NOT NULL THEN 'xrp_blockchain'
|
||||
WHEN p_native.price IS NOT NULL THEN 'native_unwrapped'
|
||||
WHEN p_wrapped_eth.price IS NOT NULL THEN 'ethereum_unwrapped'
|
||||
WHEN p_cosmos.price IS NOT NULL THEN 'cosmos_fallback'
|
||||
WHEN p_stablecoin.price IS NOT NULL THEN 'ethereum_stablecoin_fallback'
|
||||
ELSE NULL
|
||||
END AS price_source,
|
||||
|
||||
g.sender_address,
|
||||
g.destination_contract_address,
|
||||
g.recipient_address,
|
||||
g.source_tx_hash,
|
||||
g.source_event,
|
||||
g.source_block_number,
|
||||
g.token_contract_address,
|
||||
g.data_source,
|
||||
g.ez_gmp_activity_id,
|
||||
g.inserted_timestamp,
|
||||
g.modified_timestamp
|
||||
|
||||
FROM gmp_with_interchain_transfer_single_base g
|
||||
-- Join to token metadata to get coingecko_id
|
||||
LEFT JOIN token_meta tm
|
||||
ON g.token_symbol = tm.token_symbol
|
||||
-- Join to CoinGecko prices (FIRST priority for singular)
|
||||
LEFT JOIN coingecko_prices p_coingecko
|
||||
ON DATE_TRUNC('hour', g.created_at) = p_coingecko.hour
|
||||
AND tm.coingecko_id = p_coingecko.coingecko_id
|
||||
-- First try: Direct blockchain match
|
||||
LEFT JOIN prices p_direct
|
||||
ON DATE_TRUNC('hour', g.created_at) = p_direct.hour
|
||||
AND (
|
||||
g.token_symbol = p_direct.symbol
|
||||
OR (
|
||||
g.token_symbol LIKE 'axl%'
|
||||
AND REGEXP_REPLACE(g.token_symbol, '^axl', '') = p_direct.symbol
|
||||
)
|
||||
)
|
||||
AND CASE
|
||||
WHEN g.source_chain = 'binance' THEN 'bsc'
|
||||
WHEN g.source_chain = 'moonbeam' THEN 'moonbeam'
|
||||
WHEN g.source_chain = 'kava' THEN 'kava'
|
||||
ELSE g.source_chain
|
||||
END = p_direct.blockchain
|
||||
-- Second try: XRP on xrp blockchain
|
||||
LEFT JOIN prices p_xrp
|
||||
ON p_direct.price IS NULL
|
||||
AND DATE_TRUNC('hour', g.created_at) = p_xrp.hour
|
||||
AND g.token_symbol IN ('XRP', 'mXRP')
|
||||
AND p_xrp.symbol = 'XRP'
|
||||
AND p_xrp.blockchain = 'xrp'
|
||||
-- Third try: Unwrap Axelar tokens and match to native chain
|
||||
LEFT JOIN prices p_native
|
||||
ON p_direct.price IS NULL
|
||||
AND p_xrp.price IS NULL
|
||||
AND DATE_TRUNC('hour', g.created_at) = p_native.hour
|
||||
AND g.token_symbol LIKE 'axl%'
|
||||
AND REGEXP_REPLACE(g.token_symbol, '^axl', '') = p_native.symbol
|
||||
AND CASE
|
||||
WHEN REGEXP_REPLACE(g.token_symbol, '^axl', '') IN ('BNB', 'BUSD', 'BTCB') THEN p_native.blockchain = 'bsc'
|
||||
WHEN REGEXP_REPLACE(g.token_symbol, '^axl', '') IN ('MATIC', 'PolygonUSDC', 'PolygonUSDT') THEN p_native.blockchain = 'polygon'
|
||||
WHEN REGEXP_REPLACE(g.token_symbol, '^axl', '') IN ('AVAX', 'AvalancheUSDC') THEN p_native.blockchain = 'avalanche'
|
||||
WHEN REGEXP_REPLACE(g.token_symbol, '^axl', '') IN ('FTM') THEN p_native.blockchain = 'fantom'
|
||||
WHEN REGEXP_REPLACE(g.token_symbol, '^axl', '') IN ('OP', 'OptimismUSDT', 'OptimismUSDC') THEN p_native.blockchain = 'optimism'
|
||||
WHEN REGEXP_REPLACE(g.token_symbol, '^axl', '') IN ('ARB', 'ArbitrumUSDT', 'ArbitrumUSDC') THEN p_native.blockchain = 'arbitrum'
|
||||
WHEN REGEXP_REPLACE(g.token_symbol, '^axl', '') = 'REGEN' THEN p_native.blockchain = 'regen'
|
||||
WHEN REGEXP_REPLACE(g.token_symbol, '^axl', '') = 'DEUS' THEN p_native.blockchain = 'fantom'
|
||||
ELSE FALSE
|
||||
END
|
||||
-- Fourth try: Unwrap Axelar tokens and use Ethereum price
|
||||
LEFT JOIN prices p_wrapped_eth
|
||||
ON p_direct.price IS NULL
|
||||
AND p_xrp.price IS NULL
|
||||
AND p_native.price IS NULL
|
||||
AND DATE_TRUNC('hour', g.created_at) = p_wrapped_eth.hour
|
||||
AND g.token_symbol LIKE 'axl%'
|
||||
AND REGEXP_REPLACE(g.token_symbol, '^axl', '') = p_wrapped_eth.symbol
|
||||
AND p_wrapped_eth.blockchain = 'ethereum'
|
||||
-- Fifth try: Cosmos ecosystem fallback
|
||||
LEFT JOIN prices p_cosmos
|
||||
ON p_direct.price IS NULL
|
||||
AND p_xrp.price IS NULL
|
||||
AND p_native.price IS NULL
|
||||
AND p_wrapped_eth.price IS NULL
|
||||
AND DATE_TRUNC('hour', g.created_at) = p_cosmos.hour
|
||||
AND (
|
||||
g.token_symbol = p_cosmos.symbol
|
||||
OR (
|
||||
g.token_symbol LIKE 'axl%'
|
||||
AND REGEXP_REPLACE(g.token_symbol, '^axl', '') = p_cosmos.symbol
|
||||
)
|
||||
)
|
||||
AND p_cosmos.blockchain = 'cosmos'
|
||||
AND g.source_chain IN (
|
||||
'axelar', 'osmosis', 'cosmos', 'cosmoshub', 'neutron',
|
||||
'terra', 'terra-2', 'agoric', 'kujira', 'umee', 'stride',
|
||||
'regen', 'juno', 'evmos', 'crescent', 'kava', 'secret',
|
||||
'injective', 'sei', 'celestia', 'dydx'
|
||||
)
|
||||
-- Sixth try: Ethereum stablecoin fallback
|
||||
LEFT JOIN prices p_stablecoin
|
||||
ON p_direct.price IS NULL
|
||||
AND p_xrp.price IS NULL
|
||||
AND p_native.price IS NULL
|
||||
AND p_wrapped_eth.price IS NULL
|
||||
AND p_cosmos.price IS NULL
|
||||
AND DATE_TRUNC('hour', g.created_at) = p_stablecoin.hour
|
||||
AND (
|
||||
g.token_symbol = p_stablecoin.symbol
|
||||
OR (
|
||||
g.token_symbol LIKE 'axl%'
|
||||
AND REGEXP_REPLACE(g.token_symbol, '^axl', '') = p_stablecoin.symbol
|
||||
)
|
||||
)
|
||||
AND p_stablecoin.blockchain = 'ethereum'
|
||||
AND (
|
||||
p_stablecoin.symbol IN ('USDC', 'USDT', 'DAI', 'BUSD', 'FRAX', 'TUSD', 'USDP', 'GUSD', 'LUSD')
|
||||
OR REGEXP_REPLACE(g.token_symbol, '^axl', '') IN ('USDC', 'USDT', 'DAI', 'BUSD', 'FRAX', 'TUSD', 'USDP', 'GUSD', 'LUSD')
|
||||
)
|
||||
),
|
||||
|
||||
-- GMPs without ANY interchain transfer data - build from fact_gmp with inline parsing
|
||||
gmp_without_interchain_transfers_base AS (
|
||||
SELECT
|
||||
f.id AS gmp_id,
|
||||
f.created_at,
|
||||
f.status,
|
||||
f.simplified_status,
|
||||
|
||||
-- Source chain from call
|
||||
TRIM(REPLACE(LOWER(f.call:chain::STRING), 'axelarnet', 'axelar')) AS source_chain,
|
||||
|
||||
-- Destination chain from call returnValues
|
||||
TRIM(REPLACE(LOWER(f.call:returnValues:destinationChain::STRING), 'axelarnet', 'axelar')) AS destination_chain,
|
||||
|
||||
-- Callback information
|
||||
LOWER(f.data:callback:chain::STRING) AS callback_chain,
|
||||
LOWER(f.data:callback:destinationAddress::STRING) AS callback_destination_address,
|
||||
|
||||
-- Parent-child relationship
|
||||
f.call:parentMessageID::STRING AS parent_message_id,
|
||||
|
||||
-- Token information (with fallback to DATA object for ContractCall events)
|
||||
COALESCE(
|
||||
f.call:returnValues:symbol::STRING,
|
||||
f.data:symbol::STRING
|
||||
) AS token_symbol,
|
||||
|
||||
COALESCE(
|
||||
f.amount,
|
||||
f.data:amount::FLOAT
|
||||
) AS token_amount,
|
||||
|
||||
-- Address information
|
||||
LOWER(f.call:transaction:from::STRING) AS sender_address,
|
||||
LOWER(f.call:returnValues:destinationContractAddress::STRING) AS destination_contract_address,
|
||||
NULL AS recipient_address,
|
||||
|
||||
-- Transaction identifiers
|
||||
f.call:transactionHash::STRING AS source_tx_hash,
|
||||
f.call:event::STRING AS source_event,
|
||||
f.call:blockNumber::NUMBER AS source_block_number,
|
||||
|
||||
-- Additional metadata
|
||||
NULL AS token_contract_address,
|
||||
|
||||
-- Source type flag
|
||||
'standard_gmp' AS data_source,
|
||||
|
||||
-- Generate unique ID
|
||||
{{ dbt_utils.generate_surrogate_key([
|
||||
'f.id'
|
||||
]) }} AS ez_gmp_activity_id,
|
||||
|
||||
f.inserted_timestamp,
|
||||
SYSDATE() AS modified_timestamp
|
||||
|
||||
FROM {{ ref('axelscan__fact_gmp') }} f
|
||||
WHERE NOT (
|
||||
-- Exclude GMPs that have interchain_transfers ARRAY
|
||||
(f.data:interchain_transfers IS NOT NULL AND ARRAY_SIZE(f.data:interchain_transfers) > 0)
|
||||
OR
|
||||
-- Exclude GMPs that have interchain_transfer OBJECT
|
||||
(f.data:interchain_transfer IS NOT NULL AND f.data:interchain_transfer != '{}')
|
||||
)
|
||||
|
||||
{% if is_incremental() %}
|
||||
AND f.modified_timestamp >= (
|
||||
SELECT MAX(modified_timestamp) - INTERVAL '3 DAYS'
|
||||
FROM {{ this }}
|
||||
)
|
||||
{% endif %}
|
||||
),
|
||||
|
||||
-- Apply price lookups to standard GMPs
|
||||
gmp_without_interchain_transfers AS (
|
||||
SELECT
|
||||
g.gmp_id,
|
||||
g.created_at,
|
||||
g.status,
|
||||
g.simplified_status,
|
||||
g.source_chain,
|
||||
g.destination_chain,
|
||||
g.callback_chain,
|
||||
g.callback_destination_address,
|
||||
g.parent_message_id,
|
||||
g.token_symbol,
|
||||
g.token_amount,
|
||||
|
||||
-- Calculate USD amount using price waterfall
|
||||
CASE
|
||||
WHEN g.token_amount IS NOT NULL
|
||||
AND COALESCE(
|
||||
p_coingecko.price,
|
||||
p_direct.price,
|
||||
p_xrp.price,
|
||||
p_native.price,
|
||||
p_wrapped_eth.price,
|
||||
p_cosmos.price,
|
||||
p_stablecoin.price
|
||||
) IS NOT NULL THEN g.token_amount * COALESCE(
|
||||
p_coingecko.price,
|
||||
p_direct.price,
|
||||
p_xrp.price,
|
||||
p_native.price,
|
||||
p_wrapped_eth.price,
|
||||
p_cosmos.price,
|
||||
p_stablecoin.price
|
||||
)
|
||||
ELSE NULL
|
||||
END AS token_amount_usd,
|
||||
|
||||
-- Get the price
|
||||
COALESCE(
|
||||
p_coingecko.price,
|
||||
p_direct.price,
|
||||
p_xrp.price,
|
||||
p_native.price,
|
||||
p_wrapped_eth.price,
|
||||
p_cosmos.price,
|
||||
p_stablecoin.price
|
||||
) AS token_price_usd,
|
||||
|
||||
-- Track price source
|
||||
CASE
|
||||
WHEN p_coingecko.price IS NOT NULL THEN 'coingecko_lookup'
|
||||
WHEN p_direct.price IS NOT NULL THEN 'direct'
|
||||
WHEN p_xrp.price IS NOT NULL THEN 'xrp_blockchain'
|
||||
WHEN p_native.price IS NOT NULL THEN 'native_unwrapped'
|
||||
WHEN p_wrapped_eth.price IS NOT NULL THEN 'ethereum_unwrapped'
|
||||
WHEN p_cosmos.price IS NOT NULL THEN 'cosmos_fallback'
|
||||
WHEN p_stablecoin.price IS NOT NULL THEN 'ethereum_stablecoin_fallback'
|
||||
ELSE NULL
|
||||
END AS price_source,
|
||||
|
||||
g.sender_address,
|
||||
g.destination_contract_address,
|
||||
g.recipient_address,
|
||||
g.source_tx_hash,
|
||||
g.source_event,
|
||||
g.source_block_number,
|
||||
g.token_contract_address,
|
||||
g.data_source,
|
||||
g.ez_gmp_activity_id,
|
||||
g.inserted_timestamp,
|
||||
g.modified_timestamp
|
||||
|
||||
FROM gmp_without_interchain_transfers_base g
|
||||
-- Join to token metadata to get coingecko_id
|
||||
LEFT JOIN token_meta tm
|
||||
ON g.token_symbol = tm.token_symbol
|
||||
-- Join to CoinGecko prices
|
||||
LEFT JOIN coingecko_prices p_coingecko
|
||||
ON DATE_TRUNC('hour', g.created_at) = p_coingecko.hour
|
||||
AND tm.coingecko_id = p_coingecko.coingecko_id
|
||||
-- Direct blockchain match
|
||||
LEFT JOIN prices p_direct
|
||||
ON DATE_TRUNC('hour', g.created_at) = p_direct.hour
|
||||
AND (
|
||||
g.token_symbol = p_direct.symbol
|
||||
OR (
|
||||
g.token_symbol LIKE 'axl%'
|
||||
AND REGEXP_REPLACE(g.token_symbol, '^axl', '') = p_direct.symbol
|
||||
)
|
||||
)
|
||||
AND CASE
|
||||
WHEN g.source_chain = 'binance' THEN 'bsc'
|
||||
WHEN g.source_chain = 'moonbeam' THEN 'moonbeam'
|
||||
WHEN g.source_chain = 'kava' THEN 'kava'
|
||||
ELSE g.source_chain
|
||||
END = p_direct.blockchain
|
||||
-- XRP fallback
|
||||
LEFT JOIN prices p_xrp
|
||||
ON p_direct.price IS NULL
|
||||
AND DATE_TRUNC('hour', g.created_at) = p_xrp.hour
|
||||
AND g.token_symbol IN ('XRP', 'mXRP')
|
||||
AND p_xrp.symbol = 'XRP'
|
||||
AND p_xrp.blockchain = 'xrp'
|
||||
-- Unwrap Axelar tokens - native chain
|
||||
LEFT JOIN prices p_native
|
||||
ON p_direct.price IS NULL
|
||||
AND p_xrp.price IS NULL
|
||||
AND DATE_TRUNC('hour', g.created_at) = p_native.hour
|
||||
AND g.token_symbol LIKE 'axl%'
|
||||
AND REGEXP_REPLACE(g.token_symbol, '^axl', '') = p_native.symbol
|
||||
AND CASE
|
||||
WHEN REGEXP_REPLACE(g.token_symbol, '^axl', '') IN ('BNB', 'BUSD', 'BTCB') THEN p_native.blockchain = 'bsc'
|
||||
WHEN REGEXP_REPLACE(g.token_symbol, '^axl', '') IN ('MATIC', 'PolygonUSDC', 'PolygonUSDT') THEN p_native.blockchain = 'polygon'
|
||||
WHEN REGEXP_REPLACE(g.token_symbol, '^axl', '') IN ('AVAX', 'AvalancheUSDC') THEN p_native.blockchain = 'avalanche'
|
||||
WHEN REGEXP_REPLACE(g.token_symbol, '^axl', '') IN ('FTM') THEN p_native.blockchain = 'fantom'
|
||||
WHEN REGEXP_REPLACE(g.token_symbol, '^axl', '') IN ('OP', 'OptimismUSDT', 'OptimismUSDC') THEN p_native.blockchain = 'optimism'
|
||||
WHEN REGEXP_REPLACE(g.token_symbol, '^axl', '') IN ('ARB', 'ArbitrumUSDT', 'ArbitrumUSDC') THEN p_native.blockchain = 'arbitrum'
|
||||
WHEN REGEXP_REPLACE(g.token_symbol, '^axl', '') = 'REGEN' THEN p_native.blockchain = 'regen'
|
||||
WHEN REGEXP_REPLACE(g.token_symbol, '^axl', '') = 'DEUS' THEN p_native.blockchain = 'fantom'
|
||||
ELSE FALSE
|
||||
END
|
||||
-- Unwrap Axelar tokens - Ethereum fallback
|
||||
LEFT JOIN prices p_wrapped_eth
|
||||
ON p_direct.price IS NULL
|
||||
AND p_xrp.price IS NULL
|
||||
AND p_native.price IS NULL
|
||||
AND DATE_TRUNC('hour', g.created_at) = p_wrapped_eth.hour
|
||||
AND g.token_symbol LIKE 'axl%'
|
||||
AND REGEXP_REPLACE(g.token_symbol, '^axl', '') = p_wrapped_eth.symbol
|
||||
AND p_wrapped_eth.blockchain = 'ethereum'
|
||||
-- Cosmos ecosystem fallback
|
||||
LEFT JOIN prices p_cosmos
|
||||
ON p_direct.price IS NULL
|
||||
AND p_xrp.price IS NULL
|
||||
AND p_native.price IS NULL
|
||||
AND p_wrapped_eth.price IS NULL
|
||||
AND DATE_TRUNC('hour', g.created_at) = p_cosmos.hour
|
||||
AND (
|
||||
g.token_symbol = p_cosmos.symbol
|
||||
OR (
|
||||
g.token_symbol LIKE 'axl%'
|
||||
AND REGEXP_REPLACE(g.token_symbol, '^axl', '') = p_cosmos.symbol
|
||||
)
|
||||
)
|
||||
AND p_cosmos.blockchain = 'cosmos'
|
||||
AND g.source_chain IN (
|
||||
'axelar', 'osmosis', 'cosmos', 'cosmoshub', 'neutron',
|
||||
'terra', 'terra-2', 'agoric', 'kujira', 'umee', 'stride',
|
||||
'regen', 'juno', 'evmos', 'crescent', 'kava', 'secret',
|
||||
'injective', 'sei', 'celestia', 'dydx'
|
||||
)
|
||||
-- Ethereum stablecoin fallback
|
||||
LEFT JOIN prices p_stablecoin
|
||||
ON p_direct.price IS NULL
|
||||
AND p_xrp.price IS NULL
|
||||
AND p_native.price IS NULL
|
||||
AND p_wrapped_eth.price IS NULL
|
||||
AND p_cosmos.price IS NULL
|
||||
AND DATE_TRUNC('hour', g.created_at) = p_stablecoin.hour
|
||||
AND (
|
||||
g.token_symbol = p_stablecoin.symbol
|
||||
OR (
|
||||
g.token_symbol LIKE 'axl%'
|
||||
AND REGEXP_REPLACE(g.token_symbol, '^axl', '') = p_stablecoin.symbol
|
||||
)
|
||||
)
|
||||
AND p_stablecoin.blockchain = 'ethereum'
|
||||
AND (
|
||||
p_stablecoin.symbol IN ('USDC', 'USDT', 'DAI', 'BUSD', 'FRAX', 'TUSD', 'USDP', 'GUSD', 'LUSD')
|
||||
OR REGEXP_REPLACE(g.token_symbol, '^axl', '') IN ('USDC', 'USDT', 'DAI', 'BUSD', 'FRAX', 'TUSD', 'USDP', 'GUSD', 'LUSD')
|
||||
)
|
||||
),
|
||||
|
||||
-- Combine all three sources
|
||||
combined AS (
|
||||
SELECT * FROM gmp_with_interchain_transfers_array
|
||||
UNION ALL
|
||||
SELECT * FROM gmp_with_interchain_transfer_single
|
||||
UNION ALL
|
||||
SELECT * FROM gmp_without_interchain_transfers
|
||||
)
|
||||
|
||||
SELECT
|
||||
-- Core identifiers
|
||||
gmp_id,
|
||||
ez_gmp_activity_id,
|
||||
|
||||
-- Timestamps
|
||||
created_at,
|
||||
|
||||
-- Status
|
||||
status,
|
||||
simplified_status,
|
||||
|
||||
-- Chain routing
|
||||
source_chain,
|
||||
destination_chain,
|
||||
callback_chain,
|
||||
callback_destination_address,
|
||||
|
||||
-- Parent-child relationships
|
||||
parent_message_id,
|
||||
|
||||
-- Token and amounts
|
||||
token_symbol,
|
||||
token_amount,
|
||||
token_amount_usd,
|
||||
token_price_usd,
|
||||
price_source,
|
||||
|
||||
-- Addresses
|
||||
sender_address,
|
||||
destination_contract_address,
|
||||
recipient_address,
|
||||
token_contract_address,
|
||||
|
||||
-- Transaction details
|
||||
source_tx_hash,
|
||||
source_event,
|
||||
source_block_number,
|
||||
|
||||
-- Metadata
|
||||
data_source,
|
||||
|
||||
-- System timestamps
|
||||
inserted_timestamp,
|
||||
modified_timestamp
|
||||
FROM combined
|
||||
WHERE parent_message_id IS NULL -- Exclude child GMPs to match metrics counting methodology
|
||||
AND source_chain IS NOT NULL -- Exclude records with missing chain data (87 historical records)
|
||||
AND destination_chain IS NOT NULL
|
||||
QUALIFY ROW_NUMBER() OVER (
|
||||
PARTITION BY ez_gmp_activity_id
|
||||
ORDER BY modified_timestamp DESC
|
||||
) = 1
|
||||
160
models/gold/axelscan/axelscan__ez_gmp_activity.yml
Normal file
160
models/gold/axelscan/axelscan__ez_gmp_activity.yml
Normal file
@ -0,0 +1,160 @@
|
||||
version: 2
|
||||
|
||||
models:
|
||||
- name: axelscan__ez_gmp_activity
|
||||
description: |
|
||||
**Enhanced Combined EZ GMP Model - The definitive model for all GMP transfer analysis**
|
||||
|
||||
This model provides a complete, unified view of **user-initiated GMP transfers** across the Axelar network by
|
||||
intelligently combining three complementary data sources. Child GMPs (routing hops with parent_message_id) are
|
||||
excluded to match the metrics API counting methodology, resulting in 99.95% accuracy vs ez_bridge_metrics.
|
||||
|
||||
1. **interchain_transfers (array)** - Multiple transfers within a single GMP transaction
|
||||
- Primarily from Fantom multi-hop transfers
|
||||
- Contains embedded prices from the API
|
||||
- ~3% of total transfers (448 records over 7 days)
|
||||
|
||||
2. **interchain_transfer (singular object)** - Single transfer within a GMP transaction
|
||||
- Primarily from XRPL, Axelar hub, and cross-chain transfers
|
||||
- Requires price lookups (added in this enhanced version)
|
||||
- ~21% of total transfers (3,059 records over 7 days)
|
||||
|
||||
3. **standard_gmp** - Traditional GMP transactions without interchain transfer metadata
|
||||
- All other GMP transactions
|
||||
- Uses price lookups from ez_gmp_activity
|
||||
- ~76% of total transfers
|
||||
|
||||
**Key Improvements Over Previous Models:**
|
||||
- ✅ Price lookups added for singular interchain_transfers (previously NULL)
|
||||
- ✅ Callback chain information included
|
||||
- ✅ Parent-child relationship tracking for multi-hop GMPs
|
||||
- ✅ **Excludes child GMPs** (parent_message_id IS NULL) to match metrics methodology
|
||||
- ✅ Accurate destination chains from interchain transfer objects
|
||||
- ✅ One row per individual transfer (proper granularity)
|
||||
- ✅ No double-counting (exclusion logic prevents overlap)
|
||||
- ✅ Transparent data source tracking
|
||||
|
||||
**Use Cases:**
|
||||
- Volume analysis by path (source → destination)
|
||||
- Multi-hop transfer analysis
|
||||
- Callback pattern analysis (especially XRPL)
|
||||
- Cross-chain token flow tracking
|
||||
- Emerging chain adoption tracking
|
||||
|
||||
columns:
|
||||
- name: ez_gmp_activity_id
|
||||
description: Unique identifier for each transfer (primary key)
|
||||
tests:
|
||||
- unique
|
||||
- not_null
|
||||
|
||||
- name: gmp_id
|
||||
description: Parent GMP transaction ID (multiple transfers can share same gmp_id)
|
||||
|
||||
- name: created_at
|
||||
description: Timestamp when the GMP transaction was created
|
||||
tests:
|
||||
- not_null
|
||||
|
||||
- name: status
|
||||
description: Detailed status of the GMP transaction (executed, error, etc.)
|
||||
|
||||
- name: simplified_status
|
||||
description: Simplified status category
|
||||
|
||||
- name: source_chain
|
||||
description: Origin blockchain of the transfer
|
||||
|
||||
- name: destination_chain
|
||||
description: |
|
||||
Destination blockchain of the transfer. For interchain_transfers, this is the ACTUAL
|
||||
destination from the transfer object (more accurate than parent GMP destination)
|
||||
|
||||
- name: callback_chain
|
||||
description: |
|
||||
Chain where callback should be executed. Primarily populated for XRPL transfers.
|
||||
~9% of transfers have callback information.
|
||||
|
||||
- name: callback_destination_address
|
||||
description: Destination address for callback execution
|
||||
|
||||
- name: parent_message_id
|
||||
description: |
|
||||
ID of the parent GMP transaction if this GMP is part of a parent-child chain.
|
||||
**NOTE: This field will always be NULL in this model** because we filter out child GMPs
|
||||
(WHERE parent_message_id IS NULL) to match metrics counting methodology.
|
||||
This field is retained for reference and potential future analysis needs.
|
||||
|
||||
- name: token_symbol
|
||||
description: Symbol of the token being transferred (USDC, USDT, XRP, etc.)
|
||||
|
||||
- name: token_amount
|
||||
description: Raw token amount transferred (in token's native units)
|
||||
|
||||
- name: token_amount_usd
|
||||
description: |
|
||||
USD value of the transfer. Calculated using price_source waterfall:
|
||||
- For interchain_transfers (array): embedded price from API
|
||||
- For interchain_transfer (single): price lookup waterfall
|
||||
- For standard_gmp: price lookup from ez_gmp_activity
|
||||
|
||||
- name: token_price_usd
|
||||
description: USD price per token unit at the time of transfer
|
||||
|
||||
- name: price_source
|
||||
description: |
|
||||
Source of the price data:
|
||||
- interchain_transfers_embedded: From array transfer API (most accurate)
|
||||
- coingecko_lookup: From CoinGecko via token metadata
|
||||
- direct: Direct blockchain match in ez_prices_hourly
|
||||
- xrp_blockchain: XRP-specific pricing
|
||||
- native_unwrapped: Native chain pricing for wrapped tokens
|
||||
- ethereum_unwrapped: Ethereum pricing for wrapped tokens
|
||||
- cosmos_fallback: Cosmos ecosystem fallback
|
||||
- ethereum_stablecoin_fallback: Ethereum stablecoin pricing
|
||||
|
||||
- name: sender_address
|
||||
description: Address initiating the transfer on source chain
|
||||
|
||||
- name: destination_contract_address
|
||||
description: Destination contract address on destination chain
|
||||
|
||||
- name: recipient_address
|
||||
description: |
|
||||
Final recipient address. Only populated for interchain_transfers where
|
||||
recipient may differ from destination_contract_address
|
||||
|
||||
- name: token_contract_address
|
||||
description: Contract address of the token being transferred
|
||||
|
||||
- name: source_tx_hash
|
||||
description: Transaction hash on the source chain
|
||||
|
||||
- name: source_event
|
||||
description: Event type that triggered the GMP
|
||||
|
||||
- name: source_block_number
|
||||
description: Block number on source chain where transaction occurred
|
||||
|
||||
- name: data_source
|
||||
description: |
|
||||
Indicates which of the three data sources provided this transfer:
|
||||
- interchain_transfers: From array structure (multi-hop)
|
||||
- interchain_transfer_single: From singular object structure
|
||||
- standard_gmp: From standard GMP activity without interchain transfer metadata
|
||||
tests:
|
||||
- not_null
|
||||
- accepted_values:
|
||||
values: ['interchain_transfers', 'interchain_transfer_single', 'standard_gmp']
|
||||
|
||||
- name: inserted_timestamp
|
||||
description: When the record was first inserted
|
||||
|
||||
- name: modified_timestamp
|
||||
description: When the record was last modified
|
||||
|
||||
tests:
|
||||
- dbt_utils.recency:
|
||||
datepart: day
|
||||
field: created_at
|
||||
interval: 3
|
||||
79
models/gold/axelscan/axelscan__ez_transfer_activity.sql
Normal file
79
models/gold/axelscan/axelscan__ez_transfer_activity.sql
Normal file
@ -0,0 +1,79 @@
|
||||
{{ config(
|
||||
materialized = 'incremental',
|
||||
unique_key = 'ez_transfer_activity_id',
|
||||
incremental_strategy = 'delete+insert',
|
||||
cluster_by = 'created_at::DATE',
|
||||
tags = ['daily']
|
||||
) }}
|
||||
|
||||
SELECT
|
||||
-- Transaction identifiers
|
||||
id AS transfer_id,
|
||||
created_at,
|
||||
status,
|
||||
simplified_status,
|
||||
|
||||
-- Source chain info
|
||||
TRIM(REPLACE(LOWER(source_chain), 'axelarnet', 'axelar')) AS source_chain,
|
||||
LOWER(sender_address) AS sender_address,
|
||||
|
||||
-- Destination chain info
|
||||
TRIM(REPLACE(LOWER(destination_chain), 'axelarnet', 'axelar')) AS destination_chain,
|
||||
LOWER(recipient_address) AS recipient_address,
|
||||
|
||||
-- Token and amount details
|
||||
send_denom AS token_denom,
|
||||
send_amount AS amount,
|
||||
send_amount_received AS amount_received,
|
||||
send_fee AS fee,
|
||||
|
||||
-- Additional transaction details from nested send object
|
||||
send:txhash::STRING AS source_tx_hash,
|
||||
send:height::NUMBER AS source_height,
|
||||
send:type::STRING AS transfer_type,
|
||||
send:insufficient_fee::BOOLEAN AS insufficient_fee,
|
||||
send:value::FLOAT AS amount_value_usd,
|
||||
send:fee_value::FLOAT AS fee_value_usd,
|
||||
|
||||
-- Confirmation details from nested confirm object
|
||||
DATA:confirm:txhash::STRING AS confirm_tx_hash,
|
||||
DATA:confirm:height::NUMBER AS confirm_height,
|
||||
LOWER(DATA:confirm:deposit_address::STRING) AS deposit_address,
|
||||
DATA:confirm:transfer_id::NUMBER AS axelar_transfer_id,
|
||||
|
||||
-- IBC send details from nested ibc_send object
|
||||
DATA:ibc_send:ack_txhash::STRING AS ibc_ack_tx_hash,
|
||||
DATA:ibc_send:failed_txhash::STRING AS ibc_failed_tx_hash,
|
||||
DATA:ibc_send:height::NUMBER AS ibc_height,
|
||||
DATA:ibc_send:packet:packet_sequence::NUMBER AS ibc_packet_sequence,
|
||||
DATA:ibc_send:packet:packet_src_channel::STRING AS ibc_src_channel,
|
||||
DATA:ibc_send:packet:packet_dst_channel::STRING AS ibc_dst_channel,
|
||||
|
||||
-- Link information
|
||||
link,
|
||||
|
||||
-- Full nested objects for reference
|
||||
send,
|
||||
DATA:confirm AS confirm,
|
||||
DATA:ibc_send AS ibc_send,
|
||||
|
||||
-- Generated fields
|
||||
{{ dbt_utils.generate_surrogate_key(
|
||||
['id']
|
||||
) }} AS ez_transfer_activity_id,
|
||||
fact_transfers_id,
|
||||
sysdate() inserted_timestamp,
|
||||
sysdate() modified_timestamp
|
||||
|
||||
FROM
|
||||
{{ ref('axelscan__fact_transfers') }}
|
||||
|
||||
{% if is_incremental() %}
|
||||
WHERE
|
||||
modified_timestamp >= (
|
||||
SELECT
|
||||
MAX(modified_timestamp)
|
||||
FROM
|
||||
{{ this }}
|
||||
)
|
||||
{% endif %}
|
||||
85
models/gold/axelscan/axelscan__ez_transfer_activity.yml
Normal file
85
models/gold/axelscan/axelscan__ez_transfer_activity.yml
Normal file
@ -0,0 +1,85 @@
|
||||
version: 2
|
||||
models:
|
||||
- name: axelscan__ez_transfer_activity
|
||||
description: Flattened view of Axelar token transfer activity with parsed transaction details, source/destination chains, sender/receiver information, and IBC transfer metadata. This table provides an easy-to-query format for analyzing cross-chain token transfers on Axelar.
|
||||
columns:
|
||||
- name: TRANSFER_ID
|
||||
description: The unique identifier for the transfer transaction
|
||||
tests:
|
||||
- not_null
|
||||
- unique
|
||||
- name: CREATED_AT
|
||||
description: The timestamp when the transfer was created
|
||||
tests:
|
||||
- not_null
|
||||
- name: STATUS
|
||||
description: The detailed status of the transfer (e.g., success, failed)
|
||||
- name: SIMPLIFIED_STATUS
|
||||
description: Simplified transfer status
|
||||
- name: SOURCE_CHAIN
|
||||
description: The name of the source blockchain (lowercase, 'axelarnet' replaced with 'axelar')
|
||||
- name: SENDER_ADDRESS
|
||||
description: The address that initiated the transfer on the source chain
|
||||
- name: DESTINATION_CHAIN
|
||||
description: The name of the destination blockchain (lowercase, 'axelarnet' replaced with 'axelar')
|
||||
- name: RECIPIENT_ADDRESS
|
||||
description: The address that receives the transfer on the destination chain
|
||||
- name: TOKEN_DENOM
|
||||
description: The denomination of the token being transferred (e.g., uusdc, bnb-wei)
|
||||
- name: AMOUNT
|
||||
description: The transfer amount
|
||||
- name: AMOUNT_RECEIVED
|
||||
description: The amount received after fees
|
||||
- name: FEE
|
||||
description: The fee charged for the transfer
|
||||
- name: SOURCE_TX_HASH
|
||||
description: The transaction hash on the source chain
|
||||
- name: SOURCE_HEIGHT
|
||||
description: The block height on the source chain
|
||||
- name: TRANSFER_TYPE
|
||||
description: The type of transfer (e.g., ibc, bridge)
|
||||
- name: INSUFFICIENT_FEE
|
||||
description: Flag indicating whether the fee was insufficient
|
||||
- name: AMOUNT_VALUE_USD
|
||||
description: The USD value of the transfer amount
|
||||
- name: FEE_VALUE_USD
|
||||
description: The USD value of the fee
|
||||
- name: CONFIRM_TX_HASH
|
||||
description: The transaction hash for the deposit confirmation on Axelar
|
||||
- name: CONFIRM_HEIGHT
|
||||
description: The block height for the deposit confirmation
|
||||
- name: DEPOSIT_ADDRESS
|
||||
description: The deposit address on Axelar network
|
||||
- name: AXELAR_TRANSFER_ID
|
||||
description: The internal Axelar transfer ID
|
||||
- name: IBC_ACK_TX_HASH
|
||||
description: The IBC acknowledgment transaction hash
|
||||
- name: IBC_FAILED_TX_HASH
|
||||
description: The IBC failed transaction hash if applicable
|
||||
- name: IBC_HEIGHT
|
||||
description: The block height for the IBC send
|
||||
- name: IBC_PACKET_SEQUENCE
|
||||
description: The IBC packet sequence number
|
||||
- name: IBC_SRC_CHANNEL
|
||||
description: The IBC source channel identifier
|
||||
- name: IBC_DST_CHANNEL
|
||||
description: The IBC destination channel identifier
|
||||
- name: LINK
|
||||
description: Link object containing transaction URLs and references
|
||||
- name: SEND
|
||||
description: Full send object with all source transaction details
|
||||
- name: CONFIRM
|
||||
description: Full confirm object with deposit confirmation details
|
||||
- name: IBC_SEND
|
||||
description: Full IBC send object with inter-blockchain communication details
|
||||
- name: EZ_TRANSFER_ACTIVITY_ID
|
||||
description: '{{ doc("pk") }}'
|
||||
tests:
|
||||
- not_null
|
||||
- unique
|
||||
- name: FACT_TRANSFERS_ID
|
||||
description: Foreign key reference to axelscan__fact_transfers
|
||||
- name: INSERTED_TIMESTAMP
|
||||
description: '{{ doc("inserted_timestamp") }}'
|
||||
- name: MODIFIED_TIMESTAMP
|
||||
description: '{{ doc("modified_timestamp") }}'
|
||||
@ -1,8 +1,6 @@
|
||||
{{ config(
|
||||
materialized = 'view',
|
||||
meta ={ 'database_tags':{ 'table':{ 'PURPOSE': 'AXELSCAN',
|
||||
}} },
|
||||
tags = ['noncore']
|
||||
tags = ['daily']
|
||||
) }}
|
||||
|
||||
SELECT
|
||||
|
||||
@ -1,8 +1,6 @@
|
||||
{{ config(
|
||||
materialized = 'view',
|
||||
meta ={ 'database_tags':{ 'table':{ 'PURPOSE': 'AXELSCAN',
|
||||
}} },
|
||||
tags = ['noncore']
|
||||
tags = ['daily']
|
||||
) }}
|
||||
|
||||
SELECT
|
||||
|
||||
@ -0,0 +1,45 @@
|
||||
{{ config(
|
||||
materialized = 'incremental',
|
||||
unique_key = ['date_day', 'source_chain', 'destination_chain'],
|
||||
incremental_strategy = 'delete+insert',
|
||||
tags = ['daily']
|
||||
) }}
|
||||
|
||||
WITH parsed_gmp_stats AS (
|
||||
SELECT
|
||||
date_day,
|
||||
b.value :key :: STRING AS source_chain,
|
||||
C.value :key :: STRING AS destination_chain,
|
||||
C.value :num_txs :: INT AS num_txs,
|
||||
C.value :volume :: DECIMAL(18, 2) AS volume_usd,
|
||||
_inserted_timestamp
|
||||
FROM
|
||||
{{ ref('bronze__axelscan_gmp_stats_by_chains') }},
|
||||
LATERAL FLATTEN(resp :data :source_chains) b,
|
||||
LATERAL FLATTEN(b.value :destination_chains) C
|
||||
|
||||
{% if is_incremental() %}
|
||||
WHERE
|
||||
date_day >= (
|
||||
SELECT
|
||||
COALESCE(MAX(date_day), '1970-01-01' :: DATE) - 3
|
||||
FROM
|
||||
{{ this }}
|
||||
)
|
||||
{% endif %}
|
||||
)
|
||||
SELECT
|
||||
date_day,
|
||||
source_chain,
|
||||
destination_chain,
|
||||
num_txs,
|
||||
volume_usd,
|
||||
_inserted_timestamp,
|
||||
SYSDATE() AS modified_timestamp
|
||||
FROM
|
||||
parsed_gmp_stats
|
||||
QUALIFY
|
||||
ROW_NUMBER() OVER (
|
||||
PARTITION BY date_day, source_chain, destination_chain
|
||||
ORDER BY _inserted_timestamp DESC
|
||||
) = 1
|
||||
37
models/silver/defi/axelscan/silver__axelscan_token_meta.sql
Normal file
37
models/silver/defi/axelscan/silver__axelscan_token_meta.sql
Normal file
@ -0,0 +1,37 @@
|
||||
{{ config(
|
||||
materialized = 'incremental',
|
||||
unique_key = "token_symbol",
|
||||
tags = ['daily']
|
||||
) }}
|
||||
|
||||
SELECT
|
||||
token_symbol,
|
||||
coingecko_id,
|
||||
decimals,
|
||||
api_symbol,
|
||||
token_name,
|
||||
native_chain,
|
||||
token_type,
|
||||
chains_data,
|
||||
SYSDATE() AS modified_timestamp
|
||||
FROM
|
||||
{{ ref('bronze__axelscan_tokens_meta') }}
|
||||
|
||||
{% if is_incremental() %}
|
||||
WHERE
|
||||
modified_timestamp >= (
|
||||
SELECT
|
||||
COALESCE(MAX(modified_timestamp), '1970-01-01' :: TIMESTAMP) - INTERVAL '1 DAY'
|
||||
FROM
|
||||
{{ this }})
|
||||
{% endif %}
|
||||
|
||||
qualify ROW_NUMBER() over (
|
||||
PARTITION BY token_symbol
|
||||
ORDER BY
|
||||
CASE
|
||||
WHEN coingecko_id IS NOT NULL THEN 1
|
||||
ELSE 0
|
||||
END,
|
||||
_inserted_timestamp DESC
|
||||
) = 1
|
||||
@ -0,0 +1,44 @@
|
||||
{{ config(
|
||||
materialized = 'incremental',
|
||||
unique_key = ['date_day', 'source_chain', 'destination_chain'],
|
||||
incremental_strategy = 'delete+insert',
|
||||
tags = ['daily']
|
||||
) }}
|
||||
|
||||
WITH parsed_transfer_stats AS (
|
||||
SELECT
|
||||
date_day,
|
||||
b.value :source_chain :: STRING AS source_chain,
|
||||
b.value :destination_chain :: STRING AS destination_chain,
|
||||
b.value :num_txs :: INT AS num_txs,
|
||||
b.value :volume :: DECIMAL(18, 2) AS volume_usd,
|
||||
_inserted_timestamp
|
||||
FROM
|
||||
{{ ref('bronze__axelscan_transfer_stats_by_chains') }},
|
||||
LATERAL FLATTEN(resp :data :data) b
|
||||
|
||||
{% if is_incremental() %}
|
||||
WHERE
|
||||
date_day >= (
|
||||
SELECT
|
||||
COALESCE(MAX(date_day), '1970-01-01' :: DATE) - 3
|
||||
FROM
|
||||
{{ this }}
|
||||
)
|
||||
{% endif %}
|
||||
)
|
||||
SELECT
|
||||
date_day,
|
||||
source_chain,
|
||||
destination_chain,
|
||||
num_txs,
|
||||
volume_usd,
|
||||
_inserted_timestamp,
|
||||
SYSDATE() AS modified_timestamp
|
||||
FROM
|
||||
parsed_transfer_stats
|
||||
QUALIFY
|
||||
ROW_NUMBER() OVER (
|
||||
PARTITION BY date_day, source_chain, destination_chain
|
||||
ORDER BY _inserted_timestamp DESC
|
||||
) = 1
|
||||
@ -31,6 +31,12 @@ sources:
|
||||
- name: dim_labels
|
||||
- name: dim_dates
|
||||
- name: dim_date_hours
|
||||
- name: crosschain_price
|
||||
database: "{{ 'crosschain' if target.database == 'AXELAR' else 'crosschain_dev' }}"
|
||||
schema: price
|
||||
tables:
|
||||
- name: fact_prices_ohlc_hourly
|
||||
- name: ez_prices_hourly
|
||||
- name: crosschain_silver
|
||||
database: "{{ 'crosschain' if target.database == 'AXELAR' else 'crosschain_dev' }}"
|
||||
schema: silver
|
||||
@ -152,4 +158,10 @@ sources:
|
||||
database: axelar
|
||||
schema: bronze_api
|
||||
tables:
|
||||
- name: axelscan_searchgmp
|
||||
- name: axelscan_searchgmp
|
||||
- name: axelar_silver
|
||||
database: axelar
|
||||
schema: silver
|
||||
tables:
|
||||
- name: silver__axelscan_token_meta
|
||||
description: Token metadata with coingecko_id mappings from Axelarscan API
|
||||
Loading…
Reference in New Issue
Block a user