upd rules for docs review

This commit is contained in:
Jack Forgash 2025-07-15 12:46:41 -06:00
parent 60ee794e1b
commit 1a087255ee
2 changed files with 12 additions and 64 deletions

View File

@ -42,7 +42,7 @@ The quicklinks section must be organized by schema and use the relative link to
### Schema Organization Rules:
1. **Group by Schema**: Organize models by their schema (core, defi, nft, price, social, governance, etc.)
2. **Use Exact Schema Names**: Use the exact schema names as they appear in the database (e.g., `NEAR.CORE`, `NEAR.DEFI`, `NEAR.NFT`)
2. **Use Exact Schema Names**: Use the exact schema names as they appear in the database (e.g., `<blockchain_database>.CORE`, `<blockchain_database>.DEFI`, `<blockchain_database>.NFT`)
3. **Model Type Subgrouping**: Within each schema, subgroup by model type:
- **Dimension Tables:** (dim_* models)
- **Fact Tables:** (fact_* models)
@ -53,15 +53,10 @@ The quicklinks section must be organized by schema and use the relative link to
### Implementation Guidelines for Coding Agents:
1. **Scan Directory Structure**: Read `models/gold/` directory to identify all schema subdirectories
2. **Extract Model Names**: For each schema directory, list all `.sql` files and remove the `.sql` extension
3. **Determine Schema Mapping**: Map directory names to database schema names:
- `gold/core/` → `NEAR.CORE`
- `gold/defi/` → `NEAR.DEFI`
- `gold/nft/` → `NEAR.NFT`
- `gold/price/` → `NEAR.PRICE`
- `gold/social/` → `NEAR.SOCIAL`
- `gold/governance/` → `NEAR.GOV`
- `gold/stats/` → `NEAR.STATS`
- `gold/atlas/` → `NEAR.ATLAS`
3. **Determine Schema Mapping**: Map model names to database schema names:
dbt models in this project utilize a double underscore in the model name to denote schema vs table <schema>__<table_name>.sql:
- `core__fact_blocks` → `<blockchain_database>.CORE.FACT_BLOCKS`
- `defi__ez_dex_swaps` → `<blockchain_database>.DEFI.EZ_DEX_SWAPS`
4. **Categorize Models**: Group models by prefix:
- `dim_*` → Dimension Tables
- `fact_*` → Fact Tables
@ -164,22 +159,10 @@ Every `__overview__.md` file MUST include structured `<llm>` XML tags for easy i
</expert>
</llm>
```
Place these XML tags at the end of the documentation (BUT STILL BEFORE THE JINJA ENDDOCS TAG),
Place these XML tags at the end of the documentation (BUT STILL BEFORE THE JINJA ENDDOCS TAG).
## Automatic __overview__.md Updates
The `__overview__.md` file MUST be automatically updated whenever any gold model is created, modified, or deleted to maintain documentation consistency.
### Trigger Conditions:
The following actions MUST trigger an automatic update to `__overview__.md`:
1. **New Gold Model Created**: When a new `.sql` file is added to any `models/gold/*/` directory
2. **Gold Model Modified**: When any existing `.sql` file in `models/gold/*/` directories is changed
3. **Gold Model Deleted**: When any `.sql` file is removed from `models/gold/*/` directories
4. **Gold Model Renamed**: When any `.sql` file in `models/gold/*/` directories is renamed
5. **Schema Directory Changes**: When new schema directories are added to or removed from `models/gold/`
### Update Process for Coding Agents:
When any of the above trigger conditions occur, the coding agent MUST:
## Update Process for Coding Agents:
To update the overview, the coding agent MUST:
1. **Scan Current Gold Models**:
- Read the entire `models/gold/` directory structure
@ -187,53 +170,17 @@ When any of the above trigger conditions occur, the coding agent MUST:
- Extract model names (remove `.sql` extension)
2. **Generate Updated Quicklinks Section**:
- Follow the Implementation Guidelines for Coding Agents (section above)
- Follow these implementation guidelines
- Create a complete new quicklinks section with all current gold models
- Maintain proper schema organization and model type grouping
3. **Update __overview__.md**:
- Preserve the XML tag section at the top
- Replace the entire "Quick Links to Table Documentation" section with the newly generated content
- Maintain all other existing content in the file
- Ensure proper markdown formatting and link structure
- Create or update the XML tag block
4. **Validation Check**:
- Verify all gold models have corresponding links
- Confirm links use correct dbt docs format
- Check that schema names and model names are accurate
- Ensure alphabetical ordering within categories
### Implementation Commands:
When a gold model change is detected, the coding agent should execute these steps:
```bash
# 1. Scan for all gold models
find models/gold/ -name "*.sql" -type f
# 2. Generate updated quicklinks content
# (Follow the Implementation Guidelines for Coding Agents)
# 3. Update __overview__.md file
# (Replace the Quick Links section while preserving other content)
```
### File Preservation Rules:
When updating `__overview__.md`, the coding agent MUST:
- **Preserve XML Tags**: Keep the `<llm>` XML section exactly as is
- **Preserve Introduction**: Maintain the project description and usage instructions
- **Preserve Other Sections**: Keep all sections after the Quick Links (Data Model Overview, Using dbt docs, etc.)
- **Update Only Quick Links**: Replace only the "Quick Links to Table Documentation" section
- **Maintain Formatting**: Preserve markdown formatting and structure
### Error Handling:
If the update process encounters issues:
1. **Backup Original**: Create a backup of the original `__overview__.md` before making changes
2. **Partial Updates**: If full regeneration fails, attempt to add/remove only the specific model that changed
3. **Validation**: After any update, verify the file is still valid markdown and all links are functional
4. **Rollback**: If validation fails, restore from backup and report the issue
### Integration with dbt Workflow:
This automatic update process should integrate with standard dbt workflows:
- **Pre-commit**: Update `__overview__.md` before committing gold model changes
- **Post-run**: Verify `__overview__.md` is current after dbt model runs
- **CI/CD**: Include `__overview__.md` validation in continuous integration checks

View File

@ -50,7 +50,8 @@ This document outlines the comprehensive process for reviewing and improving col
**Lineage Analysis**
- Map the complete data lineage from source to gold model
- Identify key transformations and their purposes
- Understand relationships between related models
- Understand relationships between related models for the sole purpose of generating a robust description
- Do not include data lineage analysis in the table description
### Step 2: Column Description Review
**Individual Column Analysis**