flip/livequery-models
1
0
Fork 0
mirror of https://github.com/FlipsideCrypto/livequery-models.git synced 2026-02-06 10:56:46 +00:00
Code Issues Actions 1 Packages Projects Releases Wiki Activity

update slack

Browse Source
This commit is contained in:
Jensen Yap 2025-08-05 23:38:14 +09:00
parent 4f16bc3b51
commit 811ab78bfa
2 changed files with 158 additions and 378 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

530
macros/marketplace/slack/README.md
View File

@ -1,6 +1,6 @@
# Slack Integration for Livequery
A straightforward Slack integration that lets you send exactly what you want to Slack. You construct the payload according to Slack's API spec, and Livequery delivers it.
A straightforward Slack integration that uses secure vault-stored credentials. You construct the payload according to Slack's API spec, and Livequery delivers it using credentials stored in the vault.
## Prerequisites & Setup
@ -15,7 +15,8 @@ A straightforward Slack integration that lets you send exactly what you want to
4. Click "Add New Webhook to Workspace"
5. Select the channel and click "Allow"
6. Copy the webhook URL (starts with `https://hooks.slack.com/services/...`)
7. Use `slack.webhook_send(url, payload)`
7. Store the webhook URL in the vault under a secret name (e.g., 'alerts', 'notifications')
8. Use `slack_utils.post_webhook(secret_name, payload)`
**Limitations:**
- ❌ No threading support (cannot use `slack.post_reply()`)
@ -37,14 +38,15 @@ A straightforward Slack integration that lets you send exactly what you want to
5. Click "Install to Workspace" at the top
6. Click "Allow" to grant permissions
7. Copy the "Bot User OAuth Token" (starts with `xoxb-...`)
8. **Important:** Invite the bot to your channel:
8. Store the bot token in the vault (Livequery handles this automatically)
9. **Important:** Invite the bot to your channel:
- Go to your Slack channel
- Type `/invite @YourBotName` (replace with your bot's name)
- Or go to channel settings → Integrations → Add apps → Select your bot
9. Get the channel ID:
- Right-click your channel name → "Copy Link"
- Extract the ID from URL: `https://yourworkspace.slack.com/archives/C087GJQ1ZHQ` → `C087GJQ1ZHQ`
10. Use `slack.post_message(token, channel, payload)` and `slack.post_reply()` for threading
10. Get the channel ID:
- Right-click your channel name → "Copy Link"
- Extract the ID from URL: `https://yourworkspace.slack.com/archives/C087GJQ1ZHQ` → `C087GJQ1ZHQ`
11. Use `slack.post_message(channel, payload)` and `slack.post_reply()` for threading
**Features:**
- ✅ Threading support with `slack.post_reply()`
@ -54,223 +56,11 @@ A straightforward Slack integration that lets you send exactly what you want to
## Quick Start
### 1. Add to dbt_project.yml (Recommended)
The easiest way to get Slack notifications for your entire dbt project:
```yaml
# dbt_project.yml
on-run-end:
- "{{ slack_notify_on_run_end(results) }}"
```
Then configure individual models with Slack settings (see Per-Model Configuration below).
**How it works:**
- ✅ **Per-model notifications** - Each model controls its own Slack settings
- ✅ **Custom message formats** - Models can define completely custom Slack payloads
- ✅ **Flexible triggers** - Different models can notify on success, error, or both
- ✅ **Variable substitution** - Use `{model_name}`, `{status}`, `{execution_time}` in custom messages
- ✅ **Environment overrides** - Models can override global Slack webhook/channel settings
- ✅ **Default fallback** - Models without config are ignored (no spam)
### 2. Per-Model Configuration
Configure Slack notifications individually for each model by adding `slack_config` to the model's `meta` section:
#### Basic Model Configuration
### Basic Webhook Message
```sql
-- models/critical/dim_customers.sql
{{ config(
meta={
'slack_config': {
'enabled': true,
'notification_mode': 'error_only', # success_only, error_only, both
'mention': '@here' # Optional: notify team members
}
}
) }}
SELECT * FROM {{ ref('raw_customers') }}
```
#### Custom Message Format
```sql
-- models/critical/fact_revenue.sql
{{ config(
meta={
'slack_config': {
'enabled': true,
'notification_mode': 'both',
'channel': 'C1234567890', # Override default channel
'custom_message': {
'text': '💰 Revenue model {model_name} {status_emoji}',
'username': 'Revenue Bot',
'icon_emoji': ':money_with_wings:',
'attachments': [
{
'color': 'good' if '{status}' == 'success' else 'danger',
'title': 'Critical Revenue Model Alert',
'fields': [
{'title': 'Model', 'value': '{model_name}', 'short': true},
{'title': 'Status', 'value': '{status_emoji} {status}', 'short': true},
{'title': 'Environment', 'value': '{environment}', 'short': true},
{'title': 'Duration', 'value': '{execution_time}s', 'short': true}
],
'footer': 'Revenue Team • {repository}'
}
]
}
}
}
) }}
SELECT * FROM {{ ref('raw_transactions') }}
```
#### Different Slack Channels per Model
```sql
-- models/marketing/marketing_metrics.sql
{{ config(
meta={
'slack_config': {
'enabled': true,
'channel': '#marketing-alerts', # Marketing team channel
'webhook_url': 'https://hooks.slack.com/services/MARKETING/WEBHOOK/URL',
'notification_mode': 'error_only',
'mention': '<@U1234567890>' # Mention specific user by ID
}
}
) }}
```
#### Mention Options
You can notify specific people or groups using the `mention` parameter:
```sql
-- Different mention formats
{{ config(
meta={
'slack_config': {
'enabled': true,
'mention': '@here' # Notify all active members
}
}
) }}
{{ config(
meta={
'slack_config': {
'enabled': true,
'mention': '@channel' # Notify all channel members
}
}
) }}
{{ config(
meta={
'slack_config': {
'enabled': true,
'mention': '<@U1234567890>' # Mention specific user by ID
}
}
) }}
{{ config(
meta={
'slack_config': {
'enabled': true,
'mention': '@username' # Mention by username (if supported)
}
}
) }}
```
#### Available Variables for Custom Messages
Use these variables in your `custom_message` templates:
| Variable | Description | Example |
|----------|-------------|---------|
| `{model_name}` | Model name | `dim_customers` |
| `{status}` | Model status | `success`, `error` |
| `{status_emoji}` | Status emoji | `✅`, `❌` |
| `{environment}` | dbt target | `prod`, `dev` |
| `{repository}` | GitHub repository | `FlipsideCrypto/analytics` |
| `{execution_time}` | Execution seconds | `12.5` |
### 3. Example Notifications
With per-model configuration, each model sends its own notification using Slack's modern Block Kit layout with colored sidebars. Here are some examples:
**Default Model Notification (Success with Mention):**
```
🟢 ┌─────────────────────────────────────┐
│ Hi @here, ✅ Model: dim_customers │
├─────────────────────────────────────┤
│ Success execution completed │
│ │
│ Environment: Execution Time: │
│ prod 12.5s │
│ │
│ Repository: │
│ FlipsideCrypto/analytics │
│ │
│ dbt via Livequery │
└─────────────────────────────────────┘
```
**Custom Revenue Model Notification (Error):**
```
🔴 ┌─────────────────────────────────────┐
│ 💰 Revenue model fact_revenue ❌ │
├─────────────────────────────────────┤
│ Critical Revenue Model Alert │
│ Model: fact_revenue │
│ Status: ❌ Error │
│ Environment: prod │
│ Duration: 45.2s │
│ │
│ Error Message: │
│ Division by zero in line 23... │
│ │
│ Revenue Team • FlipsideCrypto/analytics │
└─────────────────────────────────────┘
```
**Marketing Model Notification (Success with User Mention):**
```
🟢 ┌─────────────────────────────────────┐
│ Hi <@U1234567890>, ✅ Model: marketing_metrics │
├─────────────────────────────────────┤
│ Success execution completed │
│ │
│ Environment: Execution Time: │
│ prod 8.1s │
│ │
│ Repository: │
│ FlipsideCrypto/analytics │
│ │
│ dbt via Livequery │
└─────────────────────────────────────┘
```
*Note: The colored circles (🟢🔴) represent Slack's colored sidebar. The actual messages will display as rich Block Kit layouts with colored left borders in Slack.*
## Advanced Usage
### Manual Function Calls
For custom use cases, call functions directly:
#### Basic Webhook Message
```sql
SELECT slack.webhook_send(
'https://hooks.slack.com/services/YOUR/WEBHOOK/URL',
-- Send a simple message via webhook
SELECT slack_utils.post_webhook(
'alerts', -- Secret name in vault
{
'text': 'Hello from Livequery!',
'username': 'Data Bot'
@ -278,11 +68,11 @@ SELECT slack.webhook_send(
);
```
#### Rich Web API Message with Blocks
### Web API Message
```sql
-- Send message to a channel
SELECT slack.post_message(
'xoxb-your-bot-token',
'C087GJQ1ZHQ',
'C087GJQ1ZHQ', -- Channel ID
{
'text': 'Pipeline completed!',
'blocks': [
@ -292,32 +82,23 @@ SELECT slack.post_message(
'type': 'plain_text',
'text': ':white_check_mark: Pipeline Success'
}
},
{
'type': 'section',
'fields': [
{'type': 'mrkdwn', 'text': '*Repository:*\nFlipsideCrypto/my-repo'},
{'type': 'mrkdwn', 'text': '*Duration:*\n15m 30s'}
]
}
]
}
);
```
#### Threading Example (Web API Only)
### Threading Example
```sql
-- First send main message
WITH main_message AS (
SELECT slack.post_message(
'xoxb-your-bot-token',
'C087GJQ1ZHQ',
{'text': 'Pipeline failed with 3 errors. Details in thread...'}
) as response
)
-- Then send threaded replies
-- Then send threaded reply
SELECT slack.post_reply(
'xoxb-your-bot-token',
'C087GJQ1ZHQ',
main_message.response:data:ts::STRING, -- Use timestamp from main message
{'text': 'Error 1: Database connection timeout'}
@ -325,190 +106,189 @@ SELECT slack.post_reply(
FROM main_message;
```
### Conditional Notifications
Add conditions to control when notifications are sent:
```yaml
# dbt_project.yml
on-run-end:
# Only send notifications in production
- "{% if target.name == 'prod' %}{{ slack_notify_on_run_end(results) }}{% endif %}"
# Or use environment variable control
- "{% if env_var('SEND_SLACK_NOTIFICATIONS', 'false') == 'true' %}{{ slack_notify_on_run_end(results) }}{% endif %}"
```
### Advanced: Custom Message Format
For full control over the message format, use the lower-level functions:
```yaml
on-run-end: |
{% if execute %}
{% set status = 'success' if results|selectattr('status', 'equalto', 'error')|list|length == 0 else 'failed' %}
SELECT slack.webhook_send(
'{{ env_var("SLACK_WEBHOOK_URL") }}',
{
'text': 'dbt run {{ status }}',
'attachments': [
{
'color': '{{ "#36a64f" if status == "success" else "#ff0000" }}',
'title': 'dbt {{ status|title }}',
'fields': [
{'title': 'Models', 'value': '{{ results|length }}', 'short': true},
{'title': 'Failed', 'value': '{{ results|selectattr("status", "equalto", "error")|list|length }}', 'short': true}
]
}
]
}
);
{% endif %}
```
## Configuration Reference
### Global Environment Variables (Optional)
Models can override these global settings. Only set these if you want fallback defaults:
```bash
# Default Slack connection (models can override)
export SLACK_WEBHOOK_URL="https://hooks.slack.com/services/YOUR/WEBHOOK/URL"
# OR
export SLACK_BOT_TOKEN="xoxb-your-bot-token"
export SLACK_CHANNEL="C087GJQ1ZHQ"
# Optional global settings
export SLACK_BOT_USERNAME="dbt Bot"
export GITHUB_REPOSITORY="your-org/your-repo"
```
### Notification Modes
- **`error_only`** (default) - Only notify on failures
- **`success_only`** - Only notify on successful runs
- **`both`** - Notify on both success and failure
### Mention Options
The `mention` parameter allows you to notify specific users or groups:
- **`@here`** - Notify all active members in the channel
- **`@channel`** - Notify all members in the channel (use sparingly)
- **`<@U1234567890>`** - Mention specific user by Slack user ID (recommended)
- **`<@U1234567890|username>`** - Mention user with display name
- **`@username`** - Mention by username (may not work in all workspaces)
**Note:** To find a user's Slack ID, right-click their profile → "Copy member ID"
## Functions Reference
### `slack.webhook_send(webhook_url, payload)`
Send messages via Slack Incoming Webhooks.
### `slack_utils.post_webhook(secret_name, payload)`
Send messages via Slack Incoming Webhooks using vault-stored webhook URL.
**Parameters:**
- `webhook_url` - Your Slack webhook URL
- `secret_name` - Name of webhook secret stored in vault (e.g., 'alerts', 'notifications')
- `payload` - JSON object following [Slack webhook format](https://api.slack.com/messaging/webhooks)
### `slack.post_message(bot_token, channel, payload)`
Send messages via Slack Web API (chat.postMessage).
**Example:**
```sql
SELECT slack_utils.post_webhook(
'notifications',
{
'text': 'dbt run completed successfully',
'username': 'dbt Bot',
'icon_emoji': ':white_check_mark:'
}
);
```
### `slack.post_message(channel, payload)`
Send messages via Slack Web API (chat.postMessage) using vault-stored bot token.
**Parameters:**
- `bot_token` - Your Slack bot token (xoxb-...)
- `channel` - Channel ID (C...) or name (#channel)
- `payload` - JSON object following [Slack chat.postMessage format](https://api.slack.com/methods/chat.postMessage)
### `slack.post_reply(bot_token, channel, thread_ts, payload)`
Send threaded replies via Slack Web API.
**Example:**
```sql
SELECT slack.post_message(
'C087GJQ1ZHQ',
{
'text': 'Model update complete',
'attachments': [
{
'color': 'good',
'title': 'Success',
'fields': [
{'title': 'Models', 'value': '15', 'short': true},
{'title': 'Duration', 'value': '2m 30s', 'short': true}
]
}
]
}
);
```
### `slack.post_reply(channel, thread_ts, payload)`
Send threaded replies via Slack Web API using vault-stored bot token.
**Parameters:**
- `bot_token` - Your Slack bot token
- `channel` - Channel ID or name
- `thread_ts` - Parent message timestamp for threading
- `payload` - JSON object following Slack chat.postMessage format
**Example:**
```sql
SELECT slack.post_reply(
'C087GJQ1ZHQ',
'1698765432.123456', -- Parent message timestamp
{'text': 'Additional details in this thread'}
);
```
### `slack.webhook_send(secret_name, payload)`
Alias for `slack_utils.post_webhook()` - sends webhook messages using vault-stored URL.
**Parameters:**
- `secret_name` - Name of webhook secret stored in vault
- `payload` - JSON object following Slack webhook format
### Validation Functions
- `slack_utils.validate_webhook_url(url)` - Check if webhook URL is valid
- `slack_utils.validate_bot_token(token)` - Check if bot token is valid
- `slack_utils.validate_webhook_url(url)` - Check if webhook URL format is valid
- `slack_utils.validate_bot_token(token)` - Check if bot token format is valid
- `slack_utils.validate_channel(channel)` - Check if channel format is valid
## Vault Configuration
### Webhook Secrets
Store your webhook URLs in the vault with meaningful names:
- `alerts` - For critical alerts
- `notifications` - For general notifications
- `marketing` - For marketing team updates
### Bot Token
The bot token is automatically managed by Livequery and stored securely in the vault. You don't need to provide it in function calls.
## Testing Without Spamming Slack
### Built-in Tests
The integration includes comprehensive tests that use mock endpoints instead of real Slack channels:
- **httpbin.org** - Tests HTTP mechanics and payload formatting
- **Validation functions** - Test URL/token/channel format validation
- **Error scenarios** - Test authentication failures and invalid endpoints
The integration includes comprehensive tests that verify functionality without hitting real Slack channels.
### Manual Testing Options
#### 1. Test with httpbin.org (Recommended for Development)
```sql
-- Test webhook functionality without hitting Slack
SELECT slack.webhook_send(
'https://httpbin.org/post',
-- (Note: This bypasses vault and uses direct URL for testing)
SELECT slack_utils.post_webhook(
'test-httpbin', -- Create test secret pointing to httpbin.org
{'text': 'Test message', 'username': 'Test Bot'}
);
-- Verify the request was formatted correctly
-- httpbin.org returns the request data in the response
```
#### 2. Test with webhook.site (Inspect Real Payloads)
```sql
-- Create a unique URL at https://webhook.site/ and use it
SELECT slack.webhook_send(
'https://webhook.site/your-unique-id',
{'text': 'Test message with full Slack formatting'}
);
-- View the captured request at webhook.site to see exactly what Slack would receive
```
#### 3. Test Workspace (Real Slack Testing)
#### 2. Test Workspace (Real Slack Testing)
Create a dedicated test workspace or use a private test channel:
- Store test webhook URLs in vault with names like `test-alerts`
- Use test channel IDs for `post_message()` calls
- Set up separate vault secrets for testing vs production
## Advanced Usage
### Rich Message Formatting
```sql
-- Use environment variables to switch between test and prod
SELECT slack.webhook_send(
'{{ env_var("SLACK_TEST_WEBHOOK_URL") }}', -- Test webhook
{'text': 'Safe test in dedicated channel'}
-- Advanced Block Kit message
SELECT slack.post_message(
'C087GJQ1ZHQ',
{
'text': 'Data Pipeline Report',
'blocks': [
{
'type': 'header',
'text': {
'type': 'plain_text',
'text': '📊 Daily Data Pipeline Report'
}
},
{
'type': 'section',
'fields': [
{'type': 'mrkdwn', 'text': '*Environment:*\nProduction'},
{'type': 'mrkdwn', 'text': '*Models Run:*\n25'},
{'type': 'mrkdwn', 'text': '*Duration:*\n12m 34s'},
{'type': 'mrkdwn', 'text': '*Status:*\n✅ Success'}
]
},
{
'type': 'actions',
'elements': [
{
'type': 'button',
'text': {'type': 'plain_text', 'text': 'View Logs'},
'url': 'https://your-logs-url.com'
}
]
}
]
}
);
```
#### 4. Conditional Testing
```yaml
# dbt_project.yml - Only send notifications in specific environments
on-run-end:
- "{% if target.name == 'prod' %}{{ slack_notify_on_run_end(results) }}{% endif %}"
- "{% if env_var('SLACK_TESTING_MODE', 'false') == 'true' %}{{ slack_notify_on_run_end(results) }}{% endif %}"
```
### Environment Variables for Testing
```bash
# Production Slack
export SLACK_WEBHOOK_URL="https://hooks.slack.com/services/YOUR/PROD/WEBHOOK"
# Testing alternatives
export SLACK_TEST_WEBHOOK_URL="https://webhook.site/your-unique-id"
export SLACK_HTTPBIN_TEST_URL="https://httpbin.org/post"
export SLACK_TESTING_MODE="true"
### Error Handling
```sql
-- Check response for errors
WITH slack_result AS (
SELECT slack_utils.post_webhook(
'alerts',
{'text': 'Test message'}
) as response
)
SELECT
response:ok::BOOLEAN as success,
response:error::STRING as error_message,
CASE
WHEN response:ok::BOOLEAN THEN 'Message sent successfully'
ELSE 'Failed: ' || response:error::STRING
END as status
FROM slack_result;
```
## How It Works
1. **You construct the payload** - Use Slack's official API documentation to build your JSON
2. **Livequery delivers it** - We handle the HTTP request to Slack
3. **Get the response** - Standard Slack API response with success/error info
1. **Secure credential storage** - Webhook URLs and bot tokens are stored in Livequery's vault
2. **You construct the payload** - Use Slack's official API documentation to build your JSON
3. **Livequery delivers it** - We handle authentication and HTTP requests to Slack
4. **Get the response** - Standard Slack API response with success/error info
## Slack API Documentation
- [Webhook Format](https://api.slack.com/messaging/webhooks) - For webhook_send()
- [chat.postMessage](https://api.slack.com/methods/chat.postMessage) - For post_message()
- [Webhook Format](https://api.slack.com/messaging/webhooks) - For webhook messages
- [chat.postMessage](https://api.slack.com/methods/chat.postMessage) - For Web API messages
- [Block Kit](https://api.slack.com/block-kit) - For rich interactive messages
- [Message Formatting](https://api.slack.com/reference/surfaces/formatting) - Text formatting guide
That's it! No complex configurations, no templates to learn. Just Slack's API delivered through Livequery.
That's it! Secure, simple Slack integration with vault-managed credentials.

6
macros/marketplace/slack/messaging_udfs.yaml.sql
View File

@ -13,7 +13,7 @@
options: |
COMMENT = 'Send a message to Slack via webhook [API docs: Webhooks](https://api.slack.com/messaging/webhooks)'
sql: |
SELECT {{ utils_schema_name }}.post_webhook(
SELECT slack_utils.post_webhook(
WEBHOOK_SECRET_NAME,
PAYLOAD
) as response
@ -28,7 +28,7 @@
options: |
COMMENT = 'Send a message to Slack via Web API [API docs: chat.postMessage](https://api.slack.com/methods/chat.postMessage)'
sql: |
SELECT {{ utils_schema_name }}.post_message(
SELECT slack_utils.post_message(
CHANNEL,
PAYLOAD
) as response
@ -44,7 +44,7 @@
options: |
COMMENT = 'Send a threaded reply to Slack via Web API [API docs: chat.postMessage](https://api.slack.com/methods/chat.postMessage)'
sql: |
SELECT {{ utils_schema_name }}.post_reply(
SELECT slack_utils.post_reply(
CHANNEL,
THREAD_TS,
PAYLOAD