AN-5747 stellar gold (#3)

* first commit, added all docs but no transformations yet

* added all gold models, plus enhanced (EZ) operations and docs

* Silver incremental logic (#2)

* check in

* check in, gte in progress

* bronze & silver

* updated docs and gold models

* updated working gold models

* adding ez__operations as well

* SO, cluster, fix silver accounts

* updated ez config

* updated ez config formatting (minor)

* changed filenames and dir structure

* fixed and ran ez_operations, updated a line in the readme

---------

Co-authored-by: Mike Stepanovic <mike.stepanovic@flipsidecrypto.com>
This commit is contained in:
eric-laurello 2025-02-11 12:50:57 -05:00 committed by GitHub
parent 549a31b281
commit d561255434
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
225 changed files with 2938 additions and 11 deletions

View File

@ -23,14 +23,13 @@ stellar:
```
### Resources:
- Learn more about Stellar's [Hubble dataset](https://developers.stellar.org/docs/data/hubble)
- Learn more about dbt [in the docs](https://docs.getdbt.com/docs/introduction)
- Check out [Discourse](https://discourse.getdbt.com/) for commonly asked questions and answers
- Join the [chat](https://community.getdbt.com/) on Slack for live discussions and support
- Find [dbt events](https://events.getdbt.com) near you
- Check out [the blog](https://blog.getdbt.com/) for the latest news on dbt's development and best practices
- Check out [the blog](https://blog.getdbt.com/) for the latest news on dbt's development and best practices
## Applying Model Tags
### Database / Schema level tags
@ -63,12 +62,12 @@ To add/update a model's snowflake tags, add/modify the `meta` model property und
By default, model tags are not pushed to snowflake on each load. You can push a tag update for a model by specifying the `UPDATE_SNOWFLAKE_TAGS` project variable during a run.
```
dbt run --var '{"UPDATE_SNOWFLAKE_TAGS":True}' -s models/core/core__fact_swaps.sql
dbt run --var '{"UPDATE_SNOWFLAKE_TAGS":True}' -s models/core/core__fact_trades.sql
```
### Querying for existing tags on a model in snowflake
```
select *
from table(stellar.information_schema.tag_references('stellar.core.fact_blocks', 'table'));
from table(stellar.information_schema.tag_references('stellar.core.fact_ledgers', 'table'));
```

View File

@ -0,0 +1,75 @@
{% docs __overview__ %}
# Welcome to the Flipside Crypto's Stellar Models Documentation
## **What does this documentation cover?**
The documentation included here details the design of Stellar tables and views available via [Flipside Crypto.](https://flipsidecrypto.xyz/) For more information on how these models are built, please see [the github repository.](https://github.com/flipsideCrypto/stellar-models/)
## **How do I use these docs?**
The easiest way to navigate this documentation is to use the Quick Links below. These links will take you to the documentation for each table, which contains a description, a list of the columns, and other helpful information.
If you are experienced with dbt docs, feel free to use the sidebar to navigate the documentation, as well as explore the relationships between tables and the logic building them.
There is more information on how to use dbt docs in the last section of this document.
## **Quick Links to Table Documentation**
**Click on the links below to jump to the documentation for each schema.**
### Core Tables (`stellar`.`CORE`.`<table_name>`)
**Dimension Tables:**
- [core.dim_assets](https://flipsidecrypto.github.io/stellar-models/#!/model/model.stellar_models.core__dim_assets)
**Fact Tables:**
- [core.fact_accounts](https://flipsidecrypto.github.io/stellar-models/#!/model/model.stellar_models.core__fact_accounts)
- [core.fact_ledgers](https://flipsidecrypto.github.io/stellar-models/#!/model/model.stellar_models.core__fact_ledgers)
- [core.fact_operations](https://flipsidecrypto.github.io/stellar-models/#!/model/model.stellar_models.core__fact_operations)
- [core.fact_transactions](https://flipsidecrypto.github.io/stellar-models/#!/model/model.stellar_models.core__fact_transactions)
### DeFi Tables (`stellar`.`DEFI`.`<table_name>`)
**Fact Tables:**
- [defi.fact_liquidity_pools](https://flipsidecrypto.github.io/stellar-models/#!/model/model.stellar_models.defi__fact_liquidity_pools)
- [defi.fact_trades](https://flipsidecrypto.github.io/stellar-models/#!/model/model.stellar_models.defi__fact_trades)
## **Data Model Overview**
These Stellar models are built a few different ways, but the core fact tables are built using three layers of sql models: **bronze, silver, and gold (or core).**
- Bronze: Data is loaded in from the source as a view
- Silver: All necessary parsing, filtering, de-duping, and other transformations are done here
- Gold (or core): Final views and tables that are available publicly
The dimension tables are sourced from a variety of on-chain and off-chain sources.
Convenience views (denoted ez_) are a combination of different fact and dimension tables. These views are built to make it easier to query the data.
## **Using dbt docs**
### Navigation
You can use the ```Project``` and ```Database``` navigation tabs on the left side of the window to explore the models in the project.
### Database Tab
This view shows relations (tables and views) grouped into database schemas. Note that ephemeral models are *not* shown in this interface, as they do not exist in the database.
### Graph Exploration
You can click the blue icon on the bottom-right corner of the page to view the lineage graph of your models.
On model pages, you'll see the immediate parents and children of the model you're exploring. By clicking the Expand button at the top-right of this lineage pane, you'll be able to see all of the models that are used to build, or are built from, the model you're exploring.
Once expanded, you'll be able to use the ```--models``` and ```--exclude``` model selection syntax to filter the models in the graph. For more information on model selection, check out the [dbt docs](https://docs.getdbt.com/docs/model-selection-syntax).
Note that you can also right-click on models to interactively filter and explore the graph.
### **More information**
- [Flipside](https://flipsidecrypto.xyz/)
- [Velocity](https://app.flipsidecrypto.com/velocity?nav=Discover)
- [Tutorials](https://docs.flipsidecrypto.com/our-data/tutorials)
- [Github](https://github.com/FlipsideCrypto/stellar-models)
- [What is dbt?](https://docs.getdbt.com/docs/introduction)
{% enddocs %}

View File

@ -0,0 +1,3 @@
{% docs account %}
The account involved in the operation.
{% enddocs %}

View File

@ -0,0 +1,3 @@
{% docs account_id %}
The address of the account. The address is the account's public key encoded in base32. All account addresses start with a 'G'.
{% enddocs %}

View File

@ -0,0 +1,3 @@
{% docs account_muxed %}
Muxed account identifier.
{% enddocs %}

View File

@ -0,0 +1,3 @@
{% docs account_sequence %}
Sequence number for the account.
{% enddocs %}

View File

@ -0,0 +1,3 @@
{% docs address %}
Address for Soroban contract creation.
{% enddocs %}

View File

@ -0,0 +1,3 @@
{% docs amount %}
Float representation of the amount of an asset sent/offered/etc.
{% enddocs %}

View File

@ -0,0 +1,3 @@
{% docs asset_a_amount %}
The raw number of tokens locked in the pool for one of the two asset pairs in the liquidity pool.
{% enddocs %}

View File

@ -0,0 +1,3 @@
{% docs asset_b_amount %}
The raw number of tokens locked in the pool for one of the two asset pairs in the liquidity pool.
{% enddocs %}

View File

@ -0,0 +1,3 @@
{% docs asset_balance_changes %}
Changes in asset balances.
{% enddocs %}

View File

@ -0,0 +1,3 @@
{% docs asset_code %}
The 4 or 12 character code representation of the asset on the network.
{% enddocs %}

View File

@ -0,0 +1,3 @@
{% docs asset_id %}
The Farm Hash encoding of Asset Code + Asset Issuer + Asset Type. This field is optimized for cross table joins since integer joins are less expensive than the original asset id components.
{% enddocs %}

View File

@ -0,0 +1,3 @@
{% docs asset_issuer %}
The account address of the original asset issuer that created the asset.
{% enddocs %}

View File

@ -0,0 +1,3 @@
{% docs asset_type %}
The identifier for type of asset code, can be an alphanumeric with 4 characters, 12 characters or the native asset to the network, XLM.
{% enddocs %}

View File

@ -0,0 +1,3 @@
{% docs authorize %}
Indicates whether the trustline is authorized.
{% enddocs %}

View File

@ -0,0 +1,3 @@
{% docs authorize_to_maintain_liabilities %}
Indicates whether the trustline is authorized for maintaining liabilities.
{% enddocs %}

View File

@ -0,0 +1,3 @@
{% docs balance %}
The number of units of XLM held by the account.
{% enddocs %}

View File

@ -0,0 +1,3 @@
{% docs balance_id %}
The balance id of the claimable balance.
{% enddocs %}

View File

@ -0,0 +1,3 @@
{% docs base_fee %}
The fee (in stroops) the network charges per operation in a transaction for the given ledger. The minimum base fee is 100, with the ability to increase if transaction demand exceeds ledger capacity. When this occurs, the ledger enters surge pricing.
{% enddocs %}

View File

@ -0,0 +1,3 @@
{% docs base_reserve %}
The reserve (in stroops) the network requires an account to retain as a minimum balance in order to be a valid account on the network. The current minimum reserve is 10 XLM.
{% enddocs %}

View File

@ -0,0 +1,3 @@
{% docs batch_id %}
String representation of the run id for a given DAG in Airflow. Takes the form of "scheduled__<batch_end_date>-<dag_alias>". Batch ids are unique to the batch and help with monitoring and rerun capabilities.
{% enddocs %}

View File

@ -0,0 +1,3 @@
{% docs batch_insert_ts %}
The timestamp in UTC when a batch of records was inserted into the database. This field can help identify if a batch executed in real time or as part of a backfill. The timestamp should not be used during ad hoc analysis and is useful for data engineering purposes.
{% enddocs %}

View File

@ -0,0 +1,3 @@
{% docs batch_run_date %}
The start date for the batch interval. When taken with the date in the batch_id, the date represents the interval of ledgers processed. The batch run date can be seen as a proxy of closed_at for a ledger.
{% enddocs %}

View File

@ -0,0 +1,3 @@
{% docs begin_sponsor %}
The account address that initiated the sponsorship.
{% enddocs %}

View File

@ -0,0 +1,3 @@
{% docs bump_to %}
The new desired value of the source account's sequence number.
{% enddocs %}

View File

@ -0,0 +1,3 @@
{% docs buying_account_address %}
The account address of the buying party.
{% enddocs %}

View File

@ -0,0 +1,3 @@
{% docs buying_amount %}
The amount of purchased asset that was moved from the seller account into the buying account, reported in terms of the bought asset.
{% enddocs %}

View File

@ -0,0 +1,3 @@
{% docs buying_asset_code %}
The 4 or 12 character code representation of the asset being bought.
{% enddocs %}

View File

@ -0,0 +1,3 @@
{% docs buying_asset_id %}
The identifier of the asset being bought.
{% enddocs %}

View File

@ -0,0 +1,3 @@
{% docs buying_asset_issuer %}
The account address of the original asset issuer for the buying asset.
{% enddocs %}

View File

@ -0,0 +1,3 @@
{% docs buying_asset_type %}
The identifier for type of buying asset code.
{% enddocs %}

View File

@ -0,0 +1,3 @@
{% docs buying_liabilities %}
The sum of all buy offers owned by this account for XLM only.
{% enddocs %}

View File

@ -0,0 +1,3 @@
{% docs buying_offer_id %}
The offer ID in the orderbook of the buying offer. If this offer was immediately and fully consumed, this will be a synthetic ID.
{% enddocs %}

View File

@ -0,0 +1,3 @@
{% docs clawback_enabled %}
Indicates whether the asset can be clawed back by the issuer.
{% enddocs %}

View File

@ -0,0 +1,3 @@
{% docs clear_flags %}
Array of numeric values of flags cleared for a trustline.
{% enddocs %}

View File

@ -0,0 +1,3 @@
{% docs clear_flags_s %}
Array of string values of flags cleared for a trustline.
{% enddocs %}

View File

@ -0,0 +1,3 @@
{% docs closed_at %}
Timestamp when the ledger was closed and committed to the network. Ledgers are expected to close ~every 5 seconds.
{% enddocs %}

View File

@ -0,0 +1,3 @@
{% docs contract_code_hash %}
Hash of Soroban contract code.
{% enddocs %}

View File

@ -0,0 +1,3 @@
{% docs contract_id %}
Soroban contract identifier.
{% enddocs %}

View File

@ -0,0 +1,3 @@
{% docs core_accounts_id %}
Unique identifier for this record
{% enddocs %}

3
models/descriptions/d.md Normal file
View File

@ -0,0 +1,3 @@
{% docs d %}
Denominator in price ratio calculations.
{% enddocs %}

View File

@ -0,0 +1,3 @@
{% docs deleted %}
Indicates whether the ledger entry (account, claimable balance, trust line, offer, liquidity pool) has been deleted or not. Once an entry is deleted, it cannot be recovered.
{% enddocs %}

View File

@ -0,0 +1,3 @@
{% docs destination_min %}
The minimum amount to be received in destination asset.
{% enddocs %}

View File

@ -0,0 +1,3 @@
{% docs details_json %}
JSON record containing operation-specific details.
{% enddocs %}

View File

@ -0,0 +1,3 @@
{% docs extend_to %}
Ledger extension target.
{% enddocs %}

View File

@ -0,0 +1,3 @@
{% docs extra_signers %}
Additional transaction signers.
{% enddocs %}

View File

@ -0,0 +1,3 @@
{% docs failed_transaction_count %}
The number of failed transactions submitted to the network in this ledger. The transaction was still paid for but contained an error that prevented it from executing
{% enddocs %}

View File

@ -0,0 +1,3 @@
{% docs fee %}
The number of basis points charged as a percentage of the trade in order to complete the transaction. The fees earned on all trades are divided amongst pool shareholders and distributed as an incentive to keep money in the pools.
{% enddocs %}

View File

@ -0,0 +1,3 @@
{% docs fee_account %}
Account charged the fee.
{% enddocs %}

View File

@ -0,0 +1,3 @@
{% docs fee_account_muxed %}
Muxed fee account identifier.
{% enddocs %}

View File

@ -0,0 +1,3 @@
{% docs fee_charged %}
Actual fee charged for the operation.
{% enddocs %}

View File

@ -0,0 +1,3 @@
{% docs fee_pool %}
The sum of all transaction fees
{% enddocs %}

View File

@ -0,0 +1,3 @@
{% docs flags %}
Denotes the enabling and disabling of certain asset issuer privileges.
{% enddocs %}

View File

@ -0,0 +1,3 @@
{% docs from %}
The account address from which the payment/contract originates.
{% enddocs %}

View File

@ -0,0 +1,3 @@
{% docs function %}
Function type for invoke_host_function.
{% enddocs %}

View File

@ -0,0 +1,3 @@
{% docs funder %}
The account address that funds a new account.
{% enddocs %}

View File

@ -0,0 +1,3 @@
{% docs history_operation_id %}
The operation id associated with the executed trade. The total amount traded in an operation can be broken up into multiple smaller trades spread across multiple orders by multiple parties.
{% enddocs %}

View File

@ -0,0 +1,3 @@
{% docs home_domain %}
The domain that hosts this account's stellar.toml file.
{% enddocs %}

View File

@ -0,0 +1,3 @@
{% docs id %}
Unique identifier for the ledger
{% enddocs %}

View File

@ -0,0 +1,3 @@
{% docs inclusion_fee_bid %}
Bid for inclusion fee.
{% enddocs %}

View File

@ -0,0 +1,3 @@
{% docs inclusion_fee_charged %}
Actual inclusion fee charged.
{% enddocs %}

View File

@ -0,0 +1,3 @@
{% docs inflation_destination %}
Deprecated: The account address to receive an inflation payment when they are disbursed on the network.
{% enddocs %}

View File

@ -0,0 +1,3 @@
{% docs inner_transaction_hash %}
A transaction hash of a transaction wrapped with its signatures for fee-bump transactions
{% enddocs %}

View File

@ -0,0 +1,3 @@
{% docs inserted_timestamp %}
Timestamp when this record was inserted.
{% enddocs %}

View File

@ -0,0 +1,3 @@
{% docs into %}
The account address receiving the deleted account's lumens.
{% enddocs %}

View File

@ -0,0 +1,3 @@
{% docs invocation_id %}
Unique identifier of the dbt job that inserted or modified this record.
{% enddocs %}

View File

@ -0,0 +1,3 @@
{% docs last_modified_ledger %}
The ledger sequence number when the ledger entry was last modified. Deletions do not count as a modification and will report the prior modification sequence number.
{% enddocs %}

View File

@ -0,0 +1,3 @@
{% docs ledger_bounds %}
Bounds of the ledger.
{% enddocs %}

View File

@ -0,0 +1,3 @@
{% docs ledger_entry_change %}
Code that describes the ledger entry change type that was applied to the ledger entry.
{% enddocs %}

View File

@ -0,0 +1,3 @@
{% docs ledger_hash %}
The hex-encoded SHA-256 hash that represents the ledger's XDR-encoded form.
{% enddocs %}

View File

@ -0,0 +1,3 @@
{% docs ledger_header %}
A base64-encoded string of the raw LedgerHeader xdr struct for this ledger
{% enddocs %}

View File

@ -0,0 +1,3 @@
{% docs ledger_operation_count %}
Number of operations in the ledger.
{% enddocs %}

View File

@ -0,0 +1,3 @@
{% docs ledger_sequence %}
The sequence number of the ledger.
{% enddocs %}

View File

@ -0,0 +1,3 @@
{% docs limit %}
The upper bound amount of an asset that an account can hold.
{% enddocs %}

View File

@ -0,0 +1,3 @@
{% docs liquidity_pool_fee %}
The percentage fee (in basis points) of the total fee collected by the liquidity pool for executing the trade.
{% enddocs %}

View File

@ -0,0 +1,3 @@
{% docs liquidity_pool_id %}
Unique identifier for a liquidity pool. There cannot be duplicate pools for the same asset pair. Once a pool has been created for the asset pair, another cannot be created.
{% enddocs %}

View File

@ -0,0 +1,3 @@
{% docs liquidity_pools_id %}
Unique identifier for the liquidity pool record.
{% enddocs %}

View File

@ -0,0 +1,3 @@
{% docs master_weight %}
The weight of the master key, which is the private key for this account. If a master key is '0', the account is locked and cannot be used.
{% enddocs %}

View File

@ -0,0 +1,3 @@
{% docs max_fee %}
Maximum fee for the transaction.
{% enddocs %}

View File

@ -0,0 +1,3 @@
{% docs max_price %}
Maximum exchange rate for deposit operation.
{% enddocs %}

View File

@ -0,0 +1,3 @@
{% docs max_price_r %}
A fractional representation of the prices of the two assets in a pool. The n is the numerator (value of asset a) and the d is the denominator (value of asset b).
{% enddocs %}

View File

@ -0,0 +1,3 @@
{% docs max_tx_set_size %}
The maximum number of operations that Stellar validator nodes have agreed to process in a given ledger. Since Protocol 11, ledger capacity has been measured in operations rather than transactions.
{% enddocs %}

View File

@ -0,0 +1,3 @@
{% docs memo %}
Memo content of the transaction.
{% enddocs %}

View File

@ -0,0 +1,3 @@
{% docs memo_type %}
Type of memo attached to transaction.
{% enddocs %}

View File

@ -0,0 +1,3 @@
{% docs min_account_sequence %}
Minimum account sequence number.
{% enddocs %}

View File

@ -0,0 +1,3 @@
{% docs min_account_sequence_age %}
Minimum age of account sequence.
{% enddocs %}

View File

@ -0,0 +1,3 @@
{% docs min_account_sequence_ledger_gap %}
Minimum gap in ledger sequences.
{% enddocs %}

View File

@ -0,0 +1,3 @@
{% docs min_price %}
Minimum exchange rate for deposit operation.
{% enddocs %}

View File

@ -0,0 +1,3 @@
{% docs min_price_r %}
A fractional representation of the prices of the two assets in a pool. The n is the numerator (value of asset a) and the d is the denominator (value of asset b).
{% enddocs %}

View File

@ -0,0 +1,3 @@
{% docs modified_timestamp %}
Timestamp when this record was last modified.
{% enddocs %}

3
models/descriptions/n.md Normal file
View File

@ -0,0 +1,3 @@
{% docs n %}
Numerator in price ratio calculations.
{% enddocs %}

View File

@ -0,0 +1,3 @@
{% docs name %}
The key for a data entry in manage data operations.
{% enddocs %}

View File

@ -0,0 +1,3 @@
{% docs new_max_fee %}
Updated maximum fee.
{% enddocs %}

View File

@ -0,0 +1,3 @@
{% docs node_id %}
The id of winning validator node which is allowed to write transaction set to the network. The winning validator is decided by the network.
{% enddocs %}

View File

@ -0,0 +1,3 @@
{% docs non_refundable_resource_fee_charged %}
The amount of the resource fee charged for the transaction that is non-refundable.
{% enddocs %}

View File

@ -0,0 +1,3 @@
{% docs num_sponsored %}
The number of reserves sponsored for this account (meaning another account is paying for the minimum balance). Sponsored entries do not incur any reserve requirement on the account that owns the entry.
{% enddocs %}

View File

@ -0,0 +1,3 @@
{% docs num_sponsoring %}
The number of reserves sponsored by this account. Entries sponsored by this account incur a reserve requirement.
{% enddocs %}

View File

@ -0,0 +1,3 @@
{% docs num_subentries %}
The total number of ledger entries connected to this account. Ledger entries include: trustlines, offers, signers, and data entries. (Claimable balances are counted under sponsoring entries, not subentries). Any newly created trustline, offer, signer or data entry will increase the number of subentries by 1. Accounts may have up to 1,000 subentries.
{% enddocs %}

View File

@ -0,0 +1,3 @@
{% docs offer_id %}
The unique id for the offer.
{% enddocs %}

Some files were not shown because too many files have changed in this diff Show More