added readmes

This commit is contained in:
artin rebekale 2024-06-17 12:51:10 -07:00
parent da183e8167
commit 27457a96bd
4 changed files with 28 additions and 3 deletions

View File

@ -1,6 +1,7 @@
### OpenBlocks Takehome ###
## Summary ##
This is a quickly constructed setup for a mature method to consume data from various sources and present the data to any team within the org.
There is automated consumers of data, an observability and discoverability stack, transformation via the industry standard dbt, and a modular enough design to be scalable.

View File

@ -6,9 +6,14 @@ In this case, airbyte was able to quickly trigger the execute command on Dune, g
It then pushes the data as raw into the database, keeping track of any errors and avoiding duplicates.
## Demo ##
the demo is at [Airbye](https://airbyte.nuvari.com)
username: airbyte
password: password
(i kept the defaults
(i kept the defaults)
* note the airbyte instance and the open-metadata instance must be seperate machines/vms/lxcs with the supplied docker-compose files)

View File

@ -3,17 +3,25 @@
## Quickstart: ##
# Install #
`python3 -m pip install dbt dbt-postgres`
# Running #
To run all transforms
`dbt run`
to run all of Dune staging:
`dbt build --select staging.dune`
to run bitcoin json staging:
`dbt build --select staging.bitcoin_json`
## Models ##
Models are in the `models` directory. They are organized `models -> phase -> source`. Currently the only phase is staging, the onlys sources Dune or json data
Models are in the `models` directory. They are organized `models -> phase -> source`.
Currently the only phase is staging, the onlys sources Dune or json data

View File

@ -1,19 +1,30 @@
### Open Metadata & Airflow ###
## Open Metadata ##
# Demo #
[Open Metadata](https://open-metadata.nuvari.com)
Username: admin
password: admin
# Usage #
Open-Metadata has a complete view of all the available data. It allows for the teams to communicate about the tables/views, share queries, monitor data freshness, quickly grasp the data lineaage (what tables were merged to finally become the desiered dataset), as well as allow the data team to monitor which datasets are in active use and by whom.
## Airflow ##
# Demo #
[Airflow](https://airflow.nuvari.com)
Username: admin
password: admin
# Usage #
Airflow on the basic level is an advanced task/job orchestrator, able to trigger airbyte jobs, dbt jobs, perform ETL/ELT and more
Airflow on the basic level is an advanced task/job orchestrator, able to trigger airbyte jobs, dbt jobs, perform ETL/ELT and more