added readmes
This commit is contained in:
parent
da183e8167
commit
27457a96bd
@ -1,6 +1,7 @@
|
||||
### OpenBlocks Takehome ###
|
||||
|
||||
## Summary ##
|
||||
|
||||
This is a quickly constructed setup for a mature method to consume data from various sources and present the data to any team within the org.
|
||||
There is automated consumers of data, an observability and discoverability stack, transformation via the industry standard dbt, and a modular enough design to be scalable.
|
||||
|
||||
|
||||
@ -6,9 +6,14 @@ In this case, airbyte was able to quickly trigger the execute command on Dune, g
|
||||
It then pushes the data as raw into the database, keeping track of any errors and avoiding duplicates.
|
||||
|
||||
## Demo ##
|
||||
|
||||
the demo is at [Airbye](https://airbyte.nuvari.com)
|
||||
|
||||
username: airbyte
|
||||
|
||||
password: password
|
||||
(i kept the defaults
|
||||
|
||||
(i kept the defaults)
|
||||
|
||||
|
||||
* note the airbyte instance and the open-metadata instance must be seperate machines/vms/lxcs with the supplied docker-compose files)
|
||||
@ -3,17 +3,25 @@
|
||||
## Quickstart: ##
|
||||
|
||||
# Install #
|
||||
|
||||
`python3 -m pip install dbt dbt-postgres`
|
||||
|
||||
# Running #
|
||||
|
||||
To run all transforms
|
||||
|
||||
`dbt run`
|
||||
|
||||
to run all of Dune staging:
|
||||
|
||||
`dbt build --select staging.dune`
|
||||
|
||||
to run bitcoin json staging:
|
||||
|
||||
`dbt build --select staging.bitcoin_json`
|
||||
|
||||
## Models ##
|
||||
Models are in the `models` directory. They are organized `models -> phase -> source`. Currently the only phase is staging, the onlys sources Dune or json data
|
||||
|
||||
Models are in the `models` directory. They are organized `models -> phase -> source`.
|
||||
|
||||
Currently the only phase is staging, the onlys sources Dune or json data
|
||||
@ -1,19 +1,30 @@
|
||||
### Open Metadata & Airflow ###
|
||||
|
||||
## Open Metadata ##
|
||||
|
||||
# Demo #
|
||||
|
||||
[Open Metadata](https://open-metadata.nuvari.com)
|
||||
|
||||
Username: admin
|
||||
|
||||
password: admin
|
||||
|
||||
|
||||
# Usage #
|
||||
|
||||
Open-Metadata has a complete view of all the available data. It allows for the teams to communicate about the tables/views, share queries, monitor data freshness, quickly grasp the data lineaage (what tables were merged to finally become the desiered dataset), as well as allow the data team to monitor which datasets are in active use and by whom.
|
||||
|
||||
## Airflow ##
|
||||
|
||||
# Demo #
|
||||
|
||||
[Airflow](https://airflow.nuvari.com)
|
||||
|
||||
Username: admin
|
||||
|
||||
password: admin
|
||||
|
||||
# Usage #
|
||||
Airflow on the basic level is an advanced task/job orchestrator, able to trigger airbyte jobs, dbt jobs, perform ETL/ELT and more
|
||||
|
||||
Airflow on the basic level is an advanced task/job orchestrator, able to trigger airbyte jobs, dbt jobs, perform ETL/ELT and more
|
||||
|
||||
Loading…
Reference in New Issue
Block a user