25 lines
713 B
Markdown
25 lines
713 B
Markdown
# Abstract
|
|
|
|
This is a demo to use [Prefect 2.0](https://github.com/PrefectHQ/prefect) and [Minio](https://github.com/minio/minio) to build a service for data pipeline orchestration🎼
|
|
|
|
# Requirements
|
|
|
|
* MacOS 12.5
|
|
* Docker 20.10
|
|
|
|
# Dirs
|
|
|
|
* **service**
|
|
* House Prefect server data and Minio storage
|
|
* **env**
|
|
* **prefect.yaml**
|
|
* conda env yaml to run this repo
|
|
|
|
# Files
|
|
|
|
* **docker-compose.yaml**
|
|
* a yaml to apply docker-compose to start data pipeline service with basic configuration (run ```docker-compose -f docker-compose.yaml up```)
|
|
* **build_flow.py**
|
|
* a script to create a small flow and deploy it to the data pipeline server
|
|
|
|
###### tags: `Data pipeline` `Object Storage` `Data Engineering` |