713 B
713 B
Abstract
This is a demo to use Prefect 2.0 and Minio to build a service for data pipeline orchestration🎼
Requirements
- MacOS 12.5
- Docker 20.10
Dirs
- service
- House Prefect server data and Minio storage
- env
- prefect.yaml
- conda env yaml to run this repo
- prefect.yaml
Files
- docker-compose.yaml
- a yaml to apply docker-compose to start data pipeline service with basic configuration (run
docker-compose -f docker-compose.yaml up)
- a yaml to apply docker-compose to start data pipeline service with basic configuration (run
- build_flow.py
- a script to create a small flow and deploy it to the data pipeline server