adopt Prefect and Minio to build data pipeline service

Go to file

deng 94e98d3b9e untrack asyncio testing script		2023-04-07 15:30:26 +08:00
env	init	2023-03-29 08:45:56 +08:00
service	update service	2023-04-07 12:43:35 +08:00
.gitignore	init	2023-03-29 08:45:56 +08:00
.prefectignore	init	2023-03-29 08:45:56 +08:00
README.md	init	2023-03-29 08:45:56 +08:00
build_flow.py	fix minio connection issue	2023-04-07 12:43:17 +08:00
docker-compose.yaml	fix minio connection issue	2023-04-07 12:43:17 +08:00
test_prefect-deployment.yaml	fix minio connection issue	2023-04-07 12:43:17 +08:00

README.md

Abstract

This is a demo to use Prefect 2.0 and Minio to build a service for data pipeline orchestration🎼

Requirements

MacOS 12.5
Docker 20.10

Dirs

service
- House Prefect server data and Minio storage
env
- prefect.yaml
  - conda env yaml to run this repo

Files

docker-compose.yaml
- a yaml to apply docker-compose to start data pipeline service with basic configuration (run docker-compose -f docker-compose.yaml up)
build_flow.py
- a script to create a small flow and deploy it to the data pipeline server

tags: `Data pipeline` `Object Storage` `Data Engineering`