deng/test_prefect
deng
/
test_prefect
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
adopt Prefect and Minio to build data pipeline service
4 Commits 1 Branch 0 Tags 200 KiB
Python 100%
Go to file
deng 94e98d3b9e untrack asyncio testing script 2023-04-07 15:30:26 +08:00
env init 2023-03-29 08:45:56 +08:00
service update service 2023-04-07 12:43:35 +08:00
.gitignore init 2023-03-29 08:45:56 +08:00
.prefectignore init 2023-03-29 08:45:56 +08:00
README.md init 2023-03-29 08:45:56 +08:00
build_flow.py fix minio connection issue 2023-04-07 12:43:17 +08:00
docker-compose.yaml fix minio connection issue 2023-04-07 12:43:17 +08:00
test_prefect-deployment.yaml fix minio connection issue 2023-04-07 12:43:17 +08:00

README.md

Abstract

This is a demo to use Prefect 2.0 and Minio to build a service for data pipeline orchestration🎼

Requirements

  • MacOS 12.5
  • Docker 20.10

Dirs

  • service
    • House Prefect server data and Minio storage
  • env
    • prefect.yaml
      • conda env yaml to run this repo

Files

  • docker-compose.yaml
    • a yaml to apply docker-compose to start data pipeline service with basic configuration (run docker-compose -f docker-compose.yaml up)
  • build_flow.py
    • a script to create a small flow and deploy it to the data pipeline server
tags: Data pipeline Object Storage Data Engineering