Getting Started

Data Pipelines & ETL


Data pipelines are one of the strongest use cases for durable workflow orchestration. Orcher lets you write ETL logic as regular async functions with automatic checkpointing at every task boundary — no duplicate processing, no manual state management.

Checkpointed by default

Every batch you process through execute_task is journaled in PostgreSQL. If your pipeline crashes midway through a million records, it resumes from the exact batch where it left off — completed batches replay from the journal instead of re-running.

Durable Batch Pipeline
✓
Batch 001Processed — journaled
✓
Batch 002Processed — journaled
✗
Batch 003Worker crashed mid-batch
↻
Batch 003Retried — resumes exactly here
○
Batch 004Pending
Batches 001–002 are never re-executed — their results replay from the journal.

Parallel where it counts

Combine checkpointing with fan-out/fan-in to process partitions concurrently: a parent workflow fans out one child task per partition, workers pull them in parallel, and the parent gathers the results before loading.

Fan-Out / Fan-In ETL
Extract
▼ fan-out
Partition A
transform
Partition B
transform
Partition C
transform
▼ fan-in
Aggregate & Load
Each partition is an independent, journaled task — one failure retries alone.

Multi-source aggregation runs each source as a child workflow, scheduled syncs use durable timers instead of external cron — and all of it is backed by a single PostgreSQL instance.


Patterns