Data Pipelines & ETL
Data pipelines are one of the strongest use cases for durable workflow orchestration. Orcher lets you write ETL logic as regular async functions with automatic checkpointing at every task boundary — no duplicate processing, no manual state management.
Checkpointed by default
Every batch you process through execute_task is journaled in PostgreSQL. If your pipeline crashes midway through a million records, it resumes from the exact batch where it left off — completed batches replay from the journal instead of re-running.
Parallel where it counts
Combine checkpointing with fan-out/fan-in to process partitions concurrently: a parent workflow fans out one child task per partition, workers pull them in parallel, and the parent gathers the results before loading.
Multi-source aggregation runs each source as a child workflow, scheduled syncs use durable timers instead of external cron — and all of it is backed by a single PostgreSQL instance.
Patterns
Batch Processing
Break large datasets into manageable chunks with automatic checkpointing at every task boundary.
Parallel ETL
Process data in parallel with fan-out/fan-in patterns for maximum throughput.
Scheduled Sync
Incremental data sync using change data capture patterns with durable timers.
Data Validation
Validate, transform, and quarantine data with configurable rules and error handling.
Best Practices & Performance
Production tips, performance optimization, and real-world examples for data pipelines.