Parallel ETL (Fan-Out / Fan-In)
When you need to process multiple data sources concurrently — or split a single large dataset across parallel workers — use child workflows to fan out work and aggregate results. Each child runs independently with its own journal, so a failure in one source doesn't block the others.
Where Batch Processing runs one sequential loop with per-step checkpoints, this pattern trades ordering for throughput: every unit of work becomes its own workflow, running wherever a worker is free. Reach for it when each unit is a multi-step pipeline of its own — extract, transform, and load per source — rather than a uniform chunk of rows.
How it works
The parent workflow does no data work at all. It starts one child workflow per source, then waits for their results. Each child is a full workflow with its own journal, its own retries, and its own lifecycle — the parent only orchestrates.
The order of operations in the parent matters: it starts all children first, collecting their handles, and only then awaits results. Awaiting each child inside the start loop would serialize the whole pipeline — you'd have parallel infrastructure running one source at a time.
Example: multi-source aggregation
The pipeline below pulls from N independent data sources into one warehouse. Each source runs the full extract → transform → load sequence as a child workflow; the parent gathers per-source results and reports which sources failed.
1use orcher::prelude::*;2use anyhow::Result;34#[derive(Debug, Clone, Serialize, Deserialize)]5struct DataSource {6 name: String,7 connection_url: String,8 query: String,9}1011#[derive(Debug, Clone, Serialize, Deserialize)]12struct SourceResult {13 source_name: String,14 records_loaded: usize,15 duration_ms: u64,16}1718#[derive(Debug, Clone, Serialize, Deserialize)]19struct AggregatedResult {20 total_records: usize,21 sources_completed: usize,22 sources_failed: Vec<String>,23}2425#[task]26async fn extract_from_source(source: DataSource) -> Result<Vec<serde_json::Value>> {27 // Connect to the source and extract data28 let records = external::fetch_data(&source.connection_url, &source.query).await?;29 Ok(records)30}3132#[task]33async fn transform_records(records: Vec<serde_json::Value>) -> Result<Vec<serde_json::Value>> {34 // Normalize, deduplicate, enrich records35 let transformed = records.into_iter()36 .filter(|r| validate_record(r))37 .map(|r| normalize_record(r))38 .collect();39 Ok(transformed)40}4142#[task]43async fn load_to_warehouse(table: String, records: Vec<serde_json::Value>) -> Result<usize> {44 let count = warehouse::bulk_upsert(&table, &records).await?;45 Ok(count)46}4748// Child workflow: processes a single data source end-to-end49#[workflow(name = "process_source", version = "1.0.0")]50pub async fn process_source(51 ctx: WorkflowContext,52 source: DataSource,53) -> Result<SourceResult> {54 let start = std::time::Instant::now();5556 let raw = ctx.execute_task(extract_from_source, source.clone()).await?;57 let transformed = ctx.execute_task(transform_records, raw).await?;58 let count = ctx.execute_task(59 load_to_warehouse, ("analytics".to_string(), transformed)60 ).await?;6162 Ok(SourceResult {63 source_name: source.name,64 records_loaded: count,65 duration_ms: start.elapsed().as_millis() as u64,66 })67}6869// Parent workflow: fans out across all sources, then aggregates70#[workflow(name = "multi_source_etl", version = "1.0.0")]71pub async fn multi_source_etl(72 ctx: WorkflowContext,73 sources: Vec<DataSource>,74) -> Result<AggregatedResult> {75 // Fan out: start a child workflow per source76 let mut handles = Vec::new();77 for source in &sources {78 let handle = ctx.start_child_workflow(79 process_source, source.clone()80 ).await?;81 handles.push((source.name.clone(), handle));82 }8384 // Fan in: collect results85 let mut total_records = 0;86 let mut sources_failed = Vec::new();8788 for (name, handle) in handles {89 match handle.result().await {90 Ok(result) => total_records += result.records_loaded,91 Err(e) => {92 println!("Source {} failed: {}", name, e);93 sources_failed.push(name);94 }95 }96 }9798 Ok(AggregatedResult {99 total_records,100 sources_completed: sources.len() - sources_failed.len(),101 sources_failed,102 })103}Failure isolation
Because every source is its own workflow, a failure is contained to the child where it happened. The other children never notice — and the parent still produces a complete result.
Crashes are just as contained. If a child crashes mid-pipeline, it replays its own journal and resumes at its own next step — exactly as described in crash recovery for batch processing. If the parent crashes, its start_child_workflow calls are journaled like any other step: on replay it reattaches to the same children instead of starting duplicates, and children that were already running were never interrupted at all.
Design notes
- Start first, await second. Collecting all handles before awaiting any result is what makes the fan-out parallel. This is the single most common mistake with this pattern — an
awaitinside the start loop silently serializes it. - Child workflows vs. parallel tasks. Use plain tasks when units are uniform, single-step chunks (see Batch Processing). Use child workflows when each unit is a multi-step pipeline that deserves its own journal, retries, and failure boundary — like the per-source E-T-L here. See Use Child Workflows for Source Isolation.
- Fan-out width is bounded by workers, not the parent. Starting many children is cheap — each is a queued workflow, not a held thread. They execute as worker capacity allows, and the suspended parent consumes nothing while it waits.
Next Steps
- Next pattern: Scheduled Sync
- Previous: Batch Processing
- Back to Data Pipelines & ETL overview