Getting Started

Parallel ETL (Fan-Out / Fan-In)


When you need to process multiple data sources concurrently — or split a single large dataset across parallel workers — use child workflows to fan out work and aggregate results. Each child runs independently with its own journal, so a failure in one source doesn't block the others.

Where Batch Processing runs one sequential loop with per-step checkpoints, this pattern trades ordering for throughput: every unit of work becomes its own workflow, running wherever a worker is free. Reach for it when each unit is a multi-step pipeline of its own — extract, transform, and load per source — rather than a uniform chunk of rows.

How it works

The parent workflow does no data work at all. It starts one child workflow per source, then waits for their results. Each child is a full workflow with its own journal, its own retries, and its own lifecycle — the parent only orchestrates.

Parent / Child Fan-Out
multi_source_etl
parent — starts children, aggregates
▼ start_child_workflow × N
process_source
Source A
extract → transform → load
own journal
process_source
Source B
extract → transform → load
own journal
process_source
Source C
extract → transform → load
own journal
▼ handle.result() × N
AggregatedResult
totals + failed sources
Children run concurrently on any available workers — the parent holds no thread while waiting.

The order of operations in the parent matters: it starts all children first, collecting their handles, and only then awaits results. Awaiting each child inside the start loop would serialize the whole pipeline — you'd have parallel infrastructure running one source at a time.

Example: multi-source aggregation

The pipeline below pulls from N independent data sources into one warehouse. Each source runs the full extract → transform → load sequence as a child workflow; the parent gathers per-source results and reports which sources failed.

1
use orcher::prelude::*;
2
use anyhow::Result;
3
4
#[derive(Debug, Clone, Serialize, Deserialize)]
5
struct DataSource {
6
name: String,
7
connection_url: String,
8
query: String,
9
}
10
11
#[derive(Debug, Clone, Serialize, Deserialize)]
12
struct SourceResult {
13
source_name: String,
14
records_loaded: usize,
15
duration_ms: u64,
16
}
17
18
#[derive(Debug, Clone, Serialize, Deserialize)]
19
struct AggregatedResult {
20
total_records: usize,
21
sources_completed: usize,
22
sources_failed: Vec<String>,
23
}
24
25
#[task]
26
async fn extract_from_source(source: DataSource) -> Result<Vec<serde_json::Value>> {
27
// Connect to the source and extract data
28
let records = external::fetch_data(&source.connection_url, &source.query).await?;
29
Ok(records)
30
}
31
32
#[task]
33
async fn transform_records(records: Vec<serde_json::Value>) -> Result<Vec<serde_json::Value>> {
34
// Normalize, deduplicate, enrich records
35
let transformed = records.into_iter()
36
.filter(|r| validate_record(r))
37
.map(|r| normalize_record(r))
38
.collect();
39
Ok(transformed)
40
}
41
42
#[task]
43
async fn load_to_warehouse(table: String, records: Vec<serde_json::Value>) -> Result<usize> {
44
let count = warehouse::bulk_upsert(&table, &records).await?;
45
Ok(count)
46
}
47
48
// Child workflow: processes a single data source end-to-end
49
#[workflow(name = "process_source", version = "1.0.0")]
50
pub async fn process_source(
51
ctx: WorkflowContext,
52
source: DataSource,
53
) -> Result<SourceResult> {
54
let start = std::time::Instant::now();
55
56
let raw = ctx.execute_task(extract_from_source, source.clone()).await?;
57
let transformed = ctx.execute_task(transform_records, raw).await?;
58
let count = ctx.execute_task(
59
load_to_warehouse, ("analytics".to_string(), transformed)
60
).await?;
61
62
Ok(SourceResult {
63
source_name: source.name,
64
records_loaded: count,
65
duration_ms: start.elapsed().as_millis() as u64,
66
})
67
}
68
69
// Parent workflow: fans out across all sources, then aggregates
70
#[workflow(name = "multi_source_etl", version = "1.0.0")]
71
pub async fn multi_source_etl(
72
ctx: WorkflowContext,
73
sources: Vec<DataSource>,
74
) -> Result<AggregatedResult> {
75
// Fan out: start a child workflow per source
76
let mut handles = Vec::new();
77
for source in &sources {
78
let handle = ctx.start_child_workflow(
79
process_source, source.clone()
80
).await?;
81
handles.push((source.name.clone(), handle));
82
}
83
84
// Fan in: collect results
85
let mut total_records = 0;
86
let mut sources_failed = Vec::new();
87
88
for (name, handle) in handles {
89
match handle.result().await {
90
Ok(result) => total_records += result.records_loaded,
91
Err(e) => {
92
println!("Source {} failed: {}", name, e);
93
sources_failed.push(name);
94
}
95
}
96
}
97
98
Ok(AggregatedResult {
99
total_records,
100
sources_completed: sources.len() - sources_failed.len(),
101
sources_failed,
102
})
103
}

Failure isolation

Because every source is its own workflow, a failure is contained to the child where it happened. The other children never notice — and the parent still produces a complete result.

One Source Fails
✓
process_source(A)48,201 records loaded
✗
process_source(B)Extract failed — retries exhausted
✓
process_source(C)112,864 records loaded
=
multi_source_etlAggregates A + C, records B in sources_failed
B's failure never touches A or C. The failed-sources list is your input for a targeted re-run.

Crashes are just as contained. If a child crashes mid-pipeline, it replays its own journal and resumes at its own next step — exactly as described in crash recovery for batch processing. If the parent crashes, its start_child_workflow calls are journaled like any other step: on replay it reattaches to the same children instead of starting duplicates, and children that were already running were never interrupted at all.

Design notes

  • Start first, await second. Collecting all handles before awaiting any result is what makes the fan-out parallel. This is the single most common mistake with this pattern — an await inside the start loop silently serializes it.
  • Child workflows vs. parallel tasks. Use plain tasks when units are uniform, single-step chunks (see Batch Processing). Use child workflows when each unit is a multi-step pipeline that deserves its own journal, retries, and failure boundary — like the per-source E-T-L here. See Use Child Workflows for Source Isolation.
  • Fan-out width is bounded by workers, not the parent. Starting many children is cheap — each is a queued workflow, not a held thread. They execute as worker capacity allows, and the suspended parent consumes nothing while it waits.

Next Steps