Guides

Dynamic Fan-Out


Dynamic Fan-Out

When the number of parallel operations isn't known at compile time, dynamic fan-out determines the degree of parallelism from runtime data. This is common when aggregating data from a configurable set of sources, processing items from a database query, or distributing work based on user input. The workflow iterates over the dynamic collection, spawns a task for each item, and then collects all results.

Fetch Items
DB query · API call · user input
N items returned
fan-out
Item 1
Item 2
Item 3
...
Item N
fan-in
Collect Results
1
use orcher::prelude::*;
2
use anyhow::Result;
3
4
#[derive(Debug, Clone, Serialize, Deserialize)]
5
struct DataSource {
6
source_id: String,
7
endpoint: String,
8
}
9
10
#[derive(Debug, Clone, Serialize, Deserialize)]
11
struct AggregatedData {
12
total_sources: usize,
13
total_records: usize,
14
data: Vec<SourceData>,
15
}
16
17
#[derive(Debug, Clone, Serialize, Deserialize)]
18
struct SourceData {
19
source_id: String,
20
record_count: usize,
21
}
22
23
// Task to fetch data from a single source
24
#[task(name = "fetch_source_data")]
25
pub async fn fetch_source_data(
26
_ctx: TaskContext,
27
source: DataSource,
28
) -> Result<SourceData> {
29
let record_count = (source.source_id.len() * 100) as usize;
30
31
Ok(SourceData {
32
source_id: source.source_id,
33
record_count,
34
})
35
}
36
37
// Dynamic fan-out based on runtime data
38
#[workflow(name = "data_aggregator", version = "1.0.0")]
39
pub async fn data_aggregator(
40
ctx: WorkflowContext,
41
sources: Vec<DataSource>,
42
) -> Result<AggregatedData> {
43
println!("Aggregating data from {} sources", sources.len());
44
45
// Fan-Out: Query all sources in parallel
46
let mut task_handles = Vec::new();
47
48
for source in &sources {
49
println!("Starting query for source: {}", source.source_id);
50
let task = ctx.execute_task(fetch_source_data, source.clone());
51
task_handles.push(task);
52
}
53
54
// Fan-In: Collect data from all sources
55
let mut all_data = Vec::new();
56
let mut total_records = 0;
57
58
for task in task_handles {
59
let data: SourceData = task.await?;
60
total_records += data.record_count;
61
all_data.push(data);
62
}
63
64
println!("Aggregation complete: {} records from {} sources",
65
total_records, all_data.len());
66
67
Ok(AggregatedData {
68
total_sources: all_data.len(),
69
total_records,
70
data: all_data,
71
})
72
}

Next Steps