Dynamic Fan-Out
Dynamic Fan-Out
When the number of parallel operations isn't known at compile time, dynamic fan-out determines the degree of parallelism from runtime data. This is common when aggregating data from a configurable set of sources, processing items from a database query, or distributing work based on user input. The workflow iterates over the dynamic collection, spawns a task for each item, and then collects all results.
Fetch Items
DB query · API call · user input
N items returned
fan-out
Item 1
Item 2
Item 3
...
Item N
fan-in
Collect Results
1use orcher::prelude::*;2use anyhow::Result;34#[derive(Debug, Clone, Serialize, Deserialize)]5struct DataSource {6 source_id: String,7 endpoint: String,8}910#[derive(Debug, Clone, Serialize, Deserialize)]11struct AggregatedData {12 total_sources: usize,13 total_records: usize,14 data: Vec<SourceData>,15}1617#[derive(Debug, Clone, Serialize, Deserialize)]18struct SourceData {19 source_id: String,20 record_count: usize,21}2223// Task to fetch data from a single source24#[task(name = "fetch_source_data")]25pub async fn fetch_source_data(26 _ctx: TaskContext,27 source: DataSource,28) -> Result<SourceData> {29 let record_count = (source.source_id.len() * 100) as usize;3031 Ok(SourceData {32 source_id: source.source_id,33 record_count,34 })35}3637// Dynamic fan-out based on runtime data38#[workflow(name = "data_aggregator", version = "1.0.0")]39pub async fn data_aggregator(40 ctx: WorkflowContext,41 sources: Vec<DataSource>,42) -> Result<AggregatedData> {43 println!("Aggregating data from {} sources", sources.len());4445 // Fan-Out: Query all sources in parallel46 let mut task_handles = Vec::new();4748 for source in &sources {49 println!("Starting query for source: {}", source.source_id);50 let task = ctx.execute_task(fetch_source_data, source.clone());51 task_handles.push(task);52 }5354 // Fan-In: Collect data from all sources55 let mut all_data = Vec::new();56 let mut total_records = 0;5758 for task in task_handles {59 let data: SourceData = task.await?;60 total_records += data.record_count;61 all_data.push(data);62 }6364 println!("Aggregation complete: {} records from {} sources",65 total_records, all_data.len());6667 Ok(AggregatedData {68 total_sources: all_data.len(),69 total_records,70 data: all_data,71 })72}