Guides

Best Practices & Performance


Best Practices

1. Control Concurrency

Always limit the number of parallel operations to avoid overwhelming downstream systems, exhausting database connections, or running out of memory. Start with a conservative limit (e.g., 50) and tune based on monitoring.

1
// Good - Controlled concurrency
2
const MAX_CONCURRENT: usize = 50;
3
4
for chunk in items.chunks(MAX_CONCURRENT) {
5
let tasks: Vec<_> = chunk
6
.iter()
7
.map(|item| ctx.execute_task(process_item, item.clone()))
8
.collect();
9
10
for task in tasks {
11
results.push(task.await?);
12
}
13
}
14
15
// Bad - Unbounded parallelism (could be 10,000+ items!)
16
let tasks: Vec<_> = items
17
.iter()
18
.map(|item| ctx.execute_task(process_item, item.clone()))
19
.collect();

2. Handle Partial Failures

Don't let one failed item abort the entire batch. Catch errors per-item and collect both successes and failures, giving the caller enough information to retry just the failed items or accept the partial result.

1
// Good - Continue on partial failures
2
for (item_id, task) in task_handles {
3
match task.await {
4
Ok(result) => successful.push(result),
5
Err(e) => {
6
eprintln!("Task failed: {}", e);
7
failed.push((item_id, e));
8
}
9
}
10
}
11
12
// Bad - One failure stops everything
13
for task in task_handles {
14
let result = task.await?; // Aborts at first error
15
results.push(result);
16
}

3. Aggregate Results Efficiently

For large result sets, avoid collecting all results into memory before aggregating. Instead, compute running aggregates (sum, count, min, max) as results arrive. This keeps memory usage constant regardless of how many items you process.

1
// Good - Stream aggregation (constant memory)
2
let mut total = 0;
3
for task in tasks {
4
let value = task.await?;
5
total += value;
6
}
7
8
// Bad - Collect all then aggregate (memory proportional to item count)
9
let results: Vec<_> = /* collect all results */;
10
let total: i32 = results.iter().sum();

4. Set Appropriate Timeouts

Each fanned-out task should have a timeout that matches its expected duration. Without timeouts, a single hung task can block the entire fan-in phase indefinitely.

1
// Good - Task-appropriate timeout
2
#[task(name = "api_call", timeout_mins = 1)] // API calls: 1 minute
3
#[task(name = "batch_import", timeout_mins = 30)] // Imports: 30 minutes
4
5
// Bad - Timeout too short for the operation
6
#[task(name = "long_computation", timeout_mins = 1)] // Actually takes hours!

5. Monitor Progress

For long-running fan-out operations, log progress at regular intervals so operators can see how much work remains and estimate completion time.

1
// Good - Progress tracking
2
let total = items.len();
3
let mut completed = 0;
4
5
for task in tasks {
6
task.await?;
7
completed += 1;
8
if completed % 100 == 0 || completed == total {
9
println!("Progress: {}/{} ({:.1}%)",
10
completed, total, (completed as f64 / total as f64) * 100.0);
11
}
12
}

6. Choose Tasks vs Child Workflows

Use tasks for lightweight, fast operations (under a few seconds) and child workflows for complex, multi-step operations that need independent error handling, their own journal, or may take minutes to complete.

1
// Good - Tasks for simple, fast operations
2
for item in items {
3
ctx.execute_task(validate_item, item).await?; // Milliseconds
4
}
5
6
// Good - Child workflows for complex operations
7
for order in orders {
8
ctx.start_child_workflow("process_order", order).await?; // Minutes, multi-step
9
}
10
11
// Bad - Child workflows for trivial operations (unnecessary overhead)
12
for item in items {
13
ctx.start_child_workflow("validate_item", item).await?; // Overkill
14
}

Performance Considerations

Parallelism Limits

The degree of parallelism directly impacts resource usage. Choose a concurrency level appropriate for your workload and downstream system capacity.

Concurrent OperationsMemory UsageNetwork ConnectionsRecommended For
1-10MinimalLowSmall batches
10-50LowModerateStandard processing
50-100ModerateHighHigh throughput
100+HighVery HighCarefully tune

Rule of thumb: Start with 50 concurrent operations and tune based on monitoring.

Latency Analysis

Fan-out/fan-in latency is determined by three components: the time to dispatch all tasks (fan-out overhead), the time for the slowest task to complete (parallel execution), and the time to collect results (fan-in overhead).

Serial vs. Parallel — 10 ops × 100ms each
Serial
1000ms

10 × 100ms — each op waits for the one before it.

Parallel fan-out
111ms · 9× faster

10ms fan-out + 100ms slowest op + 1ms fan-in.

total = fan-out + MAX(op times) + fan-in

Result Collection Strategies

Choose a collection strategy based on the number of results and available memory.

1
// Strategy 1: Collect all (good for < 10K results)
2
let results: Vec<_> = /* await all tasks */;
3
4
// Strategy 2: Stream aggregation (good for any size, constant memory)
5
let mut aggregate = Aggregate::new();
6
for task in tasks {
7
let result = task.await?;
8
aggregate.add(result);
9
}
10
11
// Strategy 3: Batch collection (good for > 10K results)
12
for chunk in task_handles.chunks(1000) {
13
let batch_results = /* await chunk */;
14
persist_batch(batch_results).await?;
15
}

Common Patterns Summary

PatternUse CaseExample
Basic Fan-Out/Fan-InIndependent operationsParallel API calls
Rate-Limited Fan-OutProtect downstream systemsAPI with rate limits
Partial Failure HandlingResilient processingOptional validations
Dynamic Fan-OutVariable workloadMulti-source aggregation
Nested Fan-OutHierarchical processingRegion → Store → Product
Streaming Fan-InLarge result setsBig data aggregation

Real-World Use Cases

Multi-API Data Aggregation

API Aggregator
fan-out · parallel
Call Weather API
Call News API
Call Stock API
Call Traffic API
Call Events API
fan-in · gather
Aggregate All Data
Return Dashboard

Batch Email Sending

Email Campaign
fan-out · parallel
Send to segment 1 · 1000 emails
Send to segment 2 · 1000 emails
Send to segment 3 · 1000 emails
Send to segment 4 · 1000 emails
Send to segment 5 · 1000 emails
fan-in · gather
Collect Delivery Stats
Generate Report

Distributed Data Processing

ETL Pipeline
fan-out · parallel
Extract from DB1
Extract from DB2
Extract from S3
Extract from API
Extract from CSV
fan-in · gather
Transform & Validate
Load to Data Warehouse

Next Steps