Developer Patterns
Orcher provides powerful primitives for building distributed systems. These patterns show you how to compose workflows, handle failures, process data at scale, and build resilient applications.
Pattern Categories
Orchestration Patterns
Parent-Child Orchestration
Compose complex workflows from reusable child components. Coordinate distributed services, process items in parallel, and build hierarchical orchestration trees.
• Sequential and parallel child execution • Dynamic child spawning • Event-based coordination • Hierarchical workflows • Map-reduce processing
Saga & Compensation
Implement distributed transactions with the saga pattern and compensating actions. Ensure data consistency across multiple services without distributed locks.
• Forward recovery • Backward compensation • Saga coordinators • Idempotent operations
Fan-Out/Fan-In
Split work across multiple parallel operations and aggregate results. Perfect for batch processing, multi-service calls, and data aggregation.
• Parallel task execution • Result aggregation • Partial failure handling • Rate limiting
Long-Running Processes
Build workflows that span hours, days, or weeks. Handle human approvals, scheduled operations, and multi-stage business processes.
• Durable timers • Human-in-the-loop • State persistence • Process checkpoints
Resilience Patterns
Error Handling
Retry Policies
Configure automatic retries with exponential backoff and jitter.
1#[task(2 name = "flaky_api_call",3 retry = 5,4 retry_delay = "exponential",5 max_retry_delay = 3006)]7pub async fn flaky_api_call(ctx: TaskContext, input: Data) -> Result<Response> {8 // Automatically retried on failure9 external_api::call(input).await10}Circuit Breaker
Prevent cascading failures by detecting when a service is down.
1#[workflow(name = "with_circuit_breaker", version = "1.0.0")]2pub async fn with_circuit_breaker(3 ctx: WorkflowContext,4 input: Data,5) -> Result<Response> {6 let circuit = ctx.get_state::<CircuitState>("circuit").await?;7 8 if circuit.is_open() {9 return Err(Error::CircuitOpen);10 }11 12 match ctx.execute_task("call_service", input).await {13 Ok(response) => {14 circuit.record_success();15 Ok(response)16 }17 Err(e) => {18 circuit.record_failure();19 Err(e)20 }21 }22}Graceful Degradation
Provide fallback responses when primary services fail.
1#[workflow(name = "resilient_workflow", version = "1.0.0")]2pub async fn resilient_workflow(3 ctx: WorkflowContext,4 query: String,5) -> Result<Response> {6 // Try primary service7 match ctx.execute_task("primary_service", query.clone()).await {8 Ok(response) => Ok(response),9 Err(_) => {10 // Fall back to cached response11 ctx.execute_task("get_cached_response", query).await12 }13 }14}Data Processing Patterns
Batch Processing
Process large datasets by splitting work across multiple workers.
1#[workflow(name = "batch_processor", version = "1.0.0")]2pub async fn batch_processor(3 ctx: WorkflowContext,4 items: Vec<Item>,5) -> Result<BatchResult> {6 const BATCH_SIZE: usize = 100;7 8 let batches: Vec<_> = items9 .chunks(BATCH_SIZE)10 .map(|chunk| chunk.to_vec())11 .collect();12 13 let mut handles = Vec::new();14 for batch in batches {15 let handle = ctx16 .start_child_workflow("process_batch", batch)17 .await?;18 handles.push(handle);19 }20 21 // Collect results22 let mut results = Vec::new();23 for handle in handles {24 results.push(handle.result::<Result>().await?);25 }26 27 Ok(BatchResult::from(results))28}Stream Processing
Process continuous streams of events with durable state.
1#[workflow(name = "stream_processor", version = "1.0.0")]2pub async fn stream_processor(3 ctx: WorkflowContext,4 stream_id: String,5) -> Result<()> {6 loop {7 // Wait for next batch of events8 let events = ctx9 .wait_for_event_with_timeout::<Vec<Event>>(10 "batch",11 Duration::from_secs(60)12 )13 .await?;14 15 if let Some(events) = events {16 // Process batch17 for event in events {18 ctx.execute_task("process_event", event).await?;19 }20 21 // Update checkpoint22 ctx.set_state("last_processed", ctx.now()).await?;23 }24 }25}Integration Patterns
External API Calls
Make reliable API calls with automatic retries and timeouts.
1#[task(2 name = "call_external_api",3 retry = 3,4 timeout = 305)]6pub async fn call_external_api(7 ctx: TaskContext,8 request: ApiRequest,9) -> Result<ApiResponse> {10 let client = reqwest::Client::new();11 12 let response = client13 .post(&request.url)14 .json(&request.body)15 .send()16 .await?17 .json()18 .await?;19 20 Ok(response)21}Webhook Handling
Wait for webhook callbacks with timeouts.
1#[workflow(name = "payment_with_webhook", version = "1.0.0")]2pub async fn payment_with_webhook(3 ctx: WorkflowContext,4 payment: PaymentData,5) -> Result<PaymentResult> {6 // Initiate payment7 let transaction_id = ctx8 .execute_task("initiate_payment", payment)9 .await?;10 11 // Wait for webhook callback (30 minute timeout)12 let callback = ctx13 .wait_for_event_with_timeout::<WebhookData>(14 "payment_callback",15 Duration::from_secs(1800)16 )17 .await?;18 19 match callback {20 Some(data) if data.success => {21 Ok(PaymentResult::success(transaction_id))22 }23 Some(data) => {24 Ok(PaymentResult::failed(data.error))25 }26 None => {27 // Timeout - check payment status28 ctx.execute_task("check_payment_status", transaction_id).await29 }30 }31}Coordination Patterns
Leader Election
Ensure only one workflow instance is active at a time.
1#[workflow(name = "singleton_job", version = "1.0.0")]2pub async fn singleton_job(3 ctx: WorkflowContext,4 job_id: String,5) -> Result<()> {6 // Try to acquire lock7 let lock_acquired = ctx8 .execute_task("try_acquire_lock", job_id.clone())9 .await?;10 11 if !lock_acquired {12 return Err(Error::AlreadyRunning);13 }14 15 // Do work16 ctx.execute_task("perform_job", job_id.clone()).await?;17 18 // Release lock19 ctx.execute_task("release_lock", job_id).await?;20 21 Ok(())22}Mutex Pattern
Coordinate access to shared resources across workflows.
1#[workflow(name = "with_mutex", version = "1.0.0")]2pub async fn with_mutex(3 ctx: WorkflowContext,4 resource: String,5) -> Result<()> {6 // Acquire mutex7 loop {8 let acquired = ctx9 .execute_task("try_acquire_mutex", resource.clone())10 .await?;11 12 if acquired {13 break;14 }15 16 // Wait and retry17 ctx.sleep(Duration::from_secs(1)).await?;18 }19 20 // Critical section21 ctx.execute_task("use_resource", resource.clone()).await?;22 23 // Release mutex24 ctx.execute_task("release_mutex", resource).await?;25 26 Ok(())27}Testing Patterns
Unit Testing Workflows
Test workflows in isolation with mock dependencies.
1#[cfg(test)]2mod tests {3 use super::*;4 use orcher::testing::*;5 6 #[tokio::test]7 async fn test_workflow() {8 let runtime = OrcherTestRuntime::new();9 let ctx = runtime.workflow_context();10 11 // Mock task responses12 runtime13 .mock_task("external_api_call")14 .returns(ApiResponse { status: "ok" });15 16 // Execute workflow17 let result = my_workflow(ctx, input).await;18 19 assert!(result.is_ok());20 }21}Integration Testing
Test workflows end-to-end with real dependencies.
1#[cfg(test)]2mod integration_tests {3 use super::*;4 5 #[tokio::test]6 async fn test_full_workflow() {7 let client = OrcherClient::new("postgresql://localhost/test").await?;8 9 // Start workflow10 let handle = client11 .start_workflow("my_workflow", "test-123", input)12 .await?;13 14 // Send events15 client16 .send_event(handle.id(), "approval", true)17 .await?;18 19 // Wait for completion20 let result = handle.result::<Output>().await?;21 22 assert_eq!(result.status, "completed");23 }24}Best Practices
Pattern Selection Guide
| Scenario | Pattern | Why |
|---|---|---|
| Multi-service coordination | Parent-Child | Independent lifecycles |
| Distributed transaction | Saga | Compensating actions |
| Batch processing | Fan-Out/Fan-In | Parallel execution |
| Human approval | Long-Running | Durable waits |
| External webhook | Event-based | Async callbacks |
| Retry with backoff | Task retry policy | Automatic recovery |
Performance Tips
- Use tasks for fast operations (under 1 second)
- Use child workflows for long operations (over 1 second)
- Limit parallel children (max 50-100 concurrent)
- Batch small items (group into chunks)
- Use timeouts everywhere (prevent infinite waits)
Common Anti-Patterns
Don't use workflows for CPU-intensive work Use background jobs or serverless functions instead.
Don't create deep hierarchies (>3 levels) Flatten the structure for better observability.
Don't share state between workflows Use events or database for communication.
Don't spawn unlimited children Use batching and rate limiting.
Next Steps
Parent-Child Orchestration
Start with the most fundamental pattern for composing workflows.
Workflow Fundamentals
Learn the basics of workflows, tasks, and execution model.
Event Handling
Master events and signals for workflow coordination.
Error Handling
Build resilient workflows with retry policies and circuit breakers.
Related Resources
• Core Workflows - Workflow fundamentals • Tasks - Task definition and configuration • State Management - Managing workflow state • Use Cases - Real-world examples