Resilience Patterns
Circuit Breaker Pattern
The circuit breaker pattern prevents cascading failures by stopping requests to a failing service once a failure threshold is reached. When the circuit is "open," operations are skipped instead of being sent to the failing service, giving it time to recover. After a cooldown period or a number of consecutive successes, the circuit "closes" and normal operation resumes. This pattern is especially useful when calling external APIs or third-party services that may become temporarily unavailable.
1use orcher::prelude::*;2use anyhow::Result;3use std::sync::atomic::{AtomicUsize, Ordering};45static FAILURE_COUNT: AtomicUsize = AtomicUsize::new(0);6const FAILURE_THRESHOLD: usize = 3;7const SUCCESS_THRESHOLD: usize = 2;89#[workflow(name = "circuit_breaker_workflow", version = "1.0.0")]10pub async fn circuit_breaker_workflow(11 ctx: WorkflowContext,12 operations: Vec<String>,13) -> Result<CircuitBreakerResult> {14 let mut successful = 0;15 let mut failed = 0;16 let mut circuit_open = false;1718 for (idx, operation) in operations.iter().enumerate() {19 if circuit_open {20 tracing::warn!("Circuit OPEN - skipping operation {}", idx + 1);21 continue;22 }2324 // Try operation25 let result = ctx26 .execute_task(external_service_call, operation.clone())27 .await;2829 match result {30 Ok(_) => {31 successful += 1;3233 // Reset failure count on success34 let current_failures = FAILURE_COUNT.load(Ordering::SeqCst);35 if current_failures > 0 {36 FAILURE_COUNT.store(0, Ordering::SeqCst);3738 if successful >= SUCCESS_THRESHOLD {39 circuit_open = false;40 tracing::info!("Circuit CLOSED - recovery detected");41 }42 }43 }44 Err(e) => {45 failed += 1;46 let failures = FAILURE_COUNT.fetch_add(1, Ordering::SeqCst) + 1;4748 if failures >= FAILURE_THRESHOLD {49 circuit_open = true;50 tracing::error!("Circuit OPENED - {} failures", failures);51 }52 }53 }54 }5556 Ok(CircuitBreakerResult {57 successful,58 failed,59 skipped: if circuit_open {60 operations.len() - (successful + failed)61 } else {62 063 },64 circuit_opened: circuit_open,65 })66}Graceful Degradation
Graceful degradation ensures your workflow returns useful results even when primary services are unavailable. Instead of failing outright, you define a chain of fallback data sources — try the primary source first, then a secondary replica, and finally a local cache. Each level may return less fresh or less complete data, but the workflow completes successfully. This pattern is essential for user-facing workflows where partial data is better than an error.
1use orcher::prelude::*;2use anyhow::Result;34#[workflow(name = "degradation_workflow", version = "1.0.0")]5pub async fn degradation_workflow(6 ctx: WorkflowContext,7 input: DataRequest,8) -> Result<DataResponse> {9 // Try primary data source10 let primary_result = ctx11 .execute_task(fetch_from_primary, input.clone())12 .await;1314 match primary_result {15 Ok(data) => {16 tracing::info!("Primary source successful");17 return Ok(data);18 }19 Err(e) => {20 tracing::warn!("Primary source failed: {}, trying fallback", e);21 }22 }2324 // Try secondary data source25 let secondary_result = ctx26 .execute_task(fetch_from_secondary, input.clone())27 .await;2829 match secondary_result {30 Ok(data) => {31 tracing::info!("Secondary source successful");32 return Ok(data);33 }34 Err(e) => {35 tracing::warn!("Secondary source failed: {}, using cache", e);36 }37 }3839 // Use cached data as last resort40 let cached_data: DataResponse = ctx41 .execute_task(fetch_from_cache, input.query)42 .await?;4344 tracing::info!("Using cached data");45 Ok(DataResponse {46 data: cached_data.data,47 source: "cache".to_string(),48 fresh: false,49 })50}Timeout Handling
Timeouts prevent workflows from hanging indefinitely when a task or external call takes longer than expected. Orcher supports two approaches: workflow-level timeouts that race an operation against a timer, and task-level timeouts configured as attributes on the task definition itself. Workflow-level timeouts give you the opportunity to run cleanup logic before failing, while task-level timeouts are simpler and terminate the task automatically.
1use orcher::prelude::*;2use anyhow::Result;3use std::time::Duration;45// ⚠️ CAUTION: Do NOT race an `execute_task` against `ctx.sleep` with6// `tokio::select!` for a "workflow-level timeout". Both `execute_task` and7// `ctx.sleep` are suspend points — on first execution they each return the8// Suspended signal immediately rather than a value, so the `select!` does not9// race the way plain async futures would, and splitting suspend points across10// a `select!` arm is not replay-safe (the journal cannot deterministically11// reconstruct which arm "won"). Enforce timeouts at the task level instead.1213// ✔ Recommended: Task-level timeout — the engine terminates the task14// deterministically when it exceeds the configured duration.15#[task(16 name = "operation_with_timeout",17 timeout_mins = 5 // 5 minute timeout18)]19pub async fn operation_with_timeout(20 _ctx: TaskContext,21 input: OperationInput,22) -> Result<OperationResult> {23 // Task will fail if execution exceeds 5 minutes24 long_running_operation(input).await25}2627#[workflow(name = "timeout_workflow", version = "1.0.0")]28pub async fn timeout_workflow(29 ctx: WorkflowContext,30 input: OperationInput,31) -> Result<ProcessingResult> {32 // The task carries its own timeout; the workflow just runs it and handles33 // the resulting error (which can include cleanup) in a replay-safe way.34 match ctx.execute_task(operation_with_timeout, input).await {35 Ok(data) => Ok(data),36 Err(e) => {37 tracing::error!("Operation timed out or failed: {}", e);3839 // Cleanup — this is a normal, journaled task call40 let _ = ctx.execute_task(cleanup_resources, ()).await;4142 Err(e)43 }44 }45}