Guides

Resilience Patterns


Circuit Breaker Pattern

The circuit breaker pattern prevents cascading failures by stopping requests to a failing service once a failure threshold is reached. When the circuit is "open," operations are skipped instead of being sent to the failing service, giving it time to recover. After a cooldown period or a number of consecutive successes, the circuit "closes" and normal operation resumes. This pattern is especially useful when calling external APIs or third-party services that may become temporarily unavailable.

1
use orcher::prelude::*;
2
use anyhow::Result;
3
use std::sync::atomic::{AtomicUsize, Ordering};
4
5
static FAILURE_COUNT: AtomicUsize = AtomicUsize::new(0);
6
const FAILURE_THRESHOLD: usize = 3;
7
const SUCCESS_THRESHOLD: usize = 2;
8
9
#[workflow(name = "circuit_breaker_workflow", version = "1.0.0")]
10
pub async fn circuit_breaker_workflow(
11
ctx: WorkflowContext,
12
operations: Vec<String>,
13
) -> Result<CircuitBreakerResult> {
14
let mut successful = 0;
15
let mut failed = 0;
16
let mut circuit_open = false;
17
18
for (idx, operation) in operations.iter().enumerate() {
19
if circuit_open {
20
tracing::warn!("Circuit OPEN - skipping operation {}", idx + 1);
21
continue;
22
}
23
24
// Try operation
25
let result = ctx
26
.execute_task(external_service_call, operation.clone())
27
.await;
28
29
match result {
30
Ok(_) => {
31
successful += 1;
32
33
// Reset failure count on success
34
let current_failures = FAILURE_COUNT.load(Ordering::SeqCst);
35
if current_failures > 0 {
36
FAILURE_COUNT.store(0, Ordering::SeqCst);
37
38
if successful >= SUCCESS_THRESHOLD {
39
circuit_open = false;
40
tracing::info!("Circuit CLOSED - recovery detected");
41
}
42
}
43
}
44
Err(e) => {
45
failed += 1;
46
let failures = FAILURE_COUNT.fetch_add(1, Ordering::SeqCst) + 1;
47
48
if failures >= FAILURE_THRESHOLD {
49
circuit_open = true;
50
tracing::error!("Circuit OPENED - {} failures", failures);
51
}
52
}
53
}
54
}
55
56
Ok(CircuitBreakerResult {
57
successful,
58
failed,
59
skipped: if circuit_open {
60
operations.len() - (successful + failed)
61
} else {
62
0
63
},
64
circuit_opened: circuit_open,
65
})
66
}

Graceful Degradation

Graceful degradation ensures your workflow returns useful results even when primary services are unavailable. Instead of failing outright, you define a chain of fallback data sources — try the primary source first, then a secondary replica, and finally a local cache. Each level may return less fresh or less complete data, but the workflow completes successfully. This pattern is essential for user-facing workflows where partial data is better than an error.

1
use orcher::prelude::*;
2
use anyhow::Result;
3
4
#[workflow(name = "degradation_workflow", version = "1.0.0")]
5
pub async fn degradation_workflow(
6
ctx: WorkflowContext,
7
input: DataRequest,
8
) -> Result<DataResponse> {
9
// Try primary data source
10
let primary_result = ctx
11
.execute_task(fetch_from_primary, input.clone())
12
.await;
13
14
match primary_result {
15
Ok(data) => {
16
tracing::info!("Primary source successful");
17
return Ok(data);
18
}
19
Err(e) => {
20
tracing::warn!("Primary source failed: {}, trying fallback", e);
21
}
22
}
23
24
// Try secondary data source
25
let secondary_result = ctx
26
.execute_task(fetch_from_secondary, input.clone())
27
.await;
28
29
match secondary_result {
30
Ok(data) => {
31
tracing::info!("Secondary source successful");
32
return Ok(data);
33
}
34
Err(e) => {
35
tracing::warn!("Secondary source failed: {}, using cache", e);
36
}
37
}
38
39
// Use cached data as last resort
40
let cached_data: DataResponse = ctx
41
.execute_task(fetch_from_cache, input.query)
42
.await?;
43
44
tracing::info!("Using cached data");
45
Ok(DataResponse {
46
data: cached_data.data,
47
source: "cache".to_string(),
48
fresh: false,
49
})
50
}

Timeout Handling

Timeouts prevent workflows from hanging indefinitely when a task or external call takes longer than expected. Orcher supports two approaches: workflow-level timeouts that race an operation against a timer, and task-level timeouts configured as attributes on the task definition itself. Workflow-level timeouts give you the opportunity to run cleanup logic before failing, while task-level timeouts are simpler and terminate the task automatically.

1
use orcher::prelude::*;
2
use anyhow::Result;
3
use std::time::Duration;
4
5
// ⚠️ CAUTION: Do NOT race an `execute_task` against `ctx.sleep` with
6
// `tokio::select!` for a "workflow-level timeout". Both `execute_task` and
7
// `ctx.sleep` are suspend points — on first execution they each return the
8
// Suspended signal immediately rather than a value, so the `select!` does not
9
// race the way plain async futures would, and splitting suspend points across
10
// a `select!` arm is not replay-safe (the journal cannot deterministically
11
// reconstruct which arm "won"). Enforce timeouts at the task level instead.
12
13
// ✔ Recommended: Task-level timeout — the engine terminates the task
14
// deterministically when it exceeds the configured duration.
15
#[task(
16
name = "operation_with_timeout",
17
timeout_mins = 5 // 5 minute timeout
18
)]
19
pub async fn operation_with_timeout(
20
_ctx: TaskContext,
21
input: OperationInput,
22
) -> Result<OperationResult> {
23
// Task will fail if execution exceeds 5 minutes
24
long_running_operation(input).await
25
}
26
27
#[workflow(name = "timeout_workflow", version = "1.0.0")]
28
pub async fn timeout_workflow(
29
ctx: WorkflowContext,
30
input: OperationInput,
31
) -> Result<ProcessingResult> {
32
// The task carries its own timeout; the workflow just runs it and handles
33
// the resulting error (which can include cleanup) in a replay-safe way.
34
match ctx.execute_task(operation_with_timeout, input).await {
35
Ok(data) => Ok(data),
36
Err(e) => {
37
tracing::error!("Operation timed out or failed: {}", e);
38
39
// Cleanup — this is a normal, journaled task call
40
let _ = ctx.execute_task(cleanup_resources, ()).await;
41
42
Err(e)
43
}
44
}
45
}

Next Steps