Getting Started

Distributed Systems Deep Dive


Before we explore more patterns, let's understand the fundamental distributed systems challenges that Orcher addresses. This knowledge will help you make better architectural decisions.

The CAP Theorem and Orcher's Trade-offs

CAP Theorem states that a distributed system can provide at most two of:

  • Consistency (C): All nodes see the same data at the same time
  • Availability (A): Every request receives a response (success or failure)
  • Partition Tolerance (P): System continues operating despite network failures

Orcher's Position:

Orcher is built on PostgreSQL, which is a CP system (Consistency + Partition tolerance). Here's what that means for you:

What You Get:

  • ✓ Strong consistency: Workflow state is always correct
  • ✓ ACID transactions: No lost or corrupted workflow data
  • ✓ Reliable journaling: Every step is durably recorded
  • ✓ Partition tolerance: Can handle network failures (with retries)

What You Trade:

  • ✗ Not available during PostgreSQL outages
  • ✗ Requires database connectivity for all operations
  • ✗ Single database is a bottleneck (though horizontal scaling is possible)

Why This Trade-off Makes Sense:

For workflow orchestration, correctness is more important than availability. Consider:

1
// Scenario: Processing a $10,000 payment
2
let charge = ctx.execute_task(charge_stripe, payment).await?;
3
ctx.execute_task(save_to_db, charge.id).await?;

If we prioritized availability over consistency:

  • We might accept the charge even though the database is down
  • We'd lose the payment record
  • Customer's money disappears

With Orcher's CP approach:

  • If database is down, workflow fails immediately
  • No charge is created (no money is lost)
  • Workflow retries when database recovers
  • Customer gets consistent experience

Consistency Models Explained

Understanding consistency models helps you reason about concurrent workflows and race conditions.

Read Your Own Writes

Orcher guarantees that a workflow always sees its own writes:

1
// Workflow always sees its own task results
2
let user_id = ctx.execute_task(create_user, user_data=user_data).await?;
3
let balance = ctx.execute_task(get_balance, user_id=user_id).await?; // ✓ Sees created user

Monotonic Reads

Once a workflow reads a value, it never sees an older value:

1
let balance1 = ctx.execute_task(get_balance, user_id=user_id).await?; // Returns 100
2
ctx.execute_task(add_credits, (user_id, 50)).await?; // Balance now 150
3
let balance2 = ctx.execute_task(get_balance, user_id=user_id).await?; // Returns 150 (never goes back to 100)

Causal Consistency

If task B depends on task A, Orcher ensures B sees A's effects:

1
// Task A causes task B
2
let order_id = ctx.execute_task(create_order, order_data).await?;
3
let inventory = ctx.execute_task(reserve_inventory, order_id).await?; // ✓ Sees the order

What Orcher Does NOT Guarantee

Isolation Between Workflows:

Two workflows running concurrently can see inconsistent state:

1
// Workflow 1
2
let balance = ctx.execute_task(get_balance, user_id="user_123").await?; // 100
3
ctx.execute_task(add_credits, ("user_123", 50)).await?; // Balance = 150
4
5
// Workflow 2 (running at same time)
6
let balance = ctx.execute_task(get_balance, user_id="user_123").await?; // Might be 100 or 150

Solution: Use database-level locking:

1
#[task]
2
async fn add_credits_atomic(user_id: String, amount: i64) -> Result<()> {
3
let mut tx = db.begin().await?;
4
5
// SELECT FOR UPDATE locks the row
6
tx.query("SELECT * FROM users WHERE id = $1 FOR UPDATE", user_id).await?;
7
tx.query("UPDATE users SET balance = balance + $1 WHERE id = $2", amount, user_id).await?;
8
9
tx.commit().await?;
10
Ok(())
11
}

Saga Pattern vs. Two-Phase Commit (2PC)

Both handle distributed transactions, but with different trade-offs.

Two-Phase Commit (2PC)

How it works:

  1. Prepare Phase: Coordinator asks all participants: "Can you commit?"
  2. Commit Phase: If all say yes, coordinator tells everyone to commit
2PC: Two Round-Trips, Locks Held Throughout
prepare
coordinator → A, B"can you commit?" — participants lock & vote
prepare
A, B → coordinatorboth reply "yes" — resources stay locked
commit
coordinator → A, B"commit" — everyone finalizes, then unlocks
if crash
coordinator dies mid-commitA and B block indefinitely, still holding locks
Atomic, but the coordinator is a single point of failure and locks block progress.

Problems with 2PC:

  1. Blocking: If coordinator crashes during commit phase, participants are stuck waiting
  2. Single point of failure: Coordinator is critical
  3. Network partitions: Can cause indefinite blocking
  4. Performance: Requires multiple round-trips
  5. Lock contention: Resources locked during entire protocol

Saga Pattern (Orcher's Approach)

How it works:

  1. Execute each step sequentially
  2. If step fails, execute compensating transactions for completed steps
Saga: No Locks, Compensate on Failure
Success path
step 1 → step 2 → step 3 → step 4

Each step commits independently — nothing held across the whole transaction.

Failure path
step 3 ✗
↺ compensate 2
↺ compensate 1

Undo the completed steps in reverse. No coordinator to block on.

Trades 2PC's instant atomicity for availability: brief inconsistency, resolved by compensation.

Orcher makes the saga the natural choice: the "completed steps" list and the compensations are just journaled tasks, so a crash mid-rollback resumes the rollback — the coordinator-crash problem that sinks 2PC simply doesn't exist.

Orcher's Saga Implementation:

1
#[workflow]
2
pub async fn order_saga(ctx: WorkflowContext, order: Order) -> Result<String> {
3
let mut completed_steps = Vec::new();
4
5
// Try each step, track what succeeded
6
match ctx.execute_task(reserve_inventory, order.items.clone()).await {
7
Ok(reservation) => {
8
completed_steps.push(CompensationStep::ReleaseInventory(reservation.id));
9
10
match ctx.execute_task(charge_payment, order.amount).await {
11
Ok(payment) => {
12
completed_steps.push(CompensationStep::RefundPayment(payment.id));
13
14
// Success! No compensation needed
15
return Ok(order.id);
16
}
17
Err(e) => {
18
// Compensate in reverse order
19
for step in completed_steps.iter().rev() {
20
compensate(ctx, step).await?;
21
}
22
return Err(e);
23
}
24
}
25
}
26
Err(e) => return Err(e),
27
}
28
}

Saga vs. 2PC Comparison:

AspectTwo-Phase CommitSaga Pattern
IsolationFull (locks held)None (eventual consistency)
AvailabilityLow (blocking)High (non-blocking)
ComplexityProtocol complexityBusiness logic complexity
ScalabilityPoor (coordinator bottleneck)Good (independent steps)
Failure handlingAutomatic rollbackManual compensation
Network partitionsDeadlocks possibleContinues executing
PerformanceSlower (multiple round-trips)Faster (single-direction flow)

When to use each:

  • 2PC: Banking transfers between accounts (need atomicity)
  • Saga: E-commerce orders, booking systems, workflow orchestration

Why Orcher uses Sagas:

  1. Better availability: Don't block on coordinator
  2. Simpler infrastructure: No distributed coordinator needed
  3. Flexibility: Compensation logic tailored to business needs
  4. Observability: Can see exactly where a saga is in its progression
  5. Failure recovery: Can retry individual steps

Event Sourcing and Orcher's Journal

Orcher's workflow journal is similar to event sourcing, but optimized for workflows.

What is Event Sourcing?

Store all changes as a sequence of events instead of current state:

1
// Traditional (current state)
2
struct Account {
3
id: String,
4
balance: i64, // Current balance only
5
}
6
7
// Event sourcing (full history)
8
enum AccountEvent {
9
Created { id: String },
10
Deposited { amount: i64 },
11
Withdrawn { amount: i64 },
12
}
13
14
// Rebuild state by replaying events
15
let events = vec![
16
AccountEvent::Created { id: "acc_123" },
17
AccountEvent::Deposited { amount: 1000 },
18
AccountEvent::Withdrawn { amount: 200 },
19
];
20
21
let balance = events.iter().fold(0, |bal, event| {
22
match event {
23
AccountEvent::Deposited { amount } => bal + amount,
24
AccountEvent::Withdrawn { amount } => bal - amount,
25
_ => bal,
26
}
27
}); // balance = 800

Orcher's Journal as Event Store

The workflow journal is an event store for workflow execution:

1
-- Each journal entry is an event
2
SELECT * FROM workflow_execution_journal WHERE workflow_id = 'wf_123' ORDER BY step_id;
3
4
| step_id | task_name | result | created_at |
5
|---------|-----------------|----------------------|-------------|
6
| 1 | create_user | {"id": "user_456"} | 10:00:00 |
7
| 2 | send_email | {"sent": true} | 10:00:01 |
8
| 3 | create_order | {"id": "order_789"} | 10:00:02 |

Rebuilding workflow state:

When a workflow resumes after a crash, the worker re-runs your workflow function from the top. Replay is not a server-side loop over journal rows — instead, the orchestrator hands the journal to the worker, and each task call consults it: any step that already completed returns its recorded result without executing again.

1
// The worker re-executes your workflow function. Each task call
2
// checks the journal before running:
3
async fn process_order(ctx: WorkflowContext, order: Order) -> Result<Receipt> {
4
// Already in the journal → returns {"id": "user_123"}, does NOT run again
5
let user = ctx.execute_task(create_user, order.clone()).await?;
6
7
// Already in the journal → returns the cached result, email not re-sent
8
ctx.execute_task(send_email, user.id.clone()).await?;
9
10
// NOT in the journal → executes for the first time and is recorded
11
let receipt = ctx.execute_task(create_order, user.id).await?;
12
Ok(receipt)
13
}

Benefits of Orcher's Approach

1. Time Travel Debugging

See exactly what happened at any point:

1
-- What did the workflow do before crashing?
2
SELECT task_name, result, completed_at
3
FROM workflow_execution_journal
4
WHERE workflow_id = 'wf_failed_123'
5
ORDER BY step_id;

2. Audit Trail

Perfect record of all operations:

1
-- Did we charge this customer?
2
SELECT * FROM workflow_execution_journal
3
WHERE workflow_id = 'wf_payment_456'
4
AND task_name = 'charge_stripe';

3. Deterministic Replay

Because the journal is an ordered, immutable record of what happened, re-running the workflow function against it always reproduces the same decisions in the same order. Replay happens automatically when a worker recovers an execution — there is no separate "replay this workflow" call. To exercise replay determinism in tests, use the test environment to run a workflow and assert on its outcome.

4. Performance Analysis

Find bottlenecks:

1
SELECT
2
task_name,
3
AVG(EXTRACT(EPOCH FROM (completed_at - created_at))) as avg_seconds
4
FROM workflow_execution_journal
5
GROUP BY task_name
6
ORDER BY avg_seconds DESC;

Eventual Consistency in Practice

Orcher workflows are eventually consistent—here's what that means and how to handle it.

What is Eventual Consistency?

A system guarantees that, given enough time without new updates, all replicas will converge to the same value.

Example in Orcher:

1
// Workflow updates user balance
2
ctx.execute_task(add_credits, ("user_123", 100)).await?;
3
4
// Immediately after:
5
// - Primary database has new balance (500)
6
// - Read replica might still show old balance (400)
7
// - Cache might show old balance (400)
8
// - Eventually (seconds later), all show 500

Patterns for Eventual Consistency

Pattern 1: Read After Write from Primary

1
#[task]
2
async fn update_and_read(user_id: String, amount: i64) -> Result<i64> {
3
let mut tx = db.begin().await?;
4
5
// Write to primary
6
tx.query("UPDATE users SET balance = balance + $1 WHERE id = $2", amount, user_id).await?;
7
8
// Read from same transaction (sees own write)
9
let balance: i64 = tx.query_one(
10
"SELECT balance FROM users WHERE id = $1",
11
user_id
12
).await?.get(0);
13
14
tx.commit().await?;
15
Ok(balance)
16
}

Pattern 2: Conflict-Free Replicated Data Types (CRDTs)

Use data structures that converge without coordination:

1
// Instead of setting absolute values (conflicts possible)
2
// Use operations that commute
3
4
// ❌ Conflicting updates
5
update_balance(user_id, 500); // Set to 500
6
update_balance(user_id, 600); // Set to 600 (which wins?)
7
8
// ✓ Commutative updates
9
add_to_balance(user_id, 100); // Add 100
10
add_to_balance(user_id, 200); // Add 200 (order doesn't matter)

Pattern 3: Version Vectors

Detect and handle conflicts:

1
#[derive(Serialize, Deserialize)]
2
struct VersionedData {
3
value: String,
4
version: u64,
5
}
6
7
#[task]
8
async fn update_with_version(key: String, new_value: String, expected_version: u64) -> Result<()> {
9
let current = db.get(key).await?;
10
11
if current.version != expected_version {
12
// Conflict! Handle it
13
return Err(anyhow::anyhow!("Version conflict"));
14
}
15
16
db.set(key, VersionedData {
17
value: new_value,
18
version: expected_version + 1,
19
}).await?;
20
21
Ok(())
22
}

The Write-Ahead Log (WAL) Pattern

Orcher's journal implements the WAL pattern used in databases.

How WAL Works:

  1. Before executing operation, write intent to log
  2. Execute the operation
  3. Mark log entry as completed

Why It's Reliable:

1
// Step 1: Write to journal (durable)
2
journal.append(Entry {
3
step_id: 5,
4
task_name: "charge_stripe",
5
input: payment_data,
6
status: "pending",
7
}).await?;
8
9
// Step 2: Execute task
10
let result = charge_stripe_api(payment_data).await?;
11
12
// Step 3: Update journal (durable)
13
journal.update(5, JournalUpdate {
14
result: result,
15
status: "completed",
16
}).await?;
17
18
// If crash happens between step 2 and 3:
19
// - Journal shows "pending"
20
// - On restart, can retry or check if it actually succeeded
21
// - Never lose track of what was attempted

Compare to non-WAL approach:

1
// ❌ Without WAL
2
let result = charge_stripe_api(payment_data).await?; // ← CRASH HERE = lost money
3
db.save_payment(result).await?;

If crash happens after charging but before saving, the payment is lost forever.

With Orcher's WAL:

1
// ✓ With WAL
2
journal.append("charge_stripe", payment_data).await?; // ← Intent recorded
3
let result = charge_stripe_api(payment_data).await?; // ← CRASH HERE = can recover
4
journal.complete("charge_stripe", result).await?;

Journal shows we attempted to charge. On recovery, check Stripe API to see if charge succeeded, then continue.


Pattern 4: Long-Running Workflows with Human Approval

Some workflows need to pause and wait for human input before continuing. Orcher handles this with durable sleeps and external signals.

The Problem

1
// Traditional approach - doesn't survive restarts
2
async function processExpenseReport(report: ExpenseReport) {
3
await saveReport(report);
4
await sendApprovalEmail(report.managerId);
5
6
// ❌ Can't wait here - server will restart, lose state
7
await waitForApproval(); // How long? What if server crashes?
8
9
await processPayment(report);
10
}

The Orcher Solution

1
#[workflow(name = "expense_approval")]
2
pub async fn expense_approval_workflow(
3
ctx: WorkflowContext,
4
report: ExpenseReport,
5
) -> Result<String> {
6
// Step 1: Save report
7
let report_id = ctx.execute_task(save_report, report.clone()).await?;
8
9
// Step 2: Send approval request
10
ctx.execute_task(send_approval_email, (report_id.clone(), report.manager_id.clone())).await?;
11
12
// Step 3: Wait for approval event (durable wait) - returns the deserialized payload
13
let approval_result: ApprovalDecision = ctx.wait_for_event("approval_decision").await?;
14
15
// Step 4: Process based on approval
16
match approval_result.approved {
17
true => {
18
let payment_id = ctx.execute_task(process_payment, report.amount).await?;
19
ctx.execute_task(notify_employee, (report.employee_id, "approved", payment_id)).await?;
20
Ok(format!("Approved and paid: {}", payment_id))
21
}
22
false => {
23
ctx.execute_task(notify_employee, (report.employee_id, "rejected", approval_result.reason)).await?;
24
Err(anyhow::anyhow!("Report rejected: {}", approval_result.reason))
25
}
26
}
27
}
28
29
// External API to send approval decision (via a workflow handle)
30
async fn approve_expense(
31
orcher: Client,
32
workflow_id: String,
33
approved: bool,
34
reason: Option<String>,
35
) -> Result<()> {
36
let handle = orcher.get_workflow_handle(workflow_id).await?;
37
handle.send_event(
38
"approval_decision",
39
ApprovalDecision { approved, reason },
40
).await?;
41
42
Ok(())
43
}

How it works:

  1. Workflow starts, saves report, sends email
  2. Workflow pauses at wait_for_event() (worker is freed)
  3. Days pass, servers restart, deployments happen (workflow survives)
  4. Manager clicks "Approve" in email → API call to send_event()
  5. Workflow resumes from exact position, processes payment

Key insight: The workflow can wait for hours, days, or weeks. It's not consuming resources—it's just a row in PostgreSQL waiting for a signal.

Pattern 5: Child Workflows and Composition

Break complex workflows into smaller, reusable pieces.

When to Use Child Workflows

Use child workflows when you have:

  • Reusable business processes
  • Long-running sub-processes that should be monitored independently
  • Need for parallel execution of complex workflows
  • Different retry policies for sub-processes
1
// Parent workflow: Onboard new customer
2
#[workflow(name = "onboard_customer")]
3
pub async fn onboard_customer(
4
ctx: WorkflowContext,
5
customer: NewCustomer,
6
) -> Result<OnboardingResult> {
7
// Step 1: Create account
8
let account_id = ctx.execute_task(create_account, customer.clone()).await?;
9
10
// Step 2: Start child workflows by name (each returns a handle immediately)
11
let billing_handle = ctx.start_child_workflow(
12
"setup_billing",
13
BillingSetup {
14
account_id: account_id.clone(),
15
plan: customer.plan,
16
payment_method: customer.payment_method,
17
},
18
).await?;
19
let provisioning_handle = ctx.start_child_workflow(
20
"provision_resources",
21
ResourceRequest {
22
account_id: account_id.clone(),
23
region: customer.region,
24
tier: customer.tier,
25
},
26
).await?;
27
let notification_handle = ctx.start_child_workflow(
28
"welcome_sequence",
29
WelcomeSequence {
30
account_id: account_id.clone(),
31
email: customer.email,
32
},
33
).await?;
34
35
// Await all three results in parallel
36
let (billing_result, provisioning_result, notification_result): (
37
BillingSetup,
38
ProvisioningResult,
39
NotificationResult,
40
) = tokio::try_join!(
41
billing_handle.result(),
42
provisioning_handle.result(),
43
notification_handle.result(),
44
)?;
45
46
// Step 3: Finalize onboarding
47
ctx.execute_task(activate_account, account_id.clone()).await?;
48
49
Ok(OnboardingResult {
50
account_id,
51
billing_id: billing_result.billing_id,
52
resources: provisioning_result.resources,
53
welcome_emails_sent: notification_result.emails_sent,
54
})
55
}
56
57
// Child workflow: Setup billing (independent, reusable)
58
#[workflow(name = "setup_billing")]
59
pub async fn setup_billing_workflow(
60
ctx: WorkflowContext,
61
setup: BillingSetup,
62
) -> Result<BillingSetup> {
63
// Create Stripe customer
64
let stripe_customer_id = ctx.execute_task(create_stripe_customer, setup.account_id.clone()).await?;
65
66
// Attach payment method
67
ctx.execute_task(attach_payment_method, (stripe_customer_id.clone(), setup.payment_method)).await?;
68
69
// Create subscription
70
let subscription_id = ctx.execute_task(create_subscription, (stripe_customer_id, setup.plan)).await?;
71
72
// Save billing info
73
let billing_id = ctx.execute_task(save_billing_info, (setup.account_id, subscription_id)).await?;
74
75
Ok(BillingSetup { billing_id })
76
}

Benefits of child workflows:

  1. Composition: Build complex flows from simple, reusable pieces
  2. Independent monitoring: Each child workflow has its own ID, status, and logs
  3. Parallel execution: Multiple child workflows run concurrently
  4. Isolated retries: Child workflow failures don't affect siblings
  5. Better testing: Test child workflows independently

Pattern 6: Advanced Error Handling Strategies

Retry Policies Per Task

Different tasks need different retry strategies:

1
// Retry policy is declared per task on the #[task] macro. Intervals are in
2
// whole seconds; a backoff_coefficient of 1.0 means fixed (non-growing) delays.
3
4
// Task 1: Quick retries for transient network issues
5
#[task(retry_policy(max_attempts = 5, initial_interval = 1, max_interval = 5, backoff_coefficient = 2.0))]
6
async fn fetch_data(url: String) -> Result<Data> { /* ... */ }
7
8
// Task 2: Slow retries for rate-limited APIs (fixed 60s between attempts)
9
#[task(retry_policy(max_attempts = 10, initial_interval = 60, max_interval = 60, backoff_coefficient = 1.0))]
10
async fn enrich_data(data: Data) -> Result<Data> { /* ... */ }
11
12
// Task 3: No retries for validation (fail fast)
13
#[task(retry_policy(max_attempts = 1))]
14
async fn validate_data(data: Data) -> Result<Data> { /* ... */ }
15
16
#[workflow(name = "complex_workflow")]
17
pub async fn complex_workflow(ctx: WorkflowContext, input: Input) -> Result<Output> {
18
let data = ctx.execute_task(fetch_data, input.url).await?;
19
let enriched = ctx.execute_task(enrich_data, data).await?;
20
let validated = ctx.execute_task(validate_data, enriched).await?;
21
22
Ok(validated)
23
}

Error Classification and Handling

Distinguish between different error types and handle them appropriately:

1
use anyhow::Context;
2
3
#[derive(Debug, thiserror::Error)]
4
pub enum WorkflowError {
5
#[error("Transient error: {0}")]
6
Transient(String),
7
8
#[error("Permanent error: {0}")]
9
Permanent(String),
10
11
#[error("Business rule violation: {0}")]
12
BusinessRule(String),
13
}
14
15
#[workflow(name = "resilient_payment")]
16
pub async fn resilient_payment_workflow(
17
ctx: WorkflowContext,
18
payment: Payment,
19
) -> Result<Receipt> {
20
// Validate input (permanent failure if invalid)
21
if payment.amount <= 0 {
22
return Err(WorkflowError::Permanent("Invalid amount".to_string()).into());
23
}
24
25
// Check fraud (business rule)
26
let fraud_score = ctx.execute_task(check_fraud, payment.user_id.clone()).await?;
27
if fraud_score > 0.9 {
28
return Err(WorkflowError::BusinessRule("High fraud risk".to_string()).into());
29
}
30
31
// Charge payment (may be transient failure)
32
let charge = match ctx.execute_task(charge_stripe, payment.clone()).await {
33
Ok(charge) => charge,
34
Err(e) if is_transient_error(&e) => {
35
warn!("Transient payment error: {}", e);
36
return Err(WorkflowError::Transient(e.to_string()).into());
37
}
38
Err(e) => {
39
return Err(WorkflowError::Permanent(format!("Payment failed: {}", e)).into());
40
}
41
};
42
43
ctx.execute_task(save_payment, charge.id.clone())
44
.await
45
.context("Failed to save payment")?;
46
47
Ok(Receipt {
48
charge_id: charge.id,
49
amount: payment.amount,
50
})
51
}
52
53
fn is_transient_error(error: &anyhow::Error) -> bool {
54
let error_str = error.to_string().to_lowercase();
55
error_str.contains("timeout") ||
56
error_str.contains("connection") ||
57
error_str.contains("503") ||
58
error_str.contains("rate limit")
59
}

Circuit Breaker Pattern

Protect external services from being overwhelmed during outages:

1
use std::sync::Arc;
2
use tokio::sync::RwLock;
3
4
pub struct CircuitBreaker {
5
failure_threshold: u32,
6
success_threshold: u32,
7
timeout: Duration,
8
state: Arc<RwLock<CircuitState>>,
9
}
10
11
enum CircuitState {
12
Closed { failures: u32 },
13
Open { opened_at: Instant },
14
HalfOpen { successes: u32 },
15
}
16
17
impl CircuitBreaker {
18
pub async fn call<F, T>(&self, f: F) -> Result<T>
19
where
20
F: FnOnce() -> Result<T>,
21
{
22
let mut state = self.state.write().await;
23
24
match *state {
25
CircuitState::Open { opened_at } => {
26
// Check if timeout elapsed
27
if opened_at.elapsed() > self.timeout {
28
*state = CircuitState::HalfOpen { successes: 0 };
29
} else {
30
return Err(anyhow::anyhow!("Circuit breaker is OPEN"));
31
}
32
}
33
_ => {}
34
}
35
36
drop(state); // Release lock before calling function
37
38
let result = f();
39
40
let mut state = self.state.write().await;
41
match result {
42
Ok(value) => {
43
match *state {
44
CircuitState::HalfOpen { successes } => {
45
if successes + 1 >= self.success_threshold {
46
*state = CircuitState::Closed { failures: 0 };
47
} else {
48
*state = CircuitState::HalfOpen { successes: successes + 1 };
49
}
50
}
51
CircuitState::Closed { .. } => {
52
*state = CircuitState::Closed { failures: 0 };
53
}
54
_ => {}
55
}
56
Ok(value)
57
}
58
Err(e) => {
59
match *state {
60
CircuitState::Closed { failures } => {
61
if failures + 1 >= self.failure_threshold {
62
*state = CircuitState::Open { opened_at: Instant::now() };
63
} else {
64
*state = CircuitState::Closed { failures: failures + 1 };
65
}
66
}
67
CircuitState::HalfOpen { .. } => {
68
*state = CircuitState::Open { opened_at: Instant::now() };
69
}
70
_ => {}
71
}
72
Err(e)
73
}
74
}
75
}
76
}
77
78
// Use in task
79
#[task]
80
async fn call_external_service(data: String) -> Result<String> {
81
static CIRCUIT_BREAKER: Lazy<CircuitBreaker> = Lazy::new(|| {
82
CircuitBreaker::new(5, 2, Duration::from_secs(60))
83
});
84
85
CIRCUIT_BREAKER.call(|| {
86
// Make actual API call
87
external_api::process(data)
88
}).await
89
}

Pattern 7: Scheduled and Recurring Workflows

Orcher has no separate cron scheduler process. Instead, model recurrence as a long-running workflow that does its work, sleeps durably until the next run, then restarts with a fresh history. The durable workflow is the schedule — it survives restarts and frees the worker while it waits:

1
use std::time::Duration;
2
3
#[workflow(name = "daily_report")]
4
pub async fn daily_report_workflow(
5
ctx: WorkflowContext,
6
input: DailyReportInput,
7
) -> Result<()> {
8
// Generate report
9
let report_data = ctx.execute_task(generate_report, input.report_type.clone()).await?;
10
11
// Send to stakeholders
12
ctx.execute_task(send_report_email, report_data).await?;
13
14
// Durable sleep until the next run (survives restarts, frees the worker)
15
ctx.sleep(Duration::from_secs(24 * 60 * 60)).await?;
16
17
// Restart with a fresh history for the next cycle
18
ctx.restart_fresh_with(input, None)
19
}

Next Steps