Distributed Systems Deep Dive
Before we explore more patterns, let's understand the fundamental distributed systems challenges that Orcher addresses. This knowledge will help you make better architectural decisions.
The CAP Theorem and Orcher's Trade-offs
CAP Theorem states that a distributed system can provide at most two of:
- Consistency (C): All nodes see the same data at the same time
- Availability (A): Every request receives a response (success or failure)
- Partition Tolerance (P): System continues operating despite network failures
Orcher's Position:
Orcher is built on PostgreSQL, which is a CP system (Consistency + Partition tolerance). Here's what that means for you:
What You Get:
- ✓ Strong consistency: Workflow state is always correct
- ✓ ACID transactions: No lost or corrupted workflow data
- ✓ Reliable journaling: Every step is durably recorded
- ✓ Partition tolerance: Can handle network failures (with retries)
What You Trade:
- ✗ Not available during PostgreSQL outages
- ✗ Requires database connectivity for all operations
- ✗ Single database is a bottleneck (though horizontal scaling is possible)
Why This Trade-off Makes Sense:
For workflow orchestration, correctness is more important than availability. Consider:
1// Scenario: Processing a $10,000 payment2let charge = ctx.execute_task(charge_stripe, payment).await?;3ctx.execute_task(save_to_db, charge.id).await?;If we prioritized availability over consistency:
- We might accept the charge even though the database is down
- We'd lose the payment record
- Customer's money disappears
With Orcher's CP approach:
- If database is down, workflow fails immediately
- No charge is created (no money is lost)
- Workflow retries when database recovers
- Customer gets consistent experience
Consistency Models Explained
Understanding consistency models helps you reason about concurrent workflows and race conditions.
Read Your Own Writes
Orcher guarantees that a workflow always sees its own writes:
1// Workflow always sees its own task results2let user_id = ctx.execute_task(create_user, user_data=user_data).await?;3let balance = ctx.execute_task(get_balance, user_id=user_id).await?; // ✓ Sees created userMonotonic Reads
Once a workflow reads a value, it never sees an older value:
1let balance1 = ctx.execute_task(get_balance, user_id=user_id).await?; // Returns 1002ctx.execute_task(add_credits, (user_id, 50)).await?; // Balance now 1503let balance2 = ctx.execute_task(get_balance, user_id=user_id).await?; // Returns 150 (never goes back to 100)Causal Consistency
If task B depends on task A, Orcher ensures B sees A's effects:
1// Task A causes task B2let order_id = ctx.execute_task(create_order, order_data).await?;3let inventory = ctx.execute_task(reserve_inventory, order_id).await?; // ✓ Sees the orderWhat Orcher Does NOT Guarantee
Isolation Between Workflows:
Two workflows running concurrently can see inconsistent state:
1// Workflow 12let balance = ctx.execute_task(get_balance, user_id="user_123").await?; // 1003ctx.execute_task(add_credits, ("user_123", 50)).await?; // Balance = 15045// Workflow 2 (running at same time)6let balance = ctx.execute_task(get_balance, user_id="user_123").await?; // Might be 100 or 150Solution: Use database-level locking:
1#[task]2async fn add_credits_atomic(user_id: String, amount: i64) -> Result<()> {3 let mut tx = db.begin().await?;45 // SELECT FOR UPDATE locks the row6 tx.query("SELECT * FROM users WHERE id = $1 FOR UPDATE", user_id).await?;7 tx.query("UPDATE users SET balance = balance + $1 WHERE id = $2", amount, user_id).await?;89 tx.commit().await?;10 Ok(())11}Saga Pattern vs. Two-Phase Commit (2PC)
Both handle distributed transactions, but with different trade-offs.
Two-Phase Commit (2PC)
How it works:
- Prepare Phase: Coordinator asks all participants: "Can you commit?"
- Commit Phase: If all say yes, coordinator tells everyone to commit
Problems with 2PC:
- Blocking: If coordinator crashes during commit phase, participants are stuck waiting
- Single point of failure: Coordinator is critical
- Network partitions: Can cause indefinite blocking
- Performance: Requires multiple round-trips
- Lock contention: Resources locked during entire protocol
Saga Pattern (Orcher's Approach)
How it works:
- Execute each step sequentially
- If step fails, execute compensating transactions for completed steps
Each step commits independently — nothing held across the whole transaction.
↺ compensate 2
↺ compensate 1
Undo the completed steps in reverse. No coordinator to block on.
Orcher makes the saga the natural choice: the "completed steps" list and the compensations are just journaled tasks, so a crash mid-rollback resumes the rollback — the coordinator-crash problem that sinks 2PC simply doesn't exist.
Orcher's Saga Implementation:
1#[workflow]2pub async fn order_saga(ctx: WorkflowContext, order: Order) -> Result<String> {3 let mut completed_steps = Vec::new();45 // Try each step, track what succeeded6 match ctx.execute_task(reserve_inventory, order.items.clone()).await {7 Ok(reservation) => {8 completed_steps.push(CompensationStep::ReleaseInventory(reservation.id));910 match ctx.execute_task(charge_payment, order.amount).await {11 Ok(payment) => {12 completed_steps.push(CompensationStep::RefundPayment(payment.id));1314 // Success! No compensation needed15 return Ok(order.id);16 }17 Err(e) => {18 // Compensate in reverse order19 for step in completed_steps.iter().rev() {20 compensate(ctx, step).await?;21 }22 return Err(e);23 }24 }25 }26 Err(e) => return Err(e),27 }28}Saga vs. 2PC Comparison:
| Aspect | Two-Phase Commit | Saga Pattern |
|---|---|---|
| Isolation | Full (locks held) | None (eventual consistency) |
| Availability | Low (blocking) | High (non-blocking) |
| Complexity | Protocol complexity | Business logic complexity |
| Scalability | Poor (coordinator bottleneck) | Good (independent steps) |
| Failure handling | Automatic rollback | Manual compensation |
| Network partitions | Deadlocks possible | Continues executing |
| Performance | Slower (multiple round-trips) | Faster (single-direction flow) |
When to use each:
- 2PC: Banking transfers between accounts (need atomicity)
- Saga: E-commerce orders, booking systems, workflow orchestration
Why Orcher uses Sagas:
- Better availability: Don't block on coordinator
- Simpler infrastructure: No distributed coordinator needed
- Flexibility: Compensation logic tailored to business needs
- Observability: Can see exactly where a saga is in its progression
- Failure recovery: Can retry individual steps
Event Sourcing and Orcher's Journal
Orcher's workflow journal is similar to event sourcing, but optimized for workflows.
What is Event Sourcing?
Store all changes as a sequence of events instead of current state:
1// Traditional (current state)2struct Account {3 id: String,4 balance: i64, // Current balance only5}67// Event sourcing (full history)8enum AccountEvent {9 Created { id: String },10 Deposited { amount: i64 },11 Withdrawn { amount: i64 },12}1314// Rebuild state by replaying events15let events = vec![16 AccountEvent::Created { id: "acc_123" },17 AccountEvent::Deposited { amount: 1000 },18 AccountEvent::Withdrawn { amount: 200 },19];2021let balance = events.iter().fold(0, |bal, event| {22 match event {23 AccountEvent::Deposited { amount } => bal + amount,24 AccountEvent::Withdrawn { amount } => bal - amount,25 _ => bal,26 }27}); // balance = 800Orcher's Journal as Event Store
The workflow journal is an event store for workflow execution:
1-- Each journal entry is an event2SELECT * FROM workflow_execution_journal WHERE workflow_id = 'wf_123' ORDER BY step_id;34| step_id | task_name | result | created_at |5|---------|-----------------|----------------------|-------------|6| 1 | create_user | {"id": "user_456"} | 10:00:00 |7| 2 | send_email | {"sent": true} | 10:00:01 |8| 3 | create_order | {"id": "order_789"} | 10:00:02 |Rebuilding workflow state:
When a workflow resumes after a crash, the worker re-runs your workflow function from the top. Replay is not a server-side loop over journal rows — instead, the orchestrator hands the journal to the worker, and each task call consults it: any step that already completed returns its recorded result without executing again.
1// The worker re-executes your workflow function. Each task call2// checks the journal before running:3async fn process_order(ctx: WorkflowContext, order: Order) -> Result<Receipt> {4 // Already in the journal → returns {"id": "user_123"}, does NOT run again5 let user = ctx.execute_task(create_user, order.clone()).await?;67 // Already in the journal → returns the cached result, email not re-sent8 ctx.execute_task(send_email, user.id.clone()).await?;910 // NOT in the journal → executes for the first time and is recorded11 let receipt = ctx.execute_task(create_order, user.id).await?;12 Ok(receipt)13}Benefits of Orcher's Approach
1. Time Travel Debugging
See exactly what happened at any point:
1-- What did the workflow do before crashing?2SELECT task_name, result, completed_at3FROM workflow_execution_journal4WHERE workflow_id = 'wf_failed_123'5ORDER BY step_id;2. Audit Trail
Perfect record of all operations:
1-- Did we charge this customer?2SELECT * FROM workflow_execution_journal3WHERE workflow_id = 'wf_payment_456'4 AND task_name = 'charge_stripe';3. Deterministic Replay
Because the journal is an ordered, immutable record of what happened, re-running the workflow function against it always reproduces the same decisions in the same order. Replay happens automatically when a worker recovers an execution — there is no separate "replay this workflow" call. To exercise replay determinism in tests, use the test environment to run a workflow and assert on its outcome.
4. Performance Analysis
Find bottlenecks:
1SELECT2 task_name,3 AVG(EXTRACT(EPOCH FROM (completed_at - created_at))) as avg_seconds4FROM workflow_execution_journal5GROUP BY task_name6ORDER BY avg_seconds DESC;Eventual Consistency in Practice
Orcher workflows are eventually consistent—here's what that means and how to handle it.
What is Eventual Consistency?
A system guarantees that, given enough time without new updates, all replicas will converge to the same value.
Example in Orcher:
1// Workflow updates user balance2ctx.execute_task(add_credits, ("user_123", 100)).await?;34// Immediately after:5// - Primary database has new balance (500)6// - Read replica might still show old balance (400)7// - Cache might show old balance (400)8// - Eventually (seconds later), all show 500Patterns for Eventual Consistency
Pattern 1: Read After Write from Primary
1#[task]2async fn update_and_read(user_id: String, amount: i64) -> Result<i64> {3 let mut tx = db.begin().await?;45 // Write to primary6 tx.query("UPDATE users SET balance = balance + $1 WHERE id = $2", amount, user_id).await?;78 // Read from same transaction (sees own write)9 let balance: i64 = tx.query_one(10 "SELECT balance FROM users WHERE id = $1",11 user_id12 ).await?.get(0);1314 tx.commit().await?;15 Ok(balance)16}Pattern 2: Conflict-Free Replicated Data Types (CRDTs)
Use data structures that converge without coordination:
1// Instead of setting absolute values (conflicts possible)2// Use operations that commute34// ❌ Conflicting updates5update_balance(user_id, 500); // Set to 5006update_balance(user_id, 600); // Set to 600 (which wins?)78// ✓ Commutative updates9add_to_balance(user_id, 100); // Add 10010add_to_balance(user_id, 200); // Add 200 (order doesn't matter)Pattern 3: Version Vectors
Detect and handle conflicts:
1#[derive(Serialize, Deserialize)]2struct VersionedData {3 value: String,4 version: u64,5}67#[task]8async fn update_with_version(key: String, new_value: String, expected_version: u64) -> Result<()> {9 let current = db.get(key).await?;1011 if current.version != expected_version {12 // Conflict! Handle it13 return Err(anyhow::anyhow!("Version conflict"));14 }1516 db.set(key, VersionedData {17 value: new_value,18 version: expected_version + 1,19 }).await?;2021 Ok(())22}The Write-Ahead Log (WAL) Pattern
Orcher's journal implements the WAL pattern used in databases.
How WAL Works:
- Before executing operation, write intent to log
- Execute the operation
- Mark log entry as completed
Why It's Reliable:
1// Step 1: Write to journal (durable)2journal.append(Entry {3 step_id: 5,4 task_name: "charge_stripe",5 input: payment_data,6 status: "pending",7}).await?;89// Step 2: Execute task10let result = charge_stripe_api(payment_data).await?;1112// Step 3: Update journal (durable)13journal.update(5, JournalUpdate {14 result: result,15 status: "completed",16}).await?;1718// If crash happens between step 2 and 3:19// - Journal shows "pending"20// - On restart, can retry or check if it actually succeeded21// - Never lose track of what was attemptedCompare to non-WAL approach:
1// ❌ Without WAL2let result = charge_stripe_api(payment_data).await?; // ← CRASH HERE = lost money3db.save_payment(result).await?;If crash happens after charging but before saving, the payment is lost forever.
With Orcher's WAL:
1// ✓ With WAL2journal.append("charge_stripe", payment_data).await?; // ← Intent recorded3let result = charge_stripe_api(payment_data).await?; // ← CRASH HERE = can recover4journal.complete("charge_stripe", result).await?;Journal shows we attempted to charge. On recovery, check Stripe API to see if charge succeeded, then continue.
Pattern 4: Long-Running Workflows with Human Approval
Some workflows need to pause and wait for human input before continuing. Orcher handles this with durable sleeps and external signals.
The Problem
1// Traditional approach - doesn't survive restarts2async function processExpenseReport(report: ExpenseReport) {3 await saveReport(report);4 await sendApprovalEmail(report.managerId);56 // ❌ Can't wait here - server will restart, lose state7 await waitForApproval(); // How long? What if server crashes?89 await processPayment(report);10}The Orcher Solution
1#[workflow(name = "expense_approval")]2pub async fn expense_approval_workflow(3 ctx: WorkflowContext,4 report: ExpenseReport,5) -> Result<String> {6 // Step 1: Save report7 let report_id = ctx.execute_task(save_report, report.clone()).await?;89 // Step 2: Send approval request10 ctx.execute_task(send_approval_email, (report_id.clone(), report.manager_id.clone())).await?;1112 // Step 3: Wait for approval event (durable wait) - returns the deserialized payload13 let approval_result: ApprovalDecision = ctx.wait_for_event("approval_decision").await?;1415 // Step 4: Process based on approval16 match approval_result.approved {17 true => {18 let payment_id = ctx.execute_task(process_payment, report.amount).await?;19 ctx.execute_task(notify_employee, (report.employee_id, "approved", payment_id)).await?;20 Ok(format!("Approved and paid: {}", payment_id))21 }22 false => {23 ctx.execute_task(notify_employee, (report.employee_id, "rejected", approval_result.reason)).await?;24 Err(anyhow::anyhow!("Report rejected: {}", approval_result.reason))25 }26 }27}2829// External API to send approval decision (via a workflow handle)30async fn approve_expense(31 orcher: Client,32 workflow_id: String,33 approved: bool,34 reason: Option<String>,35) -> Result<()> {36 let handle = orcher.get_workflow_handle(workflow_id).await?;37 handle.send_event(38 "approval_decision",39 ApprovalDecision { approved, reason },40 ).await?;4142 Ok(())43}How it works:
- Workflow starts, saves report, sends email
- Workflow pauses at
wait_for_event()(worker is freed) - Days pass, servers restart, deployments happen (workflow survives)
- Manager clicks "Approve" in email → API call to
send_event() - Workflow resumes from exact position, processes payment
Key insight: The workflow can wait for hours, days, or weeks. It's not consuming resources—it's just a row in PostgreSQL waiting for a signal.
Pattern 5: Child Workflows and Composition
Break complex workflows into smaller, reusable pieces.
When to Use Child Workflows
Use child workflows when you have:
- Reusable business processes
- Long-running sub-processes that should be monitored independently
- Need for parallel execution of complex workflows
- Different retry policies for sub-processes
1// Parent workflow: Onboard new customer2#[workflow(name = "onboard_customer")]3pub async fn onboard_customer(4 ctx: WorkflowContext,5 customer: NewCustomer,6) -> Result<OnboardingResult> {7 // Step 1: Create account8 let account_id = ctx.execute_task(create_account, customer.clone()).await?;910 // Step 2: Start child workflows by name (each returns a handle immediately)11 let billing_handle = ctx.start_child_workflow(12 "setup_billing",13 BillingSetup {14 account_id: account_id.clone(),15 plan: customer.plan,16 payment_method: customer.payment_method,17 },18 ).await?;19 let provisioning_handle = ctx.start_child_workflow(20 "provision_resources",21 ResourceRequest {22 account_id: account_id.clone(),23 region: customer.region,24 tier: customer.tier,25 },26 ).await?;27 let notification_handle = ctx.start_child_workflow(28 "welcome_sequence",29 WelcomeSequence {30 account_id: account_id.clone(),31 email: customer.email,32 },33 ).await?;3435 // Await all three results in parallel36 let (billing_result, provisioning_result, notification_result): (37 BillingSetup,38 ProvisioningResult,39 NotificationResult,40 ) = tokio::try_join!(41 billing_handle.result(),42 provisioning_handle.result(),43 notification_handle.result(),44 )?;4546 // Step 3: Finalize onboarding47 ctx.execute_task(activate_account, account_id.clone()).await?;4849 Ok(OnboardingResult {50 account_id,51 billing_id: billing_result.billing_id,52 resources: provisioning_result.resources,53 welcome_emails_sent: notification_result.emails_sent,54 })55}5657// Child workflow: Setup billing (independent, reusable)58#[workflow(name = "setup_billing")]59pub async fn setup_billing_workflow(60 ctx: WorkflowContext,61 setup: BillingSetup,62) -> Result<BillingSetup> {63 // Create Stripe customer64 let stripe_customer_id = ctx.execute_task(create_stripe_customer, setup.account_id.clone()).await?;6566 // Attach payment method67 ctx.execute_task(attach_payment_method, (stripe_customer_id.clone(), setup.payment_method)).await?;6869 // Create subscription70 let subscription_id = ctx.execute_task(create_subscription, (stripe_customer_id, setup.plan)).await?;7172 // Save billing info73 let billing_id = ctx.execute_task(save_billing_info, (setup.account_id, subscription_id)).await?;7475 Ok(BillingSetup { billing_id })76}Benefits of child workflows:
- Composition: Build complex flows from simple, reusable pieces
- Independent monitoring: Each child workflow has its own ID, status, and logs
- Parallel execution: Multiple child workflows run concurrently
- Isolated retries: Child workflow failures don't affect siblings
- Better testing: Test child workflows independently
Pattern 6: Advanced Error Handling Strategies
Retry Policies Per Task
Different tasks need different retry strategies:
1// Retry policy is declared per task on the #[task] macro. Intervals are in2// whole seconds; a backoff_coefficient of 1.0 means fixed (non-growing) delays.34// Task 1: Quick retries for transient network issues5#[task(retry_policy(max_attempts = 5, initial_interval = 1, max_interval = 5, backoff_coefficient = 2.0))]6async fn fetch_data(url: String) -> Result<Data> { /* ... */ }78// Task 2: Slow retries for rate-limited APIs (fixed 60s between attempts)9#[task(retry_policy(max_attempts = 10, initial_interval = 60, max_interval = 60, backoff_coefficient = 1.0))]10async fn enrich_data(data: Data) -> Result<Data> { /* ... */ }1112// Task 3: No retries for validation (fail fast)13#[task(retry_policy(max_attempts = 1))]14async fn validate_data(data: Data) -> Result<Data> { /* ... */ }1516#[workflow(name = "complex_workflow")]17pub async fn complex_workflow(ctx: WorkflowContext, input: Input) -> Result<Output> {18 let data = ctx.execute_task(fetch_data, input.url).await?;19 let enriched = ctx.execute_task(enrich_data, data).await?;20 let validated = ctx.execute_task(validate_data, enriched).await?;2122 Ok(validated)23}Error Classification and Handling
Distinguish between different error types and handle them appropriately:
1use anyhow::Context;23#[derive(Debug, thiserror::Error)]4pub enum WorkflowError {5 #[error("Transient error: {0}")]6 Transient(String),78 #[error("Permanent error: {0}")]9 Permanent(String),1011 #[error("Business rule violation: {0}")]12 BusinessRule(String),13}1415#[workflow(name = "resilient_payment")]16pub async fn resilient_payment_workflow(17 ctx: WorkflowContext,18 payment: Payment,19) -> Result<Receipt> {20 // Validate input (permanent failure if invalid)21 if payment.amount <= 0 {22 return Err(WorkflowError::Permanent("Invalid amount".to_string()).into());23 }2425 // Check fraud (business rule)26 let fraud_score = ctx.execute_task(check_fraud, payment.user_id.clone()).await?;27 if fraud_score > 0.9 {28 return Err(WorkflowError::BusinessRule("High fraud risk".to_string()).into());29 }3031 // Charge payment (may be transient failure)32 let charge = match ctx.execute_task(charge_stripe, payment.clone()).await {33 Ok(charge) => charge,34 Err(e) if is_transient_error(&e) => {35 warn!("Transient payment error: {}", e);36 return Err(WorkflowError::Transient(e.to_string()).into());37 }38 Err(e) => {39 return Err(WorkflowError::Permanent(format!("Payment failed: {}", e)).into());40 }41 };4243 ctx.execute_task(save_payment, charge.id.clone())44 .await45 .context("Failed to save payment")?;4647 Ok(Receipt {48 charge_id: charge.id,49 amount: payment.amount,50 })51}5253fn is_transient_error(error: &anyhow::Error) -> bool {54 let error_str = error.to_string().to_lowercase();55 error_str.contains("timeout") ||56 error_str.contains("connection") ||57 error_str.contains("503") ||58 error_str.contains("rate limit")59}Circuit Breaker Pattern
Protect external services from being overwhelmed during outages:
1use std::sync::Arc;2use tokio::sync::RwLock;34pub struct CircuitBreaker {5 failure_threshold: u32,6 success_threshold: u32,7 timeout: Duration,8 state: Arc<RwLock<CircuitState>>,9}1011enum CircuitState {12 Closed { failures: u32 },13 Open { opened_at: Instant },14 HalfOpen { successes: u32 },15}1617impl CircuitBreaker {18 pub async fn call<F, T>(&self, f: F) -> Result<T>19 where20 F: FnOnce() -> Result<T>,21 {22 let mut state = self.state.write().await;2324 match *state {25 CircuitState::Open { opened_at } => {26 // Check if timeout elapsed27 if opened_at.elapsed() > self.timeout {28 *state = CircuitState::HalfOpen { successes: 0 };29 } else {30 return Err(anyhow::anyhow!("Circuit breaker is OPEN"));31 }32 }33 _ => {}34 }3536 drop(state); // Release lock before calling function3738 let result = f();3940 let mut state = self.state.write().await;41 match result {42 Ok(value) => {43 match *state {44 CircuitState::HalfOpen { successes } => {45 if successes + 1 >= self.success_threshold {46 *state = CircuitState::Closed { failures: 0 };47 } else {48 *state = CircuitState::HalfOpen { successes: successes + 1 };49 }50 }51 CircuitState::Closed { .. } => {52 *state = CircuitState::Closed { failures: 0 };53 }54 _ => {}55 }56 Ok(value)57 }58 Err(e) => {59 match *state {60 CircuitState::Closed { failures } => {61 if failures + 1 >= self.failure_threshold {62 *state = CircuitState::Open { opened_at: Instant::now() };63 } else {64 *state = CircuitState::Closed { failures: failures + 1 };65 }66 }67 CircuitState::HalfOpen { .. } => {68 *state = CircuitState::Open { opened_at: Instant::now() };69 }70 _ => {}71 }72 Err(e)73 }74 }75 }76}7778// Use in task79#[task]80async fn call_external_service(data: String) -> Result<String> {81 static CIRCUIT_BREAKER: Lazy<CircuitBreaker> = Lazy::new(|| {82 CircuitBreaker::new(5, 2, Duration::from_secs(60))83 });8485 CIRCUIT_BREAKER.call(|| {86 // Make actual API call87 external_api::process(data)88 }).await89}Pattern 7: Scheduled and Recurring Workflows
Orcher has no separate cron scheduler process. Instead, model recurrence as a long-running workflow that does its work, sleeps durably until the next run, then restarts with a fresh history. The durable workflow is the schedule — it survives restarts and frees the worker while it waits:
1use std::time::Duration;23#[workflow(name = "daily_report")]4pub async fn daily_report_workflow(5 ctx: WorkflowContext,6 input: DailyReportInput,7) -> Result<()> {8 // Generate report9 let report_data = ctx.execute_task(generate_report, input.report_type.clone()).await?;1011 // Send to stakeholders12 ctx.execute_task(send_report_email, report_data).await?;1314 // Durable sleep until the next run (survives restarts, frees the worker)15 ctx.sleep(Duration::from_secs(24 * 60 * 60)).await?;1617 // Restart with a fresh history for the next cycle18 ctx.restart_fresh_with(input, None)19}Next Steps
- Part 3: Production Excellence — Monitoring, testing, and deployment
- Back to Overview — Learning path overview