Core Concepts
Understanding Durable Execution
Let's build a mental model of how durable execution works. This is the foundational concept that makes Orcher different from traditional backend architectures.
What Is Durable Execution?
Durable execution means your code execution survives process crashes, network failures, and machine restarts. It's like having a built-in "save and resume" system for your business logic.
In traditional programming, when your process crashes, you lose:
- All in-memory state
- The exact position in your code
- Which operations completed vs. failed
- Any partial results
With durable execution, Orcher automatically preserves:
- Execution position: Which line of code was executing
- Variable state: Values of all task results
- Decision history: Which branches (if/else) were taken
- Retry state: How many attempts have been made
This is achieved through automatic journaling - every significant operation is recorded to durable storage (PostgreSQL) before, during, and after execution.
The Problem: Partial Failures
Traditional code executes top-to-bottom. When it crashes, you lose everything:
1async function processOrder(order) {2 const charge = await stripe.charges.create({ amount: order.total }); // ← Succeeds3 await db.orders.update({ status: 'paid' }); // ← Succeeds4 await emailService.send({ template: 'receipt' }); // ← CRASHES5 await inventory.reserve(order.items); // ← Never runs6 await analytics.track('order_completed'); // ← Never runs7}What happened?
- Customer was charged ✓
- Database updated ✓
- Email failed ✗
- Inventory not reserved ✗
- Analytics not tracked ✗
What do you do?
- Manually check Stripe for the charge
- Manually reserve inventory
- Manually send the email
- Manually track analytics
- Hope you didn't miss anything
This is called a partial failure, and it's the source of countless production incidents.
The Solution: Automatic Journaling
Orcher wraps each step in a ctx.execute_task() call. This does three things:
- Before execution: Checks if this step was already completed
- During execution: Runs your code
- After execution: Saves the result to the journal
Here's what the journal looks like in PostgreSQL:
1SELECT * FROM workflow_execution_journal WHERE workflow_id = 'wf_abc123';23| step_id | task_name | status | result | created_at |4|---------|----------------|-----------|---------------------------|------------|5| 1 | charge_card | completed | {"id": "ch_xyz", ...} | 10:00:00 |6| 2 | update_db | completed | null | 10:00:01 |7| 3 | send_email | failed | "SMTP timeout" | 10:00:02 |On retry, Orcher reads the journal:
- Step 1: Already completed → Return cached result
- Step 2: Already completed → Return cached result
- Step 3: Failed → Try again
- Step 4+: Not in journal → Execute normally
This is called deterministic replay.
Execution Guarantees
Orcher provides three execution guarantees:
1. A Completed Task Never Runs Again on Replay
Once a task's result is journaled, replay reuses that result instead of running the task again. A task that has not recorded its result yet can run again, if its worker dies or it times out, so make its side effects idempotent.
1// On replay, a completed charge_stripe returns its journaled result2let charge = ctx.execute_task(charge_stripe, payment).await?; // ← not run again once completed3ctx.execute_task(update_db, charge.id).await?; // ← not run again once completed2. Deterministic Replay
When a workflow resumes after a crash, it follows the exact same execution path. Decisions made in previous runs are preserved.
1let fraud_score = ctx.execute_task(check_fraud, user_id).await?;23// On replay, this if-statement will take the SAME branch as the original run4if fraud_score > 0.8 {5 return Err(anyhow::anyhow!("Fraud detected"));6}This prevents bizarre bugs where a retry takes a different code path and produces inconsistent results.
3. Crash Recovery with Progress Preservation
If your workflow crashes at step 5 of 10, the retry starts at step 5. Steps 1-4 return their cached results instantly.
1// Original run crashes here ↓2let step1 = ctx.execute_task(step_1, data).await?; // ✓ Completed3let step2 = ctx.execute_task(step_2, step1).await?; // ✓ Completed4let step3 = ctx.execute_task(step_3, step2).await?; // ✗ CRASH56// Retry run:7let step1 = ctx.execute_task(step_1, data).await?; // ← Returns cached result (instant)8let step2 = ctx.execute_task(step_2, step1).await?; // ← Returns cached result (instant)9let step3 = ctx.execute_task(step_3, step2).await?; // ← Executes for realWhy These Guarantees Matter
Traditional distributed systems achieve these guarantees through complex infrastructure:
- Two-phase commit protocols
- Distributed transaction coordinators
- Consensus algorithms (Paxos, Raft)
- Complex state machines
Orcher achieves them through a simple journal in PostgreSQL. The complexity is hidden in the runtime, not your code.
Key Principle: Idempotency
Because Orcher may replay your workflow, each task must be idempotent: running it twice produces the same result.
What is Idempotency?
An operation is idempotent if executing it multiple times has the same effect as executing it once. Think of it like pressing an elevator button - pressing it 10 times doesn't call 10 elevators.
Why Idempotency Matters in Orcher
Even though Orcher never repeats a completed step within a workflow, your tasks might be called more than once in other scenarios:
- Workflow replays during development/testing
- Manual workflow retries from the admin UI
- Race conditions during deployments
- External retries (e.g., user clicking "submit" twice)
Making your tasks idempotent provides defense in depth.
Good (idempotent):
1#[task]2async fn update_user_balance(user_id: String, amount: i64) -> Result<()> {3 // Uses database transaction - idempotent4 db.query(5 "UPDATE users SET balance = balance + $1 WHERE id = $2",6 amount, user_id7 ).await?;8 Ok(())9}Bad (not idempotent):
1#[task]2async fn update_user_balance(user_id: String, amount: i64) -> Result<()> {3 // Fetches, modifies, writes - NOT idempotent if replayed4 let user = db.get_user(user_id).await?;5 user.balance += amount; // ← Could double-add on replay!6 db.save_user(user).await?;7 Ok(())8}Why this matters: If your task is idempotent, retries and replays have the same effect as running it once.
Deep Dive: The Journal Structure
Let's look at what Orcher actually stores in PostgreSQL. Understanding this helps you reason about performance and debugging.
Journal Table Schema:
1CREATE TABLE workflow_execution_journal (2 id UUID PRIMARY KEY,3 workflow_id UUID NOT NULL,4 step_id INTEGER NOT NULL,5 task_name TEXT NOT NULL,6 input JSONB NOT NULL,7 result JSONB,8 error TEXT,9 status TEXT NOT NULL, -- 'pending', 'completed', 'failed'10 attempt INTEGER NOT NULL DEFAULT 1,11 created_at TIMESTAMPTZ NOT NULL,12 completed_at TIMESTAMPTZ,1314 UNIQUE(workflow_id, step_id)15);Example Journal Entries:
1-- Workflow: process_payment (wf_abc123)2SELECT step_id, task_name, status, result FROM workflow_execution_journal3WHERE workflow_id = 'wf_abc123'4ORDER BY step_id;56| step_id | task_name | status | result |7|---------|-----------------|-----------|-------------------------------------------|8| 1 | validate_input | completed | {"valid": true} |9| 2 | check_fraud | completed | {"score": 0.12, "risk": "low"} |10| 3 | charge_stripe | completed | {"id": "ch_xyz", "amount": 5000, ...} |11| 4 | save_to_db | completed | {"payment_id": "pay_123"} |12| 5 | send_email | failed | null (error: "SMTP timeout") |How Orcher Uses the Journal:
- Before task execution: Check if
(workflow_id, step_id)exists - If exists and completed: Return
resultcolumn (no execution) - If not exists: Execute task and INSERT new row
- If exists and failed: Re-execute task and UPDATE row
Performance Implications:
- Each task execution = 1 SELECT + 1 INSERT/UPDATE query
- Journal queries use a unique index (very fast)
- Cached results return in ~1ms
- Trade-off: Durability for slight overhead
Debugging with the Journal:
1-- Find all failed workflows2SELECT workflow_id, task_name, error3FROM workflow_execution_journal4WHERE status = 'failed';56-- See what a workflow did before crashing7SELECT task_name, result, completed_at8FROM workflow_execution_journal9WHERE workflow_id = 'wf_abc123'10ORDER BY step_id;1112-- Find slow tasks13SELECT task_name, AVG(EXTRACT(EPOCH FROM (completed_at - created_at))) as avg_seconds14FROM workflow_execution_journal15WHERE status = 'completed'16GROUP BY task_name17ORDER BY avg_seconds DESC;The Workflow Context
The WorkflowContext is your interface to Orcher's durability system. Think of it as a "time-traveling executor" that knows which steps have already run.
The Context API:
The WorkflowContext provides several key methods:
1pub struct WorkflowContext {2 // Execute a task durably (most common)3 pub async fn execute_task<F, I, O>(&self, task_fn: F, input: I) -> Result<O>;45 // Start a child workflow6 pub async fn start_child_workflow<F, I, O>(&self, workflow_fn: F, input: I) -> Result<O>;78 // Sleep durably (survives crashes)9 pub async fn sleep(&self, duration: Duration) -> Result<()>;1011 // Wait for an external event/signal12 pub async fn wait_for_event(&self, event_name: &str) -> Result<Vec<u8>>;1314 // Get workflow metadata15 pub fn workflow_id(&self) -> &str;16 pub fn run_id(&self) -> &str;17 pub fn attempt(&self) -> u32;18}1#[workflow(name = "my_workflow")]2pub async fn my_workflow(3 ctx: WorkflowContext, // ← Your durability interface4 input: MyInput,5) -> Result<MyOutput> {6 // Use ctx.execute_task() for durable steps - pass the task function directly7 let result = ctx.execute_task(my_task, data).await?;8 Ok(result)9}What ctx.execute_task() does:
- Generates a deterministic task ID from the task name and workflow position
- Checks the journal for an existing result
- If found: Returns the cached result instantly (no execution)
- If not found: Executes the task and journals the result
- On failure: Throws an error (workflow can be retried)
Important: Task results must be serializable (JSON). Orcher stores them in PostgreSQL.
Workflows vs Tasks
Understanding the difference is crucial:
Workflow = The orchestration logic
- Defines the sequence of steps
- Makes decisions (if/else, loops)
- Handles errors and retries
- Coordinates multiple tasks
- Can call other workflows (child workflows)
Task = A single unit of work
- Performs one specific action
- Should be idempotent
- Returns a serializable result
- Can be retried independently
Example:
1// This is a WORKFLOW - orchestrates tasks2#[workflow(name = "process_payment")]3pub async fn process_payment(4 ctx: WorkflowContext,5 payment: Payment,6) -> Result<Receipt> {7 // Decision logic lives in the workflow8 let fraud_score = ctx.execute_task(check_fraud, payment.user_id).await?;910 if fraud_score > 0.8 {11 // Workflow decides to reject12 return Err(anyhow::anyhow!("Payment flagged"));13 }1415 // Workflow orchestrates multiple tasks16 let charge = ctx.execute_task(charge_stripe, payment).await?;17 ctx.execute_task(save_to_db, charge.id).await?;18 ctx.execute_task(send_email, charge.receipt_url).await?;1920 Ok(Receipt { charge_id: charge.id })21}2223// This is a TASK - does one thing24#[task]25async fn check_fraud(user_id: String) -> Result<f64> {26 let response = fraud_api::check(user_id).await?;27 Ok(response.score)28}Consistency Models Explained
Understanding Orcher's consistency guarantees helps you build correct distributed systems.
Workflow Isolation
Each workflow execution is isolated - they don't see each other's uncommitted state.
1// Workflow A2let balance = ctx.execute_task(get_balance, user_id).await?; // Returns 1003ctx.execute_task(add_credits, (user_id, 50)).await?; // Balance now 15045// Workflow B (running concurrently for same user)6let balance = ctx.execute_task(get_balance, user_id).await?; // Still returns 100!7ctx.execute_task(add_credits, (user_id, 25)).await?; // Balance now 125 (lost update!)Solution: Use database-level locking or transactions within your tasks:
1#[task]2async fn add_credits_safely(user_id: String, amount: i64) -> Result<()> {3 let mut tx = db.begin().await?;45 // SELECT FOR UPDATE locks the row6 tx.query("SELECT balance FROM users WHERE id = $1 FOR UPDATE", user_id).await?;7 tx.query("UPDATE users SET balance = balance + $1 WHERE id = $2", amount, user_id).await?;89 tx.commit().await?;10 Ok(())11}Key Insight: Orcher provides workflow-level consistency, but you still need database transactions for data-level consistency.
Once per Step, at Least Once per Task
A workflow never re-runs a step whose result is journaled, but a task itself runs at least once: if its worker dies before reporting, or a timeout fires, it runs again, so its side effects must be idempotent.
What does this mean?
1#[task]2async fn send_notification(user_id: String, message: String) -> Result<()> {3 // This code can run more than once if its worker dies before it reports...4 let response = http_client.post("https://api.pushservice.com/send")5 .json(&json!({6 "user_id": user_id,7 "message": message8 }))9 .send()10 .await?;1112 // ...but if this task succeeds, then workflow fails later,13 // the notification was already sent (can't be un-sent).14 Ok(())15}The task executes once, but the side effect persists. This is unavoidable in distributed systems (you can't "unsend" a push notification).
Solution: Make external APIs idempotent by including an idempotency key:
1#[task]2async fn send_notification(workflow_id: String, user_id: String, message: String) -> Result<()> {3 let idempotency_key = format!("{}-notification", workflow_id);4 let response = http_client.post("https://api.pushservice.com/send")5 .json(&json!({6 "idempotency_key": idempotency_key, // Key based on workflow7 "user_id": user_id,8 "message": message9 }))10 .send()11 .await?;1213 Ok(())14}Now if the task is ever re-executed, the external API deduplicates using the idempotency key.
Failure Recovery Patterns
Let's explore how Orcher handles different failure scenarios.
Pattern 1: Transient Failures (Network Blips)
1#[task]2async fn call_external_api(data: String) -> Result<String> {3 // Temporary network issue, DNS failure, etc.4 let response = http_client.get("https://api.example.com/data")5 .send()6 .await?; // ← May fail temporarily78 Ok(response.text().await?)9}Orcher's Behavior:
- Task fails, throws error
- Workflow pauses
- Retry policy kicks in (exponential backoff)
- Task re-executes after delay
- Eventually succeeds
Pattern 2: Permanent Failures (Invalid Input)
1#[task]2async fn process_payment(amount: i64) -> Result<String> {3 if amount <= 0 {4 // This will never succeed, no matter how many retries5 return Err(anyhow::anyhow!("Invalid amount"));6 }78 // ... process payment9}Orcher's Behavior:
- Task fails immediately
- Workflow fails
- Retries happen but keep failing
- Eventually hits max retry limit
- Workflow moves to "failed" state
- Requires manual intervention or compensation
Pattern 3: Partial Progress (Long-Running Tasks)
1#[task]2async fn process_large_file(file_url: String) -> Result<String> {3 // Download 1GB file4 let data = download_file(file_url).await?; // ← Crashes here56 // Transform data (expensive)7 let transformed = transform_data(data).await?;89 // Upload result10 let result_url = upload_file(transformed).await?;1112 Ok(result_url)13}Problem: If crash happens after download, retry downloads again (wastes time/bandwidth).
Solution: Break into smaller tasks:
1#[workflow]2pub async fn process_large_file_workflow(ctx: WorkflowContext, file_url: String) -> Result<String> {3 // Each task is a checkpoint4 let downloaded_path = ctx.execute_task(download, file_url).await?;5 let transformed_path = ctx.execute_task(transform, downloaded_path).await?;6 let result_url = ctx.execute_task(upload, transformed_path).await?;78 Ok(result_url)9}Now if crash happens after download, the retry skips re-downloading.
Architecture Deep Dive
Let's understand how Orcher fits into your system and the engineering decisions behind its design.
Component Overview
Orcher uses a client-server architecture: an Orchestrator server handles workflow state, journaling, and queue management, while SDK Workers connect to it via gRPC to execute your workflow and task code. The architecture has three layers:
Workers are stateless and disposable; all durability lives in the Orchestrator's Postgres. No message broker.
Layer 1 — Orchestrator Server
- gRPC API (port 50051): Accepts workflow start requests, polls from workers, and delivers task assignments
- HTTP API (port 8001): Provides management endpoints, health checks, and observability
- Workflow Engine: Manages workflow state, journaling, deterministic replay, retry policies, and queue dispatch
The Orchestrator server is a standalone process that you deploy alongside your application infrastructure.
Layer 2 — SDK Workers (your application process)
- Worker: Connects to the Orchestrator server via gRPC and registers your workflows and tasks
- Worker Pool: Long-polls the Orchestrator for pending workflows and tasks, then executes them locally
- API Server (optional): Your HTTP server can use the Orcher client SDK to start workflows by calling the Orchestrator's gRPC API
Workers run your business logic. They do not store workflow state themselves -- all state lives in the Orchestrator's database.
Layer 3 — PostgreSQL
- Workflow journal (execution history)
- Queue table (pending workflows)
- Task execution tracking
- Managed entirely by the Orchestrator server
Key Points:
- Client-Server Model: The Orchestrator server is a separate process; SDK Workers connect to it over the network
- Single Database: The Orchestrator stores everything in PostgreSQL (simpler operations)
- Worker Pattern: Your application runs as one or more SDK Workers that long-poll the Orchestrator for work
- No Message Queue: PostgreSQL acts as the queue (SKIP LOCKED queries), managed by the Orchestrator
Execution Flow
Let's trace a request through the system — from the HTTP call that starts a workflow to its completion or retry:
Step 1: API receives request
1// API handler2async fn create_payment(3 Extension(orcher): Extension<Orcher>,4 Json(payment): Json<PaymentRequest>,5) -> Result<Json<Response>, StatusCode> {6 // Start workflow (non-blocking)7 let workflow_id = orcher8 .start_workflow("process_payment", payment)9 .await10 .map_err(|_| StatusCode::INTERNAL_SERVER_ERROR)?;1112 // Return immediately (workflow runs in background)13 Ok(Json(Response { workflow_id }))14}Step 2: Workflow queued
1-- Orchestrator writes to queue table2INSERT INTO workflow_queue (id, workflow_name, input, status, created_at)3VALUES ('wf_abc123', 'process_payment', '{"amount": 5000, ...}', 'pending', NOW());Step 3: Worker picks up workflow
1// SDK Worker long-polls the Orchestrator via gRPC2// When a workflow is available, the Orchestrator assigns it to the worker3// The worker executes the workflow function locallyStep 4: Workflow executes with journaling
1// Each execute_task() call journals to PostgreSQL via the Orchestrator2ctx.execute_task(charge_stripe, payment).await?;3// Orchestrator writes: INSERT INTO workflow_execution_journal (workflow_id, step_id, task_name, result, ...)45ctx.execute_task(update_db, charge_id).await?;6// Orchestrator writes: INSERT INTO workflow_execution_journal (workflow_id, step_id, task_name, result, ...)Step 5: Completion or retry
If successful:
1UPDATE workflow_queue SET status = 'completed', completed_at = NOW() WHERE id = 'wf_abc123';If failed:
1UPDATE workflow_queue2SET status = 'pending', retry_count = retry_count + 1, next_retry_at = NOW() + INTERVAL '30 seconds'3WHERE id = 'wf_abc123';Why This Architecture Matters
Single Point of Truth: Everything is in PostgreSQL
- Workflow state
- Execution history
- Queue management
- Application data
Benefits:
- Debugging: Query the journal table to see exactly what happened
- Consistency: Database transactions ensure atomic updates
- No clock skew: Single source of time
- Simple backups: One database to backup
- Easy local dev: Just PostgreSQL, no Redis/Kafka/etc.
Retry Policies and Error Handling
Orcher provides sophisticated retry mechanisms out of the box.
Default Retry Behavior
When a task fails, Orcher automatically retries with exponential backoff:
1Attempt 1: Immediate2Attempt 2: 1 second later3Attempt 3: 2 seconds later4Attempt 4: 4 seconds later5Attempt 5: 8 seconds later6Attempt 6: 16 seconds later7...8Max: 24 hours between retriesWhy exponential backoff?
- Transient errors (network blip) resolve quickly → fast retry
- Service outages need time to recover → don't overwhelm
- Cascading failures prevented by increasing delays
Custom Retry Policies
You can configure retries per task:
1#[task(2 retry_policy(3 max_attempts = 5,4 initial_interval = 1,5 max_interval = 60,6 backoff_coefficient = 2.0,7 )8)]9async fn call_external_api(data: Data) -> Result<Response> {10 // This task will retry up to 5 times with exponential backoff11 Ok(external_api::call(data).await?)12}Handling Different Error Types
Not all errors should retry:
1#[task]2async fn process_payment(payment: Payment) -> Result<Charge> {3 match stripe::charge(&payment).await {4 Ok(charge) => Ok(charge),56 // Transient error - will retry automatically7 Err(e) if e.is_network_error() => Err(e.into()),89 // Permanent error - don't retry10 Err(e) if e.is_card_declined() => {11 // Log and return a permanent failure12 log_payment_declined(&payment, &e).await?;13 Err(anyhow::anyhow!("Card declined: {}", e))14 }1516 Err(e) => Err(e.into()),17 }18}Best Practice: Distinguish between:
- Retriable errors: Network timeouts, 5xx responses, rate limits
- Permanent errors: Invalid input, 4xx responses, business rule violations
Compensating Actions
For multi-step workflows, you need to handle partial completion:
1#[workflow(name = "book_flight")]2pub async fn book_flight(ctx: WorkflowContext, booking: Booking) -> Result<Confirmation> {3 // Step 1: Reserve seat4 let reservation = ctx.execute_task(reserve_seat, booking.seat_id).await?;56 // Step 2: Charge payment7 let charge = match ctx.execute_task(charge_card, booking.payment).await {8 Ok(c) => c,9 Err(e) => {10 // Payment failed - compensate by releasing seat11 ctx.execute_task(release_seat, reservation.id).await?;12 return Err(e);13 }14 };1516 // Step 3: Send confirmation17 ctx.execute_task(send_confirmation, charge.receipt).await?;1819 Ok(Confirmation { booking_id: charge.id })20}This is the Saga pattern: Each step has a compensating action that undoes it.
Next Steps
- Part 2: Building Patterns -- Real-world patterns
- Back to Overview -- Learning path overview