Getting Started

Core Concepts


Understanding Durable Execution

Let's build a mental model of how durable execution works. This is the foundational concept that makes Orcher different from traditional backend architectures.

What Is Durable Execution?

Durable execution means your code execution survives process crashes, network failures, and machine restarts. It's like having a built-in "save and resume" system for your business logic.

In traditional programming, when your process crashes, you lose:

  • All in-memory state
  • The exact position in your code
  • Which operations completed vs. failed
  • Any partial results

With durable execution, Orcher automatically preserves:

  • Execution position: Which line of code was executing
  • Variable state: Values of all task results
  • Decision history: Which branches (if/else) were taken
  • Retry state: How many attempts have been made

This is achieved through automatic journaling - every significant operation is recorded to durable storage (PostgreSQL) before, during, and after execution.

The Problem: Partial Failures

Traditional code executes top-to-bottom. When it crashes, you lose everything:

1
async function processOrder(order) {
2
const charge = await stripe.charges.create({ amount: order.total }); // ← Succeeds
3
await db.orders.update({ status: 'paid' }); // ← Succeeds
4
await emailService.send({ template: 'receipt' }); // ← CRASHES
5
await inventory.reserve(order.items); // ← Never runs
6
await analytics.track('order_completed'); // ← Never runs
7
}

What happened?

  • Customer was charged ✓
  • Database updated ✓
  • Email failed ✗
  • Inventory not reserved ✗
  • Analytics not tracked ✗

What do you do?

  • Manually check Stripe for the charge
  • Manually reserve inventory
  • Manually send the email
  • Manually track analytics
  • Hope you didn't miss anything

This is called a partial failure, and it's the source of countless production incidents.

The Solution: Automatic Journaling

Orcher wraps each step in a ctx.execute_task() call. This does three things:

  1. Before execution: Checks if this step was already completed
  2. During execution: Runs your code
  3. After execution: Saves the result to the journal

Here's what the journal looks like in PostgreSQL:

1
SELECT * FROM workflow_execution_journal WHERE workflow_id = 'wf_abc123';
2
3
| step_id | task_name | status | result | created_at |
4
|---------|----------------|-----------|---------------------------|------------|
5
| 1 | charge_card | completed | {"id": "ch_xyz", ...} | 10:00:00 |
6
| 2 | update_db | completed | null | 10:00:01 |
7
| 3 | send_email | failed | "SMTP timeout" | 10:00:02 |

On retry, Orcher reads the journal:

  • Step 1: Already completed → Return cached result
  • Step 2: Already completed → Return cached result
  • Step 3: Failed → Try again
  • Step 4+: Not in journal → Execute normally

This is called deterministic replay.

Execution Guarantees

Orcher provides three execution guarantees:

1. A Completed Task Never Runs Again on Replay

Once a task's result is journaled, replay reuses that result instead of running the task again. A task that has not recorded its result yet can run again, if its worker dies or it times out, so make its side effects idempotent.

1
// On replay, a completed charge_stripe returns its journaled result
2
let charge = ctx.execute_task(charge_stripe, payment).await?; // ← not run again once completed
3
ctx.execute_task(update_db, charge.id).await?; // ← not run again once completed

2. Deterministic Replay

When a workflow resumes after a crash, it follows the exact same execution path. Decisions made in previous runs are preserved.

1
let fraud_score = ctx.execute_task(check_fraud, user_id).await?;
2
3
// On replay, this if-statement will take the SAME branch as the original run
4
if fraud_score > 0.8 {
5
return Err(anyhow::anyhow!("Fraud detected"));
6
}

This prevents bizarre bugs where a retry takes a different code path and produces inconsistent results.

3. Crash Recovery with Progress Preservation

If your workflow crashes at step 5 of 10, the retry starts at step 5. Steps 1-4 return their cached results instantly.

1
// Original run crashes here ↓
2
let step1 = ctx.execute_task(step_1, data).await?; // ✓ Completed
3
let step2 = ctx.execute_task(step_2, step1).await?; // ✓ Completed
4
let step3 = ctx.execute_task(step_3, step2).await?; // ✗ CRASH
5
6
// Retry run:
7
let step1 = ctx.execute_task(step_1, data).await?; // ← Returns cached result (instant)
8
let step2 = ctx.execute_task(step_2, step1).await?; // ← Returns cached result (instant)
9
let step3 = ctx.execute_task(step_3, step2).await?; // ← Executes for real

Why These Guarantees Matter

Traditional distributed systems achieve these guarantees through complex infrastructure:

  • Two-phase commit protocols
  • Distributed transaction coordinators
  • Consensus algorithms (Paxos, Raft)
  • Complex state machines

Orcher achieves them through a simple journal in PostgreSQL. The complexity is hidden in the runtime, not your code.

Key Principle: Idempotency

Because Orcher may replay your workflow, each task must be idempotent: running it twice produces the same result.

What is Idempotency?

An operation is idempotent if executing it multiple times has the same effect as executing it once. Think of it like pressing an elevator button - pressing it 10 times doesn't call 10 elevators.

Why Idempotency Matters in Orcher

Even though Orcher never repeats a completed step within a workflow, your tasks might be called more than once in other scenarios:

  • Workflow replays during development/testing
  • Manual workflow retries from the admin UI
  • Race conditions during deployments
  • External retries (e.g., user clicking "submit" twice)

Making your tasks idempotent provides defense in depth.

Good (idempotent):

1
#[task]
2
async fn update_user_balance(user_id: String, amount: i64) -> Result<()> {
3
// Uses database transaction - idempotent
4
db.query(
5
"UPDATE users SET balance = balance + $1 WHERE id = $2",
6
amount, user_id
7
).await?;
8
Ok(())
9
}

Bad (not idempotent):

1
#[task]
2
async fn update_user_balance(user_id: String, amount: i64) -> Result<()> {
3
// Fetches, modifies, writes - NOT idempotent if replayed
4
let user = db.get_user(user_id).await?;
5
user.balance += amount; // ← Could double-add on replay!
6
db.save_user(user).await?;
7
Ok(())
8
}

Why this matters: If your task is idempotent, retries and replays have the same effect as running it once.

Deep Dive: The Journal Structure

Let's look at what Orcher actually stores in PostgreSQL. Understanding this helps you reason about performance and debugging.

Journal Table Schema:

1
CREATE TABLE workflow_execution_journal (
2
id UUID PRIMARY KEY,
3
workflow_id UUID NOT NULL,
4
step_id INTEGER NOT NULL,
5
task_name TEXT NOT NULL,
6
input JSONB NOT NULL,
7
result JSONB,
8
error TEXT,
9
status TEXT NOT NULL, -- 'pending', 'completed', 'failed'
10
attempt INTEGER NOT NULL DEFAULT 1,
11
created_at TIMESTAMPTZ NOT NULL,
12
completed_at TIMESTAMPTZ,
13
14
UNIQUE(workflow_id, step_id)
15
);

Example Journal Entries:

1
-- Workflow: process_payment (wf_abc123)
2
SELECT step_id, task_name, status, result FROM workflow_execution_journal
3
WHERE workflow_id = 'wf_abc123'
4
ORDER BY step_id;
5
6
| step_id | task_name | status | result |
7
|---------|-----------------|-----------|-------------------------------------------|
8
| 1 | validate_input | completed | {"valid": true} |
9
| 2 | check_fraud | completed | {"score": 0.12, "risk": "low"} |
10
| 3 | charge_stripe | completed | {"id": "ch_xyz", "amount": 5000, ...} |
11
| 4 | save_to_db | completed | {"payment_id": "pay_123"} |
12
| 5 | send_email | failed | null (error: "SMTP timeout") |

How Orcher Uses the Journal:

  1. Before task execution: Check if (workflow_id, step_id) exists
  2. If exists and completed: Return result column (no execution)
  3. If not exists: Execute task and INSERT new row
  4. If exists and failed: Re-execute task and UPDATE row

Performance Implications:

  • Each task execution = 1 SELECT + 1 INSERT/UPDATE query
  • Journal queries use a unique index (very fast)
  • Cached results return in ~1ms
  • Trade-off: Durability for slight overhead

Debugging with the Journal:

1
-- Find all failed workflows
2
SELECT workflow_id, task_name, error
3
FROM workflow_execution_journal
4
WHERE status = 'failed';
5
6
-- See what a workflow did before crashing
7
SELECT task_name, result, completed_at
8
FROM workflow_execution_journal
9
WHERE workflow_id = 'wf_abc123'
10
ORDER BY step_id;
11
12
-- Find slow tasks
13
SELECT task_name, AVG(EXTRACT(EPOCH FROM (completed_at - created_at))) as avg_seconds
14
FROM workflow_execution_journal
15
WHERE status = 'completed'
16
GROUP BY task_name
17
ORDER BY avg_seconds DESC;

The Workflow Context

The WorkflowContext is your interface to Orcher's durability system. Think of it as a "time-traveling executor" that knows which steps have already run.

The Context API:

The WorkflowContext provides several key methods:

1
pub struct WorkflowContext {
2
// Execute a task durably (most common)
3
pub async fn execute_task<F, I, O>(&self, task_fn: F, input: I) -> Result<O>;
4
5
// Start a child workflow
6
pub async fn start_child_workflow<F, I, O>(&self, workflow_fn: F, input: I) -> Result<O>;
7
8
// Sleep durably (survives crashes)
9
pub async fn sleep(&self, duration: Duration) -> Result<()>;
10
11
// Wait for an external event/signal
12
pub async fn wait_for_event(&self, event_name: &str) -> Result<Vec<u8>>;
13
14
// Get workflow metadata
15
pub fn workflow_id(&self) -> &str;
16
pub fn run_id(&self) -> &str;
17
pub fn attempt(&self) -> u32;
18
}
1
#[workflow(name = "my_workflow")]
2
pub async fn my_workflow(
3
ctx: WorkflowContext, // ← Your durability interface
4
input: MyInput,
5
) -> Result<MyOutput> {
6
// Use ctx.execute_task() for durable steps - pass the task function directly
7
let result = ctx.execute_task(my_task, data).await?;
8
Ok(result)
9
}

What ctx.execute_task() does:

  1. Generates a deterministic task ID from the task name and workflow position
  2. Checks the journal for an existing result
  3. If found: Returns the cached result instantly (no execution)
  4. If not found: Executes the task and journals the result
  5. On failure: Throws an error (workflow can be retried)

Important: Task results must be serializable (JSON). Orcher stores them in PostgreSQL.

Workflows vs Tasks

Understanding the difference is crucial:

Workflow = The orchestration logic

  • Defines the sequence of steps
  • Makes decisions (if/else, loops)
  • Handles errors and retries
  • Coordinates multiple tasks
  • Can call other workflows (child workflows)

Task = A single unit of work

  • Performs one specific action
  • Should be idempotent
  • Returns a serializable result
  • Can be retried independently

Example:

1
// This is a WORKFLOW - orchestrates tasks
2
#[workflow(name = "process_payment")]
3
pub async fn process_payment(
4
ctx: WorkflowContext,
5
payment: Payment,
6
) -> Result<Receipt> {
7
// Decision logic lives in the workflow
8
let fraud_score = ctx.execute_task(check_fraud, payment.user_id).await?;
9
10
if fraud_score > 0.8 {
11
// Workflow decides to reject
12
return Err(anyhow::anyhow!("Payment flagged"));
13
}
14
15
// Workflow orchestrates multiple tasks
16
let charge = ctx.execute_task(charge_stripe, payment).await?;
17
ctx.execute_task(save_to_db, charge.id).await?;
18
ctx.execute_task(send_email, charge.receipt_url).await?;
19
20
Ok(Receipt { charge_id: charge.id })
21
}
22
23
// This is a TASK - does one thing
24
#[task]
25
async fn check_fraud(user_id: String) -> Result<f64> {
26
let response = fraud_api::check(user_id).await?;
27
Ok(response.score)
28
}

Consistency Models Explained

Understanding Orcher's consistency guarantees helps you build correct distributed systems.

Workflow Isolation

Each workflow execution is isolated - they don't see each other's uncommitted state.

1
// Workflow A
2
let balance = ctx.execute_task(get_balance, user_id).await?; // Returns 100
3
ctx.execute_task(add_credits, (user_id, 50)).await?; // Balance now 150
4
5
// Workflow B (running concurrently for same user)
6
let balance = ctx.execute_task(get_balance, user_id).await?; // Still returns 100!
7
ctx.execute_task(add_credits, (user_id, 25)).await?; // Balance now 125 (lost update!)

Solution: Use database-level locking or transactions within your tasks:

1
#[task]
2
async fn add_credits_safely(user_id: String, amount: i64) -> Result<()> {
3
let mut tx = db.begin().await?;
4
5
// SELECT FOR UPDATE locks the row
6
tx.query("SELECT balance FROM users WHERE id = $1 FOR UPDATE", user_id).await?;
7
tx.query("UPDATE users SET balance = balance + $1 WHERE id = $2", amount, user_id).await?;
8
9
tx.commit().await?;
10
Ok(())
11
}

Key Insight: Orcher provides workflow-level consistency, but you still need database transactions for data-level consistency.

Once per Step, at Least Once per Task

A workflow never re-runs a step whose result is journaled, but a task itself runs at least once: if its worker dies before reporting, or a timeout fires, it runs again, so its side effects must be idempotent.

What does this mean?

1
#[task]
2
async fn send_notification(user_id: String, message: String) -> Result<()> {
3
// This code can run more than once if its worker dies before it reports...
4
let response = http_client.post("https://api.pushservice.com/send")
5
.json(&json!({
6
"user_id": user_id,
7
"message": message
8
}))
9
.send()
10
.await?;
11
12
// ...but if this task succeeds, then workflow fails later,
13
// the notification was already sent (can't be un-sent).
14
Ok(())
15
}

The task executes once, but the side effect persists. This is unavoidable in distributed systems (you can't "unsend" a push notification).

Solution: Make external APIs idempotent by including an idempotency key:

1
#[task]
2
async fn send_notification(workflow_id: String, user_id: String, message: String) -> Result<()> {
3
let idempotency_key = format!("{}-notification", workflow_id);
4
let response = http_client.post("https://api.pushservice.com/send")
5
.json(&json!({
6
"idempotency_key": idempotency_key, // Key based on workflow
7
"user_id": user_id,
8
"message": message
9
}))
10
.send()
11
.await?;
12
13
Ok(())
14
}

Now if the task is ever re-executed, the external API deduplicates using the idempotency key.

Failure Recovery Patterns

Let's explore how Orcher handles different failure scenarios.

Pattern 1: Transient Failures (Network Blips)

1
#[task]
2
async fn call_external_api(data: String) -> Result<String> {
3
// Temporary network issue, DNS failure, etc.
4
let response = http_client.get("https://api.example.com/data")
5
.send()
6
.await?; // ← May fail temporarily
7
8
Ok(response.text().await?)
9
}

Orcher's Behavior:

  • Task fails, throws error
  • Workflow pauses
  • Retry policy kicks in (exponential backoff)
  • Task re-executes after delay
  • Eventually succeeds

Pattern 2: Permanent Failures (Invalid Input)

1
#[task]
2
async fn process_payment(amount: i64) -> Result<String> {
3
if amount <= 0 {
4
// This will never succeed, no matter how many retries
5
return Err(anyhow::anyhow!("Invalid amount"));
6
}
7
8
// ... process payment
9
}

Orcher's Behavior:

  • Task fails immediately
  • Workflow fails
  • Retries happen but keep failing
  • Eventually hits max retry limit
  • Workflow moves to "failed" state
  • Requires manual intervention or compensation

Pattern 3: Partial Progress (Long-Running Tasks)

1
#[task]
2
async fn process_large_file(file_url: String) -> Result<String> {
3
// Download 1GB file
4
let data = download_file(file_url).await?; // ← Crashes here
5
6
// Transform data (expensive)
7
let transformed = transform_data(data).await?;
8
9
// Upload result
10
let result_url = upload_file(transformed).await?;
11
12
Ok(result_url)
13
}

Problem: If crash happens after download, retry downloads again (wastes time/bandwidth).

Solution: Break into smaller tasks:

1
#[workflow]
2
pub async fn process_large_file_workflow(ctx: WorkflowContext, file_url: String) -> Result<String> {
3
// Each task is a checkpoint
4
let downloaded_path = ctx.execute_task(download, file_url).await?;
5
let transformed_path = ctx.execute_task(transform, downloaded_path).await?;
6
let result_url = ctx.execute_task(upload, transformed_path).await?;
7
8
Ok(result_url)
9
}

Now if crash happens after download, the retry skips re-downloading.

Architecture Deep Dive

Let's understand how Orcher fits into your system and the engineering decisions behind its design.

Component Overview

Orcher uses a client-server architecture: an Orchestrator server handles workflow state, journaling, and queue management, while SDK Workers connect to it via gRPC to execute your workflow and task code. The architecture has three layers:

SDK Workers
your app process — runs workflows & tasks, holds no state
gRPC :50051 — long-poll for work
Orchestrator Server
workflow engine · journaling · retries · queue dispatch
gRPC :50051 · HTTP :8001
reads & writes
PostgreSQL
journal · queue (SKIP LOCKED) · app data — one database

Workers are stateless and disposable; all durability lives in the Orchestrator's Postgres. No message broker.

Layer 1 — Orchestrator Server

  • gRPC API (port 50051): Accepts workflow start requests, polls from workers, and delivers task assignments
  • HTTP API (port 8001): Provides management endpoints, health checks, and observability
  • Workflow Engine: Manages workflow state, journaling, deterministic replay, retry policies, and queue dispatch

The Orchestrator server is a standalone process that you deploy alongside your application infrastructure.

Layer 2 — SDK Workers (your application process)

  • Worker: Connects to the Orchestrator server via gRPC and registers your workflows and tasks
  • Worker Pool: Long-polls the Orchestrator for pending workflows and tasks, then executes them locally
  • API Server (optional): Your HTTP server can use the Orcher client SDK to start workflows by calling the Orchestrator's gRPC API

Workers run your business logic. They do not store workflow state themselves -- all state lives in the Orchestrator's database.

Layer 3 — PostgreSQL

  • Workflow journal (execution history)
  • Queue table (pending workflows)
  • Task execution tracking
  • Managed entirely by the Orchestrator server

Key Points:

  1. Client-Server Model: The Orchestrator server is a separate process; SDK Workers connect to it over the network
  2. Single Database: The Orchestrator stores everything in PostgreSQL (simpler operations)
  3. Worker Pattern: Your application runs as one or more SDK Workers that long-poll the Orchestrator for work
  4. No Message Queue: PostgreSQL acts as the queue (SKIP LOCKED queries), managed by the Orchestrator

Execution Flow

Let's trace a request through the system — from the HTTP call that starts a workflow to its completion or retry:

A Request's Lifecycle
API
start_workflownon-blocking — returns a workflow_id immediately
Orchestrator
enqueuerow written to workflow_queue as pending
Worker
long-poll → assignedpicks up the workflow, runs it locally
Worker
execute_task × Neach result journaled to Postgres via the Orchestrator
Orchestrator
complete / retrymark completed — or re-queue as pending with a backoff
The API returns in milliseconds; the workflow runs durably in the background.

Step 1: API receives request

1
// API handler
2
async fn create_payment(
3
Extension(orcher): Extension<Orcher>,
4
Json(payment): Json<PaymentRequest>,
5
) -> Result<Json<Response>, StatusCode> {
6
// Start workflow (non-blocking)
7
let workflow_id = orcher
8
.start_workflow("process_payment", payment)
9
.await
10
.map_err(|_| StatusCode::INTERNAL_SERVER_ERROR)?;
11
12
// Return immediately (workflow runs in background)
13
Ok(Json(Response { workflow_id }))
14
}

Step 2: Workflow queued

1
-- Orchestrator writes to queue table
2
INSERT INTO workflow_queue (id, workflow_name, input, status, created_at)
3
VALUES ('wf_abc123', 'process_payment', '{"amount": 5000, ...}', 'pending', NOW());

Step 3: Worker picks up workflow

1
// SDK Worker long-polls the Orchestrator via gRPC
2
// When a workflow is available, the Orchestrator assigns it to the worker
3
// The worker executes the workflow function locally

Step 4: Workflow executes with journaling

1
// Each execute_task() call journals to PostgreSQL via the Orchestrator
2
ctx.execute_task(charge_stripe, payment).await?;
3
// Orchestrator writes: INSERT INTO workflow_execution_journal (workflow_id, step_id, task_name, result, ...)
4
5
ctx.execute_task(update_db, charge_id).await?;
6
// Orchestrator writes: INSERT INTO workflow_execution_journal (workflow_id, step_id, task_name, result, ...)

Step 5: Completion or retry

If successful:

1
UPDATE workflow_queue SET status = 'completed', completed_at = NOW() WHERE id = 'wf_abc123';

If failed:

1
UPDATE workflow_queue
2
SET status = 'pending', retry_count = retry_count + 1, next_retry_at = NOW() + INTERVAL '30 seconds'
3
WHERE id = 'wf_abc123';

Why This Architecture Matters

Single Point of Truth: Everything is in PostgreSQL

  • Workflow state
  • Execution history
  • Queue management
  • Application data

Benefits:

  • Debugging: Query the journal table to see exactly what happened
  • Consistency: Database transactions ensure atomic updates
  • No clock skew: Single source of time
  • Simple backups: One database to backup
  • Easy local dev: Just PostgreSQL, no Redis/Kafka/etc.

Retry Policies and Error Handling

Orcher provides sophisticated retry mechanisms out of the box.

Default Retry Behavior

When a task fails, Orcher automatically retries with exponential backoff:

1
Attempt 1: Immediate
2
Attempt 2: 1 second later
3
Attempt 3: 2 seconds later
4
Attempt 4: 4 seconds later
5
Attempt 5: 8 seconds later
6
Attempt 6: 16 seconds later
7
...
8
Max: 24 hours between retries

Why exponential backoff?

  • Transient errors (network blip) resolve quickly → fast retry
  • Service outages need time to recover → don't overwhelm
  • Cascading failures prevented by increasing delays

Custom Retry Policies

You can configure retries per task:

1
#[task(
2
retry_policy(
3
max_attempts = 5,
4
initial_interval = 1,
5
max_interval = 60,
6
backoff_coefficient = 2.0,
7
)
8
)]
9
async fn call_external_api(data: Data) -> Result<Response> {
10
// This task will retry up to 5 times with exponential backoff
11
Ok(external_api::call(data).await?)
12
}

Handling Different Error Types

Not all errors should retry:

1
#[task]
2
async fn process_payment(payment: Payment) -> Result<Charge> {
3
match stripe::charge(&payment).await {
4
Ok(charge) => Ok(charge),
5
6
// Transient error - will retry automatically
7
Err(e) if e.is_network_error() => Err(e.into()),
8
9
// Permanent error - don't retry
10
Err(e) if e.is_card_declined() => {
11
// Log and return a permanent failure
12
log_payment_declined(&payment, &e).await?;
13
Err(anyhow::anyhow!("Card declined: {}", e))
14
}
15
16
Err(e) => Err(e.into()),
17
}
18
}

Best Practice: Distinguish between:

  • Retriable errors: Network timeouts, 5xx responses, rate limits
  • Permanent errors: Invalid input, 4xx responses, business rule violations

Compensating Actions

For multi-step workflows, you need to handle partial completion:

1
#[workflow(name = "book_flight")]
2
pub async fn book_flight(ctx: WorkflowContext, booking: Booking) -> Result<Confirmation> {
3
// Step 1: Reserve seat
4
let reservation = ctx.execute_task(reserve_seat, booking.seat_id).await?;
5
6
// Step 2: Charge payment
7
let charge = match ctx.execute_task(charge_card, booking.payment).await {
8
Ok(c) => c,
9
Err(e) => {
10
// Payment failed - compensate by releasing seat
11
ctx.execute_task(release_seat, reservation.id).await?;
12
return Err(e);
13
}
14
};
15
16
// Step 3: Send confirmation
17
ctx.execute_task(send_confirmation, charge.receipt).await?;
18
19
Ok(Confirmation { booking_id: charge.id })
20
}

This is the Saga pattern: Each step has a compensating action that undoes it.


Next Steps