Guides

Understanding Orcher


Orcher is a durable execution engine — it makes your code survive failures automatically. You write normal functions, and Orcher records each step's result to PostgreSQL as it runs. If a process crashes, another worker picks up the execution and replays completed steps from the saved results without re-executing them.

  • Automatic checkpointing — Every task result is saved before being returned to your workflow
  • Crash recovery — Any worker can resume any workflow from its last checkpoint
  • No duplicate side effects — Completed tasks return cached results on replay
  • Complete audit trail — The journal records every step, its result, and when it happened

The Problem Orcher Solves

Traditional code execution is ephemeral. When your process crashes, you lose all in-memory state, your position in the execution flow, and any partial results:

1
// Without Orcher — loses state on crash
2
async fn process_order(order: Order) -> Result<()> {
3
let payment = charge_card(order.total).await?; // ← Succeeds
4
update_inventory(&order.items).await?; // ← Succeeds
5
send_confirmation_email(&order.email).await?; // ← CRASHES HERE
6
notify_warehouse(&order.id).await?; // ← Never runs
7
8
// Customer was charged but never got confirmation.
9
// You don't know which steps completed. No automatic way to resume.
10
Ok(())
11
}

With Orcher, every step is journaled. After a crash, the workflow resumes from the first incomplete step:

1
use orcher::prelude::*;
2
3
#[workflow(name = "process_order")]
4
async fn process_order(ctx: WorkflowContext, order: Order) -> Result<OrderResult> {
5
let payment = ctx.execute_task("charge_card", order.total).await?; // Journaled
6
ctx.execute_task("update_inventory", &order.items).await?; // Journaled
7
ctx.execute_task("send_confirmation", &order.email).await?; // Journaled
8
ctx.execute_task("notify_warehouse", &order.id).await?; // Journaled
9
10
// If crash happens after step 2, next run:
11
// - charge_card: returns cached result (no duplicate charge)
12
// - update_inventory: returns cached result
13
// - send_confirmation: executes for the first time
14
// - notify_warehouse: executes next
15
Ok(OrderResult { order_id: order.id, payment_id: payment.id })
16
}

Core Concepts

Workflows

A workflow is a durable function that orchestrates tasks into a business process. It defines the order of operations, makes decisions, and handles errors — but never performs I/O directly. All external work is delegated to tasks.

Workflow code must be deterministic — given the same inputs and task results, it always follows the same execution path. This is what makes replay work. I/O operations are non-deterministic (they can return different results each time), so they must go through tasks whose results are cached in the journal.

1
use orcher::prelude::*;
2
3
#[workflow(name = "onboard_customer")]
4
async fn onboard_customer(ctx: WorkflowContext, customer: NewCustomer) -> Result<OnboardResult> {
5
// Workflows make decisions
6
if customer.tier == "enterprise" {
7
ctx.execute_task("assign_account_manager", &customer.id).await?;
8
}
9
10
// Workflows orchestrate tasks
11
let account_id = ctx.execute_task("create_account", &customer).await?;
12
ctx.execute_task("send_welcome_email", &customer.email).await?;
13
ctx.execute_task("provision_resources", &account_id).await?;
14
15
// Workflows handle errors with compensation
16
match ctx.execute_task("charge_setup_fee", &customer.payment_method).await {
17
Ok(_) => {}
18
Err(e) => {
19
ctx.execute_task("delete_account", &account_id).await?;
20
return Err(e);
21
}
22
}
23
24
Ok(OnboardResult { account_id, status: "active".into() })
25
}

Workflows should:

  • Orchestrate tasks — decide which tasks to run and in what order
  • Make decisions — branch logic based on task results
  • Handle errors — catch task failures and run compensation tasks
  • Coordinate systems — chain tasks that talk to different services

Workflows should NOT:

  • Perform I/O directly — network calls, file reads, database queries must go through tasks
  • Use non-deterministic functions — Math.random(), Date.now(), Uuid::new_v4() produce different values on replay; wrap them in tasks so the result is cached
  • Store large data in workflow state — pass references (URLs, IDs) instead of large payloads

Tasks

A task is a unit of work that performs one specific action — calling an API, writing to a database, or sending an email. Tasks are where all side effects happen. By isolating side effects in tasks, Orcher can cache their results for replay, retry them independently, and run them on different workers.

1
use orcher::prelude::*;
2
3
#[task(name = "charge_card", timeout = 30, retry_policy(max_attempts = 3, backoff_coefficient = 2.0))]
4
async fn charge_card(ctx: TaskContext, input: ChargeInput) -> Result<ChargeResult> {
5
let stripe = Stripe::new(&std::env::var("STRIPE_SECRET_KEY")?);
6
7
let charge = stripe.charges().create(CreateCharge {
8
amount: input.amount,
9
currency: &input.currency,
10
source: &input.payment_method,
11
idempotency_key: Some(&input.idempotency_key),
12
..Default::default()
13
}).await?;
14
15
Ok(ChargeResult { charge_id: charge.id, status: charge.status })
16
}

Task design principles:

  • Idempotent — Running the same task twice with the same input should produce the same result. Use idempotency keys for external APIs (e.g., Stripe). This matters because Orcher may retry tasks that fail partway through.
  • Focused — One task, one action. "Charge the card" not "charge the card, update inventory, and send email." Smaller tasks mean more granular checkpoints.
  • Stateless — Each invocation is independent. State across steps is managed by the workflow.
  • Serializable — Input and output must be JSON-serializable (stored in PostgreSQL JSONB and sent over gRPC).

The Journal

The journal is an append-only event log in PostgreSQL that records every step of a workflow execution. It is the source of truth for what has happened.

Workflow process_payment · execution exec_abc123

Event 1
WorkflowStarted
input: { orderId: "ord_456", total: 5000 }
Event 2
StepCompleted
validate_card → { valid: true }
Event 3
StepCompleted
check_fraud → { score: 0.15, risk: "low" }
Event 4
StepCompleted
charge_stripe → { id: "ch_xyz", status: "succeeded" }
Event 5
StepCompleted
save_payment → { payment_id: "pay_123" }
Event 6
StepCompleted
send_receipt → FAILED: "SMTP timeout" ← crash here

When the workflow resumes on a different worker, ctx.execute_task() checks the journal before executing:

ctx.execute_task(charge_card, input)

StepCompleted entry exists
in journal for this step?
YES
Return cached result
No API call — instant
NO
Execute task
Save to journal → return result

Deterministic Replay

When a workflow resumes after a crash, it re-runs the entire workflow function from the beginning. Completed steps return cached results from the journal instantly. Because the same task results lead to the same decisions, the workflow follows the exact same execution path and reaches the point where the crash occurred.

First ExecutionDeterministic Replay
1validate_card→ {valid: true}✓
2check_fraud→ {score: 0.15}✓
3charge_stripe→ {id: "ch_xyz"}✓
4send_receiptCRASH before execution✗
ReplayNew worker picks up
1validate_cardcached → {valid: true}↺
2check_fraudcached → {score: 0.15}↺
3charge_stripecached → {id: "ch_xyz"}↺
4send_receiptExecute fresh✓

This is why workflow code must be deterministic. If you use Math.random() directly in a workflow, the original run might take one branch and the replay might take a different branch, causing the replay to diverge from the journal. Anything non-deterministic must go inside a task:

1
use orcher::prelude::*;
2
3
// BAD: Non-deterministic — different value on replay breaks journal alignment
4
#[workflow(name = "bad_workflow")]
5
async fn bad_workflow(ctx: WorkflowContext, input: Input) -> Result<Output> {
6
let random: f64 = rand::random(); // Different each run!
7
if random > 0.5 {
8
ctx.execute_task("do_something", &input).await?;
9
}
10
Ok(Output {})
11
}
12
13
// GOOD: Random value captured in a task and cached in journal
14
#[workflow(name = "good_workflow")]
15
async fn good_workflow(ctx: WorkflowContext, input: Input) -> Result<Output> {
16
let random: f64 = ctx.execute_task("generate_random", &()).await?;
17
if random > 0.5 {
18
ctx.execute_task("do_something", &input).await?; // Same branch on replay
19
}
20
Ok(Output {})
21
}

Non-deterministic operations to avoid in workflows:

  • Random values — Math.random(), rand::random(), random.random()
  • Current time — Date.now(), Instant::now(), datetime.now()
  • UUID generation — Uuid::new_v4(), crypto.randomUUID()
  • Direct I/O — HTTP requests, file reads, database queries
  • Global mutable state — reading from variables that change between runs

Architecture

Orcher follows a client-server model with three components:

Your Application (SDK Workers)
Worker 1
Workflows · Tasks
Worker 2
Workflows · Tasks
Worker N
Workflows · Tasks
gRPC
Orchestrator
:50051:8080
Workflow scheduling
Task dispatch
Journal management
Claim-based locking
Timeout detection
Eager return
PostgreSQL
Journal · State · Queue
Redis
Fast dispatch · optional

Orchestrator — A stateless server (gRPC on port 50051, REST API on port 8080) that coordinates workflow execution, task scheduling, and worker polling. All state lives in PostgreSQL, so you can run multiple orchestrator instances for high availability.

SDK Workers — Your application processes running the Orcher SDK. Workers register with the orchestrator, poll for work via gRPC long-polling, execute your workflow and task handlers, and report results. Workers are stateless — if one crashes, another picks up the work.

PostgreSQL — The primary data store for workflow state, journal entries, task executions, and the work queue (SELECT FOR UPDATE SKIP LOCKED).

Redis (optional) — Wakes workers via LPUSH/BRPOP as soon as work is queued. Without Redis, the system falls back to PostgreSQL-based polling.

Execution Flow

  1. Start workflow — Your application calls the orchestrator's StartWorkflowExecution gRPC endpoint (or POST /api/v1/workflows/execute on the orchestrator's REST API). The orchestrator creates a workflow execution record in PostgreSQL.

  2. Worker polls — SDK workers long-poll the orchestrator via PollWorkflowExecution. When work is available, the orchestrator assigns the workflow to a worker using SELECT FOR UPDATE SKIP LOCKED.

  3. Replay and execute — The worker loads the journal from the orchestrator, replays completed steps (returning cached results), and executes the next pending step. Each ctx.execute_task() call schedules a task on the orchestrator.

  4. Task dispatch — The orchestrator places the task on the worker's task queue. A worker polls via PollTaskExecution, executes the task handler, and reports the result via CompleteTaskExecution.

  5. Continue or complete — After a task completes, the orchestrator re-queues the workflow for the next step. This repeats until the workflow finishes or fails.


Compensation Pattern (Sagas)

When operations span multiple services, you can't use a single database transaction. Workflows solve this with compensation — if a later step fails, you run "undo" tasks for the steps that already succeeded:

1
use orcher::prelude::*;
2
3
#[workflow(name = "transfer_money")]
4
async fn transfer_money(ctx: WorkflowContext, transfer: Transfer) -> Result<TransferResult> {
5
// Debit source account
6
ctx.execute_task("debit_account", &DebitInput {
7
account_id: transfer.from.clone(),
8
amount: transfer.amount,
9
}).await?;
10
11
// Credit destination — if this fails, compensate
12
match ctx.execute_task("credit_account", &CreditInput {
13
account_id: transfer.to.clone(),
14
amount: transfer.amount,
15
}).await {
16
Ok(_) => Ok(TransferResult { status: "completed".into() }),
17
Err(e) => {
18
// Refund the debit
19
ctx.execute_task("refund_account", &RefundInput {
20
account_id: transfer.from.clone(),
21
amount: transfer.amount,
22
}).await?;
23
Err(e)
24
}
25
}
26
}

When to Use Orcher

Good fit:

  • Multi-step business processes — order processing, user onboarding, payment flows
  • Distributed transactions — cross-service operations with compensation (Saga pattern)
  • Long-running operations — data migrations, batch processing, report generation
  • External dependencies — API integrations with retries, human approvals that take days
  • Critical operations — financial transactions, data pipelines, compliance workflows

Not ideal for:

  • Real-time streaming — use a stream processor; journaling every step adds latency a per-event stream may not afford
  • Simple CRUD — direct database access is simpler when the operation is already atomic
  • Stateless HTTP APIs — no orchestration needed if each request is independent

Next Steps