Operations

Understanding Replay


Orcher uses deterministic replay for crash recovery. Understanding replay is essential for debugging, because it explains why some steps don't re-execute after a failure.

How It Works

1
Normal Execution:
2
Step 1: validate_order → Result A (journaled)
3
Step 2: process_payment → Result B (journaled)
4
Step 3: ship_order → Result C (journaled)
5
✓ Workflow completes
6
7
Crash During Step 3:
8
Server crashes or worker restarts
9
10
Recovery (Replay):
11
Step 1: validate_order → Read Result A from journal (NOT re-executed)
12
Step 2: process_payment → Read Result B from journal (NOT re-executed)
13
Step 3: ship_order → Resume execution from crash point
14
✓ Workflow completes without duplicate side effects

Key points:

  • Completed steps are never re-executed — results are read from the PostgreSQL journal
  • Only the incomplete step is resumed
  • No side effects are duplicated (payments aren't charged twice)
  • The journal is the source of truth for what happened

Debugging Replay Issues

Use the journal view to understand what was replayed:

1
# View the journal for an execution
2
orcher logs workflow/my-workflow --journal
3
4
# Or use workflow history
5
orcher workflow history <execution-id>

If a workflow behaves differently on replay, check:

  1. Non-deterministic code in workflows — Workflows must be deterministic. Don't use rand(), SystemTime::now(), or external API calls directly in workflow code. Use tasks for side effects.
  2. Missing journal entries — If the journal is incomplete, the step will re-execute. Check for database issues.
  3. Task idempotency — Tasks that call external APIs should use idempotency keys to prevent duplicate operations on replay.

Writing Replay-Safe Tasks

Tasks that interact with external systems should be idempotent:

1
#[task]
2
async fn charge_payment(ctx: TaskCtx, payment: PaymentRequest) -> Result<ChargeResult> {
3
// Use a deterministic idempotency key
4
let idempotency_key = format!("{}:{}", ctx.execution_id, ctx.task_id);
5
6
// Check for existing charge (handles replay)
7
if let Some(existing) = find_charge_by_key(&idempotency_key).await? {
8
tracing::info!("Returning cached result (replay detected)");
9
return Ok(existing);
10
}
11
12
// Proceed with new charge
13
gateway::charge(&payment, &idempotency_key).await
14
}

Next Steps

  • Debugging Overview — Inspecting executions with describe, list, and get commands
  • Viewing Logs — Text logs, journal entries, task execution details, and workflow history
  • Scenarios & Management — Managing running workflows, common debugging scenarios, and CLI reference