Understanding Replay
Orcher uses deterministic replay for crash recovery. Understanding replay is essential for debugging, because it explains why some steps don't re-execute after a failure.
How It Works
1Normal Execution:2 Step 1: validate_order → Result A (journaled)3 Step 2: process_payment → Result B (journaled)4 Step 3: ship_order → Result C (journaled)5 ✓ Workflow completes67Crash During Step 3:8 Server crashes or worker restarts910Recovery (Replay):11 Step 1: validate_order → Read Result A from journal (NOT re-executed)12 Step 2: process_payment → Read Result B from journal (NOT re-executed)13 Step 3: ship_order → Resume execution from crash point14 ✓ Workflow completes without duplicate side effectsKey points:
- Completed steps are never re-executed — results are read from the PostgreSQL journal
- Only the incomplete step is resumed
- No side effects are duplicated (payments aren't charged twice)
- The journal is the source of truth for what happened
Debugging Replay Issues
Use the journal view to understand what was replayed:
1# View the journal for an execution2orcher logs workflow/my-workflow --journal34# Or use workflow history5orcher workflow history <execution-id>If a workflow behaves differently on replay, check:
- Non-deterministic code in workflows — Workflows must be deterministic. Don't use
rand(),SystemTime::now(), or external API calls directly in workflow code. Use tasks for side effects. - Missing journal entries — If the journal is incomplete, the step will re-execute. Check for database issues.
- Task idempotency — Tasks that call external APIs should use idempotency keys to prevent duplicate operations on replay.
Writing Replay-Safe Tasks
Tasks that interact with external systems should be idempotent:
1#[task]2async fn charge_payment(ctx: TaskCtx, payment: PaymentRequest) -> Result<ChargeResult> {3 // Use a deterministic idempotency key4 let idempotency_key = format!("{}:{}", ctx.execution_id, ctx.task_id);56 // Check for existing charge (handles replay)7 if let Some(existing) = find_charge_by_key(&idempotency_key).await? {8 tracing::info!("Returning cached result (replay detected)");9 return Ok(existing);10 }1112 // Proceed with new charge13 gateway::charge(&payment, &idempotency_key).await14}Next Steps
- Debugging Overview — Inspecting executions with describe, list, and get commands
- Viewing Logs — Text logs, journal entries, task execution details, and workflow history
- Scenarios & Management — Managing running workflows, common debugging scenarios, and CLI reference