Scenarios & Management
Managing Running Workflows
Cancel a Workflow
Gracefully cancel a running workflow:
1orcher workflow cancel <execution-id>Terminate a Workflow
Immediately terminate a workflow:
1orcher workflow terminate <execution-id>Send an Event
Send an event to a running workflow (for signal-based coordination):
1orcher workflow event <execution-id> --name "approval_received" --data '{"approved": true}'Common Debugging Scenarios
Task Failing Intermittently
Symptoms: A task succeeds sometimes but fails with timeouts or connection errors.
1# View recent task failures2orcher logs workflow/my-workflow --tasks3orcher logs workflow/my-workflow --level error --grep "timeout"Common causes:
- External service instability — add retries with exponential backoff via the
retrytask attribute - Network timeouts — increase task timeout or use task heartbeats for long-running operations
- Resource contention — check database connection pool utilization via metrics
Workflow Stuck / Not Progressing
Symptoms: Workflow stays in running status but doesn't complete.
1# Check the journal to see where it stopped2orcher logs workflow/my-workflow --journal34# Check task execution status5orcher logs workflow/my-workflow --tasks67# Check queue depth (tasks might not be claimed)8orcher queue depth9orcher queue pollers <queue-name>Common causes:
- No workers polling the queue — Check
orcher queue pollers <queue>for connected workers - Task stuck in
startedstate — Worker may have crashed. The reclaimer will reset stuck tasks after the claim timeout expires. - Waiting for an event — The workflow called
wait_for_event()and the event hasn't arrived yet. Useorcher workflow eventto send it. - Wrong namespace — Worker and workflow are in different namespaces
Workflow Failed
Symptoms: Workflow completed with failed status.
1# Get execution details2orcher workflow get <execution-id>34# View error logs5orcher logs workflow/my-workflow --level error67# View the journal to see which step failed8orcher logs workflow/my-workflow --journalCommon causes:
- Task exceeded max retries — check the dead letter queue via
orcher_dlq_current_depthmetric - Unhandled error in workflow code — check the error message in execution details
- Timeout — workflow or task exceeded its configured timeout
Slow Workflow Execution
Symptoms: Workflow takes longer than expected.
1# View task durations2orcher logs workflow/my-workflow --tasksLook at which task takes the most time. Common fixes:
- Sequential tasks that could be parallel — Use fan-out/fan-in pattern
- Slow external API calls — Consider caching or async patterns
- Queue latency — Check queue depth and worker count. Enable Redis so workers are woken when work is queued.
- No eager dispatch — Sequential workflows benefit from eager task return (enabled by default)
Diagnostic Logging in Code
Add strategic tracing calls to aid debugging:
1#[workflow]2async fn order_processing(ctx: Ctx, order: Order) -> Result<Receipt> {3 tracing::info!(4 order_id = %order.id,5 items = order.items.len(),6 "Starting order processing"7 );89 let validation = ctx.execute_task(validate_order, order.clone()).await?;1011 tracing::info!(12 valid = validation.is_valid,13 checks = validation.checks_passed,14 "Validation completed"15 );1617 if !validation.is_valid {18 tracing::warn!(19 reason = %validation.rejection_reason,20 "Order rejected"21 );22 return Err(anyhow!("Order validation failed"));23 }2425 let payment = ctx.execute_task(charge_payment, order.amount).await?;2627 tracing::info!(28 transaction_id = %payment.id,29 "Payment processed"30 );3132 ctx.execute_task(send_confirmation, payment.id).await33}Use structured fields (not string interpolation) so logs are machine-parseable in JSON format. See Logging for format configuration.
CLI Commands Summary
1# Inspect2orcher describe <resource-type> <name>3orcher workflow list [--status <status>] [--workflow <name>]4orcher workflow get <execution-id>5orcher workflow history <execution-id>67# Logs8orcher logs <resource> [--follow] [--level <level>] [--grep <pattern>]9orcher logs <resource> --journal # Event-sourced journal10orcher logs <resource> --tasks # Task execution details11orcher logs <resource> --tail <n> # Last N entries12orcher logs <resource> --since <time> # Time filter1314# Manage15orcher workflow cancel <execution-id>16orcher workflow terminate <execution-id>17orcher workflow event <execution-id> --name <event> --data <json>1819# Queue inspection20orcher queue depth21orcher queue stats <queue-name>22orcher queue pollers <queue-name>Next Steps
- Debugging Overview — Inspecting executions with describe, list, and get commands
- Viewing Logs — Text logs, journal entries, task execution details, and workflow history
- Understanding Replay — How deterministic replay works and how to debug it
- Logging — Configure structured logging formats
- Metrics — Monitor performance with Prometheus
- Tracing — Distributed tracing with OpenTelemetry
- Queues — Queue monitoring and troubleshooting