Operations

Scenarios & Management


Managing Running Workflows

Cancel a Workflow

Gracefully cancel a running workflow:

1
orcher workflow cancel <execution-id>

Terminate a Workflow

Immediately terminate a workflow:

1
orcher workflow terminate <execution-id>

Send an Event

Send an event to a running workflow (for signal-based coordination):

1
orcher workflow event <execution-id> --name "approval_received" --data '{"approved": true}'

Common Debugging Scenarios

Task Failing Intermittently

Symptoms: A task succeeds sometimes but fails with timeouts or connection errors.

1
# View recent task failures
2
orcher logs workflow/my-workflow --tasks
3
orcher logs workflow/my-workflow --level error --grep "timeout"

Common causes:

  • External service instability — add retries with exponential backoff via the retry task attribute
  • Network timeouts — increase task timeout or use task heartbeats for long-running operations
  • Resource contention — check database connection pool utilization via metrics

Workflow Stuck / Not Progressing

Symptoms: Workflow stays in running status but doesn't complete.

1
# Check the journal to see where it stopped
2
orcher logs workflow/my-workflow --journal
3
4
# Check task execution status
5
orcher logs workflow/my-workflow --tasks
6
7
# Check queue depth (tasks might not be claimed)
8
orcher queue depth
9
orcher queue pollers <queue-name>

Common causes:

  • No workers polling the queue — Check orcher queue pollers <queue> for connected workers
  • Task stuck in started state — Worker may have crashed. The reclaimer will reset stuck tasks after the claim timeout expires.
  • Waiting for an event — The workflow called wait_for_event() and the event hasn't arrived yet. Use orcher workflow event to send it.
  • Wrong namespace — Worker and workflow are in different namespaces

Workflow Failed

Symptoms: Workflow completed with failed status.

1
# Get execution details
2
orcher workflow get <execution-id>
3
4
# View error logs
5
orcher logs workflow/my-workflow --level error
6
7
# View the journal to see which step failed
8
orcher logs workflow/my-workflow --journal

Common causes:

  • Task exceeded max retries — check the dead letter queue via orcher_dlq_current_depth metric
  • Unhandled error in workflow code — check the error message in execution details
  • Timeout — workflow or task exceeded its configured timeout

Slow Workflow Execution

Symptoms: Workflow takes longer than expected.

1
# View task durations
2
orcher logs workflow/my-workflow --tasks

Look at which task takes the most time. Common fixes:

  • Sequential tasks that could be parallel — Use fan-out/fan-in pattern
  • Slow external API calls — Consider caching or async patterns
  • Queue latency — Check queue depth and worker count. Enable Redis so workers are woken when work is queued.
  • No eager dispatch — Sequential workflows benefit from eager task return (enabled by default)

Diagnostic Logging in Code

Add strategic tracing calls to aid debugging:

1
#[workflow]
2
async fn order_processing(ctx: Ctx, order: Order) -> Result<Receipt> {
3
tracing::info!(
4
order_id = %order.id,
5
items = order.items.len(),
6
"Starting order processing"
7
);
8
9
let validation = ctx.execute_task(validate_order, order.clone()).await?;
10
11
tracing::info!(
12
valid = validation.is_valid,
13
checks = validation.checks_passed,
14
"Validation completed"
15
);
16
17
if !validation.is_valid {
18
tracing::warn!(
19
reason = %validation.rejection_reason,
20
"Order rejected"
21
);
22
return Err(anyhow!("Order validation failed"));
23
}
24
25
let payment = ctx.execute_task(charge_payment, order.amount).await?;
26
27
tracing::info!(
28
transaction_id = %payment.id,
29
"Payment processed"
30
);
31
32
ctx.execute_task(send_confirmation, payment.id).await
33
}

Use structured fields (not string interpolation) so logs are machine-parseable in JSON format. See Logging for format configuration.


CLI Commands Summary

1
# Inspect
2
orcher describe <resource-type> <name>
3
orcher workflow list [--status <status>] [--workflow <name>]
4
orcher workflow get <execution-id>
5
orcher workflow history <execution-id>
6
7
# Logs
8
orcher logs <resource> [--follow] [--level <level>] [--grep <pattern>]
9
orcher logs <resource> --journal # Event-sourced journal
10
orcher logs <resource> --tasks # Task execution details
11
orcher logs <resource> --tail <n> # Last N entries
12
orcher logs <resource> --since <time> # Time filter
13
14
# Manage
15
orcher workflow cancel <execution-id>
16
orcher workflow terminate <execution-id>
17
orcher workflow event <execution-id> --name <event> --data <json>
18
19
# Queue inspection
20
orcher queue depth
21
orcher queue stats <queue-name>
22
orcher queue pollers <queue-name>

Next Steps

  • Debugging Overview — Inspecting executions with describe, list, and get commands
  • Viewing Logs — Text logs, journal entries, task execution details, and workflow history
  • Understanding Replay — How deterministic replay works and how to debug it
  • Logging — Configure structured logging formats
  • Metrics — Monitor performance with Prometheus
  • Tracing — Distributed tracing with OpenTelemetry
  • Queues — Queue monitoring and troubleshooting