Guides

Best Practices & Troubleshooting


Best Practices

1. Use Appropriate Retry Policies

Not all errors should be retried. Transient errors like network timeouts, rate limits, and temporary service outages benefit from retries. But deterministic errors — validation failures, authentication errors, or malformed input — will never succeed no matter how many times you retry. Only configure retries for operations that have a reasonable chance of succeeding on a subsequent attempt.

1
// ✔ Good: Transient errors (network, service unavailable)
2
#[task(name = "api_call", retry = 5)]
3
4
// ✔ Good: Critical operations with bounded retries
5
#[task(name = "payment", retry = 3)]
6
7
// ❌ Bad: Validation errors (will never succeed on retry)
8
#[task(name = "validate_email", retry = 5)] // Don't retry validation

2. Implement Idempotent Compensation

Compensation actions may themselves be retried if they fail, so they must be idempotent — safe to execute multiple times with the same result. Always check the current state before performing the compensation action. If the resource is already in the compensated state, return success immediately.

1
// ✔ Good: Idempotent cancellation
2
async fn cancel_booking(ctx: &WorkflowContext, booking_id: String) -> Result<()> {
3
// Check if already canceled
4
let status = get_booking_status(&booking_id).await?;
5
if status == "canceled" {
6
return Ok(()); // Already canceled, safe to skip
7
}
8
9
// Cancel the booking
10
booking_service::cancel(&booking_id).await
11
}

3. Log Error Context

When errors occur in workflows, rich contextual logging is essential for debugging. Include the operation name, relevant identifiers, the current attempt number, and the error itself. Structured logging (key-value pairs rather than string interpolation) makes it much easier to search and filter logs in production monitoring tools.

1
// ✔ Good: Detailed error context
2
Err(e) => {
3
tracing::error!(
4
operation = "book_flight",
5
flight_id = %input.flight_id,
6
attempt = ctx.attempt(),
7
error = %e,
8
"Flight booking failed"
9
);
10
return Err(e);
11
}

4. Set Reasonable Timeouts

Timeouts should reflect the actual expected duration of the operation plus a safety margin. A task that normally completes in 10 seconds might have a 60-second timeout to account for slow networks, but a 1-second timeout would cause false failures. Conversely, setting no timeout at all risks workflows hanging indefinitely if an external service stops responding.

1
// ✔ Good: Match timeout to expected duration
2
#[task(name = "quick_validation", timeout_mins = 1)]
3
#[task(name = "batch_import", timeout_mins = 30)]
4
#[task(name = "ml_training", timeout_mins = 180)]
5
6
// ❌ Bad: Timeout too short for operation
7
#[task(name = "large_file_upload", timeout_mins = 1)] // Will always timeout

5. Handle Partial Failures

When processing a batch of items, a single failure shouldn't necessarily abort the entire batch. Collect results and errors separately, then return a summary that includes both successful outcomes and failure details. This lets downstream consumers decide how to handle the partial results — retry just the failed items, alert on the errors, or accept the partial dataset.

1
// ✔ Good: Continue processing on partial failure
2
let mut results = Vec::new();
3
let mut errors = Vec::new();
4
5
for item in items {
6
match ctx.execute_task(process_item, item).await {
7
Ok(result) => results.push(result),
8
Err(e) => errors.push(e),
9
}
10
}
11
12
// Return partial results with error summary
13
Ok(BatchResult { results, errors })

Troubleshooting

Below are common error handling issues you may encounter when building Orcher workflows, along with their root causes and recommended solutions.

Retry Not Working

Symptom: Task fails without retrying

Causes:

  • Task configured with retry = 0 or no retry policy
  • Error is a non-retryable type (e.g., validation error returned as permanent failure)
  • The workflow itself failed before the task could be retried

Solutions:

  • Verify the task's retry configuration in its decorator/attribute
  • Ensure the error is a transient type that can succeed on retry
  • Add logging with ctx.attempt() to confirm retry attempts are being made

Compensation Not Running

Symptom: Resources not cleaned up after failure

Causes:

  • Compensation logic placed outside the correct catch/error block
  • The compensation task itself failed silently
  • Workflow crashed or was terminated before reaching compensation code

Solutions:

  • Ensure compensation calls are inside the catch/error block for each step
  • Make compensation tasks idempotent so they can be safely retried
  • Add logging to both the start and completion of compensation tasks

Circuit Breaker Not Opening

Symptom: Failures continue despite reaching the threshold

Causes:

  • Failure count stored in local memory but workflow runs on different instances
  • Threshold set too high relative to the number of operations
  • Each workflow instance maintains its own counter instead of a shared one

Solutions:

  • Use shared state (database, Redis, or workflow state) for the failure counter
  • Lower the failure threshold to match your availability requirements
  • Consider implementing the circuit breaker as a separate workflow that other workflows query

Next Steps