Best Practices & Troubleshooting
Best Practices
1. Use Appropriate Retry Policies
Not all errors should be retried. Transient errors like network timeouts, rate limits, and temporary service outages benefit from retries. But deterministic errors — validation failures, authentication errors, or malformed input — will never succeed no matter how many times you retry. Only configure retries for operations that have a reasonable chance of succeeding on a subsequent attempt.
1// ✔ Good: Transient errors (network, service unavailable)2#[task(name = "api_call", retry = 5)]34// ✔ Good: Critical operations with bounded retries5#[task(name = "payment", retry = 3)]67// ❌ Bad: Validation errors (will never succeed on retry)8#[task(name = "validate_email", retry = 5)] // Don't retry validation2. Implement Idempotent Compensation
Compensation actions may themselves be retried if they fail, so they must be idempotent — safe to execute multiple times with the same result. Always check the current state before performing the compensation action. If the resource is already in the compensated state, return success immediately.
1// ✔ Good: Idempotent cancellation2async fn cancel_booking(ctx: &WorkflowContext, booking_id: String) -> Result<()> {3 // Check if already canceled4 let status = get_booking_status(&booking_id).await?;5 if status == "canceled" {6 return Ok(()); // Already canceled, safe to skip7 }89 // Cancel the booking10 booking_service::cancel(&booking_id).await11}3. Log Error Context
When errors occur in workflows, rich contextual logging is essential for debugging. Include the operation name, relevant identifiers, the current attempt number, and the error itself. Structured logging (key-value pairs rather than string interpolation) makes it much easier to search and filter logs in production monitoring tools.
1// ✔ Good: Detailed error context2Err(e) => {3 tracing::error!(4 operation = "book_flight",5 flight_id = %input.flight_id,6 attempt = ctx.attempt(),7 error = %e,8 "Flight booking failed"9 );10 return Err(e);11}4. Set Reasonable Timeouts
Timeouts should reflect the actual expected duration of the operation plus a safety margin. A task that normally completes in 10 seconds might have a 60-second timeout to account for slow networks, but a 1-second timeout would cause false failures. Conversely, setting no timeout at all risks workflows hanging indefinitely if an external service stops responding.
1// ✔ Good: Match timeout to expected duration2#[task(name = "quick_validation", timeout_mins = 1)]3#[task(name = "batch_import", timeout_mins = 30)]4#[task(name = "ml_training", timeout_mins = 180)]56// ❌ Bad: Timeout too short for operation7#[task(name = "large_file_upload", timeout_mins = 1)] // Will always timeout5. Handle Partial Failures
When processing a batch of items, a single failure shouldn't necessarily abort the entire batch. Collect results and errors separately, then return a summary that includes both successful outcomes and failure details. This lets downstream consumers decide how to handle the partial results — retry just the failed items, alert on the errors, or accept the partial dataset.
1// ✔ Good: Continue processing on partial failure2let mut results = Vec::new();3let mut errors = Vec::new();45for item in items {6 match ctx.execute_task(process_item, item).await {7 Ok(result) => results.push(result),8 Err(e) => errors.push(e),9 }10}1112// Return partial results with error summary13Ok(BatchResult { results, errors })Troubleshooting
Below are common error handling issues you may encounter when building Orcher workflows, along with their root causes and recommended solutions.
Retry Not Working
Symptom: Task fails without retrying
Causes:
- Task configured with
retry = 0or no retry policy - Error is a non-retryable type (e.g., validation error returned as permanent failure)
- The workflow itself failed before the task could be retried
Solutions:
- Verify the task's retry configuration in its decorator/attribute
- Ensure the error is a transient type that can succeed on retry
- Add logging with
ctx.attempt()to confirm retry attempts are being made
Compensation Not Running
Symptom: Resources not cleaned up after failure
Causes:
- Compensation logic placed outside the correct catch/error block
- The compensation task itself failed silently
- Workflow crashed or was terminated before reaching compensation code
Solutions:
- Ensure compensation calls are inside the catch/error block for each step
- Make compensation tasks idempotent so they can be safely retried
- Add logging to both the start and completion of compensation tasks
Circuit Breaker Not Opening
Symptom: Failures continue despite reaching the threshold
Causes:
- Failure count stored in local memory but workflow runs on different instances
- Threshold set too high relative to the number of operations
- Each workflow instance maintains its own counter instead of a shared one
Solutions:
- Use shared state (database, Redis, or workflow state) for the failure counter
- Lower the failure threshold to match your availability requirements
- Consider implementing the circuit breaker as a separate workflow that other workflows query