Guides

Best Practices & Pitfalls


Best Practices

1. Make Compensation Idempotent

Compensation actions may be retried if they fail or if the workflow replays. Always check the current state of the resource before performing the compensation. If the resource has already been compensated (e.g., a payment is already refunded), return success without performing the action again.

1
// Good - Idempotent compensation
2
#[task(name = "refund_payment")]
3
pub async fn refund_payment(
4
ctx: TaskContext,
5
payment_id: String,
6
) -> Result<()> {
7
// Check if already refunded
8
let payment_status = get_payment_status(&payment_id).await?;
9
10
if payment_status == "refunded" {
11
println!("Payment {} already refunded, skipping", payment_id);
12
return Ok(());
13
}
14
15
// Issue refund
16
payment_gateway::refund(payment_id).await?;
17
Ok(())
18
}
19
20
// Bad - Not idempotent (could double-refund)
21
#[task(name = "refund_payment_bad")]
22
pub async fn refund_payment_bad(
23
ctx: TaskContext,
24
payment_id: String,
25
) -> Result<()> {
26
// Blindly refunds without checking current state
27
payment_gateway::refund(payment_id).await?;
28
Ok(())
29
}

2. Compensate in Reverse Order

Always undo operations in the opposite order they were executed. This is important because later steps may depend on the state created by earlier steps. For example, canceling a shipment before refunding the payment ensures the payment refund doesn't trigger a conflicting re-shipment.

1
// Good - Reverse order compensation
2
async fn compensate_saga(ctx: &WorkflowContext, state: &SagaState) -> Result<()> {
3
// Last operation first
4
if state.shipment_created {
5
cancel_shipment(ctx, &state.shipment_id).await?;
6
}
7
8
// Then payment
9
if state.payment_captured {
10
refund_payment(ctx, &state.payment_id).await?;
11
}
12
13
// First operation last
14
if state.inventory_reserved {
15
release_inventory(ctx, &state.inventory_id).await?;
16
}
17
18
Ok(())
19
}

3. Store Compensation Data

Every saga step must store the IDs and metadata needed for its compensation action. Without these identifiers, compensation becomes impossible. Store resource IDs, amounts (for partial refunds), and timestamps (for audit trails) immediately after each step succeeds.

1
// Good - Stores all data needed for compensation
2
#[derive(Debug, Clone, Serialize, Deserialize)]
3
struct SagaState {
4
payment_id: Option<String>,
5
inventory_id: Option<String>,
6
shipment_id: Option<String>,
7
// Store amounts for partial refunds
8
payment_amount: Option<f64>,
9
// Store timestamps for audit
10
payment_captured_at: Option<DateTime<Utc>>,
11
}
12
13
// Bad - Missing compensation data
14
#[derive(Debug, Clone, Serialize, Deserialize)]
15
struct SagaStateBad {
16
payment_captured: bool,
17
inventory_reserved: bool,
18
// No IDs stored - can't compensate!
19
}

4. Handle Compensation Failures

Compensation itself can fail — a refund API might be temporarily down, or a cancellation service might time out. When this happens, don't stop compensating the remaining steps. Instead, attempt all compensations, collect any failures, and alert the operations team for manual resolution. This ensures partial compensation doesn't leave the system in a worse state.

1
// Good - Handles compensation failures without stopping
2
async fn compensate_saga_robust(
3
ctx: &WorkflowContext,
4
state: &SagaState,
5
) -> Result<()> {
6
let mut failures = Vec::new();
7
8
if state.payment_captured {
9
if let Err(e) = refund_payment(ctx, &state.payment_id).await {
10
println!("Refund failed: {}", e);
11
failures.push(("payment", e));
12
// Continue with other compensations
13
}
14
}
15
16
if state.inventory_reserved {
17
if let Err(e) = release_inventory(ctx, &state.inventory_id).await {
18
println!("Inventory release failed: {}", e);
19
failures.push(("inventory", e));
20
}
21
}
22
23
// If any compensation failed, alert operations team
24
if !failures.is_empty() {
25
ctx.execute_task(alert_ops_team, failures).await?;
26
}
27
28
Ok(())
29
}

5. Use Semantic Compensation

Prefer semantic undo (marking a resource as "cancelled" with a timestamp) over physical undo (deleting the record). Semantic compensation preserves the audit trail and makes it possible to analyze what happened, replay events, or manually recover. Physical deletion loses this history permanently.

1
// Good - Semantic compensation (preserves audit trail)
2
#[task(name = "cancel_reservation")]
3
pub async fn cancel_reservation(
4
ctx: TaskContext,
5
reservation_id: String,
6
) -> Result<()> {
7
database::update(
8
"reservations",
9
&reservation_id,
10
json!({ "status": "cancelled", "cancelled_at": Utc::now() })
11
).await?;
12
Ok(())
13
}
14
15
// Less good - Physical undo (loses audit trail)
16
#[task(name = "delete_reservation")]
17
pub async fn delete_reservation(
18
ctx: TaskContext,
19
reservation_id: String,
20
) -> Result<()> {
21
database::delete("reservations", &reservation_id).await?;
22
Ok(())
23
}

6. Set Appropriate Timeouts

When waiting for external systems in an event-driven saga, always set a timeout. Without one, the workflow could hang indefinitely if the external service never responds. When a timeout fires, treat it as a step failure and trigger compensation for all previously completed steps.

1
// Good - Timeouts for each step
2
let payment_event = ctx
3
.wait_for_event_with_timeout("payment_result", Duration::from_secs(300))
4
.await?;
5
6
match payment_event {
7
Some(event) => handle_event(event).await?,
8
None => {
9
// Timeout - compensate and fail gracefully
10
compensate_saga(&ctx, &state).await?;
11
return Err(Error::PaymentTimeout);
12
}
13
}

Saga vs Two-Phase Commit

Understanding when to use a saga versus a traditional two-phase commit helps you choose the right consistency model for your use case. Sagas trade immediate consistency for higher availability and throughput, making them the better choice for microservices architectures and long-running transactions.

AspectSaga PatternTwo-Phase Commit
LockingNo distributed locksRequires locks on all participants
AvailabilityHigh - services independentLow - all must be available
ConsistencyEventual consistencyImmediate consistency
PerformanceHigh throughputLower throughput
ComplexityModerate (compensation logic)High (coordinator + protocol)
Use CaseMicroservices, long transactionsTraditional databases, short transactions

Common Pitfalls

Below are the most frequent mistakes developers make when implementing sagas. Each pitfall can lead to data inconsistency, stuck workflows, or double-charged customers.

Non-Idempotent Operations

Without idempotency keys, retrying a step can cause duplicate side effects like charging a customer twice.

1
// Bad - Can charge customer multiple times on retry
2
async fn charge_customer(amount: f64) -> Result<String> {
3
stripe::charge(amount).await
4
}
5
6
// Good - Idempotent with unique key
7
async fn charge_customer(amount: f64, idempotency_key: String) -> Result<String> {
8
stripe::charge_with_key(amount, idempotency_key).await
9
}

Lost Compensation Data

If you forget to store the result ID from a step, you won't be able to compensate it later.

1
// Bad - Can't compensate without IDs
2
let payment_result = charge_payment(amount).await?;
3
// Forgot to store payment_result.id!
4
5
// Good - Store compensation data immediately
6
let payment_id = charge_payment(amount).await?;
7
state.payment_id = Some(payment_id);

Partial Compensation

If the first compensation step fails and throws an error, subsequent compensation steps never run — leaving the system in a partially compensated state.

1
// Bad - Stops compensating on first error
2
if state.payment_captured {
3
refund_payment(&state.payment_id).await?; // Throws error
4
}
5
if state.inventory_reserved {
6
release_inventory(&state.inventory_id).await?; // Never runs!
7
}
8
9
// Good - Try all compensations regardless of failures
10
let mut errors = Vec::new();
11
if state.payment_captured {
12
if let Err(e) = refund_payment(&state.payment_id).await {
13
errors.push(e);
14
}
15
}
16
if state.inventory_reserved {
17
if let Err(e) = release_inventory(&state.inventory_id).await {
18
errors.push(e);
19
}
20
}

Performance Considerations

Saga Overhead

Each saga step adds latency due to task execution, state persistence, and journal recording. The total time depends on your saga type:

  • Sequential saga — Total time = sum of all step durations
  • Parallel saga — Total time = duration of the slowest step
  • Compensation — Adds additional time on the failure path

Rule of thumb: Use sagas for operations that take >100ms per step. For faster operations, consider optimistic locking or single-transaction approaches.

State Storage

Saga state is persisted after each step for durability. Keep state small by storing references (IDs) rather than full objects:

1
// Good - Store references (~100 bytes)
2
struct SagaState {
3
order_id: String,
4
payment_id: String,
5
inventory_id: String,
6
}
7
8
// Bad - Store full objects (~65KB)
9
struct SagaStateBad {
10
order: Order, // Full object (10KB)
11
payment_details: Payment, // Full object (5KB)
12
inventory: Vec<Item>, // Array of objects (50KB)
13
}

Real-World Use Cases

E-commerce Order Processing

Order Saga · forward
Reserve inventory
Charge payment card
Create shipment label
Update order status
Send confirmation email
Compensation · on failure
Cancel shipment
Refund payment
Release inventory

If any step fails, only the already-committed steps compensate, in reverse order.

Travel Booking System

Travel Booking · forward
Reserve flight seats
Reserve hotel room
Book rental car
Process payment
Compensation · on failure
Release rental car
Cancel hotel reservation
Cancel flight seats

Compensation mirrors the forward path in reverse, releasing each reserved resource.

Financial Transaction Processing

Money Transfer · forward
Debit source account
Credit destination account
Record transaction
Send notifications
Compensation · on failure
Reverse credit
Reverse debit
Log failure

Reversing entries undo the transfer while preserving a durable audit trail.


Next Steps