Best Practices & Pitfalls
Best Practices
1. Make Compensation Idempotent
Compensation actions may be retried if they fail or if the workflow replays. Always check the current state of the resource before performing the compensation. If the resource has already been compensated (e.g., a payment is already refunded), return success without performing the action again.
1// Good - Idempotent compensation2#[task(name = "refund_payment")]3pub async fn refund_payment(4 ctx: TaskContext,5 payment_id: String,6) -> Result<()> {7 // Check if already refunded8 let payment_status = get_payment_status(&payment_id).await?;910 if payment_status == "refunded" {11 println!("Payment {} already refunded, skipping", payment_id);12 return Ok(());13 }1415 // Issue refund16 payment_gateway::refund(payment_id).await?;17 Ok(())18}1920// Bad - Not idempotent (could double-refund)21#[task(name = "refund_payment_bad")]22pub async fn refund_payment_bad(23 ctx: TaskContext,24 payment_id: String,25) -> Result<()> {26 // Blindly refunds without checking current state27 payment_gateway::refund(payment_id).await?;28 Ok(())29}2. Compensate in Reverse Order
Always undo operations in the opposite order they were executed. This is important because later steps may depend on the state created by earlier steps. For example, canceling a shipment before refunding the payment ensures the payment refund doesn't trigger a conflicting re-shipment.
1// Good - Reverse order compensation2async fn compensate_saga(ctx: &WorkflowContext, state: &SagaState) -> Result<()> {3 // Last operation first4 if state.shipment_created {5 cancel_shipment(ctx, &state.shipment_id).await?;6 }78 // Then payment9 if state.payment_captured {10 refund_payment(ctx, &state.payment_id).await?;11 }1213 // First operation last14 if state.inventory_reserved {15 release_inventory(ctx, &state.inventory_id).await?;16 }1718 Ok(())19}3. Store Compensation Data
Every saga step must store the IDs and metadata needed for its compensation action. Without these identifiers, compensation becomes impossible. Store resource IDs, amounts (for partial refunds), and timestamps (for audit trails) immediately after each step succeeds.
1// Good - Stores all data needed for compensation2#[derive(Debug, Clone, Serialize, Deserialize)]3struct SagaState {4 payment_id: Option<String>,5 inventory_id: Option<String>,6 shipment_id: Option<String>,7 // Store amounts for partial refunds8 payment_amount: Option<f64>,9 // Store timestamps for audit10 payment_captured_at: Option<DateTime<Utc>>,11}1213// Bad - Missing compensation data14#[derive(Debug, Clone, Serialize, Deserialize)]15struct SagaStateBad {16 payment_captured: bool,17 inventory_reserved: bool,18 // No IDs stored - can't compensate!19}4. Handle Compensation Failures
Compensation itself can fail — a refund API might be temporarily down, or a cancellation service might time out. When this happens, don't stop compensating the remaining steps. Instead, attempt all compensations, collect any failures, and alert the operations team for manual resolution. This ensures partial compensation doesn't leave the system in a worse state.
1// Good - Handles compensation failures without stopping2async fn compensate_saga_robust(3 ctx: &WorkflowContext,4 state: &SagaState,5) -> Result<()> {6 let mut failures = Vec::new();78 if state.payment_captured {9 if let Err(e) = refund_payment(ctx, &state.payment_id).await {10 println!("Refund failed: {}", e);11 failures.push(("payment", e));12 // Continue with other compensations13 }14 }1516 if state.inventory_reserved {17 if let Err(e) = release_inventory(ctx, &state.inventory_id).await {18 println!("Inventory release failed: {}", e);19 failures.push(("inventory", e));20 }21 }2223 // If any compensation failed, alert operations team24 if !failures.is_empty() {25 ctx.execute_task(alert_ops_team, failures).await?;26 }2728 Ok(())29}5. Use Semantic Compensation
Prefer semantic undo (marking a resource as "cancelled" with a timestamp) over physical undo (deleting the record). Semantic compensation preserves the audit trail and makes it possible to analyze what happened, replay events, or manually recover. Physical deletion loses this history permanently.
1// Good - Semantic compensation (preserves audit trail)2#[task(name = "cancel_reservation")]3pub async fn cancel_reservation(4 ctx: TaskContext,5 reservation_id: String,6) -> Result<()> {7 database::update(8 "reservations",9 &reservation_id,10 json!({ "status": "cancelled", "cancelled_at": Utc::now() })11 ).await?;12 Ok(())13}1415// Less good - Physical undo (loses audit trail)16#[task(name = "delete_reservation")]17pub async fn delete_reservation(18 ctx: TaskContext,19 reservation_id: String,20) -> Result<()> {21 database::delete("reservations", &reservation_id).await?;22 Ok(())23}6. Set Appropriate Timeouts
When waiting for external systems in an event-driven saga, always set a timeout. Without one, the workflow could hang indefinitely if the external service never responds. When a timeout fires, treat it as a step failure and trigger compensation for all previously completed steps.
1// Good - Timeouts for each step2let payment_event = ctx3 .wait_for_event_with_timeout("payment_result", Duration::from_secs(300))4 .await?;56match payment_event {7 Some(event) => handle_event(event).await?,8 None => {9 // Timeout - compensate and fail gracefully10 compensate_saga(&ctx, &state).await?;11 return Err(Error::PaymentTimeout);12 }13}Saga vs Two-Phase Commit
Understanding when to use a saga versus a traditional two-phase commit helps you choose the right consistency model for your use case. Sagas trade immediate consistency for higher availability and throughput, making them the better choice for microservices architectures and long-running transactions.
| Aspect | Saga Pattern | Two-Phase Commit |
|---|---|---|
| Locking | No distributed locks | Requires locks on all participants |
| Availability | High - services independent | Low - all must be available |
| Consistency | Eventual consistency | Immediate consistency |
| Performance | High throughput | Lower throughput |
| Complexity | Moderate (compensation logic) | High (coordinator + protocol) |
| Use Case | Microservices, long transactions | Traditional databases, short transactions |
Common Pitfalls
Below are the most frequent mistakes developers make when implementing sagas. Each pitfall can lead to data inconsistency, stuck workflows, or double-charged customers.
Non-Idempotent Operations
Without idempotency keys, retrying a step can cause duplicate side effects like charging a customer twice.
1// Bad - Can charge customer multiple times on retry2async fn charge_customer(amount: f64) -> Result<String> {3 stripe::charge(amount).await4}56// Good - Idempotent with unique key7async fn charge_customer(amount: f64, idempotency_key: String) -> Result<String> {8 stripe::charge_with_key(amount, idempotency_key).await9}Lost Compensation Data
If you forget to store the result ID from a step, you won't be able to compensate it later.
1// Bad - Can't compensate without IDs2let payment_result = charge_payment(amount).await?;3// Forgot to store payment_result.id!45// Good - Store compensation data immediately6let payment_id = charge_payment(amount).await?;7state.payment_id = Some(payment_id);Partial Compensation
If the first compensation step fails and throws an error, subsequent compensation steps never run — leaving the system in a partially compensated state.
1// Bad - Stops compensating on first error2if state.payment_captured {3 refund_payment(&state.payment_id).await?; // Throws error4}5if state.inventory_reserved {6 release_inventory(&state.inventory_id).await?; // Never runs!7}89// Good - Try all compensations regardless of failures10let mut errors = Vec::new();11if state.payment_captured {12 if let Err(e) = refund_payment(&state.payment_id).await {13 errors.push(e);14 }15}16if state.inventory_reserved {17 if let Err(e) = release_inventory(&state.inventory_id).await {18 errors.push(e);19 }20}Performance Considerations
Saga Overhead
Each saga step adds latency due to task execution, state persistence, and journal recording. The total time depends on your saga type:
- Sequential saga — Total time = sum of all step durations
- Parallel saga — Total time = duration of the slowest step
- Compensation — Adds additional time on the failure path
Rule of thumb: Use sagas for operations that take >100ms per step. For faster operations, consider optimistic locking or single-transaction approaches.
State Storage
Saga state is persisted after each step for durability. Keep state small by storing references (IDs) rather than full objects:
1// Good - Store references (~100 bytes)2struct SagaState {3 order_id: String,4 payment_id: String,5 inventory_id: String,6}78// Bad - Store full objects (~65KB)9struct SagaStateBad {10 order: Order, // Full object (10KB)11 payment_details: Payment, // Full object (5KB)12 inventory: Vec<Item>, // Array of objects (50KB)13}Real-World Use Cases
E-commerce Order Processing
If any step fails, only the already-committed steps compensate, in reverse order.
Travel Booking System
Compensation mirrors the forward path in reverse, releasing each reserved resource.
Financial Transaction Processing
Reversing entries undo the transfer while preserving a durable audit trail.