Idempotency Patterns
Tasks should be designed to be safely retried without causing duplicate side effects.
Using Idempotency Keys
Pass a unique idempotency key with external API calls so that retries don't create duplicate side effects. The TaskContext provides a task token and attempt number that you can combine to build a stable key for each execution attempt.
1#[task(2name = "charge_card",3retry = 3 // ← Task may be retried up to 3 times4)]5pub async fn charge_card(6_ctx: TaskContext,7input: PaymentInput,8) -> Result<ChargeResult> {9// Get Stripe API client10let stripe_client = get_stripe_client();1112// Create charge with idempotency key13let charge = stripe::Charge::create(14 &stripe_client,15 ChargeParams {16 amount: input.amount, // e.g., 5000 (= $50.00)17 currency: "usd", // USD currency18 source: input.card_token, // Customer's card token1920 // ⭐ THE KEY LINE: Idempotency key prevents duplicate charges21 // Format: "order_12345"22 // If Stripe sees this key again, it returns the original charge23 // without creating a new one24 idempotency_key: Some(format!("order_{}", input.order_id)),25 ..Default::default()26 }27).await?;2829// Return charge details30Ok(ChargeResult {31 charge_id: charge.id, // e.g., "ch_abc123"32 amount: charge.amount, // 500033 status: "succeeded".to_string(),34})35}What happens on retry:
1// First execution (workflow run #1):2// 1. Orcher executes task3// 2. Stripe receives request with idempotency_key="order_12345"4// 3. Stripe creates charge, stores key5// 4. Returns charge_id="ch_abc123"6// 5. Orcher saves result to journal78// Second execution (if workflow retried):9// 1. Orcher checks journal first10// 2. Finds cached result: charge_id="ch_abc123"11// 3. Returns cached result immediately (NO API call to Stripe)12// 4. Customer charged only once ✓1314// Third execution (if task manually re-run outside workflow):15// 1. Task executes again16// 2. Stripe receives request with same idempotency_key="order_12345"17// 3. Stripe recognizes the key18// 4. Returns original charge_id="ch_abc123" (NO new charge created)19// 5. Customer still charged only once ✓Key Takeaways:
- Idempotency key should be based on workflow/order ID, not timestamps
- Use same key format consistently:
order_{orderId},payment_{paymentId} - External system (Stripe) deduplicates based on key
- Orcher's journal provides first layer of deduplication
- Idempotency key provides second layer (if task runs outside workflow)
Strategy 2: Database Upsert Pattern
Concept: Use database operations that are naturally idempotent (INSERT ON CONFLICT, UPSERT, atomic UPDATE).
When to Use:
- Database writes
- Inventory management
- Counter updates
- Status changes
How It Works:
Instead of read-modify-write (which is NOT idempotent), use atomic database operations that produce the same result when run multiple times.
Complete Example with Breakdown:
1#[task(name = "update_inventory")]2pub async fn update_inventory(3_ctx: TaskContext,4input: InventoryUpdate,5) -> Result<()> {6let db = get_database();78// Use UPSERT (INSERT ... ON CONFLICT) for idempotency9db.execute(10 "INSERT INTO inventory (product_id, quantity, updated_at)11 VALUES ($1, $2, NOW())12 ON CONFLICT (product_id)13 DO UPDATE SET14 quantity = $2,15 updated_at = NOW()",16 &[&input.product_id, &input.quantity]17).await?;1819Ok(())20}Check-Before-Execute Pattern
Before performing a non-idempotent operation, check whether it has already been done. This pattern is useful when the external system doesn't support idempotency keys — you query the current state first and skip the operation if it was already applied.
1#[task(name = "send_welcome_email")]2pub async fn send_welcome_email(3_ctx: TaskContext,4user_id: String,5) -> Result<EmailResult> {6let db = get_database();78// Check if email was already sent9let already_sent: bool = db.query_one(10 "SELECT EXISTS(11 SELECT 1 FROM sent_emails12 WHERE user_id = $1 AND email_type = 'welcome'13 )",14 &[&user_id]15).await?;1617if already_sent {18 tracing::info!("Email already sent to user {}", user_id);19 return Ok(EmailResult {20 message_id: "already_sent".to_string(),21 sent: false,22 });23}2425// Send email26let result = email_service::send_welcome(&user_id).await?;2728// Record that email was sent29db.execute(30 "INSERT INTO sent_emails (user_id, email_type, message_id)31 VALUES ($1, 'welcome', $2)",32 &[&user_id, &result.message_id]33).await?;3435Ok(result)36}Error Handling
Tasks should handle errors appropriately and return meaningful error messages.
1use anyhow::{Context, Result};23#[task(4name = "fetch_external_data",5retry = 56)]7pub async fn fetch_external_data(8ctx: TaskContext,9url: String,10) -> Result<ApiResponse> {11let response = reqwest::get(&url)12 .await13 .context("Failed to connect to API")?;1415match response.status() {16 status if status.is_success() => {17 let data = response.json().await?;18 Ok(data)19 }20 status if status.is_server_error() => {21 // 5xx errors are transient - should retry22 Err(anyhow::anyhow!(23 "Server error (will retry): HTTP {}",24 status25 ))26 }27 status => {28 // 4xx errors are permanent - don't retry29 Err(anyhow::anyhow!(30 "Client error (permanent): HTTP {}",31 status32 ))33 }34}35}