Guides

Idempotency Patterns


Tasks should be designed to be safely retried without causing duplicate side effects.

Using Idempotency Keys

Pass a unique idempotency key with external API calls so that retries don't create duplicate side effects. The TaskContext provides a task token and attempt number that you can combine to build a stable key for each execution attempt.

1
#[task(
2
name = "charge_card",
3
retry = 3 // ← Task may be retried up to 3 times
4
)]
5
pub async fn charge_card(
6
_ctx: TaskContext,
7
input: PaymentInput,
8
) -> Result<ChargeResult> {
9
// Get Stripe API client
10
let stripe_client = get_stripe_client();
11
12
// Create charge with idempotency key
13
let charge = stripe::Charge::create(
14
&stripe_client,
15
ChargeParams {
16
amount: input.amount, // e.g., 5000 (= $50.00)
17
currency: "usd", // USD currency
18
source: input.card_token, // Customer's card token
19
20
// ⭐ THE KEY LINE: Idempotency key prevents duplicate charges
21
// Format: "order_12345"
22
// If Stripe sees this key again, it returns the original charge
23
// without creating a new one
24
idempotency_key: Some(format!("order_{}", input.order_id)),
25
..Default::default()
26
}
27
).await?;
28
29
// Return charge details
30
Ok(ChargeResult {
31
charge_id: charge.id, // e.g., "ch_abc123"
32
amount: charge.amount, // 5000
33
status: "succeeded".to_string(),
34
})
35
}

What happens on retry:

1
// First execution (workflow run #1):
2
// 1. Orcher executes task
3
// 2. Stripe receives request with idempotency_key="order_12345"
4
// 3. Stripe creates charge, stores key
5
// 4. Returns charge_id="ch_abc123"
6
// 5. Orcher saves result to journal
7
8
// Second execution (if workflow retried):
9
// 1. Orcher checks journal first
10
// 2. Finds cached result: charge_id="ch_abc123"
11
// 3. Returns cached result immediately (NO API call to Stripe)
12
// 4. Customer charged only once ✓
13
14
// Third execution (if task manually re-run outside workflow):
15
// 1. Task executes again
16
// 2. Stripe receives request with same idempotency_key="order_12345"
17
// 3. Stripe recognizes the key
18
// 4. Returns original charge_id="ch_abc123" (NO new charge created)
19
// 5. Customer still charged only once ✓

Key Takeaways:

  • Idempotency key should be based on workflow/order ID, not timestamps
  • Use same key format consistently: order_{orderId}, payment_{paymentId}
  • External system (Stripe) deduplicates based on key
  • Orcher's journal provides first layer of deduplication
  • Idempotency key provides second layer (if task runs outside workflow)

Strategy 2: Database Upsert Pattern

Concept: Use database operations that are naturally idempotent (INSERT ON CONFLICT, UPSERT, atomic UPDATE).

When to Use:

  • Database writes
  • Inventory management
  • Counter updates
  • Status changes

How It Works:

Instead of read-modify-write (which is NOT idempotent), use atomic database operations that produce the same result when run multiple times.

Complete Example with Breakdown:

1
#[task(name = "update_inventory")]
2
pub async fn update_inventory(
3
_ctx: TaskContext,
4
input: InventoryUpdate,
5
) -> Result<()> {
6
let db = get_database();
7
8
// Use UPSERT (INSERT ... ON CONFLICT) for idempotency
9
db.execute(
10
"INSERT INTO inventory (product_id, quantity, updated_at)
11
VALUES ($1, $2, NOW())
12
ON CONFLICT (product_id)
13
DO UPDATE SET
14
quantity = $2,
15
updated_at = NOW()",
16
&[&input.product_id, &input.quantity]
17
).await?;
18
19
Ok(())
20
}

Check-Before-Execute Pattern

Before performing a non-idempotent operation, check whether it has already been done. This pattern is useful when the external system doesn't support idempotency keys — you query the current state first and skip the operation if it was already applied.

1
#[task(name = "send_welcome_email")]
2
pub async fn send_welcome_email(
3
_ctx: TaskContext,
4
user_id: String,
5
) -> Result<EmailResult> {
6
let db = get_database();
7
8
// Check if email was already sent
9
let already_sent: bool = db.query_one(
10
"SELECT EXISTS(
11
SELECT 1 FROM sent_emails
12
WHERE user_id = $1 AND email_type = 'welcome'
13
)",
14
&[&user_id]
15
).await?;
16
17
if already_sent {
18
tracing::info!("Email already sent to user {}", user_id);
19
return Ok(EmailResult {
20
message_id: "already_sent".to_string(),
21
sent: false,
22
});
23
}
24
25
// Send email
26
let result = email_service::send_welcome(&user_id).await?;
27
28
// Record that email was sent
29
db.execute(
30
"INSERT INTO sent_emails (user_id, email_type, message_id)
31
VALUES ($1, 'welcome', $2)",
32
&[&user_id, &result.message_id]
33
).await?;
34
35
Ok(result)
36
}

Error Handling

Tasks should handle errors appropriately and return meaningful error messages.

1
use anyhow::{Context, Result};
2
3
#[task(
4
name = "fetch_external_data",
5
retry = 5
6
)]
7
pub async fn fetch_external_data(
8
ctx: TaskContext,
9
url: String,
10
) -> Result<ApiResponse> {
11
let response = reqwest::get(&url)
12
.await
13
.context("Failed to connect to API")?;
14
15
match response.status() {
16
status if status.is_success() => {
17
let data = response.json().await?;
18
Ok(data)
19
}
20
status if status.is_server_error() => {
21
// 5xx errors are transient - should retry
22
Err(anyhow::anyhow!(
23
"Server error (will retry): HTTP {}",
24
status
25
))
26
}
27
status => {
28
// 4xx errors are permanent - don't retry
29
Err(anyhow::anyhow!(
30
"Client error (permanent): HTTP {}",
31
status
32
))
33
}
34
}
35
}

Next Steps