Getting Started

Retry with Exponential Backoff


External API calls fail — rate limits, transient errors, network timeouts. Rather than letting a single failure kill your entire workflow, catch the error and retry with exponential backoff using durable timers. The retry loop and backoff delays are all journaled, so if the process crashes between retries, it resumes at the correct attempt count without re-executing successful calls.

How it works

Each failed attempt doubles the wait before the next one — 1s, 2s, 4s, 8s, 16s — giving a struggling API room to recover instead of hammering it. Every attempt and every backoff sleep is a journaled step, so the attempt counter survives a crash: a restart resumes at the same attempt with the same remaining budget, never resetting the ladder to zero.

The Backoff Ladder
✗
call_external_apiattempt 0 — 503 Service Unavailable
⏱
sleep(1s)2^0 — durable timer
✗
call_external_apiattempt 1 — 503 again
⏱
sleep(2s)2^1
⏱
sleep(4s) · sleep(8s)2^2, 2^3 — attempts 2, 3 also fail
✓
call_external_apiattempt 4 — 200 OK
✓
store_resultSuccess cached — the loop returns
After max_retries the loop alerts and fails — the delays never double past your ceiling.

Example: a resilient API call

1
import {
2
Workflow,
3
Tasks,
4
Task,
5
WorkflowContext,
6
createTaskRefs,
7
} from '@orcher/sdk';
8
9
@Tasks()
10
class ApiTasks {
11
@Task()
12
async callExternalApi(request: ApiRequest): Promise<ApiResponse> {
13
return await externalApi.post(request.url, request.body);
14
}
15
16
@Task()
17
async storeResult(result: ApiResponse): Promise<void> {
18
await db.insertApiResult(result);
19
}
20
21
@Task()
22
async sendFailureAlert(input: FailureAlert): Promise<void> {
23
await alerting.send(input);
24
}
25
}
26
27
const tasks = createTaskRefs(ApiTasks);
28
29
@Workflow({ name: 'resilient_api_call', version: '1.0.0' })
30
class ResilientApiCall {
31
async run(
32
ctx: WorkflowContext,
33
request: ApiRequest,
34
): Promise<ApiResponse> {
35
const maxRetries = 5;
36
37
for (let attempt = 0; attempt <= maxRetries; attempt++) {
38
try {
39
const response = await ctx.executeTask(
40
tasks.callExternalApi, request,
41
);
42
await ctx.executeTask(tasks.storeResult, response);
43
return response;
44
} catch (error) {
45
if (attempt < maxRetries) {
46
// Exponential backoff: 1s, 2s, 4s, 8s, 16s
47
const delay = Math.pow(2, attempt) * 1000;
48
await ctx.sleep(delay);
49
} else {
50
// All retries exhausted — alert and fail
51
await ctx.executeTask(tasks.sendFailureAlert, {
52
url: request.url,
53
attempts: maxRetries + 1,
54
lastError: String(error),
55
});
56
throw error;
57
}
58
}
59
}
60
61
throw new Error('Unreachable');
62
}
63
}

Each retry attempt is a separate journaled task execution. If attempt 3 succeeds, the result is cached — on replay, Orcher returns the cached response without hitting the external API again. The exponential backoff delays use durable timers, so they're also replay-safe.

Don't reach for this too soon

Orcher already retries a failing task automatically before the error ever reaches your workflow code. The manual loop on this page is for the cases the automatic policy can't express — reach for it deliberately, not by default.

Automatic Task Retry vs. a Manual Loop
Automatic (default)
Transient faults

The platform retries the task on failure with its own backoff. Invisible to workflow code. Right for the common case — a flaky call that just needs another go.

Manual loop
Retry as business logic

You need side effects between attempts — alerting, a fallback provider, a custom ceiling, or a specific failure path. The loop makes the policy explicit and journaled.

Write the loop when what happens between and after attempts is part of the workflow's job.

The example earns its manual loop: it sends a send_failure_alert when retries exhaust and returns a specific error. If all you needed was "try a few more times," the automatic policy would already cover it.

Design notes

  • Add jitter to the delay. Pure 2^n backoff means every workflow that failed at the same moment retries at the same moment — a thundering herd that re-overloads the API you're backing off from. Add a random fraction to each delay so retries spread out. (Randomness must come from a task, not inline in the workflow — see the determinism rules.)
  • Retry only what retrying can fix. A 429 or 503 is worth waiting on; a 400 or 401 will fail identically every attempt. Inspect the error and fail fast on permanent ones instead of burning the whole ladder — 31 seconds spent re-sending a malformed request is 31 seconds wasted.
  • Cap total time, not just attempt count. Five attempts with doubling delays is ~31s of backoff; raising the ceiling to 10 attempts is ~17 minutes. If a caller is waiting on this workflow, bound the wall-clock budget explicitly rather than reasoning about it from the exponent.
  • Retries assume idempotency. A retried POST that already succeeded server-side (but whose response was lost) double-submits. Where the call isn't naturally idempotent, send an idempotency key — the same defense as payment processing.

Next Steps