Retry with Exponential Backoff
External API calls fail — rate limits, transient errors, network timeouts. Rather than letting a single failure kill your entire workflow, catch the error and retry with exponential backoff using durable timers. The retry loop and backoff delays are all journaled, so if the process crashes between retries, it resumes at the correct attempt count without re-executing successful calls.
How it works
Each failed attempt doubles the wait before the next one — 1s, 2s, 4s, 8s, 16s — giving a struggling API room to recover instead of hammering it. Every attempt and every backoff sleep is a journaled step, so the attempt counter survives a crash: a restart resumes at the same attempt with the same remaining budget, never resetting the ladder to zero.
Example: a resilient API call
1import {2 Workflow,3 Tasks,4 Task,5 WorkflowContext,6 createTaskRefs,7} from '@orcher/sdk';89@Tasks()10class ApiTasks {11 @Task()12 async callExternalApi(request: ApiRequest): Promise<ApiResponse> {13 return await externalApi.post(request.url, request.body);14 }1516 @Task()17 async storeResult(result: ApiResponse): Promise<void> {18 await db.insertApiResult(result);19 }2021 @Task()22 async sendFailureAlert(input: FailureAlert): Promise<void> {23 await alerting.send(input);24 }25}2627const tasks = createTaskRefs(ApiTasks);2829@Workflow({ name: 'resilient_api_call', version: '1.0.0' })30class ResilientApiCall {31 async run(32 ctx: WorkflowContext,33 request: ApiRequest,34 ): Promise<ApiResponse> {35 const maxRetries = 5;3637 for (let attempt = 0; attempt <= maxRetries; attempt++) {38 try {39 const response = await ctx.executeTask(40 tasks.callExternalApi, request,41 );42 await ctx.executeTask(tasks.storeResult, response);43 return response;44 } catch (error) {45 if (attempt < maxRetries) {46 // Exponential backoff: 1s, 2s, 4s, 8s, 16s47 const delay = Math.pow(2, attempt) * 1000;48 await ctx.sleep(delay);49 } else {50 // All retries exhausted — alert and fail51 await ctx.executeTask(tasks.sendFailureAlert, {52 url: request.url,53 attempts: maxRetries + 1,54 lastError: String(error),55 });56 throw error;57 }58 }59 }6061 throw new Error('Unreachable');62 }63}Each retry attempt is a separate journaled task execution. If attempt 3 succeeds, the result is cached — on replay, Orcher returns the cached response without hitting the external API again. The exponential backoff delays use durable timers, so they're also replay-safe.
Don't reach for this too soon
Orcher already retries a failing task automatically before the error ever reaches your workflow code. The manual loop on this page is for the cases the automatic policy can't express — reach for it deliberately, not by default.
The platform retries the task on failure with its own backoff. Invisible to workflow code. Right for the common case — a flaky call that just needs another go.
You need side effects between attempts — alerting, a fallback provider, a custom ceiling, or a specific failure path. The loop makes the policy explicit and journaled.
The example earns its manual loop: it sends a send_failure_alert when retries exhaust and returns a specific error. If all you needed was "try a few more times," the automatic policy would already cover it.
Design notes
- Add jitter to the delay. Pure
2^nbackoff means every workflow that failed at the same moment retries at the same moment — a thundering herd that re-overloads the API you're backing off from. Add a random fraction to each delay so retries spread out. (Randomness must come from a task, not inline in the workflow — see the determinism rules.) - Retry only what retrying can fix. A
429or503is worth waiting on; a400or401will fail identically every attempt. Inspect the error and fail fast on permanent ones instead of burning the whole ladder — 31 seconds spent re-sending a malformed request is 31 seconds wasted. - Cap total time, not just attempt count. Five attempts with doubling delays is ~31s of backoff; raising the ceiling to 10 attempts is ~17 minutes. If a caller is waiting on this workflow, bound the wall-clock budget explicitly rather than reasoning about it from the exponent.
- Retries assume idempotency. A retried
POSTthat already succeeded server-side (but whose response was lost) double-submits. Where the call isn't naturally idempotent, send an idempotency key — the same defense as payment processing.