Parallel Processing & Error Handling
A pipeline that works isn't the same as one that scales. This page is about the two things that break first under load: doing independent work in parallel instead of one row at a time, and keeping one poison record from taking down an entire run. Then a short note on running any of it on a schedule.
Parallel Fan-Out Processing
When your data is already partitioned — S3 date prefixes, table shards, per-tenant files — the batches are independent, so there's no reason to process them one after another. You've scaled this out before with Spark partitions or a Dask bag. The shape is always the same: fan out the work, then fan in the results.
Each partition is its own journaled task — one failing partition retries alone; the rest keep their cached results.
Step 1 — Discover and process one partition
Two tasks: list_partitions finds the independent units, process_partition does the full read-transform-load for exactly one. Because a partition is self-contained, its task is the natural retry boundary.
1@Task()2async listPartitions(ctx: TaskContext, source: string): Promise<string[]> {3 // e.g., list S3 prefixes, date partitions, or table shards4 return await storage.listPrefixes(source);5}67@Task()8async processPartition(ctx: TaskContext, partition: string): Promise<PartitionResult> {9 const data = await storage.readPartition(partition);10 const transformed = data.map(transformRecord);11 const loaded = await db.upsertRows('analytics', transformed);12 return { partition, rowsLoaded: loaded };13}Step 2 — Fan out, then fan in
The workflow starts every partition task without awaiting them one by one — Promise.all / try_join_all / asyncio.gather — so they run concurrently across your worker pool. Then it awaits them all and sums the results. Throughput scales with workers, not with a rewrite.
1@Workflow({ name: 'parallel_etl', version: '1.0.0' })2class ParallelEtl {3 async run(ctx: WorkflowContext, source: string): Promise<EtlResult> {4 const partitions = await ctx.executeTask(tasks.listPartitions, source);56 // Fan-out: start all partitions at once, then await them together7 const results = await Promise.all(8 partitions.map(p => ctx.executeTask(tasks.processPartition, p))9 );1011 // Fan-in: aggregate12 const totalRows = results.reduce((sum, r) => sum + r.rowsLoaded, 0);13 return { totalRows, partitions: results.length };14 }15}One caution the code hides: Promise.all starts every partition at once. With thousands of partitions that can overwhelm a downstream database — the Parallel ETL reference shows how to cap concurrency and why child workflows are the better tool once each partition grows beyond a single task.
Error Handling for Data Pipelines
Real data fails in two completely different ways, and conflating them is the classic pipeline mistake. A transient failure — a network blip, a 429 rate limit — will succeed if you just try again. A permanent failure — a malformed row, a schema mismatch — will fail identically forever. Retrying a permanent failure wastes time; giving up on a transient one loses data. Each needs its own answer.
Timeout, rate limit. Set a retry policy on the task; Orcher backs off and retries automatically. Your workflow code never sees the blip.
Bad data. Catch it, write it to a dead-letter table with the reason, and keep going. One poison row can't sink a million-row run.
Step 1 — Let transient failures retry themselves
The fetch task talks to a flaky API, so it carries a retry policy: up to five attempts with exponential backoff. A 429 or timeout just throws — Orcher catches it, waits, and re-runs the task. No retry loop in your code, because retrying transient faults is the runtime's job (the Retry with Backoff reference goes deep on this).
1// Transient failures: Orcher retries automatically2@Task({ retryPolicy: { maxAttempts: 5, initialInterval: 1000, backoffCoefficient: 2.0 } })3async fetchFromApi(ctx: TaskContext, input: { endpoint: string; page: number }): Promise<ApiResponse> {4 const response = await fetch(5 input.endpoint + '?page=' + input.page + '&size=100',6 { signal: AbortSignal.timeout(30_000) },7 );8 if (response.status === 429) throw new Error('Rate limited'); // retried with backoff9 if (!response.ok) throw new Error('API error: ' + response.status);10 return response.json();11}Step 2 — Quarantine permanent failures and continue
The processing task does the opposite: it catches the error so it never propagates. A row that can't be transformed goes to a failed_records dead-letter table — with its payload and the reason — and the task returns failed instead of throwing. The run continues; you triage the dead-letter table later, out of band.
1// Permanent failures: catch, log, continue2@Task()3async processRecordSafely(ctx: TaskContext, record: RawRecord): Promise<ProcessResult> {4 try {5 const transformed = transformRecord(record);6 await db.upsert('records', transformed);7 return { status: 'success', id: record.id };8 } catch (error) {9 // Log to dead-letter table for manual review10 await db.insert('failed_records', {11 record_id: record.id,12 error: String(error),13 payload: JSON.stringify(record),14 failed_at: new Date().toISOString(),15 });16 return { status: 'failed', id: record.id, error: String(error) };17 }18}Step 3 — Orchestrate: page through, tally both outcomes
The workflow pages through the API (each page a retry-protected fetch) and runs every record through the safe processor, counting successes and failures. It always finishes with a summary — never a crash — because the only errors that can escape are the transient ones the platform already retried away.
1@Workflow({ name: 'resilient_api_sync', version: '1.0.0' })2class ResilientApiSync {3 async run(ctx: WorkflowContext, endpoint: string): Promise<SyncSummary> {4 let page = 0, hasMore = true, succeeded = 0, failed = 0;56 while (hasMore) {7 const response = await ctx.executeTask(tasks.fetchFromApi, { endpoint, page });8 hasMore = response.hasMore;910 for (const record of response.data) {11 const result = await ctx.executeTask(tasks.processRecordSafely, record);12 if (result.status === 'success') succeeded++;13 else failed++;14 }15 page++;16 }1718 return { succeeded, failed, totalPages: page };19 }20}That returned failed count is your signal to go work the dead-letter table — fix the mapping, then re-run just those rows. A separate workflow can even do it on a timer.
Scheduled Recurring Pipelines
Most pipelines run on a clock — hourly syncs, nightly aggregations, weekly reports. Reflexively you'd reach for cron or a separate always-on scheduler that triggers the job. Orcher can be that scheduler itself. Model the recurrence as a long-running workflow that does its work, then ctx.sleep()s until the next run — a durable timer that releases the worker while it waits and fires on schedule even across restarts.
1@Workflow({ name: 'hourly_sync', version: '1.0.0' })2class HourlySyncScheduler {3 async run(ctx: WorkflowContext, config: SyncConfig): Promise<void> {4 while (true) {5 // Run the sync6 await ctx.executeTask(tasks.runIncrementalSync, config);78 // Sleep for 1 hour (durable — survives restarts)9 await ctx.sleep(Duration.fromHours(1));10 }11 }12}Start this workflow once and it runs forever — no cron entry, no scheduler service. Two things to know before you rely on it: a fixed sleep(1h) drifts (it fires an hour after the work finishes, not on the hour), and an unbounded loop grows its journal. Both are handled in the Scheduled Jobs reference, which also covers the Schedule API for when you want true cron semantics and pause/resume controls.