Getting Started

Parallel Processing & Error Handling


A pipeline that works isn't the same as one that scales. This page is about the two things that break first under load: doing independent work in parallel instead of one row at a time, and keeping one poison record from taking down an entire run. Then a short note on running any of it on a schedule.

Parallel Fan-Out Processing

When your data is already partitioned — S3 date prefixes, table shards, per-tenant files — the batches are independent, so there's no reason to process them one after another. You've scaled this out before with Spark partitions or a Dask bag. The shape is always the same: fan out the work, then fan in the results.

list_partitions
discover the independent units
fan out — all at once
process_partition
2024-01
own journal & retries
process_partition
2024-02
own journal & retries
process_partition
2024-03
own journal & retries
fan in — sum the results
totalRows

Each partition is its own journaled task — one failing partition retries alone; the rest keep their cached results.

Step 1 — Discover and process one partition

Two tasks: list_partitions finds the independent units, process_partition does the full read-transform-load for exactly one. Because a partition is self-contained, its task is the natural retry boundary.

1
@Task()
2
async listPartitions(ctx: TaskContext, source: string): Promise<string[]> {
3
// e.g., list S3 prefixes, date partitions, or table shards
4
return await storage.listPrefixes(source);
5
}
6
7
@Task()
8
async processPartition(ctx: TaskContext, partition: string): Promise<PartitionResult> {
9
const data = await storage.readPartition(partition);
10
const transformed = data.map(transformRecord);
11
const loaded = await db.upsertRows('analytics', transformed);
12
return { partition, rowsLoaded: loaded };
13
}

Step 2 — Fan out, then fan in

The workflow starts every partition task without awaiting them one by one — Promise.all / try_join_all / asyncio.gather — so they run concurrently across your worker pool. Then it awaits them all and sums the results. Throughput scales with workers, not with a rewrite.

1
@Workflow({ name: 'parallel_etl', version: '1.0.0' })
2
class ParallelEtl {
3
async run(ctx: WorkflowContext, source: string): Promise<EtlResult> {
4
const partitions = await ctx.executeTask(tasks.listPartitions, source);
5
6
// Fan-out: start all partitions at once, then await them together
7
const results = await Promise.all(
8
partitions.map(p => ctx.executeTask(tasks.processPartition, p))
9
);
10
11
// Fan-in: aggregate
12
const totalRows = results.reduce((sum, r) => sum + r.rowsLoaded, 0);
13
return { totalRows, partitions: results.length };
14
}
15
}

One caution the code hides: Promise.all starts every partition at once. With thousands of partitions that can overwhelm a downstream database — the Parallel ETL reference shows how to cap concurrency and why child workflows are the better tool once each partition grows beyond a single task.


Error Handling for Data Pipelines

Real data fails in two completely different ways, and conflating them is the classic pipeline mistake. A transient failure — a network blip, a 429 rate limit — will succeed if you just try again. A permanent failure — a malformed row, a schema mismatch — will fail identically forever. Retrying a permanent failure wastes time; giving up on a transient one loses data. Each needs its own answer.

Two Failures, Two Answers
Transient
Retry — the platform's job

Timeout, rate limit. Set a retry policy on the task; Orcher backs off and retries automatically. Your workflow code never sees the blip.

Permanent
Quarantine — your job

Bad data. Catch it, write it to a dead-letter table with the reason, and keep going. One poison row can't sink a million-row run.

Where a traditional pipeline needs a retry library and a DLQ topic, these are a task policy and a try/catch.

Step 1 — Let transient failures retry themselves

The fetch task talks to a flaky API, so it carries a retry policy: up to five attempts with exponential backoff. A 429 or timeout just throws — Orcher catches it, waits, and re-runs the task. No retry loop in your code, because retrying transient faults is the runtime's job (the Retry with Backoff reference goes deep on this).

1
// Transient failures: Orcher retries automatically
2
@Task({ retryPolicy: { maxAttempts: 5, initialInterval: 1000, backoffCoefficient: 2.0 } })
3
async fetchFromApi(ctx: TaskContext, input: { endpoint: string; page: number }): Promise<ApiResponse> {
4
const response = await fetch(
5
input.endpoint + '?page=' + input.page + '&size=100',
6
{ signal: AbortSignal.timeout(30_000) },
7
);
8
if (response.status === 429) throw new Error('Rate limited'); // retried with backoff
9
if (!response.ok) throw new Error('API error: ' + response.status);
10
return response.json();
11
}

Step 2 — Quarantine permanent failures and continue

The processing task does the opposite: it catches the error so it never propagates. A row that can't be transformed goes to a failed_records dead-letter table — with its payload and the reason — and the task returns failed instead of throwing. The run continues; you triage the dead-letter table later, out of band.

1
// Permanent failures: catch, log, continue
2
@Task()
3
async processRecordSafely(ctx: TaskContext, record: RawRecord): Promise<ProcessResult> {
4
try {
5
const transformed = transformRecord(record);
6
await db.upsert('records', transformed);
7
return { status: 'success', id: record.id };
8
} catch (error) {
9
// Log to dead-letter table for manual review
10
await db.insert('failed_records', {
11
record_id: record.id,
12
error: String(error),
13
payload: JSON.stringify(record),
14
failed_at: new Date().toISOString(),
15
});
16
return { status: 'failed', id: record.id, error: String(error) };
17
}
18
}

Step 3 — Orchestrate: page through, tally both outcomes

The workflow pages through the API (each page a retry-protected fetch) and runs every record through the safe processor, counting successes and failures. It always finishes with a summary — never a crash — because the only errors that can escape are the transient ones the platform already retried away.

1
@Workflow({ name: 'resilient_api_sync', version: '1.0.0' })
2
class ResilientApiSync {
3
async run(ctx: WorkflowContext, endpoint: string): Promise<SyncSummary> {
4
let page = 0, hasMore = true, succeeded = 0, failed = 0;
5
6
while (hasMore) {
7
const response = await ctx.executeTask(tasks.fetchFromApi, { endpoint, page });
8
hasMore = response.hasMore;
9
10
for (const record of response.data) {
11
const result = await ctx.executeTask(tasks.processRecordSafely, record);
12
if (result.status === 'success') succeeded++;
13
else failed++;
14
}
15
page++;
16
}
17
18
return { succeeded, failed, totalPages: page };
19
}
20
}

That returned failed count is your signal to go work the dead-letter table — fix the mapping, then re-run just those rows. A separate workflow can even do it on a timer.


Scheduled Recurring Pipelines

Most pipelines run on a clock — hourly syncs, nightly aggregations, weekly reports. Reflexively you'd reach for cron or a separate always-on scheduler that triggers the job. Orcher can be that scheduler itself. Model the recurrence as a long-running workflow that does its work, then ctx.sleep()s until the next run — a durable timer that releases the worker while it waits and fires on schedule even across restarts.

1
@Workflow({ name: 'hourly_sync', version: '1.0.0' })
2
class HourlySyncScheduler {
3
async run(ctx: WorkflowContext, config: SyncConfig): Promise<void> {
4
while (true) {
5
// Run the sync
6
await ctx.executeTask(tasks.runIncrementalSync, config);
7
8
// Sleep for 1 hour (durable — survives restarts)
9
await ctx.sleep(Duration.fromHours(1));
10
}
11
}
12
}

Start this workflow once and it runs forever — no cron entry, no scheduler service. Two things to know before you rely on it: a fixed sleep(1h) drifts (it fires an hour after the work finishes, not on the hour), and an unbounded loop grows its journal. Both are handled in the Scheduled Jobs reference, which also covers the Schedule API for when you want true cron semantics and pause/resume controls.


Next Steps