Getting Started

Batch Processing


The most common data pipeline pattern: break a large dataset into manageable chunks and process each batch as a separate task. Every batch completion is journaled, so crashes only require reprocessing the current in-flight batch — not the entire dataset.

Use this pattern when a dataset is too large to process in one step, when you want progress to survive crashes and deployments, or when partial failures should not force a full re-run. If throughput matters more than ordering, combine it with Parallel ETL to run batches concurrently.

How it works

The workflow is a plain loop. Each iteration extracts, transforms, and loads one batch — and because each of those steps runs through execute_task, each step's result is journaled the moment it completes. There is no explicit checkpointing code: the journal is the checkpoint.

One Batch Cycle
1
fetch_batchExtract records at offset — result journaled
2
transform_batchClean, normalize, enrich — result journaled
3
load_batchIdempotent upsert — result journaled
↻ repeat for batches 001 … N — every completed step is a checkpoint

Example: a checkpointed ETL pipeline

The pipeline below copies total_records from a source table to a destination table in fixed-size batches. Note what it does not contain: no checkpoint bookkeeping, no resume logic, no state table — the loop index and all intermediate results live in the journal.

1
use orcher::prelude::*;
2
use anyhow::Result;
3
use serde::{Deserialize, Serialize};
4
5
#[derive(Debug, Clone, Serialize, Deserialize)]
6
struct BatchConfig {
7
source_table: String,
8
dest_table: String,
9
batch_size: usize,
10
total_records: usize,
11
}
12
13
#[derive(Debug, Clone, Serialize, Deserialize)]
14
struct BatchResult {
15
records_processed: usize,
16
batches_completed: usize,
17
failed_batches: Vec<usize>,
18
}
19
20
#[derive(Debug, Clone, Serialize, Deserialize)]
21
struct RecordBatch {
22
records: Vec<serde_json::Value>,
23
batch_index: usize,
24
}
25
26
#[task]
27
async fn fetch_batch(source: String, offset: usize, limit: usize) -> Result<RecordBatch> {
28
// Query source database for a batch of records
29
let records = db::query(&source, offset, limit).await?;
30
Ok(RecordBatch { records, batch_index: offset / limit })
31
}
32
33
#[task]
34
async fn transform_batch(batch: RecordBatch) -> Result<RecordBatch> {
35
// Apply transformations: clean, normalize, enrich
36
let transformed = batch.records.into_iter()
37
.map(|r| apply_transforms(r))
38
.collect();
39
Ok(RecordBatch { records: transformed, batch_index: batch.batch_index })
40
}
41
42
#[task]
43
async fn load_batch(dest: String, batch: RecordBatch) -> Result<usize> {
44
// Upsert into destination — idempotent for safe retries
45
let count = db::upsert_batch(&dest, &batch.records).await?;
46
Ok(count)
47
}
48
49
#[workflow(name = "batch_etl_pipeline", version = "1.0.0")]
50
pub async fn batch_etl_pipeline(
51
ctx: WorkflowContext,
52
config: BatchConfig,
53
) -> Result<BatchResult> {
54
let num_batches = (config.total_records + config.batch_size - 1) / config.batch_size;
55
let mut records_processed = 0;
56
let mut failed_batches = Vec::new();
57
58
for batch_idx in 0..num_batches {
59
let offset = batch_idx * config.batch_size;
60
61
// Extract
62
let batch = ctx.execute_task(fetch_batch, (
63
config.source_table.clone(), offset, config.batch_size
64
)).await?;
65
66
// Transform
67
let transformed = ctx.execute_task(transform_batch, batch).await?;
68
69
// Load (idempotent upsert)
70
match ctx.execute_task(load_batch, (
71
config.dest_table.clone(), transformed
72
)).await {
73
Ok(count) => records_processed += count,
74
Err(e) => {
75
println!("Batch {} failed: {}", batch_idx, e);
76
failed_batches.push(batch_idx);
77
}
78
}
79
80
// Each loop iteration is automatically checkpointed
81
}
82
83
Ok(BatchResult {
84
records_processed,
85
batches_completed: num_batches - failed_batches.len(),
86
failed_batches,
87
})
88
}

Crash recovery, step by step

Suppose a worker dies while loading batch 042. Here is exactly what happens when the workflow is re-dispatched:

Recovery of Batch 042
✓
fetch_batch(042)Completed — journaled
✓
transform_batch(042)Completed — journaled
✗
load_batch(042)Worker dies — no journal entry
↻
replayfetch & transform return journaled results — not re-executed
✓
load_batch(042)Re-executed — journaled on success
Only the step without a journal entry re-executes. Batches 001–041 replay the same way — instantly.

Replay walks the workflow function from the top, but every execute_task call that already has a journal entry returns its recorded result without running the task again. The loop fast-forwards through batches 001–041 and the completed steps of batch 042 in milliseconds, then resumes real work at the failed load_batch call.

This is also why the load step must be idempotent: if the worker died after the upsert reached the database but before the journal entry was written, the retry will run the upsert again. An idempotent write (upsert by key) makes that harmless.

Design notes

  • Batch size is a checkpoint-granularity dial. Smaller batches mean finer checkpoints but more journal overhead; larger batches lose more work on failure. See Choose the Right Batch Size for concrete numbers.
  • Collect failures, don't abort. The example records failed batch indexes and keeps going, so one bad batch doesn't sink a million-record run. The returned failed_batches list is your input for a targeted reprocessing pass. If any failure should stop the pipeline, propagate the error instead of catching it. See Handle Partial Failures Gracefully.
  • Keep task payloads lean. Task inputs and results flow through the journal. Passing offsets and table names (as this example does with fetch_batch) keeps journal entries small; avoid threading multi-megabyte blobs through task results when a reference will do.

Next Steps