Batch Processing
The most common data pipeline pattern: break a large dataset into manageable chunks and process each batch as a separate task. Every batch completion is journaled, so crashes only require reprocessing the current in-flight batch — not the entire dataset.
Use this pattern when a dataset is too large to process in one step, when you want progress to survive crashes and deployments, or when partial failures should not force a full re-run. If throughput matters more than ordering, combine it with Parallel ETL to run batches concurrently.
How it works
The workflow is a plain loop. Each iteration extracts, transforms, and loads one batch — and because each of those steps runs through execute_task, each step's result is journaled the moment it completes. There is no explicit checkpointing code: the journal is the checkpoint.
Example: a checkpointed ETL pipeline
The pipeline below copies total_records from a source table to a destination table in fixed-size batches. Note what it does not contain: no checkpoint bookkeeping, no resume logic, no state table — the loop index and all intermediate results live in the journal.
1use orcher::prelude::*;2use anyhow::Result;3use serde::{Deserialize, Serialize};45#[derive(Debug, Clone, Serialize, Deserialize)]6struct BatchConfig {7 source_table: String,8 dest_table: String,9 batch_size: usize,10 total_records: usize,11}1213#[derive(Debug, Clone, Serialize, Deserialize)]14struct BatchResult {15 records_processed: usize,16 batches_completed: usize,17 failed_batches: Vec<usize>,18}1920#[derive(Debug, Clone, Serialize, Deserialize)]21struct RecordBatch {22 records: Vec<serde_json::Value>,23 batch_index: usize,24}2526#[task]27async fn fetch_batch(source: String, offset: usize, limit: usize) -> Result<RecordBatch> {28 // Query source database for a batch of records29 let records = db::query(&source, offset, limit).await?;30 Ok(RecordBatch { records, batch_index: offset / limit })31}3233#[task]34async fn transform_batch(batch: RecordBatch) -> Result<RecordBatch> {35 // Apply transformations: clean, normalize, enrich36 let transformed = batch.records.into_iter()37 .map(|r| apply_transforms(r))38 .collect();39 Ok(RecordBatch { records: transformed, batch_index: batch.batch_index })40}4142#[task]43async fn load_batch(dest: String, batch: RecordBatch) -> Result<usize> {44 // Upsert into destination — idempotent for safe retries45 let count = db::upsert_batch(&dest, &batch.records).await?;46 Ok(count)47}4849#[workflow(name = "batch_etl_pipeline", version = "1.0.0")]50pub async fn batch_etl_pipeline(51 ctx: WorkflowContext,52 config: BatchConfig,53) -> Result<BatchResult> {54 let num_batches = (config.total_records + config.batch_size - 1) / config.batch_size;55 let mut records_processed = 0;56 let mut failed_batches = Vec::new();5758 for batch_idx in 0..num_batches {59 let offset = batch_idx * config.batch_size;6061 // Extract62 let batch = ctx.execute_task(fetch_batch, (63 config.source_table.clone(), offset, config.batch_size64 )).await?;6566 // Transform67 let transformed = ctx.execute_task(transform_batch, batch).await?;6869 // Load (idempotent upsert)70 match ctx.execute_task(load_batch, (71 config.dest_table.clone(), transformed72 )).await {73 Ok(count) => records_processed += count,74 Err(e) => {75 println!("Batch {} failed: {}", batch_idx, e);76 failed_batches.push(batch_idx);77 }78 }7980 // Each loop iteration is automatically checkpointed81 }8283 Ok(BatchResult {84 records_processed,85 batches_completed: num_batches - failed_batches.len(),86 failed_batches,87 })88}Crash recovery, step by step
Suppose a worker dies while loading batch 042. Here is exactly what happens when the workflow is re-dispatched:
Replay walks the workflow function from the top, but every execute_task call that already has a journal entry returns its recorded result without running the task again. The loop fast-forwards through batches 001–041 and the completed steps of batch 042 in milliseconds, then resumes real work at the failed load_batch call.
This is also why the load step must be idempotent: if the worker died after the upsert reached the database but before the journal entry was written, the retry will run the upsert again. An idempotent write (upsert by key) makes that harmless.
Design notes
- Batch size is a checkpoint-granularity dial. Smaller batches mean finer checkpoints but more journal overhead; larger batches lose more work on failure. See Choose the Right Batch Size for concrete numbers.
- Collect failures, don't abort. The example records failed batch indexes and keeps going, so one bad batch doesn't sink a million-record run. The returned
failed_batcheslist is your input for a targeted reprocessing pass. If any failure should stop the pipeline, propagate the error instead of catching it. See Handle Partial Failures Gracefully. - Keep task payloads lean. Task inputs and results flow through the journal. Passing offsets and table names (as this example does with
fetch_batch) keeps journal entries small; avoid threading multi-megabyte blobs through task results when a reference will do.
Next Steps
- Next pattern: Parallel ETL
- Back to Data Pipelines & ETL overview