Data Validation Pipeline
Before loading data into a warehouse, validate records against a schema, quarantine invalid rows for review, and only load clean data. This pattern separates validation from loading so you can fix and re-process quarantined records without rerunning the entire pipeline.
The key mindset: an invalid record is data, not an error. The workflow doesn't fail when it meets bad rows — it forks the stream, loads what's clean, and files the rest with a reason attached. Failure is reserved for real faults (the source is down, the warehouse rejects writes), which retry as usual.
How it works
One extract, one validation pass, then the stream splits. Both destinations are ordinary tasks, so both paths get the same durability guarantees.
Example: a validated ETL pipeline
Note the quarantine step is conditional — it only runs when the validation task reported invalid records. Branching like this is replay-safe because the condition derives from a journaled task result: on replay, validate_records returns the same recorded result, so the workflow always takes the same path. See the determinism rules.
1use orcher::prelude::*;2use anyhow::Result;34#[derive(Debug, Clone, Serialize, Deserialize)]5struct ValidationResult {6 valid_records: Vec<serde_json::Value>,7 invalid_records: Vec<(serde_json::Value, String)>, // (record, reason)8}910#[derive(Debug, Clone, Serialize, Deserialize)]11struct PipelineResult {12 loaded: usize,13 quarantined: usize,14 total: usize,15}1617#[task]18async fn extract_records(source: String, batch_size: usize) -> Result<Vec<serde_json::Value>> {19 let records = db::fetch_all(&source, batch_size).await?;20 Ok(records)21}2223#[task]24async fn validate_records(records: Vec<serde_json::Value>) -> Result<ValidationResult> {25 let mut valid = Vec::new();26 let mut invalid = Vec::new();2728 for record in records {29 match schema::validate(&record) {30 Ok(_) => valid.push(record),31 Err(reason) => invalid.push((record, reason.to_string())),32 }33 }3435 Ok(ValidationResult {36 valid_records: valid,37 invalid_records: invalid,38 })39}4041#[task]42async fn load_valid_records(dest: String, records: Vec<serde_json::Value>) -> Result<usize> {43 let count = db::upsert_batch(&dest, &records).await?;44 Ok(count)45}4647#[task]48async fn quarantine_invalid(records: Vec<(serde_json::Value, String)>) -> Result<usize> {49 // Store invalid records with their failure reasons for manual review50 let count = db::insert_quarantine(&records).await?;51 Ok(count)52}5354#[workflow(name = "validated_etl", version = "1.0.0")]55pub async fn validated_etl(56 ctx: WorkflowContext,57 source: String,58) -> Result<PipelineResult> {59 // Extract60 let records = ctx.execute_task(extract_records, (source, 5000usize)).await?;61 let total = records.len();6263 // Validate64 let validation = ctx.execute_task(validate_records, records).await?;6566 // Load valid records67 let loaded = ctx.execute_task(68 load_valid_records, ("warehouse.clean_data".to_string(), validation.valid_records)69 ).await?;7071 // Quarantine invalid records72 let quarantined = if !validation.invalid_records.is_empty() {73 ctx.execute_task(quarantine_invalid, validation.invalid_records).await?74 } else {75 076 };7778 Ok(PipelineResult { loaded, quarantined, total })79}The quarantine loop
Quarantine is only useful if records leave it. Because each quarantined row carries its failure reason, the review loop is targeted: fix the cause, then re-run the same pipeline with the quarantine table as the source — only the previously-bad rows flow through again.
Crash recovery follows the usual replay rules. The valid/invalid split is stable across replays because it lives in the journaled validation result — and if the worker dies between loading and quarantining, the load result is already journaled, so only the quarantine step retries.
Design notes
- Store the reason with the record. A quarantine table without failure reasons is a dead-letter queue; with them (as
quarantine_invaliddoes here) it's an actionable work queue you can group, triage, and fix in bulk. - Validate in a task, not in the workflow. Per-record schema checks are real compute. Keeping them inside
validate_recordsmeans the work runs once, its verdict is journaled, and the workflow only orchestrates — which is also what keeps the conditional quarantine branch deterministic. - Scale it with batching or fan-out. This example validates one 5,000-record extract. For millions of rows, run this pipeline per chunk with Batch Processing, or per source with Parallel ETL — the fork works the same at any width.
- Decide what "too dirty" means. Routing 1% of rows to quarantine is routine; routing 90% usually means the schema or mapping is wrong. Consider failing the workflow when the invalid ratio crosses a threshold so a systemic problem surfaces as an error instead of a full quarantine table.
Next Steps
- Next: Best Practices & Performance
- Previous: Scheduled Sync
- Back to Data Pipelines & ETL overview