Getting Started

Data Validation Pipeline


Before loading data into a warehouse, validate records against a schema, quarantine invalid rows for review, and only load clean data. This pattern separates validation from loading so you can fix and re-process quarantined records without rerunning the entire pipeline.

The key mindset: an invalid record is data, not an error. The workflow doesn't fail when it meets bad rows — it forks the stream, loads what's clean, and files the rest with a reason attached. Failure is reserved for real faults (the source is down, the warehouse rejects writes), which retry as usual.

How it works

One extract, one validation pass, then the stream splits. Both destinations are ordinary tasks, so both paths get the same durability guarantees.

Validate, Then Fork
extract_records
5,000 records
▼
validate_records
schema check per record
▼ split
load_valid_records
4,940 clean rows
→ warehouse.clean_data
quarantine_invalid
60 rows + failure reasons
→ quarantine table
The workflow succeeds either way — bad rows are routed, not fatal.

Example: a validated ETL pipeline

Note the quarantine step is conditional — it only runs when the validation task reported invalid records. Branching like this is replay-safe because the condition derives from a journaled task result: on replay, validate_records returns the same recorded result, so the workflow always takes the same path. See the determinism rules.

1
use orcher::prelude::*;
2
use anyhow::Result;
3
4
#[derive(Debug, Clone, Serialize, Deserialize)]
5
struct ValidationResult {
6
valid_records: Vec<serde_json::Value>,
7
invalid_records: Vec<(serde_json::Value, String)>, // (record, reason)
8
}
9
10
#[derive(Debug, Clone, Serialize, Deserialize)]
11
struct PipelineResult {
12
loaded: usize,
13
quarantined: usize,
14
total: usize,
15
}
16
17
#[task]
18
async fn extract_records(source: String, batch_size: usize) -> Result<Vec<serde_json::Value>> {
19
let records = db::fetch_all(&source, batch_size).await?;
20
Ok(records)
21
}
22
23
#[task]
24
async fn validate_records(records: Vec<serde_json::Value>) -> Result<ValidationResult> {
25
let mut valid = Vec::new();
26
let mut invalid = Vec::new();
27
28
for record in records {
29
match schema::validate(&record) {
30
Ok(_) => valid.push(record),
31
Err(reason) => invalid.push((record, reason.to_string())),
32
}
33
}
34
35
Ok(ValidationResult {
36
valid_records: valid,
37
invalid_records: invalid,
38
})
39
}
40
41
#[task]
42
async fn load_valid_records(dest: String, records: Vec<serde_json::Value>) -> Result<usize> {
43
let count = db::upsert_batch(&dest, &records).await?;
44
Ok(count)
45
}
46
47
#[task]
48
async fn quarantine_invalid(records: Vec<(serde_json::Value, String)>) -> Result<usize> {
49
// Store invalid records with their failure reasons for manual review
50
let count = db::insert_quarantine(&records).await?;
51
Ok(count)
52
}
53
54
#[workflow(name = "validated_etl", version = "1.0.0")]
55
pub async fn validated_etl(
56
ctx: WorkflowContext,
57
source: String,
58
) -> Result<PipelineResult> {
59
// Extract
60
let records = ctx.execute_task(extract_records, (source, 5000usize)).await?;
61
let total = records.len();
62
63
// Validate
64
let validation = ctx.execute_task(validate_records, records).await?;
65
66
// Load valid records
67
let loaded = ctx.execute_task(
68
load_valid_records, ("warehouse.clean_data".to_string(), validation.valid_records)
69
).await?;
70
71
// Quarantine invalid records
72
let quarantined = if !validation.invalid_records.is_empty() {
73
ctx.execute_task(quarantine_invalid, validation.invalid_records).await?
74
} else {
75
0
76
};
77
78
Ok(PipelineResult { loaded, quarantined, total })
79
}

The quarantine loop

Quarantine is only useful if records leave it. Because each quarantined row carries its failure reason, the review loop is targeted: fix the cause, then re-run the same pipeline with the quarantine table as the source — only the previously-bad rows flow through again.

Reprocessing Quarantined Records
1
review60 rows, grouped by failure reason
2
fix the causePatch the mapping, backfill the field, or amend the schema
3
validated_etl(quarantine)Re-run the same workflow, quarantine table as source
✓
load / re-quarantine58 now clean → warehouse; 2 still invalid → back to quarantine
Only the 60 quarantined rows re-run — never the 4,940 that already loaded.

Crash recovery follows the usual replay rules. The valid/invalid split is stable across replays because it lives in the journaled validation result — and if the worker dies between loading and quarantining, the load result is already journaled, so only the quarantine step retries.

Design notes

  • Store the reason with the record. A quarantine table without failure reasons is a dead-letter queue; with them (as quarantine_invalid does here) it's an actionable work queue you can group, triage, and fix in bulk.
  • Validate in a task, not in the workflow. Per-record schema checks are real compute. Keeping them inside validate_records means the work runs once, its verdict is journaled, and the workflow only orchestrates — which is also what keeps the conditional quarantine branch deterministic.
  • Scale it with batching or fan-out. This example validates one 5,000-record extract. For millions of rows, run this pipeline per chunk with Batch Processing, or per source with Parallel ETL — the fork works the same at any width.
  • Decide what "too dirty" means. Routing 1% of rows to quarantine is routine; routing 90% usually means the schema or mapping is wrong. Consider failing the workflow when the invalid ratio crosses a threshold so a systemic problem surfaces as an error instead of a full quarantine table.

Next Steps