Scheduled Sync (CDC Pattern)
For pipelines that need to run continuously — syncing changes from a source system every few minutes — use a long-running workflow with durable timers. The workflow tracks a watermark (timestamp or offset) so it always picks up exactly where it left off, even after restarts.
The conventional version of this needs three moving parts: a cron scheduler, a job runner, and a state store for the cursor. Here all three collapse into one workflow — ctx.sleep is the schedule, the workflow is the runner, and the journaled watermark is the cursor.
How it works
Each cycle fetches everything changed since the watermark, applies it, advances the watermark, and goes to sleep. The sleep is a durable timer: the workflow fully suspends — holding no thread, no connection, no memory — and the server re-dispatches it when the timer fires.
Example: incremental sync with a watermark
The workflow below syncs changes on a fixed interval. The watermark is an ordinary local variable — it needs no database row or config entry, because every step that produced it is journaled and restored on replay.
1use orcher::prelude::*;2use anyhow::Result;3use std::time::Duration;45#[derive(Debug, Clone, Serialize, Deserialize)]6struct SyncConfig {7 source_url: String,8 dest_table: String,9 sync_interval_secs: u64,10 max_iterations: u32,11}1213#[derive(Debug, Clone, Serialize, Deserialize)]14struct SyncState {15 watermark: String, // Tracks last synced timestamp or offset16 total_synced: usize,17 iterations: u32,18}1920#[task]21async fn fetch_changes(source_url: String, since: String) -> Result<(Vec<serde_json::Value>, String)> {22 // Query source for records changed since the watermark23 let (records, new_watermark) = source::get_changes(&source_url, &since).await?;24 Ok((records, new_watermark))25}2627#[task]28async fn apply_changes(dest_table: String, records: Vec<serde_json::Value>) -> Result<usize> {29 // Upsert changed records into destination30 let count = dest::upsert(&dest_table, &records).await?;31 Ok(count)32}3334#[workflow(name = "incremental_sync", version = "1.0.0")]35pub async fn incremental_sync(36 ctx: WorkflowContext,37 config: SyncConfig,38) -> Result<SyncState> {39 let mut state = SyncState {40 watermark: "1970-01-01T00:00:00Z".to_string(),41 total_synced: 0,42 iterations: 0,43 };4445 loop {46 // Fetch changes since last watermark47 let (changes, new_watermark) = ctx.execute_task(48 fetch_changes, (config.source_url.clone(), state.watermark.clone())49 ).await?;5051 if !changes.is_empty() {52 // Apply changes to destination53 let count = ctx.execute_task(54 apply_changes, (config.dest_table.clone(), changes)55 ).await?;56 state.total_synced += count;57 println!("Synced {} records (total: {})", count, state.total_synced);58 }5960 state.watermark = new_watermark;61 state.iterations += 1;6263 if state.iterations >= config.max_iterations {64 break;65 }6667 // Sleep until next sync — durable timer, zero resource consumption68 ctx.sleep(Duration::from_secs(config.sync_interval_secs)).await?;69 }7071 Ok(state)72}Surviving a deploy mid-sleep
The usual failure mode for cron-based sync is the gap: the scheduler fires while the runner is being deployed, the run is missed, and the cursor drifts. With a durable timer there is nothing to miss — the timer lives in the server, not the process:
Crashes mid-cycle behave like any workflow crash: completed steps replay from the journal, and the in-flight step retries. If the process dies right after apply_changes but before the watermark advances, replay returns the journaled fetch and apply results and continues — the watermark never skips ahead of work that wasn't applied.
Design notes
- Durable timers, not external cron. No scheduler infrastructure, no missed runs during deploys, and no overlapping runs — the next cycle can't start until the previous one finishes. See Schedule with Durable Timers, Not External Cron.
- Make
apply_changesidempotent. A retry can re-apply a batch of changes whose upsert already reached the destination. Upserting by key makes the second application a no-op — the same rule as batch processing's load step. - Bound the loop. Every cycle appends journal entries, so an unbounded loop grows its journal forever. The example stops after
max_iterationsand returns its final state — restart the workflow with the returned watermark to continue with a fresh journal. - Watermark choice matters. A timestamp is simple but can miss rows written concurrently with the query; a monotonic offset or sequence ID from the source (as
fetch_changesreturns here) is safer for true change-data-capture.
Next Steps
- Next pattern: Data Validation
- Previous: Parallel ETL
- Back to Data Pipelines & ETL overview