Windowed Stream Aggregation
Processing high-volume event streams often requires aggregating data over time windows — counting page views per minute, summing transaction amounts per hour, or calculating average sensor readings per 5-minute interval. Orcher models this as a recurring workflow that collects events over a time window, aggregates them, and emits the result. The durable timer ensures the window fires on schedule even if the process restarts.
How it works
Each turn of the loop closes one tumbling window: fetch the window's events, fold them into an aggregate, store it, fire an alert if a threshold trips, then sleep until the next window. The durable timer makes the cadence survive restarts; the journal records each window's result once, so recovery never folds a window twice.
Example: a windowed aggregator
1import {2 Workflow,3 Tasks,4 Task,5 WorkflowContext,6 createTaskRefs,7} from '@orcher/sdk';89@Tasks()10class AggregationTasks {11 @Task()12 async fetchEventsInWindow(input: WindowQuery): Promise<AnalyticsEvent[]> {13 return await db.queryEvents(input.start, input.end, input.eventType);14 }1516 @Task()17 async computeAggregation(events: AnalyticsEvent[]): Promise<AggregationResult> {18 const count = events.length;19 const uniqueUsers = new Set(events.map((e) => e.userId)).size;20 const totalValue = events.reduce((sum, e) => sum + e.value, 0);21 return {22 count,23 uniqueUsers,24 totalValue,25 avgValue: count > 0 ? totalValue / count : 0,26 };27 }2829 @Task()30 async storeAggregation(input: StoredAggregation): Promise<void> {31 await db.insertAggregation(input);32 }3334 @Task()35 async checkThresholdAlert(input: AlertCheck): Promise<Alert | null> {36 if (input.value > input.threshold) {37 return { metric: input.metric, value: input.value };38 }39 return null;40 }4142 @Task()43 async sendAlert(alert: Alert): Promise<void> {44 await alerting.send(alert);45 }46}4748const tasks = createTaskRefs(AggregationTasks);4950// Recurring 5-minute aggregation window51@Workflow({ name: 'analytics_aggregator', version: '1.0.0' })52class AnalyticsAggregator {53 async run(ctx: WorkflowContext, config: AggregatorConfig): Promise<void> {54 const windowMs = 5 * 60 * 1000; // 5 minutes5556 while (true) {57 const windowEnd = new Date();58 const windowStart = new Date(windowEnd.getTime() - windowMs);5960 // Fetch events in this window61 const events = await ctx.executeTask(tasks.fetchEventsInWindow, {62 start: windowStart.toISOString(),63 end: windowEnd.toISOString(),64 eventType: config.eventType,65 });6667 // Compute aggregation68 const agg = await ctx.executeTask(tasks.computeAggregation, events);6970 // Store result71 await ctx.executeTask(tasks.storeAggregation, {72 windowStart: windowStart.toISOString(),73 windowEnd: windowEnd.toISOString(),74 eventType: config.eventType,75 result: agg,76 });7778 // Check alert thresholds79 const alert = await ctx.executeTask(tasks.checkThresholdAlert, {80 metric: config.eventType,81 value: agg.count,82 threshold: config.alertThreshold,83 });84 if (alert) {85 await ctx.executeTask(tasks.sendAlert, alert);86 }8788 // Sleep until next window89 await ctx.sleep(windowMs);90 }91 }92}Start this workflow once and it runs continuously, aggregating a new 5-minute window on each iteration. The durable timer at the end of each loop ensures the window fires reliably even across restarts. Each aggregation result is stored as a separate task, creating an audit trail of all computed metrics.
Mind the gap: anchor windows to a grid
The example computes each window from now() — window_end = now(), window_start = now() - 5min — and then sleeps 5 minutes. But the work between waking and sleeping takes time, so the next window starts a little later than the last one ended. Events that land in that sliver belong to no window at all.
window_end forward as the next window_start. Windows tile a fixed grid, immune to processing time.Anchor windows to a grid, not to the wall clock at wake time: keep the previous window_end as state and use it as the next window_start. Because that boundary is journaled, it also survives crashes — a resume continues the same grid instead of restarting it at a fresh now().
Design notes
- Grid boundaries beat
now(). The fix above is the difference between "roughly every 5 minutes" and "exactly the[:00, :05)buckets." Only wall-clock-anchored grids let you compare a window against the same window yesterday. - Decide what to do about late events. An event written just after its window closed is missed by a query that already ran. If completeness matters, aggregate a window a grace period after it closes (fetch
[10:00, 10:05)at 10:06), trading freshness for correctness — a watermark, the same idea as scheduled sync's watermark. - Query-driven vs. accumulate-in-actor. This workflow re-queries stored events each window — simple, and the DB is the source of truth. For very high volume, feed events into a per-key actor that folds them incrementally, so no window ever re-scans raw rows.
- Bound the loop. Like every recurring workflow, an unbounded aggregator grows its journal forever — stop after N windows and restart carrying the last boundary, the same reasoning as scheduled jobs.