Getting Started

Windowed Stream Aggregation


Processing high-volume event streams often requires aggregating data over time windows — counting page views per minute, summing transaction amounts per hour, or calculating average sensor readings per 5-minute interval. Orcher models this as a recurring workflow that collects events over a time window, aggregates them, and emits the result. The durable timer ensures the window fires on schedule even if the process restarts.

How it works

Each turn of the loop closes one tumbling window: fetch the window's events, fold them into an aggregate, store it, fire an alert if a threshold trips, then sleep until the next window. The durable timer makes the cadence survive restarts; the journal records each window's result once, so recovery never folds a window twice.

One Window, Then the Next
1
fetch_events_in_window[10:00, 10:05) — 8,412 events
2
compute_aggregationcount · unique users · total · avg
3
store_aggregationOne row per window — journaled
4
check_threshold_alertFire only if the metric trips
⏱
sleep(5 min)Durable timer → the [10:05, 10:10) window
Consecutive windows should tile the timeline exactly — no gaps, no overlaps.

Example: a windowed aggregator

1
import {
2
Workflow,
3
Tasks,
4
Task,
5
WorkflowContext,
6
createTaskRefs,
7
} from '@orcher/sdk';
8
9
@Tasks()
10
class AggregationTasks {
11
@Task()
12
async fetchEventsInWindow(input: WindowQuery): Promise<AnalyticsEvent[]> {
13
return await db.queryEvents(input.start, input.end, input.eventType);
14
}
15
16
@Task()
17
async computeAggregation(events: AnalyticsEvent[]): Promise<AggregationResult> {
18
const count = events.length;
19
const uniqueUsers = new Set(events.map((e) => e.userId)).size;
20
const totalValue = events.reduce((sum, e) => sum + e.value, 0);
21
return {
22
count,
23
uniqueUsers,
24
totalValue,
25
avgValue: count > 0 ? totalValue / count : 0,
26
};
27
}
28
29
@Task()
30
async storeAggregation(input: StoredAggregation): Promise<void> {
31
await db.insertAggregation(input);
32
}
33
34
@Task()
35
async checkThresholdAlert(input: AlertCheck): Promise<Alert | null> {
36
if (input.value > input.threshold) {
37
return { metric: input.metric, value: input.value };
38
}
39
return null;
40
}
41
42
@Task()
43
async sendAlert(alert: Alert): Promise<void> {
44
await alerting.send(alert);
45
}
46
}
47
48
const tasks = createTaskRefs(AggregationTasks);
49
50
// Recurring 5-minute aggregation window
51
@Workflow({ name: 'analytics_aggregator', version: '1.0.0' })
52
class AnalyticsAggregator {
53
async run(ctx: WorkflowContext, config: AggregatorConfig): Promise<void> {
54
const windowMs = 5 * 60 * 1000; // 5 minutes
55
56
while (true) {
57
const windowEnd = new Date();
58
const windowStart = new Date(windowEnd.getTime() - windowMs);
59
60
// Fetch events in this window
61
const events = await ctx.executeTask(tasks.fetchEventsInWindow, {
62
start: windowStart.toISOString(),
63
end: windowEnd.toISOString(),
64
eventType: config.eventType,
65
});
66
67
// Compute aggregation
68
const agg = await ctx.executeTask(tasks.computeAggregation, events);
69
70
// Store result
71
await ctx.executeTask(tasks.storeAggregation, {
72
windowStart: windowStart.toISOString(),
73
windowEnd: windowEnd.toISOString(),
74
eventType: config.eventType,
75
result: agg,
76
});
77
78
// Check alert thresholds
79
const alert = await ctx.executeTask(tasks.checkThresholdAlert, {
80
metric: config.eventType,
81
value: agg.count,
82
threshold: config.alertThreshold,
83
});
84
if (alert) {
85
await ctx.executeTask(tasks.sendAlert, alert);
86
}
87
88
// Sleep until next window
89
await ctx.sleep(windowMs);
90
}
91
}
92
}

Start this workflow once and it runs continuously, aggregating a new 5-minute window on each iteration. The durable timer at the end of each loop ensures the window fires reliably even across restarts. Each aggregation result is stored as a separate task, creating an audit trail of all computed metrics.


Mind the gap: anchor windows to a grid

The example computes each window from now() — window_end = now(), window_start = now() - 5min — and then sleeps 5 minutes. But the work between waking and sleeping takes time, so the next window starts a little later than the last one ended. Events that land in that sliver belong to no window at all.

now()-Anchored Windows Drift
✓
window 1[10:00:00, 10:05:00) — anchored to now()
⏱
aggregate + storeTakes ~800ms of wall-clock
⏱
sleep(5 min)Fires at 10:10:00.8, not 10:10:00
✗
window 2[10:05:00.8, 10:10:00.8) — starts 800ms late
✗
the gapEvents in [10:05:00, 10:05:00.8) counted by neither window
Fix: carry window_end forward as the next window_start. Windows tile a fixed grid, immune to processing time.

Anchor windows to a grid, not to the wall clock at wake time: keep the previous window_end as state and use it as the next window_start. Because that boundary is journaled, it also survives crashes — a resume continues the same grid instead of restarting it at a fresh now().

Design notes

  • Grid boundaries beat now(). The fix above is the difference between "roughly every 5 minutes" and "exactly the [:00, :05) buckets." Only wall-clock-anchored grids let you compare a window against the same window yesterday.
  • Decide what to do about late events. An event written just after its window closed is missed by a query that already ran. If completeness matters, aggregate a window a grace period after it closes (fetch [10:00, 10:05) at 10:06), trading freshness for correctness — a watermark, the same idea as scheduled sync's watermark.
  • Query-driven vs. accumulate-in-actor. This workflow re-queries stored events each window — simple, and the DB is the source of truth. For very high volume, feed events into a per-key actor that folds them incrementally, so no window ever re-scans raw rows.
  • Bound the loop. Like every recurring workflow, an unbounded aggregator grows its journal forever — stop after N windows and restart carrying the last boundary, the same reasoning as scheduled jobs.

Next Steps