Operations

Scaling Workers


Statelessness and Recovery

Workers hold no workflow state in memory. All state — journal entries, execution status, task results — lives in the orchestrator's PostgreSQL database. This has important implications:

  • Any worker can resume any workflow — If a worker crashes mid-execution, the orchestrator's reclaimer releases the claim after a timeout (60s dev, 300s production). Another worker picks up the workflow and replays completed steps from the journal without re-executing them.
  • Workers can handle more workflows than they have memory for — A blocked workflow (waiting for an event, timer, or child workflow) consumes no worker resources. The worker only needs memory for actively executing workflows.
  • Rolling deployments are safe — Old workers finish in-flight work and shut down. New workers pick up new work immediately. No state migration needed.

Scaling

Horizontal Scaling

Add more Worker instances to increase throughput. Each instance polls independently:

1
# Run multiple instances
2
./target/release/order-worker &
3
./target/release/order-worker &
4
./target/release/order-worker &

Or scale with container orchestration:

1
# Docker Compose
2
services:
3
order-worker:
4
build: .
5
deploy:
6
replicas: 4
7
environment:
8
ORCHER_SERVER_URL: http://orchestrator:50051
1
# Kubernetes
2
apiVersion: apps/v1
3
kind: Deployment
4
metadata:
5
name: order-worker
6
spec:
7
replicas: 4
8
template:
9
spec:
10
containers:
11
- name: worker
12
image: your-registry/order-worker:latest
13
env:
14
- name: ORCHER_SERVER_URL
15
value: "http://orcher-orchestrator:50051"
16
- name: ORCHER_TASK_QUEUE
17
value: "orders"
18
resources:
19
requests:
20
memory: "256Mi"
21
cpu: "500m"
22
limits:
23
memory: "1Gi"
24
cpu: "2000m"

When to Scale

Scale up when:

  • Queue depth is consistently growing (orcher_queue_depth metric)
  • Task latency exceeds your SLA (orcher_task_duration_seconds)
  • CPU usage is above 70% across workers

Scale down when:

  • Queue depth is consistently zero
  • CPU usage is below 30% for several minutes
  • Most pollers return empty (no work available)

Performance Tuning

ScenarioAdjustment
High task latencyIncrease max_concurrent_tasks (I/O-bound tasks can go much higher)
Pollers competing for workReduce task_poller_count per worker, add more worker instances instead
Sequential workflows are slowEnsure eager dispatch is working — check orcher_tasks_completed_total vs poll counts
Queue depth growingAdd more workers or increase max_concurrent_workflows

Next Steps

  • Creating a Worker — Builder API and configuration options
  • Connectivity — Registration, polling, heartbeats, and eager dispatch
  • Operations — Graceful shutdown, monitoring, and troubleshooting