Scaling Workers
Statelessness and Recovery
Workers hold no workflow state in memory. All state — journal entries, execution status, task results — lives in the orchestrator's PostgreSQL database. This has important implications:
- Any worker can resume any workflow — If a worker crashes mid-execution, the orchestrator's reclaimer releases the claim after a timeout (60s dev, 300s production). Another worker picks up the workflow and replays completed steps from the journal without re-executing them.
- Workers can handle more workflows than they have memory for — A blocked workflow (waiting for an event, timer, or child workflow) consumes no worker resources. The worker only needs memory for actively executing workflows.
- Rolling deployments are safe — Old workers finish in-flight work and shut down. New workers pick up new work immediately. No state migration needed.
Scaling
Horizontal Scaling
Add more Worker instances to increase throughput. Each instance polls independently:
1# Run multiple instances2./target/release/order-worker &3./target/release/order-worker &4./target/release/order-worker &Or scale with container orchestration:
1# Docker Compose2services:3 order-worker:4 build: .5 deploy:6 replicas: 47 environment:8 ORCHER_SERVER_URL: http://orchestrator:500511# Kubernetes2apiVersion: apps/v13kind: Deployment4metadata:5 name: order-worker6spec:7 replicas: 48 template:9 spec:10 containers:11 - name: worker12 image: your-registry/order-worker:latest13 env:14 - name: ORCHER_SERVER_URL15 value: "http://orcher-orchestrator:50051"16 - name: ORCHER_TASK_QUEUE17 value: "orders"18 resources:19 requests:20 memory: "256Mi"21 cpu: "500m"22 limits:23 memory: "1Gi"24 cpu: "2000m"When to Scale
Scale up when:
- Queue depth is consistently growing (
orcher_queue_depthmetric) - Task latency exceeds your SLA (
orcher_task_duration_seconds) - CPU usage is above 70% across workers
Scale down when:
- Queue depth is consistently zero
- CPU usage is below 30% for several minutes
- Most pollers return empty (no work available)
Performance Tuning
| Scenario | Adjustment |
|---|---|
| High task latency | Increase max_concurrent_tasks (I/O-bound tasks can go much higher) |
| Pollers competing for work | Reduce task_poller_count per worker, add more worker instances instead |
| Sequential workflows are slow | Ensure eager dispatch is working — check orcher_tasks_completed_total vs poll counts |
| Queue depth growing | Add more workers or increase max_concurrent_workflows |
Next Steps
- Creating a Worker — Builder API and configuration options
- Connectivity — Registration, polling, heartbeats, and eager dispatch
- Operations — Graceful shutdown, monitoring, and troubleshooting