Monitoring & Operations
Health Checks
Orchestrator Endpoints
The orchestrator's HTTP listener (port 8080) serves the health endpoints alongside the REST API:
1# Health2curl http://localhost:8080/health34# Liveness — is the orchestrator process running?5curl http://localhost:8080/health/live67# Readiness8curl http://localhost:8080/health/ready910# Prometheus metrics11curl http://localhost:8080/metricsKubernetes Probes
1apiVersion: apps/v12kind: Deployment3metadata:4 name: orcher-orchestrator5spec:6 replicas: 37 template:8 spec:9 containers:10 - name: orchestrator11 ports:12 - containerPort: 5005113 name: grpc14 - containerPort: 808015 name: http16 livenessProbe:17 httpGet:18 path: /health/live19 port: 808020 initialDelaySeconds: 1021 periodSeconds: 1022 failureThreshold: 323 readinessProbe:24 httpGet:25 path: /health/ready26 port: 808027 initialDelaySeconds: 528 periodSeconds: 529 failureThreshold: 330 resources:31 requests:32 memory: "512Mi"33 cpu: "500m"34 limits:35 memory: "2Gi"36 cpu: "2000m"Database Schema
The orchestrator uses these core tables:
| Table | Purpose |
|---|---|
workflow_executions | Workflow instances with claim tracking (claimed_by, claim_timeout_at) |
task_executions | Task lifecycle (scheduled, started, completed, failed) |
journal | Event-sourced journal for replay protection |
execution_logs | Real-time execution logs |
outbox_queue | Atomic task dispatch with lease-based claiming |
timer_queue | Durable timers (sleep, timeouts, schedules) |
dead_letter_queue | Failed tasks after max retries |
namespaces | Logical isolation for multi-tenancy |
organizations | Organization-level multi-tenancy |
Key design patterns:
- Idempotency —
idempotency_keyunique constraints prevent duplicate operations - Claim-based polling —
claimed_byandclaim_timeout_atfields enable safe distributed polling without database locks - Shard routing —
shard_idcolumn for horizontal partitioning - JSONB attributes — Flexible metadata with GIN indexes for fast queries
Performance Tuning
Connection Pool Sizing
The orchestrator defaults to 80 PostgreSQL connections (ORCHER_PG_MAX_CONNECTIONS, with ORCHER_PG_MIN_CONNECTIONS defaulting to 20). For production, tune based on your instance count:
1# Per-node pool size (default 80); lower it when running many nodes on one database2ORCHER_PG_MAX_CONNECTIONS=803ORCHER_PG_MIN_CONNECTIONS=2045# Total connections needed = orchestrator_instances * ORCHER_PG_MAX_CONNECTIONS6# Ensure PostgreSQL max_connections > total needed + buffer7# With PgBouncer, each orchestrator gets multiplexed connectionsUsing PgBouncer in transaction mode is strongly recommended — it supports 1000+ concurrent clients via connection multiplexing.
Memory
| Concurrent Workflows | Recommended RAM |
|---|---|
| Up to 100 | 512 MB |
| 100 - 500 | 1 GB |
| 500 - 1,000 | 2 GB |
| 1,000 - 5,000 | 4 GB |
| 5,000+ | 8 GB+ |
The HighPerfCache keeps hot workflow state in memory. Memory usage scales with the number of active (not total) workflows.
Scaling Characteristics
Single node: Up to ~100 concurrent workflows with direct PostgreSQL, more with PgBouncer.
Multi-node cluster: Horizontal scaling via sharding. Each node owns a subset of shards and processes them independently. PostgreSQL becomes the bottleneck at very high throughput — mitigate with PgBouncer and read replicas.
Eager dispatch optimization: When a task completes, the orchestrator can eagerly return the next task in the same gRPC response, saving a poll round trip for sequential workflows.
Graceful Shutdown
The orchestrator handles SIGINT/SIGTERM gracefully:
- Stops accepting new gRPC connections
- Cancels background processors (outbox, timer, schedule)
- Releases all shard leases (other nodes pick them up)
- Drains in-flight requests
- Closes database connections
The default shutdown timeout is 30 seconds.
Troubleshooting
Server Won't Start
Check database connectivity:
1# Test PostgreSQL connection2psql $DATABASE_URL -c "SELECT 1"Check port availability:
1lsof -i :500512lsof -i :8080Slow Performance
Check database query performance:
1-- Find slow queries2SELECT query, mean_exec_time, calls3FROM pg_stat_statements4ORDER BY mean_exec_time DESC5LIMIT 10;Check queue depth via the CLI:
1orcher queue depth --watch2orcher queue stats defaultCommon causes:
- High outbox depth — OutboxProcessor can't keep up. Scale orchestrator instances.
- Timer backlog — Many timers firing simultaneously. Increase timer processor throughput.
- Connection pool exhaustion — Use PgBouncer or increase pool size.
Next Steps
- Orchestrator Overview — Architecture and components
- Configuration — Environment variables and database connection
- Deployment — Starting the server in development, Docker, and systemd
- High Availability — Multi-instance deployment and load balancing
- Worker Management — Deploy and manage workers
- Queue Configuration — Configure task queues and priority
- Storage Setup — Configure PostgreSQL
- Observability — Set up metrics, tracing, and logging