Operations

Monitoring & Operations


Health Checks

Orchestrator Endpoints

The orchestrator's HTTP listener (port 8080) serves the health endpoints alongside the REST API:

1
# Health
2
curl http://localhost:8080/health
3
4
# Liveness — is the orchestrator process running?
5
curl http://localhost:8080/health/live
6
7
# Readiness
8
curl http://localhost:8080/health/ready
9
10
# Prometheus metrics
11
curl http://localhost:8080/metrics

Kubernetes Probes

1
apiVersion: apps/v1
2
kind: Deployment
3
metadata:
4
name: orcher-orchestrator
5
spec:
6
replicas: 3
7
template:
8
spec:
9
containers:
10
- name: orchestrator
11
ports:
12
- containerPort: 50051
13
name: grpc
14
- containerPort: 8080
15
name: http
16
livenessProbe:
17
httpGet:
18
path: /health/live
19
port: 8080
20
initialDelaySeconds: 10
21
periodSeconds: 10
22
failureThreshold: 3
23
readinessProbe:
24
httpGet:
25
path: /health/ready
26
port: 8080
27
initialDelaySeconds: 5
28
periodSeconds: 5
29
failureThreshold: 3
30
resources:
31
requests:
32
memory: "512Mi"
33
cpu: "500m"
34
limits:
35
memory: "2Gi"
36
cpu: "2000m"

Database Schema

The orchestrator uses these core tables:

TablePurpose
workflow_executionsWorkflow instances with claim tracking (claimed_by, claim_timeout_at)
task_executionsTask lifecycle (scheduled, started, completed, failed)
journalEvent-sourced journal for replay protection
execution_logsReal-time execution logs
outbox_queueAtomic task dispatch with lease-based claiming
timer_queueDurable timers (sleep, timeouts, schedules)
dead_letter_queueFailed tasks after max retries
namespacesLogical isolation for multi-tenancy
organizationsOrganization-level multi-tenancy

Key design patterns:

  • Idempotency — idempotency_key unique constraints prevent duplicate operations
  • Claim-based polling — claimed_by and claim_timeout_at fields enable safe distributed polling without database locks
  • Shard routing — shard_id column for horizontal partitioning
  • JSONB attributes — Flexible metadata with GIN indexes for fast queries

Performance Tuning

Connection Pool Sizing

The orchestrator defaults to 80 PostgreSQL connections (ORCHER_PG_MAX_CONNECTIONS, with ORCHER_PG_MIN_CONNECTIONS defaulting to 20). For production, tune based on your instance count:

1
# Per-node pool size (default 80); lower it when running many nodes on one database
2
ORCHER_PG_MAX_CONNECTIONS=80
3
ORCHER_PG_MIN_CONNECTIONS=20
4
5
# Total connections needed = orchestrator_instances * ORCHER_PG_MAX_CONNECTIONS
6
# Ensure PostgreSQL max_connections > total needed + buffer
7
# With PgBouncer, each orchestrator gets multiplexed connections

Using PgBouncer in transaction mode is strongly recommended — it supports 1000+ concurrent clients via connection multiplexing.

Memory

Concurrent WorkflowsRecommended RAM
Up to 100512 MB
100 - 5001 GB
500 - 1,0002 GB
1,000 - 5,0004 GB
5,000+8 GB+

The HighPerfCache keeps hot workflow state in memory. Memory usage scales with the number of active (not total) workflows.

Scaling Characteristics

Single node: Up to ~100 concurrent workflows with direct PostgreSQL, more with PgBouncer.

Multi-node cluster: Horizontal scaling via sharding. Each node owns a subset of shards and processes them independently. PostgreSQL becomes the bottleneck at very high throughput — mitigate with PgBouncer and read replicas.

Eager dispatch optimization: When a task completes, the orchestrator can eagerly return the next task in the same gRPC response, saving a poll round trip for sequential workflows.


Graceful Shutdown

The orchestrator handles SIGINT/SIGTERM gracefully:

  1. Stops accepting new gRPC connections
  2. Cancels background processors (outbox, timer, schedule)
  3. Releases all shard leases (other nodes pick them up)
  4. Drains in-flight requests
  5. Closes database connections

The default shutdown timeout is 30 seconds.


Troubleshooting

Server Won't Start

Check database connectivity:

1
# Test PostgreSQL connection
2
psql $DATABASE_URL -c "SELECT 1"

Check port availability:

1
lsof -i :50051
2
lsof -i :8080

Slow Performance

Check database query performance:

1
-- Find slow queries
2
SELECT query, mean_exec_time, calls
3
FROM pg_stat_statements
4
ORDER BY mean_exec_time DESC
5
LIMIT 10;

Check queue depth via the CLI:

1
orcher queue depth --watch
2
orcher queue stats default

Common causes:

  • High outbox depth — OutboxProcessor can't keep up. Scale orchestrator instances.
  • Timer backlog — Many timers firing simultaneously. Increase timer processor throughput.
  • Connection pool exhaustion — Use PgBouncer or increase pool size.

Next Steps