Operations

Cloud Operations


High Availability

The orchestrator is designed for active-active deployment with multiple replicas:

  • Stateless — No shared in-process state between instances. All coordination happens through PostgreSQL.
  • Claim-based work distribution — SELECT FOR UPDATE SKIP LOCKED ensures each workflow execution is processed by exactly one orchestrator instance.
  • Automatic failover — If an instance crashes, the reclaimer releases its claims after the claim timeout (default: 300 seconds in production) so another instance picks up the work.

Recommended production topology:

ComponentReplicasRationale
Orchestrator3+Minimum for HPA and PDB (minAvailable: 2)
PostgreSQL2 (primary + standby)Multi-AZ for automated failover
Redis2 (primary + replica)Failover without dispatch interruption

Monitoring in the Cloud

Prometheus on Managed Kubernetes

All orchestrator pods expose metrics at /metrics on port 8080 with Prometheus scrape annotations. If you use the Prometheus Operator (common on EKS, GKE, AKS), add a ServiceMonitor:

1
apiVersion: monitoring.coreos.com/v1
2
kind: ServiceMonitor
3
metadata:
4
name: orchestrator
5
namespace: orcher
6
spec:
7
selector:
8
matchLabels:
9
app: orchestrator
10
endpoints:
11
- port: http
12
interval: 10s
13
path: /metrics

Cloud-Native Monitoring

Alternatively, use your cloud provider's monitoring:

  • AWS CloudWatch — Export metrics via CloudWatch Agent or ADOT Collector. JSON logs on stdout are natively ingested by CloudWatch Logs.
  • GCP Cloud Monitoring — Use the Managed Service for Prometheus (GMP) which scrapes Prometheus endpoints automatically on GKE.
  • Azure Monitor — Azure Monitor managed Prometheus scrapes pods with standard annotations on AKS.

The orchestrator outputs structured JSON logs to stdout (when LOG_FORMAT=json), which is the standard format for all cloud log aggregation services.


Secrets Management

For production deployments, avoid plain Kubernetes secrets. Use your cloud provider's secrets manager:

ProviderServiceIntegration
AWSSecrets ManagerExternal Secrets Operator or ECS native secrets
GCPSecret ManagerExternal Secrets Operator or workload identity
AzureKey VaultExternal Secrets Operator or CSI Secret Store Driver
Multi-cloudHashiCorp VaultCSI driver or sidecar injector

The orchestrator reads DATABASE_URL and REDIS_URL from environment variables, so any mechanism that injects environment variables works.


Next Steps