Operating in Production
Now that Orcher is deployed and tuned, let's cover day-to-day operations.
Monitoring and Alerting
The Essential Metrics (monitor these from day one):
1. Workflow Queue Depth
1-- Current queue depth by status2SELECT status, COUNT(*) as count3FROM workflow_executions4WHERE status IN ('pending', 'queued', 'running')5GROUP BY status;Alert when: Pending count exceeds 2x your normal baseline for more than 5 minutes. This means workflows are arriving faster than workers can process them.
2. Workflow Completion Rate
1-- Completion rate over the last hour2SELECT3 COUNT(*) FILTER (WHERE status = 'completed') AS completed,4 COUNT(*) FILTER (WHERE status = 'failed') AS failed,5 ROUND(6 100.0 * COUNT(*) FILTER (WHERE status = 'completed') /7 NULLIF(COUNT(*), 0), 28 ) AS success_rate_pct9FROM workflow_executions10WHERE completed_at > NOW() - INTERVAL '1 hour';Alert when: Success rate drops below 95% over a 15-minute window.
3. Workflow Duration (P95)
1-- P95 duration by workflow type2SELECT3 workflow_name,4 PERCENTILE_CONT(0.95) WITHIN GROUP (ORDER BY duration_ms) AS p95_ms,5 PERCENTILE_CONT(0.50) WITHIN GROUP (ORDER BY duration_ms) AS p50_ms,6 COUNT(*) AS total7FROM workflow_executions8WHERE status = 'completed'9 AND completed_at > NOW() - INTERVAL '1 hour'10GROUP BY workflow_name;Alert when: P95 exceeds 3x the normal baseline.
4. Database Connection Count
1-- Active connections2SELECT COUNT(*) AS active_connections,3 MAX(pg_stat_activity.backend_type) AS connection_types4FROM pg_stat_activity5WHERE state = 'active';Alert when: Connections exceed 80% of max_connections.
5. Dead Letter Queue Growth
1-- DLQ entries in the last hour2SELECT COUNT(*) AS dlq_entries3FROM dead_letter_queue4WHERE failed_at > NOW() - INTERVAL '1 hour';Alert when: Any new entries appear (DLQ entries indicate tasks that failed beyond retry limits).
Prometheus Metrics
Export these metrics from your application for Grafana dashboards:
1# Workflow throughput2orcher_workflows_started_total{workflow_name, namespace}3orcher_workflows_completed_total{workflow_name, namespace, status}45# Workflow duration6orcher_workflow_duration_seconds{workflow_name, namespace, quantile}78# Queue depth9orcher_queue_depth{status, namespace}1011# Worker utilization12orcher_worker_active_workflows{worker_id}13orcher_worker_capacity{worker_id}1415# Database health16orcher_db_connections_active17orcher_db_query_duration_seconds{query_type}Debugging Production Issues
Problem 1: Workflows Are Stuck
Symptoms: Workflows stay in 'running' for too long.
1-- Find stuck workflows2SELECT3 execution_id,4 workflow_name,5 status,6 claimed_by,7 claimed_at,8 claim_timeout_at,9 EXTRACT(EPOCH FROM (NOW() - claimed_at)) / 60 AS running_minutes10FROM workflow_executions11WHERE status = 'running'12 AND claimed_at < NOW() - INTERVAL '10 minutes'13ORDER BY claimed_at;Common causes:
- Worker crashed without releasing claim (wait for claim timeout, then auto-reclaim)
- Task making external call that never returns (check task timeouts)
- Deadlock between workflows (check for circular dependencies)
Resolution: If a workflow is truly stuck, release its claim:
1-- Release a stuck workflow's claim (it will be picked up by another worker)2UPDATE workflow_executions3SET claimed_by = NULL,4 claimed_at = NULL,5 claim_timeout_at = NULL,6 status = 'pending'7WHERE execution_id = 'the-stuck-execution-id'8 AND status = 'running';Problem 2: High Failure Rate
1-- Find the most common errors2SELECT3 workflow_name,4 error,5 COUNT(*) AS failure_count,6 MAX(completed_at) AS last_failure7FROM workflow_executions8WHERE status = 'failed'9 AND completed_at > NOW() - INTERVAL '1 hour'10GROUP BY workflow_name, error11ORDER BY failure_count DESC12LIMIT 20;Common causes:
- External service outage (check error messages for HTTP timeouts, connection refused)
- Database connection exhaustion (check connection count)
- Code bug in task implementation (check application logs)
Problem 3: Increasing Latency
1-- Compare current vs. historical duration2SELECT3 workflow_name,4 AVG(duration_ms) FILTER (WHERE completed_at > NOW() - INTERVAL '1 hour') AS current_avg_ms,5 AVG(duration_ms) FILTER (WHERE completed_at BETWEEN NOW() - INTERVAL '25 hours' AND NOW() - INTERVAL '24 hours') AS yesterday_avg_ms6FROM workflow_executions7WHERE status = 'completed'8GROUP BY workflow_name9HAVING AVG(duration_ms) FILTER (WHERE completed_at > NOW() - INTERVAL '1 hour') IS NOT NULL;Common causes:
- Database bloat (run VACUUM ANALYZE)
- Too many concurrent workflows (check worker concurrency settings)
- External service degradation (check task-level durations)
- Missing indexes (check
pg_stat_user_tablesfor sequential scans)
Backup and Disaster Recovery
Since PostgreSQL is your only stateful component, backup strategy is straightforward.
Automated Backups (Managed PostgreSQL):
- Enable daily automated backups with 7-30 day retention
- Enable point-in-time recovery (PITR) for sub-second RPO
- Test restores quarterly
Self-Managed Backups:
1# Daily logical backup2pg_dump -Fc -h postgres -U orcher -d orcher > backup_$(date +%Y%m%d).dump34# Restore from backup5pg_restore -h postgres -U orcher -d orcher_restored backup_20260317.dumpFor near-zero RPO, use WAL archiving:
1# postgresql.conf2archive_mode = on3archive_command = 'aws s3 cp %p s3://your-bucket/wal/%f'Recovery Time Objectives (RTO):
| Scenario | RTO | Strategy |
|---|---|---|
| Worker pod crash | Seconds | Kubernetes auto-restart |
| All workers down | 1-2 minutes | HPA scales up, workflows resume from journal |
| Database failover (managed) | 30-60 seconds | Automatic Multi-AZ failover |
| Database corruption | 5-30 minutes | Restore from backup + WAL replay |
| Full region failure | 15-60 minutes | Restore in new region from cross-region backup |
Upgrade Strategies
Application Upgrades (zero-downtime):
1# Rolling update (zero downtime)2kubectl set image deployment/orcher-api api=your-registry.com/orcher-app:v2.03kubectl set image deployment/orcher-worker worker=your-registry.com/orcher-app:v2.045# Monitor the rollout6kubectl rollout status deployment/orcher-api -n orcher7kubectl rollout status deployment/orcher-worker -n orcher89# Rollback if something goes wrong10kubectl rollout undo deployment/orcher-api -n orcher11kubectl rollout undo deployment/orcher-worker -n orcherKey principle: Since workers are stateless, you can upgrade them independently, and old and new workers can run simultaneously. But Orcher routes work by workflow name and task queue — not by code version — so during a rolling deploy an in-flight workflow may be picked up and continued by either an old or a new worker. That is safe only when the change is replay-compatible (it doesn't alter the step sequence of workflows already in progress). For a breaking change to workflow logic, drain in-flight executions first or register the new logic under a distinct workflow name (see Versioning).
Database Schema Migrations:
Orcher manages its own schema migrations automatically on startup. When upgrading:
- Deploy new application version (it runs migrations on first connect)
- Migrations are forward-compatible (old version can still read the schema)
- Never run migrations manually unless Orcher docs specifically instruct you to
Common Failure Modes and Prevention
| Failure Mode | Symptoms | Prevention | Resolution |
|---|---|---|---|
| Database connection exhaustion | Timeouts, "too many connections" errors | Use PgBouncer, set appropriate max_connections | Restart PgBouncer, reduce worker concurrency |
| Table bloat | Increasing query latency, disk usage growth | Aggressive autovacuum settings (see above) | Run VACUUM FULL during maintenance window |
| Worker memory leak | OOMKilled pods, increasing memory usage | Set resource limits, monitor memory per workflow | Restart workers, investigate task code |
| Long-running transactions | Lock contention, vacuum blocked | Set statement_timeout, idle_in_transaction_session_timeout | Terminate long transactions, investigate cause |
| Network partition (app to DB) | Connection errors, workflow timeouts | Health checks, connection retry logic | Restore connectivity, workflows auto-resume |
On-Call Runbook
Severity 1: All Workflows Stopped
- Check database connectivity:
psql -h postgres -U orcher -c 'SELECT 1' - Check worker pods:
kubectl get pods -n orcher -l component=worker - Check worker logs:
kubectl logs -l component=worker -n orcher --tail=50 - If DB is down: check cloud provider status page, initiate failover if needed
- If workers are crash-looping: check recent deployment, rollback if needed
Severity 2: High Failure Rate
- Check error distribution: Run the "most common errors" query above
- If external service: Check status page of the failing service, consider circuit breaker
- If database: Check connections, run
VACUUM ANALYZE, check slow query log - If code bug: Rollback to previous version
Severity 3: Increasing Latency
- Check queue depth: If growing, add workers
- Check database: Run
EXPLAIN ANALYZEon slow queries, check for missing indexes - Check external services: Measure task-level durations
- Check vacuum: Ensure autovacuum is running and keeping up
Production Checklist
Before going live, verify each item:
Infrastructure:
- PostgreSQL deployed with HA (Multi-AZ or replication)
- Automated backups enabled with tested restore procedure
- PgBouncer or connection pooling configured
- Resource limits set on all pods/processes
- Health checks configured for API and worker deployments
Database:
- Required indexes verified (
idx_workflow_exec_*) - Autovacuum tuned for write-heavy workload
-
max_connectionssized for expected worker count - Slow query logging enabled (
log_min_duration_statement) - Journal retention/cleanup strategy implemented
Monitoring:
- Queue depth metric with alerting
- Workflow success rate metric with alerting
- P95 workflow duration tracked
- Database connection count monitored
- Dead letter queue growth monitored
- Disk space alerts configured
Operations:
- Rolling update strategy tested
- Rollback procedure documented and tested
- On-call runbook written and accessible
- Disaster recovery plan documented
- Team trained on debugging stuck workflows (SQL queries above)
Next Steps
What You've Learned
Congratulations on completing the Platform Engineers learning path! Here's what you now know:
- Orcher's two-component architecture — stateless app processes + PostgreSQL — and why it's radically simpler to operate than traditional orchestration platforms
- How work flows through the system — from API request to queue insert to worker poll to journal-backed execution with automatic recovery
- Deployment strategies — Kubernetes manifests, Dockerfiles, and configuration for API and worker deployments across Rust, TypeScript, and Python
- Database selection and sizing — managed vs. self-managed PostgreSQL, capacity planning formulas, and realistic cost estimates
- Scaling model — the two independent dimensions (throughput vs. concurrency), worker pool tuning, and when to scale what
- Database performance tuning — PostgreSQL configuration for write-heavy workloads, autovacuum strategy, connection pooling with PgBouncer
- Monitoring and alerting — the five essential metrics (queue depth, success rate, P95 duration, connections, DLQ growth) and when to alert
- Debugging production issues — SQL queries for stuck workflows, high failure rates, and latency investigations
- Backup and disaster recovery — RTO targets, WAL archiving, and tested restore procedures
- Upgrade strategies — zero-downtime rolling updates with rollback capability
- On-call runbooks — severity-based response procedures for common failure modes