Getting Started

Operating in Production


Now that Orcher is deployed and tuned, let's cover day-to-day operations.

Monitoring and Alerting

The Essential Metrics (monitor these from day one):

1. Workflow Queue Depth

1
-- Current queue depth by status
2
SELECT status, COUNT(*) as count
3
FROM workflow_executions
4
WHERE status IN ('pending', 'queued', 'running')
5
GROUP BY status;

Alert when: Pending count exceeds 2x your normal baseline for more than 5 minutes. This means workflows are arriving faster than workers can process them.

2. Workflow Completion Rate

1
-- Completion rate over the last hour
2
SELECT
3
COUNT(*) FILTER (WHERE status = 'completed') AS completed,
4
COUNT(*) FILTER (WHERE status = 'failed') AS failed,
5
ROUND(
6
100.0 * COUNT(*) FILTER (WHERE status = 'completed') /
7
NULLIF(COUNT(*), 0), 2
8
) AS success_rate_pct
9
FROM workflow_executions
10
WHERE completed_at > NOW() - INTERVAL '1 hour';

Alert when: Success rate drops below 95% over a 15-minute window.

3. Workflow Duration (P95)

1
-- P95 duration by workflow type
2
SELECT
3
workflow_name,
4
PERCENTILE_CONT(0.95) WITHIN GROUP (ORDER BY duration_ms) AS p95_ms,
5
PERCENTILE_CONT(0.50) WITHIN GROUP (ORDER BY duration_ms) AS p50_ms,
6
COUNT(*) AS total
7
FROM workflow_executions
8
WHERE status = 'completed'
9
AND completed_at > NOW() - INTERVAL '1 hour'
10
GROUP BY workflow_name;

Alert when: P95 exceeds 3x the normal baseline.

4. Database Connection Count

1
-- Active connections
2
SELECT COUNT(*) AS active_connections,
3
MAX(pg_stat_activity.backend_type) AS connection_types
4
FROM pg_stat_activity
5
WHERE state = 'active';

Alert when: Connections exceed 80% of max_connections.

5. Dead Letter Queue Growth

1
-- DLQ entries in the last hour
2
SELECT COUNT(*) AS dlq_entries
3
FROM dead_letter_queue
4
WHERE failed_at > NOW() - INTERVAL '1 hour';

Alert when: Any new entries appear (DLQ entries indicate tasks that failed beyond retry limits).

Prometheus Metrics

Export these metrics from your application for Grafana dashboards:

1
# Workflow throughput
2
orcher_workflows_started_total{workflow_name, namespace}
3
orcher_workflows_completed_total{workflow_name, namespace, status}
4
5
# Workflow duration
6
orcher_workflow_duration_seconds{workflow_name, namespace, quantile}
7
8
# Queue depth
9
orcher_queue_depth{status, namespace}
10
11
# Worker utilization
12
orcher_worker_active_workflows{worker_id}
13
orcher_worker_capacity{worker_id}
14
15
# Database health
16
orcher_db_connections_active
17
orcher_db_query_duration_seconds{query_type}

Debugging Production Issues

Problem 1: Workflows Are Stuck

Symptoms: Workflows stay in 'running' for too long.

1
-- Find stuck workflows
2
SELECT
3
execution_id,
4
workflow_name,
5
status,
6
claimed_by,
7
claimed_at,
8
claim_timeout_at,
9
EXTRACT(EPOCH FROM (NOW() - claimed_at)) / 60 AS running_minutes
10
FROM workflow_executions
11
WHERE status = 'running'
12
AND claimed_at < NOW() - INTERVAL '10 minutes'
13
ORDER BY claimed_at;

Common causes:

  • Worker crashed without releasing claim (wait for claim timeout, then auto-reclaim)
  • Task making external call that never returns (check task timeouts)
  • Deadlock between workflows (check for circular dependencies)

Resolution: If a workflow is truly stuck, release its claim:

1
-- Release a stuck workflow's claim (it will be picked up by another worker)
2
UPDATE workflow_executions
3
SET claimed_by = NULL,
4
claimed_at = NULL,
5
claim_timeout_at = NULL,
6
status = 'pending'
7
WHERE execution_id = 'the-stuck-execution-id'
8
AND status = 'running';

Problem 2: High Failure Rate

1
-- Find the most common errors
2
SELECT
3
workflow_name,
4
error,
5
COUNT(*) AS failure_count,
6
MAX(completed_at) AS last_failure
7
FROM workflow_executions
8
WHERE status = 'failed'
9
AND completed_at > NOW() - INTERVAL '1 hour'
10
GROUP BY workflow_name, error
11
ORDER BY failure_count DESC
12
LIMIT 20;

Common causes:

  • External service outage (check error messages for HTTP timeouts, connection refused)
  • Database connection exhaustion (check connection count)
  • Code bug in task implementation (check application logs)

Problem 3: Increasing Latency

1
-- Compare current vs. historical duration
2
SELECT
3
workflow_name,
4
AVG(duration_ms) FILTER (WHERE completed_at > NOW() - INTERVAL '1 hour') AS current_avg_ms,
5
AVG(duration_ms) FILTER (WHERE completed_at BETWEEN NOW() - INTERVAL '25 hours' AND NOW() - INTERVAL '24 hours') AS yesterday_avg_ms
6
FROM workflow_executions
7
WHERE status = 'completed'
8
GROUP BY workflow_name
9
HAVING AVG(duration_ms) FILTER (WHERE completed_at > NOW() - INTERVAL '1 hour') IS NOT NULL;

Common causes:

  • Database bloat (run VACUUM ANALYZE)
  • Too many concurrent workflows (check worker concurrency settings)
  • External service degradation (check task-level durations)
  • Missing indexes (check pg_stat_user_tables for sequential scans)

Backup and Disaster Recovery

Since PostgreSQL is your only stateful component, backup strategy is straightforward.

Automated Backups (Managed PostgreSQL):

  • Enable daily automated backups with 7-30 day retention
  • Enable point-in-time recovery (PITR) for sub-second RPO
  • Test restores quarterly

Self-Managed Backups:

1
# Daily logical backup
2
pg_dump -Fc -h postgres -U orcher -d orcher > backup_$(date +%Y%m%d).dump
3
4
# Restore from backup
5
pg_restore -h postgres -U orcher -d orcher_restored backup_20260317.dump

For near-zero RPO, use WAL archiving:

1
# postgresql.conf
2
archive_mode = on
3
archive_command = 'aws s3 cp %p s3://your-bucket/wal/%f'

Recovery Time Objectives (RTO):

ScenarioRTOStrategy
Worker pod crashSecondsKubernetes auto-restart
All workers down1-2 minutesHPA scales up, workflows resume from journal
Database failover (managed)30-60 secondsAutomatic Multi-AZ failover
Database corruption5-30 minutesRestore from backup + WAL replay
Full region failure15-60 minutesRestore in new region from cross-region backup

Upgrade Strategies

Application Upgrades (zero-downtime):

1
# Rolling update (zero downtime)
2
kubectl set image deployment/orcher-api api=your-registry.com/orcher-app:v2.0
3
kubectl set image deployment/orcher-worker worker=your-registry.com/orcher-app:v2.0
4
5
# Monitor the rollout
6
kubectl rollout status deployment/orcher-api -n orcher
7
kubectl rollout status deployment/orcher-worker -n orcher
8
9
# Rollback if something goes wrong
10
kubectl rollout undo deployment/orcher-api -n orcher
11
kubectl rollout undo deployment/orcher-worker -n orcher

Key principle: Since workers are stateless, you can upgrade them independently, and old and new workers can run simultaneously. But Orcher routes work by workflow name and task queue — not by code version — so during a rolling deploy an in-flight workflow may be picked up and continued by either an old or a new worker. That is safe only when the change is replay-compatible (it doesn't alter the step sequence of workflows already in progress). For a breaking change to workflow logic, drain in-flight executions first or register the new logic under a distinct workflow name (see Versioning).

Database Schema Migrations:

Orcher manages its own schema migrations automatically on startup. When upgrading:

  1. Deploy new application version (it runs migrations on first connect)
  2. Migrations are forward-compatible (old version can still read the schema)
  3. Never run migrations manually unless Orcher docs specifically instruct you to

Common Failure Modes and Prevention

Failure ModeSymptomsPreventionResolution
Database connection exhaustionTimeouts, "too many connections" errorsUse PgBouncer, set appropriate max_connectionsRestart PgBouncer, reduce worker concurrency
Table bloatIncreasing query latency, disk usage growthAggressive autovacuum settings (see above)Run VACUUM FULL during maintenance window
Worker memory leakOOMKilled pods, increasing memory usageSet resource limits, monitor memory per workflowRestart workers, investigate task code
Long-running transactionsLock contention, vacuum blockedSet statement_timeout, idle_in_transaction_session_timeoutTerminate long transactions, investigate cause
Network partition (app to DB)Connection errors, workflow timeoutsHealth checks, connection retry logicRestore connectivity, workflows auto-resume

On-Call Runbook

Severity 1: All Workflows Stopped

  1. Check database connectivity: psql -h postgres -U orcher -c 'SELECT 1'
  2. Check worker pods: kubectl get pods -n orcher -l component=worker
  3. Check worker logs: kubectl logs -l component=worker -n orcher --tail=50
  4. If DB is down: check cloud provider status page, initiate failover if needed
  5. If workers are crash-looping: check recent deployment, rollback if needed

Severity 2: High Failure Rate

  1. Check error distribution: Run the "most common errors" query above
  2. If external service: Check status page of the failing service, consider circuit breaker
  3. If database: Check connections, run VACUUM ANALYZE, check slow query log
  4. If code bug: Rollback to previous version

Severity 3: Increasing Latency

  1. Check queue depth: If growing, add workers
  2. Check database: Run EXPLAIN ANALYZE on slow queries, check for missing indexes
  3. Check external services: Measure task-level durations
  4. Check vacuum: Ensure autovacuum is running and keeping up

Production Checklist

Before going live, verify each item:

Infrastructure:

  • PostgreSQL deployed with HA (Multi-AZ or replication)
  • Automated backups enabled with tested restore procedure
  • PgBouncer or connection pooling configured
  • Resource limits set on all pods/processes
  • Health checks configured for API and worker deployments

Database:

  • Required indexes verified (idx_workflow_exec_*)
  • Autovacuum tuned for write-heavy workload
  • max_connections sized for expected worker count
  • Slow query logging enabled (log_min_duration_statement)
  • Journal retention/cleanup strategy implemented

Monitoring:

  • Queue depth metric with alerting
  • Workflow success rate metric with alerting
  • P95 workflow duration tracked
  • Database connection count monitored
  • Dead letter queue growth monitored
  • Disk space alerts configured

Operations:

  • Rolling update strategy tested
  • Rollback procedure documented and tested
  • On-call runbook written and accessible
  • Disaster recovery plan documented
  • Team trained on debugging stuck workflows (SQL queries above)

Next Steps


What You've Learned

Congratulations on completing the Platform Engineers learning path! Here's what you now know:

  • Orcher's two-component architecture — stateless app processes + PostgreSQL — and why it's radically simpler to operate than traditional orchestration platforms
  • How work flows through the system — from API request to queue insert to worker poll to journal-backed execution with automatic recovery
  • Deployment strategies — Kubernetes manifests, Dockerfiles, and configuration for API and worker deployments across Rust, TypeScript, and Python
  • Database selection and sizing — managed vs. self-managed PostgreSQL, capacity planning formulas, and realistic cost estimates
  • Scaling model — the two independent dimensions (throughput vs. concurrency), worker pool tuning, and when to scale what
  • Database performance tuning — PostgreSQL configuration for write-heavy workloads, autovacuum strategy, connection pooling with PgBouncer
  • Monitoring and alerting — the five essential metrics (queue depth, success rate, P95 duration, connections, DLQ growth) and when to alert
  • Debugging production issues — SQL queries for stuck workflows, high failure rates, and latency investigations
  • Backup and disaster recovery — RTO targets, WAL archiving, and tested restore procedures
  • Upgrade strategies — zero-downtime rolling updates with rollback capability
  • On-call runbooks — severity-based response procedures for common failure modes

Back to Overview