Operations

Troubleshooting


Health Checks

All services include health checks:

ServiceCheckInterval
PostgreSQLpg_isready -U orcher -d orcher5s
PgBouncerpg_isready -h localhost -p 5432 -U orcher5s
Redisredis-cli ping5s
Orchestratorcurl -f http://localhost:8080/health/live30s

Volumes

Docker Compose creates named volumes for data persistence:

VolumeMount PointPurpose
orcher-postgres-data/var/lib/postgresql/dataWorkflow state, journal, task executions
orcher-redis-data/dataRedis AOF persistence
orcher-prometheus-data/prometheusMetrics history (30-day retention)
orcher-grafana-data/var/lib/grafanaDashboard state

To reset all data:

1
cd docker
2
docker compose down -v

Troubleshooting

PgBouncer Connection Refused

PgBouncer waits for PostgreSQL to be healthy before starting. If connections fail:

1
# Check service health
2
docker compose ps
3
4
# Check PgBouncer logs
5
docker compose logs pgbouncer
6
7
# Test direct PostgreSQL connection
8
psql postgresql://orcher:orcher@localhost:5432/orcher -c "SELECT 1"
9
10
# Test PgBouncer connection
11
psql postgresql://orcher:orcher@localhost:6432/orcher -c "SELECT 1"

Orchestrator Can't Connect to Database

When running the orchestrator on the host, use localhost ports. When running in a container on orcher-network, use container names:

ContextPostgreSQLPgBouncerRedis
Hostlocalhost:5432localhost:6432localhost:6379
Container (orcher-network)orcher-postgres:5432orcher-pgbouncer:5432orcher-redis:6379

Redis Optional — Fallback to PostgreSQL

If Redis is unavailable, the orchestrator automatically falls back to PostgreSQL-based polling. You'll see the orcher_postgres_fallback_total metric incrementing. This works correctly but with higher dispatch latency: workers find work at their next poll.


Next Steps

  • Docker Overview — Quick start and Docker Compose services
  • Configuration — Environment variables, CLI flags, and TOML configuration
  • Production — Production Dockerfile and full production stack
  • Storage — PostgreSQL schema and migration details
  • Metrics — Prometheus metrics reference