Operations

Prometheus Setup


Scrape Configuration

Add the orchestrator to your prometheus.yml:

1
global:
2
scrape_interval: 15s
3
evaluation_interval: 15s
4
5
scrape_configs:
6
# Orcher orchestrator metrics
7
- job_name: 'orcher-orchestrator'
8
static_configs:
9
- targets: ['orchestrator:8001']
10
labels:
11
service: 'orchestrator'
12
metrics_path: /metrics
13
scrape_interval: 10s
14
15
# Orchestrator health (for up/down detection)
16
- job_name: 'orcher-orchestrator-health'
17
static_configs:
18
- targets: ['orchestrator:8001']
19
metrics_path: /health
20
scrape_interval: 30s

For infrastructure metrics, add exporters for your backing services:

1
# Redis (via redis_exporter)
2
- job_name: 'redis'
3
static_configs:
4
- targets: ['redis-exporter:9121']
5
6
# PostgreSQL (via postgres_exporter)
7
- job_name: 'postgres'
8
static_configs:
9
- targets: ['postgres-exporter:9187']
10
11
# PgBouncer (via pgbouncer_exporter)
12
- job_name: 'pgbouncer'
13
static_configs:
14
- targets: ['pgbouncer-exporter:9127']

Docker Compose

The monitoring stack is available via the observability profile:

1
cd docker
2
docker compose --profile observability up -d

This starts Prometheus (port 9090) and Grafana (port 3000) alongside the orchestrator.

Kubernetes

Use a ServiceMonitor with the Prometheus Operator:

1
apiVersion: monitoring.coreos.com/v1
2
kind: ServiceMonitor
3
metadata:
4
name: orcher-orchestrator
5
spec:
6
selector:
7
matchLabels:
8
app: orcher-orchestrator
9
endpoints:
10
- port: metrics
11
interval: 10s
12
path: /metrics

Or annotate pods for discovery:

1
metadata:
2
annotations:
3
prometheus.io/scrape: "true"
4
prometheus.io/port: "8001"
5
prometheus.io/path: "/metrics"

Useful PromQL Queries

Workflow Performance

1
# Workflow success rate (5-minute window)
2
sum(rate(orcher_workflows_completed_total[5m]))
3
/
4
(sum(rate(orcher_workflows_completed_total[5m])) + sum(rate(orcher_workflows_failed_total[5m])))
5
6
# Average workflow duration
7
rate(orcher_workflow_duration_seconds_sum[5m])
8
/
9
rate(orcher_workflow_duration_seconds_count[5m])
10
11
# Active workflows over time
12
orcher_workflows_executing

Task Performance

1
# Task failure rate
2
sum(rate(orcher_tasks_failed_total[5m]))
3
/
4
sum(rate(orcher_tasks_submitted_total[5m]))
5
6
# Average task duration
7
rate(orcher_task_duration_seconds_sum[5m])
8
/
9
rate(orcher_task_duration_seconds_count[5m])
10
11
# Task throughput (tasks/second)
12
sum(rate(orcher_tasks_completed_total[5m]))

Queue Health

1
# Queue depth by queue
2
orcher_queue_depth
3
4
# Publish latency (p99)
5
histogram_quantile(0.99, rate(orcher_queue_publish_latency_seconds_bucket[5m]))
6
7
# End-to-end dispatch latency (LPUSH to BRPOP)
8
histogram_quantile(0.99, rate(orcher_push_to_claim_latency_seconds_bucket[5m]))

Database Health

1
# Connection pool utilization
2
orcher_db_pool_utilization_ratio
3
4
# Query latency (p95)
5
histogram_quantile(0.95, rate(orcher_db_query_duration_seconds_bucket[5m]))
6
7
# Pool acquisition timeouts (should be 0)
8
rate(orcher_db_pool_acquisition_timeouts_total[5m])

Cache Efficiency

1
# Journal cache hit ratio
2
orcher_journal_cache_hit_ratio
3
4
# DB reads avoided by cache
5
rate(orcher_journal_cache_db_reads_avoided[5m])
6
7
# Storage cache hit ratio
8
orcher_storage_cache_hit_ratio

Retry & DLQ

1
# Retry exhaustion rate (tasks going to DLQ)
2
rate(orcher_retry_exhausted_total[5m])
3
4
# DLQ depth
5
orcher_dlq_current_depth
6
7
# DLQ net growth trend
8
rate(orcher_dlq_net_growth[1h])

Next Steps