Prometheus Setup
Scrape Configuration
Add the orchestrator to your prometheus.yml:
1global:2 scrape_interval: 15s3 evaluation_interval: 15s45scrape_configs:6 # Orcher orchestrator metrics7 - job_name: 'orcher-orchestrator'8 static_configs:9 - targets: ['orchestrator:8001']10 labels:11 service: 'orchestrator'12 metrics_path: /metrics13 scrape_interval: 10s1415 # Orchestrator health (for up/down detection)16 - job_name: 'orcher-orchestrator-health'17 static_configs:18 - targets: ['orchestrator:8001']19 metrics_path: /health20 scrape_interval: 30sFor infrastructure metrics, add exporters for your backing services:
1# Redis (via redis_exporter)2 - job_name: 'redis'3 static_configs:4 - targets: ['redis-exporter:9121']56 # PostgreSQL (via postgres_exporter)7 - job_name: 'postgres'8 static_configs:9 - targets: ['postgres-exporter:9187']1011 # PgBouncer (via pgbouncer_exporter)12 - job_name: 'pgbouncer'13 static_configs:14 - targets: ['pgbouncer-exporter:9127']Docker Compose
The monitoring stack is available via the observability profile:
1cd docker2docker compose --profile observability up -dThis starts Prometheus (port 9090) and Grafana (port 3000) alongside the orchestrator.
Kubernetes
Use a ServiceMonitor with the Prometheus Operator:
1apiVersion: monitoring.coreos.com/v12kind: ServiceMonitor3metadata:4 name: orcher-orchestrator5spec:6 selector:7 matchLabels:8 app: orcher-orchestrator9 endpoints:10 - port: metrics11 interval: 10s12 path: /metricsOr annotate pods for discovery:
1metadata:2 annotations:3 prometheus.io/scrape: "true"4 prometheus.io/port: "8001"5 prometheus.io/path: "/metrics"Useful PromQL Queries
Workflow Performance
1# Workflow success rate (5-minute window)2sum(rate(orcher_workflows_completed_total[5m]))3/4(sum(rate(orcher_workflows_completed_total[5m])) + sum(rate(orcher_workflows_failed_total[5m])))56# Average workflow duration7rate(orcher_workflow_duration_seconds_sum[5m])8/9rate(orcher_workflow_duration_seconds_count[5m])1011# Active workflows over time12orcher_workflows_executingTask Performance
1# Task failure rate2sum(rate(orcher_tasks_failed_total[5m]))3/4sum(rate(orcher_tasks_submitted_total[5m]))56# Average task duration7rate(orcher_task_duration_seconds_sum[5m])8/9rate(orcher_task_duration_seconds_count[5m])1011# Task throughput (tasks/second)12sum(rate(orcher_tasks_completed_total[5m]))Queue Health
1# Queue depth by queue2orcher_queue_depth34# Publish latency (p99)5histogram_quantile(0.99, rate(orcher_queue_publish_latency_seconds_bucket[5m]))67# End-to-end dispatch latency (LPUSH to BRPOP)8histogram_quantile(0.99, rate(orcher_push_to_claim_latency_seconds_bucket[5m]))Database Health
1# Connection pool utilization2orcher_db_pool_utilization_ratio34# Query latency (p95)5histogram_quantile(0.95, rate(orcher_db_query_duration_seconds_bucket[5m]))67# Pool acquisition timeouts (should be 0)8rate(orcher_db_pool_acquisition_timeouts_total[5m])Cache Efficiency
1# Journal cache hit ratio2orcher_journal_cache_hit_ratio34# DB reads avoided by cache5rate(orcher_journal_cache_db_reads_avoided[5m])67# Storage cache hit ratio8orcher_storage_cache_hit_ratioRetry & DLQ
1# Retry exhaustion rate (tasks going to DLQ)2rate(orcher_retry_exhausted_total[5m])34# DLQ depth5orcher_dlq_current_depth67# DLQ net growth trend8rate(orcher_dlq_net_growth[1h])Next Steps
- Alerting — Recommended alert rules
- Metric Reference — Full list of all exposed metrics
- Metrics Overview — Endpoint configuration