Alerting
Recommended Alert Rules
1groups:2 - name: orcher3 interval: 30s4 rules:5 # High workflow failure rate6 - alert: HighWorkflowFailureRate7 expr: |8 sum(rate(orcher_workflows_failed_total[5m]))9 /10 (sum(rate(orcher_workflows_completed_total[5m])) + sum(rate(orcher_workflows_failed_total[5m])))11 > 0.0512 for: 5m13 labels:14 severity: warning15 annotations:16 summary: "Workflow failure rate above 5%"1718 # Queue depth growing19 - alert: QueueDepthHigh20 expr: orcher_queue_depth > 100021 for: 5m22 labels:23 severity: warning24 annotations:25 summary: "Queue {{ $labels.queue }} has {{ $value }} pending items"2627 # Database pool near exhaustion28 - alert: DBPoolExhausted29 expr: orcher_db_pool_utilization_ratio > 0.9030 for: 5m31 labels:32 severity: critical33 annotations:34 summary: "Database pool at {{ $value | humanizePercentage }} utilization"3536 # Journal cache hit ratio low37 - alert: JournalCacheHitRateLow38 expr: orcher_journal_cache_hit_ratio < 0.8039 for: 15m40 labels:41 severity: warning42 annotations:43 summary: "Journal cache hit ratio is {{ $value | humanizePercentage }}"4445 # DLQ growing46 - alert: DLQGrowing47 expr: orcher_dlq_current_depth > 10048 for: 10m49 labels:50 severity: warning51 annotations:52 summary: "Dead letter queue has {{ $value }} entries"5354 # PostgreSQL fallback active (Redis down)55 - alert: RedisUnavailable56 expr: rate(orcher_postgres_fallback_total[5m]) > 057 for: 5m58 labels:59 severity: warning60 annotations:61 summary: "Workers falling back to PostgreSQL polling (Redis may be down)"Next Steps
- Prometheus Setup — Scrape configuration and useful PromQL queries
- Metric Reference — Full list of all exposed metrics
- Metrics Overview — Endpoint configuration