Operations

Metric Reference


All metric names are defined in the orcher-monitoring crate and use the orcher_ prefix.

Workflow Metrics

MetricTypeDescription
orcher_workflows_executingGaugeCurrently executing workflows
orcher_workflows_completed_totalCounterTotal workflows completed successfully
orcher_workflows_failed_totalCounterTotal workflows failed
orcher_workflow_duration_secondsHistogramEnd-to-end workflow execution duration
orcher_workflow_step_duration_secondsHistogramDuration of individual workflow steps

Task Metrics

MetricTypeDescription
orcher_tasks_submitted_totalCounterTotal tasks submitted for execution
orcher_tasks_runningGaugeCurrently executing tasks
orcher_tasks_completed_totalCounterTotal tasks completed successfully
orcher_tasks_failed_totalCounterTotal tasks failed
orcher_task_duration_secondsHistogramTask execution duration
orcher_task_queue_sizeGaugeNumber of tasks waiting in queue

Actor Metrics

MetricTypeDescription
orcher_actor_operations_totalCounterTotal actor operations executed
orcher_actor_operations_failed_totalCounterTotal actor operations failed
orcher_actor_operations_executingGaugeCurrently executing actor operations
orcher_actor_operation_duration_secondsHistogramActor operation duration

Queue Metrics

MetricTypeDescription
orcher_queue_depthGaugeCurrent number of pending items per queue
orcher_queue_messages_published_totalCounterTotal messages published to queues
orcher_queue_messages_consumed_totalCounterTotal messages consumed from queues
orcher_queue_publish_errors_totalCounterTotal publish errors
orcher_queue_consume_errors_totalCounterTotal consume errors
orcher_queue_publish_latency_secondsHistogramTime to publish a message to a queue

Polling Metrics

MetricTypeDescription
orcher_poll_duration_secondsHistogramDuration of poll operations
orcher_poll_totalCounterTotal poll attempts
orcher_poll_success_totalCounterPolls that returned work
orcher_poll_timeout_totalCounterPolls that timed out (no work available)
orcher_poll_error_totalCounterPolls that errored
orcher_claim_duration_secondsHistogramTime to claim an execution after poll
orcher_push_to_claim_latency_secondsHistogramEnd-to-end latency from LPUSH to BRPOP return
orcher_redis_push_duration_secondsHistogramTime for Redis LPUSH
orcher_redis_pop_duration_secondsHistogramTime for Redis BRPOP
orcher_postgres_fallback_totalCounterTimes PostgreSQL fallback was used (Redis unavailable)
orcher_redis_retry_totalCounterRedis operation retries
orcher_batch_claim_sizeHistogramNumber of items claimed per batch
orcher_batch_claim_success_ratioGaugeRatio of successful claims in a batch

Database Pool Metrics

MetricTypeDescription
orcher_db_connections_activeGaugeActive database connections
orcher_db_connections_idleGaugeIdle database connections
orcher_db_connections_maxGaugeMaximum pool size
orcher_db_connections_totalCounterTotal connections created
orcher_db_pool_utilization_ratioGaugeactive / max (0.0–1.0)
orcher_db_pool_acquisitions_totalCounterTotal pool acquisitions
orcher_db_pool_acquisition_timeouts_totalCounterPool acquisition timeouts
orcher_db_pool_acquisition_duration_secondsHistogramTime to acquire a connection from pool
orcher_db_query_duration_secondsHistogramQuery execution duration
orcher_db_queries_totalCounterTotal queries executed
orcher_db_query_errors_totalCounterTotal query errors
orcher_db_health_statusGauge1 = healthy, 0 = unhealthy

Journal Cache Metrics

MetricTypeDescription
orcher_journal_cache_hits_totalCounterCache hits
orcher_journal_cache_misses_totalCounterCache misses (full DB load)
orcher_journal_cache_hit_ratioGaugeHit ratio (0.0–1.0)
orcher_journal_cache_incremental_loads_totalCounterCache hit but needed DB load for new entries
orcher_journal_cache_incremental_entries_loadedCounterEntries loaded incrementally
orcher_journal_cache_db_reads_avoidedCounterDatabase reads avoided due to cache
orcher_journal_cache_sizeGaugeNumber of cached journals

Retry Metrics

MetricTypeDescription
orcher_retry_total_retriesCounterTotal retry attempts
orcher_retry_by_attemptCounterRetries broken down by attempt number
orcher_retry_successes_totalCounterRetries that succeeded
orcher_retry_exhausted_totalCounterRetries exhausted (moved to DLQ)
orcher_retry_success_rateGaugeRetry success rate (0.0–1.0)
orcher_retry_average_per_taskGaugeAverage retries per task
orcher_retry_backoff_duration_secondsHistogramBackoff wait duration

Dead Letter Queue Metrics

MetricTypeDescription
orcher_dlq_current_depthGaugeCurrent DLQ depth
orcher_dlq_tasks_moved_totalCounterTasks moved to DLQ
orcher_dlq_tasks_replayed_totalCounterTasks replayed from DLQ
orcher_dlq_replay_failures_totalCounterDLQ replay failures
orcher_dlq_tasks_purged_totalCounterTasks purged from DLQ
orcher_dlq_replay_success_rateGaugeReplay success rate (0.0–1.0)
orcher_dlq_net_growthGaugeNet DLQ growth (moved - replayed - purged)

Orchestrator Metrics

MetricTypeDescription
orcher_orchestrator_active_workflowsGaugeActive workflows across all queues
orcher_orchestrator_active_tasksGaugeActive tasks across all queues
orcher_orchestrator_tasks_submitted_totalCounterTasks submitted by orchestrator
orcher_orchestrator_tasks_cancelled_totalCounterTasks cancelled
orcher_orchestrator_workflows_paused_totalCounterWorkflows paused
orcher_orchestrator_workflows_resumed_totalCounterWorkflows resumed
orcher_orchestrator_workflows_cancelled_totalCounterWorkflows cancelled

Scheduler Metrics

MetricTypeDescription
orcher_scheduler_queue_sizeGaugeScheduled triggers waiting
orcher_scheduler_triggers_fired_totalCounterTotal scheduled triggers fired
orcher_scheduler_next_fire_timeGaugeNext trigger time (epoch seconds)

System Metrics

MetricTypeDescription
orcher_system_cpu_usageGaugeCPU usage
orcher_system_memory_usageGaugeMemory usage
orcher_system_disk_usageGaugeDisk usage
orcher_system_network_io_bytesCounterNetwork I/O bytes

Next Steps