Logging
Orcher uses the tracing crate for structured, leveled logging across all services. Logs are output to stdout in a configurable format — compact single-line for development, structured JSON for production, or pretty multi-line for debugging.
- Structured fields — Key-value pairs attached to every log event via
tracingspans and fields - Four output formats — Compact (default), JSON (production), Pretty (debugging), Bunyan (Node.js compatibility)
- Environment variable config —
LOG_LEVELandLOG_FORMATcontrol behavior, withRUST_LOGfor fine-grained filtering - Per-crate filtering — Set different log levels for Orcher modules, SQLx, Tokio, and other dependencies
- Zero-config defaults — Sensible defaults that work out of the box
Configuration
Environment Variables
| Variable | Default | Description |
|---|---|---|
LOG_LEVEL | info | Log level: trace, debug, info, warn, error |
LOG_FORMAT | compact | Output format: compact, json, pretty, bunyan |
RUST_LOG | — | Fine-grained filter (overrides LOG_LEVEL when set) |
SERVICE_NAME | auto-detected | Service name included in log output |
ENVIRONMENT / ORCHER_ENV | — | Set to production for production defaults |
1# Development (default)2LOG_LEVEL=debug LOG_FORMAT=compact ./orcher-orchestrator34# Production5LOG_LEVEL=info LOG_FORMAT=json ./orcher-orchestrator67# Debugging8LOG_LEVEL=trace LOG_FORMAT=pretty ./orcher-orchestratorPer-Crate Filtering
Use RUST_LOG to set different levels per module. This takes priority over LOG_LEVEL:
1# Verbose workflow logging, quiet storage and SQLx2RUST_LOG="orcher_core=debug,orcher_storage=warn,sqlx=warn,orcher=info"34# Trace polling internals only5RUST_LOG="orcher_dispatch::polling=trace,orcher=info"67# Quiet everything except errors, with debug for your workflow8RUST_LOG="warn,orcher_core::engine=debug"Initialization
Services initialize logging early in main():
1use orcher_monitoring::logging::initialize_logging;23fn main() -> Result<()> {4 // Initialize with service name, level, and format5 initialize_logging("orchestrator", "info", "json")?;67 tracing::info!("Service started");8 Ok(())9}Or read from environment variables automatically:
1use orcher_monitoring::logging::init_from_env;23fn main() -> Result<()> {4 // Reads LOG_LEVEL, LOG_FORMAT, and SERVICE_NAME from env5 init_from_env()?;6 Ok(())7}Preset configurations are also available:
1use orcher_monitoring::logging::{init_logging, LoggingConfig};23// Production: JSON format, info level4let config = LoggingConfig::for_production("orchestrator");56// Development: Compact format, debug level7let config = LoggingConfig::for_development("orchestrator");89// Debugging: Pretty format, trace level, full metadata10let config = LoggingConfig::for_debugging("orchestrator");1112init_logging(config)?;Log Formats
Compact (Default)
Single-line, human-readable output. Best for local development:
12024-01-15T10:30:45Z INFO orchestrator: Starting service version=0.1.022024-01-15T10:30:45Z INFO orcher_core: Task submitted task_id=123 priority=high32024-01-15T10:30:46Z DEBUG orcher_core: Processing workflow workflow_id=abcJSON (Production)
Structured JSON for log aggregation systems (ELK, Splunk, CloudWatch, Loki):
1{"timestamp":"2024-01-15T10:30:45Z","level":"INFO","target":"orchestrator","message":"Starting service","fields":{"version":"0.1.0"}}2{"timestamp":"2024-01-15T10:30:45Z","level":"INFO","target":"orcher_core","message":"Task submitted","fields":{"task_id":"123","priority":"high"}}Pretty (Debugging)
Multi-line, colorized output with file locations and span context:
12024-01-15T10:30:45.123456Z INFO orchestrator2 at src/main.rs:423 in orchestrator::main4 Starting service5 version: 0.1.0Bunyan
JSON output compatible with the Bunyan log format used by Node.js applications. Useful for mixed-language stacks.
Log Levels
From most to least verbose:
| Level | Use Case | Example |
|---|---|---|
trace | Deep internals — poll loops, cache lookups, individual SQL queries | Polling queue: attempting BRPOP on redis key |
debug | Detailed operational info — execution steps, durations, state changes | Workflow execution completed execution_id=abc duration_ms=1234 |
info | Normal operations — service startup, workflow starts, task completions | Starting workflow execution workflow=order_processing |
warn | Potential problems — high queue depth, retry attempts, degraded state | Queue depth exceeds threshold queue=payments depth=5000 |
error | Failures requiring attention — execution failures, connection errors | Workflow execution failed execution_id=abc error=timeout |
Production recommendation: Use info level. Switch to debug when investigating specific issues. Avoid trace in production — it generates high log volume.
Next Steps
- Structured Logging — Using tracing fields,
#[instrument], and span context - Viewing Logs — CLI commands, Docker, and database queries
- Log Aggregation — Kubernetes, Grafana Loki, Elasticsearch, and performance
- Metrics — Prometheus metrics for monitoring
- Tracing — Distributed tracing with OpenTelemetry
- Debugging — Debug workflow execution