Operations

Logging


Orcher uses the tracing crate for structured, leveled logging across all services. Logs are output to stdout in a configurable format — compact single-line for development, structured JSON for production, or pretty multi-line for debugging.

  • Structured fields — Key-value pairs attached to every log event via tracing spans and fields
  • Four output formats — Compact (default), JSON (production), Pretty (debugging), Bunyan (Node.js compatibility)
  • Environment variable config — LOG_LEVEL and LOG_FORMAT control behavior, with RUST_LOG for fine-grained filtering
  • Per-crate filtering — Set different log levels for Orcher modules, SQLx, Tokio, and other dependencies
  • Zero-config defaults — Sensible defaults that work out of the box

Configuration

Environment Variables

VariableDefaultDescription
LOG_LEVELinfoLog level: trace, debug, info, warn, error
LOG_FORMATcompactOutput format: compact, json, pretty, bunyan
RUST_LOG—Fine-grained filter (overrides LOG_LEVEL when set)
SERVICE_NAMEauto-detectedService name included in log output
ENVIRONMENT / ORCHER_ENV—Set to production for production defaults
1
# Development (default)
2
LOG_LEVEL=debug LOG_FORMAT=compact ./orcher-orchestrator
3
4
# Production
5
LOG_LEVEL=info LOG_FORMAT=json ./orcher-orchestrator
6
7
# Debugging
8
LOG_LEVEL=trace LOG_FORMAT=pretty ./orcher-orchestrator

Per-Crate Filtering

Use RUST_LOG to set different levels per module. This takes priority over LOG_LEVEL:

1
# Verbose workflow logging, quiet storage and SQLx
2
RUST_LOG="orcher_core=debug,orcher_storage=warn,sqlx=warn,orcher=info"
3
4
# Trace polling internals only
5
RUST_LOG="orcher_dispatch::polling=trace,orcher=info"
6
7
# Quiet everything except errors, with debug for your workflow
8
RUST_LOG="warn,orcher_core::engine=debug"

Initialization

Services initialize logging early in main():

1
use orcher_monitoring::logging::initialize_logging;
2
3
fn main() -> Result<()> {
4
// Initialize with service name, level, and format
5
initialize_logging("orchestrator", "info", "json")?;
6
7
tracing::info!("Service started");
8
Ok(())
9
}

Or read from environment variables automatically:

1
use orcher_monitoring::logging::init_from_env;
2
3
fn main() -> Result<()> {
4
// Reads LOG_LEVEL, LOG_FORMAT, and SERVICE_NAME from env
5
init_from_env()?;
6
Ok(())
7
}

Preset configurations are also available:

1
use orcher_monitoring::logging::{init_logging, LoggingConfig};
2
3
// Production: JSON format, info level
4
let config = LoggingConfig::for_production("orchestrator");
5
6
// Development: Compact format, debug level
7
let config = LoggingConfig::for_development("orchestrator");
8
9
// Debugging: Pretty format, trace level, full metadata
10
let config = LoggingConfig::for_debugging("orchestrator");
11
12
init_logging(config)?;

Log Formats

Compact (Default)

Single-line, human-readable output. Best for local development:

1
2024-01-15T10:30:45Z INFO orchestrator: Starting service version=0.1.0
2
2024-01-15T10:30:45Z INFO orcher_core: Task submitted task_id=123 priority=high
3
2024-01-15T10:30:46Z DEBUG orcher_core: Processing workflow workflow_id=abc

JSON (Production)

Structured JSON for log aggregation systems (ELK, Splunk, CloudWatch, Loki):

1
{"timestamp":"2024-01-15T10:30:45Z","level":"INFO","target":"orchestrator","message":"Starting service","fields":{"version":"0.1.0"}}
2
{"timestamp":"2024-01-15T10:30:45Z","level":"INFO","target":"orcher_core","message":"Task submitted","fields":{"task_id":"123","priority":"high"}}

Pretty (Debugging)

Multi-line, colorized output with file locations and span context:

1
2024-01-15T10:30:45.123456Z INFO orchestrator
2
at src/main.rs:42
3
in orchestrator::main
4
Starting service
5
version: 0.1.0

Bunyan

JSON output compatible with the Bunyan log format used by Node.js applications. Useful for mixed-language stacks.


Log Levels

From most to least verbose:

LevelUse CaseExample
traceDeep internals — poll loops, cache lookups, individual SQL queriesPolling queue: attempting BRPOP on redis key
debugDetailed operational info — execution steps, durations, state changesWorkflow execution completed execution_id=abc duration_ms=1234
infoNormal operations — service startup, workflow starts, task completionsStarting workflow execution workflow=order_processing
warnPotential problems — high queue depth, retry attempts, degraded stateQueue depth exceeds threshold queue=payments depth=5000
errorFailures requiring attention — execution failures, connection errorsWorkflow execution failed execution_id=abc error=timeout

Production recommendation: Use info level. Switch to debug when investigating specific issues. Avoid trace in production — it generates high log volume.


Next Steps

  • Structured Logging — Using tracing fields, #[instrument], and span context
  • Viewing Logs — CLI commands, Docker, and database queries
  • Log Aggregation — Kubernetes, Grafana Loki, Elasticsearch, and performance
  • Metrics — Prometheus metrics for monitoring
  • Tracing — Distributed tracing with OpenTelemetry
  • Debugging — Debug workflow execution