Log Aggregation
Since Orcher outputs structured JSON to stdout, it integrates with any log aggregation system that can read container or process output.
Kubernetes
In Kubernetes, stdout logs are automatically collected by the node's log driver. Use a DaemonSet-based collector:
- Fluentd / Fluent Bit — Parse JSON from container stdout, ship to Elasticsearch, CloudWatch, or Loki
- Vector — High-performance log collection with built-in JSON parsing
- Promtail — Ship logs to Grafana Loki for PromQL-like querying
No sidecar or file output is needed — JSON to stdout is the standard pattern.
Grafana Loki
Query Orcher logs with LogQL:
1# All error logs2{job="orcher-orchestrator"} | json | level="ERROR"34# Workflow execution events5{job="orcher-orchestrator"} | json | target="orcher_core"67# Rate of errors over time8rate({job="orcher-orchestrator"} | json | level="ERROR" [5m])Elasticsearch / CloudWatch
JSON logs are natively parseable by Elasticsearch and CloudWatch Insights without additional transformation.
1# CloudWatch Insights example2fields @timestamp, level, message3| filter level = "ERROR"4| sort @timestamp desc5| limit 100Performance
The logging system is designed for minimal overhead:
| Format | Approximate Cost |
|---|---|
| Compact | ~5-10 us per log statement |
| JSON | ~10-15 us per log statement |
| Pretty | ~20-30 us per log statement |
Disabled log levels have near-zero runtime cost thanks to tracing's compile-time callsite optimization. A debug!() call at info level compiles to a single branch check.
Troubleshooting
Logs Not Appearing
- Check log level — Ensure
LOG_LEVELis set low enough (e.g.,debugto seedebug!()messages) - Check
RUST_LOG— If set, it overridesLOG_LEVEL. Unset it to use the simpler config. - Check initialization order — Logging must be initialized before any
tracingmacros are called
Too Much Noise from Dependencies
Use RUST_LOG to quiet noisy crates while keeping Orcher verbose:
1RUST_LOG="orcher=debug,sqlx=warn,hyper=warn,tower=warn,tonic=warn"Logs Not JSON in Production
Set LOG_FORMAT=json explicitly. The default format is compact (not JSON):
1LOG_FORMAT=json ./orcher-orchestratorNext Steps
- Logging Overview — Configuration, formats, and log levels
- Structured Logging — Using tracing fields,
#[instrument], and span context - Viewing Logs — CLI commands, Docker, and database queries
- Metrics — Prometheus metrics for monitoring
- Tracing — Distributed tracing with OpenTelemetry
- Debugging — Debug workflow execution
- Storage — PostgreSQL and Redis configuration