Operations

Log Aggregation


Since Orcher outputs structured JSON to stdout, it integrates with any log aggregation system that can read container or process output.

Kubernetes

In Kubernetes, stdout logs are automatically collected by the node's log driver. Use a DaemonSet-based collector:

  • Fluentd / Fluent Bit — Parse JSON from container stdout, ship to Elasticsearch, CloudWatch, or Loki
  • Vector — High-performance log collection with built-in JSON parsing
  • Promtail — Ship logs to Grafana Loki for PromQL-like querying

No sidecar or file output is needed — JSON to stdout is the standard pattern.

Grafana Loki

Query Orcher logs with LogQL:

1
# All error logs
2
{job="orcher-orchestrator"} | json | level="ERROR"
3
4
# Workflow execution events
5
{job="orcher-orchestrator"} | json | target="orcher_core"
6
7
# Rate of errors over time
8
rate({job="orcher-orchestrator"} | json | level="ERROR" [5m])

Elasticsearch / CloudWatch

JSON logs are natively parseable by Elasticsearch and CloudWatch Insights without additional transformation.

1
# CloudWatch Insights example
2
fields @timestamp, level, message
3
| filter level = "ERROR"
4
| sort @timestamp desc
5
| limit 100

Performance

The logging system is designed for minimal overhead:

FormatApproximate Cost
Compact~5-10 us per log statement
JSON~10-15 us per log statement
Pretty~20-30 us per log statement

Disabled log levels have near-zero runtime cost thanks to tracing's compile-time callsite optimization. A debug!() call at info level compiles to a single branch check.


Troubleshooting

Logs Not Appearing

  1. Check log level — Ensure LOG_LEVEL is set low enough (e.g., debug to see debug!() messages)
  2. Check RUST_LOG — If set, it overrides LOG_LEVEL. Unset it to use the simpler config.
  3. Check initialization order — Logging must be initialized before any tracing macros are called

Too Much Noise from Dependencies

Use RUST_LOG to quiet noisy crates while keeping Orcher verbose:

1
RUST_LOG="orcher=debug,sqlx=warn,hyper=warn,tower=warn,tonic=warn"

Logs Not JSON in Production

Set LOG_FORMAT=json explicitly. The default format is compact (not JSON):

1
LOG_FORMAT=json ./orcher-orchestrator

Next Steps

  • Logging Overview — Configuration, formats, and log levels
  • Structured Logging — Using tracing fields, #[instrument], and span context
  • Viewing Logs — CLI commands, Docker, and database queries
  • Metrics — Prometheus metrics for monitoring
  • Tracing — Distributed tracing with OpenTelemetry
  • Debugging — Debug workflow execution
  • Storage — PostgreSQL and Redis configuration