Operations

Distributed Tracing


Orcher provides distributed tracing built on OpenTelemetry standards. The orcher-monitoring crate includes span creation helpers for workflows, tasks, workers, and storage operations, with context propagation across service boundaries and export to Jaeger or any OTLP-compatible backend.

  • OpenTelemetry native — Standard span creation, context propagation, and W3C trace context headers
  • Span helpers — create_workflow_span, create_task_span, create_worker_span, create_storage_span for consistent instrumentation
  • Multiple exporters — Jaeger, OTLP (gRPC, HTTP/Protobuf, HTTP/JSON), and console output
  • Context propagation — inject_context / extract_context for trace continuity across HTTP boundaries
  • Feature-gated — Enabled via the tracing-export Cargo feature (on by default)

Setup

Monitoring Builder

The MonitoringBuilder API provides integrated initialization of metrics, tracing, and health checks:

1
use orcher_monitoring::MonitoringBuilder;
2
3
let mut monitoring = MonitoringBuilder::new()
4
.with_service_name("orchestrator")
5
.with_prometheus("0.0.0.0:8001".parse()?)
6
.with_jaeger("orchestrator", "localhost:6831")
7
.build()
8
.await?;
9
10
monitoring.start().await?;

Presets are available for common configurations:

1
use orcher_monitoring::presets;
2
3
// Minimal: Prometheus metrics only
4
let monitoring = presets::minimal_monitoring();
5
6
// Full: Prometheus + Jaeger + health checks
7
let monitoring = presets::full_observability("orchestrator", "localhost:6831");
8
9
// Cloud-native: OTLP for both metrics and traces
10
let monitoring = presets::cloud_native_otlp("http://otel-collector:4317");

Jaeger

Export traces to Jaeger for visualization:

1
use orcher_monitoring::exporters::jaeger::JaegerConfig;
2
3
let config = JaegerConfig {
4
service_name: "orchestrator".to_string(),
5
agent_endpoint: Some("localhost:6831".to_string()), // UDP agent
6
collector_endpoint: Some("http://localhost:14268/api/traces".to_string()), // HTTP collector
7
sample_rate: 1.0, // 1.0 = 100%, 0.1 = 10%
8
max_batch_size: 512,
9
batch_timeout: Duration::from_secs(5),
10
};

Run Jaeger locally:

1
docker run -d --name jaeger \
2
-p 6831:6831/udp \
3
-p 16686:16686 \
4
-p 14268:14268 \
5
jaegertracing/all-in-one:latest
6
7
# Jaeger UI at http://localhost:16686

OTLP

Export to any OpenTelemetry Collector:

1
use orcher_monitoring::exporters::otlp::OtlpConfig;
2
3
let config = OtlpConfig {
4
endpoint: "http://otel-collector:4317".to_string(),
5
protocol: OtlpProtocol::Grpc, // or HttpProtobuf, HttpJson
6
export_traces: true,
7
export_metrics: true,
8
timeout: Duration::from_secs(10),
9
max_batch_size: 512,
10
batch_timeout: Duration::from_secs(5),
11
headers: HashMap::new(), // Custom headers (API keys, etc.)
12
};

OpenTelemetry Collector config:

1
# otel-collector-config.yaml
2
receivers:
3
otlp:
4
protocols:
5
grpc:
6
endpoint: 0.0.0.0:4317
7
8
processors:
9
batch:
10
timeout: 10s
11
send_batch_size: 1024
12
13
exporters:
14
jaeger:
15
endpoint: jaeger-collector:14250
16
tls:
17
insecure: true
18
19
service:
20
pipelines:
21
traces:
22
receivers: [otlp]
23
processors: [batch]
24
exporters: [jaeger]

Next Steps

  • Creating Spans — Typed span helpers for workflows, tasks, storage, and workers
  • Context Propagation — W3C trace context across service boundaries
  • Best Practices — Sampling, visualization, and troubleshooting
  • Metrics — Prometheus metrics for monitoring
  • Logging — Structured logging configuration