Distributed Tracing
Orcher provides distributed tracing built on OpenTelemetry standards. The orcher-monitoring crate includes span creation helpers for workflows, tasks, workers, and storage operations, with context propagation across service boundaries and export to Jaeger or any OTLP-compatible backend.
- OpenTelemetry native — Standard span creation, context propagation, and W3C trace context headers
- Span helpers —
create_workflow_span,create_task_span,create_worker_span,create_storage_spanfor consistent instrumentation - Multiple exporters — Jaeger, OTLP (gRPC, HTTP/Protobuf, HTTP/JSON), and console output
- Context propagation —
inject_context/extract_contextfor trace continuity across HTTP boundaries - Feature-gated — Enabled via the
tracing-exportCargo feature (on by default)
Setup
Monitoring Builder
The MonitoringBuilder API provides integrated initialization of metrics, tracing, and health checks:
1use orcher_monitoring::MonitoringBuilder;23let mut monitoring = MonitoringBuilder::new()4 .with_service_name("orchestrator")5 .with_prometheus("0.0.0.0:8001".parse()?)6 .with_jaeger("orchestrator", "localhost:6831")7 .build()8 .await?;910monitoring.start().await?;Presets are available for common configurations:
1use orcher_monitoring::presets;23// Minimal: Prometheus metrics only4let monitoring = presets::minimal_monitoring();56// Full: Prometheus + Jaeger + health checks7let monitoring = presets::full_observability("orchestrator", "localhost:6831");89// Cloud-native: OTLP for both metrics and traces10let monitoring = presets::cloud_native_otlp("http://otel-collector:4317");Jaeger
Export traces to Jaeger for visualization:
1use orcher_monitoring::exporters::jaeger::JaegerConfig;23let config = JaegerConfig {4 service_name: "orchestrator".to_string(),5 agent_endpoint: Some("localhost:6831".to_string()), // UDP agent6 collector_endpoint: Some("http://localhost:14268/api/traces".to_string()), // HTTP collector7 sample_rate: 1.0, // 1.0 = 100%, 0.1 = 10%8 max_batch_size: 512,9 batch_timeout: Duration::from_secs(5),10};Run Jaeger locally:
1docker run -d --name jaeger \2 -p 6831:6831/udp \3 -p 16686:16686 \4 -p 14268:14268 \5 jaegertracing/all-in-one:latest67# Jaeger UI at http://localhost:16686OTLP
Export to any OpenTelemetry Collector:
1use orcher_monitoring::exporters::otlp::OtlpConfig;23let config = OtlpConfig {4 endpoint: "http://otel-collector:4317".to_string(),5 protocol: OtlpProtocol::Grpc, // or HttpProtobuf, HttpJson6 export_traces: true,7 export_metrics: true,8 timeout: Duration::from_secs(10),9 max_batch_size: 512,10 batch_timeout: Duration::from_secs(5),11 headers: HashMap::new(), // Custom headers (API keys, etc.)12};OpenTelemetry Collector config:
1# otel-collector-config.yaml2receivers:3 otlp:4 protocols:5 grpc:6 endpoint: 0.0.0.0:431778processors:9 batch:10 timeout: 10s11 send_batch_size: 10241213exporters:14 jaeger:15 endpoint: jaeger-collector:1425016 tls:17 insecure: true1819service:20 pipelines:21 traces:22 receivers: [otlp]23 processors: [batch]24 exporters: [jaeger]Next Steps
- Creating Spans — Typed span helpers for workflows, tasks, storage, and workers
- Context Propagation — W3C trace context across service boundaries
- Best Practices — Sampling, visualization, and troubleshooting
- Metrics — Prometheus metrics for monitoring
- Logging — Structured logging configuration