Operations

Tracing Best Practices


Sampling

Adjust sampling rate based on environment and traffic volume. A sample rate of 1.0 traces everything; 0.01 traces 1%.

EnvironmentRecommended RateRationale
Development1.0 (100%)Full visibility for debugging
Staging0.5 (50%)Good coverage without overwhelming storage
Production (low traffic)0.1 (10%)Balanced cost and observability
Production (high traffic)0.01 (1%)Keep storage costs manageable

Configure via JaegerConfig.sample_rate or OtlpConfig sampling settings.


Trace Visualization

Jaeger UI

Access the Jaeger UI at http://localhost:16686 to:

  • Search traces — Filter by service name, operation, tags, min/max duration
  • View timelines — See span relationships and timing in a waterfall chart
  • Inspect details — View span attributes, events, and logs
  • Compare traces — Side-by-side comparison of different executions

Querying Traces

Useful searches in Jaeger:

QueryPurpose
Service: orchestrator, Operation: workflow.startAll workflow executions
Service: orchestrator, Min Duration: 5sSlow workflows
Tags: error=trueFailed operations
Tags: workflow.id=<id>Specific workflow trace

Best Practices

Instrument at meaningful boundaries. Create spans for workflows, tasks, external API calls, and database operations — not for every function call. Too many spans add overhead and noise.

1
// Good: span for the task, not for each line
2
let span = create_task_span("execute", &task_id, Some("process_order"));
3
let _guard = span.enter();
4
5
validate_order(&order)?; // No separate span needed
6
calculate_total(&order)?; // No separate span needed
7
charge_payment(&order).await?; // External call — could warrant its own span

Always propagate context. When calling external services from tasks, inject trace context into headers. Without it, traces are broken across service boundaries.

Record errors in spans. When an operation fails, log the error within the span so it appears in trace visualization:

1
match operation().await {
2
Ok(result) => Ok(result),
3
Err(err) => {
4
tracing::error!(error = %err, "Operation failed");
5
Err(err)
6
}
7
}

Use descriptive span names. Include the operation type and specific handler name:

1
// Good
2
create_task_span("execute", &task_id, Some("charge_payment"));
3
create_workflow_span("step", &workflow_id, Some("validate_orders"));
4
5
// Bad
6
create_task_span("do_thing", &task_id, None);

Lower sampling in production. Full tracing at scale generates enormous data volumes. Start with 1–10% and increase for specific services or workflows as needed.


Troubleshooting

No Traces Appearing

  1. Check exporter connectivity — Verify the Jaeger/OTLP endpoint is reachable from the orchestrator
  2. Check feature flag — Ensure tracing-export feature is enabled in Cargo.toml
  3. Check sampling rate — A rate of 0.0 will drop all traces
  4. Check Jaeger UI — Make sure you're searching the correct service name
1
# Test Jaeger collector connectivity
2
curl -s http://localhost:14268/api/traces
3
4
# Check Jaeger UI
5
open http://localhost:16686

Broken Traces (Missing Parent Spans)

Ensure context propagation is correctly wired:

1
// Outgoing: inject before sending
2
let mut headers = HeaderMap::new();
3
inject_context(&mut headers); // Don't forget this!
4
client.post(url).headers(headers).send().await?;
5
6
// Incoming: extract from request headers
7
let context = extract_context(req.headers());

High Overhead

  • Lower sampling rate — Reduce from 1.0 to 0.01–0.1
  • Reduce span count — Only instrument significant operations
  • Batch exports — Increase max_batch_size and batch_timeout to reduce export frequency

Next Steps