Tracing Best Practices
Sampling
Adjust sampling rate based on environment and traffic volume. A sample rate of 1.0 traces everything; 0.01 traces 1%.
| Environment | Recommended Rate | Rationale |
|---|---|---|
| Development | 1.0 (100%) | Full visibility for debugging |
| Staging | 0.5 (50%) | Good coverage without overwhelming storage |
| Production (low traffic) | 0.1 (10%) | Balanced cost and observability |
| Production (high traffic) | 0.01 (1%) | Keep storage costs manageable |
Configure via JaegerConfig.sample_rate or OtlpConfig sampling settings.
Trace Visualization
Jaeger UI
Access the Jaeger UI at http://localhost:16686 to:
- Search traces — Filter by service name, operation, tags, min/max duration
- View timelines — See span relationships and timing in a waterfall chart
- Inspect details — View span attributes, events, and logs
- Compare traces — Side-by-side comparison of different executions
Querying Traces
Useful searches in Jaeger:
| Query | Purpose |
|---|---|
Service: orchestrator, Operation: workflow.start | All workflow executions |
Service: orchestrator, Min Duration: 5s | Slow workflows |
Tags: error=true | Failed operations |
Tags: workflow.id=<id> | Specific workflow trace |
Best Practices
Instrument at meaningful boundaries. Create spans for workflows, tasks, external API calls, and database operations — not for every function call. Too many spans add overhead and noise.
1// Good: span for the task, not for each line2let span = create_task_span("execute", &task_id, Some("process_order"));3let _guard = span.enter();45validate_order(&order)?; // No separate span needed6calculate_total(&order)?; // No separate span needed7charge_payment(&order).await?; // External call — could warrant its own spanAlways propagate context. When calling external services from tasks, inject trace context into headers. Without it, traces are broken across service boundaries.
Record errors in spans. When an operation fails, log the error within the span so it appears in trace visualization:
1match operation().await {2 Ok(result) => Ok(result),3 Err(err) => {4 tracing::error!(error = %err, "Operation failed");5 Err(err)6 }7}Use descriptive span names. Include the operation type and specific handler name:
1// Good2create_task_span("execute", &task_id, Some("charge_payment"));3create_workflow_span("step", &workflow_id, Some("validate_orders"));45// Bad6create_task_span("do_thing", &task_id, None);Lower sampling in production. Full tracing at scale generates enormous data volumes. Start with 1–10% and increase for specific services or workflows as needed.
Troubleshooting
No Traces Appearing
- Check exporter connectivity — Verify the Jaeger/OTLP endpoint is reachable from the orchestrator
- Check feature flag — Ensure
tracing-exportfeature is enabled inCargo.toml - Check sampling rate — A rate of
0.0will drop all traces - Check Jaeger UI — Make sure you're searching the correct service name
1# Test Jaeger collector connectivity2curl -s http://localhost:14268/api/traces34# Check Jaeger UI5open http://localhost:16686Broken Traces (Missing Parent Spans)
Ensure context propagation is correctly wired:
1// Outgoing: inject before sending2let mut headers = HeaderMap::new();3inject_context(&mut headers); // Don't forget this!4client.post(url).headers(headers).send().await?;56// Incoming: extract from request headers7let context = extract_context(req.headers());High Overhead
- Lower sampling rate — Reduce from 1.0 to 0.01–0.1
- Reduce span count — Only instrument significant operations
- Batch exports — Increase
max_batch_sizeandbatch_timeoutto reduce export frequency
Next Steps
- Tracing Overview — Setup and exporter configuration
- Creating Spans — Typed span helpers for workflows, tasks, storage, and workers
- Context Propagation — W3C trace context across service boundaries
- Metrics — Prometheus metrics for monitoring
- Logging — Structured logging configuration
- Debugging — Debug workflow execution
- Orchestrator — Server configuration and deployment