Queue Monitoring & Troubleshooting
Queue Monitoring
CLI Commands
Use the CLI to inspect queue state:
1# List all queues with connected workers2orcher queue list34# Check queue depth across all queues5orcher queue depth --watch67# Queue statistics for a specific queue8orcher queue stats payments910# Show connected pollers for a queue11orcher queue pollers payments --verboseScaling Based on Queue Depth
Monitor queue depth to decide when to scale workers:
Scale up when:
- Queue depth is consistently growing
- Task latency exceeds your SLA
- Workers are at max concurrency (all slots full)
Scale down when:
- Queue depth is consistently zero
- Most pollers are idle (no work returned)
- CPU usage is below 30% for several minutes
Kubernetes Auto-Scaling
1apiVersion: autoscaling/v22kind: HorizontalPodAutoscaler3metadata:4 name: payment-worker-hpa5spec:6 scaleTargetRef:7 apiVersion: apps/v18 kind: Deployment9 name: payment-worker10 minReplicas: 211 maxReplicas: 2012 metrics:13 - type: Resource14 resource:15 name: cpu16 target:17 type: Utilization18 averageUtilization: 70Troubleshooting
Tasks Not Being Picked Up
- Wrong task queue — Ensure the worker's
task_queuematches what the workflow dispatches to - Wrong namespace — Worker and orchestrator namespace must match
- Handler not registered — In Rust, ensure
#[workflow]and#[task]macros are applied. In Python, ensureauto_discoverpaths include your handler files - No workers running — Check
orcher queue pollers <queue-name>for connected workers
1# Verify workers are connected to the right queue2orcher queue pollers payments --verbose34# Check queue depth to see if work is queued5orcher queue depth67# Test gRPC connectivity8grpcurl -plaintext localhost:50051 listHigh Queue Latency
- Not enough pollers — Increase
workflow_poller_count/task_poller_count(default: 4 each) - Not enough concurrency — Increase
max_concurrent_tasksif tasks are I/O-bound - Not enough workers — Add more Worker instances if queue depth is growing
- No Redis — Enable Redis so workers are notified of work instead of finding it at the next poll
Queue Depth Growing
Common causes and fixes:
- Input rate exceeds capacity — Scale workers horizontally (more instances) or vertically (higher
max_concurrent_tasks) - Slow tasks blocking workers — Move slow tasks to a dedicated queue with its own workers
- Worker crashes — Check worker logs; unclaimed tasks are automatically re-queued after claim timeout expires
- Database contention — Use PgBouncer for connection multiplexing
Best Practices
Use descriptive queue names that reflect the workload:
1payments # Payment processing2email-sendgrid # Email via SendGrid3ml-inference # ML model inference4video-encoding # Video transcodingSeparate queues by resource profile. CPU-bound tasks (ML inference, encoding) should run on different workers than I/O-bound tasks (API calls, email). This lets you tune concurrency and instance types independently.
One queue per Worker instance. Each Worker polls a single task_queue. To handle multiple workloads, run multiple Worker instances — one per queue.
Use namespaces for environment isolation. Development, staging, and production should use different namespaces, not different queue names. This keeps queue names consistent across environments.
Monitor queue depth. A consistently growing queue depth means workers can't keep up. Either scale workers or investigate slow task execution.
Next Steps
- Queues Overview — How queues work in Orcher
- Configuring Queues — Assign workers, isolate workloads, and use namespaces
- Polling & Dead Letter Queue — Tune pollers, configure retries, and enable Redis fast path
- Workers — Deploy and manage workers
- Orchestrator Server — How the coordination engine works
- Storage Setup — Configure PostgreSQL and Redis
- Observability — Metrics, tracing, and logging