Operations

Queue Monitoring & Troubleshooting


Queue Monitoring

CLI Commands

Use the CLI to inspect queue state:

1
# List all queues with connected workers
2
orcher queue list
3
4
# Check queue depth across all queues
5
orcher queue depth --watch
6
7
# Queue statistics for a specific queue
8
orcher queue stats payments
9
10
# Show connected pollers for a queue
11
orcher queue pollers payments --verbose

Scaling Based on Queue Depth

Monitor queue depth to decide when to scale workers:

Scale up when:

  • Queue depth is consistently growing
  • Task latency exceeds your SLA
  • Workers are at max concurrency (all slots full)

Scale down when:

  • Queue depth is consistently zero
  • Most pollers are idle (no work returned)
  • CPU usage is below 30% for several minutes

Kubernetes Auto-Scaling

1
apiVersion: autoscaling/v2
2
kind: HorizontalPodAutoscaler
3
metadata:
4
name: payment-worker-hpa
5
spec:
6
scaleTargetRef:
7
apiVersion: apps/v1
8
kind: Deployment
9
name: payment-worker
10
minReplicas: 2
11
maxReplicas: 20
12
metrics:
13
- type: Resource
14
resource:
15
name: cpu
16
target:
17
type: Utilization
18
averageUtilization: 70

Troubleshooting

Tasks Not Being Picked Up

  1. Wrong task queue — Ensure the worker's task_queue matches what the workflow dispatches to
  2. Wrong namespace — Worker and orchestrator namespace must match
  3. Handler not registered — In Rust, ensure #[workflow] and #[task] macros are applied. In Python, ensure auto_discover paths include your handler files
  4. No workers running — Check orcher queue pollers <queue-name> for connected workers
1
# Verify workers are connected to the right queue
2
orcher queue pollers payments --verbose
3
4
# Check queue depth to see if work is queued
5
orcher queue depth
6
7
# Test gRPC connectivity
8
grpcurl -plaintext localhost:50051 list

High Queue Latency

  1. Not enough pollers — Increase workflow_poller_count / task_poller_count (default: 4 each)
  2. Not enough concurrency — Increase max_concurrent_tasks if tasks are I/O-bound
  3. Not enough workers — Add more Worker instances if queue depth is growing
  4. No Redis — Enable Redis so workers are notified of work instead of finding it at the next poll

Queue Depth Growing

Common causes and fixes:

  • Input rate exceeds capacity — Scale workers horizontally (more instances) or vertically (higher max_concurrent_tasks)
  • Slow tasks blocking workers — Move slow tasks to a dedicated queue with its own workers
  • Worker crashes — Check worker logs; unclaimed tasks are automatically re-queued after claim timeout expires
  • Database contention — Use PgBouncer for connection multiplexing

Best Practices

Use descriptive queue names that reflect the workload:

1
payments # Payment processing
2
email-sendgrid # Email via SendGrid
3
ml-inference # ML model inference
4
video-encoding # Video transcoding

Separate queues by resource profile. CPU-bound tasks (ML inference, encoding) should run on different workers than I/O-bound tasks (API calls, email). This lets you tune concurrency and instance types independently.

One queue per Worker instance. Each Worker polls a single task_queue. To handle multiple workloads, run multiple Worker instances — one per queue.

Use namespaces for environment isolation. Development, staging, and production should use different namespaces, not different queue names. This keeps queue names consistent across environments.

Monitor queue depth. A consistently growing queue depth means workers can't keep up. Either scale workers or investigate slow task execution.


Next Steps