Operations

Polling Configuration


Polling Configuration

Each Worker spawns multiple parallel pollers to maximize throughput:

OptionDefaultDescription
workflow_poller_count4Parallel workflow pollers
task_poller_count4Parallel task pollers
actor_poller_count4Parallel actor pollers (if actors registered)
poll_interval100msInterval between poll attempts
max_concurrent_workflows100Max simultaneous workflow executions
max_concurrent_tasks200 (Rust), 100 (Python)Max simultaneous task executions

Concurrency is controlled through channel backpressure, not explicit semaphores. Each driver creates a channel with capacity equal to the max concurrency setting:

1
Pollers → [channel (capacity=100)] → Handler executors

If all slots are occupied by in-flight handlers, the pollers block on send — naturally capping concurrency without coordination overhead.

Tuning Poller Count

Increase poller count when you have high-throughput queues where a single poller can't claim work fast enough:

1
let worker = WorkerBuilder::new()
2
.task_queue("high-throughput")
3
.workflow_poller_count(8) // More workflow pollers
4
.task_poller_count(8) // More task pollers
5
.max_concurrent_tasks(500) // Higher task capacity
6
.build()
7
.await?;

Dead Letter Queue

Tasks that exceed their maximum retry count are moved to the dead letter queue (DLQ) in PostgreSQL. The DLQ preserves the full task payload, error details, and execution history for debugging.

Tasks are retried with exponential backoff before reaching the DLQ. The retry policy is configured per-task:

1
#[task(retry = 3)]
2
async fn charge_payment(ctx: TaskContext, amount: f64) -> Result<PaymentId> {
3
// Retried up to 3 times with exponential backoff
4
// After 3 failures → moved to DLQ
5
Ok(PaymentId::new("pay_123"))
6
}

Redis Fast Path

Redis is optional but recommended for high-throughput production deployments. It provides near-instant task notification instead of periodic PostgreSQL polling.

How It Works

  1. When new work is available, the orchestrator pushes the execution ID to a Redis list (LPUSH)
  2. Workers block on BRPOP — Redis handles the blocking efficiently at the connection level
  3. When an ID arrives, the worker claims the execution in PostgreSQL with SELECT FOR UPDATE SKIP LOCKED
  4. If Redis is unavailable, workers automatically fall back to PostgreSQL polling

Enabling Redis

Configure the orchestrator with a Redis URL:

1
# Orchestrator environment variable
2
REDIS_URL=redis://localhost:6379

No worker-side configuration is needed — the orchestrator handles the Redis integration.

Reliability

Redis is used only as a notification layer. All durable state lives in PostgreSQL. If Redis goes down:

  • Workers fall back to PostgreSQL polling automatically
  • No work is lost — PostgreSQL is the source of truth
  • When Redis recovers, workers switch back to the fast path

Next Steps