Scaling and Performance
Now that Orcher is deployed, let's understand how to scale it and optimize performance.
First, How the Pieces Fit
Before tuning anything, it's essential to know which component does what — because it determines where each bottleneck lives:
- Clients call the orchestrator over gRPC to start workflows and send events.
- The orchestrator is the only component that talks to PostgreSQL. It owns the queue, the journal, and every read/write, using its own database connection pool.
- Workers are stateless gRPC clients. They connect to the orchestrator, pull workflow/task work, execute your code, and report results back — all over gRPC. A worker never opens a database connection.
That last point is the one people get wrong. When you "add workers," you are not adding database clients — you are adding execution capacity that talks to the orchestrator. Database connections are driven by the orchestrator, not by your worker count. Keep this in mind for every knob below.
Understanding Orcher's Scaling Model
Orcher scales in two dimensions, and it's important to understand the difference:
Dimension 1: Workflow Throughput (How Fast Can You Start Workflows?)
What it means: How many workflows per second can you enqueue?
Limited by: Orchestrator + PostgreSQL write capacity
Example: Your API receives 1,000 POST requests per second to create orders. Each request calls the orchestrator (start_workflow over gRPC), and the orchestrator enqueues it with one INSERT into workflow_queue. Can the orchestrator + database sustain 1,000 starts/second?
How to scale (all of this is the orchestrator tier — workers are not involved):
- Vertical scale the database: More CPU = more IOPS
- Scale the orchestrator horizontally: run multiple orchestrator nodes against the same database (see Scaling the Orchestrator and Database)
- Use connection pooling: put PgBouncer in front of PostgreSQL for the orchestrator's connections
- Add read replicas: offload analytics queries from the primary
Measure, don't guess. Start throughput depends on your database, its storage and the workflow's shape, so size from a measurement: the sdk-core benchmark suite runs standard scenarios against any engine and reports runs per second.
Dimension 2: Concurrent Workflow Execution (How Many Workflows Can Run Simultaneously?)
What it means: How many workflows are actively being executed right now?
Limited by: Worker pool size x concurrency per worker
Example: You have 10 worker pods, each configured to run 10 workflows concurrently. Your maximum concurrency is 10 x 10 = 100 workflows.
How to scale:
- Horizontal scale workers: Add more worker pods
- Vertical scale workers: Increase concurrency per worker (if tasks are I/O-bound)
- Optimize slow tasks: If a task takes 10 seconds, you can only process 6/minute per worker
Worked arithmetic (assuming an average task duration of 500ms):
- 5 workers x 10 concurrency = 50 concurrent, ~6,000 workflows/minute
- 10 workers x 10 concurrency = 100 concurrent, ~12,000 workflows/minute
- 20 workers x 20 concurrency = 400 concurrent, ~48,000 workflows/minute
Real-world example:
- Payment processing with 3-second average workflow duration
- Need to handle 100 workflows/second sustained = 300 concurrent workflows
- Deploy 15 workers x 20 concurrency = 300 concurrent capacity
The Key Insight: These dimensions are independent. You can start 10,000 workflows/second but only execute 100 concurrently (they queue up). Or you can execute 1,000 concurrently but only start 100/second (workers sit idle).
Worker Pool Configuration Deep Dive
Workers are where the work happens. Let's understand how to tune them.
The Worker Configuration Parameters
1use orcher::prelude::*;2use std::time::Duration;34#[tokio::main]5async fn main() -> Result<()> {6 // A worker is a stateless gRPC client of the orchestrator — no database here.7 // Workflows and tasks are auto-registered from their decorators/macros.8 WorkerBuilder::new()9 .server_url("https://orchestrator.internal:50051") // gRPC endpoint of the orchestrator10 .namespace("production")11 .task_queue("default")12 .max_concurrent_workflows(20) // workflows executing at once in this process13 .max_concurrent_tasks(40) // tasks executing at once in this process14 .poll_interval(Duration::from_millis(100)) // how often to pull work when idle15 .workflow_poller_count(2) // parallel pollers pulling workflow work16 .task_poller_count(4) // parallel pollers pulling task work17 .heartbeat_interval(Duration::from_secs(30)) // liveness reported to the orchestrator18 .build()19 .await?20 .run()21 .await?;2223 Ok(())24}Let's understand each parameter:
1. Concurrency — max_concurrent_workflows / max_concurrent_tasks (most important)
This controls how many workflows (and tasks) execute simultaneously within one worker process.
Low concurrency (1-5):
- Use when: Your tasks are CPU-intensive
- Example: Image processing, video encoding, complex calculations
- Reasoning: CPU-bound tasks don't benefit from parallelism (they're already using the CPU). More concurrency just causes context switching overhead.
Medium concurrency (10-20):
- Use when: Your tasks are mixed I/O and CPU
- Example: API calls with some local processing
- Reasoning: While waiting for API responses, other workflows can execute. But not too many or you'll thrash.
High concurrency (30-50):
- Use when: Your tasks are pure I/O (network calls, database queries)
- Example: Webhooks that just call external services
- Reasoning: You spend 95% of time waiting for I/O. High concurrency keeps workers busy.
Rule of thumb: Start with 10. Monitor CPU utilization:
- CPU under 50%? Increase concurrency (you're waiting on I/O)
- CPU above 80%? Decrease concurrency (you're CPU-bound)
2. Poll Interval
How often an idle worker asks the orchestrator (over gRPC) for new work. This is a poll to the orchestrator, not to the database — the orchestrator is what touches PostgreSQL.
Fast (50-100ms):
- Pros: Workflows start quickly (low latency)
- Cons: More gRPC poll traffic to the orchestrator
- Use when: You have real-time requirements, low volume
Moderate (100-500ms):
- Pros: Balanced latency and orchestrator load
- Cons: None really, this is the sweet spot
- Use when: Most production workloads
Slow (1000ms+):
- Pros: Minimal poll traffic
- Cons: Workflows can wait up to 1 second to start
- Use when: You have very high volume, batch-oriented workflows
Impact: With 100 workers polling every 100ms, the orchestrator fields ~1,000 poll RPCs/sec. It's the orchestrator (and its database queries), not the workers, that absorbs this — another reason worker count is decoupled from database connections.
3. Execution timeouts (set per workflow/task — not a worker knob)
Runaway executions are bounded by timeouts you set on the workflow or task, which the orchestrator enforces. There is no worker-level "max duration" setting: pass an execution timeout when you start a workflow, or a task timeout in the task's options, and the engine fails the execution if it exceeds it — freeing the slot for other work.
Why it matters: sometimes an execution gets stuck (an unbounded loop, a wedged external call). A timeout lets the orchestrator fail it instead of holding a slot forever.
How to set it: take your longest normal duration and multiply by 3-5x for headroom. Example: a workflow that normally finishes in 5 minutes → a 20-minute execution timeout.
4. Heartbeat Interval
How often the worker reports "I'm still alive" to the orchestrator — a gRPC heartbeat, not a database write.
What it does: workers periodically send a heartbeat RPC. If a worker stops heartbeating (crash, network partition), the orchestrator notices and re-dispatches that worker's in-flight work to a healthy worker.
Why it matters: enables detection of silent failures (a worker dying mid-execution) so work isn't stranded in a slot that will never finish.
How to set it: the default of 30 seconds is good. Don't go below 10 seconds (needless RPC traffic) or above 60 seconds (slower failure detection). (In the Python SDK this interval isn't builder-configurable yet — a sensible default is used.)
Calculating the Right Worker Count
Here's a formula to determine how many workers you need:
1Required Workers = (Peak Concurrent Workflows) / (Concurrency per Worker)23Example:4- Peak load: 500 workflows executing simultaneously5- Concurrency per worker: 206- Required workers: 500 / 20 = 25 workersBut add headroom:
- For auto-scaling: Deploy 50% of required, let HPA scale to 200%
- For fixed capacity: Deploy 150% of required (for buffer)
Real-world sizing:
| Workload | Peak Concurrent | Concurrency | Workers | Reasoning |
|---|---|---|---|---|
| Low-volume SaaS | 50 | 10 | 5-10 | Mostly idle, low cost |
| E-commerce (moderate) | 200 | 20 | 10-20 | Handle peak hours |
| High-volume API | 1,000 | 20 | 50-100 | Consistent high load |
| Batch processing | 5,000 | 50 | 100-200 | Long-running, I/O-heavy |
Scaling the Orchestrator and Database
Everything below is the orchestrator tier — the only part of the system that touches PostgreSQL. This is where start-throughput and overall capacity are won or lost.
Scale it two ways:
- Vertically — a bigger PostgreSQL instance (more CPU/IOPS) and a larger orchestrator connection pool. The orchestrator's pool size is set via the
ORCHER_PG_MAX_CONNECTIONSenvironment variable (default 80 per node;ORCHER_PG_MIN_CONNECTIONSsets the warm minimum, default 20), not in worker code. - Horizontally — run multiple orchestrator nodes against the same database. Nodes coordinate through cluster membership and claim work atomically with PostgreSQL
SELECT ... FOR UPDATE SKIP LOCKED, so two nodes never hand out the same workflow. This is how you scale start-throughput and dispatch beyond a single orchestrator process, and it survives node failure without losing or duplicating a run — see High Availability for how to enable a cluster.
The database is the shared foundation for all of this, so optimizing it matters most. Let's tune it.
Database Performance Tuning
Essential Indexes (Orcher Creates These Automatically)
These indexes are critical for performance:
1-- For worker polling (finding pending workflows)2CREATE INDEX idx_workflow_queue_status3 ON workflow_queue(status, created_at);45-- For retry logic (finding workflows ready to retry)6CREATE INDEX idx_workflow_queue_retry7 ON workflow_queue(next_retry_at)8 WHERE status = 'pending';910-- For journal lookups (checking completed steps)11CREATE INDEX idx_workflow_execution_journal_lookup12 ON workflow_execution_journal(workflow_id, step_id);1314-- For analytics (finding workflows by name)15CREATE INDEX idx_workflow_queue_name16 ON workflow_queue(workflow_name, created_at);Verify they exist:
1SELECT2 schemaname,3 tablename,4 indexname,5 indexdef6FROM pg_indexes7WHERE tablename LIKE 'workflow%'8ORDER BY tablename, indexname;If any are missing, create them immediately. Performance degrades significantly without proper indexes.
PostgreSQL Configuration for Orcher Workloads
Orcher has a specific access pattern: many small writes (INSERTs), many small reads (SELECTs), occasional large reads (journal replay).
Recommended settings (adjust based on your database size):
1# postgresql.conf23# === Memory Settings ===4# General rule: 25% of RAM for shared_buffers5shared_buffers = 8GB # For 32GB RAM instance67# How much memory Postgres can use for sorting/joins8# Rule: 10-20% of RAM9effective_cache_size = 24GB # Tells Postgres how much cache the OS has1011# Per-query memory for sorts/joins12work_mem = 64MB # Increase if you see temp file warnings1314# Memory for maintenance operations (VACUUM, CREATE INDEX)15maintenance_work_mem = 2GB1617# === Connection Settings ===18# Only the orchestrator connects to PostgreSQL — workers never do.19# Size for: (orchestrator nodes x ORCHER_PG_MAX_CONNECTIONS) + headroom for admin/CLI/analytics.20# The per-node pool defaults to 80; lower ORCHER_PG_MAX_CONNECTIONS when running many nodes.21# e.g. 8 nodes x 20 = 160, so 200 leaves comfortable headroom.22max_connections = 200 # driven by orchestrator nodes x per-node pool, NOT worker count2324# === Write Performance ===25# How often to write WAL to disk (trades durability for performance)26# Keep default (1s) for durability, increase to 3s for more performance27wal_writer_delay = 1000ms2829# When to trigger a checkpoint (write to disk)30checkpoint_timeout = 15min31checkpoint_completion_target = 0.9 # Spread checkpoint I/O over 90% of interval3233# Maximum WAL size before forcing checkpoint34max_wal_size = 4GB3536# === Query Performance ===37# Cost of random page access (lower for SSD)38random_page_cost = 1.1 # Default is 4.0 for spinning disks3940# How many I/O operations can run in parallel41effective_io_concurrency = 200 # For SSDs (spinning disks: 2)4243# === Autovacuum (Critical for Write-Heavy Workloads) ===44# Orcher does lots of inserts, so vacuum must keep up45autovacuum = on46autovacuum_max_workers = 447autovacuum_naptime = 10s # Check for vacuum work every 10 seconds4849# Trigger vacuum more aggressively50autovacuum_vacuum_scale_factor = 0.05 # Default is 0.251autovacuum_analyze_scale_factor = 0.02 # Default is 0.15253# === Logging (For Debugging) ===54# Log slow queries55log_min_duration_statement = 1000 # Log queries taking > 1 second5657# Detailed log format58log_line_prefix = '%t [%p]: [%l-1] user=%u,db=%d,app=%a,client=%h '5960# Log checkpoints (helps tune checkpoint settings)61log_checkpoints = onHow to apply:
For managed databases (RDS, Cloud SQL):
- Go to parameter groups
- Modify the parameters above
- Apply changes (may require restart)
For self-managed:
- Edit
/var/lib/postgresql/data/postgresql.conf - Restart PostgreSQL:
sudo systemctl restart postgresql
Verify:
1-- Check current settings2SHOW shared_buffers;3SHOW effective_cache_size;4SHOW max_connections;56-- Check if settings require restart7SELECT name, setting, pending_restart8FROM pg_settings9WHERE pending_restart = true;Vacuum Strategy (Critical!)
Orcher generates many inserts. PostgreSQL needs VACUUM to reclaim space and update statistics.
The Problem: Without regular vacuuming, tables bloat (dead tuples accumulate), queries slow down, and disk fills up.
The Solution: Aggressive autovacuum configuration.
1-- Set per-table autovacuum parameters (more aggressive than global)2ALTER TABLE workflow_queue SET (3 autovacuum_vacuum_scale_factor = 0.02, -- Vacuum when 2% of rows are dead4 autovacuum_analyze_scale_factor = 0.01, -- Analyze when 1% of rows change5 autovacuum_vacuum_cost_delay = 2 -- Faster vacuuming6);78ALTER TABLE workflow_execution_journal SET (9 autovacuum_vacuum_scale_factor = 0.05, -- Journal is append-only, vacuum less often10 autovacuum_analyze_scale_factor = 0.02,11 autovacuum_vacuum_cost_delay = 512);Monitoring vacuum health:
1-- Check for table bloat2SELECT3 relname AS table_name,4 n_live_tup AS live_rows,5 n_dead_tup AS dead_rows,6 ROUND(100.0 * n_dead_tup / NULLIF(n_live_tup + n_dead_tup, 0), 2) AS dead_pct,7 last_autovacuum,8 last_autoanalyze9FROM pg_stat_user_tables10WHERE relname LIKE 'workflow%'11ORDER BY dead_pct DESC;If dead_pct exceeds 10%, your autovacuum isn't keeping up. Consider increasing autovacuum_max_workers or reducing autovacuum_vacuum_cost_delay.
Journal Retention and Cleanup
The workflow journal grows continuously. You need a retention strategy:
1-- Option 1: Delete completed workflow journals older than 30 days2DELETE FROM workflow_execution_journal3WHERE workflow_id IN (4 SELECT id FROM workflow_executions5 WHERE status = 'completed'6 AND completed_at < NOW() - INTERVAL '30 days'7);89-- Option 2: Archive to a separate table first (recommended)10INSERT INTO workflow_execution_journal_archive11SELECT * FROM workflow_execution_journal12WHERE workflow_id IN (13 SELECT id FROM workflow_executions14 WHERE status = 'completed'15 AND completed_at < NOW() - INTERVAL '30 days'16);1718-- Then delete the archived entries19DELETE FROM workflow_execution_journal20WHERE workflow_id IN (21 SELECT id FROM workflow_executions22 WHERE status = 'completed'23 AND completed_at < NOW() - INTERVAL '30 days'24);Recommended schedule: Run cleanup weekly via pg_cron or an external scheduler. Choose a retention period based on your debugging and compliance needs.
Connection Pooling
As you add orchestrator nodes (each with its own pool), the connections into PostgreSQL add up. PostgreSQL handles each connection as a separate process, and performance degrades above ~500 connections. A pooler in front of the database keeps this in check. (Workers are never part of this math — they hold no database connections.)
Use PgBouncer (connection pooler), between the orchestrator(s) and PostgreSQL:
1# pgbouncer.ini2[databases]3orcher = host=postgres port=5432 dbname=orcher45[pgbouncer]6listen_port = 64327listen_addr = 0.0.0.089# Pool mode: transaction (recommended for Orcher)10pool_mode = transaction1112# Pool sizing13default_pool_size = 25 # Connections per database per user14max_client_conn = 2000 # Max client connections to PgBouncer15max_db_connections = 100 # Max connections to PostgreSQL1617# Timeouts18server_idle_timeout = 60019client_idle_timeout = 0Why transaction mode: Orcher uses short transactions (claim workflow, write journal entry, release claim). Transaction mode returns the connection to the pool after each transaction, maximizing reuse.
Deployment: Run PgBouncer as a sidecar to each orchestrator node, or as a standalone service between the orchestrator and the database. Workers never connect to it — they reach the orchestrator over gRPC.
Next Steps
- Operating in Production -- Monitoring, debugging, DR
- Back to Overview