Getting Started

Scaling and Performance


Now that Orcher is deployed, let's understand how to scale it and optimize performance.

First, How the Pieces Fit

Before tuning anything, it's essential to know which component does what — because it determines where each bottleneck lives:

  • Clients call the orchestrator over gRPC to start workflows and send events.
  • The orchestrator is the only component that talks to PostgreSQL. It owns the queue, the journal, and every read/write, using its own database connection pool.
  • Workers are stateless gRPC clients. They connect to the orchestrator, pull workflow/task work, execute your code, and report results back — all over gRPC. A worker never opens a database connection.

That last point is the one people get wrong. When you "add workers," you are not adding database clients — you are adding execution capacity that talks to the orchestrator. Database connections are driven by the orchestrator, not by your worker count. Keep this in mind for every knob below.

Understanding Orcher's Scaling Model

Orcher scales in two dimensions, and it's important to understand the difference:

Dimension 1: Workflow Throughput (How Fast Can You Start Workflows?)

What it means: How many workflows per second can you enqueue?

Limited by: Orchestrator + PostgreSQL write capacity

Example: Your API receives 1,000 POST requests per second to create orders. Each request calls the orchestrator (start_workflow over gRPC), and the orchestrator enqueues it with one INSERT into workflow_queue. Can the orchestrator + database sustain 1,000 starts/second?

How to scale (all of this is the orchestrator tier — workers are not involved):

  1. Vertical scale the database: More CPU = more IOPS
  2. Scale the orchestrator horizontally: run multiple orchestrator nodes against the same database (see Scaling the Orchestrator and Database)
  3. Use connection pooling: put PgBouncer in front of PostgreSQL for the orchestrator's connections
  4. Add read replicas: offload analytics queries from the primary

Measure, don't guess. Start throughput depends on your database, its storage and the workflow's shape, so size from a measurement: the sdk-core benchmark suite runs standard scenarios against any engine and reports runs per second.

Dimension 2: Concurrent Workflow Execution (How Many Workflows Can Run Simultaneously?)

What it means: How many workflows are actively being executed right now?

Limited by: Worker pool size x concurrency per worker

Example: You have 10 worker pods, each configured to run 10 workflows concurrently. Your maximum concurrency is 10 x 10 = 100 workflows.

How to scale:

  1. Horizontal scale workers: Add more worker pods
  2. Vertical scale workers: Increase concurrency per worker (if tasks are I/O-bound)
  3. Optimize slow tasks: If a task takes 10 seconds, you can only process 6/minute per worker

Worked arithmetic (assuming an average task duration of 500ms):

  • 5 workers x 10 concurrency = 50 concurrent, ~6,000 workflows/minute
  • 10 workers x 10 concurrency = 100 concurrent, ~12,000 workflows/minute
  • 20 workers x 20 concurrency = 400 concurrent, ~48,000 workflows/minute

Real-world example:

  • Payment processing with 3-second average workflow duration
  • Need to handle 100 workflows/second sustained = 300 concurrent workflows
  • Deploy 15 workers x 20 concurrency = 300 concurrent capacity

The Key Insight: These dimensions are independent. You can start 10,000 workflows/second but only execute 100 concurrently (they queue up). Or you can execute 1,000 concurrently but only start 100/second (workers sit idle).

Worker Pool Configuration Deep Dive

Workers are where the work happens. Let's understand how to tune them.

The Worker Configuration Parameters

1
use orcher::prelude::*;
2
use std::time::Duration;
3
4
#[tokio::main]
5
async fn main() -> Result<()> {
6
// A worker is a stateless gRPC client of the orchestrator — no database here.
7
// Workflows and tasks are auto-registered from their decorators/macros.
8
WorkerBuilder::new()
9
.server_url("https://orchestrator.internal:50051") // gRPC endpoint of the orchestrator
10
.namespace("production")
11
.task_queue("default")
12
.max_concurrent_workflows(20) // workflows executing at once in this process
13
.max_concurrent_tasks(40) // tasks executing at once in this process
14
.poll_interval(Duration::from_millis(100)) // how often to pull work when idle
15
.workflow_poller_count(2) // parallel pollers pulling workflow work
16
.task_poller_count(4) // parallel pollers pulling task work
17
.heartbeat_interval(Duration::from_secs(30)) // liveness reported to the orchestrator
18
.build()
19
.await?
20
.run()
21
.await?;
22
23
Ok(())
24
}

Let's understand each parameter:

1. Concurrency — max_concurrent_workflows / max_concurrent_tasks (most important)

This controls how many workflows (and tasks) execute simultaneously within one worker process.

Low concurrency (1-5):

  • Use when: Your tasks are CPU-intensive
  • Example: Image processing, video encoding, complex calculations
  • Reasoning: CPU-bound tasks don't benefit from parallelism (they're already using the CPU). More concurrency just causes context switching overhead.

Medium concurrency (10-20):

  • Use when: Your tasks are mixed I/O and CPU
  • Example: API calls with some local processing
  • Reasoning: While waiting for API responses, other workflows can execute. But not too many or you'll thrash.

High concurrency (30-50):

  • Use when: Your tasks are pure I/O (network calls, database queries)
  • Example: Webhooks that just call external services
  • Reasoning: You spend 95% of time waiting for I/O. High concurrency keeps workers busy.

Rule of thumb: Start with 10. Monitor CPU utilization:

  • CPU under 50%? Increase concurrency (you're waiting on I/O)
  • CPU above 80%? Decrease concurrency (you're CPU-bound)

2. Poll Interval

How often an idle worker asks the orchestrator (over gRPC) for new work. This is a poll to the orchestrator, not to the database — the orchestrator is what touches PostgreSQL.

Fast (50-100ms):

  • Pros: Workflows start quickly (low latency)
  • Cons: More gRPC poll traffic to the orchestrator
  • Use when: You have real-time requirements, low volume

Moderate (100-500ms):

  • Pros: Balanced latency and orchestrator load
  • Cons: None really, this is the sweet spot
  • Use when: Most production workloads

Slow (1000ms+):

  • Pros: Minimal poll traffic
  • Cons: Workflows can wait up to 1 second to start
  • Use when: You have very high volume, batch-oriented workflows

Impact: With 100 workers polling every 100ms, the orchestrator fields ~1,000 poll RPCs/sec. It's the orchestrator (and its database queries), not the workers, that absorbs this — another reason worker count is decoupled from database connections.

3. Execution timeouts (set per workflow/task — not a worker knob)

Runaway executions are bounded by timeouts you set on the workflow or task, which the orchestrator enforces. There is no worker-level "max duration" setting: pass an execution timeout when you start a workflow, or a task timeout in the task's options, and the engine fails the execution if it exceeds it — freeing the slot for other work.

Why it matters: sometimes an execution gets stuck (an unbounded loop, a wedged external call). A timeout lets the orchestrator fail it instead of holding a slot forever.

How to set it: take your longest normal duration and multiply by 3-5x for headroom. Example: a workflow that normally finishes in 5 minutes → a 20-minute execution timeout.

4. Heartbeat Interval

How often the worker reports "I'm still alive" to the orchestrator — a gRPC heartbeat, not a database write.

What it does: workers periodically send a heartbeat RPC. If a worker stops heartbeating (crash, network partition), the orchestrator notices and re-dispatches that worker's in-flight work to a healthy worker.

Why it matters: enables detection of silent failures (a worker dying mid-execution) so work isn't stranded in a slot that will never finish.

How to set it: the default of 30 seconds is good. Don't go below 10 seconds (needless RPC traffic) or above 60 seconds (slower failure detection). (In the Python SDK this interval isn't builder-configurable yet — a sensible default is used.)

Calculating the Right Worker Count

Here's a formula to determine how many workers you need:

1
Required Workers = (Peak Concurrent Workflows) / (Concurrency per Worker)
2
3
Example:
4
- Peak load: 500 workflows executing simultaneously
5
- Concurrency per worker: 20
6
- Required workers: 500 / 20 = 25 workers

But add headroom:

  • For auto-scaling: Deploy 50% of required, let HPA scale to 200%
  • For fixed capacity: Deploy 150% of required (for buffer)

Real-world sizing:

WorkloadPeak ConcurrentConcurrencyWorkersReasoning
Low-volume SaaS50105-10Mostly idle, low cost
E-commerce (moderate)2002010-20Handle peak hours
High-volume API1,0002050-100Consistent high load
Batch processing5,00050100-200Long-running, I/O-heavy

Scaling the Orchestrator and Database

Everything below is the orchestrator tier — the only part of the system that touches PostgreSQL. This is where start-throughput and overall capacity are won or lost.

Scale it two ways:

  • Vertically — a bigger PostgreSQL instance (more CPU/IOPS) and a larger orchestrator connection pool. The orchestrator's pool size is set via the ORCHER_PG_MAX_CONNECTIONS environment variable (default 80 per node; ORCHER_PG_MIN_CONNECTIONS sets the warm minimum, default 20), not in worker code.
  • Horizontally — run multiple orchestrator nodes against the same database. Nodes coordinate through cluster membership and claim work atomically with PostgreSQL SELECT ... FOR UPDATE SKIP LOCKED, so two nodes never hand out the same workflow. This is how you scale start-throughput and dispatch beyond a single orchestrator process, and it survives node failure without losing or duplicating a run — see High Availability for how to enable a cluster.

The database is the shared foundation for all of this, so optimizing it matters most. Let's tune it.

Database Performance Tuning

Essential Indexes (Orcher Creates These Automatically)

These indexes are critical for performance:

1
-- For worker polling (finding pending workflows)
2
CREATE INDEX idx_workflow_queue_status
3
ON workflow_queue(status, created_at);
4
5
-- For retry logic (finding workflows ready to retry)
6
CREATE INDEX idx_workflow_queue_retry
7
ON workflow_queue(next_retry_at)
8
WHERE status = 'pending';
9
10
-- For journal lookups (checking completed steps)
11
CREATE INDEX idx_workflow_execution_journal_lookup
12
ON workflow_execution_journal(workflow_id, step_id);
13
14
-- For analytics (finding workflows by name)
15
CREATE INDEX idx_workflow_queue_name
16
ON workflow_queue(workflow_name, created_at);

Verify they exist:

1
SELECT
2
schemaname,
3
tablename,
4
indexname,
5
indexdef
6
FROM pg_indexes
7
WHERE tablename LIKE 'workflow%'
8
ORDER BY tablename, indexname;

If any are missing, create them immediately. Performance degrades significantly without proper indexes.

PostgreSQL Configuration for Orcher Workloads

Orcher has a specific access pattern: many small writes (INSERTs), many small reads (SELECTs), occasional large reads (journal replay).

Recommended settings (adjust based on your database size):

1
# postgresql.conf
2
3
# === Memory Settings ===
4
# General rule: 25% of RAM for shared_buffers
5
shared_buffers = 8GB # For 32GB RAM instance
6
7
# How much memory Postgres can use for sorting/joins
8
# Rule: 10-20% of RAM
9
effective_cache_size = 24GB # Tells Postgres how much cache the OS has
10
11
# Per-query memory for sorts/joins
12
work_mem = 64MB # Increase if you see temp file warnings
13
14
# Memory for maintenance operations (VACUUM, CREATE INDEX)
15
maintenance_work_mem = 2GB
16
17
# === Connection Settings ===
18
# Only the orchestrator connects to PostgreSQL — workers never do.
19
# Size for: (orchestrator nodes x ORCHER_PG_MAX_CONNECTIONS) + headroom for admin/CLI/analytics.
20
# The per-node pool defaults to 80; lower ORCHER_PG_MAX_CONNECTIONS when running many nodes.
21
# e.g. 8 nodes x 20 = 160, so 200 leaves comfortable headroom.
22
max_connections = 200 # driven by orchestrator nodes x per-node pool, NOT worker count
23
24
# === Write Performance ===
25
# How often to write WAL to disk (trades durability for performance)
26
# Keep default (1s) for durability, increase to 3s for more performance
27
wal_writer_delay = 1000ms
28
29
# When to trigger a checkpoint (write to disk)
30
checkpoint_timeout = 15min
31
checkpoint_completion_target = 0.9 # Spread checkpoint I/O over 90% of interval
32
33
# Maximum WAL size before forcing checkpoint
34
max_wal_size = 4GB
35
36
# === Query Performance ===
37
# Cost of random page access (lower for SSD)
38
random_page_cost = 1.1 # Default is 4.0 for spinning disks
39
40
# How many I/O operations can run in parallel
41
effective_io_concurrency = 200 # For SSDs (spinning disks: 2)
42
43
# === Autovacuum (Critical for Write-Heavy Workloads) ===
44
# Orcher does lots of inserts, so vacuum must keep up
45
autovacuum = on
46
autovacuum_max_workers = 4
47
autovacuum_naptime = 10s # Check for vacuum work every 10 seconds
48
49
# Trigger vacuum more aggressively
50
autovacuum_vacuum_scale_factor = 0.05 # Default is 0.2
51
autovacuum_analyze_scale_factor = 0.02 # Default is 0.1
52
53
# === Logging (For Debugging) ===
54
# Log slow queries
55
log_min_duration_statement = 1000 # Log queries taking > 1 second
56
57
# Detailed log format
58
log_line_prefix = '%t [%p]: [%l-1] user=%u,db=%d,app=%a,client=%h '
59
60
# Log checkpoints (helps tune checkpoint settings)
61
log_checkpoints = on

How to apply:

For managed databases (RDS, Cloud SQL):

  1. Go to parameter groups
  2. Modify the parameters above
  3. Apply changes (may require restart)

For self-managed:

  1. Edit /var/lib/postgresql/data/postgresql.conf
  2. Restart PostgreSQL: sudo systemctl restart postgresql

Verify:

1
-- Check current settings
2
SHOW shared_buffers;
3
SHOW effective_cache_size;
4
SHOW max_connections;
5
6
-- Check if settings require restart
7
SELECT name, setting, pending_restart
8
FROM pg_settings
9
WHERE pending_restart = true;

Vacuum Strategy (Critical!)

Orcher generates many inserts. PostgreSQL needs VACUUM to reclaim space and update statistics.

The Problem: Without regular vacuuming, tables bloat (dead tuples accumulate), queries slow down, and disk fills up.

The Solution: Aggressive autovacuum configuration.

1
-- Set per-table autovacuum parameters (more aggressive than global)
2
ALTER TABLE workflow_queue SET (
3
autovacuum_vacuum_scale_factor = 0.02, -- Vacuum when 2% of rows are dead
4
autovacuum_analyze_scale_factor = 0.01, -- Analyze when 1% of rows change
5
autovacuum_vacuum_cost_delay = 2 -- Faster vacuuming
6
);
7
8
ALTER TABLE workflow_execution_journal SET (
9
autovacuum_vacuum_scale_factor = 0.05, -- Journal is append-only, vacuum less often
10
autovacuum_analyze_scale_factor = 0.02,
11
autovacuum_vacuum_cost_delay = 5
12
);

Monitoring vacuum health:

1
-- Check for table bloat
2
SELECT
3
relname AS table_name,
4
n_live_tup AS live_rows,
5
n_dead_tup AS dead_rows,
6
ROUND(100.0 * n_dead_tup / NULLIF(n_live_tup + n_dead_tup, 0), 2) AS dead_pct,
7
last_autovacuum,
8
last_autoanalyze
9
FROM pg_stat_user_tables
10
WHERE relname LIKE 'workflow%'
11
ORDER BY dead_pct DESC;

If dead_pct exceeds 10%, your autovacuum isn't keeping up. Consider increasing autovacuum_max_workers or reducing autovacuum_vacuum_cost_delay.

Journal Retention and Cleanup

The workflow journal grows continuously. You need a retention strategy:

1
-- Option 1: Delete completed workflow journals older than 30 days
2
DELETE FROM workflow_execution_journal
3
WHERE workflow_id IN (
4
SELECT id FROM workflow_executions
5
WHERE status = 'completed'
6
AND completed_at < NOW() - INTERVAL '30 days'
7
);
8
9
-- Option 2: Archive to a separate table first (recommended)
10
INSERT INTO workflow_execution_journal_archive
11
SELECT * FROM workflow_execution_journal
12
WHERE workflow_id IN (
13
SELECT id FROM workflow_executions
14
WHERE status = 'completed'
15
AND completed_at < NOW() - INTERVAL '30 days'
16
);
17
18
-- Then delete the archived entries
19
DELETE FROM workflow_execution_journal
20
WHERE workflow_id IN (
21
SELECT id FROM workflow_executions
22
WHERE status = 'completed'
23
AND completed_at < NOW() - INTERVAL '30 days'
24
);

Recommended schedule: Run cleanup weekly via pg_cron or an external scheduler. Choose a retention period based on your debugging and compliance needs.

Connection Pooling

As you add orchestrator nodes (each with its own pool), the connections into PostgreSQL add up. PostgreSQL handles each connection as a separate process, and performance degrades above ~500 connections. A pooler in front of the database keeps this in check. (Workers are never part of this math — they hold no database connections.)

Use PgBouncer (connection pooler), between the orchestrator(s) and PostgreSQL:

1
# pgbouncer.ini
2
[databases]
3
orcher = host=postgres port=5432 dbname=orcher
4
5
[pgbouncer]
6
listen_port = 6432
7
listen_addr = 0.0.0.0
8
9
# Pool mode: transaction (recommended for Orcher)
10
pool_mode = transaction
11
12
# Pool sizing
13
default_pool_size = 25 # Connections per database per user
14
max_client_conn = 2000 # Max client connections to PgBouncer
15
max_db_connections = 100 # Max connections to PostgreSQL
16
17
# Timeouts
18
server_idle_timeout = 600
19
client_idle_timeout = 0

Why transaction mode: Orcher uses short transactions (claim workflow, write journal entry, release claim). Transaction mode returns the connection to the pool after each transaction, maximizing reuse.

Deployment: Run PgBouncer as a sidecar to each orchestrator node, or as a standalone service between the orchestrator and the database. Workers never connect to it — they reach the orchestrator over gRPC.


Next Steps