Operations

Orchestrator Server


The Orchestrator Server is the central coordination component of Orcher. It is one process that serves a gRPC API (port 50051) for SDKs, workers and the CLI, and a REST API under /api/v1 plus health checks and metrics (port 8080). It does not execute workflow or task code — it only coordinates execution; workers pull work from it over gRPC.

  • Manages Workflow State — Tracks execution status, journaling, and checkpoints
  • Coordinates Work Distribution — Hands tasks and workflows to workers via queues and the outbox pattern
  • Handles Durable Timers — Manages sleep, timeouts, and scheduled execution
  • Ensures Durability — Persists all state to PostgreSQL for crash recovery
  • Shards Work — Uses consistent hashing with PostgreSQL advisory locks to distribute work across multiple instances

Architecture

The orchestrator (binary orchestrator, image ghcr.io/orcher-io/orcher) is a single process with two listeners:

  • gRPC on GRPC_BIND_ADDR / --grpc-addr (default 127.0.0.1:50051) — used by SDK clients, workers (which long-poll for work) and the CLI.
  • HTTP on HTTP_BIND_ADDR / --http-addr (default 127.0.0.1:8080) — serves the REST API under /api/v1, plus /health, /health/ready, /health/live and /metrics. The dashboard and any HTTP client talk to this listener directly.

Both listeners share one authentication layer, configured in the [security] section of the orchestrator's TOML config. PostgreSQL is the only data store the orchestrator needs.

Clients
SDK clients · CLI
Dashboard · HTTP clients
gRPC :50051
►
REST :8080
►
Orchestrator
:50051 gRPC · :8080 HTTP
Workflow state
Task dispatch
REST API · health · metrics
Background Processors
Outbox
Timers
Schedules
Sharding
Polling
Cache
PostgreSQL
State · Journal · Outbox · Timers
SDK Workers (your application)
Worker 1
Worker 2
Worker N

gRPC :50051

Internal Components

The orchestrator runs several background processors:

  • OutboxProcessor — Polls the outbox_queue table and dispatches tasks atomically. The outbox pattern guarantees at-least-once delivery: workflow state changes and task inserts happen in the same database transaction.
  • TimerProcessor — Fires durable timers from the timer_queue table when they expire.
  • ScheduleProcessor — Handles cron-style scheduled workflow triggers.
  • ShardCoordinator — Acquires and renews shard leases using PostgreSQL advisory locks. Each orchestrator node owns a subset of shards for parallel processing.
  • PollingManager — Manages long-polling connections from workers, with notification-based wake-up when new work arrives.
  • HighPerfCache — LRU in-memory cache for hot workflow state, reducing database roundtrips.

Task Lifecycle

Tasks flow through the system as follows:

  1. Workflow state change + outbox task insert happen in a single database transaction
  2. OutboxProcessor claims the outbox entry (lease-based, with expiry for crash safety)
  3. Task is dispatched to workers via the polling manager or eager dispatch
  4. Worker completes the task and reports back via gRPC
  5. On failure, tasks retry with exponential backoff, or move to the dead letter queue after max retries
1
State change + outbox insert
Single DB transaction
2
OutboxProcessor claims entry
Lease-based with expiry
3
Task dispatched to worker
Polling or eager dispatch
4
Worker executes & reports
gRPC completion
5
Result journaled
Retry on failure · DLQ after max

Installation

From Source

1
git clone <your-repo-url>
2
cd orcher
3
cargo build --release
4
5
# Orchestrator binary (serves gRPC and the REST API)
6
DATABASE_URL=postgres://... ./target/release/orchestrator

Local Development

The quickest way to run the stack locally is the CLI:

1
orcher dev start

This runs the orchestrator and PostgreSQL in Docker with no authentication, with its ports published on 127.0.0.1.


Next Steps

  • Configuration — Environment variables and database connection
  • Deployment — Starting the server in development, Docker, and systemd
  • High Availability — Multi-instance deployment and load balancing
  • Monitoring — Health checks, performance tuning, and troubleshooting