Getting Started

Understanding the System


Before you deploy anything, you need a mental model of how Orcher works at the infrastructure level.

The Two-Component Architecture

Orcher has exactly two components:

  1. Application Processes (stateless, horizontally scalable)
  2. PostgreSQL Database (stateful, your only persistence layer)

That's it. No queues, no schedulers, no caches, no coordinators. Let's understand each.

Component 1: Application Processes

Your application is a standard process (binary, container, whatever) that embeds the Orcher runtime. Think of Orcher as a library, not a separate service.

Anatomy of an Application Process:

Your application process has three layers:

Layer 1 — Business Logic (Your Code)

  • API handlers (e.g., POST /api/orders)
  • Workflow definitions
  • Task implementations
  • Database models

Layer 2 — Orcher Runtime (Embedded Library)

  • Workflow executor engine
  • Journal reader/writer
  • Queue poller (polls DB for work)
  • Retry coordinator

Layer 3 — Database Driver (SQLx, pg, etc.)

  • Connects to PostgreSQL for all state management

Critical Characteristics:

Stateless: The application process keeps zero persistent state. You can kill it at any time and start a new one. The new instance picks up exactly where the old one left off by reading from the database.

Why this matters: Stateless apps are the easiest thing to operate. They scale horizontally (just add instances). They're easy to update (rolling deployment). They're easy to debug (no local state to inspect).

Mode Agnostic: The same binary can run in "API mode" (serving HTTP requests) or "worker mode" (executing workflows). You just set an environment variable.

Why this matters: You build one Docker image, deploy it twice. API mode handles inbound traffic. Worker mode processes the queue. Same code, different configuration.

No Coordination: Application instances don't know about each other. They don't elect leaders, they don't gossip, they don't coordinate. Each instance independently polls the database for work.

Why this matters: No split-brain scenarios. No quorum requirements. No consensus protocols. If two instances try to grab the same work, PostgreSQL's FOR UPDATE SKIP LOCKED ensures only one gets it.

Fast Startup: No warm-up period, no cache priming, no leader election. An instance starts and is immediately productive.

Why this matters: Auto-scaling is fast. Deployments are fast. Recovery from crashes is fast.

Component 2: PostgreSQL Database

PostgreSQL is your entire state layer. Everything is here.

What's Stored:

1
-- 1. Workflow Queue (pending work)
2
CREATE TABLE workflow_queue (
3
id UUID PRIMARY KEY,
4
workflow_name TEXT NOT NULL, -- e.g., "process_payment"
5
input JSONB, -- workflow input data
6
status TEXT NOT NULL, -- 'pending', 'running', 'completed', 'failed'
7
retry_count INT DEFAULT 0,
8
next_retry_at TIMESTAMP, -- for exponential backoff
9
created_at TIMESTAMP DEFAULT NOW(),
10
updated_at TIMESTAMP DEFAULT NOW(),
11
completed_at TIMESTAMP
12
);
13
14
-- 2. Workflow Journal (execution history)
15
CREATE TABLE workflow_execution_journal (
16
workflow_id UUID NOT NULL,
17
step_id INT NOT NULL, -- sequential step number
18
task_name TEXT NOT NULL, -- e.g., "charge_stripe"
19
input JSONB, -- task input
20
output JSONB, -- task output (cached result)
21
status TEXT NOT NULL, -- 'completed', 'failed'
22
error TEXT, -- error message if failed
23
created_at TIMESTAMP DEFAULT NOW(),
24
PRIMARY KEY (workflow_id, step_id)
25
);
26
27
-- 3. Your Application Data
28
-- (whatever tables your app needs)
29
CREATE TABLE orders (...);
30
CREATE TABLE users (...);
31
-- etc.

Critical Characteristics:

Single Source of Truth: All state is here. If you want to know if a workflow completed, query this database. No need to check a queue, a cache, and a state store.

Why this matters: Debugging is simple. Backup is simple. Disaster recovery is simple. You have one system to understand.

ACID Guarantees: Workflow state changes are transactional. Either a step fully completes and is journaled, or it doesn't. No partial states.

Why this matters: You can't have a workflow that charged a credit card but didn't record it. Either both happen (transaction commits) or neither happens (transaction rolls back).

Standard PostgreSQL: This isn't some custom database. It's PostgreSQL. You already know how to operate it.

Why this matters: Use your existing tools (pg_dump, pgAdmin, AWS RDS). Use your existing expertise. Use your existing monitoring. Use your existing backup strategy.

The Queue Is Just a Table: There's no separate queue service. Workers poll SELECT ... FROM workflow_queue WHERE status = 'pending' ORDER BY created_at FOR UPDATE SKIP LOCKED LIMIT 1.

Why this matters: No queue-specific problems. No queue backpressure issues. No queue rebalancing. It's just SELECT queries, which PostgreSQL is very good at.

How Work Flows Through the System

Let's trace a complete workflow execution to see how the pieces interact.

Scenario: A user places an order on your e-commerce site. The order needs to charge payment, update inventory, and send confirmation. Here's the whole lifecycle at a glance before we walk each step in detail — notice that PostgreSQL is the only thing every actor talks to:

A Workflow's Lifecycle
API
enqueueINSERT into workflow_queue → returns 202 in ~15ms
Worker
claimpoll with FOR UPDATE SKIP LOCKED → status = running
Worker
execute + journaleach task result INSERTed into workflow_execution_journal
Worker
completestatus = completed
if crash
left runningno worker updates the row
Any Worker
reclaimstale running row (> 5 min) is picked up; journal replays finished steps
No scheduler, no broker — the queue and the journal are both just Postgres tables.

Step 1: API Request (Workflow Creation)

1
User Browser
2
↓
3
HTTP POST /api/orders
4
↓
5
Load Balancer
6
↓
7
API Pod (any instance)
8
↓
9
[Business logic validates the order]
10
↓
11
[Call: orcher.start_workflow("process_order", order_data)]
12
↓
13
PostgreSQL
14
INSERT INTO workflow_queue (
15
id, workflow_name, input, status, created_at
16
) VALUES (
17
'wf_abc123', 'process_order', '{"order_id": "..."}', 'pending', NOW()
18
);
19
↓
20
API Pod
21
↓
22
[Return HTTP 202 Accepted with workflow_id]
23
↓
24
User Browser (gets {"workflow_id": "wf_abc123"})

Timeline: This entire flow takes 10-20 milliseconds.

Key Points:

  • The API doesn't execute the workflow. It just enqueues it (inserts a row).
  • The response is immediate. The user doesn't wait for the workflow.
  • Any API instance can handle this. No routing or coordination needed.
  • If the API pod crashes after the INSERT, the workflow is safe in the database.

Step 2: Worker Polls for Work

1
Worker Pod (background loop)
2
↓
3
[Every 100ms, poll for work]
4
↓
5
PostgreSQL Query:
6
SELECT * FROM workflow_queue
7
WHERE status = 'pending'
8
AND (next_retry_at IS NULL OR next_retry_at <= NOW())
9
ORDER BY created_at ASC
10
FOR UPDATE SKIP LOCKED
11
LIMIT 1;
12
↓
13
PostgreSQL Returns: workflow_id 'wf_abc123'
14
↓
15
Worker Pod
16
↓
17
UPDATE workflow_queue
18
SET status = 'running', updated_at = NOW()
19
WHERE id = 'wf_abc123';

Timeline: The SELECT + UPDATE takes 2-5 milliseconds.

Key Points:

  • FOR UPDATE SKIP LOCKED is the magic. If another worker is checking the same row, it's skipped. No conflicts.
  • The worker "claims" the workflow by updating the status to 'running'.
  • If the worker crashes before updating, the workflow remains 'pending' and another worker will pick it up.
  • Polling every 100ms means workflows start within 100ms of being enqueued.

Step 3: Workflow Execution

1
Worker Pod
2
↓
3
[Load workflow definition for "process_order"]
4
↓
5
[Check journal for existing steps]
6
SELECT * FROM workflow_execution_journal WHERE workflow_id = 'wf_abc123';
7
(empty result - this is a new workflow)
8
↓
9
[Execute task 1: charge_payment]
10
↓
11
[Call Stripe API to charge card]
12
Stripe returns: {"charge_id": "ch_xyz", "status": "succeeded"}
13
↓
14
INSERT INTO workflow_execution_journal (
15
workflow_id, step_id, task_name, input, output, status, created_at
16
) VALUES (
17
'wf_abc123', 1, 'charge_payment', '{"amount": 5000}', '{"charge_id": "ch_xyz"}', 'completed', NOW()
18
);
19
↓
20
[Execute task 2: update_inventory]
21
↓
22
[Call inventory service]
23
Inventory service returns: {"reserved": true}
24
↓
25
INSERT INTO workflow_execution_journal (
26
workflow_id, step_id, task_name, input, output, status, created_at
27
) VALUES (
28
'wf_abc123', 2, 'update_inventory', '{"order_id": "..."}', '{"reserved": true}', 'completed', NOW()
29
);
30
↓
31
[Execute task 3: send_confirmation_email]
32
↓
33
[Call email service]
34
↓
35
INSERT INTO workflow_execution_journal (...);
36
↓
37
[All tasks complete]
38
↓
39
UPDATE workflow_queue
40
SET status = 'completed', completed_at = NOW()
41
WHERE id = 'wf_abc123';

Timeline: This varies based on the tasks. Let's say 500ms total (300ms for Stripe, 100ms for inventory, 100ms for email).

Key Points:

  • Each task execution is immediately journaled before moving to the next.
  • If the worker crashes after task 1, the journal has task 1's result. On retry, task 1 returns the cached result instantly.
  • The workflow code looks like normal sequential code, but it's durable.
  • The database commits each journal entry independently. ACID guarantees no partial states.

Step 4: Failure Scenario (What If the Worker Crashes?)

Let's say the worker crashes after completing task 1 (charge_payment) but before starting task 2.

1
Worker Pod
2
↓
3
[Completes task 1: charge_payment]
4
↓
5
INSERT INTO workflow_execution_journal (step_id = 1, status = 'completed');
6
↓
7
[Worker crashes - pod evicted, out of memory, network issue, etc.]
8
↓
9
[workflow_queue.status is still 'running']
10
[workflow_execution_journal has 1 entry for step 1]

What happens next?

After a timeout (configurable, default 5 minutes), the workflow is marked as stale:

1
Another Worker Pod (or the same one restarted)
2
↓
3
[Polls for work]
4
↓
5
SELECT * FROM workflow_queue
6
WHERE status = 'running'
7
AND updated_at < NOW() - INTERVAL '5 minutes';
8
(finds 'wf_abc123')
9
↓
10
UPDATE workflow_queue
11
SET status = 'pending', retry_count = retry_count + 1
12
WHERE id = 'wf_abc123';
13
↓
14
[Another poll loop picks it up]
15
↓
16
[Load workflow definition]
17
↓
18
[Check journal]
19
SELECT * FROM workflow_execution_journal WHERE workflow_id = 'wf_abc123';
20
(finds step 1: charge_payment, status = 'completed')
21
↓
22
[Replay step 1 from journal - return cached result]
23
(no Stripe API call, no duplicate charge!)
24
↓
25
[Execute step 2: update_inventory - first time]
26
↓
27
[Execute step 3: send_confirmation_email]
28
↓
29
[Mark completed]

Timeline: The retry happens 5 minutes after the crash. Execution resumes and completes in ~200ms (no re-execution of step 1).

Key Points:

  • The charge was not repeated (step 1's journaled result was reused).
  • No manual intervention required. The system self-healed.
  • The journal provides automatic idempotency.
  • From the user's perspective, the workflow just completed (with a 5-minute delay).

Why This Architecture Works for Operations

Simplicity Enables Reliability

Traditional orchestration platforms fail in complex ways because they have many failure modes. Orcher fails in simple ways:

  1. App crashes: Kubernetes restarts it. Workflows resume from journal.
  2. Database goes down: Apps wait. When DB comes back, everything resumes.
  3. Network partition: Apps can't reach DB, so they wait. When connectivity returns, everything resumes.

That's it. Three failure modes. Compare to traditional platforms with dozens.

Observability Is Built-In

Want to know what's happening? Query the database:

1
-- How many workflows are pending?
2
SELECT COUNT(*) FROM workflow_queue WHERE status = 'pending';
3
4
-- What's taking so long?
5
SELECT workflow_name, AVG(EXTRACT(EPOCH FROM (completed_at - created_at))) as avg_duration
6
FROM workflow_queue
7
WHERE status = 'completed'
8
AND completed_at > NOW() - INTERVAL '1 hour'
9
GROUP BY workflow_name;
10
11
-- Show me the failed workflows
12
SELECT id, workflow_name, retry_count, updated_at
13
FROM workflow_queue
14
WHERE status = 'failed'
15
ORDER BY updated_at DESC;

No custom UIs, no special tools. Just SQL.

Scaling Is Simple

Need more throughput? Add worker pods. Need to scale down? Remove worker pods. No rebalancing, no coordination, no complicated procedures.

1
# Scale up
2
kubectl scale deployment orcher-worker --replicas=20
3
4
# Scale down
5
kubectl scale deployment orcher-worker --replicas=5

Orcher's architecture makes operations straightforward by removing complexity. That's the fundamental insight.


Next Steps