Understanding the System
Before you deploy anything, you need a mental model of how Orcher works at the infrastructure level.
The Two-Component Architecture
Orcher has exactly two components:
- Application Processes (stateless, horizontally scalable)
- PostgreSQL Database (stateful, your only persistence layer)
That's it. No queues, no schedulers, no caches, no coordinators. Let's understand each.
Component 1: Application Processes
Your application is a standard process (binary, container, whatever) that embeds the Orcher runtime. Think of Orcher as a library, not a separate service.
Anatomy of an Application Process:
Your application process has three layers:
Layer 1 — Business Logic (Your Code)
- API handlers (e.g.,
POST /api/orders) - Workflow definitions
- Task implementations
- Database models
Layer 2 — Orcher Runtime (Embedded Library)
- Workflow executor engine
- Journal reader/writer
- Queue poller (polls DB for work)
- Retry coordinator
Layer 3 — Database Driver (SQLx, pg, etc.)
- Connects to PostgreSQL for all state management
Critical Characteristics:
Stateless: The application process keeps zero persistent state. You can kill it at any time and start a new one. The new instance picks up exactly where the old one left off by reading from the database.
Why this matters: Stateless apps are the easiest thing to operate. They scale horizontally (just add instances). They're easy to update (rolling deployment). They're easy to debug (no local state to inspect).
Mode Agnostic: The same binary can run in "API mode" (serving HTTP requests) or "worker mode" (executing workflows). You just set an environment variable.
Why this matters: You build one Docker image, deploy it twice. API mode handles inbound traffic. Worker mode processes the queue. Same code, different configuration.
No Coordination: Application instances don't know about each other. They don't elect leaders, they don't gossip, they don't coordinate. Each instance independently polls the database for work.
Why this matters: No split-brain scenarios. No quorum requirements. No consensus protocols. If two instances try to grab the same work, PostgreSQL's FOR UPDATE SKIP LOCKED ensures only one gets it.
Fast Startup: No warm-up period, no cache priming, no leader election. An instance starts and is immediately productive.
Why this matters: Auto-scaling is fast. Deployments are fast. Recovery from crashes is fast.
Component 2: PostgreSQL Database
PostgreSQL is your entire state layer. Everything is here.
What's Stored:
1-- 1. Workflow Queue (pending work)2CREATE TABLE workflow_queue (3 id UUID PRIMARY KEY,4 workflow_name TEXT NOT NULL, -- e.g., "process_payment"5 input JSONB, -- workflow input data6 status TEXT NOT NULL, -- 'pending', 'running', 'completed', 'failed'7 retry_count INT DEFAULT 0,8 next_retry_at TIMESTAMP, -- for exponential backoff9 created_at TIMESTAMP DEFAULT NOW(),10 updated_at TIMESTAMP DEFAULT NOW(),11 completed_at TIMESTAMP12);1314-- 2. Workflow Journal (execution history)15CREATE TABLE workflow_execution_journal (16 workflow_id UUID NOT NULL,17 step_id INT NOT NULL, -- sequential step number18 task_name TEXT NOT NULL, -- e.g., "charge_stripe"19 input JSONB, -- task input20 output JSONB, -- task output (cached result)21 status TEXT NOT NULL, -- 'completed', 'failed'22 error TEXT, -- error message if failed23 created_at TIMESTAMP DEFAULT NOW(),24 PRIMARY KEY (workflow_id, step_id)25);2627-- 3. Your Application Data28-- (whatever tables your app needs)29CREATE TABLE orders (...);30CREATE TABLE users (...);31-- etc.Critical Characteristics:
Single Source of Truth: All state is here. If you want to know if a workflow completed, query this database. No need to check a queue, a cache, and a state store.
Why this matters: Debugging is simple. Backup is simple. Disaster recovery is simple. You have one system to understand.
ACID Guarantees: Workflow state changes are transactional. Either a step fully completes and is journaled, or it doesn't. No partial states.
Why this matters: You can't have a workflow that charged a credit card but didn't record it. Either both happen (transaction commits) or neither happens (transaction rolls back).
Standard PostgreSQL: This isn't some custom database. It's PostgreSQL. You already know how to operate it.
Why this matters: Use your existing tools (pg_dump, pgAdmin, AWS RDS). Use your existing expertise. Use your existing monitoring. Use your existing backup strategy.
The Queue Is Just a Table: There's no separate queue service. Workers poll SELECT ... FROM workflow_queue WHERE status = 'pending' ORDER BY created_at FOR UPDATE SKIP LOCKED LIMIT 1.
Why this matters: No queue-specific problems. No queue backpressure issues. No queue rebalancing. It's just SELECT queries, which PostgreSQL is very good at.
How Work Flows Through the System
Let's trace a complete workflow execution to see how the pieces interact.
Scenario: A user places an order on your e-commerce site. The order needs to charge payment, update inventory, and send confirmation. Here's the whole lifecycle at a glance before we walk each step in detail — notice that PostgreSQL is the only thing every actor talks to:
Step 1: API Request (Workflow Creation)
1User Browser2 ↓3 HTTP POST /api/orders4 ↓5Load Balancer6 ↓7API Pod (any instance)8 ↓9 [Business logic validates the order]10 ↓11 [Call: orcher.start_workflow("process_order", order_data)]12 ↓13PostgreSQL14 INSERT INTO workflow_queue (15 id, workflow_name, input, status, created_at16 ) VALUES (17 'wf_abc123', 'process_order', '{"order_id": "..."}', 'pending', NOW()18 );19 ↓20API Pod21 ↓22 [Return HTTP 202 Accepted with workflow_id]23 ↓24User Browser (gets {"workflow_id": "wf_abc123"})Timeline: This entire flow takes 10-20 milliseconds.
Key Points:
- The API doesn't execute the workflow. It just enqueues it (inserts a row).
- The response is immediate. The user doesn't wait for the workflow.
- Any API instance can handle this. No routing or coordination needed.
- If the API pod crashes after the INSERT, the workflow is safe in the database.
Step 2: Worker Polls for Work
1Worker Pod (background loop)2 ↓3 [Every 100ms, poll for work]4 ↓5PostgreSQL Query:6 SELECT * FROM workflow_queue7 WHERE status = 'pending'8 AND (next_retry_at IS NULL OR next_retry_at <= NOW())9 ORDER BY created_at ASC10 FOR UPDATE SKIP LOCKED11 LIMIT 1;12 ↓13PostgreSQL Returns: workflow_id 'wf_abc123'14 ↓15Worker Pod16 ↓17 UPDATE workflow_queue18 SET status = 'running', updated_at = NOW()19 WHERE id = 'wf_abc123';Timeline: The SELECT + UPDATE takes 2-5 milliseconds.
Key Points:
FOR UPDATE SKIP LOCKEDis the magic. If another worker is checking the same row, it's skipped. No conflicts.- The worker "claims" the workflow by updating the status to 'running'.
- If the worker crashes before updating, the workflow remains 'pending' and another worker will pick it up.
- Polling every 100ms means workflows start within 100ms of being enqueued.
Step 3: Workflow Execution
1Worker Pod2 ↓3 [Load workflow definition for "process_order"]4 ↓5 [Check journal for existing steps]6 SELECT * FROM workflow_execution_journal WHERE workflow_id = 'wf_abc123';7 (empty result - this is a new workflow)8 ↓9 [Execute task 1: charge_payment]10 ↓11 [Call Stripe API to charge card]12 Stripe returns: {"charge_id": "ch_xyz", "status": "succeeded"}13 ↓14 INSERT INTO workflow_execution_journal (15 workflow_id, step_id, task_name, input, output, status, created_at16 ) VALUES (17 'wf_abc123', 1, 'charge_payment', '{"amount": 5000}', '{"charge_id": "ch_xyz"}', 'completed', NOW()18 );19 ↓20 [Execute task 2: update_inventory]21 ↓22 [Call inventory service]23 Inventory service returns: {"reserved": true}24 ↓25 INSERT INTO workflow_execution_journal (26 workflow_id, step_id, task_name, input, output, status, created_at27 ) VALUES (28 'wf_abc123', 2, 'update_inventory', '{"order_id": "..."}', '{"reserved": true}', 'completed', NOW()29 );30 ↓31 [Execute task 3: send_confirmation_email]32 ↓33 [Call email service]34 ↓35 INSERT INTO workflow_execution_journal (...);36 ↓37 [All tasks complete]38 ↓39 UPDATE workflow_queue40 SET status = 'completed', completed_at = NOW()41 WHERE id = 'wf_abc123';Timeline: This varies based on the tasks. Let's say 500ms total (300ms for Stripe, 100ms for inventory, 100ms for email).
Key Points:
- Each task execution is immediately journaled before moving to the next.
- If the worker crashes after task 1, the journal has task 1's result. On retry, task 1 returns the cached result instantly.
- The workflow code looks like normal sequential code, but it's durable.
- The database commits each journal entry independently. ACID guarantees no partial states.
Step 4: Failure Scenario (What If the Worker Crashes?)
Let's say the worker crashes after completing task 1 (charge_payment) but before starting task 2.
1Worker Pod2 ↓3 [Completes task 1: charge_payment]4 ↓5 INSERT INTO workflow_execution_journal (step_id = 1, status = 'completed');6 ↓7 [Worker crashes - pod evicted, out of memory, network issue, etc.]8 ↓9 [workflow_queue.status is still 'running']10 [workflow_execution_journal has 1 entry for step 1]What happens next?
After a timeout (configurable, default 5 minutes), the workflow is marked as stale:
1Another Worker Pod (or the same one restarted)2 ↓3 [Polls for work]4 ↓5 SELECT * FROM workflow_queue6 WHERE status = 'running'7 AND updated_at < NOW() - INTERVAL '5 minutes';8 (finds 'wf_abc123')9 ↓10 UPDATE workflow_queue11 SET status = 'pending', retry_count = retry_count + 112 WHERE id = 'wf_abc123';13 ↓14 [Another poll loop picks it up]15 ↓16 [Load workflow definition]17 ↓18 [Check journal]19 SELECT * FROM workflow_execution_journal WHERE workflow_id = 'wf_abc123';20 (finds step 1: charge_payment, status = 'completed')21 ↓22 [Replay step 1 from journal - return cached result]23 (no Stripe API call, no duplicate charge!)24 ↓25 [Execute step 2: update_inventory - first time]26 ↓27 [Execute step 3: send_confirmation_email]28 ↓29 [Mark completed]Timeline: The retry happens 5 minutes after the crash. Execution resumes and completes in ~200ms (no re-execution of step 1).
Key Points:
- The charge was not repeated (step 1's journaled result was reused).
- No manual intervention required. The system self-healed.
- The journal provides automatic idempotency.
- From the user's perspective, the workflow just completed (with a 5-minute delay).
Why This Architecture Works for Operations
Simplicity Enables Reliability
Traditional orchestration platforms fail in complex ways because they have many failure modes. Orcher fails in simple ways:
- App crashes: Kubernetes restarts it. Workflows resume from journal.
- Database goes down: Apps wait. When DB comes back, everything resumes.
- Network partition: Apps can't reach DB, so they wait. When connectivity returns, everything resumes.
That's it. Three failure modes. Compare to traditional platforms with dozens.
Observability Is Built-In
Want to know what's happening? Query the database:
1-- How many workflows are pending?2SELECT COUNT(*) FROM workflow_queue WHERE status = 'pending';34-- What's taking so long?5SELECT workflow_name, AVG(EXTRACT(EPOCH FROM (completed_at - created_at))) as avg_duration6FROM workflow_queue7WHERE status = 'completed'8 AND completed_at > NOW() - INTERVAL '1 hour'9GROUP BY workflow_name;1011-- Show me the failed workflows12SELECT id, workflow_name, retry_count, updated_at13FROM workflow_queue14WHERE status = 'failed'15ORDER BY updated_at DESC;No custom UIs, no special tools. Just SQL.
Scaling Is Simple
Need more throughput? Add worker pods. Need to scale down? Remove worker pods. No rebalancing, no coordination, no complicated procedures.
1# Scale up2kubectl scale deployment orcher-worker --replicas=2034# Scale down5kubectl scale deployment orcher-worker --replicas=5Orcher's architecture makes operations straightforward by removing complexity. That's the fundamental insight.
Next Steps
- Deploying to Production — Kubernetes, database, HA
- Back to Overview