Getting Started

Production Excellence


Now that you understand the core concepts and patterns, let's explore how to run Orcher workflows in production with confidence.

Monitoring and Observability

Understanding Workflow State

Orcher provides rich visibility into workflow execution through the workflow journal and metadata tables. Every workflow moves through a small set of states — and monitoring is mostly about watching for the two that mean trouble.

Workflow States — and What to Watch
•
queuedwaiting for a worker
normal
•
runningexecuting tasks
normal — alert if stuck for too long
•
suspendedawaiting a timer or event — zero resources
normal
✓
completedall steps finished
terminal ✓
✗
failedretries exhausted
terminal — alert
Alert on failed, and on running that never terminates — the rest are healthy by design.

Query Workflow Status:

1
-- Get workflow overview
2
SELECT
3
id,
4
workflow_name,
5
status,
6
created_at,
7
completed_at,
8
attempt,
9
error
10
FROM workflows
11
WHERE status IN ('running', 'failed')
12
ORDER BY created_at DESC
13
LIMIT 100;

Monitor Task Performance:

1
-- Find slow tasks across all workflows
2
SELECT
3
task_name,
4
COUNT(*) as execution_count,
5
AVG(EXTRACT(EPOCH FROM (completed_at - created_at))) as avg_duration_seconds,
6
MAX(EXTRACT(EPOCH FROM (completed_at - created_at))) as max_duration_seconds,
7
PERCENTILE_CONT(0.95) WITHIN GROUP (ORDER BY EXTRACT(EPOCH FROM (completed_at - created_at))) as p95_duration_seconds
8
FROM workflow_execution_journal
9
WHERE status = 'completed'
10
AND completed_at > NOW() - INTERVAL '24 hours'
11
GROUP BY task_name
12
ORDER BY avg_duration_seconds DESC;

Track Failure Rates:

1
-- Calculate task failure rates
2
SELECT
3
task_name,
4
COUNT(*) FILTER (WHERE status = 'completed') as success_count,
5
COUNT(*) FILTER (WHERE status = 'failed') as failure_count,
6
ROUND(
7
100.0 * COUNT(*) FILTER (WHERE status = 'failed') / COUNT(*),
8
2
9
) as failure_rate_percent
10
FROM workflow_execution_journal
11
WHERE created_at > NOW() - INTERVAL '7 days'
12
GROUP BY task_name
13
HAVING COUNT(*) > 10
14
ORDER BY failure_rate_percent DESC;

Structured Logging

Add contextual logging to your workflows and tasks:

1
use tracing::{info, warn, error, instrument};
2
3
#[workflow(name = "process_payment")]
4
#[instrument(skip(ctx), fields(workflow_id = %ctx.workflow_id()))]
5
pub async fn process_payment(
6
ctx: WorkflowContext,
7
payment: PaymentInput,
8
) -> Result<PaymentResult> {
9
info!(
10
user_id = %payment.user_id,
11
amount = payment.amount,
12
"Starting payment processing"
13
);
14
15
// Log decision points
16
let fraud_score = ctx.execute_task(check_fraud, payment.user_id.clone()).await?;
17
18
if fraud_score > 0.8 {
19
warn!(
20
user_id = %payment.user_id,
21
fraud_score = fraud_score,
22
"Payment flagged as fraudulent"
23
);
24
return Err(anyhow::anyhow!("Payment flagged"));
25
}
26
27
info!(fraud_score = fraud_score, "Fraud check passed");
28
29
// Execute payment
30
let charge = ctx.execute_task(charge_stripe, payment.clone()).await?;
31
32
info!(
33
charge_id = %charge.id,
34
amount = payment.amount,
35
"Payment charged successfully"
36
);
37
38
Ok(PaymentResult {
39
charge_id: charge.id,
40
status: "completed".to_string(),
41
receipt_url: charge.receipt_url,
42
})
43
}
44
45
#[task]
46
#[instrument]
47
async fn charge_stripe(ctx: TaskContext, payment: PaymentInput) -> Result<StripeCharge> {
48
info!("Calling Stripe API");
49
50
let start = std::time::Instant::now();
51
let charge = make_stripe_call(payment).await?;
52
let duration = start.elapsed();
53
54
info!(
55
duration_ms = duration.as_millis(),
56
charge_id = %charge.id,
57
"Stripe API call completed"
58
);
59
60
Ok(charge)
61
}

Logging Best Practices:

✓ Always include workflow_id for correlation ✓ Log at decision points (if/else branches) ✓ Log external API calls with duration ✓ Log business-significant events ✗ Don't log sensitive data (credit cards, passwords) ✗ Don't log inside tight loops

Metrics and Alerting

Key Metrics to Track:

  1. Workflow Completion Rate

    1
    SELECT
    2
    workflow_name,
    3
    COUNT(*) FILTER (WHERE status = 'completed') * 100.0 / COUNT(*) as completion_rate
    4
    FROM workflows
    5
    WHERE created_at > NOW() - INTERVAL '1 hour'
    6
    GROUP BY workflow_name;
  2. Workflow Duration (P50, P95, P99)

    1
    SELECT
    2
    workflow_name,
    3
    PERCENTILE_CONT(0.5) WITHIN GROUP (ORDER BY EXTRACT(EPOCH FROM (completed_at - created_at))) as p50_seconds,
    4
    PERCENTILE_CONT(0.95) WITHIN GROUP (ORDER BY EXTRACT(EPOCH FROM (completed_at - created_at))) as p95_seconds,
    5
    PERCENTILE_CONT(0.99) WITHIN GROUP (ORDER BY EXTRACT(EPOCH FROM (completed_at - created_at))) as p99_seconds
    6
    FROM workflows
    7
    WHERE status = 'completed'
    8
    AND created_at > NOW() - INTERVAL '1 hour'
    9
    GROUP BY workflow_name;
  3. Queue Depth (Pending Workflows)

    1
    SELECT COUNT(*) as pending_workflows
    2
    FROM workflows
    3
    WHERE status = 'pending';
  4. Retry Rate

    1
    SELECT
    2
    workflow_name,
    3
    AVG(attempt) as avg_attempts,
    4
    MAX(attempt) as max_attempts
    5
    FROM workflows
    6
    WHERE created_at > NOW() - INTERVAL '1 hour'
    7
    GROUP BY workflow_name;

Alerting Rules:

  • Critical: Workflow failure rate > 5% over 5 minutes
  • Warning: P95 duration > 2x baseline
  • Warning: Queue depth > 1000 workflows
  • Critical: Any workflow stuck for > 1 hour

Example Prometheus Metrics Export:

1
use prometheus::{register_histogram_vec, register_counter_vec};
2
3
// In your application initialization
4
let workflow_duration = register_histogram_vec!(
5
"orcher_workflow_duration_seconds",
6
"Workflow execution duration",
7
&["workflow_name", "status"],
8
vec![0.1, 0.5, 1.0, 5.0, 10.0, 30.0, 60.0]
9
).unwrap();
10
11
let workflow_executions = register_counter_vec!(
12
"orcher_workflow_executions_total",
13
"Total workflow executions",
14
&["workflow_name", "status"]
15
).unwrap();
16
17
// In your workflow completion handler
18
workflow_duration
19
.with_label_values(&[workflow_name, "completed"])
20
.observe(duration_seconds);
21
22
workflow_executions
23
.with_label_values(&[workflow_name, "completed"])
24
.inc();

Testing Strategies

Unit Testing Tasks

Tasks are just async functions, so they're straightforward to test:

1
#[cfg(test)]
2
mod tests {
3
use super::*;
4
5
#[tokio::test]
6
async fn test_charge_stripe_success() {
7
// Arrange
8
let payment = PaymentInput {
9
user_id: "user_123".to_string(),
10
amount: 5000,
11
currency: "usd".to_string(),
12
payment_method: "pm_test_card".to_string(),
13
};
14
15
// Act
16
let result = charge_stripe(payment).await;
17
18
// Assert
19
assert!(result.is_ok());
20
let charge = result.unwrap();
21
assert_eq!(charge.amount, 5000);
22
assert!(!charge.id.is_empty());
23
}
24
25
#[tokio::test]
26
async fn test_charge_stripe_invalid_card() {
27
let payment = PaymentInput {
28
user_id: "user_123".to_string(),
29
amount: 5000,
30
currency: "usd".to_string(),
31
payment_method: "pm_invalid".to_string(),
32
};
33
34
let result = charge_stripe(payment).await;
35
36
assert!(result.is_err());
37
assert!(result.unwrap_err().to_string().contains("card"));
38
}
39
40
#[tokio::test]
41
async fn test_task_idempotency() {
42
// Test that running task twice produces same result
43
let user_id = "user_123";
44
let amount = 100;
45
46
// First execution
47
let result1 = update_user_balance(user_id.to_string(), amount).await.unwrap();
48
49
// Get balance
50
let balance1 = get_user_balance(user_id).await.unwrap();
51
52
// Second execution (should be idempotent)
53
let result2 = update_user_balance(user_id.to_string(), amount).await.unwrap();
54
55
// Balance should be the same
56
let balance2 = get_user_balance(user_id).await.unwrap();
57
assert_eq!(balance1, balance2);
58
}
59
}

Integration Testing Workflows

Test workflows with a real Orcher instance and database:

1
#[cfg(test)]
2
mod integration_tests {
3
use super::*;
4
use orcher::test_utils::TestOrcher;
5
6
#[tokio::test]
7
async fn test_payment_workflow_success() {
8
// Setup test Orcher instance with in-memory DB
9
let orcher = TestOrcher::new().await;
10
11
// Register workflow
12
orcher.register_workflow(process_payment).await;
13
14
// Start workflow
15
let payment = PaymentInput {
16
user_id: "user_123".to_string(),
17
amount: 5000,
18
currency: "usd".to_string(),
19
payment_method: "pm_test_card".to_string(),
20
};
21
22
let workflow_id = orcher
23
.start_workflow("process_payment", payment)
24
.await
25
.unwrap();
26
27
// Wait for completion (with timeout)
28
let result = orcher
29
.wait_for_workflow::<PaymentResult>(workflow_id, Duration::from_secs(30))
30
.await
31
.unwrap();
32
33
// Assert result
34
assert_eq!(result.status, "completed");
35
assert!(!result.charge_id.is_empty());
36
37
// Verify journal entries
38
let journal = orcher.get_workflow_journal(workflow_id).await.unwrap();
39
assert_eq!(journal.len(), 4); // validate, fraud_check, charge, save
40
assert!(journal.iter().all(|e| e.status == "completed"));
41
}
42
43
#[tokio::test]
44
async fn test_payment_workflow_fraud_rejection() {
45
let orcher = TestOrcher::new().await;
46
orcher.register_workflow(process_payment).await;
47
48
// Use high-risk user ID that triggers fraud detection
49
let payment = PaymentInput {
50
user_id: "high_risk_user".to_string(),
51
amount: 5000,
52
currency: "usd".to_string(),
53
payment_method: "pm_test_card".to_string(),
54
};
55
56
let workflow_id = orcher
57
.start_workflow("process_payment", payment)
58
.await
59
.unwrap();
60
61
// Wait for failure
62
let result = orcher
63
.wait_for_workflow::<PaymentResult>(workflow_id, Duration::from_secs(30))
64
.await;
65
66
assert!(result.is_err());
67
68
// Verify workflow failed at fraud check
69
let workflow = orcher.get_workflow(workflow_id).await.unwrap();
70
assert_eq!(workflow.status, "failed");
71
assert!(workflow.error.unwrap().contains("fraud"));
72
}
73
74
#[tokio::test]
75
async fn test_workflow_retry_on_transient_failure() {
76
let orcher = TestOrcher::new().await;
77
orcher.register_workflow(process_payment).await;
78
79
// Inject transient failure in stripe task
80
orcher.inject_task_failure("charge_stripe", 2); // Fail first 2 attempts
81
82
let payment = PaymentInput {
83
user_id: "user_123".to_string(),
84
amount: 5000,
85
currency: "usd".to_string(),
86
payment_method: "pm_test_card".to_string(),
87
};
88
89
let workflow_id = orcher
90
.start_workflow("process_payment", payment)
91
.await
92
.unwrap();
93
94
// Should eventually succeed after retries
95
let result = orcher
96
.wait_for_workflow::<PaymentResult>(workflow_id, Duration::from_secs(60))
97
.await
98
.unwrap();
99
100
assert_eq!(result.status, "completed");
101
102
// Verify retry attempts
103
let workflow = orcher.get_workflow(workflow_id).await.unwrap();
104
assert!(workflow.attempt >= 3);
105
}
106
}

End-to-End Testing

Test complete flows with real external services (use sandbox/test mode):

1
#[tokio::test]
2
#[ignore] // Run with --ignored flag for E2E tests
3
async fn test_payment_flow_e2e() {
4
// Use real Stripe test mode API
5
std::env::set_var("STRIPE_SECRET_KEY", "sk_test_...");
6
7
let orcher = Orcher::new(get_test_db_url()).await;
8
orcher.register_workflow(process_payment).await;
9
10
// Real test card from Stripe
11
let payment = PaymentInput {
12
user_id: "user_123".to_string(),
13
amount: 100, // $1.00
14
currency: "usd".to_string(),
15
payment_method: "pm_card_visa".to_string(), // Stripe test card
16
};
17
18
let workflow_id = orcher
19
.start_workflow("process_payment", payment)
20
.await
21
.unwrap();
22
23
let result = orcher
24
.wait_for_workflow::<PaymentResult>(workflow_id, Duration::from_secs(60))
25
.await
26
.unwrap();
27
28
assert_eq!(result.status, "completed");
29
30
// Verify in real Stripe dashboard
31
let stripe = stripe::Client::new(std::env::var("STRIPE_SECRET_KEY").unwrap());
32
let charge = stripe.charges.retrieve(&result.charge_id).await.unwrap();
33
assert_eq!(charge.amount, 100);
34
assert_eq!(charge.status, "succeeded");
35
}

Performance Optimization

Database Tuning

Index Optimization:

1
-- Essential indexes for Orcher tables
2
CREATE INDEX CONCURRENTLY idx_workflows_status_created
3
ON workflows(status, created_at)
4
WHERE status IN ('pending', 'running');
5
6
CREATE INDEX CONCURRENTLY idx_workflow_execution_journal_lookup
7
ON workflow_execution_journal(workflow_id, step_id);
8
9
CREATE INDEX CONCURRENTLY idx_workflow_execution_journal_task_perf
10
ON workflow_execution_journal(task_name, status, created_at, completed_at)
11
WHERE completed_at IS NOT NULL;
12
13
-- Index for queue polling (critical for performance)
14
CREATE INDEX CONCURRENTLY idx_workflows_queue_poll
15
ON workflows(status, created_at)
16
WHERE status = 'pending';

Connection Pooling:

The database connection pool belongs to the orchestrator, not to your workflow or worker code — there is no Orcher.builder().with_pool() API. Workers are stateless gRPC clients and never open a database connection; only the orchestrator talks to PostgreSQL. You size the pool where the orchestrator runs, via environment variables:

1
# Orchestrator connection pool (set per orchestrator node)
2
ORCHER_PG_MAX_CONNECTIONS=80 # max pooled connections (default 80)
3
ORCHER_PG_MIN_CONNECTIONS=20 # warm connections kept open (default 20)

Sizing rule — every orchestrator node holds up to ORCHER_PG_MAX_CONNECTIONS, so when you run N nodes against one database:

1
N × ORCHER_PG_MAX_CONNECTIONS ≤ PostgreSQL max_connections (default 100)

To run several orchestrator nodes against a single PostgreSQL, lower ORCHER_PG_MAX_CONNECTIONS per node (e.g. three nodes at 30 each = 90 < 100). For higher aggregate concurrency without exhausting PostgreSQL, put PgBouncer in front of the database in transaction-pooling mode — the orchestrator's connections then multiplex over a small number of real backend connections. If you need more headroom directly, raise max_connections in postgresql.conf.

Because the pool is an orchestrator concern, adding workers never adds database connections — workers scale execution capacity over gRPC, decoupled from the database. See the scaling guide for the full worker-vs-orchestrator model.

Workflow Optimization Patterns

Pattern 1: Parallel Task Execution

When tasks don't depend on each other, execute them in parallel:

1
// Sequential (slow)
2
let email_result = ctx.execute_task(send_email, email_data).await?;
3
let sms_result = ctx.execute_task(send_sms, sms_data).await?;
4
let push_result = ctx.execute_task(send_push, push_data).await?;
5
6
// Parallel (fast)
7
let (email_result, sms_result, push_result) = tokio::join!(
8
ctx.execute_task(send_email, email_data),
9
ctx.execute_task(send_sms, sms_data),
10
ctx.execute_task(send_push, push_data),
11
);
12
13
let email_result = email_result?;
14
let sms_result = sms_result?;
15
let push_result = push_result?;

Pattern 2: Batch Processing

Process multiple items together instead of one at a time:

1
// One at a time (slow)
2
for user_id in user_ids {
3
ctx.execute_task(send_email, user_id).await?;
4
}
5
6
// Batch processing (fast)
7
ctx.execute_task(send_emails_batch, user_ids).await?;
8
9
#[task]
10
async fn send_emails_batch(ctx: TaskContext, user_ids: Vec<String>) -> Result<Vec<String>> {
11
// Send all emails in one task execution
12
let mut results = Vec::new();
13
for user_id in user_ids {
14
let result = send_email_to_user(user_id).await?;
15
results.push(result);
16
}
17
Ok(results)
18
}

Pattern 3: Result Caching

Cache expensive computations in task results:

1
// The result of this task is cached in the journal
2
let expensive_data = ctx.execute_task(load_large_dataset, query).await?;
3
4
// Use cached data multiple times (no re-computation)
5
let analysis1 = ctx.execute_task(analyze_data, (expensive_data.clone(), "method1")).await?;
6
let analysis2 = ctx.execute_task(analyze_data, (expensive_data.clone(), "method2")).await?;
7
let report = ctx.execute_task(generate_report, (analysis1, analysis2)).await?;

Scaling Workers

Horizontal Scaling:

1
# Kubernetes deployment for Orcher workers
2
apiVersion: apps/v1
3
kind: Deployment
4
metadata:
5
name: orcher-workers
6
spec:
7
replicas: 5 # Scale up worker count
8
selector:
9
matchLabels:
10
app: orcher-worker
11
template:
12
metadata:
13
labels:
14
app: orcher-worker
15
spec:
16
containers:
17
- name: worker
18
image: my-app:latest
19
env:
20
- name: ORCHER_SERVER_URL
21
value: "http://orchestrator:50051" # gRPC endpoint of the orchestrator
22
- name: ORCHER_NAMESPACE
23
value: "production"
24
- name: ORCHER_TASK_QUEUE
25
value: "default"
26
resources:
27
requests:
28
cpu: 1000m
29
memory: 2Gi
30
limits:
31
cpu: 2000m
32
memory: 4Gi

Worker Configuration:

A worker is its own process: a stateless gRPC client of the orchestrator. It opens no database connection — it pulls work over gRPC and runs your registered workflows and tasks. Scale out by running more replicas (as in the Deployment above); scale up within a process with the max_concurrent_* knobs.

1
use orcher::prelude::*;
2
use std::time::Duration;
3
4
#[tokio::main]
5
async fn main() -> Result<()> {
6
// Workflows and tasks are auto-registered from their macros — no DB pool here.
7
WorkerBuilder::new()
8
.server_url(
9
std::env::var("ORCHER_SERVER_URL")
10
.unwrap_or_else(|_| "http://orchestrator:50051".to_string()),
11
)
12
.namespace("production")
13
.task_queue("default")
14
.max_concurrent_workflows(20) // workflows running at once in this process
15
.max_concurrent_tasks(40) // tasks running at once in this process
16
.heartbeat_interval(Duration::from_secs(30)) // liveness to the orchestrator
17
.build()
18
.await?
19
.run() // runs until shutdown
20
.await?;
21
22
Ok(())
23
}

Deployment Best Practices

Configuration Management

Environment-specific Configuration:

1
use config::{Config, Environment, File};
2
3
#[derive(Debug, Deserialize)]
4
struct OrcherConfig {
5
database_url: String,
6
worker_count: usize,
7
max_retry_attempts: u32,
8
retry_backoff_ms: u64,
9
log_level: String,
10
}
11
12
fn load_config() -> Result<OrcherConfig> {
13
let env = std::env::var("ENV").unwrap_or_else(|_| "development".to_string());
14
15
let config = Config::builder()
16
// Default config
17
.add_source(File::with_name("config/default"))
18
// Environment-specific config
19
.add_source(File::with_name(&format!("config/{}", env)).required(false))
20
// Environment variables (override everything)
21
.add_source(Environment::with_prefix("ORCHER"))
22
.build()?;
23
24
Ok(config.try_deserialize()?)
25
}

Configuration Files:

1
# config/production.yaml
2
database_url: "postgresql://user:pass@prod-db:5432/orcher"
3
worker_count: 10
4
max_retry_attempts: 10
5
retry_backoff_ms: 5000
6
log_level: "info"
7
8
# config/staging.yaml
9
database_url: "postgresql://user:pass@staging-db:5432/orcher"
10
worker_count: 2
11
max_retry_attempts: 5
12
retry_backoff_ms: 1000
13
log_level: "debug"

Database Migrations

Migration Strategy:

1
-- migrations/001_initial_schema.sql
2
CREATE TABLE workflows (
3
id UUID PRIMARY KEY,
4
workflow_name TEXT NOT NULL,
5
input JSONB NOT NULL,
6
result JSONB,
7
status TEXT NOT NULL,
8
error TEXT,
9
attempt INTEGER NOT NULL DEFAULT 1,
10
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
11
started_at TIMESTAMPTZ,
12
completed_at TIMESTAMPTZ,
13
next_retry_at TIMESTAMPTZ
14
);
15
16
CREATE TABLE workflow_execution_journal (
17
id UUID PRIMARY KEY,
18
workflow_id UUID NOT NULL REFERENCES workflows(id) ON DELETE CASCADE,
19
step_id INTEGER NOT NULL,
20
task_name TEXT NOT NULL,
21
input JSONB NOT NULL,
22
result JSONB,
23
error TEXT,
24
status TEXT NOT NULL,
25
attempt INTEGER NOT NULL DEFAULT 1,
26
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
27
completed_at TIMESTAMPTZ,
28
29
UNIQUE(workflow_id, step_id)
30
);
31
32
-- migrations/002_add_indexes.sql
33
CREATE INDEX idx_workflows_status_created ON workflows(status, created_at);
34
CREATE INDEX idx_workflow_execution_journal_lookup ON workflow_execution_journal(workflow_id, step_id);
35
36
-- migrations/003_add_workflow_metadata.sql
37
ALTER TABLE workflows ADD COLUMN metadata JSONB DEFAULT '{}';
38
CREATE INDEX idx_workflows_metadata ON workflows USING GIN(metadata);

Run Migrations:

1
use sqlx::migrate::Migrator;
2
3
static MIGRATOR: Migrator = sqlx::migrate!("./migrations");
4
5
async fn run_migrations(pool: &PgPool) -> Result<()> {
6
MIGRATOR.run(pool).await?;
7
Ok(())
8
}

Health Checks

Liveness and Readiness Probes:

1
use axum::{Router, routing::get, Json};
2
use serde_json::json;
3
4
async fn health_check() -> Json<serde_json::Value> {
5
Json(json!({
6
"status": "healthy",
7
"timestamp": chrono::Utc::now().to_rfc3339(),
8
}))
9
}
10
11
async fn readiness_check(
12
Extension(pool): Extension<PgPool>,
13
) -> Json<serde_json::Value> {
14
// Check database connectivity
15
let db_ok = sqlx::query("SELECT 1").fetch_one(&pool).await.is_ok();
16
17
// Check queue depth
18
let queue_depth: i64 = sqlx::query_scalar(
19
"SELECT COUNT(*) FROM workflows WHERE status = 'pending'"
20
)
21
.fetch_one(&pool)
22
.await
23
.unwrap_or(0);
24
25
let ready = db_ok && queue_depth < 10000;
26
27
Json(json!({
28
"ready": ready,
29
"database": db_ok,
30
"queue_depth": queue_depth,
31
"timestamp": chrono::Utc::now().to_rfc3339(),
32
}))
33
}
34
35
// Add to router
36
let app = Router::new()
37
.route("/health", get(health_check))
38
.route("/ready", get(readiness_check));

Kubernetes Configuration:

1
apiVersion: v1
2
kind: Pod
3
spec:
4
containers:
5
- name: orcher-worker
6
image: my-app:latest
7
livenessProbe:
8
httpGet:
9
path: /health
10
port: 8080
11
initialDelaySeconds: 30
12
periodSeconds: 10
13
timeoutSeconds: 5
14
failureThreshold: 3
15
16
readinessProbe:
17
httpGet:
18
path: /ready
19
port: 8080
20
initialDelaySeconds: 10
21
periodSeconds: 5
22
timeoutSeconds: 3
23
failureThreshold: 2

Graceful Shutdown

Handle shutdown signals properly to avoid interrupting workflows:

1
use tokio::signal;
2
3
async fn shutdown_signal() {
4
let ctrl_c = async {
5
signal::ctrl_c()
6
.await
7
.expect("failed to install Ctrl+C handler");
8
};
9
10
#[cfg(unix)]
11
let terminate = async {
12
signal::unix::signal(signal::unix::SignalKind::terminate())
13
.expect("failed to install signal handler")
14
.recv()
15
.await;
16
};
17
18
#[cfg(not(unix))]
19
let terminate = std::future::pending::<()>();
20
21
tokio::select! {
22
_ = ctrl_c => {},
23
_ = terminate => {},
24
}
25
26
info!("Shutdown signal received, starting graceful shutdown...");
27
}
28
29
#[tokio::main]
30
async fn main() -> Result<()> {
31
let orcher = Orcher::new().await?;
32
33
// Run server with graceful shutdown
34
let server = axum::Server::bind(&"0.0.0.0:8080".parse()?)
35
.serve(app.into_make_service())
36
.with_graceful_shutdown(shutdown_signal());
37
38
// Shutdown workers when signal received
39
tokio::select! {
40
result = server => {
41
result?;
42
}
43
_ = shutdown_signal() => {
44
info!("Stopping workers...");
45
orcher.stop_workers().await?;
46
info!("Workers stopped gracefully");
47
}
48
}
49
50
Ok(())
51
}

Production Checklist

Before deploying Orcher to production, verify:

Infrastructure:

  • ✓ PostgreSQL 13+ with sufficient resources (CPU, memory, disk)
  • ✓ Database backups configured (continuous + daily snapshots)
  • ✓ Connection pooling properly configured
  • ✓ Database indexes created
  • ✓ High availability setup (optional: read replicas, failover)

Application:

  • ✓ All tasks are idempotent
  • ✓ Retry policies configured appropriately
  • ✓ Error handling covers all edge cases
  • ✓ Logging includes workflow_id for correlation
  • ✓ Metrics exported (workflow duration, failure rate, queue depth)
  • ✓ Health checks implemented (liveness + readiness)
  • ✓ Graceful shutdown handling

Monitoring:

  • ✓ Alerts configured for high failure rate
  • ✓ Alerts configured for stuck workflows
  • ✓ Alerts configured for queue backlog
  • ✓ Dashboard showing key metrics
  • ✓ Log aggregation setup (e.g., ELK, Datadog)

Testing:

  • ✓ Unit tests for all tasks
  • ✓ Integration tests for workflows
  • ✓ Load testing completed
  • ✓ Failure scenario testing (network, database outages)
  • ✓ Idempotency verified

Operations:

  • ✓ Runbook created for common issues
  • ✓ On-call rotation established
  • ✓ Rollback procedure documented
  • ✓ Database migration strategy tested
  • ✓ Disaster recovery plan in place

Next Steps


What You've Learned

Congratulations! You now understand:

  • Durable execution fundamentals - How journaling eliminates entire classes of bugs
  • Payment processing - survive crashes without losing track of a charge or repeating it
  • Webhook handling - Reliable event processing at scale
  • Distributed sagas - Transactions across services without 2PC
  • The Orcher mental model - Thinking in workflows and tasks
  • Production excellence - Monitoring, testing, performance, and deployment best practices

The key insight: Orcher makes reliability the default, not something you have to build yourself.


Back to Overview — Learning path overview