Production Excellence
Now that you understand the core concepts and patterns, let's explore how to run Orcher workflows in production with confidence.
Monitoring and Observability
Understanding Workflow State
Orcher provides rich visibility into workflow execution through the workflow journal and metadata tables. Every workflow moves through a small set of states — and monitoring is mostly about watching for the two that mean trouble.
failed, and on running that never terminates — the rest are healthy by design.Query Workflow Status:
1-- Get workflow overview2SELECT3 id,4 workflow_name,5 status,6 created_at,7 completed_at,8 attempt,9 error10FROM workflows11WHERE status IN ('running', 'failed')12ORDER BY created_at DESC13LIMIT 100;Monitor Task Performance:
1-- Find slow tasks across all workflows2SELECT3 task_name,4 COUNT(*) as execution_count,5 AVG(EXTRACT(EPOCH FROM (completed_at - created_at))) as avg_duration_seconds,6 MAX(EXTRACT(EPOCH FROM (completed_at - created_at))) as max_duration_seconds,7 PERCENTILE_CONT(0.95) WITHIN GROUP (ORDER BY EXTRACT(EPOCH FROM (completed_at - created_at))) as p95_duration_seconds8FROM workflow_execution_journal9WHERE status = 'completed'10 AND completed_at > NOW() - INTERVAL '24 hours'11GROUP BY task_name12ORDER BY avg_duration_seconds DESC;Track Failure Rates:
1-- Calculate task failure rates2SELECT3 task_name,4 COUNT(*) FILTER (WHERE status = 'completed') as success_count,5 COUNT(*) FILTER (WHERE status = 'failed') as failure_count,6 ROUND(7 100.0 * COUNT(*) FILTER (WHERE status = 'failed') / COUNT(*),8 29 ) as failure_rate_percent10FROM workflow_execution_journal11WHERE created_at > NOW() - INTERVAL '7 days'12GROUP BY task_name13HAVING COUNT(*) > 1014ORDER BY failure_rate_percent DESC;Structured Logging
Add contextual logging to your workflows and tasks:
1use tracing::{info, warn, error, instrument};23#[workflow(name = "process_payment")]4#[instrument(skip(ctx), fields(workflow_id = %ctx.workflow_id()))]5pub async fn process_payment(6 ctx: WorkflowContext,7 payment: PaymentInput,8) -> Result<PaymentResult> {9 info!(10 user_id = %payment.user_id,11 amount = payment.amount,12 "Starting payment processing"13 );1415 // Log decision points16 let fraud_score = ctx.execute_task(check_fraud, payment.user_id.clone()).await?;1718 if fraud_score > 0.8 {19 warn!(20 user_id = %payment.user_id,21 fraud_score = fraud_score,22 "Payment flagged as fraudulent"23 );24 return Err(anyhow::anyhow!("Payment flagged"));25 }2627 info!(fraud_score = fraud_score, "Fraud check passed");2829 // Execute payment30 let charge = ctx.execute_task(charge_stripe, payment.clone()).await?;3132 info!(33 charge_id = %charge.id,34 amount = payment.amount,35 "Payment charged successfully"36 );3738 Ok(PaymentResult {39 charge_id: charge.id,40 status: "completed".to_string(),41 receipt_url: charge.receipt_url,42 })43}4445#[task]46#[instrument]47async fn charge_stripe(ctx: TaskContext, payment: PaymentInput) -> Result<StripeCharge> {48 info!("Calling Stripe API");4950 let start = std::time::Instant::now();51 let charge = make_stripe_call(payment).await?;52 let duration = start.elapsed();5354 info!(55 duration_ms = duration.as_millis(),56 charge_id = %charge.id,57 "Stripe API call completed"58 );5960 Ok(charge)61}Logging Best Practices:
✓ Always include workflow_id for correlation
✓ Log at decision points (if/else branches)
✓ Log external API calls with duration
✓ Log business-significant events
✗ Don't log sensitive data (credit cards, passwords)
✗ Don't log inside tight loops
Metrics and Alerting
Key Metrics to Track:
-
Workflow Completion Rate
1SELECT2workflow_name,3COUNT(*) FILTER (WHERE status = 'completed') * 100.0 / COUNT(*) as completion_rate4FROM workflows5WHERE created_at > NOW() - INTERVAL '1 hour'6GROUP BY workflow_name; -
Workflow Duration (P50, P95, P99)
1SELECT2workflow_name,3PERCENTILE_CONT(0.5) WITHIN GROUP (ORDER BY EXTRACT(EPOCH FROM (completed_at - created_at))) as p50_seconds,4PERCENTILE_CONT(0.95) WITHIN GROUP (ORDER BY EXTRACT(EPOCH FROM (completed_at - created_at))) as p95_seconds,5PERCENTILE_CONT(0.99) WITHIN GROUP (ORDER BY EXTRACT(EPOCH FROM (completed_at - created_at))) as p99_seconds6FROM workflows7WHERE status = 'completed'8AND created_at > NOW() - INTERVAL '1 hour'9GROUP BY workflow_name; -
Queue Depth (Pending Workflows)
1SELECT COUNT(*) as pending_workflows2FROM workflows3WHERE status = 'pending'; -
Retry Rate
1SELECT2workflow_name,3AVG(attempt) as avg_attempts,4MAX(attempt) as max_attempts5FROM workflows6WHERE created_at > NOW() - INTERVAL '1 hour'7GROUP BY workflow_name;
Alerting Rules:
- Critical: Workflow failure rate > 5% over 5 minutes
- Warning: P95 duration > 2x baseline
- Warning: Queue depth > 1000 workflows
- Critical: Any workflow stuck for > 1 hour
Example Prometheus Metrics Export:
1use prometheus::{register_histogram_vec, register_counter_vec};23// In your application initialization4let workflow_duration = register_histogram_vec!(5 "orcher_workflow_duration_seconds",6 "Workflow execution duration",7 &["workflow_name", "status"],8 vec![0.1, 0.5, 1.0, 5.0, 10.0, 30.0, 60.0]9).unwrap();1011let workflow_executions = register_counter_vec!(12 "orcher_workflow_executions_total",13 "Total workflow executions",14 &["workflow_name", "status"]15).unwrap();1617// In your workflow completion handler18workflow_duration19 .with_label_values(&[workflow_name, "completed"])20 .observe(duration_seconds);2122workflow_executions23 .with_label_values(&[workflow_name, "completed"])24 .inc();Testing Strategies
Unit Testing Tasks
Tasks are just async functions, so they're straightforward to test:
1#[cfg(test)]2mod tests {3 use super::*;45 #[tokio::test]6 async fn test_charge_stripe_success() {7 // Arrange8 let payment = PaymentInput {9 user_id: "user_123".to_string(),10 amount: 5000,11 currency: "usd".to_string(),12 payment_method: "pm_test_card".to_string(),13 };1415 // Act16 let result = charge_stripe(payment).await;1718 // Assert19 assert!(result.is_ok());20 let charge = result.unwrap();21 assert_eq!(charge.amount, 5000);22 assert!(!charge.id.is_empty());23 }2425 #[tokio::test]26 async fn test_charge_stripe_invalid_card() {27 let payment = PaymentInput {28 user_id: "user_123".to_string(),29 amount: 5000,30 currency: "usd".to_string(),31 payment_method: "pm_invalid".to_string(),32 };3334 let result = charge_stripe(payment).await;3536 assert!(result.is_err());37 assert!(result.unwrap_err().to_string().contains("card"));38 }3940 #[tokio::test]41 async fn test_task_idempotency() {42 // Test that running task twice produces same result43 let user_id = "user_123";44 let amount = 100;4546 // First execution47 let result1 = update_user_balance(user_id.to_string(), amount).await.unwrap();4849 // Get balance50 let balance1 = get_user_balance(user_id).await.unwrap();5152 // Second execution (should be idempotent)53 let result2 = update_user_balance(user_id.to_string(), amount).await.unwrap();5455 // Balance should be the same56 let balance2 = get_user_balance(user_id).await.unwrap();57 assert_eq!(balance1, balance2);58 }59}Integration Testing Workflows
Test workflows with a real Orcher instance and database:
1#[cfg(test)]2mod integration_tests {3 use super::*;4 use orcher::test_utils::TestOrcher;56 #[tokio::test]7 async fn test_payment_workflow_success() {8 // Setup test Orcher instance with in-memory DB9 let orcher = TestOrcher::new().await;1011 // Register workflow12 orcher.register_workflow(process_payment).await;1314 // Start workflow15 let payment = PaymentInput {16 user_id: "user_123".to_string(),17 amount: 5000,18 currency: "usd".to_string(),19 payment_method: "pm_test_card".to_string(),20 };2122 let workflow_id = orcher23 .start_workflow("process_payment", payment)24 .await25 .unwrap();2627 // Wait for completion (with timeout)28 let result = orcher29 .wait_for_workflow::<PaymentResult>(workflow_id, Duration::from_secs(30))30 .await31 .unwrap();3233 // Assert result34 assert_eq!(result.status, "completed");35 assert!(!result.charge_id.is_empty());3637 // Verify journal entries38 let journal = orcher.get_workflow_journal(workflow_id).await.unwrap();39 assert_eq!(journal.len(), 4); // validate, fraud_check, charge, save40 assert!(journal.iter().all(|e| e.status == "completed"));41 }4243 #[tokio::test]44 async fn test_payment_workflow_fraud_rejection() {45 let orcher = TestOrcher::new().await;46 orcher.register_workflow(process_payment).await;4748 // Use high-risk user ID that triggers fraud detection49 let payment = PaymentInput {50 user_id: "high_risk_user".to_string(),51 amount: 5000,52 currency: "usd".to_string(),53 payment_method: "pm_test_card".to_string(),54 };5556 let workflow_id = orcher57 .start_workflow("process_payment", payment)58 .await59 .unwrap();6061 // Wait for failure62 let result = orcher63 .wait_for_workflow::<PaymentResult>(workflow_id, Duration::from_secs(30))64 .await;6566 assert!(result.is_err());6768 // Verify workflow failed at fraud check69 let workflow = orcher.get_workflow(workflow_id).await.unwrap();70 assert_eq!(workflow.status, "failed");71 assert!(workflow.error.unwrap().contains("fraud"));72 }7374 #[tokio::test]75 async fn test_workflow_retry_on_transient_failure() {76 let orcher = TestOrcher::new().await;77 orcher.register_workflow(process_payment).await;7879 // Inject transient failure in stripe task80 orcher.inject_task_failure("charge_stripe", 2); // Fail first 2 attempts8182 let payment = PaymentInput {83 user_id: "user_123".to_string(),84 amount: 5000,85 currency: "usd".to_string(),86 payment_method: "pm_test_card".to_string(),87 };8889 let workflow_id = orcher90 .start_workflow("process_payment", payment)91 .await92 .unwrap();9394 // Should eventually succeed after retries95 let result = orcher96 .wait_for_workflow::<PaymentResult>(workflow_id, Duration::from_secs(60))97 .await98 .unwrap();99100 assert_eq!(result.status, "completed");101102 // Verify retry attempts103 let workflow = orcher.get_workflow(workflow_id).await.unwrap();104 assert!(workflow.attempt >= 3);105 }106}End-to-End Testing
Test complete flows with real external services (use sandbox/test mode):
1#[tokio::test]2#[ignore] // Run with --ignored flag for E2E tests3async fn test_payment_flow_e2e() {4 // Use real Stripe test mode API5 std::env::set_var("STRIPE_SECRET_KEY", "sk_test_...");67 let orcher = Orcher::new(get_test_db_url()).await;8 orcher.register_workflow(process_payment).await;910 // Real test card from Stripe11 let payment = PaymentInput {12 user_id: "user_123".to_string(),13 amount: 100, // $1.0014 currency: "usd".to_string(),15 payment_method: "pm_card_visa".to_string(), // Stripe test card16 };1718 let workflow_id = orcher19 .start_workflow("process_payment", payment)20 .await21 .unwrap();2223 let result = orcher24 .wait_for_workflow::<PaymentResult>(workflow_id, Duration::from_secs(60))25 .await26 .unwrap();2728 assert_eq!(result.status, "completed");2930 // Verify in real Stripe dashboard31 let stripe = stripe::Client::new(std::env::var("STRIPE_SECRET_KEY").unwrap());32 let charge = stripe.charges.retrieve(&result.charge_id).await.unwrap();33 assert_eq!(charge.amount, 100);34 assert_eq!(charge.status, "succeeded");35}Performance Optimization
Database Tuning
Index Optimization:
1-- Essential indexes for Orcher tables2CREATE INDEX CONCURRENTLY idx_workflows_status_created3ON workflows(status, created_at)4WHERE status IN ('pending', 'running');56CREATE INDEX CONCURRENTLY idx_workflow_execution_journal_lookup7ON workflow_execution_journal(workflow_id, step_id);89CREATE INDEX CONCURRENTLY idx_workflow_execution_journal_task_perf10ON workflow_execution_journal(task_name, status, created_at, completed_at)11WHERE completed_at IS NOT NULL;1213-- Index for queue polling (critical for performance)14CREATE INDEX CONCURRENTLY idx_workflows_queue_poll15ON workflows(status, created_at)16WHERE status = 'pending';Connection Pooling:
The database connection pool belongs to the orchestrator, not to your workflow or worker code — there is no Orcher.builder().with_pool() API. Workers are stateless gRPC clients and never open a database connection; only the orchestrator talks to PostgreSQL. You size the pool where the orchestrator runs, via environment variables:
1# Orchestrator connection pool (set per orchestrator node)2ORCHER_PG_MAX_CONNECTIONS=80 # max pooled connections (default 80)3ORCHER_PG_MIN_CONNECTIONS=20 # warm connections kept open (default 20)Sizing rule — every orchestrator node holds up to ORCHER_PG_MAX_CONNECTIONS, so when you run N nodes against one database:
1N × ORCHER_PG_MAX_CONNECTIONS ≤ PostgreSQL max_connections (default 100)To run several orchestrator nodes against a single PostgreSQL, lower ORCHER_PG_MAX_CONNECTIONS per node (e.g. three nodes at 30 each = 90 < 100). For higher aggregate concurrency without exhausting PostgreSQL, put PgBouncer in front of the database in transaction-pooling mode — the orchestrator's connections then multiplex over a small number of real backend connections. If you need more headroom directly, raise max_connections in postgresql.conf.
Because the pool is an orchestrator concern, adding workers never adds database connections — workers scale execution capacity over gRPC, decoupled from the database. See the scaling guide for the full worker-vs-orchestrator model.
Workflow Optimization Patterns
Pattern 1: Parallel Task Execution
When tasks don't depend on each other, execute them in parallel:
1// Sequential (slow)2let email_result = ctx.execute_task(send_email, email_data).await?;3let sms_result = ctx.execute_task(send_sms, sms_data).await?;4let push_result = ctx.execute_task(send_push, push_data).await?;56// Parallel (fast)7let (email_result, sms_result, push_result) = tokio::join!(8 ctx.execute_task(send_email, email_data),9 ctx.execute_task(send_sms, sms_data),10 ctx.execute_task(send_push, push_data),11);1213let email_result = email_result?;14let sms_result = sms_result?;15let push_result = push_result?;Pattern 2: Batch Processing
Process multiple items together instead of one at a time:
1// One at a time (slow)2for user_id in user_ids {3 ctx.execute_task(send_email, user_id).await?;4}56// Batch processing (fast)7ctx.execute_task(send_emails_batch, user_ids).await?;89#[task]10async fn send_emails_batch(ctx: TaskContext, user_ids: Vec<String>) -> Result<Vec<String>> {11 // Send all emails in one task execution12 let mut results = Vec::new();13 for user_id in user_ids {14 let result = send_email_to_user(user_id).await?;15 results.push(result);16 }17 Ok(results)18}Pattern 3: Result Caching
Cache expensive computations in task results:
1// The result of this task is cached in the journal2let expensive_data = ctx.execute_task(load_large_dataset, query).await?;34// Use cached data multiple times (no re-computation)5let analysis1 = ctx.execute_task(analyze_data, (expensive_data.clone(), "method1")).await?;6let analysis2 = ctx.execute_task(analyze_data, (expensive_data.clone(), "method2")).await?;7let report = ctx.execute_task(generate_report, (analysis1, analysis2)).await?;Scaling Workers
Horizontal Scaling:
1# Kubernetes deployment for Orcher workers2apiVersion: apps/v13kind: Deployment4metadata:5 name: orcher-workers6spec:7 replicas: 5 # Scale up worker count8 selector:9 matchLabels:10 app: orcher-worker11 template:12 metadata:13 labels:14 app: orcher-worker15 spec:16 containers:17 - name: worker18 image: my-app:latest19 env:20 - name: ORCHER_SERVER_URL21 value: "http://orchestrator:50051" # gRPC endpoint of the orchestrator22 - name: ORCHER_NAMESPACE23 value: "production"24 - name: ORCHER_TASK_QUEUE25 value: "default"26 resources:27 requests:28 cpu: 1000m29 memory: 2Gi30 limits:31 cpu: 2000m32 memory: 4GiWorker Configuration:
A worker is its own process: a stateless gRPC client of the orchestrator. It opens no database connection — it pulls work over gRPC and runs your registered workflows and tasks. Scale out by running more replicas (as in the Deployment above); scale up within a process with the max_concurrent_* knobs.
1use orcher::prelude::*;2use std::time::Duration;34#[tokio::main]5async fn main() -> Result<()> {6 // Workflows and tasks are auto-registered from their macros — no DB pool here.7 WorkerBuilder::new()8 .server_url(9 std::env::var("ORCHER_SERVER_URL")10 .unwrap_or_else(|_| "http://orchestrator:50051".to_string()),11 )12 .namespace("production")13 .task_queue("default")14 .max_concurrent_workflows(20) // workflows running at once in this process15 .max_concurrent_tasks(40) // tasks running at once in this process16 .heartbeat_interval(Duration::from_secs(30)) // liveness to the orchestrator17 .build()18 .await?19 .run() // runs until shutdown20 .await?;2122 Ok(())23}Deployment Best Practices
Configuration Management
Environment-specific Configuration:
1use config::{Config, Environment, File};23#[derive(Debug, Deserialize)]4struct OrcherConfig {5 database_url: String,6 worker_count: usize,7 max_retry_attempts: u32,8 retry_backoff_ms: u64,9 log_level: String,10}1112fn load_config() -> Result<OrcherConfig> {13 let env = std::env::var("ENV").unwrap_or_else(|_| "development".to_string());1415 let config = Config::builder()16 // Default config17 .add_source(File::with_name("config/default"))18 // Environment-specific config19 .add_source(File::with_name(&format!("config/{}", env)).required(false))20 // Environment variables (override everything)21 .add_source(Environment::with_prefix("ORCHER"))22 .build()?;2324 Ok(config.try_deserialize()?)25}Configuration Files:
1# config/production.yaml2database_url: "postgresql://user:pass@prod-db:5432/orcher"3worker_count: 104max_retry_attempts: 105retry_backoff_ms: 50006log_level: "info"78# config/staging.yaml9database_url: "postgresql://user:pass@staging-db:5432/orcher"10worker_count: 211max_retry_attempts: 512retry_backoff_ms: 100013log_level: "debug"Database Migrations
Migration Strategy:
1-- migrations/001_initial_schema.sql2CREATE TABLE workflows (3 id UUID PRIMARY KEY,4 workflow_name TEXT NOT NULL,5 input JSONB NOT NULL,6 result JSONB,7 status TEXT NOT NULL,8 error TEXT,9 attempt INTEGER NOT NULL DEFAULT 1,10 created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),11 started_at TIMESTAMPTZ,12 completed_at TIMESTAMPTZ,13 next_retry_at TIMESTAMPTZ14);1516CREATE TABLE workflow_execution_journal (17 id UUID PRIMARY KEY,18 workflow_id UUID NOT NULL REFERENCES workflows(id) ON DELETE CASCADE,19 step_id INTEGER NOT NULL,20 task_name TEXT NOT NULL,21 input JSONB NOT NULL,22 result JSONB,23 error TEXT,24 status TEXT NOT NULL,25 attempt INTEGER NOT NULL DEFAULT 1,26 created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),27 completed_at TIMESTAMPTZ,2829 UNIQUE(workflow_id, step_id)30);3132-- migrations/002_add_indexes.sql33CREATE INDEX idx_workflows_status_created ON workflows(status, created_at);34CREATE INDEX idx_workflow_execution_journal_lookup ON workflow_execution_journal(workflow_id, step_id);3536-- migrations/003_add_workflow_metadata.sql37ALTER TABLE workflows ADD COLUMN metadata JSONB DEFAULT '{}';38CREATE INDEX idx_workflows_metadata ON workflows USING GIN(metadata);Run Migrations:
1use sqlx::migrate::Migrator;23static MIGRATOR: Migrator = sqlx::migrate!("./migrations");45async fn run_migrations(pool: &PgPool) -> Result<()> {6 MIGRATOR.run(pool).await?;7 Ok(())8}Health Checks
Liveness and Readiness Probes:
1use axum::{Router, routing::get, Json};2use serde_json::json;34async fn health_check() -> Json<serde_json::Value> {5 Json(json!({6 "status": "healthy",7 "timestamp": chrono::Utc::now().to_rfc3339(),8 }))9}1011async fn readiness_check(12 Extension(pool): Extension<PgPool>,13) -> Json<serde_json::Value> {14 // Check database connectivity15 let db_ok = sqlx::query("SELECT 1").fetch_one(&pool).await.is_ok();1617 // Check queue depth18 let queue_depth: i64 = sqlx::query_scalar(19 "SELECT COUNT(*) FROM workflows WHERE status = 'pending'"20 )21 .fetch_one(&pool)22 .await23 .unwrap_or(0);2425 let ready = db_ok && queue_depth < 10000;2627 Json(json!({28 "ready": ready,29 "database": db_ok,30 "queue_depth": queue_depth,31 "timestamp": chrono::Utc::now().to_rfc3339(),32 }))33}3435// Add to router36let app = Router::new()37 .route("/health", get(health_check))38 .route("/ready", get(readiness_check));Kubernetes Configuration:
1apiVersion: v12kind: Pod3spec:4 containers:5 - name: orcher-worker6 image: my-app:latest7 livenessProbe:8 httpGet:9 path: /health10 port: 808011 initialDelaySeconds: 3012 periodSeconds: 1013 timeoutSeconds: 514 failureThreshold: 31516 readinessProbe:17 httpGet:18 path: /ready19 port: 808020 initialDelaySeconds: 1021 periodSeconds: 522 timeoutSeconds: 323 failureThreshold: 2Graceful Shutdown
Handle shutdown signals properly to avoid interrupting workflows:
1use tokio::signal;23async fn shutdown_signal() {4 let ctrl_c = async {5 signal::ctrl_c()6 .await7 .expect("failed to install Ctrl+C handler");8 };910 #[cfg(unix)]11 let terminate = async {12 signal::unix::signal(signal::unix::SignalKind::terminate())13 .expect("failed to install signal handler")14 .recv()15 .await;16 };1718 #[cfg(not(unix))]19 let terminate = std::future::pending::<()>();2021 tokio::select! {22 _ = ctrl_c => {},23 _ = terminate => {},24 }2526 info!("Shutdown signal received, starting graceful shutdown...");27}2829#[tokio::main]30async fn main() -> Result<()> {31 let orcher = Orcher::new().await?;3233 // Run server with graceful shutdown34 let server = axum::Server::bind(&"0.0.0.0:8080".parse()?)35 .serve(app.into_make_service())36 .with_graceful_shutdown(shutdown_signal());3738 // Shutdown workers when signal received39 tokio::select! {40 result = server => {41 result?;42 }43 _ = shutdown_signal() => {44 info!("Stopping workers...");45 orcher.stop_workers().await?;46 info!("Workers stopped gracefully");47 }48 }4950 Ok(())51}Production Checklist
Before deploying Orcher to production, verify:
Infrastructure:
- ✓ PostgreSQL 13+ with sufficient resources (CPU, memory, disk)
- ✓ Database backups configured (continuous + daily snapshots)
- ✓ Connection pooling properly configured
- ✓ Database indexes created
- ✓ High availability setup (optional: read replicas, failover)
Application:
- ✓ All tasks are idempotent
- ✓ Retry policies configured appropriately
- ✓ Error handling covers all edge cases
- ✓ Logging includes workflow_id for correlation
- ✓ Metrics exported (workflow duration, failure rate, queue depth)
- ✓ Health checks implemented (liveness + readiness)
- ✓ Graceful shutdown handling
Monitoring:
- ✓ Alerts configured for high failure rate
- ✓ Alerts configured for stuck workflows
- ✓ Alerts configured for queue backlog
- ✓ Dashboard showing key metrics
- ✓ Log aggregation setup (e.g., ELK, Datadog)
Testing:
- ✓ Unit tests for all tasks
- ✓ Integration tests for workflows
- ✓ Load testing completed
- ✓ Failure scenario testing (network, database outages)
- ✓ Idempotency verified
Operations:
- ✓ Runbook created for common issues
- ✓ On-call rotation established
- ✓ Rollback procedure documented
- ✓ Database migration strategy tested
- ✓ Disaster recovery plan in place
Next Steps
Building Workflows
Deep dive into workflow primitives, deterministic execution, and composition patterns.
Defining Tasks
Retry policies, timeouts, heartbeats, and task context APIs.
Saga Pattern
Distributed transactions with compensating actions for multi-service workflows.
Microservice Orchestration
Coordinate distributed systems with child workflows, events, and saga patterns.
What You've Learned
Congratulations! You now understand:
- Durable execution fundamentals - How journaling eliminates entire classes of bugs
- Payment processing - survive crashes without losing track of a charge or repeating it
- Webhook handling - Reliable event processing at scale
- Distributed sagas - Transactions across services without 2PC
- The Orcher mental model - Thinking in workflows and tasks
- Production excellence - Monitoring, testing, performance, and deployment best practices
The key insight: Orcher makes reliability the default, not something you have to build yourself.
Back to Overview — Learning path overview