Building Patterns
Now that you understand the concepts, let's build real systems.
Installation & Setup
First, let's get Orcher integrated into your application.
Step 1: Add dependencies
1[dependencies]2orcher = "0.1"3tokio = { version = "1", features = ["full"] }4serde = { version = "1", features = ["derive"] }Step 2: Connect to the Orcher server
Create a main.rs that connects to the orchestrator via gRPC:
1use orcher::prelude::*;23#[tokio::main]4async fn main() -> Result<()> {5 // Build a worker that connects to the Orcher server6 let worker = Worker::builder()7 .server_url("http://localhost:50051")8 .task_queue("orders")9 .build()10 .await?;1112 // Run the worker (blocks until shutdown)13 worker.run().await?;1415 Ok(())16}What's happening here:
- Server connection: The SDK connects to the Orcher server via gRPC
- Task queue: Workers poll the specified task queue for pending work
- Worker loop: The worker polls for pending workflows and tasks and executes them
Important: The Orcher server manages workflow state and dispatching. Workers connect to the server and execute workflows and tasks. You can scale workers independently by running multiple instances.
Pattern 1: Reliable Payment Processing
Let's build a payment system that survives a crash mid-payment without losing track of the money or charging twice.
The Problem We're Solving
Payment processing is high-stakes:
- Lose a charge: Company loses revenue
- Double-charge: Customer complaints, chargebacks, reputation damage
- Partial completion: Charge succeeded but email failed → customer confusion
- Fraud not checked: Financial liability
Traditional code is brittle:
1// Traditional approach (FRAGILE!)2async function processPayment(payment) {3 const charge = await stripe.charges.create({ amount: payment.amount });4 await db.payments.insert({ chargeId: charge.id });5 await sendEmail({ template: 'receipt', to: payment.email });6 // If server crashes here, email is lost forever7 await updateUserCredits(payment.userId, payment.amount);8}Problems:
- No automatic retry
- No idempotency
- No visibility into failures
- Manual debugging required
The Orcher Solution: Step by Step
Let's build this properly. We'll break it into phases:
Phase 1: The Workflow Structure
First, understand the flow. Every step is a checkpoint, and step 2 can branch to an early rejection:
Phase 2: Define Data Types
1use serde::{Deserialize, Serialize};23#[derive(Debug, Serialize, Deserialize)]4pub struct PaymentInput {5 pub user_id: String,6 pub amount: i64, // Amount in cents7 pub currency: String,8 pub payment_method: String, // Stripe payment method ID9}1011#[derive(Debug, Serialize, Deserialize)]12pub struct PaymentResult {13 pub charge_id: String,14 pub status: String,15 pub receipt_url: String,16}Phase 3: The Workflow
Now the orchestration logic:
1use orcher::prelude::*;23#[workflow(name = "process_payment")]4pub async fn process_payment(5 ctx: WorkflowContext,6 input: PaymentInput,7) -> Result<PaymentResult> {8 // Step 1: Validate (fast, local check)9 ctx.execute_task(validate_payment_method, input.payment_method.clone())10 .await?;1112 // Step 2: Fraud check (external API)13 let fraud_score: f64 = ctx.execute_task(check_fraud, (14 input.user_id.clone(),15 input.amount,16 )).await?;1718 // Decision logic in the workflow19 if fraud_score > 0.8 {20 return Err(anyhow::anyhow!("Payment flagged for fraud"));21 }2223 // Step 3: Charge card (idempotent via Stripe idempotency key)24 let charge: StripeCharge = ctx.execute_task(charge_stripe, (25 input.payment_method.clone(),26 input.amount,27 input.currency.clone(),28 )).await?;2930 // Step 4: Record in database31 ctx.execute_task(record_payment, PaymentRecord {32 user_id: input.user_id.clone(),33 charge_id: charge.id.clone(),34 amount: input.amount,35 status: "succeeded".to_string(),36 }).await?;3738 // Step 5: Send email (idempotent - won't send twice)39 ctx.execute_task(send_receipt_email, (40 input.user_id.clone(),41 charge.receipt_url.clone(),42 )).await?;4344 // Step 6: Update credits45 ctx.execute_task(update_user_credits, (46 input.user_id.clone(),47 input.amount,48 )).await?;4950 Ok(PaymentResult {51 charge_id: charge.id,52 status: "succeeded".to_string(),53 receipt_url: charge.receipt_url,54 })55}What makes this reliable:
- Each
ctx.execute_task()is checkpointed: If the workflow crashes, it resumes from the last checkpoint - Idempotent by design: Each task can be safely retried
- Decision logic isolated: The workflow controls the flow, tasks do the work
- Clear error boundaries: Any error stops the workflow; it can be retried later
Phase 4: Implementing Tasks
Now the individual task implementations. Notice how each task is focused and idempotent:
1#[task]2async fn validate_payment_method(method: String) -> Result<()> {3 // Validate format only (no external calls)4 if !method.starts_with("pm_") && !method.starts_with("card_") {5 return Err(anyhow::anyhow!("Invalid payment method format"));6 }7 Ok(())8}910#[task]11async fn check_fraud(user_id: String, amount: i64) -> Result<f64> {12 // Call fraud detection service13 let response = reqwest::Client::new()14 .post("https://fraud-api.example.com/check")15 .json(&serde_json::json!({16 "user_id": user_id,17 "amount": amount,18 }))19 .send()20 .await?;2122 let score = response.json::<serde_json::Value>().await?["score"]23 .as_f64()24 .unwrap_or(0.0);2526 Ok(score)27}2829#[task]30async fn charge_stripe(31 payment_method: String,32 amount: i64,33 currency: String,34) -> Result<StripeCharge> {35 let stripe_key = std::env::var("STRIPE_SECRET_KEY")?;3637 // Stripe's idempotency is built-in via their API38 let response = reqwest::Client::new()39 .post("https://api.stripe.com/v1/charges")40 .header("Authorization", format!("Bearer {}", stripe_key))41 .form(&[42 ("amount", amount.to_string()),43 ("currency", currency),44 ("payment_method", payment_method),45 ("confirm", "true".to_string()),46 ])47 .send()48 .await?;4950 if !response.status().is_success() {51 return Err(anyhow::anyhow!("Stripe charge failed"));52 }5354 let charge: StripeCharge = response.json().await?;55 Ok(charge)56}5758#[task]59async fn record_payment(record: PaymentRecord) -> Result<()> {60 let pool = get_db_pool();6162 // Idempotent: ON CONFLICT DO NOTHING63 sqlx::query!(64 r#"65 INSERT INTO payments (user_id, charge_id, amount, status, created_at)66 VALUES ($1, $2, $3, $4, NOW())67 ON CONFLICT (charge_id) DO NOTHING68 "#,69 record.user_id,70 record.charge_id,71 record.amount,72 record.status,73 )74 .execute(&pool)75 .await?;7677 Ok(())78}7980#[task]81async fn send_receipt_email(user_id: String, receipt_url: String) -> Result<()> {82 // Get user email83 let pool = get_db_pool();84 let user: User = sqlx::query_as!(85 User,86 "SELECT email FROM users WHERE id = $1",87 user_id88 )89 .fetch_one(&pool)90 .await?;9192 // Send via email service (idempotent - they deduplicate)93 let sendgrid_key = std::env::var("SENDGRID_API_KEY")?;9495 reqwest::Client::new()96 .post("https://api.sendgrid.com/v3/mail/send")97 .header("Authorization", format!("Bearer {}", sendgrid_key))98 .json(&serde_json::json!({99 "personalizations": [{100 "to": [{"email": user.email}],101 }],102 "from": {"email": "receipts@example.com"},103 "subject": "Payment Receipt",104 "content": [{105 "type": "text/html",106 "value": format!(107 "Your payment was successful. <a href='{}'>View Receipt</a>",108 receipt_url109 ),110 }],111 }))112 .send()113 .await?;114115 Ok(())116}117118#[task]119async fn update_user_credits(user_id: String, amount: i64) -> Result<()> {120 let pool = get_db_pool();121 let credits = amount / 100; // Convert cents to credits122123 // Idempotent: adds credits (not set)124 sqlx::query!(125 "UPDATE users SET credits = credits + $1 WHERE id = $2",126 credits,127 user_id128 )129 .execute(&pool)130 .await?;131132 Ok(())133}134135// Helper types136#[derive(Debug, Serialize, Deserialize)]137struct StripeCharge {138 id: String,139 receipt_url: String,140}141142#[derive(Debug, Serialize, Deserialize)]143struct PaymentRecord {144 user_id: String,145 charge_id: String,146 amount: i64,147 status: String,148}149150#[derive(sqlx::FromRow)]151struct User {152 email: String,153}154155fn get_db_pool() -> sqlx::PgPool {156 // Return from your app context157 todo!("Get pool from application state")158}Key Design Decisions:
validate_payment_method: No external calls, fast validationcheck_fraud: External API call, but cached on replaycharge_stripe: Stripe handles idempotency for usrecord_payment:ON CONFLICT DO NOTHINGmakes it idempotentsend_receipt_email: Email service deduplicatesupdate_user_credits: Uses+=not=, so it's additive and safe
Testing the Payment Flow
Let's see what happens in different scenarios:
Why This Is Better
Compare to traditional approaches:
Traditional:
- 200 lines of error handling
- Redis for state management
- Manual retry queues
- Hope you got the idempotency logic right
- Debugging requires checking 5 different systems
- Testing requires mocking everything
Orcher:
- 100 lines of business logic
- No manual state management
- Automatic retries
- Completed steps never repeated on recovery
- Debugging: just query the journal table
- Testing: mock individual tasks easily
Pattern 2: Webhook Processing
Webhooks are notoriously tricky. Let's build a reliable webhook handler.
The Webhook Problem
External services (Stripe, Twilio, GitHub, etc.) send HTTP requests to your server when events occur. Problems:
- Duplicate webhooks: Services retry on timeout, you process twice
- Out-of-order delivery: Event A arrives after Event B
- Lost webhooks: Your server is down, webhook is lost forever
- Slow processing: Webhook times out, service retries, you process twice
- No visibility: Did we process this webhook? What happened?
Traditional approach:
1// Traditional (FRAGILE!)2app.post('/webhooks/stripe', async (req, res) => {3 const event = req.body;45 // Quick check for duplicate6 const exists = await redis.get('webhook:' + event.id);7 if (exists) {8 return res.json({ received: true });9 }1011 // Process synchronously (blocks response)12 try {13 await processEvent(event);14 await redis.set('webhook:' + event.id, 'processed', 'EX', 86400);15 res.json({ received: true });16 } catch (error) {17 // Error - but webhook service already timed out18 // Will retry and we'll process twice!19 res.status(500).json({ error: error.message });20 }21});Problems:
- Synchronous processing (slow response)
- Redis dependency for deduplication
- If Redis is down, duplicate processing
- If processing is slow, webhook times out
- No automatic retries on failure
The Orcher Solution
With Orcher, webhooks become trivial:
Step 1: Receive and Queue
1use axum::{Json, Extension, http::StatusCode};23pub async fn handle_stripe_webhook(4 Extension(orcher): Extension<Orcher>,5 Json(event): Json<StripeWebhookEvent>,6) -> Result<Json<serde_json::Value>, StatusCode> {7 // Start workflow asynchronously (non-blocking)8 let workflow_id = orcher9 .start_workflow("stripe_webhook_processor", event)10 .await11 .map_err(|_| StatusCode::INTERNAL_SERVER_ERROR)?;1213 // Return immediately - webhook processed in background14 Ok(Json(serde_json::json!({15 "received": true,16 "workflow_id": workflow_id17 })))18}Key insight: The API handler returns immediately. Processing happens in the background worker. This keeps responses fast.
Step 2: The Processing Workflow
1#[workflow(name = "stripe_webhook_processor")]2pub async fn stripe_webhook_processor(3 ctx: WorkflowContext,4 event: StripeWebhookEvent,5) -> Result<()> {6 // Step 1: Verify webhook signature7 ctx.execute_task(verify_stripe_signature, event.id.clone())8 .await?;910 // Step 2: Check for duplicate (idempotent)11 let is_duplicate: bool = ctx.execute_task(check_duplicate, event.id.clone())12 .await?;1314 if is_duplicate {15 // Already processed - this is a retry from Stripe16 return Ok(());17 }1819 // Step 3: Route to appropriate handler20 match event.event_type.as_str() {21 "charge.succeeded" => {22 ctx.execute_task(handle_charge_succeeded, event.data).await?;23 }24 "charge.failed" => {25 ctx.execute_task(handle_charge_failed, event.data).await?;26 }27 "customer.subscription.updated" => {28 ctx.execute_task(handle_subscription_updated, event.data).await?;29 }30 _ => {31 ctx.execute_task(log_unknown_event, event.event_type).await?;32 }33 }3435 // Step 4: Mark as processed36 ctx.execute_task(mark_event_processed, event.id).await?;3738 Ok(())39}Step 3: Task Implementations
1#[task]2async fn check_duplicate(event_id: String) -> Result<bool> {3 let pool = get_db_pool();45 let result = sqlx::query!(6 "SELECT COUNT(*) as count FROM processed_webhooks WHERE event_id = $1",7 event_id8 )9 .fetch_one(&pool)10 .await?;1112 Ok(result.count.unwrap_or(0) > 0)13}1415#[task]16async fn handle_charge_succeeded(data: serde_json::Value) -> Result<()> {17 let charge_id = data["object"]["id"].as_str().unwrap();18 let customer_id = data["object"]["customer"].as_str().unwrap();1920 let pool = get_db_pool();2122 // Update order status23 sqlx::query!(24 "UPDATE orders SET status = 'paid', payment_id = $1 WHERE customer_id = $2",25 charge_id,26 customer_id27 )28 .execute(&pool)29 .await?;3031 // Send confirmation32 send_order_confirmation(customer_id, charge_id).await?;3334 Ok(())35}3637#[task]38async fn mark_event_processed(event_id: String) -> Result<()> {39 let pool = get_db_pool();4041 sqlx::query!(42 "INSERT INTO processed_webhooks (event_id, processed_at) VALUES ($1, NOW())",43 event_id44 )45 .execute(&pool)46 .await?;4748 Ok(())49}Why This Works
Fast Response:
- Webhook endpoint returns in under 50ms
- Stripe won't timeout and retry unnecessarily
- Processing happens in background
Automatic Deduplication:
check_duplicateprevents double-processing- Even if Stripe retries, we detect it
- No Redis needed - PostgreSQL is your deduplication store
Fault Tolerance:
- If processing fails, workflow retries automatically
- Each task is idempotent
- Complete audit trail in journal
Visibility:
- Query
workflow_execution_journalto see what happened - Query
processed_webhooksto see all events - Easy debugging
Pattern 3: Distributed Transactions (Saga Pattern)
Let's build a complex order fulfillment system that spans multiple services.
The Distributed Transaction Problem
Your order flow:
- Create order in your database
- Reserve inventory (external inventory service)
- Charge customer (Stripe)
- Trigger fulfillment (external warehouse service)
What if step 3 fails?
- Order created ✓
- Inventory reserved ✓
- Payment failed ✗
- Fulfillment not triggered ✗
Now you need to:
- Release the inventory reservation
- Cancel the order
- Notify the customer
This is the saga pattern: each step has a compensating action.
The Orcher Saga Implementation
Step 1: Define the Workflow
1#[workflow(name = "order_fulfillment")]2pub async fn order_fulfillment(3 ctx: WorkflowContext,4 order: OrderInput,5) -> Result<OrderResult> {6 let total_amount = order.items.iter()7 .map(|i| i.price * i.quantity as i64)8 .sum();910 // Step 1: Create order (compensatable)11 let order_id: String = ctx.execute_task(create_order, (12 order.user_id.clone(),13 order.items.clone(),14 total_amount,15 )).await?;1617 // Step 2: Reserve inventory (compensatable)18 let reservation = match ctx.execute_task(19 reserve_inventory,20 order.items.clone(),21 ).await {22 Ok(res) => res,23 Err(e) => {24 // Compensate: cancel order25 ctx.execute_task(cancel_order, order_id.clone()).await?;26 return Err(e);27 }28 };2930 // Step 3: Process payment (compensatable)31 let payment_id = match ctx.execute_task(32 charge_payment,33 (order.payment_method.clone(), total_amount),34 ).await {35 Ok(id) => id,36 Err(e) => {37 // Compensate: release inventory and cancel order38 ctx.execute_task(release_inventory, reservation.id).await?;39 ctx.execute_task(cancel_order, order_id.clone()).await?;40 return Err(e);41 }42 };4344 // Step 4: Confirm order (point of no return)45 ctx.execute_task(confirm_order, order_id.clone()).await?;4647 // Step 5: Trigger fulfillment48 ctx.execute_task(trigger_fulfillment, (49 order_id.clone(),50 reservation.id,51 )).await?;5253 Ok(OrderResult {54 order_id,55 status: "confirmed".to_string(),56 total_amount,57 })58}Key Points:
- Try-Catch Compensation: Each step that can fail has explicit compensation logic
- Ordering Matters: Steps are ordered to minimize compensation complexity
- Point of No Return: After
confirm_order, we don't compensate (customer has been charged) - Automatic Retry: If
trigger_fulfillmentfails, Orcher retries automatically
Understanding Compensation
Scenario 1: Payment fails — roll back in reverse. The charge is declined before the point of no return, so every completed step is undone by its compensation, newest first.
Scenario 2: Fulfillment is down — retry, don't roll back. The failure is after the charge, so undoing isn't an option (the customer paid). Instead the step retries until it succeeds.
Full Task Implementations:
1#[task]2async fn create_order(3 user_id: String,4 items: Vec<OrderItem>,5 total_amount: i64,6) -> Result<String> {7 let pool = get_db_pool();8 let order_id = uuid::Uuid::new_v4().to_string();910 sqlx::query!(11 "INSERT INTO orders (id, user_id, total_amount, status) VALUES ($1, $2, $3, 'pending')",12 order_id, user_id, total_amount13 )14 .execute(&pool)15 .await?;1617 // Insert items18 for item in items {19 sqlx::query!(20 "INSERT INTO order_items (order_id, product_id, quantity, price) VALUES ($1, $2, $3, $4)",21 order_id, item.product_id, item.quantity, item.price22 )23 .execute(&pool)24 .await?;25 }2627 Ok(order_id)28}2930#[task]31async fn reserve_inventory(items: Vec<OrderItem>) -> Result<InventoryReservation> {32 let reservation_id = uuid::Uuid::new_v4().to_string();3334 // Call inventory service35 let response = reqwest::Client::new()36 .post("https://inventory-service.example.com/reserve")37 .json(&serde_json::json!({38 "reservation_id": reservation_id,39 "items": items,40 }))41 .send()42 .await?;4344 if !response.status().is_success() {45 return Err(anyhow::anyhow!("Inventory reservation failed"));46 }4748 Ok(InventoryReservation {49 id: reservation_id,50 items: items.into_iter().map(|i| ReservedItem {51 product_id: i.product_id,52 quantity: i.quantity,53 }).collect(),54 })55}5657#[task]58async fn charge_payment(payment_method: String, amount: i64) -> Result<String> {59 let stripe_key = std::env::var("STRIPE_SECRET_KEY")?;6061 let response = reqwest::Client::new()62 .post("https://api.stripe.com/v1/charges")63 .header("Authorization", format!("Bearer {}", stripe_key))64 .form(&[65 ("amount", amount.to_string()),66 ("currency", "usd".to_string()),67 ("payment_method", payment_method),68 ("confirm", "true".to_string()),69 ])70 .send()71 .await?;7273 let charge: serde_json::Value = response.json().await?;74 Ok(charge["id"].as_str().unwrap().to_string())75}7677#[task]78async fn confirm_order(order_id: String) -> Result<()> {79 let pool = get_db_pool();80 sqlx::query!("UPDATE orders SET status = 'confirmed' WHERE id = $1", order_id)81 .execute(&pool)82 .await?;83 Ok(())84}8586#[task]87async fn cancel_order(order_id: String) -> Result<()> {88 let pool = get_db_pool();89 sqlx::query!("UPDATE orders SET status = 'cancelled' WHERE id = $1", order_id)90 .execute(&pool)91 .await?;92 Ok(())93}9495#[task]96async fn release_inventory(reservation_id: String) -> Result<()> {97 reqwest::Client::new()98 .post("https://inventory-service.example.com/release")99 .json(&serde_json::json!({"reservation_id": reservation_id}))100 .send()101 .await?;102 Ok(())103}104105#[task]106async fn trigger_fulfillment(order_id: String, reservation_id: String) -> Result<()> {107 reqwest::Client::new()108 .post("https://fulfillment-service.example.com/orders")109 .json(&serde_json::json!({110 "order_id": order_id,111 "reservation_id": reservation_id112 }))113 .send()114 .await?;115 Ok(())116}117118fn get_db_pool() -> sqlx::PgPool {119 todo!("Get pool from application state")120}The saga pattern in action:
This pattern ensures consistency across distributed services without a distributed transaction coordinator. Each step is compensatable, and Orcher's journaling ensures we never lose track of where we are in the process.
Next Steps
- Deep Dive: Distributed Systems — CAP theorem, event sourcing, and advanced patterns
- Part 3: Production Excellence — Monitoring, testing, and deployment
- Back to Overview — Learning path overview