Guides

Developer Patterns


Orcher provides powerful primitives for building distributed systems. These patterns show you how to compose workflows, handle failures, process data at scale, and build resilient applications.

Pattern Categories

Orchestration Patterns

Parent-Child Orchestration
Compose complex workflows from reusable child components. Coordinate distributed services, process items in parallel, and build hierarchical orchestration trees.

• Sequential and parallel child execution • Dynamic child spawning • Event-based coordination • Hierarchical workflows • Map-reduce processing

Saga & Compensation
Implement distributed transactions with the saga pattern and compensating actions. Ensure data consistency across multiple services without distributed locks.

• Forward recovery • Backward compensation • Saga coordinators • Idempotent operations

Fan-Out/Fan-In
Split work across multiple parallel operations and aggregate results. Perfect for batch processing, multi-service calls, and data aggregation.

• Parallel task execution • Result aggregation • Partial failure handling • Rate limiting

Long-Running Processes
Build workflows that span hours, days, or weeks. Handle human approvals, scheduled operations, and multi-stage business processes.

• Durable timers • Human-in-the-loop • State persistence • Process checkpoints


Resilience Patterns

Error Handling

Retry Policies
Configure automatic retries with exponential backoff and jitter.

1
#[task(
2
name = "flaky_api_call",
3
retry = 5,
4
retry_delay = "exponential",
5
max_retry_delay = 300
6
)]
7
pub async fn flaky_api_call(ctx: TaskContext, input: Data) -> Result<Response> {
8
// Automatically retried on failure
9
external_api::call(input).await
10
}

Circuit Breaker
Prevent cascading failures by detecting when a service is down.

1
#[workflow(name = "with_circuit_breaker", version = "1.0.0")]
2
pub async fn with_circuit_breaker(
3
ctx: WorkflowContext,
4
input: Data,
5
) -> Result<Response> {
6
let circuit = ctx.get_state::<CircuitState>("circuit").await?;
7
8
if circuit.is_open() {
9
return Err(Error::CircuitOpen);
10
}
11
12
match ctx.execute_task("call_service", input).await {
13
Ok(response) => {
14
circuit.record_success();
15
Ok(response)
16
}
17
Err(e) => {
18
circuit.record_failure();
19
Err(e)
20
}
21
}
22
}

Graceful Degradation
Provide fallback responses when primary services fail.

1
#[workflow(name = "resilient_workflow", version = "1.0.0")]
2
pub async fn resilient_workflow(
3
ctx: WorkflowContext,
4
query: String,
5
) -> Result<Response> {
6
// Try primary service
7
match ctx.execute_task("primary_service", query.clone()).await {
8
Ok(response) => Ok(response),
9
Err(_) => {
10
// Fall back to cached response
11
ctx.execute_task("get_cached_response", query).await
12
}
13
}
14
}

Data Processing Patterns

Batch Processing

Process large datasets by splitting work across multiple workers.

1
#[workflow(name = "batch_processor", version = "1.0.0")]
2
pub async fn batch_processor(
3
ctx: WorkflowContext,
4
items: Vec<Item>,
5
) -> Result<BatchResult> {
6
const BATCH_SIZE: usize = 100;
7
8
let batches: Vec<_> = items
9
.chunks(BATCH_SIZE)
10
.map(|chunk| chunk.to_vec())
11
.collect();
12
13
let mut handles = Vec::new();
14
for batch in batches {
15
let handle = ctx
16
.start_child_workflow("process_batch", batch)
17
.await?;
18
handles.push(handle);
19
}
20
21
// Collect results
22
let mut results = Vec::new();
23
for handle in handles {
24
results.push(handle.result::<Result>().await?);
25
}
26
27
Ok(BatchResult::from(results))
28
}

Stream Processing

Process continuous streams of events with durable state.

1
#[workflow(name = "stream_processor", version = "1.0.0")]
2
pub async fn stream_processor(
3
ctx: WorkflowContext,
4
stream_id: String,
5
) -> Result<()> {
6
loop {
7
// Wait for next batch of events
8
let events = ctx
9
.wait_for_event_with_timeout::<Vec<Event>>(
10
"batch",
11
Duration::from_secs(60)
12
)
13
.await?;
14
15
if let Some(events) = events {
16
// Process batch
17
for event in events {
18
ctx.execute_task("process_event", event).await?;
19
}
20
21
// Update checkpoint
22
ctx.set_state("last_processed", ctx.now()).await?;
23
}
24
}
25
}

Integration Patterns

External API Calls

Make reliable API calls with automatic retries and timeouts.

1
#[task(
2
name = "call_external_api",
3
retry = 3,
4
timeout = 30
5
)]
6
pub async fn call_external_api(
7
ctx: TaskContext,
8
request: ApiRequest,
9
) -> Result<ApiResponse> {
10
let client = reqwest::Client::new();
11
12
let response = client
13
.post(&request.url)
14
.json(&request.body)
15
.send()
16
.await?
17
.json()
18
.await?;
19
20
Ok(response)
21
}

Webhook Handling

Wait for webhook callbacks with timeouts.

1
#[workflow(name = "payment_with_webhook", version = "1.0.0")]
2
pub async fn payment_with_webhook(
3
ctx: WorkflowContext,
4
payment: PaymentData,
5
) -> Result<PaymentResult> {
6
// Initiate payment
7
let transaction_id = ctx
8
.execute_task("initiate_payment", payment)
9
.await?;
10
11
// Wait for webhook callback (30 minute timeout)
12
let callback = ctx
13
.wait_for_event_with_timeout::<WebhookData>(
14
"payment_callback",
15
Duration::from_secs(1800)
16
)
17
.await?;
18
19
match callback {
20
Some(data) if data.success => {
21
Ok(PaymentResult::success(transaction_id))
22
}
23
Some(data) => {
24
Ok(PaymentResult::failed(data.error))
25
}
26
None => {
27
// Timeout - check payment status
28
ctx.execute_task("check_payment_status", transaction_id).await
29
}
30
}
31
}

Coordination Patterns

Leader Election

Ensure only one workflow instance is active at a time.

1
#[workflow(name = "singleton_job", version = "1.0.0")]
2
pub async fn singleton_job(
3
ctx: WorkflowContext,
4
job_id: String,
5
) -> Result<()> {
6
// Try to acquire lock
7
let lock_acquired = ctx
8
.execute_task("try_acquire_lock", job_id.clone())
9
.await?;
10
11
if !lock_acquired {
12
return Err(Error::AlreadyRunning);
13
}
14
15
// Do work
16
ctx.execute_task("perform_job", job_id.clone()).await?;
17
18
// Release lock
19
ctx.execute_task("release_lock", job_id).await?;
20
21
Ok(())
22
}

Mutex Pattern

Coordinate access to shared resources across workflows.

1
#[workflow(name = "with_mutex", version = "1.0.0")]
2
pub async fn with_mutex(
3
ctx: WorkflowContext,
4
resource: String,
5
) -> Result<()> {
6
// Acquire mutex
7
loop {
8
let acquired = ctx
9
.execute_task("try_acquire_mutex", resource.clone())
10
.await?;
11
12
if acquired {
13
break;
14
}
15
16
// Wait and retry
17
ctx.sleep(Duration::from_secs(1)).await?;
18
}
19
20
// Critical section
21
ctx.execute_task("use_resource", resource.clone()).await?;
22
23
// Release mutex
24
ctx.execute_task("release_mutex", resource).await?;
25
26
Ok(())
27
}

Testing Patterns

Unit Testing Workflows

Test workflows in isolation with mock dependencies.

1
#[cfg(test)]
2
mod tests {
3
use super::*;
4
use orcher::testing::*;
5
6
#[tokio::test]
7
async fn test_workflow() {
8
let runtime = OrcherTestRuntime::new();
9
let ctx = runtime.workflow_context();
10
11
// Mock task responses
12
runtime
13
.mock_task("external_api_call")
14
.returns(ApiResponse { status: "ok" });
15
16
// Execute workflow
17
let result = my_workflow(ctx, input).await;
18
19
assert!(result.is_ok());
20
}
21
}

Integration Testing

Test workflows end-to-end with real dependencies.

1
#[cfg(test)]
2
mod integration_tests {
3
use super::*;
4
5
#[tokio::test]
6
async fn test_full_workflow() {
7
let client = OrcherClient::new("postgresql://localhost/test").await?;
8
9
// Start workflow
10
let handle = client
11
.start_workflow("my_workflow", "test-123", input)
12
.await?;
13
14
// Send events
15
client
16
.send_event(handle.id(), "approval", true)
17
.await?;
18
19
// Wait for completion
20
let result = handle.result::<Output>().await?;
21
22
assert_eq!(result.status, "completed");
23
}
24
}

Best Practices

Pattern Selection Guide

ScenarioPatternWhy
Multi-service coordinationParent-ChildIndependent lifecycles
Distributed transactionSagaCompensating actions
Batch processingFan-Out/Fan-InParallel execution
Human approvalLong-RunningDurable waits
External webhookEvent-basedAsync callbacks
Retry with backoffTask retry policyAutomatic recovery

Performance Tips

  1. Use tasks for fast operations (under 1 second)
  2. Use child workflows for long operations (over 1 second)
  3. Limit parallel children (max 50-100 concurrent)
  4. Batch small items (group into chunks)
  5. Use timeouts everywhere (prevent infinite waits)

Common Anti-Patterns

Don't use workflows for CPU-intensive work Use background jobs or serverless functions instead.

Don't create deep hierarchies (>3 levels) Flatten the structure for better observability.

Don't share state between workflows Use events or database for communication.

Don't spawn unlimited children Use batching and rate limiting.


Next Steps


• Core Workflows - Workflow fundamentals • Tasks - Task definition and configuration • State Management - Managing workflow state • Use Cases - Real-world examples