Debugging Workflows
Debug workflows effectively using Orcher's Admin UI, logging infrastructure, and debugging tools. Learn how to inspect workflow state, trace execution, and troubleshoot common issues.
Choose Your SDK
Debugging Tools:
• Admin UI at http://localhost:8081
• RUST_LOG environment variable
• tracing crate for structured logging
• Database queries for journal inspection
Admin UI Debugger
The Admin UI is your primary tool for debugging workflows. Access it at http://localhost:8081.
Workflow Execution View
Navigate to a workflow:
- Open
http://localhost:8081 - Click "Workflows" in the sidebar
- Select your workflow from the list
- View execution details
What you can see:
• Workflow Status - Running, completed, failed, cancelled • Execution Timeline - Visual representation of step execution • Input/Output - View workflow input and output data • Duration - Time spent in each step and total execution time • Errors - Any errors that occurred during execution
Journal Inspection
View journal entries:
- Open workflow details
- Click "Journal" tab
- See all recorded steps
Journal information:
1{2 "step_name": "charge_card",3 "step_id": "step_abc123",4 "status": "completed",5 "input": {6 "amount": 99.99,7 "card_token": "tok_visa"8 },9 "output": {10 "charge_id": "ch_123456",11 "status": "succeeded"12 },13 "started_at": "2024-01-15T10:30:00Z",14 "completed_at": "2024-01-15T10:30:02Z",15 "duration_ms": 2150,16 "retry_count": 017}Task Execution History
View task executions:
- Open workflow details
- Click "Tasks" tab
- See all task attempts including retries
For each task you can see:
• Task name and ID • Execution status (pending, running, completed, failed) • Retry attempts • Input and output data • Execution duration • Error messages (if failed)
Real-Time Monitoring
Watch workflows in real-time:
- Open workflow details
- Enable "Live Updates" toggle
- See steps complete as they execute
Use cases:
• Monitor long-running workflows • Watch for failures as they happen • Verify workflow behavior during development • Debug timing-related issues
Logging and Tracing
Enable Debug Logging
Set log level:
1# Enable debug logs for your worker2RUST_LOG=debug cargo run34# Enable trace logs (very verbose)5RUST_LOG=trace cargo run67# Enable logs for specific modules8RUST_LOG=my_workflow=debug,orcher=info cargo run910# Multiple targets11RUST_LOG=my_workflow=trace,orcher::worker=debug,orcher::executor=trace cargo runIn code:
1use tracing::{debug, info, warn, error, trace};23#[workflow]4async fn debug_workflow(ctx: WorkflowContext, input: Input) -> Result<Output> {5 info!("Starting workflow with input: {:?}", input);6 7 // Log before step8 debug!("About to charge card for amount: {}", input.amount);9 10 let payment = ctx.execute("charge_card", || async {11 trace!("Calling Stripe API");12 let result = stripe::charge(input.amount).await;13 trace!("Stripe API returned: {:?}", result);14 result15 }).await?;16 17 info!("Payment successful: charge_id={}", payment.charge_id);18 19 // Log warning for unusual conditions20 if payment.amount > 10000.0 {21 warn!("High-value payment detected: ${}", payment.amount);22 }23 24 Ok(Output { payment })25}Structured logging:
1use tracing::{info, instrument};23#[instrument(skip(ctx))]4async fn process_order(ctx: &WorkflowContext, order_id: &str) -> Result<()> {5 // Automatically logs function entry/exit with parameters6 info!(order_id = %order_id, "Processing order");7 8 let result = ctx.execute("validate", || async {9 validate_order(order_id).await10 }).await?;11 12 info!(13 order_id = %order_id,14 validation_result = ?result,15 "Order validated"16 );17 18 Ok(())19}Log to File
1use tracing_subscriber::{fmt, EnvFilter};2use tracing_appender::rolling::{RollingFileAppender, Rotation};34#[tokio::main]5async fn main() -> orcher::Result<()> {6 // Create file appender (rotates daily)7 let file_appender = RollingFileAppender::new(8 Rotation::DAILY,9 "logs",10 "workflow.log"11 );12 13 // Configure logging to file and stdout14 tracing_subscriber::fmt()15 .with_env_filter(EnvFilter::from_default_env())16 .with_writer(file_appender)17 .with_ansi(false) // No colors in file18 .json() // JSON format for parsing19 .init();20 21 // Your worker code22 let worker = WorkerBuilder::new()23 .server_url("http://localhost:9090")24 .task_queue("default")25 .build()26 .await?;27 28 worker.run().await29}Log rotation:
1use tracing_appender::rolling::Rotation;23// Rotate daily4let appender = RollingFileAppender::new(Rotation::DAILY, "logs", "workflow.log");56// Rotate hourly7let appender = RollingFileAppender::new(Rotation::HOURLY, "logs", "workflow.log");89// Rotate at specific size10let appender = RollingFileAppender::new(Rotation::NEVER, "logs", "workflow.log");11// Use external tool like logrotate for size-based rotationInspecting the Journal Database
Query the journal directly for deep debugging:
Connect to Database
1# Connect with psql2psql postgresql://postgres:postgres@localhost:5432/orcher_dev34# Or use any PostgreSQL client5pgcli postgresql://postgres:postgres@localhost:5432/orcher_devUseful Queries
View all workflow executions:
1SELECT 2 id,3 workflow_name,4 status,5 started_at,6 completed_at,7 completed_at - started_at as duration8FROM workflow_executions9ORDER BY started_at DESC10LIMIT 10;View journal for a specific workflow:
1SELECT 2 step_name,3 status,4 retry_count,5 started_at,6 completed_at,7 error_message8FROM execution_log9WHERE workflow_id = 'wf_abc123'10ORDER BY sequence_number;Find failed workflows:
1SELECT 2 we.id,3 we.workflow_name,4 we.started_at,5 el.step_name,6 el.error_message7FROM workflow_executions we8JOIN execution_log el ON we.id = el.workflow_id9WHERE we.status = 'failed'10ORDER BY we.started_at DESC;Find workflows with retries:
1SELECT 2 workflow_id,3 step_name,4 retry_count,5 error_message6FROM execution_log7WHERE retry_count > 08ORDER BY retry_count DESC;View workflow timeline:
1SELECT 2 step_name,3 status,4 started_at,5 completed_at,6 EXTRACT(EPOCH FROM (completed_at - started_at)) as duration_seconds7FROM execution_log8WHERE workflow_id = 'wf_abc123'9ORDER BY sequence_number;Find slow steps:
1SELECT 2 step_name,3 COUNT(*) as executions,4 AVG(EXTRACT(EPOCH FROM (completed_at - started_at))) as avg_duration,5 MAX(EXTRACT(EPOCH FROM (completed_at - started_at))) as max_duration6FROM execution_log7WHERE completed_at IS NOT NULL8GROUP BY step_name9ORDER BY avg_duration DESC;Common Issues and Solutions
Workflow Stuck in "Running" State
Symptoms: • Workflow shows as "running" but no progress • No recent journal entries • Tasks not executing
Debug steps:
- Check worker is running:
1# Check worker logs2docker logs -f orcher_worker34# Verify worker connected to orchestrator5curl http://localhost:9090/api/workers- Check database connection:
1# Test connection2psql $DATABASE_URL -c "SELECT 1"34# Check for locks5SELECT * FROM pg_locks WHERE NOT granted;- Inspect workflow state:
1SELECT * FROM workflow_executions WHERE id = 'wf_abc123';2SELECT * FROM execution_log WHERE workflow_id = 'wf_abc123' ORDER BY sequence_number DESC LIMIT 5;Solutions:
• Restart the worker • Check network connectivity between worker and orchestrator • Verify database isn't full or locked • Check for deadlocks in the database
Task Failing Repeatedly
Symptoms: • Task fails and retries multiple times • Same error message each time • Eventually workflow fails
Debug steps:
-
View error details in Admin UI:
- Open workflow details
- Go to "Tasks" tab
- Click on failed task
- Read full error message and stack trace
-
Check logs:
1RUST_LOG=debug cargo run 2>&1 | grep "task_name"- Query retry history:
1SELECT 2 step_name,3 retry_count,4 error_message,5 started_at6FROM execution_log7WHERE workflow_id = 'wf_abc123' AND step_name = 'failing_task'8ORDER BY started_at;Common causes:
• External service down - Check external API availability • Invalid credentials - Verify API keys and tokens • Network timeout - Increase timeout or check network • Data validation error - Check input data format • Rate limiting - Implement backoff or request quota
Solutions:
1// Add better error handling. `retry = 5` caps the number of attempts; for full2// control use a retry_policy(...) with initial_interval / backoff_coefficient.3#[task(retry = 5)]4async fn robust_task(ctx: TaskContext, input: Input) -> Result<Output> {5 match external_service::call(input).await {6 Ok(result) => Ok(result),7 Err(e) if e.is_retryable() => {8 warn!("Retryable error: {}", e);9 // Return the error to let Orcher retry per the policy10 Err(e.into())11 }12 Err(e) => {13 error!("Non-retryable error: {}", e);14 // Fail immediately without retrying15 Err(anyhow::anyhow!("non-retryable: {e}").into())16 }17 }18}Workflow Replaying Incorrectly
Symptoms: • After restart, workflow re-executes completed steps • Duplicate API calls • Inconsistent state
Debug steps:
- Verify journal integrity:
1-- Check for gaps in sequence numbers2SELECT 3 workflow_id,4 step_name,5 sequence_number,6 LAG(sequence_number) OVER (ORDER BY sequence_number) as prev_seq7FROM execution_log8WHERE workflow_id = 'wf_abc123'9ORDER BY sequence_number;- Check step IDs are deterministic:
1// Bad - non-deterministic step name2ctx.execute(&format!("step_{}", uuid::Uuid::new_v4()), || async {3 // ...4}).await?;56// Good - deterministic step name7ctx.execute("charge_card", || async {8 // ...9}).await?;- Verify workflow is deterministic:
1// Bad - uses system time directly2let timestamp = SystemTime::now();34// Good - uses workflow context time5let timestamp = ctx.time().now();Solutions:
• Use consistent, deterministic step names
• Don't use random values outside ctx.rand()
• Don't use system time outside ctx.time().now()
• Ensure workflow logic doesn't depend on external mutable state
Database Performance Issues
Symptoms: • Slow workflow execution • High database CPU usage • Timeouts connecting to database
Debug steps:
- Check active connections:
1SELECT count(*) FROM pg_stat_activity;2SELECT * FROM pg_stat_activity WHERE datname = 'orcher_dev';- Find slow queries:
1SELECT 2 query,3 calls,4 total_time,5 mean_time6FROM pg_stat_statements7ORDER BY mean_time DESC8LIMIT 10;- Check table sizes:
1SELECT 2 tablename,3 pg_size_pretty(pg_total_relation_size(schemaname||'.'||tablename)) AS size4FROM pg_tables5WHERE schemaname = 'public'6ORDER BY pg_total_relation_size(schemaname||'.'||tablename) DESC;Solutions:
• Add indexes on frequently queried columns • Increase database connection pool size • Archive old workflow executions • Vacuum and analyze tables regularly • Consider read replicas for queries
Memory Leaks
Symptoms: • Worker memory usage grows over time • Eventually crashes with OOM • Performance degrades
Debug steps:
- Monitor memory usage:
1# Watch memory over time2watch -n 1 'ps aux | grep workflow-worker'34# Use htop5htop -p $(pgrep -f workflow-worker)- Profile with heaptrack (Linux):
1heaptrack ./target/release/workflow-worker2# Run workflows3# Ctrl+C to stop4heaptrack_gui heaptrack.workflow-worker.*.gz- Check for unbounded collections:
1// Bad - grows forever2static CACHE: Lazy<HashMap<String, Data>> = Lazy::new(HashMap::new);34// Good - bounded cache5static CACHE: Lazy<LruCache<String, Data>> = Lazy::new(|| LruCache::new(1000));Solutions:
• Use bounded caches (LRU) • Clean up completed workflow state • Avoid holding large data in memory • Stream large datasets instead of loading all at once • Restart workers periodically as a safeguard
Performance Debugging
Profiling Workflow Execution
Using tracing spans:
1use tracing::{info_span, instrument};23#[workflow]4async fn profiled_workflow(ctx: WorkflowContext, input: Input) -> Result<Output> {5 let _span = info_span!("workflow_execution").entered();6 7 let step1 = {8 let _span = info_span!("step1_charge_card").entered();9 ctx.execute("charge_card", || async {10 charge_card_impl().await11 }).await?12 };13 14 let step2 = {15 let _span = info_span!("step2_reserve_inventory").entered();16 ctx.execute("reserve_inventory", || async {17 reserve_inventory_impl().await18 }).await?19 };20 21 Ok(Output { step1, step2 })22}Using cargo-flamegraph:
1# Install2cargo install flamegraph34# Run with profiling5sudo flamegraph target/release/workflow-worker67# Open flamegraph.svg in browserUsing tokio-console:
1# Add to Cargo.toml2[dependencies]3console-subscriber = "0.2"45# In code6console_subscriber::init();78# Run tokio-console9tokio-consoleFinding Bottlenecks
- Check step durations in Admin UI
- Query database for slow steps:
1SELECT 2 step_name,3 AVG(EXTRACT(EPOCH FROM (completed_at - started_at))) as avg_seconds,4 COUNT(*) as count5FROM execution_log6WHERE completed_at IS NOT NULL7GROUP BY step_name8ORDER BY avg_seconds DESC9LIMIT 10;- Enable metrics:
1use orcher::interceptor::MetricsInterceptor;23#[tokio::main]4async fn main() -> orcher::Result<()> {5 // Enable metrics via the built-in metrics interceptor6 let worker = WorkerBuilder::new()7 .server_url("http://localhost:9090")8 .task_queue("default")9 .interceptor(MetricsInterceptor::new())10 .build()11 .await?;12 13 worker.run().await14}- View metrics:
1# Prometheus format2curl http://localhost:9091/metrics34# Grep for specific metrics5curl -s http://localhost:9091/metrics | grep workflow_durationDebugging Tools
CLI Commands
1# List running workflows2orcher workflow list --status running34# Get workflow details5orcher workflow get wf_abc12367# View workflow history (journal)8orcher workflow history wf_abc123910# Cancel stuck workflow11orcher workflow cancel wf_abc1231213# View worker status14orcher worker list1516# Check orchestrator health17orcher healthHTTP API Debugging
1# Get workflow status2curl http://localhost:8080/api/workflows/wf_abc12334# List workflows5curl http://localhost:8080/api/workflows67# Get workflow journal8curl http://localhost:8080/api/workflows/wf_abc123/journal910# Health check11curl http://localhost:8080/healthBest Practices
1. Add Debug Logging Early
1#[workflow]2async fn well_instrumented_workflow(3 ctx: WorkflowContext,4 input: Input5) -> Result<Output> {6 info!("Workflow started with input: {:?}", input);7 8 debug!("Validating input");9 validate_input(&input)?;10 11 debug!("Executing step 1");12 let result1 = ctx.execute("step1", || async {13 // ...14 }).await?;15 debug!("Step 1 completed: {:?}", result1);16 17 info!("Workflow completed successfully");18 Ok(Output::from(result1))19}2. Use Descriptive Error Messages
1// Bad2return Err("failed".into());34// Good5return Err(format!(6 "Failed to charge card: card_token={}, amount={}, error={}",7 card_token, amount, e8).into());3. Add Context to Errors
1use anyhow::Context;23let payment = stripe::charge(amount, card)4 .await5 .context("Failed to charge card via Stripe")?;4. Test Error Paths
1#[tokio::test]2async fn test_handles_stripe_failure() {3 // Test what happens when Stripe fails4 // Verify logs, retries, error messages5}5. Monitor in Production
Set up alerts for: • High failure rates • Long execution times • Database errors • Memory usage spikes • Error log patterns
Next Steps
Testing Workflows
Configuration
Production Deployment
Learn More
• Workflow Development - Best practices to prevent bugs • Monitoring Guide - Production monitoring setup • Troubleshooting - Common issues and solutions