Local Development

Debugging Workflows


Debug workflows effectively using Orcher's Admin UI, logging infrastructure, and debugging tools. Learn how to inspect workflow state, trace execution, and troubleshoot common issues.

Choose Your SDK

Debugging Tools: • Admin UI at http://localhost:8081 • RUST_LOG environment variable • tracing crate for structured logging • Database queries for journal inspection


Admin UI Debugger

The Admin UI is your primary tool for debugging workflows. Access it at http://localhost:8081.

Workflow Execution View

Navigate to a workflow:

  1. Open http://localhost:8081
  2. Click "Workflows" in the sidebar
  3. Select your workflow from the list
  4. View execution details

What you can see:

• Workflow Status - Running, completed, failed, cancelled • Execution Timeline - Visual representation of step execution • Input/Output - View workflow input and output data • Duration - Time spent in each step and total execution time • Errors - Any errors that occurred during execution

Journal Inspection

View journal entries:

  1. Open workflow details
  2. Click "Journal" tab
  3. See all recorded steps

Journal information:

1
{
2
"step_name": "charge_card",
3
"step_id": "step_abc123",
4
"status": "completed",
5
"input": {
6
"amount": 99.99,
7
"card_token": "tok_visa"
8
},
9
"output": {
10
"charge_id": "ch_123456",
11
"status": "succeeded"
12
},
13
"started_at": "2024-01-15T10:30:00Z",
14
"completed_at": "2024-01-15T10:30:02Z",
15
"duration_ms": 2150,
16
"retry_count": 0
17
}

Task Execution History

View task executions:

  1. Open workflow details
  2. Click "Tasks" tab
  3. See all task attempts including retries

For each task you can see:

• Task name and ID • Execution status (pending, running, completed, failed) • Retry attempts • Input and output data • Execution duration • Error messages (if failed)

Real-Time Monitoring

Watch workflows in real-time:

  1. Open workflow details
  2. Enable "Live Updates" toggle
  3. See steps complete as they execute

Use cases:

• Monitor long-running workflows • Watch for failures as they happen • Verify workflow behavior during development • Debug timing-related issues


Logging and Tracing

Enable Debug Logging

Set log level:

1
# Enable debug logs for your worker
2
RUST_LOG=debug cargo run
3
4
# Enable trace logs (very verbose)
5
RUST_LOG=trace cargo run
6
7
# Enable logs for specific modules
8
RUST_LOG=my_workflow=debug,orcher=info cargo run
9
10
# Multiple targets
11
RUST_LOG=my_workflow=trace,orcher::worker=debug,orcher::executor=trace cargo run

In code:

1
use tracing::{debug, info, warn, error, trace};
2
3
#[workflow]
4
async fn debug_workflow(ctx: WorkflowContext, input: Input) -> Result<Output> {
5
info!("Starting workflow with input: {:?}", input);
6
7
// Log before step
8
debug!("About to charge card for amount: {}", input.amount);
9
10
let payment = ctx.execute("charge_card", || async {
11
trace!("Calling Stripe API");
12
let result = stripe::charge(input.amount).await;
13
trace!("Stripe API returned: {:?}", result);
14
result
15
}).await?;
16
17
info!("Payment successful: charge_id={}", payment.charge_id);
18
19
// Log warning for unusual conditions
20
if payment.amount > 10000.0 {
21
warn!("High-value payment detected: ${}", payment.amount);
22
}
23
24
Ok(Output { payment })
25
}

Structured logging:

1
use tracing::{info, instrument};
2
3
#[instrument(skip(ctx))]
4
async fn process_order(ctx: &WorkflowContext, order_id: &str) -> Result<()> {
5
// Automatically logs function entry/exit with parameters
6
info!(order_id = %order_id, "Processing order");
7
8
let result = ctx.execute("validate", || async {
9
validate_order(order_id).await
10
}).await?;
11
12
info!(
13
order_id = %order_id,
14
validation_result = ?result,
15
"Order validated"
16
);
17
18
Ok(())
19
}

Log to File

1
use tracing_subscriber::{fmt, EnvFilter};
2
use tracing_appender::rolling::{RollingFileAppender, Rotation};
3
4
#[tokio::main]
5
async fn main() -> orcher::Result<()> {
6
// Create file appender (rotates daily)
7
let file_appender = RollingFileAppender::new(
8
Rotation::DAILY,
9
"logs",
10
"workflow.log"
11
);
12
13
// Configure logging to file and stdout
14
tracing_subscriber::fmt()
15
.with_env_filter(EnvFilter::from_default_env())
16
.with_writer(file_appender)
17
.with_ansi(false) // No colors in file
18
.json() // JSON format for parsing
19
.init();
20
21
// Your worker code
22
let worker = WorkerBuilder::new()
23
.server_url("http://localhost:9090")
24
.task_queue("default")
25
.build()
26
.await?;
27
28
worker.run().await
29
}

Log rotation:

1
use tracing_appender::rolling::Rotation;
2
3
// Rotate daily
4
let appender = RollingFileAppender::new(Rotation::DAILY, "logs", "workflow.log");
5
6
// Rotate hourly
7
let appender = RollingFileAppender::new(Rotation::HOURLY, "logs", "workflow.log");
8
9
// Rotate at specific size
10
let appender = RollingFileAppender::new(Rotation::NEVER, "logs", "workflow.log");
11
// Use external tool like logrotate for size-based rotation

Inspecting the Journal Database

Query the journal directly for deep debugging:

Connect to Database

1
# Connect with psql
2
psql postgresql://postgres:postgres@localhost:5432/orcher_dev
3
4
# Or use any PostgreSQL client
5
pgcli postgresql://postgres:postgres@localhost:5432/orcher_dev

Useful Queries

View all workflow executions:

1
SELECT
2
id,
3
workflow_name,
4
status,
5
started_at,
6
completed_at,
7
completed_at - started_at as duration
8
FROM workflow_executions
9
ORDER BY started_at DESC
10
LIMIT 10;

View journal for a specific workflow:

1
SELECT
2
step_name,
3
status,
4
retry_count,
5
started_at,
6
completed_at,
7
error_message
8
FROM execution_log
9
WHERE workflow_id = 'wf_abc123'
10
ORDER BY sequence_number;

Find failed workflows:

1
SELECT
2
we.id,
3
we.workflow_name,
4
we.started_at,
5
el.step_name,
6
el.error_message
7
FROM workflow_executions we
8
JOIN execution_log el ON we.id = el.workflow_id
9
WHERE we.status = 'failed'
10
ORDER BY we.started_at DESC;

Find workflows with retries:

1
SELECT
2
workflow_id,
3
step_name,
4
retry_count,
5
error_message
6
FROM execution_log
7
WHERE retry_count > 0
8
ORDER BY retry_count DESC;

View workflow timeline:

1
SELECT
2
step_name,
3
status,
4
started_at,
5
completed_at,
6
EXTRACT(EPOCH FROM (completed_at - started_at)) as duration_seconds
7
FROM execution_log
8
WHERE workflow_id = 'wf_abc123'
9
ORDER BY sequence_number;

Find slow steps:

1
SELECT
2
step_name,
3
COUNT(*) as executions,
4
AVG(EXTRACT(EPOCH FROM (completed_at - started_at))) as avg_duration,
5
MAX(EXTRACT(EPOCH FROM (completed_at - started_at))) as max_duration
6
FROM execution_log
7
WHERE completed_at IS NOT NULL
8
GROUP BY step_name
9
ORDER BY avg_duration DESC;

Common Issues and Solutions

Workflow Stuck in "Running" State

Symptoms: • Workflow shows as "running" but no progress • No recent journal entries • Tasks not executing

Debug steps:

  1. Check worker is running:
1
# Check worker logs
2
docker logs -f orcher_worker
3
4
# Verify worker connected to orchestrator
5
curl http://localhost:9090/api/workers
  1. Check database connection:
1
# Test connection
2
psql $DATABASE_URL -c "SELECT 1"
3
4
# Check for locks
5
SELECT * FROM pg_locks WHERE NOT granted;
  1. Inspect workflow state:
1
SELECT * FROM workflow_executions WHERE id = 'wf_abc123';
2
SELECT * FROM execution_log WHERE workflow_id = 'wf_abc123' ORDER BY sequence_number DESC LIMIT 5;

Solutions:

• Restart the worker • Check network connectivity between worker and orchestrator • Verify database isn't full or locked • Check for deadlocks in the database

Task Failing Repeatedly

Symptoms: • Task fails and retries multiple times • Same error message each time • Eventually workflow fails

Debug steps:

  1. View error details in Admin UI:

    • Open workflow details
    • Go to "Tasks" tab
    • Click on failed task
    • Read full error message and stack trace
  2. Check logs:

1
RUST_LOG=debug cargo run 2>&1 | grep "task_name"
  1. Query retry history:
1
SELECT
2
step_name,
3
retry_count,
4
error_message,
5
started_at
6
FROM execution_log
7
WHERE workflow_id = 'wf_abc123' AND step_name = 'failing_task'
8
ORDER BY started_at;

Common causes:

• External service down - Check external API availability • Invalid credentials - Verify API keys and tokens • Network timeout - Increase timeout or check network • Data validation error - Check input data format • Rate limiting - Implement backoff or request quota

Solutions:

1
// Add better error handling. `retry = 5` caps the number of attempts; for full
2
// control use a retry_policy(...) with initial_interval / backoff_coefficient.
3
#[task(retry = 5)]
4
async fn robust_task(ctx: TaskContext, input: Input) -> Result<Output> {
5
match external_service::call(input).await {
6
Ok(result) => Ok(result),
7
Err(e) if e.is_retryable() => {
8
warn!("Retryable error: {}", e);
9
// Return the error to let Orcher retry per the policy
10
Err(e.into())
11
}
12
Err(e) => {
13
error!("Non-retryable error: {}", e);
14
// Fail immediately without retrying
15
Err(anyhow::anyhow!("non-retryable: {e}").into())
16
}
17
}
18
}

Workflow Replaying Incorrectly

Symptoms: • After restart, workflow re-executes completed steps • Duplicate API calls • Inconsistent state

Debug steps:

  1. Verify journal integrity:
1
-- Check for gaps in sequence numbers
2
SELECT
3
workflow_id,
4
step_name,
5
sequence_number,
6
LAG(sequence_number) OVER (ORDER BY sequence_number) as prev_seq
7
FROM execution_log
8
WHERE workflow_id = 'wf_abc123'
9
ORDER BY sequence_number;
  1. Check step IDs are deterministic:
1
// Bad - non-deterministic step name
2
ctx.execute(&format!("step_{}", uuid::Uuid::new_v4()), || async {
3
// ...
4
}).await?;
5
6
// Good - deterministic step name
7
ctx.execute("charge_card", || async {
8
// ...
9
}).await?;
  1. Verify workflow is deterministic:
1
// Bad - uses system time directly
2
let timestamp = SystemTime::now();
3
4
// Good - uses workflow context time
5
let timestamp = ctx.time().now();

Solutions:

• Use consistent, deterministic step names • Don't use random values outside ctx.rand() • Don't use system time outside ctx.time().now() • Ensure workflow logic doesn't depend on external mutable state

Database Performance Issues

Symptoms: • Slow workflow execution • High database CPU usage • Timeouts connecting to database

Debug steps:

  1. Check active connections:
1
SELECT count(*) FROM pg_stat_activity;
2
SELECT * FROM pg_stat_activity WHERE datname = 'orcher_dev';
  1. Find slow queries:
1
SELECT
2
query,
3
calls,
4
total_time,
5
mean_time
6
FROM pg_stat_statements
7
ORDER BY mean_time DESC
8
LIMIT 10;
  1. Check table sizes:
1
SELECT
2
tablename,
3
pg_size_pretty(pg_total_relation_size(schemaname||'.'||tablename)) AS size
4
FROM pg_tables
5
WHERE schemaname = 'public'
6
ORDER BY pg_total_relation_size(schemaname||'.'||tablename) DESC;

Solutions:

• Add indexes on frequently queried columns • Increase database connection pool size • Archive old workflow executions • Vacuum and analyze tables regularly • Consider read replicas for queries

Memory Leaks

Symptoms: • Worker memory usage grows over time • Eventually crashes with OOM • Performance degrades

Debug steps:

  1. Monitor memory usage:
1
# Watch memory over time
2
watch -n 1 'ps aux | grep workflow-worker'
3
4
# Use htop
5
htop -p $(pgrep -f workflow-worker)
  1. Profile with heaptrack (Linux):
1
heaptrack ./target/release/workflow-worker
2
# Run workflows
3
# Ctrl+C to stop
4
heaptrack_gui heaptrack.workflow-worker.*.gz
  1. Check for unbounded collections:
1
// Bad - grows forever
2
static CACHE: Lazy<HashMap<String, Data>> = Lazy::new(HashMap::new);
3
4
// Good - bounded cache
5
static CACHE: Lazy<LruCache<String, Data>> = Lazy::new(|| LruCache::new(1000));

Solutions:

• Use bounded caches (LRU) • Clean up completed workflow state • Avoid holding large data in memory • Stream large datasets instead of loading all at once • Restart workers periodically as a safeguard


Performance Debugging

Profiling Workflow Execution

Using tracing spans:

1
use tracing::{info_span, instrument};
2
3
#[workflow]
4
async fn profiled_workflow(ctx: WorkflowContext, input: Input) -> Result<Output> {
5
let _span = info_span!("workflow_execution").entered();
6
7
let step1 = {
8
let _span = info_span!("step1_charge_card").entered();
9
ctx.execute("charge_card", || async {
10
charge_card_impl().await
11
}).await?
12
};
13
14
let step2 = {
15
let _span = info_span!("step2_reserve_inventory").entered();
16
ctx.execute("reserve_inventory", || async {
17
reserve_inventory_impl().await
18
}).await?
19
};
20
21
Ok(Output { step1, step2 })
22
}

Using cargo-flamegraph:

1
# Install
2
cargo install flamegraph
3
4
# Run with profiling
5
sudo flamegraph target/release/workflow-worker
6
7
# Open flamegraph.svg in browser

Using tokio-console:

1
# Add to Cargo.toml
2
[dependencies]
3
console-subscriber = "0.2"
4
5
# In code
6
console_subscriber::init();
7
8
# Run tokio-console
9
tokio-console

Finding Bottlenecks

  1. Check step durations in Admin UI
  2. Query database for slow steps:
1
SELECT
2
step_name,
3
AVG(EXTRACT(EPOCH FROM (completed_at - started_at))) as avg_seconds,
4
COUNT(*) as count
5
FROM execution_log
6
WHERE completed_at IS NOT NULL
7
GROUP BY step_name
8
ORDER BY avg_seconds DESC
9
LIMIT 10;
  1. Enable metrics:
1
use orcher::interceptor::MetricsInterceptor;
2
3
#[tokio::main]
4
async fn main() -> orcher::Result<()> {
5
// Enable metrics via the built-in metrics interceptor
6
let worker = WorkerBuilder::new()
7
.server_url("http://localhost:9090")
8
.task_queue("default")
9
.interceptor(MetricsInterceptor::new())
10
.build()
11
.await?;
12
13
worker.run().await
14
}
  1. View metrics:
1
# Prometheus format
2
curl http://localhost:9091/metrics
3
4
# Grep for specific metrics
5
curl -s http://localhost:9091/metrics | grep workflow_duration

Debugging Tools

CLI Commands

1
# List running workflows
2
orcher workflow list --status running
3
4
# Get workflow details
5
orcher workflow get wf_abc123
6
7
# View workflow history (journal)
8
orcher workflow history wf_abc123
9
10
# Cancel stuck workflow
11
orcher workflow cancel wf_abc123
12
13
# View worker status
14
orcher worker list
15
16
# Check orchestrator health
17
orcher health

HTTP API Debugging

1
# Get workflow status
2
curl http://localhost:8080/api/workflows/wf_abc123
3
4
# List workflows
5
curl http://localhost:8080/api/workflows
6
7
# Get workflow journal
8
curl http://localhost:8080/api/workflows/wf_abc123/journal
9
10
# Health check
11
curl http://localhost:8080/health

Best Practices

1. Add Debug Logging Early

1
#[workflow]
2
async fn well_instrumented_workflow(
3
ctx: WorkflowContext,
4
input: Input
5
) -> Result<Output> {
6
info!("Workflow started with input: {:?}", input);
7
8
debug!("Validating input");
9
validate_input(&input)?;
10
11
debug!("Executing step 1");
12
let result1 = ctx.execute("step1", || async {
13
// ...
14
}).await?;
15
debug!("Step 1 completed: {:?}", result1);
16
17
info!("Workflow completed successfully");
18
Ok(Output::from(result1))
19
}

2. Use Descriptive Error Messages

1
// Bad
2
return Err("failed".into());
3
4
// Good
5
return Err(format!(
6
"Failed to charge card: card_token={}, amount={}, error={}",
7
card_token, amount, e
8
).into());

3. Add Context to Errors

1
use anyhow::Context;
2
3
let payment = stripe::charge(amount, card)
4
.await
5
.context("Failed to charge card via Stripe")?;

4. Test Error Paths

1
#[tokio::test]
2
async fn test_handles_stripe_failure() {
3
// Test what happens when Stripe fails
4
// Verify logs, retries, error messages
5
}

5. Monitor in Production

Set up alerts for: • High failure rates • Long execution times • Database errors • Memory usage spikes • Error log patterns


Next Steps


Learn More

• Workflow Development - Best practices to prevent bugs • Monitoring Guide - Production monitoring setup • Troubleshooting - Common issues and solutions