Guides

Best Practices & Performance


Performance

Journal Write Performance

  • Batch inserts: Multiple entries inserted via UNNEST in a single query rather than one per entry
  • Append-only: No updates or deletes during normal operation — write-optimized
  • Indexed: (workflow_id, execution_id, event_id) unique index for fast lookups

Journal Read Performance

  • In-memory cache: Moka-based HighPerfCache with 60s TTL and 5K max entries
  • Incremental loading: After a cache hit, only new entries (event_id > max cached) are fetched from PostgreSQL
  • Optimistic append: New entries written to cache immediately — no full reload needed
  • Sequential access: Journal read in event_id order (matches PostgreSQL index scan)

Storage Growth

Monitor journal size and implement retention policies:

1
-- Check journal size per workflow
2
SELECT
3
workflow_id,
4
COUNT(*) AS entry_count,
5
MIN(created_at) AS oldest_entry,
6
MAX(created_at) AS newest_entry
7
FROM workflow_execution_journal
8
GROUP BY workflow_id
9
ORDER BY entry_count DESC
10
LIMIT 20;
11
12
-- Archive old completed workflows
13
DELETE FROM workflow_execution_journal
14
WHERE workflow_id IN (
15
SELECT workflow_id
16
FROM workflow_executions
17
WHERE status = 'COMPLETED'
18
AND completed_at < NOW() - INTERVAL '90 days'
19
);

Best Practices

1. Keep Steps Small and Focused

Small steps mean finer-grained checkpoints — less work repeated after a crash.

1
// Good: Many small journaled steps
2
let validated = ctx.execute_task(validate, input.clone()).await?;
3
let processed = ctx.execute_task(process, validated).await?;
4
let stored = ctx.execute_task(store, processed).await?;
5
6
// Bad: One giant step (all-or-nothing — crash means full retry)
7
let result = ctx.execute_task(do_everything, input).await?;

2. Make Tasks Idempotent

Journaling prevents duplicate execution during replay, but tasks may still retry on transient failures. Design tasks to be safely retryable:

1
// Use idempotency keys for external APIs
2
let charge = stripe.create_charge(
3
amount,
4
IdempotencyKey::new(format!("order_{}", order_id))
5
);
6
7
// Use upserts for database operations
8
db.execute(
9
"INSERT INTO orders (...) VALUES (...)
10
ON CONFLICT (order_id) DO UPDATE SET ..."
11
);

3. Don't Journal Sensitive Data

Step inputs and outputs are stored as JSONB in PostgreSQL. Avoid logging secrets:

1
// Bad: Full card token stored in journal
2
ctx.execute_task(charge, card_token).await?;
3
4
// Good: Only reference IDs stored
5
ctx.execute_task(charge, (amount, payment_method_id)).await?;

4. Monitor Journal Size

Set up alerts for abnormally large journals that may indicate infinite loops or overly granular steps:

1
-- Workflows with unusually large journals
2
SELECT workflow_id, COUNT(*) AS entry_count
3
FROM workflow_execution_journal
4
GROUP BY workflow_id
5
HAVING COUNT(*) > 10000;

Troubleshooting

Workflow Keeps Replaying

Symptom: Workflow restarts from beginning repeatedly

Causes:

  • Non-deterministic workflow code (different control flow on replay)
  • External state changes between replays
  • Step that produces different result on replay

Solution: Ensure all non-deterministic operations are wrapped in tasks or ctx.execute():

1
// Problem: Uses current time (different on replay)
2
// let now = SystemTime::now();
3
4
// Solution: Wrap in a journaled step
5
let now: u64 = ctx.execute("get_time", || async {
6
Ok(SystemTime::now().duration_since(UNIX_EPOCH)?.as_secs())
7
}).await?;

Journal Too Large

Symptom: Slow workflow performance, large database

Solutions:

  • Combine related steps to reduce entry count
  • Store large payloads externally (S3, blob storage) and journal only the reference
  • Implement journal archival for completed workflows

Missing Journal Entries

Symptom: Workflow fails on replay with cache miss

Causes:

  • Database connection lost during write
  • Transaction rolled back
  • Disk full

Solutions:

  • Check PostgreSQL logs for write errors
  • Verify database connectivity and disk space
  • Monitor journal_misses metric in HighPerfCache

Next Steps


  • Workflows — Learn how workflows orchestrate tasks with journaling
  • Tasks — Define idempotent task implementations
  • Error Handling — Handle failures with retry policies
  • Events — Signal workflows with external events