Best Practices & Performance
Performance
Journal Write Performance
- Batch inserts: Multiple entries inserted via
UNNESTin a single query rather than one per entry - Append-only: No updates or deletes during normal operation — write-optimized
- Indexed:
(workflow_id, execution_id, event_id)unique index for fast lookups
Journal Read Performance
- In-memory cache: Moka-based HighPerfCache with 60s TTL and 5K max entries
- Incremental loading: After a cache hit, only new entries (event_id > max cached) are fetched from PostgreSQL
- Optimistic append: New entries written to cache immediately — no full reload needed
- Sequential access: Journal read in
event_idorder (matches PostgreSQL index scan)
Storage Growth
Monitor journal size and implement retention policies:
1-- Check journal size per workflow2SELECT3 workflow_id,4 COUNT(*) AS entry_count,5 MIN(created_at) AS oldest_entry,6 MAX(created_at) AS newest_entry7FROM workflow_execution_journal8GROUP BY workflow_id9ORDER BY entry_count DESC10LIMIT 20;1112-- Archive old completed workflows13DELETE FROM workflow_execution_journal14WHERE workflow_id IN (15 SELECT workflow_id16 FROM workflow_executions17 WHERE status = 'COMPLETED'18 AND completed_at < NOW() - INTERVAL '90 days'19);Best Practices
1. Keep Steps Small and Focused
Small steps mean finer-grained checkpoints — less work repeated after a crash.
1// Good: Many small journaled steps2let validated = ctx.execute_task(validate, input.clone()).await?;3let processed = ctx.execute_task(process, validated).await?;4let stored = ctx.execute_task(store, processed).await?;56// Bad: One giant step (all-or-nothing — crash means full retry)7let result = ctx.execute_task(do_everything, input).await?;2. Make Tasks Idempotent
Journaling prevents duplicate execution during replay, but tasks may still retry on transient failures. Design tasks to be safely retryable:
1// Use idempotency keys for external APIs2let charge = stripe.create_charge(3 amount,4 IdempotencyKey::new(format!("order_{}", order_id))5);67// Use upserts for database operations8db.execute(9 "INSERT INTO orders (...) VALUES (...)10 ON CONFLICT (order_id) DO UPDATE SET ..."11);3. Don't Journal Sensitive Data
Step inputs and outputs are stored as JSONB in PostgreSQL. Avoid logging secrets:
1// Bad: Full card token stored in journal2ctx.execute_task(charge, card_token).await?;34// Good: Only reference IDs stored5ctx.execute_task(charge, (amount, payment_method_id)).await?;4. Monitor Journal Size
Set up alerts for abnormally large journals that may indicate infinite loops or overly granular steps:
1-- Workflows with unusually large journals2SELECT workflow_id, COUNT(*) AS entry_count3FROM workflow_execution_journal4GROUP BY workflow_id5HAVING COUNT(*) > 10000;Troubleshooting
Workflow Keeps Replaying
Symptom: Workflow restarts from beginning repeatedly
Causes:
- Non-deterministic workflow code (different control flow on replay)
- External state changes between replays
- Step that produces different result on replay
Solution: Ensure all non-deterministic operations are wrapped in tasks or ctx.execute():
1// Problem: Uses current time (different on replay)2// let now = SystemTime::now();34// Solution: Wrap in a journaled step5let now: u64 = ctx.execute("get_time", || async {6 Ok(SystemTime::now().duration_since(UNIX_EPOCH)?.as_secs())7}).await?;Journal Too Large
Symptom: Slow workflow performance, large database
Solutions:
- Combine related steps to reduce entry count
- Store large payloads externally (S3, blob storage) and journal only the reference
- Implement journal archival for completed workflows
Missing Journal Entries
Symptom: Workflow fails on replay with cache miss
Causes:
- Database connection lost during write
- Transaction rolled back
- Disk full
Solutions:
- Check PostgreSQL logs for write errors
- Verify database connectivity and disk space
- Monitor
journal_missesmetric in HighPerfCache
Next Steps
Defining Tasks
Define reusable, type-safe tasks that get journaled automatically
Managing State
Manage persistent state across workflow executions
Related Guides
- Workflows — Learn how workflows orchestrate tasks with journaling
- Tasks — Define idempotent task implementations
- Error Handling — Handle failures with retry policies
- Events — Signal workflows with external events