Workflows
A workflow is an async function that orchestrates tasks, timers, events and other workflows. The engine journals the outcome of every step. When a workflow resumes, after a timer fires, a task completes or a worker restarts, the worker runs the function again from the top and each step that already has a journaled outcome returns it instead of running again. This is replay, and it is why workflow code must be deterministic: given the same journal, it must issue the same steps in the same order.
- Do side effects (network, files, databases) in tasks or
ctx.execute, never directly in the workflow. - Read time and randomness from
ctx.time()andctx.rand(), not the system clock or a thread RNG. - Always return the error from a step with
?, or match it and pass on what you don't handle (see Handling Failures).
Defining a Workflow
1use orcher_sdk::prelude::*;23#[derive(Serialize, Deserialize)]4pub struct OrderInput {5 pub order_id: String,6 pub amount_cents: u64,7}89#[derive(Serialize, Deserialize)]10pub struct OrderOutput {11 pub order_id: String,12 pub charge_id: String,13}1415#[task(retry = 3)]16async fn charge_card(_ctx: TaskContext, input: OrderInput) -> Result<String> {17 // Call your payment provider here.18 Ok(format!("ch_{}", input.order_id))19}2021// The workflow type is "process_order"; without `name` it would be the22// function name, which here is the same.23#[workflow(name = "process_order")]24async fn process_order(ctx: WorkflowContext, input: OrderInput) -> Result<OrderOutput> {25 let order_id = input.order_id.clone();26 let charge_id: String = ctx.execute_task(charge_card, input).await?;27 Ok(OrderOutput {28 order_id,29 charge_id,30 })31}A workflow function is async, takes WorkflowContext first and exactly one input after it, and returns orcher_sdk::Result<T>. The input and output are serialized as JSON, so they need Serialize and Deserialize. Use a struct when a workflow needs several values, and () when it needs none:
1#[workflow]2async fn nightly_report(ctx: WorkflowContext, _input: ()) -> Result<u64> {3 let rows: u64 = ctx.execute_task(count_rows, ()).await?;4 Ok(rows)5}67#[task]8async fn count_rows(_ctx: TaskContext, _input: ()) -> Result<u64> {9 // Query your database here.10 Ok(42)11}#[workflow] attributes
| Attribute | Effect |
|---|---|
name | The workflow type that clients start. Defaults to the function name. |
timeout, version | No effect; each produces a deprecation warning. Set the deadline per start with StartWorkflowOptions::with_workflow_execution_timeout (Client), and the code release on the worker with WorkerBuilder::version_id or ORCHER_VERSION_ID (Worker). |
namespace | A compile error. A workflow runs in whatever namespace its worker serves; set the namespace on the worker and where workflows are started. |
The macro also accepts description, task_queue, max_concurrent, tags(...), retry_policy(...), cron, schedule and enabled, but in 0.7 the worker does not apply them. Choose the task queue, retry policy and cron schedule when you start the workflow, with StartWorkflowOptions (Client).
Executing Tasks
ctx.execute_task(task, input) schedules a task and waits for its result. Name the task with the constant #[task] generates, or with a string:
1#[workflow]2async fn charge_by_name(ctx: WorkflowContext, input: OrderInput) -> Result<String> {3 // A typed reference: a misspelled or renamed task fails to compile.4 // ctx.execute_task(charge_card, input)5 // A string name: checked only when the workflow runs. Useful for a task6 // that is registered by a worker built from another crate.7 let charge_id: String = ctx.execute_task("charge_card", input).await?;8 Ok(charge_id)9}The typed reference only checks the task's name. The input and output are serialized as JSON and checked when the task runs, so the output type you ask for (let x: T = ...) must match what the task returns. Name the output type even when it is (): execute_task is generic over it.
Tasks run on the workflow's task queue. A task's retry policy and timeouts come from its #[task] attributes, which the worker running the workflow reads, so link the task's crate into that worker. A task scheduled by string name that the worker doesn't know runs with the defaults: 3 attempts and a 300-second timeout. See Tasks.
Inline steps
For a one-off side effect that doesn't need its own task, ctx.execute(step_name, closure) runs an async closure on the workflow's worker and journals its result:
1#[workflow]2async fn tag_order(ctx: WorkflowContext, order_id: String) -> Result<String> {3 // Runs on this worker, inside the activation, and journals the result.4 // On replay the journaled value is returned and the closure is skipped.5 let label: String = ctx6 .execute("make_label", move || async move {7 // A one-off side effect, for example a call to an internal service.8 Ok(format!("label-{order_id}"))9 })10 .await?;11 Ok(label)12}execute_task(task, input) | execute("name", closure) | |
|---|---|---|
| Runs on | Any worker polling the queue | The worker running the workflow, during the activation |
| Retries | Under the task's retry policy | None; an error is returned to the workflow |
| Timeout | The task's timeout (300 s by default) | None |
| Reuse and testing | Shared by workflows, mockable in tests | Local to the workflow |
Both are at least once: a step that ran but whose result was never journaled, because a worker died at the wrong moment, runs again. Make side effects idempotent (Tasks).
Timers
1#[workflow]2async fn send_reminder(ctx: WorkflowContext, user_id: String) -> Result<()> {3 // A durable timer: the engine fires it, so it survives worker restarts.4 ctx.sleep(std::time::Duration::from_secs(24 * 60 * 60)).await?;56 // The same, with an id you choose instead of one taken from the step order.7 ctx.sleep_with_id("second_reminder", std::time::Duration::from_secs(60 * 60))8 .await?;910 ctx.execute_task(notify_user, user_id).await11}1213#[task]14async fn notify_user(_ctx: TaskContext, user_id: String) -> Result<()> {15 // Send the notification here.16 let _ = user_id;17 Ok(())18}ctx.sleep starts a durable timer that the engine fires, so it holds across worker restarts and no worker is busy while it waits. Its id comes from the step's position in the code; sleep_with_id lets you choose it.
Child Workflows
1use orcher_sdk::workflow::OrphanPolicy;23#[derive(Serialize, Deserialize)]4pub struct ShipmentInput {5 pub order_id: String,6}78#[workflow]9async fn ship_order(_ctx: WorkflowContext, input: ShipmentInput) -> Result<String> {10 Ok(format!("shipped {}", input.order_id))11}1213#[workflow]14async fn fulfill_order(ctx: WorkflowContext, order_id: String) -> Result<String> {15 // Start the child and wait for its result.16 let shipped: String = ctx17 .execute_child_workflow(18 "ship_order",19 ShipmentInput {20 order_id: order_id.clone(),21 },22 )23 .await?;2425 // Or start it with options, keep the handle, and wait later.26 let options = ChildWorkflowOptions::new()27 // Derive the child's id from the parent's so it is the same on replay28 // and unique per parent.29 .workflow_id(format!("{}-invoice", ctx.workflow_id()))30 .execution_timeout(std::time::Duration::from_secs(600))31 // Leave the child running if the parent closes first.32 .orphan_policy(OrphanPolicy::Abandon);33 let invoice = ctx34 .start_child_workflow_with_options("send_invoice", order_id, options)35 .await?;36 let invoice_id: String = invoice.result().await?;3738 Ok(format!("{shipped}; invoice {invoice_id}"))39}execute_child_workflow(type, input)starts a child and waits for its result.start_child_workflow(type, input)returns aChildWorkflowHandlestraight away, so you can start several children before waiting on any of them;handle.result()waits, and the handle also hassend_eventandcancel.ChildWorkflowOptionssets the child'sworkflow_id,execution_timeout,task_queue(the parent's by default) andorphan_policy: what happens to the child when the parent closes first.OrphanPolicy::Cancel(the default) asks the child to cancel,Terminateends it at once, andAbandonleaves it running.- Without a
workflow_id, the child's id is derived from its position in the parent's code. Derive it fromctx.workflow_id()as above when you want it to be unique and recognizable. - A child that fails, is cancelled, terminated or times out surfaces in the parent as
WorkflowError::ChildWorkflowFailed.
Events
Clients send events to a running workflow with handle.send_event(name, payload) (Client). The workflow receives them by name:
1#[derive(Serialize, Deserialize)]2pub struct Approval {3 pub approved: bool,4 pub approver: String,5}67#[workflow]8async fn expense_approval(ctx: WorkflowContext, expense_id: String) -> Result<String> {9 // Suspends until a client sends an "approval" event to this workflow.10 let first: Approval = ctx.wait_for_event("approval").await?;11 if !first.approved {12 return Ok(format!("{expense_id} rejected by {}", first.approver));13 }1415 // Waits at most 24 hours. The deadline is a durable timer; `None` means it16 // fired before the event arrived.17 let second: Option<Approval> = ctx18 .wait_for_event_with_timeout("approval", std::time::Duration::from_secs(24 * 60 * 60))19 .await?;2021 Ok(match second {22 Some(a) if a.approved => format!("{expense_id} approved"),23 Some(a) => format!("{expense_id} rejected by {}", a.approver),24 None => format!("{expense_id} expired"),25 })26}- Events of one name are delivered oldest first. An event that arrives before the workflow waits for it is kept until it does.
wait_for_event_with_timeoutreturnsOk(None)when the timeout passes first. The deadline is a durable timer, so it holds while no worker is running the workflow. An event that arrives after the deadline is kept for the next wait on that name.
Handling Failures
A task that fails for good, because its attempts are used up or it failed with a non-retryable error, is returned to the workflow as Error::Workflow(WorkflowError::TaskFailed { task_type, attempts, reason, .. }). The workflow can handle it or return it, which fails the workflow:
1use orcher_sdk::error::WorkflowError;23#[task(retry = 3)]4async fn reserve_stock(_ctx: TaskContext, order_id: String) -> Result<String> {5 // Call your inventory service here.6 Ok(format!("res_{order_id}"))7}89#[workflow]10async fn reserve_or_backorder(ctx: WorkflowContext, order_id: String) -> Result<String> {11 match ctx.execute_task(reserve_stock, order_id.clone()).await {12 Ok(reservation) => Ok(reservation),13 // The task failed for good: its retries are used up, or it failed with14 // a non-retryable error.15 Err(Error::Workflow(WorkflowError::TaskFailed {16 attempts, reason, ..17 })) => Ok(format!(18 "{order_id} backordered after {attempts} attempts: {reason}"19 )),20 // Everything else, including the suspension signal the SDK uses while21 // the task is still running, must be returned unchanged.22 Err(other) => Err(other),23 }24}While a task, timer, child or event is still pending, the step returns a suspension error: Error::Workflow(WorkflowError::Suspended { .. }), for which error.is_suspend() is true. It is how the SDK hands control back to the worker, and it must reach the worker unchanged. Match the errors you mean to handle and return everything else, as above. A catch-all arm such as Err(_) => ... treats every step that hasn't finished yet as failed, so a healthy workflow would take its failure path on its first activation.
A workflow that returns an error fails. It is not retried unless it was started with a retry policy (Client).
Queries, Updates and Workflow State
ctx.set_state and ctx.get_state keep JSON values by key for the current run. The state is rebuilt by replay, like everything else in the workflow, so it is a convenient place for values that query handlers report.
1#[derive(Serialize, Deserialize)]2pub struct Address {3 pub street: String,4 pub city: String,5}67#[workflow]8async fn delivery(ctx: WorkflowContext, order_id: String) -> Result<String> {9 ctx.set_state("phase", &"packing")?;1011 // A query reads state. Its handler takes no arguments, so capture what it12 // needs; a clone of the context shares the workflow's state.13 let reader = ctx.clone();14 ctx.register_query_handler("phase", move || {15 reader16 .get_state::<String>("phase")17 .ok()18 .flatten()19 .unwrap_or_default()20 });2122 // An update may change state; its result is journaled.23 let writer = ctx.clone();24 ctx.register_update_handler("change_address", move |address: Address| {25 let writer = writer.clone();26 async move {27 writer.set_state("address", &address)?;28 Ok(true)29 }30 });3132 // Name the result type, even when it is `()`: `execute_task` is generic33 // over it.34 let _: () = ctx.execute_task(pack_order, order_id.clone()).await?;35 ctx.set_state("phase", &"shipped")?;36 Ok(order_id)37}3839#[task]40async fn pack_order(_ctx: TaskContext, order_id: String) -> Result<()> {41 // Talk to your warehouse system here.42 let _ = order_id;43 Ok(())44}- A query handler is a synchronous
Fn() -> Rthat takes no arguments. Clients call it withhandle.query(name, ()). It reads; it should not change state. - An update handler is an async
Fn(I) -> Future<Output = Result<R>>. Clients call it withhandle.update(name, args)and wait for its result, which is journaled. - Every workflow also answers the built-in queries
workflow_id,workflow_type,run_id,is_replaying,attempt,version,namespace,task_queueandworkflow_info(all of these as one JSON object).
Determinism
Deterministic helpers
1#[derive(Serialize, Deserialize)]2pub struct Ticket {3 pub id: String,4 pub created_at: i64,5 pub priority: u8,6}78#[workflow]9async fn open_ticket(ctx: WorkflowContext, subject: String) -> Result<Ticket> {10 // Seeded from the workflow id: a replay draws the same values.11 let id = format!("T-{}", ctx.rand().uuid());12 let priority: u8 = ctx.rand().random_range(1..=5);1314 // Seconds since the UNIX epoch, read from the journal rather than the15 // clock: the same at this point on every replay.16 let created_at = ctx.time().now();1718 // Not these: a replay would see different values.19 // uuid::Uuid::new_v4(), rand::random(), std::time::SystemTime::now()2021 let _ = subject;22 Ok(Ticket {23 id,24 created_at,25 priority,26 })27}ctx.rand()is seeded from the workflow id and advances with each draw:uuid(),random()(a float in[0, 1)),random_range(range),bool(),choose(&items),choose_multiple(&items, n)andshuffle(&mut items).ctx.time().now()is seconds since the UNIX epoch, read from the journal: the workflow's start, then the moment the engine journaled each result, timer or event the workflow has received. It moves forward only as the workflow receives things, and is the same at each point on every replay.started_at(),elapsed(),elapsed_secs()andelapsed_millis()use the same clock.
Logging
The workflow function runs again on every replay, so a log line in it repeats. Log from tasks, or skip logging while replaying:
1#[workflow]2async fn audited(ctx: WorkflowContext, order_id: String) -> Result<String> {3 // Workflow code runs again on every replay, so a plain log line repeats.4 if !ctx.is_replaying() {5 println!("starting {} for {order_id}", ctx.workflow_id());6 }7 ctx.execute_task(reserve_stock, order_id).await8}What happens when code changes under a running workflow
Each activation, the worker reports which journaled steps (tasks, timers and child workflows) the code reached. If a run's code skips a step the journal recorded and then issues new work or finishes, or issues different work at a position the journal already has, the activation is held: it is not applied, the run stays open with its journal unchanged, and the worker logs an error that names the workflow and run. The activation is handed back and tried again after 2 seconds, doubling up to 1 minute, until a worker whose code replays the journal picks it up.
Not every change is caught. Reordering steps of different kinds (a task and a timer swapping places) and changing a timer's duration replay without error, against a journal that no longer matches the code.
To change a workflow safely:
- Let in-flight runs finish before deploying an incompatible change, or run the new code under a new workflow type (
#[workflow(name = "process_order_v2")]) beside the old one. - Tag each worker with the code release it runs (
WorkerBuilder::version_id), which the server records on each execution. ctx.version()is a metadata label only. Don't branch on it.
See Determinism for the concepts.
Restarting with a Fresh History
Every step a workflow takes is added to its journal, and replay reads all of it. A workflow that loops for a long time should restart itself now and then:
1#[derive(Serialize, Deserialize)]2pub struct PollState {3 pub cursor: u64,4}56#[task]7async fn sync_batch(_ctx: TaskContext, cursor: u64) -> Result<u64> {8 // Copy one batch from the source system and return the next cursor.9 Ok(cursor + 100)10}1112#[workflow]13async fn sync_forever(ctx: WorkflowContext, state: PollState) -> Result<()> {14 let mut cursor = state.cursor;15 for _ in 0..100 {16 cursor = ctx.execute_task(sync_batch, cursor).await?;17 ctx.sleep(std::time::Duration::from_secs(60)).await?;18 }1920 // Start a new run with the same workflow id, an empty history and this21 // input. Return right after it.22 ctx.restart_fresh_with(PollState { cursor }, None)23}ctx.restart_fresh() ends the run and starts a new one with the same workflow id, an empty history and no input. restart_fresh_with(input, options) passes an input, and its RestartFreshOptions can change the workflow_type, task_queue or timeout of the new run; changing the type moves a long-running workflow onto new code. Return straight after calling either. A client waiting on the original run's result receives the result of the run that finally completes.
Payload Size
Each payload, a workflow or task input or result, an event or an update, is limited by the engine to 8 MiB by default; the operator can change the limit. A task whose result is too large to send fails with the error type PayloadTooLarge, which is never retried. Every activation also carries the payloads the workflow has received, and a gRPC message is limited to 32 MiB by default (ORCHER_MAX_MESSAGE_BYTES on the worker, and the engine's own setting). Store large data elsewhere and pass a reference (Patterns).
Workflow Metadata
1#[workflow]2async fn describe_self(ctx: WorkflowContext, _input: ()) -> Result<String> {3 Ok(format!(4 "{} run {} of type {} in {} on {} (attempt {})",5 ctx.workflow_id(),6 ctx.run_id(),7 ctx.workflow_type(),8 ctx.namespace(),9 ctx.task_queue(),10 ctx.attempt(),11 ))12}ctx.attempt() counts from 1 and increases when a workflow started with a retry policy runs again. ctx.is_replaying() is true while the activation is replaying journaled steps.
WorkflowContext Reference
| Method | Description |
|---|---|
execute_task(task, input) | Run a task and wait for its result |
execute(name, closure) | Run an inline step on this worker and journal its result |
sleep(duration) / sleep_with_id(id, duration) | Durable timer |
wait_for_event(name) | Wait for the next event with this name |
wait_for_event_with_timeout(name, duration) | Same, or None after the timeout |
execute_child_workflow(type, input) | Start a child workflow and wait for its result |
start_child_workflow(type, input) | Start a child workflow and return a handle |
execute_child_workflow_with_options / start_child_workflow_with_options | The same with ChildWorkflowOptions |
create_session(options) | Pin tasks to one worker (Sessions) |
set_state(key, &value) / get_state(key) | Per-run state, as JSON |
register_query_handler(name, handler) | Answer a query |
register_update_handler(name, handler) | Accept an update |
rand() / time() | Deterministic randomness and time |
restart_fresh() / restart_fresh_with(input, options) | End the run and start a fresh one |
workflow_id(), run_id(), workflow_type(), namespace(), task_queue(), attempt() | Metadata |
is_replaying() | Whether the activation is replaying journaled steps |
version() | Metadata label; don't branch on it |