Reference

Patterns


Workflow shapes that come up again and again, written with the Python SDK. Each one relies on the same guarantee: a step the journal has recorded is not repeated when the workflow resumes.


Compensation (saga)

When a workflow changes several systems and a later step fails, undo the earlier steps. Saga collects a compensation after each step that succeeds and runs them newest first:

1
@workflow(name="book-trip")
2
async def book_trip(ctx: WorkflowContext, hotel_id: str, flight_id: str, amount: int, token: str) -> dict:
3
saga = Saga()
4
try:
5
hotel = await ctx.execute_task(book_hotel, hotel_id=hotel_id)
6
saga.add_compensation(lambda: ctx.execute_task(cancel_hotel, reservation_id=hotel["id"]))
7
8
flight = await ctx.execute_task(book_flight, flight_id=flight_id)
9
saga.add_compensation(lambda: ctx.execute_task(cancel_flight, reservation_id=flight["id"]))
10
11
payment = await ctx.execute_task(charge_payment, amount=amount, token=token)
12
saga.add_compensation(lambda: ctx.execute_task(refund_payment, payment_id=payment["id"]))
13
except Exception:
14
# Undo what succeeded, newest first, then fail the workflow.
15
await saga.compensate()
16
raise
17
18
saga.commit()
19
return {"hotel": hotel["id"], "flight": flight["id"], "payment": payment["id"]}
  • Compensations are tasks, so they are journaled and retried like any task. A compensation that still fails is logged and the rest still run.
  • Register a compensation after its step succeeds, so it can use the step's result (the reservation id here). saga.add_step(action, compensation) runs an action and registers its compensation in one call when the compensation doesn't need the result.
  • except Exception is safe here: the signal the SDK uses to suspend a workflow is not an Exception, so a workflow that is merely waiting for a task never compensates.

Fan-out and fan-in

To run independent tasks at the same time, schedule them together with asyncio.gather. A loop of awaits would run them one after another:

1
@workflow(name="fetch-all")
2
async def fetch_all(ctx: WorkflowContext, urls: list[str]) -> list[dict]:
3
# Every task is scheduled before the workflow waits on any of them.
4
results = await asyncio.gather(*(ctx.execute_task(fetch_data, url=url) for url in urls))
5
return list(results)

For larger units of work, fan out to child workflows. start_child_workflow doesn't wait, so all the children start before the parent waits on any of them:

1
@workflow(name="process-regions")
2
async def process_regions(ctx: WorkflowContext, regions: list[str]) -> dict:
3
handles = [
4
await ctx.start_child_workflow("process-region", workflow_id=f"region-{r}", args=(r,))
5
for r in regions
6
]
7
totals = [await handle.result() for handle in handles]
8
return {"regions": len(regions), "total": sum(totals)}

If one fails, handle.result() raises WorkflowError for it.


Waiting for a person

A workflow can wait days for a decision without holding a worker. Ask, wait for an event with a deadline, and escalate if none comes:

1
@workflow(name="expense-approval")
2
async def expense_approval(ctx: WorkflowContext, request_id: str, approver: str) -> str:
3
await ctx.execute_task(send_approval_request, approver=approver, request_id=request_id)
4
5
try:
6
decision = await ctx.wait_for_event_with_timeout("decision", timedelta(days=1))
7
except TimeoutError:
8
await ctx.execute_task(escalate, request_id=request_id)
9
return "escalated"
10
11
return "approved" if decision.get("approved") else "rejected"

The approver's tool sends the decision with handle.send_event("decision", {"approved": True}).


State machine

A loop over an explicit state keeps a long, multi-stage process readable, and a query handler makes the current state visible from outside:

1
class OrderState(StrEnum):
2
CREATED = "created"
3
PAYMENT_PENDING = "payment_pending"
4
PAID = "paid"
5
SHIPPED = "shipped"
6
DELIVERED = "delivered"
7
CANCELLED = "cancelled"
8
9
10
@workflow(name="order-lifecycle")
11
async def order_lifecycle(ctx: WorkflowContext, order_id: str, amount: int, token: str) -> str:
12
state = OrderState.CREATED
13
ctx.register_query_handler("state", lambda qctx: qctx.get_state("state"))
14
15
while state not in (OrderState.DELIVERED, OrderState.CANCELLED):
16
ctx.set_state("state", state.value)
17
18
if state == OrderState.CREATED:
19
await ctx.execute_task(validate_order, order_id=order_id)
20
state = OrderState.PAYMENT_PENDING
21
elif state == OrderState.PAYMENT_PENDING:
22
payment = await ctx.execute_task(charge_payment, amount=amount, token=token)
23
state = OrderState.PAID if payment["success"] else OrderState.CANCELLED
24
elif state == OrderState.PAID:
25
await ctx.execute_task(ship_order, order_id=order_id)
26
state = OrderState.SHIPPED
27
elif state == OrderState.SHIPPED:
28
await ctx.wait_for_event("delivered")
29
state = OrderState.DELIVERED
30
31
ctx.set_state("state", state.value)
32
return state.value

Batches with progress

Process items one task at a time, record progress, and expose it through a query. A failed item is caught as TaskError so the rest of the batch carries on:

1
@workflow(name="import-records")
2
async def import_records(ctx: WorkflowContext, record_ids: list[str]) -> dict:
3
progress = {"total": len(record_ids), "done": 0, "failed": 0}
4
ctx.register_query_handler("progress", lambda qctx: progress)
5
6
for record_id in record_ids:
7
try:
8
await ctx.execute_task(process_record, record_id=record_id)
9
progress["done"] += 1
10
except TaskError:
11
progress["failed"] += 1 # retries are used up; carry on with the rest
12
13
return progress

One task per item gives each item its own retries and its own journal entry. For many small items, a task per chunk keeps the journal short. Each task's input and result must stay under the engine's 8 MiB payload limit:

1
@workflow(name="import-records-in-chunks")
2
async def import_records_in_chunks(ctx: WorkflowContext, record_ids: list[str]) -> int:
3
imported = 0
4
# One task per chunk keeps the journal short; keep each input well under 8 MiB.
5
for start in range(0, len(record_ids), 500):
6
imported += await ctx.execute_task(process_records, record_ids=record_ids[start:start + 500])
7
return imported

For a batch with no end, such as a sync that runs forever, process one page per run and restart with fresh history so the journal stays small.


Idempotent side effects

Tasks run at least once: a retried or reassigned attempt may repeat side effects that a previous attempt already made. Pass the services you call an idempotency key that stays the same across attempts; see Tasks run at least once.

Starting a workflow is idempotent through its id: give each piece of work a workflow id derived from it (an order id, a webhook delivery id), and a repeated start is refused rather than run twice. See duplicate starts.


Next steps

  • Workflows: the WorkflowContext API
  • Tasks: retries and idempotency
  • Testing: test these workflows in memory