Tasks
A task is a unit of work a workflow schedules and a worker runs. Unlike workflow code, a task can do anything: call other services, write to a database, send an email. The engine journals its result, so a replayed workflow reuses the result instead of running the task again.
There are two ways to define tasks:
- Functions decorated with
@task. - Methods decorated with
@taskon a class decorated with@tasks, when the tasks share dependencies such as clients or connection pools.
Defining a task
The first parameter is the TaskContext; the rest are the task's inputs, which workflows pass as keyword arguments:
1@task(2 name="charge-order", # required: the name workflows and the engine use3 timeout=30.0, # seconds one attempt may run (default 300)4 heartbeat_timeout=20.0, # seconds without a heartbeat before the attempt is lost5 retry_policy=RetryPolicy(6 max_attempts=5, # attempts in all, the first included7 initial_interval=timedelta(seconds=1),8 max_interval=timedelta(seconds=30),9 backoff_coefficient=2.0,10 ),11)12async def charge_card(ctx: TaskContext, order_id: str, amount: int) -> dict:13 # Call your payment provider here.14 return {"charge_id": f"ch_{order_id}", "amount": amount}| Option | Default | Description |
|---|---|---|
name | required | The task's name, unique in the worker |
retry_policy | engine default | A RetryPolicy, or a dict (see Retries) |
retry | none | Shorthand for RetryPolicy(max_attempts=retry); ignored when retry_policy is given |
timeout | 300 seconds | How long one attempt may run, in seconds |
heartbeat_timeout | engine default, 60 seconds | How long an attempt may go without a heartbeat, in seconds |
A workflow can override retry_policy, timeout and heartbeat_timeout for a single call; see per-call options.
1@task(name="send-receipt", retry=3) # same as RetryPolicy(max_attempts=3)2async def send_receipt(ctx: TaskContext, email: str, charge_id: str) -> bool:3 # Send the email here.4 return TrueA plain (non-async) function is a task too. The worker runs it in a thread pool, so blocking calls don't stall the worker's event loop; for CPU-bound work, hand the worker a process pool with task_executor().
1@task(name="resize-image")2def resize_image(ctx: TaskContext, path: str, width: int) -> str:3 # A plain function runs in the worker's thread pool, so blocking calls4 # don't stall the event loop.5 return f"{path}@{width}"Task classes
@tasks on a class registers each of its @task methods. Construct the class with its dependencies:
1@tasks2class PaymentTasks:3 def __init__(self, gateway: PaymentGateway) -> None:4 self.gateway = gateway56 @task(name="charge-card", timeout=30.0)7 async def charge_card(self, ctx: TaskContext, amount: int, token: str) -> dict:8 key = f"{ctx.info.workflow_id}:{ctx.task_id}"9 receipt = await self.gateway.charge(amount, token, idempotency_key=key)10 return {"charge_id": receipt.id}1112 @task(name="refund", retry=5)13 async def refund(self, ctx: TaskContext, charge_id: str) -> dict:14 receipt = await self.gateway.refund(charge_id)15 return {"refund_id": receipt.id}1617 def _helper(self) -> str:18 # Not a task: only @task methods are registered.19 return "helper"In a workflow, refer to a method through the class:
1@workflow(name="checkout")2async def checkout(ctx: WorkflowContext, total: int, card_token: str) -> str:3 charge = await ctx.execute_task(PaymentTasks.charge_card, amount=total, token=card_token)4 return str(charge["charge_id"])Give the worker the instance to run the tasks on:
1async def main() -> None:2 worker = (3 Worker.builder()4 .server_url("http://localhost:50051")5 .task_queue("payments")6 .build()7 )8 # The worker runs PaymentTasks' tasks on this instance.9 worker.register_task_instance(PaymentTasks(PaymentGateway(api_key="...")))10 await worker.run()A class whose constructor takes no arguments needs no instance: the worker creates one. A class that needs arguments and has no registered instance stops the worker at startup with an error, rather than failing its first task.
Tasks run at least once
A task can run more than once: the engine retries an attempt that failed, and an attempt whose worker died is timed out and run again. The workflow still sees one result, because only a completed task's result is journaled, but the side effects of an attempt that didn't complete may already have happened.
So make tasks idempotent. Most services accept an idempotency key; build one that is the same for every attempt of the same call, such as the workflow id and the task id (both stable across retries), as charge_card above does:
1key = f"{ctx.info.workflow_id}:{ctx.task_id}"Don't generate the key inside the task with uuid4() or the clock: a retry would get a new one.
Retries
A RetryPolicy has these fields:
| Field | Default | Description |
|---|---|---|
max_attempts | 3 | Attempts in all, the first included |
initial_interval | timedelta(seconds=1) | Wait before the second attempt |
max_interval | timedelta(seconds=60) | Cap on the wait between attempts |
backoff_coefficient | 2.0 | Multiplier applied to the wait after each attempt |
non_retryable_error_types | [] | Exception class names that are never retried |
The engine honors whole-second intervals and rounds sub-second ones down. A task with no policy gets the engine's default.
Errors that shouldn't be retried
A failure's type is the exception's class name (CardDeclined, not payments.CardDeclined). List the types never to retry in the policy, or set non_retryable = True on the exception class (or on one instance before raising it):
1class CardDeclined(Exception):2 pass345class AccountClosed(Exception):6 non_retryable = True # never retried, whatever the policy says789@task(10 name="charge",11 retry_policy=RetryPolicy(max_attempts=5, non_retryable_error_types=["CardDeclined"]),12)13async def charge(ctx: TaskContext, order_id: str, declined: bool, closed: bool) -> str:14 if closed:15 raise AccountClosed(f"account closed for {order_id}") # runs once16 if declined:17 raise CardDeclined(f"card declined for {order_id}") # runs once18 # Any other exception is retried, up to 5 attempts in all.19 return f"charged {order_id}"When a task fails for good, the workflow's execute_task call raises TaskError.
Attempts
ctx.attempt counts from 1, and ctx.is_retry is True from the second attempt on:
1@task(name="sync-account", retry=4)2async def sync_account(ctx: TaskContext, account_id: str) -> str:3 if ctx.is_retry: # ctx.attempt > 1; attempts count from 14 ctx.logger.warning("retrying %s, attempt %d", account_id, ctx.attempt)5 # Call the remote system here.6 return account_idA dict policy uses seconds and non_retryable_errors:
1@task(2 name="poll-status",3 retry_policy={4 "max_attempts": 10,5 "initial_interval_seconds": 5,6 "max_interval_seconds": 60,7 "backoff_coefficient": 1.5,8 "non_retryable_errors": ["NotFound"],9 },10)11async def poll_status(ctx: TaskContext, job_id: str) -> str:12 return "done"Heartbeats
The worker heartbeats every task it runs, on its own, for as long as the task runs. A long task needs no heartbeat code to stay alive, and if its worker dies the heartbeats stop, the attempt times out after its heartbeat timeout, and the engine runs it again.
ctx.heartbeat(details) is for reporting progress. It never waits, and however often you call it, no more heartbeats are sent than the worker's own; the latest details go with the next one:
1@task(name="reindex", timeout=3600.0)2async def reindex(ctx: TaskContext, doc_ids: list[str]) -> int:3 for done, doc_id in enumerate(doc_ids, start=1):4 await index_document(doc_id)5 # Optional: the worker heartbeats the task anyway. This attaches progress.6 await ctx.heartbeat({"done": done, "total": len(doc_ids)})7 return len(doc_ids)Cancellation
When the workflow is canceled, or the attempt is no longer wanted, the engine answers the worker's next heartbeat with a request to stop, and the task's cancellation token is set. Stop at a convenient point by raising:
1@task(name="export", timeout=3600.0)2async def export(ctx: TaskContext, doc_ids: list[str]) -> int:3 exported = 04 for doc_id in doc_ids:5 # Raises TaskError once the engine has asked the task to stop.6 ctx.cancellation_token.raise_if_cancelled()7 await index_document(doc_id)8 exported += 19 return exportedOr by checking the flag:
1@task(name="export-partial", timeout=3600.0)2async def export_partial(ctx: TaskContext, doc_ids: list[str]) -> int:3 exported = 04 for doc_id in doc_ids:5 if ctx.cancellation_token.is_cancelled:6 break # stop early, and report what was done7 await index_document(doc_id)8 exported += 19 return exportedThe token only signals; a task that never checks it runs to the end.
Inputs, results and size limits
Inputs and results travel as JSON. A task can return a dict, list, string, number, boolean, None or a dataclass (sent as a dict).
The engine accepts a single payload (one input, result or event) of up to 8 MiB unless it's configured otherwise. A task whose result is larger fails at once with a PayloadTooLarge failure that says how large it was, and it isn't retried. Store large data elsewhere, such as an object store, and pass a reference. Workers and clients send and receive gRPC messages of up to 32 MiB; set ORCHER_MAX_MESSAGE_BYTES to change that.
Task metadata
1@task(name="audit")2async def audit(ctx: TaskContext, entry: str) -> dict:3 info = ctx.info4 ctx.logger.info("auditing %s", entry) # carries task_id, workflow_id and attempt5 return {6 "task_id": ctx.task_id, # stable across retries of this call7 "task_type": ctx.task_type, # the name from @task8 "attempt": ctx.attempt, # 1 on the first attempt9 "workflow_id": info.workflow_id,10 "run_id": info.run_id,11 "queue": info.task_queue,12 "heartbeat_timeout": str(ctx.heartbeat_timeout),13 }TaskContext reference
| Method or property | Description |
|---|---|
await ctx.heartbeat(details=None) | Attach progress details to the next heartbeat |
ctx.cancellation_token | is_cancelled, raise_if_cancelled(), await wait_for_cancellation() |
ctx.task_id | The id of this call, the same on every attempt |
ctx.task_type | The task's name |
ctx.attempt | The attempt number, from 1 |
ctx.is_retry | True when attempt > 1 |
ctx.heartbeat_timeout | The heartbeat timeout this attempt runs under, or None |
ctx.start_to_close_timeout | How long this attempt may run, or None |
ctx.logger | A logging.LoggerAdapter carrying task_id, task_type, workflow_id and attempt |
ctx.info | TaskInfo: task_id, task_type, workflow_id, run_id, task_queue, namespace, attempt, scheduled_at, started_at, heartbeat_timeout, start_to_close_timeout |