Reference

Tasks


A task is a unit of work a workflow schedules and a worker runs. Unlike workflow code, a task can do anything: call other services, write to a database, send an email. The engine journals its result, so a replayed workflow reuses the result instead of running the task again.

There are two ways to define tasks:

  • Functions decorated with @task.
  • Methods decorated with @task on a class decorated with @tasks, when the tasks share dependencies such as clients or connection pools.

Defining a task

The first parameter is the TaskContext; the rest are the task's inputs, which workflows pass as keyword arguments:

1
@task(
2
name="charge-order", # required: the name workflows and the engine use
3
timeout=30.0, # seconds one attempt may run (default 300)
4
heartbeat_timeout=20.0, # seconds without a heartbeat before the attempt is lost
5
retry_policy=RetryPolicy(
6
max_attempts=5, # attempts in all, the first included
7
initial_interval=timedelta(seconds=1),
8
max_interval=timedelta(seconds=30),
9
backoff_coefficient=2.0,
10
),
11
)
12
async def charge_card(ctx: TaskContext, order_id: str, amount: int) -> dict:
13
# Call your payment provider here.
14
return {"charge_id": f"ch_{order_id}", "amount": amount}
OptionDefaultDescription
namerequiredThe task's name, unique in the worker
retry_policyengine defaultA RetryPolicy, or a dict (see Retries)
retrynoneShorthand for RetryPolicy(max_attempts=retry); ignored when retry_policy is given
timeout300 secondsHow long one attempt may run, in seconds
heartbeat_timeoutengine default, 60 secondsHow long an attempt may go without a heartbeat, in seconds

A workflow can override retry_policy, timeout and heartbeat_timeout for a single call; see per-call options.

1
@task(name="send-receipt", retry=3) # same as RetryPolicy(max_attempts=3)
2
async def send_receipt(ctx: TaskContext, email: str, charge_id: str) -> bool:
3
# Send the email here.
4
return True

A plain (non-async) function is a task too. The worker runs it in a thread pool, so blocking calls don't stall the worker's event loop; for CPU-bound work, hand the worker a process pool with task_executor().

1
@task(name="resize-image")
2
def resize_image(ctx: TaskContext, path: str, width: int) -> str:
3
# A plain function runs in the worker's thread pool, so blocking calls
4
# don't stall the event loop.
5
return f"{path}@{width}"

Task classes

@tasks on a class registers each of its @task methods. Construct the class with its dependencies:

1
@tasks
2
class PaymentTasks:
3
def __init__(self, gateway: PaymentGateway) -> None:
4
self.gateway = gateway
5
6
@task(name="charge-card", timeout=30.0)
7
async def charge_card(self, ctx: TaskContext, amount: int, token: str) -> dict:
8
key = f"{ctx.info.workflow_id}:{ctx.task_id}"
9
receipt = await self.gateway.charge(amount, token, idempotency_key=key)
10
return {"charge_id": receipt.id}
11
12
@task(name="refund", retry=5)
13
async def refund(self, ctx: TaskContext, charge_id: str) -> dict:
14
receipt = await self.gateway.refund(charge_id)
15
return {"refund_id": receipt.id}
16
17
def _helper(self) -> str:
18
# Not a task: only @task methods are registered.
19
return "helper"

In a workflow, refer to a method through the class:

1
@workflow(name="checkout")
2
async def checkout(ctx: WorkflowContext, total: int, card_token: str) -> str:
3
charge = await ctx.execute_task(PaymentTasks.charge_card, amount=total, token=card_token)
4
return str(charge["charge_id"])

Give the worker the instance to run the tasks on:

1
async def main() -> None:
2
worker = (
3
Worker.builder()
4
.server_url("http://localhost:50051")
5
.task_queue("payments")
6
.build()
7
)
8
# The worker runs PaymentTasks' tasks on this instance.
9
worker.register_task_instance(PaymentTasks(PaymentGateway(api_key="...")))
10
await worker.run()

A class whose constructor takes no arguments needs no instance: the worker creates one. A class that needs arguments and has no registered instance stops the worker at startup with an error, rather than failing its first task.


Tasks run at least once

A task can run more than once: the engine retries an attempt that failed, and an attempt whose worker died is timed out and run again. The workflow still sees one result, because only a completed task's result is journaled, but the side effects of an attempt that didn't complete may already have happened.

So make tasks idempotent. Most services accept an idempotency key; build one that is the same for every attempt of the same call, such as the workflow id and the task id (both stable across retries), as charge_card above does:

1
key = f"{ctx.info.workflow_id}:{ctx.task_id}"

Don't generate the key inside the task with uuid4() or the clock: a retry would get a new one.


Retries

A RetryPolicy has these fields:

FieldDefaultDescription
max_attempts3Attempts in all, the first included
initial_intervaltimedelta(seconds=1)Wait before the second attempt
max_intervaltimedelta(seconds=60)Cap on the wait between attempts
backoff_coefficient2.0Multiplier applied to the wait after each attempt
non_retryable_error_types[]Exception class names that are never retried

The engine honors whole-second intervals and rounds sub-second ones down. A task with no policy gets the engine's default.

Errors that shouldn't be retried

A failure's type is the exception's class name (CardDeclined, not payments.CardDeclined). List the types never to retry in the policy, or set non_retryable = True on the exception class (or on one instance before raising it):

1
class CardDeclined(Exception):
2
pass
3
4
5
class AccountClosed(Exception):
6
non_retryable = True # never retried, whatever the policy says
7
8
9
@task(
10
name="charge",
11
retry_policy=RetryPolicy(max_attempts=5, non_retryable_error_types=["CardDeclined"]),
12
)
13
async def charge(ctx: TaskContext, order_id: str, declined: bool, closed: bool) -> str:
14
if closed:
15
raise AccountClosed(f"account closed for {order_id}") # runs once
16
if declined:
17
raise CardDeclined(f"card declined for {order_id}") # runs once
18
# Any other exception is retried, up to 5 attempts in all.
19
return f"charged {order_id}"

When a task fails for good, the workflow's execute_task call raises TaskError.

Attempts

ctx.attempt counts from 1, and ctx.is_retry is True from the second attempt on:

1
@task(name="sync-account", retry=4)
2
async def sync_account(ctx: TaskContext, account_id: str) -> str:
3
if ctx.is_retry: # ctx.attempt > 1; attempts count from 1
4
ctx.logger.warning("retrying %s, attempt %d", account_id, ctx.attempt)
5
# Call the remote system here.
6
return account_id

A dict policy uses seconds and non_retryable_errors:

1
@task(
2
name="poll-status",
3
retry_policy={
4
"max_attempts": 10,
5
"initial_interval_seconds": 5,
6
"max_interval_seconds": 60,
7
"backoff_coefficient": 1.5,
8
"non_retryable_errors": ["NotFound"],
9
},
10
)
11
async def poll_status(ctx: TaskContext, job_id: str) -> str:
12
return "done"

Heartbeats

The worker heartbeats every task it runs, on its own, for as long as the task runs. A long task needs no heartbeat code to stay alive, and if its worker dies the heartbeats stop, the attempt times out after its heartbeat timeout, and the engine runs it again.

ctx.heartbeat(details) is for reporting progress. It never waits, and however often you call it, no more heartbeats are sent than the worker's own; the latest details go with the next one:

1
@task(name="reindex", timeout=3600.0)
2
async def reindex(ctx: TaskContext, doc_ids: list[str]) -> int:
3
for done, doc_id in enumerate(doc_ids, start=1):
4
await index_document(doc_id)
5
# Optional: the worker heartbeats the task anyway. This attaches progress.
6
await ctx.heartbeat({"done": done, "total": len(doc_ids)})
7
return len(doc_ids)

Cancellation

When the workflow is canceled, or the attempt is no longer wanted, the engine answers the worker's next heartbeat with a request to stop, and the task's cancellation token is set. Stop at a convenient point by raising:

1
@task(name="export", timeout=3600.0)
2
async def export(ctx: TaskContext, doc_ids: list[str]) -> int:
3
exported = 0
4
for doc_id in doc_ids:
5
# Raises TaskError once the engine has asked the task to stop.
6
ctx.cancellation_token.raise_if_cancelled()
7
await index_document(doc_id)
8
exported += 1
9
return exported

Or by checking the flag:

1
@task(name="export-partial", timeout=3600.0)
2
async def export_partial(ctx: TaskContext, doc_ids: list[str]) -> int:
3
exported = 0
4
for doc_id in doc_ids:
5
if ctx.cancellation_token.is_cancelled:
6
break # stop early, and report what was done
7
await index_document(doc_id)
8
exported += 1
9
return exported

The token only signals; a task that never checks it runs to the end.


Inputs, results and size limits

Inputs and results travel as JSON. A task can return a dict, list, string, number, boolean, None or a dataclass (sent as a dict).

The engine accepts a single payload (one input, result or event) of up to 8 MiB unless it's configured otherwise. A task whose result is larger fails at once with a PayloadTooLarge failure that says how large it was, and it isn't retried. Store large data elsewhere, such as an object store, and pass a reference. Workers and clients send and receive gRPC messages of up to 32 MiB; set ORCHER_MAX_MESSAGE_BYTES to change that.


Task metadata

1
@task(name="audit")
2
async def audit(ctx: TaskContext, entry: str) -> dict:
3
info = ctx.info
4
ctx.logger.info("auditing %s", entry) # carries task_id, workflow_id and attempt
5
return {
6
"task_id": ctx.task_id, # stable across retries of this call
7
"task_type": ctx.task_type, # the name from @task
8
"attempt": ctx.attempt, # 1 on the first attempt
9
"workflow_id": info.workflow_id,
10
"run_id": info.run_id,
11
"queue": info.task_queue,
12
"heartbeat_timeout": str(ctx.heartbeat_timeout),
13
}

TaskContext reference

Method or propertyDescription
await ctx.heartbeat(details=None)Attach progress details to the next heartbeat
ctx.cancellation_tokenis_cancelled, raise_if_cancelled(), await wait_for_cancellation()
ctx.task_idThe id of this call, the same on every attempt
ctx.task_typeThe task's name
ctx.attemptThe attempt number, from 1
ctx.is_retryTrue when attempt > 1
ctx.heartbeat_timeoutThe heartbeat timeout this attempt runs under, or None
ctx.start_to_close_timeoutHow long this attempt may run, or None
ctx.loggerA logging.LoggerAdapter carrying task_id, task_type, workflow_id and attempt
ctx.infoTaskInfo: task_id, task_type, workflow_id, run_id, task_queue, namespace, attempt, scheduled_at, started_at, heartbeat_timeout, start_to_close_timeout

Next steps

  • Workflows: schedule tasks and handle their failures
  • Testing: mock tasks in workflow tests
  • Patterns: compensation and batching