Reference

Worker Configuration


A Worker is the process that runs your code. It connects to the engine over gRPC, registers the workflows and tasks it knows, and polls one task queue for work to run.


Creating a worker

1
import asyncio
2
import os
3
import signal
4
from datetime import timedelta
5
6
from orcher import Worker
7
8
import handlers # noqa: F401 (importing registers its @workflow and @task)
9
10
11
async def main() -> None:
12
worker = (
13
Worker.builder()
14
.server_url(os.environ.get("ORCHER_URL", "http://localhost:50051"))
15
.namespace("default")
16
.task_queue("orders")
17
.identity("orders-worker-1") # default: host, pid and a random suffix
18
.version_id(os.environ.get("GIT_SHA", "dev")) # the release this code is
19
.max_concurrent_workflows(100)
20
.max_concurrent_tasks(100)
21
.shutdown_grace_time(timedelta(seconds=30))
22
.build()
23
)
24
25
loop = asyncio.get_running_loop()
26
for sig in (signal.SIGTERM, signal.SIGINT):
27
loop.add_signal_handler(sig, lambda: asyncio.create_task(worker.shutdown()))
28
29
await worker.run() # returns once shutdown has finished
30
31
32
if __name__ == "__main__":
33
asyncio.run(main())

server_url and task_queue are required; build() raises ValueError without them. run() starts polling and returns once the worker has shut down.

Registering workflows and tasks

@workflow, @task, @tasks and @actor register what they decorate when their module is imported, and the worker picks up everything registered by the time build() is called. So import your modules before building the worker, as import handlers does above. Each name can be registered once per process; registering it twice raises ValueError.

The builder can also import modules for you:

1
def build_worker() -> Worker:
2
return (
3
Worker.builder()
4
.server_url("http://localhost:50051")
5
.task_queue("orders")
6
# Imports every matching file under ./app before the worker is built.
7
.auto_discover(["./app"], exclude=["**/test_*.py", "**/*_test.py"])
8
.build()
9
)

auto_discover() imports each matching file under a generated module name, and silently skips any file that fails to import. A file that is also imported the normal way (by a workflow module that imports its tasks, say) registers its names twice, and whichever import comes second fails, so its workflows go missing without an error. Explicit imports avoid both problems.


Builder reference

MethodDefaultDescription
.server_url(url)requiredThe engine's gRPC address
.task_queue(name)requiredThe queue this worker polls
.namespace(name)"default"The namespace to work in
.api_key(key)ORCHER_API_KEYAPI key for an engine that requires one; None or blank sends none
.tls(ca_cert_path=, client_cert_path=, client_key_path=)noneCertificates for a private CA or mutual TLS; see Authentication and TLS
.version_id(id)ORCHER_VERSION_IDThe release this worker runs; see Versions
.identity(name)host, pid and a random suffixHow the worker is named to the engine
.max_concurrent_workflows(n)100Workflow activations run at once
.max_concurrent_tasks(n)100Tasks run at once; also the size of the default thread pool for plain-function tasks
.max_concurrent_actor_operations(n)100Actor operations run at once
.workflow_poller_count(n)4Concurrent workflow pollers
.task_poller_count(n)4Concurrent task pollers
.actor_poller_count(n)4Concurrent actor pollers, started only if the worker has actors
.shutdown_grace_time(timedelta)30 secondsHow long shutdown waits for in-flight work
.organization_id(id)noneSent as the X-Organization-Id header on every request
.task_executor(executor)a thread poolWhere plain-function tasks run; pass a ProcessPoolExecutor for CPU-bound work
.interceptor(i)noneAdd an interceptor; see Interceptors
.workflow_interceptor(i) / .task_interceptor(i)noneAdd an interceptor to one chain only
.auto_discover(paths, *, patterns, exclude)offImport matching files before building

Durations accept a timedelta, or an int of milliseconds. .workflow_poll_interval(), .task_poll_interval(), .force_shutdown_timeout() and .binary_checksum() are accepted but have no effect on how the worker runs; binary_checksum is only reported back by worker.get_identity().

From the environment

WorkerConfig.from_env() builds the same configuration from environment variables:

1
from orcher import Worker, WorkerConfig
2
3
import handlers # noqa: F401
4
5
6
async def main() -> None:
7
# Reads ORCHER_SERVER_URL and ORCHER_TASK_QUEUE (required), plus
8
# ORCHER_NAMESPACE, ORCHER_API_KEY, ORCHER_VERSION_ID, the ORCHER_TLS_*
9
# certificate paths and the rest.
10
worker = Worker(WorkerConfig.from_env())
11
await worker.run()

It reads ORCHER_SERVER_URL and ORCHER_TASK_QUEUE (both required), and optionally ORCHER_NAMESPACE, ORCHER_API_KEY, ORCHER_TLS_CA_CERT_PATH, ORCHER_TLS_CLIENT_CERT_PATH, ORCHER_TLS_CLIENT_KEY_PATH, ORCHER_VERSION_ID, ORCHER_IDENTITY, ORCHER_ORGANIZATION_ID, ORCHER_MAX_CONCURRENT_WORKFLOWS, ORCHER_MAX_CONCURRENT_TASKS, ORCHER_WORKFLOW_POLLER_COUNT, ORCHER_TASK_POLLER_COUNT and ORCHER_SHUTDOWN_GRACE_TIME_MS. Pass prefix= to use another prefix. A Worker built from a WorkerConfig this way has no interceptors or custom executor.


Authentication and TLS

An engine that requires authentication rejects a worker without a valid API key. The builder reads ORCHER_API_KEY from the environment, or takes the key explicitly; either way it is sent as authorization: Bearer <key> on every request the worker makes:

1
def build_worker() -> Worker:
2
return (
3
Worker.builder()
4
.server_url("https://orcher.example.com:443") # https:// connects with TLS
5
.namespace("payments")
6
.task_queue("payments")
7
.api_key(os.environ["ORCHER_API_KEY"]) # also read from ORCHER_API_KEY by default
8
.build()
9
)

TLS follows the URL scheme. An https:// URL connects over TLS and verifies the engine against the system trust store, with no other setting. An http:// URL connects in plaintext, even with certificates set, so use https:// whenever the engine expects TLS.

.tls() adds certificates to every connection the worker opens. Give a CA certificate to trust a private or self-signed authority, and a client certificate and key for mutual TLS:

1
def build_mtls_worker() -> Worker:
2
return (
3
Worker.builder()
4
.server_url("https://orcher.internal:443") # TLS needs an https:// URL
5
.task_queue("payments")
6
.tls(
7
ca_cert_path="/etc/orcher/ca.pem", # omit to use the system trust store
8
client_cert_path="/etc/orcher/client.pem", # client certificate, for mTLS
9
client_key_path="/etc/orcher/client-key.pem",
10
)
11
.build()
12
)

Leave out ca_cert_path when the engine's certificate is publicly trusted; for mutual TLS the worker then sends only its client certificate:

1
def build_client_identity_worker() -> Worker:
2
# A publicly trusted server: only the client certificate is custom.
3
return (
4
Worker.builder()
5
.server_url("https://orcher.example.com:443")
6
.task_queue("payments")
7
.tls(
8
client_cert_path="/etc/orcher/client.pem",
9
client_key_path="/etc/orcher/client-key.pem",
10
)
11
.build()
12
)

The client certificate and key go together: .tls() raises ValueError if only one is given. A WorkerConfig takes the same paths:

1
def mtls_config() -> WorkerConfig:
2
return WorkerConfig(
3
server_url="https://orcher.internal:443",
4
task_queue="payments",
5
tls_ca_cert_path="/etc/orcher/ca.pem",
6
tls_client_cert_path="/etc/orcher/client.pem",
7
tls_client_key_path="/etc/orcher/client-key.pem",
8
)

WorkerConfig.from_env() reads them from ORCHER_TLS_CA_CERT_PATH, ORCHER_TLS_CLIENT_CERT_PATH and ORCHER_TLS_CLIENT_KEY_PATH; the builder doesn't read those variables. When a handshake fails, the error names the cause, such as a certificate the worker doesn't trust or a client certificate the engine requires.

TLS and mutual TLS need orcher-sdk 0.4.4 or later. In 0.4.3 and earlier the worker had no TLS settings and couldn't connect to an https:// URL.


Versions

version_id declares which release of your code the worker runs: a git SHA, an image digest or a release tag. The engine records it on an execution the first time a worker claims it, so that execution keeps replaying against the code it started on. Set it in CI, through ORCHER_VERSION_ID, and the same build needs no code change. See Changing workflow code.


Interceptors

Interceptors wrap every workflow activation and every task attempt the worker runs, for logging, metrics and tracing. The built-in ones are factories: .interceptor() adds the workflow and the task half of each.

1
def build_worker() -> Worker:
2
metrics = MetricsInterceptor(InMemoryMetricsCollector())
3
return (
4
Worker.builder()
5
.server_url("http://localhost:50051")
6
.task_queue("orders")
7
.interceptor(LoggingInterceptor(log_input=False))
8
.interceptor(metrics)
9
.interceptor(TracingInterceptor(service_name="orders-worker"))
10
.build()
11
)
  • LoggingInterceptor logs each execution through Python's logging. Turn off log_input or log_output to keep payloads out of your logs.
  • MetricsInterceptor counts executions and failures and records durations in a MetricsCollector; InMemoryMetricsCollector is the built-in one.
  • TracingInterceptor opens an OpenTelemetry span for each execution. It needs opentelemetry-api and opentelemetry-sdk installed, and passes executions through untouched without them.

A workflow activation runs the workflow code from the start, replaying what's in the journal, so a workflow interceptor sees one call per activation, not one per workflow.

To write your own, subclass WorkflowInterceptor or TaskInterceptor and override intercept_execute, or the on_enter, on_success, on_error and on_exit hooks:

1
class SlowTaskAlert(TaskInterceptor):
2
"""Logs any task attempt that takes longer than a threshold."""
3
4
def __init__(self, threshold_seconds: float) -> None:
5
self.threshold = threshold_seconds
6
7
async def intercept_execute(
8
self, context: InterceptorContext, input_data: Any, next_fn: NextFn
9
) -> Any:
10
started = time.monotonic()
11
try:
12
return await next_fn(input_data)
13
finally:
14
elapsed = time.monotonic() - started
15
if elapsed > self.threshold:
16
print(f"task {context.task_name} took {elapsed:.1f}s")
17
18
19
def build_worker_with_alert() -> Worker:
20
return (
21
Worker.builder()
22
.server_url("http://localhost:50051")
23
.task_queue("orders")
24
.interceptor(SlowTaskAlert(threshold_seconds=10.0))
25
.build()
26
)

.interceptor() raises TypeError for anything that is neither an interceptor nor a factory with workflow() and task() methods.


Graceful shutdown

worker.shutdown() asks the worker to stop and returns at once; run() returns when shutdown has finished. Wire it to SIGTERM and SIGINT, as the example above does. On shutdown the worker:

  1. stops taking new work;
  2. waits up to the shutdown grace time for in-flight workflow activations and tasks to finish, delivering the results of work it had already claimed; and
  3. stops its pollers.

shutdown(force=True) skips the wait. Work the worker had claimed but not finished is not lost: once its claim or heartbeat times out, the engine hands it to another worker.


Scaling

Run more worker processes on the same task queue, on one machine or many; each polls for work independently and the engine hands each piece of work to one of them. Within a process, max_concurrent_workflows and max_concurrent_tasks cap how much runs at once.

Each worker also hosts up to 10 sessions at a time, a limit the builder can't change.


Next steps

  • Workflows: the code the worker runs
  • Tasks: retries, heartbeats and cancellation
  • Client: start the workflows the worker runs