Operations

Kubernetes Manifests


Manifests

All manifests live in services/orchestrator/k8s/ and are referenced by kustomization.yaml.

Deployment

3 replicas by default, with resource limits, health probes, and pod anti-affinity:

1
apiVersion: apps/v1
2
kind: Deployment
3
metadata:
4
name: orchestrator
5
namespace: orcher
6
spec:
7
replicas: 3
8
selector:
9
matchLabels:
10
app: orchestrator
11
template:
12
metadata:
13
labels:
14
app: orchestrator
15
component: coordination
16
annotations:
17
prometheus.io/scrape: "true"
18
prometheus.io/port: "8080"
19
prometheus.io/path: "/metrics"
20
spec:
21
serviceAccountName: orchestrator
22
securityContext:
23
runAsNonRoot: true
24
runAsUser: 1000
25
fsGroup: 1000
26
containers:
27
- name: orchestrator
28
image: orcher/orchestrator:latest
29
ports:
30
- name: grpc
31
containerPort: 50051
32
- name: http
33
containerPort: 8080
34
env:
35
- name: DATABASE_URL
36
valueFrom:
37
secretKeyRef:
38
name: orcher-secrets
39
key: database-url
40
- name: REDIS_URL
41
valueFrom:
42
secretKeyRef:
43
name: orcher-secrets
44
key: redis-url
45
optional: true
46
- name: LOG_LEVEL
47
value: "info"
48
- name: LOG_FORMAT
49
value: "json"
50
resources:
51
requests:
52
memory: "512Mi"
53
cpu: "500m"
54
limits:
55
memory: "2Gi"
56
cpu: "2000m"
57
livenessProbe:
58
httpGet:
59
path: /health/live
60
port: 8080
61
initialDelaySeconds: 10
62
periodSeconds: 30
63
readinessProbe:
64
httpGet:
65
path: /health/ready
66
port: 8080
67
initialDelaySeconds: 5
68
periodSeconds: 10
69
volumeMounts:
70
- name: config
71
mountPath: /etc/orchestrator
72
readOnly: true
73
volumes:
74
- name: config
75
configMap:
76
name: orchestrator-config
77
affinity:
78
podAntiAffinity:
79
preferredDuringSchedulingIgnoredDuringExecution:
80
- weight: 100
81
podAffinityTerm:
82
labelSelector:
83
matchExpressions:
84
- key: app
85
operator: In
86
values:
87
- orchestrator
88
topologyKey: kubernetes.io/hostname

Services

Two Service definitions — a standard ClusterIP for normal access and a headless service for direct pod addressing:

1
# Standard service for gRPC + HTTP
2
apiVersion: v1
3
kind: Service
4
metadata:
5
name: orchestrator
6
namespace: orcher
7
spec:
8
type: ClusterIP
9
ports:
10
- name: grpc
11
port: 50051
12
targetPort: grpc
13
- name: http
14
port: 8080
15
targetPort: http
16
selector:
17
app: orchestrator
18
---
19
# Headless service for direct pod access
20
apiVersion: v1
21
kind: Service
22
metadata:
23
name: orchestrator-headless
24
namespace: orcher
25
spec:
26
type: ClusterIP
27
clusterIP: None
28
ports:
29
- name: grpc
30
port: 50051
31
targetPort: grpc
32
- name: http
33
port: 8080
34
targetPort: http
35
selector:
36
app: orchestrator

SDK workers connect to the orchestrator via the standard service at orchestrator.orcher.svc.cluster.local:50051.

ConfigMap

TOML configuration mounted at /etc/orchestrator/config.toml:

1
apiVersion: v1
2
kind: ConfigMap
3
metadata:
4
name: orchestrator-config
5
namespace: orcher
6
data:
7
config.toml: |
8
[server]
9
grpc_addr = "0.0.0.0:50051"
10
http_addr = "0.0.0.0:8080"
11
max_concurrent_streams = 1000
12
13
[orchestrator]
14
max_workers = 100
15
max_concurrent_tasks = 1000
16
heartbeat_timeout_secs = 90
17
18
[storage]
19
database_url = "${DATABASE_URL}"
20
redis_url = "${REDIS_URL}"
21
22
[logging]
23
level = "info"
24
format = "json"
25
26
[health]
27
readiness_enabled = true
28
liveness_enabled = true

Note: DATABASE_URL and REDIS_URL in the ConfigMap are overridden by the environment variables from the Secret, which take precedence.

RBAC

ServiceAccount, Role, and RoleBinding with minimal permissions:

1
apiVersion: v1
2
kind: ServiceAccount
3
metadata:
4
name: orchestrator
5
namespace: orcher
6
---
7
apiVersion: rbac.authorization.k8s.io/v1
8
kind: Role
9
metadata:
10
name: orchestrator
11
namespace: orcher
12
rules:
13
- apiGroups: [""]
14
resources: ["configmaps"]
15
verbs: ["get", "list", "watch"]
16
- apiGroups: [""]
17
resources: ["secrets"]
18
verbs: ["get"]
19
- apiGroups: [""]
20
resources: ["pods"]
21
verbs: ["get", "list", "watch"]
22
- apiGroups: [""]
23
resources: ["services"]
24
verbs: ["get", "list"]
25
---
26
apiVersion: rbac.authorization.k8s.io/v1
27
kind: RoleBinding
28
metadata:
29
name: orchestrator
30
namespace: orcher
31
roleRef:
32
apiGroup: rbac.authorization.k8s.io
33
kind: Role
34
name: orchestrator
35
subjects:
36
- kind: ServiceAccount
37
name: orchestrator
38
namespace: orcher

Next Steps