Operations

Kubernetes Production Guide


Monitoring

Prometheus Scraping

Pods are annotated for automatic Prometheus discovery:

1
annotations:
2
prometheus.io/scrape: "true"
3
prometheus.io/port: "8080"
4
prometheus.io/path: "/metrics"

If you use the Prometheus Operator, add a ServiceMonitor:

1
apiVersion: monitoring.coreos.com/v1
2
kind: ServiceMonitor
3
metadata:
4
name: orchestrator
5
namespace: orcher
6
spec:
7
selector:
8
matchLabels:
9
app: orchestrator
10
endpoints:
11
- port: http
12
interval: 10s
13
path: /metrics

Health Endpoints

EndpointPurposeUsed By
/health/liveLiveness — is the process alive?livenessProbe
/health/readyReadiness — can it accept traffic?readinessProbe
/healthGeneral health checkManual verification
/metricsPrometheus metricsPrometheus scraper

Test health from inside the cluster:

1
kubectl port-forward -n orcher service/orchestrator 8080:8080
2
3
# In another terminal
4
curl http://localhost:8080/health/live
5
curl http://localhost:8080/health/ready
6
curl http://localhost:8080/metrics

Scaling

Manual Scaling

1
kubectl scale deployment orchestrator -n orcher --replicas=5

Note: The HPA will override manual scaling if the target utilization doesn't match. To disable autoscaling temporarily, delete the HPA.

Resource Tuning

Start with a size below and adjust requests and limits from observed usage:

WorkloadCPU RequestMemory RequestCPU LimitMemory Limit
Small250m256Mi1000m1Gi
Medium500m512Mi2000m2Gi
Large1000m1Gi4000m4Gi

Production Considerations

Secrets Management

The quick start uses kubectl create secret for simplicity. In production, use a secrets management solution:

  • External Secrets Operator — Sync secrets from AWS Secrets Manager, Vault, or GCP Secret Manager
  • Sealed Secrets — Encrypt secrets in Git for GitOps workflows
  • HashiCorp Vault — Inject secrets via CSI driver or sidecar

Image Registry

Push to a private registry and update the Kustomize image reference:

1
docker tag orcher/orchestrator:latest your-registry/orcher/orchestrator:v1.0.0
2
docker push your-registry/orcher/orchestrator:v1.0.0
1
cd services/orchestrator/k8s
2
kustomize edit set image orcher/orchestrator=your-registry/orcher/orchestrator:v1.0.0
3
kubectl apply -k .

Network Policies

Restrict traffic to only necessary connections:

1
apiVersion: networking.k8s.io/v1
2
kind: NetworkPolicy
3
metadata:
4
name: orchestrator
5
namespace: orcher
6
spec:
7
podSelector:
8
matchLabels:
9
app: orchestrator
10
policyTypes:
11
- Ingress
12
- Egress
13
ingress:
14
- from:
15
- namespaceSelector: {}
16
ports:
17
- port: 50051 # gRPC from SDK workers
18
- port: 8080 # Metrics from Prometheus
19
egress:
20
- to:
21
- namespaceSelector: {}
22
ports:
23
- port: 5432 # PostgreSQL
24
- port: 6432 # PgBouncer
25
- port: 6379 # Redis

Troubleshooting

Pods Not Starting

1
# Check events
2
kubectl describe pod -n orcher -l app=orchestrator
3
4
# Check logs
5
kubectl logs -n orcher -l app=orchestrator --tail=100

Common causes:

  • ImagePullBackOff — Image not found in registry. Check image name and registry credentials.
  • CrashLoopBackOff — Container starts and immediately exits. Check logs for missing DATABASE_URL or connection errors.
  • Pending — Insufficient resources. Check kubectl describe pod for scheduling failures.

Database Connection Failures

1
# Verify secret exists and has correct keys
2
kubectl get secret orcher-secrets -n orcher -o jsonpath='{.data.database-url}' | base64 -d
3
4
# Test connectivity from the cluster
5
kubectl run -it --rm psql --image=postgres:15 --restart=Never -n orcher -- \
6
psql "postgresql://orcher:orcher@postgres:5432/orcher" -c "SELECT 1"

gRPC Connectivity

1
# Port forward gRPC port
2
kubectl port-forward -n orcher service/orchestrator 50051:50051
3
4
# Test with grpcurl
5
grpcurl -plaintext localhost:50051 list

Cleanup

Remove all resources:

1
cd services/orchestrator/k8s
2
kubectl delete -k .
3
kubectl delete secret orcher-secrets -n orcher
4
kubectl delete namespace orcher

Next Steps

  • Kubernetes Overview — Prerequisites and quick start
  • Manifests — Deployment, Service, ConfigMap, and RBAC manifests
  • Autoscaling — HPA, PDB, and Kustomize configuration
  • Docker — Docker Compose setup for backing services
  • Configuration — Full configuration reference
  • Metrics — Prometheus metrics reference
  • Storage — PostgreSQL schema and migration details