Kubernetes Production Guide
Monitoring
Prometheus Scraping
Pods are annotated for automatic Prometheus discovery:
1annotations:2 prometheus.io/scrape: "true"3 prometheus.io/port: "8080"4 prometheus.io/path: "/metrics"If you use the Prometheus Operator, add a ServiceMonitor:
1apiVersion: monitoring.coreos.com/v12kind: ServiceMonitor3metadata:4 name: orchestrator5 namespace: orcher6spec:7 selector:8 matchLabels:9 app: orchestrator10 endpoints:11 - port: http12 interval: 10s13 path: /metricsHealth Endpoints
| Endpoint | Purpose | Used By |
|---|---|---|
/health/live | Liveness — is the process alive? | livenessProbe |
/health/ready | Readiness — can it accept traffic? | readinessProbe |
/health | General health check | Manual verification |
/metrics | Prometheus metrics | Prometheus scraper |
Test health from inside the cluster:
1kubectl port-forward -n orcher service/orchestrator 8080:808023# In another terminal4curl http://localhost:8080/health/live5curl http://localhost:8080/health/ready6curl http://localhost:8080/metricsScaling
Manual Scaling
1kubectl scale deployment orchestrator -n orcher --replicas=5Note: The HPA will override manual scaling if the target utilization doesn't match. To disable autoscaling temporarily, delete the HPA.
Resource Tuning
Start with a size below and adjust requests and limits from observed usage:
| Workload | CPU Request | Memory Request | CPU Limit | Memory Limit |
|---|---|---|---|---|
| Small | 250m | 256Mi | 1000m | 1Gi |
| Medium | 500m | 512Mi | 2000m | 2Gi |
| Large | 1000m | 1Gi | 4000m | 4Gi |
Production Considerations
Secrets Management
The quick start uses kubectl create secret for simplicity. In production, use a secrets management solution:
- External Secrets Operator — Sync secrets from AWS Secrets Manager, Vault, or GCP Secret Manager
- Sealed Secrets — Encrypt secrets in Git for GitOps workflows
- HashiCorp Vault — Inject secrets via CSI driver or sidecar
Image Registry
Push to a private registry and update the Kustomize image reference:
1docker tag orcher/orchestrator:latest your-registry/orcher/orchestrator:v1.0.02docker push your-registry/orcher/orchestrator:v1.0.01cd services/orchestrator/k8s2kustomize edit set image orcher/orchestrator=your-registry/orcher/orchestrator:v1.0.03kubectl apply -k .Network Policies
Restrict traffic to only necessary connections:
1apiVersion: networking.k8s.io/v12kind: NetworkPolicy3metadata:4 name: orchestrator5 namespace: orcher6spec:7 podSelector:8 matchLabels:9 app: orchestrator10 policyTypes:11 - Ingress12 - Egress13 ingress:14 - from:15 - namespaceSelector: {}16 ports:17 - port: 50051 # gRPC from SDK workers18 - port: 8080 # Metrics from Prometheus19 egress:20 - to:21 - namespaceSelector: {}22 ports:23 - port: 5432 # PostgreSQL24 - port: 6432 # PgBouncer25 - port: 6379 # RedisTroubleshooting
Pods Not Starting
1# Check events2kubectl describe pod -n orcher -l app=orchestrator34# Check logs5kubectl logs -n orcher -l app=orchestrator --tail=100Common causes:
- ImagePullBackOff — Image not found in registry. Check image name and registry credentials.
- CrashLoopBackOff — Container starts and immediately exits. Check logs for missing
DATABASE_URLor connection errors. - Pending — Insufficient resources. Check
kubectl describe podfor scheduling failures.
Database Connection Failures
1# Verify secret exists and has correct keys2kubectl get secret orcher-secrets -n orcher -o jsonpath='{.data.database-url}' | base64 -d34# Test connectivity from the cluster5kubectl run -it --rm psql --image=postgres:15 --restart=Never -n orcher -- \6 psql "postgresql://orcher:orcher@postgres:5432/orcher" -c "SELECT 1"gRPC Connectivity
1# Port forward gRPC port2kubectl port-forward -n orcher service/orchestrator 50051:5005134# Test with grpcurl5grpcurl -plaintext localhost:50051 listCleanup
Remove all resources:
1cd services/orchestrator/k8s2kubectl delete -k .3kubectl delete secret orcher-secrets -n orcher4kubectl delete namespace orcherNext Steps
- Kubernetes Overview — Prerequisites and quick start
- Manifests — Deployment, Service, ConfigMap, and RBAC manifests
- Autoscaling — HPA, PDB, and Kustomize configuration
- Docker — Docker Compose setup for backing services
- Configuration — Full configuration reference
- Metrics — Prometheus metrics reference
- Storage — PostgreSQL schema and migration details