Deploy and Monitor Applications on Azure Kubernetes Service

1. Deployment Manifest Structure

A Deployment tells Kubernetes: "run N copies of this container image." Kubernetes maintains that exact state — restarting crashed pods, scheduling on healthy nodes.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: ai-inference-api
spec:
  replicas: 2
  selector:
    matchLabels:
      app: inference-api       # Must match template.labels below
  template:
    metadata:
      labels:
        app: inference-api     # Must match selector above
    spec:
      containers:
      - name: api
        image: myregistry.azurecr.io/inference-api:v1.0
        ports:
        - containerPort: 8080
        resources:
          requests:
            memory: "2Gi"
            cpu: "1000m"       # 1 CPU = 1000 millicores
          limits:
            memory: "4Gi"
            cpu: "2000m"

2. Replicas for High Availability

1. 1 replica — single point of failure. App is down while Kubernetes restarts it after a crash.
2. 2 replicas — basic resilience. One can crash, the other keeps serving. Minimum for production.
3. 3+ replicas — enables rolling updates with zero downtime. One updates while two keep serving traffic.

3. Resource Requests and Limits

1. Requests = minimum guaranteed resources (scheduler uses this to pick a node). Set too low = pod on overloaded node = poor performance.
2. Limits = maximum allowed. Pod is killed (OOMKill) if memory limit exceeded. Set too low for model inference = constant restarts.

4. Injecting Secrets via Kubernetes Secrets

kubectl create secret generic api-secrets --from-literal=api-key=your-secret-key

# Reference in Deployment manifest:
env:
- name: API_KEY
  valueFrom:
    secretKeyRef:
      name: api-secrets   # Secret object name
      key: api-key        # Key within the secret

1. Service Types — When to Use Each

2. LoadBalancer Service Manifest

apiVersion: v1
kind: Service
metadata:
  name: inference-api-service
spec:
  type: LoadBalancer
  selector:
    app: inference-api    # Routes to pods with this label
  ports:
  - protocol: TCP
    port: 80              # External port (clients connect here)
    targetPort: 8080      # Container port (app listens here)

3. The Selector-Label Contract (Most Common Failure)

Service selector must exactly match pod labels. One typo = Service has no endpoints = silent traffic drop.

kubectl get pods --show-labels
kubectl describe svc inference-api-service | grep Selector

4. ClusterIP Internal DNS

ClusterIP services are reachable via stable DNS: servicename.namespace.svc.cluster.local. Your API pod calls http://vector-search-svc.default.svc.cluster.local:8080 even as pod IPs change.

1. Deploy with kubectl apply (Idempotent)

kubectl apply -f deployment.yaml
kubectl apply -f service.yaml
kubectl apply -f ./k8s/   # Apply entire directory

apply creates if not exists, updates if exists. Safe to run multiple times.

2. Verify Status

kubectl get deployments                    # READY 2/2 = both replicas healthy
kubectl get pods                           # STATUS Running/Pending/CrashLoopBackOff
kubectl get svc                            # EXTERNAL-IP for LoadBalancer (may show pending)

3. Common Failure Modes and Fixes

1. ImagePullBackOff — wrong image tag, registry URL, or missing pull credentials. Fix: kubectl describe pod to see exact error.
2. CrashLoopBackOff — app starts then crashes. Fix: kubectl logs pod-name for stack trace. Do NOT blindly restart.
3. Pending — can't schedule. Insufficient node resources. Fix: kubectl describe pod → Events section shows reason.
4. No Endpoints — Service selector doesn't match pod labels. Fix: kubectl describe svc and kubectl get pods --show-labels.

kubectl describe pod my-pod     # Events, conditions, image details
kubectl logs my-pod             # App stdout/stderr
kubectl logs my-pod --previous  # Logs from PREVIOUS crashed container

Enable Container Insights

# Enable Container Insights on existing cluster
az aks enable-addons \
  --addons monitoring \
  --name my-aks-cluster \
  --resource-group rg \
  --workspace-resource-id $LOG_ANALYTICS_ID

# KQL: pod restart count (CrashLoopBackOff signal)
KubePodInventory
| where TimeGenerated > ago(1h)
| where Namespace == "ai-apps"
| summarize restarts=sum(PodRestartCount) by PodUid, Name
| where restarts > 3
| order by restarts desc

# KQL: container CPU usage
Perf
| where ObjectName == "K8SContainer"
| where CounterName == "cpuUsageNanoCores"
| summarize avg_cpu=avg(CounterValue)
    by bin(TimeGenerated, 5m), InstanceName

Diagnose Common Failures

# CrashLoopBackOff — app crashes on start
kubectl logs my-pod --previous        # last crash logs
kubectl describe pod my-pod           # events section = root cause

# Pending — stuck scheduling
kubectl describe pod my-pod           # look for "Insufficient CPU"
kubectl get nodes                     # check node capacity
kubectl top nodes                     # live resource usage

# ImagePullBackOff — can't pull image
kubectl describe pod my-pod           # shows registry/auth error
# Fix: ensure AcrPull role on managed identity

# OOMKilled — out of memory
kubectl describe pod my-pod | grep -A5 "Last State"
# Fix: increase resources.limits.memory in deployment YAML

# Exec into running pod for debugging
kubectl exec -it my-pod -- /bin/bash
kubectl port-forward my-pod 8080:8080  # local testing

Pods Access Azure Services Securely

# Enable OIDC + Workload Identity on cluster
az aks update --enable-oidc-issuer \
  --enable-workload-identity \
  --name my-aks --resource-group rg

# Create managed identity for the workload
az identity create --name ai-workload-id --resource-group rg

# Federate: allow pod SA to use the managed identity
az identity federated-credential create \
  --name aks-federated \
  --identity-name ai-workload-id \
  --resource-group rg \
  --issuer $OIDC_ISSUER \
  --subject system:serviceaccount:ai-apps:ai-sa

# Grant managed identity access to Azure OpenAI
az role assignment create \
  --role "Cognitive Services OpenAI User" \
  --assignee $MI_CLIENT_ID \
  --scope $OPENAI_RESOURCE_ID