Critical question: Were secrets encrypted at rest in etcd? The answer determines if you have a catastrophic incident or a containable breach.

Phase 1: Immediate triage (0-5 minutes)
1. Check if etcd encryption was enabled: kubectl get configmap -n kube-system encryption-config -o yaml # If not found, encryption was NOT enabled = secrets are in plaintext in the backup

2. Check kube-apiserver flags: kubectl describe pod -n kube-system kube-apiserver-* | grep -E 'encryption|auth'

Look for: --encryption-provider-config flag

If encryption is NOT enabled (likely scenario): CRITICAL INCIDENT
All secrets are in plaintext in the etcd backup:

1. Immediately access all external services (cloud APIs, databases, SaaS)

2. Impersonate any service account

3. Decrypt traffic between services

4. Escalate to full cluster compromise

   Phase 2: Emergency response (5-30 minutes)
   ASSUME FULL COMPROMISE. Do this in parallel:

5. Revoke all secrets immediately: for secret in $(kubectl get secrets -A -o jsonpath='{.items[*].metadata.name}'); do

4. Monitor for suspicious activity: kubectl logs -n kube-system kube-apiserver-* --tail=500 | grep -E 'unauthorized|forbidden|token.*invalid' Check cloud provider (AWS) for unusual API calls Check databases for unauthorized access

   Phase 3: Investigation (30 minutes - 2 hours)
   1. Analyze the etcd backup to determine exposure:

Did attacker actually access something, or just had the file?

Phase 4: Remediation (2-8 hours)
Priority 1: Rotate HIGH-risk secrets (database creds, production API keys)

• This is expensive (requires new CA if private key was exposed)
• Can be done in parallel with other rotations
• Rolling update of TLS secrets to all affected pods

  Phase 5: Long-term hardening
  Enable encryption at rest in etcd (SHOULD HAVE BEEN DONE):

This forces etcd to re-encrypt all existing secrets with the new key

Now, if etcd backup is exfiltrated again, secrets are encrypted:
etcdctl snapshot info /backup/etcd-snapshot

Attacker has only encrypted blobs, not plaintext credentials

Phase 6: Backup security hardening
1. Encrypt backups: etcdctl snapshot save /backups/etcd-$(date +%s).db --cacert=/etc/etcd/ca.pem --cert=/etc/etcd/etcd.pem --key=/etc/etcd/etcd-key.pem

• Keep multiple versions (can’t delete all at once)
• Delete old backups after 90 days (or your retention policy)
• This limits exposure window if backup is found

  Phase 7: Post-incident checklist
  [] All HIGH-risk secrets rotated

[] All MEDIUM-risk secrets rotated [] TLS certificates replaced (if private key was exposed) [] Encryption at rest enabled in etcd [] etcd backups now encrypted [] Backup storage hardened [] No suspicious activity detected in logs (24h monitoring) [] Team trained on secret management [] Post-mortem scheduled (review how backup got leaked)

This is a subtle but critical bug: encryption configuration exists, but secrets aren't actually being encrypted. The system has no way to know which secrets are encrypted and which aren't.

Phase 1: Detection
1. Check encryption configuration: kubectl get configmap -n kube-system encryption-config -o yaml # Verify encryption-provider-config is set

2. Verify kube-apiserver is using encryption: ps aux | grep kube-apiserver | grep encryption-provider-config

Expected encrypted output: binary gibberish (AES-CBC encrypted) Actual (unencrypted): readable JSON with plaintext values

Phase 2: Quantify the damage
1. Determine which secrets are unencrypted: kubectl get secrets -A -o json |
jq -r '.items[] | "(.metadata.namespace)/(.metadata.name) (.metadata.creationTimestamp)"' > /tmp/all-secrets.txt

Count: ~30,000 unencrypted secrets

Phase 3: Root cause analysis
1. Check encryption provider logs: kubectl logs -n kube-system kube-apiserver-* | grep -i encrypt | tail -50

2. Common bugs: a) Encryption key not loaded properly:

   • Check key file exists: ls -la /etc/kubernetes/encryption-key
   • Check file permissions: stat /etc/kubernetes/encryption-key

   b) Identity provider is primary (should be fallback): apiVersion: apiserver.config.k8s.io/v1 kind: EncryptionConfiguration resources:

   • resources:
     • secrets providers:
     • identity: {} # BUG: This is first, so secrets are NOT encrypted
     • aescbc: keys:
       • name: key1 secret: …

   c) Reload wasn’t working: kube-apiserver should have --encryption-provider-config-automatic-reload=true If reload is broken, old config persists

   Phase 4: Fix (safely, without data loss)
   1. FIX the encryption configuration order:

• resources:
  • secrets providers:
  • aescbc: keys:
    • name: key1 secret:
  • identity: {} # Fallback AFTER encryption attempt
    2. Update kube-apiserver config and restart:
    kubectl set env deployment/kube-apiserver -n kube-system
ENCRYPTION_CONFIG_UPDATED="true" kubectl rollout restart deployment/kube-apiserver -n kube-system

kubectl rollout status deployment/kube-apiserver -n kube-system
3. Force re-encryption of all secrets: kubectl get secrets --all-namespaces -o json |
kubectl apply -f -

This re-writes all secrets, forcing them to be encrypted with the fixed config

4. Verify encryption worked: etcdctl get /kubernetes.io/secrets/default/my-secret

Should now show encrypted binary data, not plaintext

Phase 5: Verification that fix worked
1. Sample encrypted secrets: for secret in $(kubectl get secrets -A -o name | shuf | head -5); do echo "=== $secret ===" etcdctl get /kubernetes.io/secrets/$(echo $secret | cut -d’/’ -f1)/$(echo $secret | cut -d’/’ -f2) | hexdump -C | head -3

3. Audit log should show encryption was applied: kubectl logs -n kube-system kube-apiserver-* --tail=100 | grep -i 'secret.*encrypt|encrypt.*success'

   Phase 6: Prevention for the future
   1. Automated verification test:

This policy verifies that secrets are actually encrypted before allowing creation

Phase 7: Post-incident remediation
[] Fixed encryption configuration [] Re-encrypted all 30,000 unencrypted secrets [] Verified sample secrets are now encrypted [] Enabled hourly verification test [] Updated runbook for future encryption issues [] Team training on EncryptionConfiguration gotchas

Migrating from Kubernetes Secrets to Vault is a big architectural change. The safest approach is running both in parallel, gradually moving services to Vault.

Phase 1: Design the target architecture
┌─────────────────────────────────────────┐ │ HashiCorp Vault │ │ ┌────────────────────────────────────┐ │ │ │ /secret/data/prod/db-password │ │ │ │ /secret/data/prod/stripe-api-key │ │ │ │ /secret/data/staging/test-db-pass │ │ │ └────────────────────────────────────┘ │ │ Features: audit logs, MFA, TTL, rotation │ └─────────────────────────────────────────┘ ↑ │ (pod auth via Kubernetes ServiceAccount) │ ┌──────────────────────────┐ │ Kubernetes Pod │ │ Sidecar proxy: vault-sidekcar pulls secrets from Vault │ OR │ External Secrets operator: syncs Vault→K8s Secrets └──────────────────────────┘

Phase 2: Deploy Vault in cluster
1. Install Vault via Helm: helm repo add hashicorp https://helm.releases.hashicorp.com helm install vault hashicorp/vault \ --set server.dataStorage.size=10Gi \ --set server.statefulSet.replicas=3 \ --namespace vault \ --create-namespace

2. Initialize Vault: kubectl exec -n vault vault-0 – vault operator init
   –key-shares=5
   –key-threshold=3

3. Unseal Vault: for i in 1 2 3; do kubectl exec -n vault vault-0 – vault operator unseal KEY_$i done

4. Configure Kubernetes auth: kubectl exec -n vault vault-0 – vault auth enable kubernetes kubectl exec -n vault vault-0 – vault write auth/kubernetes/config
kubernetes_host=https://kubernetes.default.svc.cluster.local:443
kubernetes_ca_cert="@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt"
token_reviewer_jwt="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)"

   Phase 3: Setup External Secrets Operator (for parallel running)
   1. Install External Secrets:

Pods can use K8s Secret as before, but data comes from Vault

Phase 4: Migrate secrets incrementally
Wave 1: Non-critical, staging secrets (Week 1)

PARALLEL STATE (safest): Some services still use K8s Secrets Other services synced from Vault via ExternalSecret Both work simultaneously

Phase 5: Setup audit logging in Vault
apiVersion: v1 kind: ConfigMap metadata: name: vault-audit-config data: audit-config.hcl: | audit { file { path = "/vault/logs/audit.log" } }

- What action (read, create, delete)

Phase 6: Secret rotation policy
1. For database credentials (need coordinated rotation): apiVersion: pam.vault.io/v1alpha1 kind: RotationPolicy metadata: name: db-rotation spec: secretPath: secret/data/prod/db-password rotationInterval: 30d # Rotate every 30 days rotation: databaseConnection: engine: postgresql connectionUri: "postgres://root:password@db.internal:5432/postgres" rotationStatements: - "ALTER USER app_user WITH PASSWORD '{{ password }}'"

2. For API keys (can be rotated independently): kubectl exec -n vault vault-0 – vault write -f /secret/data/prod/stripe-api-key
value=$(curl -H "Authorization: Bearer $STRIPE_TOKEN" https://api.stripe.com/v1/api_keys/generate | jq -r '.key')

   Phase 7: Access control with policies
   Create fine-grained Vault policies:

admin.hcl: | path "/" { capabilities = [""] } # Only cluster admin

Phase 8: Monitoring and alerting
- alert: VaultSealed expr: vault_unsealed_status == 0 for: 1m annotations: summary: "Vault is sealed - pods can’t retrieve secrets!"

• alert: UnauthorizedVaultAccess expr: rate(vault_audit_log_error_total[5m]) > 0.1 annotations: summary: "Unauthorized access attempts to Vault"

• alert: ExternalSecretSyncFailure expr: externalsecret_status_sync_failures_total > 0 for: 5m annotations: summary: "ExternalSecret failed to sync from Vault"

  Final state after migration:
  OLD (Kubernetes Secrets only):

• Secrets scattered across namespaces

• No audit trail

• Hard to rotate

• Developers have broad Secret access via RBAC

• Centralized secret management
• Full audit trail of all access
• Automatic rotation
• Fine-grained access control per service
• Secrets encrypted at rest in Vault
• Can use for non-Kubernetes systems too

This is a critical incident. The secret is now visible to anyone who cloned the repo, and may be indexed by search engines or security scanners.

Hour 0 (immediate, first 5 minutes):
1. Confirm the leak: git log --oneline | grep -i secret git show | grep -E 'password|key|token' # Identify exactly what was leaked
2. Revoke the secret IMMEDIATELY: - If database password: change it NOW - If API key: revoke it NOW - If AWS credential: disable it NOW - If OAuth token: invalidate it NOW

Command example (PostgreSQL): ALTER USER app_user WITH PASSWORD 'new-secure-password-12345678';
3. Notify stakeholders: "Database password for production has been exposed in GitHub. Password has been rotated. No unauthorized access detected yet. Starting incident response."

Hour 0-1 (first hour):
1. Rewrite Git history (remove secret from all commits):

4. Check if secret was exposed to search engines:

   • Google Cache: https://webcache.googleusercontent.com/…
   • GitHub API history (check 3rd party services)
   • Pastebin, GitLab mirrors, etc.

   Hour 1-4 (first 4 hours):
   1. Deep forensic investigation: a) Who has access to the repo (collaborators)? b) Who cloned the repo in the last 30 days? c) When was the secret first exposed? d) Is there evidence of the secret being used by attackers?

5. Check GitHub audit logs: GitHub Settings → Security log Look for: unusual clone IPs, API calls from unknown locations

6. Query cloud provider audit logs: aws cloudtrail get-event-selectors --trail-name production-trail

   Look for: unauthorized DB access, IAM policy changes, new users created

7. Database forensics: Check connection logs for source IPs using exposed credentials: SELECT * FROM pg_stat_statements WHERE query LIKE '%app_user%'; SHOW log_statement; # Verify logging is enabled

8. Rotate all related secrets:

   • New database password
   • Update Kubernetes Secret
   • Restart pods that use this secret
   • Verify no connection errors in logs

9. Update pre-commit hooks to PREVENT future incidents: Install git-secrets or similar: git secrets --install git secrets --register-aws git secrets --add-provider 'grep -i "password|api.key|token"'

   Hour 4+ (ongoing):
   1. Incident post-mortem: - How did the secret get committed? - Why did pre-commit hooks fail? - How do we prevent this in the future? - What's the cost of this incident?

10. Implement preventive measures: a) Secret scanning in GitHub (enable free tier): GitHub Settings → Security → Secret scanning This alerts if secrets are detected in new commits

    b) Mandatory secret scanning in CI/CD:

    • truffleHog: scans for secrets
    • detect-secrets: discovers secrets in code
    • gitleaks: finds secrets in git history

    In CI/CD pipeline:

    • Pre-push check (runs locally)
    • Pre-commit check (blocks commit)
    • CI check (blocks PR merge)

    c) Access control:

    • Restrict who can push to main/master
    • Require PR reviews (including security review)
    • Only protected branches allowed

11. Monitoring going forward:

    • GitHub secret scanning alerts
    • Vault audit logs (if using Vault)
    • Database connection anomalies
    • API key usage patterns

    Assessment of damage (depends on findings):
    Scenario A: No unauthorized access detected - Risk is low if secret was rotated quickly - Attacker likely didn't get the secret in time - Action: document incident, implement preventive measures

    Scenario B: Evidence of unauthorized access

    • Critical incident: assume full compromise
    • Action: follow Phase 1-5 of the etcd breach playbook
    • Rotate ALL secrets, investigate affected systems

    Scenario C: Secret was indexed by search engines

    • Search engines: request removal via Google Search Console
    • Pastebin/mirrors: request removal
    • Risk is higher, assume secret may have been accessed
    • Rotate more aggressively

Seamless secret rotation requires: (1) dual credential support during rotation, (2) rapid secret refresh in pods, (3) health checking to verify connectivity after rotation.

Challenge: Kubernetes doesn't automatically update environment variables or mounted files when a Secret changes. Pods keep using stale values until restarted.

Solution 1: Grace period with dual credentials
1. During rotation, both old and new credentials work: a) Create new credential in target system (database, AWS, etc.) - PostgreSQL: CREATE USER app_user_new WITH PASSWORD 'new-password' - AWS: Create new access key - API provider: Generate new key

b) Update Vault with BOTH credentials: vault write secret/data/prod/db-password
username=app_user_new
password=new-password
username_old=app_user_old
password_old=old-password

• name: app image: myapp:latest env:
  • name: VAULT_ADDR value: "http://vault.vault.svc.cluster.local:8200"

  Pod runs vault-agent sidecar

  Vault-agent auto-refreshes secrets every 60 seconds

  Application reads from templated file (updated in real-time)
  

  Vault agent template (updates in real-time):
  {{ with secret "secret/data/prod/db-password" -}} export DB_USER="{{ .Data.data.username }}" export DB_PASSWORD="{{ .Data.data.password }}" {{ end }}

  Solution 2: Automatic pod restart via ExternalSecret rotation detection
  1. When ExternalSecret detects secret change, trigger pod restart:

2. Deploy a controller that watches for label changes: apiVersion: v1 kind: ConfigMap metadata: name: secret-rotation-watcher data: watcher.py: | import subprocess import hashlib import time

   def get_secret_hash(secret_name, namespace): result = subprocess.run(['kubectl', 'get', 'secret', secret_name, '-n', namespace, '-o', 'jsonpath={.data.password}'], capture_output=True) return hashlib.sha256(result.stdout).hexdigest()

   old_hash = None while True: current_hash = get_secret_hash('db-credentials', 'production') if old_hash and current_hash != old_hash: # Secret changed! Restart pods subprocess.run(['kubectl', 'rollout', 'restart', 'deployment/app', '-n', 'production']) old_hash = current_hash time.sleep(60)

3. Deploy the watcher: kubectl apply -f secret-rotation-watcher-job.yaml

   Solution 3: Service mesh integration (most elegant)
   apiVersion: security.istio.io/v1beta1

- Connection pool handles transient failures during rotation

Solution 4: Health checks + graceful degradation
During rotation, some pods may temporarily fail:

(which was updated by ExternalSecret)

Solution 5: Zero-downtime rotation orchestration
apiVersion: batch/v1 kind: CronJob metadata: name: secret-rotation-orchestrator spec: schedule: "0 2 * * *" # Daily at 2 AM jobTemplate: spec: template: spec: containers: - name: rotator image: rotation-orchestrator:latest env: - name: GRACE_PERIOD_SECONDS value: "3600" # 1 hour grace period - name: BATCH_SIZE value: "10" # Restart 10 pods at a time script: | #!/bin/bash

# Step 5: Revoke old credential vault kv patch secret/prod/db-password username_old="" password_old="" # PostgreSQL: ALTER USER app_user_old NOLOGIN;

Best practice checklist:
[] Use Vault + ExternalSecret (auto-refresh secrets) [] Implement health checks for post-rotation verification [] Grace period: keep old + new credentials during rotation [] Batch pod restarts: avoid thundering herd [] Monitor rotation process: alert on failures [] Automatic rollback if rotation fails [] Test rotation in staging first [] Document rotation procedure for manual override [] Audit all rotation events

This is a critical security question: balancing operational agility (developers need to debug) with security (minimize exposure of production secrets).

Solution: Temporary elevated access with approval, TTL, and auditing
Architecture: Developer → Request tool → Approval workflow → Vault issues temporary credential → Automatic expiry

Step 1: Developer requests temporary secret access $ vault-breakglass request secret:prod/db-password --reason "Debug connection pool issue" --duration 1h Request ID: bkg-2024-04-07-001 Status: PENDING

• name: v1 served: true storage: true schema: openAPIV3Schema: type: object properties: spec: type: object properties: requesterEmail: type: string secretPath: type: string reason: type: string durationSeconds: type: integer maxDurationSeconds: type: integer default: 3600 # Max 1 hour status: type: object properties: approved: type: boolean approverEmail: type: string approvedAt: type: string expiresAt: type: string credentialId: type: string

  Approval workflow implementation (CLI tool):
  #!/bin/bash

# Issue temporary credential from Vault vault read -format=json auth/approle/role/breakglass/secret-id
metadata="request_id=$REQUEST_ID" | jq -r '.auth.client_token' ;; esac

Audit and compliance:
apiVersion: v1 kind: ConfigMap metadata: name: vault-audit-breakglass data: audit-policy.json: | { "type": "file", "options": { "file_path": "/var/log/vault/audit.log", "log_raw": true } }

Monthly audit report: vault audit logs | jq 'select(.auth.metadata.breakglass == "true")' |
group_by(.auth.display_name) |
map({user: .[0].auth.display_name, accesses: length})

Compliance and guardrails:
[] Max duration: 1 hour [] Reason: required and audited [] Approval: required (no self-approval) [] Scope: read-only access [] Audit: all access logged [] Auto-expiry: credential revoked after TTL [] Alerting: unusual patterns trigger alerts

• alert: HighBreakglassUsage expr: count(rate(vault_breakglass_requests_total[5m])) > 5 annotations: summary: "More than 5 breakglass requests/min (unusual activity)"

• alert: LongDurationBreakglass expr: vault_breakglass_duration_seconds > 3600 annotations: summary: "Breakglass request exceeded max duration"

  Usage examples:
  # Request prod database password for 30 minutes

vault-breakglass retrieve bkg-2024-04-07-001

Secrets migration requires zero-downtime switchover. You can't cut over abruptly.

Strategy:
Phase 1: Deploy Vault and sync secrets (both old and new active)
Phase 2: Update 10% of pods to read from Vault
Phase 3: Monitor, validate, then expand to 50%, then 100%
Phase 4: Delete old ConfigMaps/env vars once 100% migrated

Implementation:
Original deployment uses env vars: spec: containers: - name: app env: - name: DB_PASSWORD valueFrom: configMapKeyRef: name: db-creds key: password
Step 1: Deploy ExternalSecret alongside:
apiVersion: external-secrets.io/v1beta1 kind: ExternalSecret metadata: name: db-credentials-vault spec: secretStoreRef: name: vault target: name: db-credentials-from-vault data: - secretKey: password remoteRef: key: prod/db-password
Step 2: Canary: 10% of pods use Vault secret
spec: containers: - name: app env: - name: DB_PASSWORD valueFrom: secretKeyRef: name: db-credentials-from-vault # NEW: from Vault key: password
Step 3: Monitor error rates, database connection errors, latency
If issues: can revert by scaling back canary
Step 4: Gradually expand to 25%, 50%, 100%
Step 5: Delete old ConfigMaps
kubectl delete configmap db-creds

Validation during each phase:
- Application logs show no connection errors
- Database connection pool healthy
- Vault audit logs show secret access
- No increase in error rate

Balance: security without friction. Developers need enough access to be productive, but with guardrails.

Model:
- Developers read staging secrets freely (low security risk)
- Production secrets: read-only access via audit logs
- Sensitive operations (modify secrets): require approval
- Access automatically revokes after TTL

Tiered access:
TIER 1: Developer (Staging) - Read all staging secrets - No TTL (permanent) - No approval needed

TIER 2: Developer (Production, Debug)

• Full access to all secrets
• No TTL
• Automatic audit logging

  Implementation with Vault:
  Dev can request staging password:

Requires approval from SRE, only then can they update

Vault policy:
path "secret/staging/*" { capabilities = ["read", "list"]

}

Monitoring suspicious patterns:
- Dev reading same secret 100x in 1 hour (exfiltration attempt?)
- Dev reading secrets outside their team
- Dev requesting modifications without reason
- Off-hours access patterns