Your production database passwords are stored in plaintext in group_vars/all.yml and committed to Git. A security audit flags this as critical risk. You need to encrypt sensitive variables immediately while maintaining CI/CD automation. How do you migrate existing secrets to Ansible Vault without downtime?
Create a new vault file for secrets, use `ansible-vault create group_vars/all/vault.yml` with a strong password. Move sensitive vars there and reference them in group_vars/all/vars.yml using `"{{ vault_db_password }}"`. Update CI/CD to pass vault password via `--vault-password-file` or environment variable. Migrate incrementally by creating separate vault files per environment. Store vault passwords in a secrets manager (HashiCorp Vault, AWS Secrets Manager) rather than files. Rotate vault password periodically using `ansible-vault rekey`.
Follow-up: How would you handle vault password distribution across 50 CI/CD agents without storing passwords in version control?
Your Ansible playbook needs to reference API keys, database credentials, and SSL certificates across multiple environments (dev, staging, prod). Currently all secrets are in one vault file, making it impossible to rotate prod secrets without affecting dev. How do you structure multi-environment secret management?
Create environment-specific vault files: group_vars/production/vault.yml, group_vars/staging/vault.yml, etc. Each has its own encryption password stored in CI/CD secrets manager. Use Ansible's variable precedence to load the correct vault based on inventory group. Implement separate decryption keys per environment—this enables rotating production secrets independently. Store vault IDs using `--vault-id` for cleaner management: `ansible-vault view --vault-id prod@prompt secrets.yml`. Alternatively, use external secret management (Vault, AWS Secrets Manager) and query them at runtime instead of managing encrypted files.
Follow-up: If a developer accidentally commits an unencrypted production secret, how would you detect and remediate this?
Your CI/CD pipeline runs Ansible deployments hourly. Recently, vault password rotations broke 30% of deployments because the password wasn't updated in the pipeline config. You need to implement a process that survives password rotations without manual intervention. What's your approach?
Decouple vault passwords from CI/CD configuration by fetching them at runtime from a secrets manager. Implement a credential provider script that queries AWS Secrets Manager, HashiCorp Vault, or Kubernetes secrets instead of storing passwords in CI/CD environment variables. Use `--vault-password-file` with a script that dynamically fetches the password. Example: create a Python script that retrieves the password from AWS Secrets Manager and pipes it to Ansible. Implement alerts when vault password retrieval fails to catch credential issues before deployments. Test password rotation workflows in staging first, then execute in production during change windows.
Follow-up: How would you implement automatic vault password rotation without service interruption?
A developer is auditing your Ansible infrastructure and discovers that vault files are being cached in temporary directories after decryption, potentially leaving secrets exposed if the machine is compromised. How do you prevent plaintext secret exposure?
Configure Ansible to use in-memory processing rather than temporary files. Set environment variables to prevent caching: `ANSIBLE_VAULT_IDENTITY_LIST` to specify vault IDs explicitly. Use `--vault-id` instead of `--vault-password-file` for better audit trails. Implement file system hardening: use tmpfs for Ansible working directories to keep secrets in RAM-only, inaccessible to disk recovery tools. Enable SELinux or AppArmor policies restricting vault file access. Ensure execution happens in containerized environments (Docker/Kubernetes) with ephemeral storage that's destroyed after execution. Regularly audit /tmp and /var/tmp for accidentally leaked vault files using automated scanning.
Follow-up: How would you implement secret rotation for database passwords that must be synchronized across running applications?
Your team manages secrets across 200 servers using Ansible Vault. An engineer accidentally checked in a vault file with a weak password (8 characters). By the time discovered, it had been in the repo for 48 hours. What's your incident response and long-term remediation?
Immediate response: Assume all secrets in that vault are compromised. Rotate every credential (database passwords, API keys, certificates) encrypted by that vault immediately. Re-encrypt vault files with new strong password. Update all CI/CD configurations with new password. Run `git log --full-history -- path/to/vault.yml` to verify how long the weak password was exposed, then audit logs for unauthorized access during that window. Implement pre-commit hooks using tools like `detect-secrets` or `git-secrets` to prevent plaintext credential commits. Add CI/CD checks to validate vault passwords meet complexity requirements. Long-term: implement vault password policy enforcement, enable secret scanning in your Git provider (GitHub/GitLab secret scanning), and consider using external secret managers to eliminate vault files entirely.
Follow-up: How would you structure Ansible Vault to support principle of least privilege where developers see only their environment's secrets?
You're migrating from managing vault passwords in environment variables to using HashiCorp Vault. During the transition, some playbooks reference vault passwords from environment variables while others fetch from HashiCorp Vault. How do you manage this hybrid approach safely?
Create a credential provider abstraction layer that accepts multiple vault backends. Implement a Python script that checks for credentials in order of precedence: HashiCorp Vault first, fall back to environment variables, then fail gracefully with clear error messages. Use `--vault-password-file` with this script across all playbooks. Gradually migrate playbooks to the new system by updating the script's credential lookup order. Add logging to track which credential source is used for each playbook run. Implement a feature flag system where playbooks can opt-in to the new vault backend. Test the hybrid configuration thoroughly in a staging environment before production deployment. Set a migration deadline and enforce the new system after that date.
Follow-up: How would you validate that secrets haven't been accidentally logged or printed in Ansible output?
Your company's security team requires all secrets to be encrypted at rest and in transit, with audit logs showing who accessed which secret and when. Ansible Vault alone doesn't provide this granular audit capability. How do you achieve this with production-grade secret management?
Integrate HashiCorp Vault or AWS Secrets Manager as your source of truth. Modify Ansible playbooks to query secrets at runtime instead of storing them. Implement a custom Ansible lookup plugin that retrieves secrets from your secret manager, which provides built-in audit logging showing access timestamp, accessor identity, and secret version. HashiCorp Vault offers dynamic secrets generation—database credentials can be auto-generated per deployment and auto-revoked after playbook completion. Enable Vault's audit backend to log all secret access. Implement secrets usage monitoring by wrapping secret retrieval in custom modules that log what secrets were accessed during each deployment. Use Vault's per-secret access policies to enforce least privilege.
Follow-up: How would you handle emergency access to secrets when the vault is temporarily unavailable during a production incident?
Your Ansible playbooks contain inline tasks that reference vault variables, and you need to validate them before runtime without decrypting them. You also want developers to review encrypted playbooks in pull requests without seeing the secrets. How do you handle this?
Use `ansible-playbook --syntax-check` which validates YAML syntax without requiring vault decryption. For security scanning, implement tools like Ansible-lint which also work without decryption. Create a CI/CD pipeline that: 1) runs syntax/lint checks on encrypted files, 2) decrypts only in secure CI/CD environment for testing, 3) never outputs decrypted values to logs. Implement pre-commit hooks that verify vault file integrity without decrypting. For pull request reviews, use git's diff attributes to obfuscate vault file diffs: add vault files to `.gitattributes` with a clean/smudge filter that shows only file metadata in diffs. Require manual approval from security team for vault file changes. Alternatively, store vault files outside git and reference them—this prevents accidental exposure through PRs entirely.
Follow-up: How would you implement a zero-trust secret management architecture where secrets are never stored in Ansible files at all?