DevOps interview questions: secrets & config management

HashiCorp Vault is a dedicated secret management tool for securely storing, controlling access to, and dynamically generating secrets like API keys, passwords, and certificates. Jenkins (CI/CD), Prometheus (monitoring), and NGINX (web server/reverse proxy) serve entirely different purposes and are not designed for secret management.

Good secret management keeps credentials out of source code and Git history. Teams store secrets in a dedicated secrets manager and inject them at runtime, rotate them regularly and after any suspected exposure, and grant read access only to the workloads that need each secret.

The correct approach is to store the database password in a managed secrets manager such as AWS Secrets Manager or HashiCorp Vault and inject it into the pod at runtime via short-lived credentials, environment variables, or mounted volumes. This keeps the secret out of both version control and the container image. Base64-encoded Kubernetes Secrets are not encryption, private Git repos still leak, and baking secrets into an image persists them in an image layer.

The statement is false. A later commit that removes the key from the working files does not erase it from Git history, so the key remains visible in past commits and in every existing clone or fetch that already contains it. The key must still be treated as compromised: revoke it, rotate it, and optionally purge it from history.

Secret rotation is the practice of periodically (or on demand) replacing a secret, such as a database password, API key, or certificate, with a new value and retiring the old one. Rotation limits the window of exposure: if a credential leaks, it becomes useless after the next rotation, and rotation forces systems to fetch secrets dynamically rather than hard-coding them. Long-lived static credentials that never change mean a leak stays exploitable indefinitely and is hard to detect or revoke safely.

The cardinal anti-pattern is committing secrets into source control — even in a private repo they live forever in git history and leak through forks, clones, and CI logs. Related anti-patterns: hardcoding secrets in the application image so every copy of the artifact carries them, baking them into Dockerfiles or build args, printing them to logs, sharing one static credential across all environments, and never rotating them. A sound approach keeps secrets out of code and out of the artifact entirely. They live in a dedicated secret store or vault (or the cloud provider's secret manager), are injected at deploy/runtime via environment variables or mounted files, and are scoped per environment with least-privilege access controls and audit logging. Crucially, secrets should be rotatable — short-lived or dynamically issued credentials limit the damage of a leak — and a leaked secret must be revoked and rotated, not just deleted from the latest commit, because the history still holds it.

Zero-downtime database password rotation starts by creating a second valid credential in the database, then saving it to the secret store as the new version. Services are rolled out to read that version, and usage of the old credential is checked. Only after that is the old credential revoked, so no running instance is cut off mid-rotation.

Architecture Overview

Secret Store: Deploy HashiCorp Vault in a dedicated, highly-available cluster (Raft storage, 3-5 nodes) not co-located with any application cluster. Vault is the single source of truth for all secrets. Each application cluster runs the Vault Agent Injector (admission webhook) or the Secrets Store CSI Driver with the Vault provider.

Why not rely on native Kubernetes Secrets: Native K8s Secrets are merely base64-encoded, not encrypted — anyone with read access to the Secret API (or the kubelet on the node) sees plaintext. Enabling EncryptionConfiguration encrypts data at rest in etcd, but it does not prevent the API server or an authorized client from decrypting and reading the secret at runtime; it only protects the raw etcd snapshot or backup. Therefore this design minimizes secrets stored in etcd by delivering them directly from Vault to the pod's memory at runtime and eliminating standalone K8s Secret objects entirely during the migration. EncryptionConfiguration is enabled on all clusters as defense-in-depth, but the architecture does not depend on it.

Authentication Path (Pod → Vault): Each pod runs under a dedicated Kubernetes ServiceAccount. Vault's Kubernetes auth method is configured per cluster with a bound ServiceAccount name and namespace, so a pod's ServiceAccount JWT token authenticates it to Vault and maps to a specific Vault policy. On EKS we additionally use IRSA (IAM Roles for Service Accounts) so the Vault pods themselves authenticate via AWS IAM; on GKE we use Workload Identity. No human token is ever used by applications.

Secret Delivery — Sidecar Agent Injector: We use the Vault Agent Injector admission webhook. When a pod is annotated with vault.hashicorp.com/agent-inject: "true" and path annotations for each secret, the injector mutates the pod spec to add a Vault Agent sidecar. The sidecar authenticates using the pod's ServiceAccount token, fetches the secrets, and writes them to a shared memory volume (/vault/secrets/) as files. The application reads these files. For dynamic database credentials, the agent template renders the username/password into a file and runs a command hook (or the application watches the file) to reload connections. The agent handles lease renewal automatically; if renewal fails, it revokes the old lease, fetches a new credential, and rewrites the file — no pod restart needed.

Why sidecar over CSI or SDK: The CSI driver is simpler but doesn't support dynamic secret renewal without remount (requires pod restart in older versions). The SDK approach requires every service to implement Vault client logic, which is impractical across 40+ services in different languages. The agent sidecar is language-agnostic, handles renewal transparently, and writes to tmpfs (shared memory) so secrets never touch disk.

Encryption Hierarchy:

• Vault's master key is protected by Shamir's Secret Sharing (unseal keys held by separate human custodians) or, preferably, by a cloud KMS auto-unseal (AWS KMS or GCP KMS). The master key encrypts the Raft storage backend where all data (including encrypted DEKs) resides.
• Vault internally uses barrier encryption: every value written to storage is encrypted with an AES-256-GCM key derived from the master key. No plaintext is ever persisted.
• For transit secrets (e.g., application-level field encryption), Vault's Transit engine acts as a KEK service: applications send plaintext to Transit, receive ciphertext, and store only ciphertext in their databases. The DEK never leaves Vault.

Key Rotation Strategy:

• Cloud KMS keys (auto-unseal KEK): rotate annually via cloud provider key rotation; Vault supports re-wrapping the master key.
• Transit keys: configure auto_rotate_period (e.g., 30 days); old key versions remain available for decryption, new writes use the latest version.
• Dynamic DB credentials: TTL of 1 hour with max_ttl of 4 hours; the agent renews at TTL/2 (30 min) and fetches a new credential if renewal fails.
• ServiceAccount tokens: short-lived (1 hour) JWTs auto-mounted by Kubernetes; the Vault agent re-authenticates as needed.

Blast-Radius Isolation (Requirement 4)

1. Per-service policies: Each ServiceAccount maps to a Vault policy scoped to exactly its secret paths (e.g., secret/data/payments-service/*, database/creds/payments-service). A compromised pod's token can only read its own paths; Vault denies all other paths.
2. No shared ServiceAccounts: Every service has a unique ServiceAccount; no default SA is used.
3. Short TTLs: Dynamic credentials expire in 1 hour; the ServiceAccount JWT expires in 1 hour. A stolen token has a narrow window.
4. No secret caching beyond pod lifetime: The agent sidecar's token and leases are revoked when the pod terminates (via the vault.hashicorp.com/agent-inject-token-volume-mount lifecycle or a preStop hook that calls vault token revoke -self).
5. Network policies: Restrict egress from each namespace so a compromised pod cannot reach other clusters' Vault endpoints or other services' databases directly.
6. Response wrapping (optional): For especially sensitive secrets, use Vault response wrapping so the secret is delivered as a single-use wrapping token; even if intercepted, it can be unwrapped only once.

Auditability (Requirement 3)

Vault audit devices are enabled in both file and syslog modes. Every request (read, lease, renew, revoke) is logged with:

• The authenticated identity (K8s ServiceAccount name + namespace + cluster)
• The request path (which secret was accessed)
• The timestamp and operation type
• Whether the request was allowed or denied

Audit logs are streamed to a centralized log aggregator (e.g., Fluent Bit → Loki/Elasticsearch) and archived to an S3 bucket with Object Lock (WORM) and 1-year retention. For the 90-day requirement, an auditor queries the log store filtered by secret path to see every identity that accessed it. Because each ServiceAccount maps 1:1 to a service, the auditor can correlate access to a specific deployment and team owner via a ServiceAccount-to-team registry.

On EKS, CloudTrail captures all AWS Secrets Manager / KMS API calls as a secondary audit trail. On GKE, Cloud Audit Logs serve the same purpose.

Migration Plan (Zero Downtime)

Phase 1 — Inventory & Backfill (1-2 weeks): Scan all Helm charts, Kustomize overlays, and CI/CD pipelines for secrets (using tools like truffleHog, gitleaks). Catalog every secret with its owner service, type (API key, DB password, TLS cert), and current storage location. Backfill all static secrets into Vault KV v2 at secret/data/<service>/<key>. Verify ACLs are in place.

Phase 2 — Dual-Read Sidecar Deployment (2-3 weeks): For each service, add Vault Agent Injector annotations alongside existing env-var/K8s Secret mounts. The application is updated to read from the injected file (/vault/secrets/db-creds) as the primary source, with a fallback to the existing env var. Deploy via rolling update — no downtime. Both paths work simultaneously during this window.

Phase 3 — Static → Dynamic DB Credentials (1-2 weeks): Configure the Vault database secrets engine for each database. Update the agent template to render dynamic credentials instead of static KV values. The application's connection pool must handle credential changes (either via a file-watch reload or a health-check-driven reconnect). Roll out per-service with canary deployments.

Phase 4 — Cleanup (1 week): After monitoring confirms all services read exclusively from Vault, remove the old Kubernetes Secret objects and Helm values references. Run a final truffleHog scan on all repos to confirm no secrets remain in Git. Update CI/CD pipelines to never inject secrets directly — only Vault references.

Rollback: At any phase, if issues arise, the application falls back to env-var/K8s Secret values which remain in place until Phase 4. No big-bang cutover is required.

In envelope encryption, each secret is sealed with its own freshly generated data encryption key, and that key is then wrapped (encrypted) by a key encryption key that never leaves the KMS. Name the two keys: the first is the per-object key that encrypts the payload, the second is the key that wraps it.

Storage: keep a single system of record, such as HashiCorp Vault or a cloud secret manager, with encryption at rest and a KMS-backed root key. Git holds only references and policy, never values. Each secret has an owner and a path, and access policies are written per workload identity, so reads are least-privilege.

Authentication: each pod authenticates with its Kubernetes service account token (or a SPIFFE SVID) through the broker's Kubernetes auth method, which validates the token against the cluster's API. CI jobs use OIDC federation from the CI provider to obtain a short-lived token, so no long-lived CI secret is stored. The initial trust comes from the platform's attestation of the pod or job identity, not from a secret placed in the image.

Dynamic credentials: database users and cloud role sessions are issued on demand by the broker with a short TTL (for example one hour), renewed by the sidecar or SDK, and revoked automatically on expiry. No static database password exists to leak. Third-party API keys that the provider cannot mint dynamically are stored as versioned secrets. Rotation creates version N+1 while N stays valid, workloads reference the latest version and reload on change, and N is revoked only after metrics show no requests using it.

Audit and detection: the broker logs every issuance and read with the authenticated identity, and logs are shipped to a central SIEM. Secret scanners run on every commit, build log, and artifact, and a hit triggers automatic alerting and revocation of the matched key through the provider's API. Applying this to the leaked key: the scanner would have flagged the build log within minutes of the first build, the revocation path would have invalidated the key, and the audit trail would show which identities had used it.