DevOps Interview Questions: Config and Secrets Management

Store secrets such as database passwords and API keys in a dedicated secrets manager like HashiCorp Vault or AWS Secrets Manager, and inject them into the application at runtime. This keeps secrets encrypted at rest, access-controlled via policies, and out of source control, whereas hard-coding them or committing plaintext config files exposes credentials and makes rotation difficult.

Good secret management keeps credentials out of source code and Git history. Teams store secrets in a dedicated secrets manager and inject them at runtime, rotate them regularly and after any suspected exposure, and grant read access only to the workloads that need each secret.

The statement is false. A later commit that removes the key from the working files does not erase it from Git history, so the key remains visible in past commits and in every existing clone or fetch that already contains it. The key must still be treated as compromised: revoke it, rotate it, and optionally purge it from history.

postgres://prod-db/app — the spread merges process.env last, so the real environment variable overrides the committed default, which is the twelve-factor pattern

Secret rotation is the practice of periodically (or on demand) replacing a secret, such as a database password, API key, or certificate, with a new value and retiring the old one. Rotation limits the window of exposure: if a credential leaks, it becomes useless after the next rotation, and rotation forces systems to fetch secrets dynamically rather than hard-coding them. Long-lived static credentials that never change mean a leak stays exploitable indefinitely and is hard to detect or revoke safely.

Both operations assume a clean host: useradd app errors (and aborts under set -e) when the user already exists, and the >> append adds a duplicate CONFIG_DIR line every run — the script must check-then-act (e.g. id app || useradd app) and write the line only if absent, so repeated runs converge to the same state

The cardinal anti-pattern is committing secrets into source control — even in a private repo they live forever in git history and leak through forks, clones, and CI logs. Related anti-patterns: hardcoding secrets in the application image so every copy of the artifact carries them, baking them into Dockerfiles or build args, printing them to logs, sharing one static credential across all environments, and never rotating them. A sound approach keeps secrets out of code and out of the artifact entirely. They live in a dedicated secret store or vault (or the cloud provider's secret manager), are injected at deploy/runtime via environment variables or mounted files, and are scoped per environment with least-privilege access controls and audit logging. Crucially, secrets should be rotatable — short-lived or dynamically issued credentials limit the damage of a leak — and a leaked secret must be revoked and rotated, not just deleted from the latest commit, because the history still holds it.

We would use a tiered approach: non-sensitive configuration (feature flags, endpoint URLs, resource defaults) stored in a GitOps-managed ConfigMap repository (e.g., Argo CD + Kustomize overlays per environment), while all secrets are stored in HashiCorp Vault (or AWS Secrets Manager if already on AWS).

Storage & Encryption (c1): Vault provides encryption at rest via its storage backend (e.g., integrated Raft with AES-256), Shamir's secret sharing for unseal keys, and a sealed/unsealed model. Raw Kubernetes Secrets are only base64-encoded — they are not encrypted at rest unless you configure a KMS provider with encryption-at-rest (which many clusters do not enable). Relying on raw K8s Secrets fails SOC 2 requirements for encryption-at-rest and granular audit. We would use Vault as the source of truth and sync secrets into K8s Secrets only as a transient delivery vehicle, or mount them directly via CSI.

Environment Isolation & Access Control (c2): Each environment gets its own Vault namespace (or path prefix, e.g., secret/dev/, secret/staging/, secret/prod/). Each cluster authenticates to Vault via Kubernetes auth backend using its own service account token, bound to a specific Vault policy that grants read access only to that environment's path. IAM/RBAC policies enforce least-privilege: a dev cluster's Vault auth role can only read secret/dev/*. All secret accesses are logged via Vault's audit devices (file or syslog) to a tamper-evident SIEM. Developers are granted read access to dev and staging paths but never prod. To allow debugging config issues without exposing secret values, non-sensitive config lives in Git (visible to all), and for staging secrets we can provide a Vault policy that allows read on metadata but not on secret data, or use a break-glass workflow with just-in-time access and automatic expiration.

Secret Delivery (c3): We use the Secrets Store CSI Driver with the Vault provider. At pod startup, the CSI driver mounts secrets as files into a tmpfs volume, fetching them from Vault using the pod's service account. This works seamlessly for scale-to-zero services because the mount happens during pod initialization — there is no long-lived sidecar process that must be running before the pod can accept traffic. An alternative like the Vault Agent sidecar adds a startup dependency (the sidecar must fetch secrets before the main container starts), which adds latency to cold starts. The External Secrets Operator is also viable: it periodically syncs Vault secrets into standard K8s Secrets, which pods consume normally — this decouples pod startup from Vault availability but introduces a window where K8s Secrets may be stale.

Rotation & Propagation (c4): For database credentials, we use Vault dynamic secrets: Vault creates short-lived database users with configurable TTLs (e.g., 1 hour). The CSI driver re-fetches the lease before expiry. For services using the External Secrets Operator sync model, rotation triggers a K8s Secret update, and we configure a Reloader or Argo CD to issue a rollout restart when the secret changes. For shared credentials used by multiple services (e.g., a third-party API key), we prefer per-service credentials where possible; if truly shared, rotation means updating the secret in Vault and triggering a coordinated rollout restart across all consuming deployments, potentially with a brief overlap where both old and new credentials are valid (dual-key rotation pattern).

Zero-downtime database password rotation starts by creating a second valid credential in the database, then saving it to the secret store as the new version. Services are rolled out to read that version, and usage of the old credential is checked. Only after that is the old credential revoked, so no running instance is cut off mid-rotation.

In envelope encryption, each secret is sealed with its own freshly generated data encryption key, and that key is then wrapped (encrypted) by a key encryption key that never leaves the KMS. Name the two keys: the first is the per-object key that encrypts the payload, the second is the key that wraps it.