Terraform State Interview Questions (tf-state)

Drift means the real-world infrastructure no longer matches what Terraform's state file records. It usually happens when someone changes cloud resources outside of Terraform — for example, through the cloud console or CLI — so terraform plan shows unexpected differences even though the .tf files haven't changed. Running terraform plan is the standard way to detect it.

A remote backend lets every team member share the same state file, so everyone works from one consistent view of the infrastructure. Many remote backends also support state locking, which helps prevent two people from applying changes at the same time.

True. A remote Terraform backend stores state in a shared location so a whole team works from one state file. Backends that support locking hold a lock during writes, so two concurrent runs cannot modify the state at the same time and corrupt it.

Most Terraform remote state backends support state locking, which prevents two Terraform operations from writing to the same state file at the same time.

The state file maps each resource address in your configuration to the real infrastructure object Terraform created (an instance id, a bucket ARN). Terraform reads it to compute the diff between desired and actual state, so it knows what to create, update, or destroy. It is not where you declare desired infrastructure (that's your .tf files), and it is not a credential or plugin cache.

Storing Terraform state in Git or another version control system is an anti-pattern. State files can hold secrets like passwords and access keys, and version control offers no state locking, so concurrent applies can corrupt state. Remote backends such as S3 with DynamoDB or Terraform Cloud avoid both problems.

In Terraform v1.x, the S3 backend marks only bucket and key as Required. region can come from environment variables or provider defaults, and dynamodb_table only enables state locking, so both are Optional. workspace_key_prefix has a default value and is also Optional.

A local state file means each engineer has their own copy, so the state diverges and is never authoritative; it also isn't backed up and can be lost or committed to git with secrets in it. A remote backend stores a single shared state in a central, durable location so the whole team plans and applies against the same source of truth. Crucially it enables state locking: while one apply holds the lock, others are blocked, preventing two people from writing state at once and corrupting it. It also gives durability/versioning (S3 versioning, backups) and keeps the (plaintext) state off laptops behind access controls and encryption at rest.

We would migrate all state to an S3 backend (one bucket per AWS account or a single bucket with KMS encryption and a strict key prefix scheme) fronted by a DynamoDB table for locking. Each team-environment pair gets its own state key path: for example s3://tf-state-prod/team-a/prod/terraform.tfstate. This gives us per-team, per-environment isolation at the object level.

For environment isolation we deliberately choose separate backend keys over Terraform Workspaces. Workspaces share a single backend configuration and make it trivial to accidentally run terraform apply against the wrong environment — the workspace is just a CLI flag. Separate keys let us enforce isolation through IAM: each team's CI role gets s3:GetObject/s3:PutObject and DynamoDB lock permissions only on its own key prefix and lock-table partition. A team-A role literally cannot read team-B's state object.

Locking is handled by the DynamoDB lock table. On every plan and apply, Terraform writes a lock item keyed by the S3 state path; if the item already exists, the run aborts with the lock ID. If a pipeline crashes without releasing the lock, the operator verifies (via CI logs and a DynamoDB DescribeTable check on lock age) that no run is genuinely in progress, then runs terraform force-unlock <lock-id> to remove the stale item.

For state recovery we enable S3 bucket versioning. If a state file is corrupted or manually edited, we identify the last known-good version via the S3 version list, download it locally, and run terraform state push -force good-state.tfstate. The -force flag is required because the good version has a lower serial than the current (corrupted) state; Terraform's serial check exists precisely to prevent silent state rollbacks, so we must acknowledge the override. All S3 and DynamoDB operations are captured in CloudTrail and S3 access logs, giving us an audit trail of who pushed the replacement state and when. As a post-incident step, we run terraform plan to confirm the restored state matches reality and investigate how the corruption occurred.

State Isolation

We replace the single shared S3 bucket + workspace-based isolation with a dedicated S3 bucket per team (e.g. tfstate-payment, tfstate-checkout). Within each team bucket, state keys are organized by environment: payment/dev/terraform.tfstate, payment/staging/terraform.tfstate, payment/prod/terraform.tfstate. This eliminates the collision class entirely: a misconfigured workspace name in one team's backend block cannot overwrite another team's state because the teams write to physically separate buckets. Workspaces are insufficient as a team-boundary isolation mechanism because they share the same backend configuration and S3 key prefix — if two teams accidentally use the same workspace name or state key, Terraform happily writes one state over the other with no warning.

Access Control & Encryption

Each team bucket uses SSE-KMS with a per-team KMS CMK (e.g. alias/tfstate-payment). The platform team owns the KMS keys and grants each team's IAM role kms:Decrypt, kms:Encrypt, and kms:GenerateDataKey on only their key. S3 bucket policies restrict s3:GetObject and s3:PutObject to the team's IAM role, scoped to the team's key prefix within the bucket. Since Terraform state is stored as a single S3 object per workspace, IAM policy conditions use the object key path in the resource ARN (e.g. arn:aws:s3:::tfstate-payment/payment/prod/terraform.tfstate) to enforce isolation. The platform team retains kms:CreateKey, kms:ScheduleKeyDeletion, and kms:RotateKey on all keys for centralized rotation. DynamoDB lock tables are either per-team or use a shared table with a per-team hash key namespace, with IAM policies scoped accordingly so that a team cannot acquire or release another team's lock.

Cross-Team State Sharing

Teams that need to consume another team's outputs use a terraform_remote_state data source:

data "terraform_remote_state" "payment" {
  backend = "s3"
  config = {
    bucket         = "tfstate-payment"
    key            = "payment/prod/terraform.tfstate"
    region         = "us-east-1"
    dynamodb_table = "tflock-payment"
  }
}

The consuming team's IAM role is granted read-only s3:GetObject on the specific state object and kms:Decrypt on the producing team's KMS key — no write access, no DynamoDB lock-table write access. terraform_remote_state exposes only the outputs block of the state, not raw resource attributes, so the producing team curates which values are shared by defining outputs explicitly. Importantly, the remote-state read does NOT acquire a state lock, so multiple consumers can read concurrently without blocking the producing team's applies.

Policy Enforcement & Audit

1. Config-level secret prevention: All CI/CD pipelines run tfsec or checkov scans that flag resources with sensitive attributes not marked sensitive = true, and tflint rules that reject output blocks exposing secrets. Additionally, a custom Sentinel policy (for Terraform Enterprise/Cloud users) or OPA policy (for open-source) checks that all variable and output declarations handling secrets use sensitive = true. These tools scan HCL configuration, not state file contents — they prevent secrets from leaking through outputs but cannot detect a secret already embedded in a resource attribute in state.
2. State-content secret scanning: Because config-level tools do not inspect state files, the platform team runs a scheduled job that pulls each team's state object via terraform state pull and pipes the JSON through trufflehog to detect plaintext secrets in resource attributes (e.g., a password hardcoded in a resource argument). If a secret is found, the job alerts the owning team and triggers credential rotation. This complements the config-level prevention above by catching secrets that slip through into state despite config policies.
3. Backend standards: The platform team publishes a versioned Terraform backend module that teams must use. This module hardcodes encryption settings, key naming conventions, and DynamoDB table configuration. A wrapper script or Makefile enforces terraform init -backend-config= with only the team-specific values, preventing manual misconfiguration.
4. Audit logging: AWS CloudTrail data events are enabled on every state bucket, logging every GetObject and PutObject call with the caller IAM principal, timestamp, and object key. This provides a tamper-resistant audit trail. S3 server access logs are also enabled as a secondary, lower-cost log for long-term retention, but CloudTrail is the primary audit mechanism because it integrates with SIEM tools and captures the IAM principal identity.

Zero-Downtime Migration

Migration is performed team-by-team in a phased rollout:

1. Prepare: The platform team creates the new per-team bucket, KMS key, and DynamoDB table for a single pilot team.
2. Pull: The team runs terraform state pull against the old shared backend, producing a local terraform.tfstate file.
3. Push: The team updates their backend configuration to point at the new bucket and runs terraform state push terraform.tfstate to write the state to the new backend.
4. Verify: The team runs terraform state pull on the new backend and diffs the JSON against the old state. The lineage UUID must match and the serial must be equal or higher. The team runs terraform plan — it should show no changes (or only the expected diff from the backend switch).
5. Cutover: The team's CI/CD pipeline is updated to use the new backend. The old state object in the shared bucket is preserved (not deleted) until all 50+ teams have confirmed migration.
6. Rollback: If any step fails, the team reverts their backend config to the old shared bucket — the old state object is untouched and immediately usable. No state loss occurs because state push to the new backend is additive; it does not delete from the old backend.

This process is repeated for each team, one at a time, with the platform team monitoring CloudTrail for any unexpected state access patterns during the migration window.