Terraform Interview Questions: Remote State & tfstate Management

True. A remote Terraform backend stores state in a shared location so a whole team works from one state file. Backends that support locking hold a lock during writes, so two concurrent runs cannot modify the state at the same time and corrupt it.

A remote backend lets every team member share the same state file, so everyone works from one consistent view of the infrastructure. Many remote backends also support state locking, which helps prevent two people from applying changes at the same time.

Most Terraform remote state backends support state locking, which prevents two Terraform operations from writing to the same state file at the same time.

A Terraform remote state backend lets the whole team share one authoritative state file and, when the backend supports locking, prevents concurrent writes from corrupting it. It does not automatically guarantee encryption, remove the need for plan, or store your module source code — those are separate concerns.

A single Terraform root module can declare only one backend block. That is why teams that need data from another backend use terraform_remote_state data sources instead. Remote backends are preferred over version control because state files may contain secrets and version control does not provide state locking.

Storing Terraform state in Git or another version control system is an anti-pattern. State files can hold secrets like passwords and access keys, and version control offers no state locking, so concurrent applies can corrupt state. Remote backends such as S3 with DynamoDB or Terraform Cloud avoid both problems.

Migrating to a remote backend such as S3 with DynamoDB solves concurrent-write corruption. Local state has no locking, so two engineers running terraform apply at once can overwrite each other's state updates. Remote backends add a lock, such as a DynamoDB table for the S3 backend, so only one write proceeds at a time.

False. The Terraform S3 backend does not lock state out of the box. Locking must be enabled: with Terraform 1.10 and later, set use_lockfile = true to use a lock file in the bucket; older versions used a DynamoDB table via dynamodb_table. Without locking, concurrent applies can overwrite each other's state.

Terraform plans to destroy the old resource at aws_s3_bucket.my_bucket and create a new resource at aws_s3_bucket.website_bucket. State is matched to configuration by resource address, so the old address no longer appears in the configuration and is marked for destruction, while the new address appears for the first time and is marked for creation. On apply, the bucket tracked by the old address would be destroyed (unless a prevent_destroy lifecycle guard blocks it) and a new bucket created under the new address. Running terraform state mv before planning would rewrite the state entry to the new address and avoid the destroy-and-create.

In Terraform v1.x, the S3 backend marks only bucket and key as Required. region can come from environment variables or provider defaults, and dynamodb_table only enables state locking, so both are Optional. workspace_key_prefix has a default value and is also Optional.

To configure remote state in Terraform, you declare a backend block of type s3 inside the terraform configuration block. For the S3 backend, the three arguments bucket, key, and region are required at minimum to specify the bucket name, the state file path, and the AWS region, respectively. Use the Terraform argument names.

Terraform fails to acquire the state lock and returns an error, so your terraform apply does not proceed. When a backend supports locking (e.g., S3 with a DynamoDB lock table), every state read/write acquires an exclusive lock. A concurrent operation finds the lock already held and fails fast rather than queuing or merging. You can inspect the lock with the reported lock ID and, if it is genuinely stale, use terraform force-unlock to release it.

The terraform_remote_state data source in Terraform exposes only the root-module output values recorded in another configuration's state, and it does so read-only. It does not give access to managed resources, their full attributes, or any write capability. Managing those resources requires importing them into your own state or sharing the module that defines them.

The DynamoDB table holds a distributed lock that Terraform acquires before any state-modifying operation such as terraform apply or terraform destroy. This prevents concurrent runs, from multiple team members or CI pipelines, from writing to the same state file at once, which could corrupt the state or produce conflicting infrastructure changes. If a run is interrupted and the lock is left behind, terraform force-unlock <lock-id>, using the lock ID shown in the lock error message, releases it manually.

In Configuration A, declare output blocks for the VPC ID, subnet IDs, and security group IDs. In Configuration B, add a terraform_remote_state data source that points at Configuration A's S3 backend, using Configuration A's bucket, state key, and region (the DynamoDB lock table is not needed to read outputs). Then reference the values as data.terraform_remote_state.networking.outputs.vpc_id, and similarly for the subnet and security group outputs. This keeps the two state files independent and exposes only the explicitly declared outputs as read-only data.

data "terraform_remote_state" "network" { backend = "s3" config = { bucket = "<networking-state-bucket>" key = "network/terraform.tfstate" region = "<region>" } }

Reference the VPC ID as data.terraform_remote_state.network.outputs.vpc_id. The terraform_remote_state data source reads the outputs recorded in the networking configuration's state file, so the application configuration gets the value the networking team published. Each output is nested under the outputs attribute, so omitting .outputs produces an invalid path.

Use an S3 backend in a dedicated, versioned, access-logged state bucket, with a lock table for concurrency control.

Isolation: give every environment-team pair its own state file through the key path, for example envs/prod/networking/terraform.tfstate and envs/dev/database/terraform.tfstate. Backend blocks do not accept variables, so each team's configuration hard-codes its own key, or the key is supplied per environment with terraform init -backend-config=envs/prod.hcl. Separate files mean a plan or apply by one team never touches another team's state.

Locking: configure dynamodb_table = "tf-state-locks" (partition key LockID, type String) so Terraform writes a lock row before any state-changing operation. On Terraform 1.10 and later, use_lockfile = true offers S3-native locking as an alternative; either way a concurrent apply on the same state fails instead of corrupting it.

Encryption: set encrypt = true in the backend, and for production set kms_key_id to a customer-managed KMS key so key policy controls who can decrypt state. Plain encrypt = true uses S3 SSE-S3 by default, so the KMS key is what adds granular control. Enable bucket versioning and block public access as well.

Least privilege: give each team's IAM role s3:GetObject, s3:PutObject and s3:DeleteObject only on its own key prefix (arn:aws:s3:::acme-tf-state/envs/*/networking/* for the networking role), plus s3:ListBucket restricted with an s3:prefix condition. Scope DynamoDB GetItem, PutItem and DeleteItem to the lock table, using a dynamodb:LeadingKeys condition so each team can only lock its own state path. Grant kms:Decrypt and kms:Encrypt on the CMK only to the roles that need them.

Cross-team sharing: the consumer reads the producer's state with the terraform_remote_state data source, for example

data "terraform_remote_state" "network" {
  backend = "s3"
  config = {
    bucket = "acme-tf-state"
    key    = "envs/prod/networking/terraform.tfstate"
    region = "us-east-1"
  }
}

then references data.terraform_remote_state.network.outputs.vpc_id. The networking team exposes only intended values through output blocks, and consumers are granted read access to that one state object. Be explicit about the caveat: this data source reads the whole state file, so granting it exposes every value in that state including secrets. For sensitive or high-volume sharing, the producer should instead publish the values to a store such as SSM Parameter Store, and consumers should read them with the matching data source, keeping the state file itself private.

A local state file means each engineer has their own copy, so the state diverges and is never authoritative; it also isn't backed up and can be lost or committed to git with secrets in it. A remote backend stores a single shared state in a central, durable location so the whole team plans and applies against the same source of truth. Crucially it enables state locking: while one apply holds the lock, others are blocked, preventing two people from writing state at once and corrupting it. It also gives durability/versioning (S3 versioning, backups) and keeps the (plaintext) state off laptops behind access controls and encryption at rest.

State isolation: give every team-and-environment pair its own state object, for example s3://tf-state-prod/teams/{team}/{env}/terraform.tfstate, with prefix-level separation (or separate buckets per environment). A terraform destroy run by one team then only touches that team's state, and shared infrastructure such as a platform-owned VPC is exposed through outputs read with the terraform_remote_state data source, not written by other teams. Locking: enable state locking on the S3 backend so concurrent applies cannot write the same state. Use a DynamoDB table (partition key LockID) via dynamodb_table, or on Terraform 1.10+ use the native S3 lock file (use_lockfile = true) and retire the DynamoDB table. Access control: give each team an IAM role whose s3:GetObject, s3:PutObject, s3:DeleteObject permissions are scoped to its own key prefix, s3:ListBucket is limited to that prefix with a condition key (Terraform needs ListBucket for init and state-version listing), and lock-table actions are limited to the needed DynamoDB operations; there is no cross-team write access, and read-only access is granted for shared outputs. Centralize backend configuration in a shared module or Terragrunt so teams cannot misconfigure their backend. Disaster recovery: enable S3 versioning on every state bucket so prior state versions can be restored after accidental deletion or corruption, enable cross-region replication to a backup region (with a KMS key available there), encrypt state with SSE-KMS, enable MFA Delete so versions cannot be permanently removed without MFA, and set lifecycle rules that retain noncurrent versions for a defined period.

DynamoDB is the AWS service typically used for state locking with the S3 backend, configured through the dynamodb_table attribute. State locking prevents concurrent terraform apply (or other state-writing) operations from reading and writing the same state file at the same time, which would otherwise cause lost updates or state corruption. (Newer Terraform versions can also lock using a lock file in S3 via use_lockfile, but DynamoDB is the traditional and most widely used choice.)

Because internal nodes hold only keys (not data), more keys fit in each node, increasing the tree's fanout. A higher fanout means a shallower tree, so point lookups require fewer disk I/O operations to traverse from root to leaf. Second, because all data values reside in the linked leaf nodes, range queries and sequential scans can be served by following the leaf-level pointers after a single root-to-leaf descent, without repeatedly walking back up and down the tree.

Terraform verifies that the local state is a valid successor of the remote state before overwriting it. It checks that both states share the same lineage (a UUID assigned when the state is first created) and that the local state's serial number is greater than or equal to the remote state's serial number. If either check fails, meaning a different lineage or a local serial lower than the remote serial, the push is refused unless the -force flag is provided. This guards against overwriting a newer remote state with a stale local copy, which could cause Terraform to lose track of resources and produce drift.

State Isolation

We replace the single shared S3 bucket + workspace-based isolation with a dedicated S3 bucket per team (e.g. tfstate-payment, tfstate-checkout). Within each team bucket, state keys are organized by environment: payment/dev/terraform.tfstate, payment/staging/terraform.tfstate, payment/prod/terraform.tfstate. This eliminates the collision class entirely: a misconfigured workspace name in one team's backend block cannot overwrite another team's state because the teams write to physically separate buckets. Workspaces are insufficient as a team-boundary isolation mechanism because they share the same backend configuration and S3 key prefix — if two teams accidentally use the same workspace name or state key, Terraform happily writes one state over the other with no warning.

Access Control & Encryption

Each team bucket uses SSE-KMS with a per-team KMS CMK (e.g. alias/tfstate-payment). The platform team owns the KMS keys and grants each team's IAM role kms:Decrypt, kms:Encrypt, and kms:GenerateDataKey on only their key. S3 bucket policies restrict s3:GetObject and s3:PutObject to the team's IAM role, scoped to the team's key prefix within the bucket. Since Terraform state is stored as a single S3 object per workspace, IAM policy conditions use the object key path in the resource ARN (e.g. arn:aws:s3:::tfstate-payment/payment/prod/terraform.tfstate) to enforce isolation. The platform team retains kms:CreateKey, kms:ScheduleKeyDeletion, and kms:RotateKey on all keys for centralized rotation. DynamoDB lock tables are either per-team or use a shared table with a per-team hash key namespace, with IAM policies scoped accordingly so that a team cannot acquire or release another team's lock.

Cross-Team State Sharing

Teams that need to consume another team's outputs use a terraform_remote_state data source:

data "terraform_remote_state" "payment" {
  backend = "s3"
  config = {
    bucket         = "tfstate-payment"
    key            = "payment/prod/terraform.tfstate"
    region         = "us-east-1"
    dynamodb_table = "tflock-payment"
  }
}

The consuming team's IAM role is granted read-only s3:GetObject on the specific state object and kms:Decrypt on the producing team's KMS key — no write access, no DynamoDB lock-table write access. terraform_remote_state exposes only the outputs block of the state, not raw resource attributes, so the producing team curates which values are shared by defining outputs explicitly. Importantly, the remote-state read does NOT acquire a state lock, so multiple consumers can read concurrently without blocking the producing team's applies.

Policy Enforcement & Audit

1. Config-level secret prevention: All CI/CD pipelines run tfsec or checkov scans that flag resources with sensitive attributes not marked sensitive = true, and tflint rules that reject output blocks exposing secrets. Additionally, a custom Sentinel policy (for Terraform Enterprise/Cloud users) or OPA policy (for open-source) checks that all variable and output declarations handling secrets use sensitive = true. These tools scan HCL configuration, not state file contents — they prevent secrets from leaking through outputs but cannot detect a secret already embedded in a resource attribute in state.
2. State-content secret scanning: Because config-level tools do not inspect state files, the platform team runs a scheduled job that pulls each team's state object via terraform state pull and pipes the JSON through trufflehog to detect plaintext secrets in resource attributes (e.g., a password hardcoded in a resource argument). If a secret is found, the job alerts the owning team and triggers credential rotation. This complements the config-level prevention above by catching secrets that slip through into state despite config policies.
3. Backend standards: The platform team publishes a versioned Terraform backend module that teams must use. This module hardcodes encryption settings, key naming conventions, and DynamoDB table configuration. A wrapper script or Makefile enforces terraform init -backend-config= with only the team-specific values, preventing manual misconfiguration.
4. Audit logging: AWS CloudTrail data events are enabled on every state bucket, logging every GetObject and PutObject call with the caller IAM principal, timestamp, and object key. This provides a tamper-resistant audit trail. S3 server access logs are also enabled as a secondary, lower-cost log for long-term retention, but CloudTrail is the primary audit mechanism because it integrates with SIEM tools and captures the IAM principal identity.

Zero-Downtime Migration

Migration is performed team-by-team in a phased rollout:

1. Prepare: The platform team creates the new per-team bucket, KMS key, and DynamoDB table for a single pilot team.
2. Pull: The team runs terraform state pull against the old shared backend, producing a local terraform.tfstate file.
3. Push: The team updates their backend configuration to point at the new bucket and runs terraform state push terraform.tfstate to write the state to the new backend.
4. Verify: The team runs terraform state pull on the new backend and diffs the JSON against the old state. The lineage UUID must match and the serial must be equal or higher. The team runs terraform plan — it should show no changes (or only the expected diff from the backend switch).
5. Cutover: The team's CI/CD pipeline is updated to use the new backend. The old state object in the shared bucket is preserved (not deleted) until all 50+ teams have confirmed migration.
6. Rollback: If any step fails, the team reverts their backend config to the old shared bucket — the old state object is untouched and immediately usable. No state loss occurs because state push to the new backend is additive; it does not delete from the old backend.

This process is repeated for each team, one at a time, with the platform team monitoring CloudTrail for any unexpected state access patterns during the migration window.

The lineage field is a UUID fixed when a state file is created, and serial is a counter that increases on each state write. During terraform state push, a local state with a lower serial than the remote state is rejected, which blocks stale overwrites independently of locking.