Terraform State Locking Under Concurrent CD: Race Conditions, Backend Contention, and the Partial-Apply Trap

Terraform's state locking mechanism is documented as a mutual-exclusion guarantee. In practice it is a distributed coordination protocol with failure modes that most CD pipelines never exercise in staging but hit regularly in production. Understanding what the lock actually protects, what it does not protect, and how partial applies corrupt drift detection is prerequisite knowledge for any team running Terraform inside a real delivery system.

What the Lock Actually Serializes

Terraform state backends that support locking—S3 with DynamoDB, GCS, Azure Blob, Terraform Cloud—implement optimistic or pessimistic locking at the state file level, not at the resource graph level. The lock prevents two apply invocations from writing conflicting state simultaneously. It does not prevent two plans from reading stale state and generating divergent diffs before either acquires the lock.

The race window looks like this:

Pipeline A: plan ──────────────────────────── acquire lock ── apply ── release
Pipeline B:      plan ────────────── acquire lock ── blocked ───────────────── apply (with stale plan)

Pipeline B planned against state snapshot T₀. Pipeline A applied and wrote state T₁. Pipeline B acquires the lock after A releases it and applies the diff it computed against T₀. That diff is now wrong. It may add resources already present, or more dangerously, it may omit changes to resources Pipeline A modified, leaving those resources in a state Terraform believes is managed but whose actual configuration no longer matches the recorded intent.

This is the partial-apply trap: not a failed apply, but a succeeded apply that produces divergence the state file does not expose.

DynamoDB Locking Mechanics and the Conditional Write

With the S3 backend, Terraform acquires a lock by performing a PutItem with a ConditionExpression requiring the lock item's absence. The item contains a JSON blob with the lock ID, who holds it, and when it was acquired.

{
  "LockID": {"S": "s3://bucket/env/terraform.tfstate"},
  "Info":   {"S": "{\"ID\":\"uuid\",\"Operation\":\"OperationTypeApply\",\"Who\":\"runner@ci\",\"Version\":\"1.7.0\",\"Created\":\"2026-09-30T10:00:00Z\",\"Path\":\"s3://bucket/env/terraform.tfstate\"}"}
}

On apply completion, Terraform deletes that item and writes the new state to S3. The critical ordering issue: S3 PutObject is not atomic with DynamoDB DeleteItem. A crash or runner termination between the two leaves the lock released but the state file potentially mid-write, or worse, the lock persists while the state file was already written. The latter is recoverable via terraform force-unlock. The former—state written but not fully flushed—can leave a corrupted or partial JSON state file that Terraform will parse as valid if the S3 multipart upload completed but the content is a prior version due to eventual consistency between regions.

Concurrent CD: The Real Topology

Short-lived CD runners, auto-scaling build fleets, and merge-triggered pipelines create concurrency patterns that single-workspace Terraform is not designed for. Consider a monorepo where infrastructure modules are shared across environments and three PRs merge within sixty seconds of each other. Each triggers a pipeline. Each pipeline runs terraform plan against the same workspace. The DynamoDB lock gates apply, so pipelines queue. But by the time the third pipeline's apply runs, its plan reflects state from two applies ago.

The practical mitigations break into three categories:

1. Plan Invalidation at Apply Time

Run terraform plan -out=plan.tfplan and capture the state serial embedded in the binary plan file. Before executing apply plan.tfplan, re-read the current state serial from the backend and compare. If the serial advanced since the plan was captured, abort and re-plan. This is not a feature Terraform exposes natively; it requires wrapping the apply step.

A Go service implementing this gate fetches state from S3, deserializes the serial field, and compares:

type TFState struct {
    Serial  int64  `json:"serial"`
    Version int    `json:"version"`
    // ...
}

func stateSerialFromS3(ctx context.Context, client *s3.Client, bucket, key string) (int64, error) {
    out, err := client.GetObject(ctx, &s3.GetObjectInput{
        Bucket: aws.String(bucket),
        Key:    aws.String(key),
    })
    if err != nil {
        return 0, fmt.Errorf("fetch state: %w", err)
    }
    defer out.Body.Close()

    var st TFState
    if err := json.NewDecoder(out.Body).Decode(&st); err != nil {
        return 0, fmt.Errorf("decode state: %w", err)
    }
    return st.Serial, nil
}

func applyIfSerialUnchanged(ctx context.Context, plannedSerial, bucket, key string) error {
    current, err := stateSerialFromS3(ctx, s3Client, bucket, key)
    if err != nil {
        return err
    }
    parsed, _ := strconv.ParseInt(plannedSerial, 10, 64)
    if current != parsed {
        return fmt.Errorf("state serial advanced from %d to %d: re-plan required", parsed, current)
    }
    return runTerraformApply(ctx)
}

This gate runs before terraform apply inside the CD step. A serial mismatch fails the pipeline with a re-plan signal rather than silently applying a stale diff.

2. Workspace Isolation per Environment Tier

Sharing a single workspace across all environment applies is the root structural problem. Terraform workspaces provide isolated state files and independent lock scopes. Mapping one workspace per environment per service eliminates cross-environment lock contention entirely. The cost is state sprawl and the operational burden of workspace lifecycle management, but for teams with more than two environments and any meaningful CD velocity, the tradeoff is clear.

The anti-pattern is using workspaces as a feature flag substitute—one workspace with var.environment = "staging" switching resource names. That collapses the isolation boundary back to a single state file and a single lock.

3. Serialized Apply Queues at the Orchestration Layer

For modules that genuinely must be shared—VPC, IAM, DNS—apply serialization belongs at the pipeline orchestration layer, not inside Terraform. An SQS FIFO queue per workspace, with a single consumer running applies in order, eliminates concurrent apply attempts. Plans are enqueued with their captured serial. The consumer validates serial freshness before dequeuing an apply. Stale plans are discarded and requeuability is signaled back to the triggering pipeline.

This architecture accepts higher apply latency in exchange for correctness. Under bursty merge activity, the queue depth becomes a leading indicator of infrastructure delivery lag—worth exposing as a metric.

The Drift Detection Corruption Problem

Terraform's plan output diffing and terraform show rely on the state file being an accurate record of what was last applied. When a partial-apply trap succeeds—apply returns exit code 0 but the applied diff was computed against a stale snapshot—the resulting state serial increments and the new state file records what Terraform intended to apply, not necessarily what the cloud API accepted.

This creates a class of drift that terraform plan will not surface on the next run: resources that exist in a configuration Terraform has recorded as authoritative, but whose actual cloud state differs from the recorded state in ways Terraform now believes are intentional. Drift detection tools running against the state file will agree with Terraform. The divergence is invisible until an incident surfaces it.

Mitigation is a post-apply validation step: after every apply, run terraform plan again and assert that the output contains zero changes. Any non-zero diff after a successful apply indicates the apply was computed against stale state. Fail the pipeline, create an incident, and require a human to reconcile before the next automated apply is permitted against that workspace.

Decision Framework

Select your locking and concurrency strategy against these dimensions:

Condition	Approach
Single environment, low CD velocity	Default DynamoDB lock; accept queue
Multiple environments, shared module	One workspace per environment; separate lock scopes
High CD velocity, merge bursts	FIFO queue per workspace; serial validation gate
Shared foundational infrastructure	Manually gated applies; no automated concurrent apply
Any apply pipeline	Post-apply zero-diff assertion; fail on non-zero

Plan staleness is not a Terraform bug. It is a distributed coordination problem: a plan is a snapshot of intent computed at time T, and the world it describes may have changed by time T+n. Treating the lock as sufficient to guarantee apply correctness is the architectural mistake. The lock prevents concurrent writes. It does not prevent applying a diff that was correct when computed and wrong when executed. Making that distinction operational—in your pipeline code, your validation steps, and your incident response—is what separates infrastructure delivery that is merely automated from infrastructure delivery that is safe.