Terraform Module Design That Survives Three Years of Drift

We inherited a Terraform module that nobody on the team would touch. It provisioned the entire VPC, subnets, NAT gateways, route tables, and a tangle of security groups for our primary AWS account. It was three years old. It had 47 input variables, 19 of which were marked # DO NOT CHANGE in comments, and a count block on the NAT gateway that, when flipped, would silently destroy and recreate every private subnet route in the account. The last engineer who understood it had left 14 months earlier.

The module worked. That was the problem. It worked exactly as long as nobody ran terraform plan against it. The moment someone did, the plan showed 31 resources to change and 8 to replace, none of which anyone had requested. The state had drifted from the code, the code had drifted from reality, and the provider had been upgraded twice underneath it without anyone re-running anything. The module had become a write-only artifact: you could append to it, but you could not safely apply it.

This is the slow death of most Terraform codebases. Not a dramatic outage, just an accumulation of small decisions that make the code progressively more dangerous to run until running it becomes a quarterly event preceded by a war room. This post is about the design choices that keep a module applyable in year three, not just year one. Most of them are unglamorous. All of them are things I wish someone had told me before I wrote a 900-line main.tf.

Drift is the default state, not the exception

Before the design rules, the mental model. Terraform's promise is that your code is the source of truth and apply reconciles the world to match it. In practice, three forces pull the world away from your code continuously.

Out-of-band changes. Someone clicks something in the console during an incident. An auto-scaling group changes its desired count. AWS adds a default tag to a resource type. A security team bolts on a managed rule. None of these go through your pipeline, and all of them show up as diffs the next time you plan.

Provider evolution. The AWS provider ships breaking changes between major versions. Attributes get deprecated, defaults change, resources get split. An aws_s3_bucket that managed its own ACL in provider 3.x needs a separate aws_s3_bucket_acl resource in 4.x. Your code did not change, but its meaning did.

Refactoring entropy. Every time you split a module, rename a resource, or extract a variable, you risk a state/code mismatch unless you migrate state deliberately. Most teams do not, so they accumulate orphaned state entries and resources Terraform wants to recreate because it lost track of their address.

A module survives three years not by preventing drift, because you cannot, but by making drift cheap to detect and safe to reconcile. Every design choice below serves that.

Rule 1: One module does one thing, and that thing has a lifecycle

The inherited VPC module failed because it bundled resources with completely different lifecycles. A VPC CIDR block changes approximately never. NAT gateways change when you optimize cost. Security group rules change weekly. Bundling all three into one module means every weekly security group change forces a plan that also evaluates the never-changing VPC and the occasionally-changing NAT setup. The blast radius of a routine change is the entire network stack.

The heuristic I use: a module should group resources that are created together, destroyed together, and changed at roughly the same frequency. If two resources have wildly different change frequencies, they belong in different modules.

For the VPC, that means splitting along lifecycle seams:

# modules/network-foundation  -- changes ~never
#   VPC, CIDR, base subnets, IGW

# modules/network-egress      -- changes when optimizing cost
#   NAT gateways, EIPs, private route tables

# modules/network-security    -- changes weekly
#   security groups, NACLs, rules

This is not micro-modularization for its own sake. The test is concrete: when a security group rule changes, the plan should touch only network-security. If it touches the VPC, your seams are wrong.

The trade-off is that you now have inter-module dependencies to wire. The egress module needs the VPC ID and subnet IDs from the foundation module. The temptation is to make foundation a child module of egress so the wiring is implicit. Resist it. Implicit wiring through nested modules is exactly how you end up with a 900-line main.tf again. Wire them explicitly at the root, passing outputs to inputs:

module "foundation" {
  source   = "./modules/network-foundation"
  vpc_cidr = "10.40.0.0/16"
  az_count = 3
}

module "egress" {
  source             = "./modules/network-egress"
  vpc_id             = module.foundation.vpc_id
  private_subnet_ids = module.foundation.private_subnet_ids
  nat_strategy       = "single" # one NAT for non-prod, per-az for prod
}

The dependency graph is visible at the root. Someone reading the root config can see what depends on what without descending into module internals.

Rule 2: Variables describe intent, not implementation

The 47-variable module had inputs like enable_nat_gateway_eip_allocation and route_table_propagation_count. These are implementation knobs. They leak the module's internal structure to the caller, which means the caller now depends on that structure. Change the internals and you break every caller, even when the externally observable behavior is identical.

A good input variable describes what the caller wants, not how the module achieves it. Compare:

# Leaky: caller has to understand NAT internals
variable "nat_gateway_count"        { type = number }
variable "nat_eip_allocation_ids"   { type = list(string) }
variable "private_route_table_count" { type = number }

# Intent-revealing: caller states the outcome they want
variable "nat_strategy" {
  type        = string
  description = "How to provide outbound internet for private subnets."
  default     = "per_az"
  validation {
    condition     = contains(["none", "single", "per_az"], var.nat_strategy)
    error_message = "nat_strategy must be one of: none, single, per_az."
  }
}

With the intent-revealing version, the module owns the decision of how many NAT gateways and EIPs and route tables to create. The caller says per_az and the module figures out the rest. Three years later, when you find a cheaper NAT arrangement, you change the module internals and every caller who said per_az gets the improvement for free. The intent did not change.

The validation block is not optional. An invalid value caught at plan time is a typo. An invalid value that slips through is a 2 AM page. Validate enums, validate CIDR formats, validate that lists are non-empty when the module needs them. Every validation you write is a class of misuse you never have to debug in production.

variable "vpc_cidr" {
  type        = string
  description = "Primary CIDR block for the VPC."
  validation {
    condition     = can(cidrhost(var.vpc_cidr, 0))
    error_message = "vpc_cidr must be a valid IPv4 CIDR block."
  }
  validation {
    condition     = tonumber(split("/", var.vpc_cidr)[1]) <= 24
    error_message = "vpc_cidr must be /24 or larger to fit the subnet plan."
  }
}

Terraform supports multiple validation blocks per variable since 0.13, and you should use them. One condition, one error message, one failure mode the caller can actually fix.

Rule 3: Pin everything, and pin it where it belongs

The single biggest source of three-year drift is the unpinned or loosely-pinned provider. A module written against AWS provider 4.x that gets applied with provider 5.x will surface a wall of diffs that have nothing to do with your code. You did not change anything. The provider changed what your code means.

Version constraints belong in required_providers, in the module that uses the provider, not just the root:

terraform {
  required_version = ">= 1.6.0"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.40"
    }
  }
}

The ~> 5.40 constraint allows 5.40 and later 5.x releases but blocks the jump to 6.0. That is the right granularity for a module: accept patches and minor versions, reject majors. Major version upgrades are deliberate migration projects, not something that should happen because someone ran terraform init -upgrade on a Tuesday.

The .terraform.lock.hcl file is the other half of this. The version constraint says what range is acceptable; the lock file pins the exact version and its hashes. Commit it. I have seen teams gitignore the lock file because it caused merge conflicts, which is like throwing away your seatbelt because it wrinkles your shirt. The conflicts are annoying. The drift from unpinned providers across a team is far worse.

# Upgrade providers deliberately, review the lock diff, commit it
terraform init -upgrade
git diff .terraform.lock.hcl
# Record hashes for every platform your team and CI run on
terraform providers lock \
  -platform=linux_amd64 \
  -platform=darwin_arm64 \
  -platform=linux_arm64

The multi-platform lock matters more than people expect. An engineer on an M-series Mac and a CI runner on linux_amd64 will fight over the lock file forever unless both platforms' hashes are recorded.

Rule 4: Use moved blocks instead of state surgery

Refactoring is where modules go to die. You rename a resource from aws_instance.web to aws_instance.app, run plan, and Terraform announces it will destroy web and create app. They are the same EC2 instance. Terraform does not know that, because it tracks resources by their address in the configuration, and you changed the address.

The old way to fix this was terraform state mv, a manual imperative command run against state outside of code. It worked, but it was a side-channel. The migration lived in someone's shell history, not in version control. The next person to apply from a clean checkout had no idea the move had happened.

Since Terraform 1.1, moved blocks put the migration in the code where it belongs:

moved {
  from = aws_instance.web
  to   = aws_instance.app
}

resource "aws_instance" "app" {
  # ...
}

Now the rename is part of the plan. Anyone who applies, from any checkout, gets the move applied to their state automatically and idempotently. The plan shows aws_instance.web has moved to aws_instance.app instead of a destroy/create. This is the difference between a refactor that survives a code review and one that detonates three months later when someone applies from a stale state.

moved blocks compose with module restructuring too. When you split that VPC module, the resources move from the old module path to the new one:

moved {
  from = module.vpc.aws_nat_gateway.this[0]
  to   = module.egress.aws_nat_gateway.this[0]
}

Leave the moved blocks in place for at least one release cycle so every state in every environment has had a chance to apply them. Then remove them in a follow-up. Removing them too early strands states that never applied the move.

Rule 5: Prefer for_each over count, always

count is a trap that springs in year two. The problem is that count indexes resources by position in a list. Resource zero is [0], resource one is [1]. When you remove an item from the middle of the list, every item after it shifts down by one, and Terraform reads that as "destroy and recreate everything after the removed item."

Concrete example. You manage three subnets with count:

resource "aws_subnet" "private" {
  count      = length(var.private_cidrs)
  cidr_block = var.private_cidrs[count.index]
  vpc_id     = var.vpc_id
}

Now you remove the middle CIDR from private_cidrs. You wanted to delete one subnet. Terraform wants to delete the last subnet and re-key the middle one, because [1] now points at what used to be [2]. If anything depends on those subnets (NAT gateways, route table associations, EKS node groups) the recreation cascades.

for_each keys resources by a stable map key instead of position:

resource "aws_subnet" "private" {
  for_each          = var.private_subnets   # map(object({ cidr, az }))
  cidr_block        = each.value.cidr
  availability_zone = each.value.az
  vpc_id            = var.vpc_id
}

Now each subnet is addressed as aws_subnet.private["app-1a"], not aws_subnet.private[2]. Remove app-1b from the map and Terraform deletes exactly that one subnet, leaving the others untouched. The key is semantic and stable. This single change eliminates an entire category of accidental destruction.

The only legitimate use of count is a boolean toggle, creating zero or one of something:

resource "aws_nat_gateway" "this" {
  count = var.nat_strategy == "single" ? 1 : 0
  # ...
}

Even here, the moment you have more than one item, switch to for_each. Three years of survival means never letting a list-indexed resource grow into a multi-element list.

Rule 6: Make outputs a contract, and keep it small

A module's outputs are its public API. Everything you output, someone might depend on, and once they depend on it you cannot remove it without breaking them. The inherited VPC module exported 23 outputs including things like nat_gateway_eip_allocation_ids that exposed internal IDs nobody should have needed.

Output what callers need to wire the module to other modules, and nothing else. For the foundation module:

output "vpc_id" {
  value       = aws_vpc.this.id
  description = "ID of the VPC, for attaching resources."
}

output "private_subnet_ids" {
  value       = [for s in aws_subnet.private : s.id]
  description = "Private subnet IDs."
}

output "private_subnets_by_az" {
  value       = { for k, s in aws_subnet.private : s.availability_zone => s.id }
  description = "Map of AZ to private subnet ID, for AZ-aware placement."
}

The last output is a map keyed by AZ. Callers that need to place a resource in a specific AZ can look it up by name instead of guessing list positions. This is the output equivalent of for_each over count: give callers stable, semantic handles instead of positional ones.

Do not output secrets or internal attributes that callers have no business depending on. Every output is a commitment. The smaller your output surface, the more freedom you retain to refactor internals.

Rule 7: Drift detection runs on a schedule, not on demand

All the design discipline in the world does not stop out-of-band changes. Someone will click something in the console. The only defense is to detect it fast, while everyone still remembers what they changed and why.

Run terraform plan on a schedule against every environment and alert on non-empty plans. This is the single highest-leverage operational practice for keeping a module applyable. A nightly plan that shows three unexpected diffs is a five-minute investigation. The same three diffs discovered six months later, buried under thirty more, is a war room.

A minimal scheduled drift check in CI:

name: drift-detection
on:
  schedule:
    - cron: "0 6 * * *"   # 06:00 UTC daily

jobs:
  plan:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        env: [staging, production]
    steps:
      - uses: actions/checkout@v4
      - uses: hashicorp/setup-terraform@v3
        with:
          terraform_version: 1.7.5
      - name: Configure AWS via OIDC
        uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::ACCOUNT:role/tf-drift-readonly
          aws-region: ap-south-1
      - name: Plan and detect drift
        working-directory: envs/${{ matrix.env }}
        run: |
          terraform init -input=false
          terraform plan -detailed-exitcode -lock=false -out=plan.tfplan
        # exit code 0 = no changes, 1 = error, 2 = drift detected
      - name: Alert on drift
        if: failure()
        run: ./scripts/notify-slack.sh "Drift detected in ${{ matrix.env }}"

The -detailed-exitcode flag is the key. It returns 2 when the plan has changes, which the workflow treats as a failure and alerts on. The role is read-only, so the drift check can never accidentally apply. Use -lock=false for the read-only plan so a scheduled run never blocks an engineer mid-apply.

The organizational discipline that makes this work: a non-empty drift plan is a ticket, not background noise. Either the console change was legitimate and should be codified, or it was a mistake and should be reverted. Both resolutions end with the plan going empty again. If you let drift plans accumulate as known noise, you have rebuilt the write-only module, just with a daily reminder of how broken it is.

Rule 8: Lifecycle blocks are scalpels, use them sparingly

When you cannot or should not let Terraform manage an attribute, lifecycle blocks let you carve out exceptions. They are powerful and dangerous in equal measure.

ignore_changes is for attributes modified out-of-band by design. An auto-scaling group's desired_capacity is managed by a scaling policy, not by you. If Terraform manages it, every plan after a scale event shows a diff:

resource "aws_autoscaling_group" "workers" {
  desired_capacity = 3   # initial only
  min_size         = 3
  max_size         = 20

  lifecycle {
    ignore_changes = [desired_capacity]
  }
}

Now Terraform sets the initial capacity and then stops fighting the scaling policy. This is correct use: an attribute that has a different owner at runtime.

Incorrect use is reaching for ignore_changes = all to silence a noisy plan you do not want to investigate. That converts the resource into an opaque blob that Terraform tracks but does not manage. It is the write-only module in miniature.

create_before_destroy is the other lifecycle workhorse, essential for resources that cannot have downtime during replacement:

resource "aws_launch_template" "workers" {
  # ...
  lifecycle {
    create_before_destroy = true
  }
}

Without it, a replacement destroys the old resource before creating the new one, which for a launch template referenced by a live ASG means a window with no valid template. With it, the new one comes up first. Know which of your resources need this; it is not the default for a reason, but for anything in a serving path, it usually should be.

prevent_destroy is the seatbelt for stateful resources: a database, an S3 bucket with production data, a KMS key. Set it and a destroy plan errors out instead of silently scheduling deletion:

resource "aws_db_instance" "ledger" {
  # ...
  lifecycle {
    prevent_destroy = true
  }
}

This has saved me from a terraform destroy against the wrong workspace more than once. The cost is that you must remove the block deliberately when you genuinely want to delete, which is exactly the friction you want for a resource that holds a fintech ledger.

Rule 9: Test the module, not just the deployment

A module nobody can test is a module nobody can refactor with confidence, and confidence is what survives three years. Terraform's native test framework, stable since 1.6, lets you assert on plans and applies without leaving HCL:

# tests/network_foundation.tftest.hcl

run "creates_expected_subnet_count" {
  command = plan

  variables {
    vpc_cidr = "10.50.0.0/16"
    az_count = 3
  }

  assert {
    condition     = length(aws_subnet.private) == 3
    error_message = "Expected one private subnet per AZ."
  }
}

run "rejects_oversized_cidr" {
  command = plan

  variables {
    vpc_cidr = "10.50.0.0/26"
    az_count = 3
  }

  expect_failures = [var.vpc_cidr]
}

The first test asserts behavior: three AZs yields three subnets. The second asserts that the validation block rejects a CIDR too small for the subnet plan. Both run in seconds against a plan, no real infrastructure required. Run them in CI on every module change:

terraform test

For higher-confidence integration tests that actually apply against a sandbox account, the command = apply variant spins up real resources and tears them down after. Reserve those for the modules whose breakage hurts most, because they are slower and cost real money. The plan-level tests are cheap enough to run on every commit, and they catch the majority of regressions: a refactor that changes resource counts, a variable rename that breaks validation, a for_each key that shifts.

The payoff compounds. When you have tests, the year-three provider upgrade becomes: bump the version, run the tests, read the diffs. Without tests, the same upgrade is a manual inspection of every plan in every environment, performed by someone who is afraid to touch the module.

Rule 10: Document the why in the code, not in a wiki

The # DO NOT CHANGE comments on the inherited module were the worst kind of documentation: a warning with no reason. Nobody knew why those variables could not change, so nobody could ever decide it was safe to change them. The knowledge died with the author.

Document the non-obvious decisions where they live, in the code, with the reason:

# NAT gateway is per-AZ in production rather than single because a
# single-AZ NAT outage during the 2023-09 incident took down all
# private-subnet egress. Per-AZ isolates that failure.
# Single NAT is acceptable in non-prod where egress downtime is tolerable.
variable "nat_strategy" {
  # ...
}

This comment turns a mysterious constraint into a decision someone can revisit. If NAT pricing changes or someone proposes a NAT instance alternative, the next engineer has the context to evaluate it. A wiki page would have been stale within a quarter and unfindable within a year. The comment travels with the code and shows up in every diff that touches the variable.

The README matters too, but for a different audience: callers who need to use the module without reading its internals. Keep it focused on inputs, outputs, and a working example, and generate the input/output tables from the code so they never go stale:

terraform-docs markdown table --output-file README.md --output-mode inject ./modules/network-foundation

Wire terraform-docs into a pre-commit hook so the README is regenerated whenever variables or outputs change. A README that drifts from the code is worse than no README, because people trust it.

What a surviving module actually looks like

Put the rules together and a three-year-survivable module has a recognizable shape. It is small, scoped to a single lifecycle. Its variables describe intent and validate themselves. Its providers are pinned with a committed lock file. Its resources use for_each with stable keys. Its refactors ship as moved blocks. Its outputs are a deliberate, minimal contract. It has tests that run on every change, scheduled drift detection that alerts on surprises, and comments that explain the decisions someone will want to revisit.

None of this is clever. There is no elegant abstraction that makes Terraform safe to run for three years. There is only the accumulation of boring discipline that keeps the gap between your code and the world small enough to close with a single apply. The module that survives is not the one with the most flexibility or the cleverest dynamic blocks. It is the one where, in year three, an engineer who has never seen it can run terraform plan, read the output, and trust it.

We rewrote that VPC module over three weeks. Split into three lifecycle-scoped modules, converted every count to for_each with moved blocks to migrate the state without recreation, pinned the provider, added plan-level tests and a nightly drift check. The first clean terraform plan after the rewrite showed No changes. Your infrastructure matches the configuration. for the first time in over a year. That line is the entire goal. Everything above is what it takes to keep it true.

The test of a Terraform module is not whether it applies cleanly the day you write it. Every module does. The test is whether someone who has never seen it can run plan in year three and believe what it tells them. Design for that engineer, not for yourself today, and the drift never gets a chance to win.