Skip to content

Practice2 min read

The console change nobody wrote down

Estates do not become unreproducible through a decision. They get there through a sequence of individually reasonable exceptions, each made at two in the morning by someone doing the right thing.

  • terraform
  • gitops
  • operations

Here is how it happens, every time.

An incident starts. Someone competent adjusts a security group in a console. Playback recovers, the incident closes, everybody sleeps. Six weeks later a rebuild from the repository produces an environment that is subtly different and nobody can say why.

No one did anything wrong. That is the problem.

The rule

If it is not in the repository, it is not in the environment. No exceptions — including, and especially, during an incident.

git switch -c raise-vod-cache-ttl
$EDITOR infra/edge/cache-policy.tf

terraform fmt -check
terraform validate
terraform plan -out=tfplan       # attached to the merge request

# review -> approve -> merge -> pipeline applies
#
# What does not exist in this workflow:
#   a console session
#   an ssh into a box to "just fix it"
#   a change nobody can replay

Emergency changes go through the same path with an expedited review. Faster, not different.

What the rule buys

  • Recovery becomes a rebuild with a known duration rather than an archaeology project with an unknown one.
  • Changes get read before they happen. A plan attached to a merge request is a diff of reality, and somebody sees the blast radius while it is still cheap.
  • The bill becomes explainable. Cost analysis is only possible when every resource has a declaration and an owner. Untracked infrastructure is where cloud bills go to hide.
  • Handover becomes possible. A repository can be read by a new team. Tribal knowledge leaves when its owner does.

The uncomfortable part

The rule is hardest to keep at the exact moment it is most valuable. An undocumented fix applied during an outage is the one most likely to be forgotten and the one most likely to matter — because it was made under pressure, on the path that was already failing.

What it does not solve

Drift from outside the tooling. A provider that changes a default, a resource mutated by a managed service, a manual change made by somebody without the context to know better. The rule reduces the surface; it does not eliminate it, and treating it as though it does is how a team stops running plan regularly and starts being surprised by it.

Is this the bill you are paying?

If the pattern in this write-up describes your setup, the next step is reading your actual numbers against it. Send twelve months of billing and your concurrency curve.