Here is how it happens, every time.
An incident starts. Someone competent adjusts a security group in a console.
Playback recovers, the incident closes, everybody sleeps. Six weeks later a
rebuild from the repository produces an environment that is subtly different and
nobody can say why.
No one did anything wrong. That is the problem.
The rule
If it is not in the repository, it is not in the environment. No exceptions —
including, and especially, during an incident.
git switch -c raise-vod-cache-ttl
$EDITOR infra/edge/cache-policy.tf
terraform fmt -check
terraform validate
terraform plan -out=tfplan # attached to the merge request
# review -> approve -> merge -> pipeline applies
#
# What does not exist in this workflow:
# a console session
# an ssh into a box to "just fix it"
# a change nobody can replay
Emergency changes go through the same path with an expedited review. Faster, not
different.
What the rule buys
- Recovery becomes a rebuild with a known duration rather than an
archaeology project with an unknown one.
- Changes get read before they happen. A plan attached to a merge request is
a diff of reality, and somebody sees the blast radius while it is still cheap.
- The bill becomes explainable. Cost analysis is only possible when every
resource has a declaration and an owner. Untracked infrastructure is where
cloud bills go to hide.
- Handover becomes possible. A repository can be read by a new team. Tribal
knowledge leaves when its owner does.
The uncomfortable part
The rule is hardest to keep at the exact moment it is most valuable. An
undocumented fix applied during an outage is the one most likely to be forgotten
and the one most likely to matter — because it was made under pressure, on the
path that was already failing.
What it does not solve
Drift from outside the tooling. A provider that changes a default, a resource
mutated by a managed service, a manual change made by somebody without the
context to know better. The rule reduces the surface; it does not eliminate it,
and treating it as though it does is how a team stops running plan regularly and
starts being surprised by it.