The console change that was never written down

Here is how it happens, every time.

An incident starts. Someone competent adjusts a security group in a console, so playback recovers and the incident closes. Six weeks later a rebuild from the repository produces an environment that is subtly different, and the reason has been lost, although everyone involved did the right thing.

The rule

If it is not in the repository, it is not in the environment. No exceptions, including, and especially, during an incident.

git switch -c raise-vod-cache-ttl
$EDITOR infra/edge/cache-policy.tf

terraform fmt -check
terraform validate
terraform plan -out=tfplan       # attached to the merge request

# review -> approve -> merge -> pipeline applies
#
# What does not exist in this workflow:
#   a console session
#   an ssh into a box to "just fix it"
#   a change that cannot be replayed

Emergency changes go through the same path with an expedited review. Faster, not different.

What the rule buys

• Recovery becomes a rebuild with a known duration, instead of an archaeology project with an unknown one.
• Changes get read before they happen. A plan attached to a merge request is a diff of reality, and somebody sees the blast radius while it is still cheap.
• The cost becomes explainable. Cost analysis is only possible when every resource has a declaration and an owner. Untracked infrastructure is where cloud spending goes to hide.
• Handover becomes possible. A repository can be read by a new team. Tribal knowledge leaves when its owner does.

What it does not solve

Drift from outside the tooling: a provider that changes a default, a resource mutated by a managed service, a manual change made without the context to know better. The rule reduces the surface without eliminating it, and treating it as a cure is how a team stops running plan regularly.