Change Window Runbook

A practical runbook for making production changes without improvising: prove readiness, move in small steps, capture evidence, and know when to stop.

Before the window opens

Most production pain comes from doing real planning during the window instead of before it. The minimum readiness bar:

• scope is explicit: hosts, services, environments, and what is out of scope
• one operator and one reviewer or watcher are assigned
• the exact command path is known and tested in staging where possible
• rollback command or rollback document is ready to paste
• monitoring dashboards and logs are bookmarked
• affected stakeholders know the window and expected user impact

Pre-window header
- Change ID: CHG-1234
- Window: 22:00-22:30 UTC
- Scope: production web tier only
- Operator: alice
- Watcher: bob
- Rollback: revert MR !442 and redeploy previous image
- Stop conditions: 5xx > 2%, login failures, or smoke test failure

If the change came from normal engineering flow, the supporting detail should already exist in Infra Change Lifecycle or the related implementation MR. The window is not where you figure out what to run for the first time.

Go / no-go pre-checks

Run these checks shortly before the declared start and record the result. A failed pre-check is a normal reason to delay. It is not a personal failure.

# A practical pre-check bundle
date -Is
git rev-parse HEAD
curl -fsS https://app.example.com/health
journalctl -u myapp --since "10 minutes ago" --no-pager | tail -40
systemctl is-active myapp
df -h
df -i

For service-specific checks, pair this page with Generic or Service Troubleshooting.

Opening comms and declaring the start

Start the window with a short message that states scope, owner, expected impact, and next update time. The goal is to prevent "is anything happening?" chatter while the operator is working.

[START][CHG-1234] 22:00 UTC
Beginning production web rollout.
Scope: web01-web04 only.
Expected impact: brief connection draining per node, no full outage expected.
Operator: alice. Watcher: bob.
Rollback: previous image + launch template.
Next update: 22:10 UTC or sooner on material change.

If there is any chance the change could mask an ongoing incident, freeze unrelated deploys during the window. That makes later attribution much easier.

Execution loop and rollback points

Move in small, reversible steps. Every meaningful step should have a checkpoint and a clear decision: continue, pause, or roll back.

1. Drain or isolate one canary node if possible.
2. Apply the change to the canary.
3. Run the defined smoke tests.
4. Check dashboards and logs for a short interval.
5. Only then continue to the next batch or the rest of the fleet.

# Example execution shape for a controlled Ansible deploy
ansible-playbook site.yml -i inventories/prod --limit web01 --tags myapp
curl -fsS https://web01.example.com/health

ansible-playbook site.yml -i inventories/prod --limit web02:web03 --tags myapp
curl -fsS https://app.example.com/health

ansible-playbook site.yml -i inventories/prod --limit web04 --tags myapp

Smoke tests and validation

Your smoke tests should prove the user-visible or operator-visible path still works. Health endpoints are useful, but not sufficient by themselves.

# Example smoke pack
curl -fsS https://app.example.com/health
curl -fsSI https://app.example.com/login
ssh bastion 'systemctl is-active myapp'
journalctl -u myapp --since "2 minutes ago" --no-pager | tail -20

Validation usually has three layers:

• service layer: process up, config loaded, no immediate errors
• application layer: real request succeeds, auth still works, queue or database still reachable
• observability layer: no silent 5xx rise, latency spike, saturation jump, or log storm

For the dashboard side, lean on Observability Overview. If the service has its own troubleshooting path, jump to Service Troubleshooting instead of guessing.

Evidence capture and closeout

Capture evidence as you go, not from memory afterwards. A minimal evidence bundle should answer: what changed, when, what was observed, and how success was validated.

mkdir -p /var/tmp/change-CHG-1234

date -Is | tee /var/tmp/change-CHG-1234/timestamps.txt
git rev-parse HEAD | tee /var/tmp/change-CHG-1234/revision.txt
curl -fsS https://app.example.com/health | tee /var/tmp/change-CHG-1234/health.txt
journalctl -u myapp --since "15 minutes ago" --no-pager \
  > /var/tmp/change-CHG-1234/journal.txt

[COMPLETE][CHG-1234] 22:23 UTC
Rollout completed successfully.
Validation: health endpoint 200, login test passed, 5xx flat, latency unchanged.
Evidence: attached logs, dashboard screenshots, commit SHA, smoke output.
Watch period: continue normal monitoring for 30 minutes.

That closure message becomes the source material for the ticket, handover note, or change record. If the change was noisy or surprising, capture what should be improved for the next run instead of waiting for the next on-call engineer to rediscover it.

Failure modes

Related pages: Generic, Infra Change Lifecycle, Service Troubleshooting, Observability Overview, Backup & Restore.