Change Verification

Change Verification

Event-driven change verification skill for structured change windows. Guides baseline capture before a change, provides change execution safety patterns, performs post-change diff analysis, and supports rollback decision-making when unexpected deviations are detected.

This skill covers a single change event lifecycle (before → during → after). For ongoing configuration drift detection and compliance auditing, use the config-management skill instead.

Commands are labeled [Cisco], [JunOS], or [EOS] where syntax diverges. Unlabeled statements apply to all three vendors.

Safety Note — Read-Write Operations: This skill includes procedures that modify device state during change execution and rollback phases. Steps that write to devices are marked with ⚠️ WRITE. Always confirm authorization, change ticket approval, and maintenance window status before executing write operations. Baseline capture and post-change verification steps are read-only and safe to run at any time.

When to Use

• Planned maintenance window requiring structured pre/post verification
• Configuration change (routing policy, ACLs, interface config) with rollback plan
• Software upgrade or patch requiring before/after state comparison
• Hardware replacement (linecard, SFP, PSU) with service validation
• Circuit turn-up or decommission with adjacency and traffic verification
• Emergency change requiring rapid baseline capture and rollback readiness
• Post-change soak period with periodic re-verification against baselines

Prerequisites

• SSH or console access to all devices in the change scope (read-only for baselines; enable/configure privilege for change execution and rollback)
• Approved change ticket with documented scope, expected impact, and rollback plan including timing criteria
• Pre-identified list of devices and interfaces in the change scope
• Knowledge of expected state changes: which routes will move, which interfaces will bounce, which adjacencies will flap
• Access to a file store (flash, SCP server, or local disk) for baseline archival
• Contact information for escalation if rollback criteria are met
• See references/checklist-templates.md for per-change-type prerequisites

Procedure

Follow these steps sequentially for each change event. Steps 1–2 are always read-only. Steps 3–4 include write operations. Steps 5–6 are analytical and drive the rollback decision.

Step 1: Pre-Change Baseline Capture

Capture device state snapshots before any changes. Store outputs with timestamps for post-change comparison.

Routing state:

[Cisco]

show ip route summary
show ip bgp summary
show ip ospf neighbor

[JunOS]

show route summary
show bgp summary
show ospf neighbor

[EOS]

show ip route summary
show ip bgp summary
show ip ospf neighbor

Interface and adjacency state:

All vendors — capture interface status, error counters, and neighbor tables:

[Cisco]

show interfaces summary
show cdp neighbors
show ip arp

[JunOS]

show interfaces terse
show lldp neighbors
show arp no-resolve

[EOS]

show interfaces status
show lldp neighbors
show ip arp

Configuration and hardware:

[Cisco]

show running-config
show environment all
show inventory

[JunOS]

show configuration
show chassis environment
show chassis hardware

[EOS]

show running-config
show environment all
show inventory

⚠️ WRITE — Archive baseline config to persistent storage:

[Cisco] copy running-config flash:pre-change-[ticket]-[date].cfg [JunOS] request system configuration save /var/tmp/pre-change-[ticket].conf [EOS] copy running-config flash:pre-change-[ticket]-[date].cfg

Record baseline metrics for comparison: total route count, BGP peer count (Established), OSPF neighbor count (Full), interface error counters, and hardware sensor readings.

Step 2: Change Scope Documentation

Before executing any changes, document:

1. Change description — what configuration lines are being added, modified, or removed
2. Expected impact — which peers will flap, which routes will shift, which interfaces will bounce, expected duration of disruption
3. Rollback trigger criteria — specific thresholds that mandate rollback (see Threshold Tables below)
4. Rollback procedure — exact commands to revert (see references/cli-reference.md for vendor-specific rollback commands)
5. Success criteria — what "done" looks like: all baselines restored, intended changes visible, no unexpected deviations
6. Soak period — how long to monitor after change before declaring success

Step 3: Change Execution

⚠️ WRITE — Apply changes using commit-confirm patterns when available.

[Cisco] — No native commit-confirm. Apply changes in config mode and immediately verify. For bulk changes, use configure replace with a prepared config file.

[JunOS] — Use commit confirmed [minutes] to auto-rollback if not confirmed within the timer window. Confirm with commit after verification.

configure
# ... apply changes ...
commit confirmed 5
# ... verify ...
commit

[EOS] — Use configure session for atomic staged changes. Review before committing.

configure session change-[ticket]
# ... apply changes ...
show session-config
commit

Staged rollout for multi-device changes: Apply to one device first, verify post-change state (Step 4), then proceed to remaining devices only after the first device passes all checks.

Step 4: Post-Change Verification

Re-capture all baseline metrics from Step 1 using identical commands. Perform a structured diff against pre-change baselines.

Key comparisons:

Config diff verification:

[Cisco]

show archive config differences flash:pre-change-[ticket]-[date].cfg system:running-config

[JunOS]

show | compare rollback 1

[EOS]

diff running-config flash:pre-change-[ticket]-[date].cfg

Review every line in the diff output. Classify each changed line as expected (directly part of the change plan) or unexpected (not in the change scope).

Step 5: Impact Assessment

Classify all deviations from baseline into categories:

1. Expected — Intended: Changes that are the direct goal of the change window (e.g., new BGP peer appearing, old ACL removed). No action needed.
2. Expected — Side Effect: Changes caused by the intended change but not the primary goal (e.g., route count increase because a new peer is now advertising). Verify they are benign.
3. Unexpected — Minor: Deviations not related to the change scope but low severity (e.g., a single interface counter increment). Investigate but do not necessarily roll back.
4. Unexpected — Critical: Deviations indicating collateral damage (e.g., adjacency loss on an unrelated interface, route withdrawal not in change scope). Evaluate rollback immediately.

If any deviation is classified as Unexpected — Critical, proceed directly to Step 6 rollback evaluation.

Step 6: Rollback Decision

Evaluate whether to accept the change or roll back using the criteria below.

Rollback if ANY of these conditions are true:

• Service-affecting outage on interfaces/peers outside the change scope
• Route count deviation exceeds threshold AND routes are not accounted for in the change plan
• Adjacency loss persists beyond the expected convergence window
• Hardware errors (PSU, fan, temperature) emerged that were not present in baseline
• The change did not achieve its intended effect (success criteria from Step 2 not met)

Accept if ALL of these conditions are true:

• All success criteria from Step 2 are met
• Config diff contains only expected change lines
• All baseline metrics are within acceptable deviation thresholds
• No unexpected critical deviations detected
• Soak period has elapsed without regression

⚠️ WRITE — If rolling back:

[Cisco] configure replace flash:pre-change-[ticket]-[date].cfg force [JunOS] rollback 1 then commit [EOS] configure replace flash:pre-change-[ticket]-[date].cfg

After rollback, re-run Step 4 post-change verification to confirm the device has returned to its pre-change state.