Skip to main content
The quarterly DR drill arrives. The voice carrier has to prove it can fail SBCs, RADIUS, and routing to the secondary site without dropping carrier-grade SLAs. The drill needs to be announced, executed, validated, reverted, and signed off — with the regulator’s evidence pack assembled the same day. Make the agent responsible for coordinating the drill record, approvals, announcements, validation evidence, and follow-up actions. Engineers execute and verify the infrastructure changes through the appropriate tools. The support loop keeps a failed check assigned to an owner and prevents a report from being mistaken for a successful recovery test.

​
Systems involved

​
Technical playbook

1

Pre-drill announcement

72 hours before, Copilot drafts the customer notice through Gmail: drill window, expected impact (none), what to do if real symptoms appear, a single point of contact for the window. Sent to the enterprise tenant contacts.
2

Open the drill war room

At T-15 minutes, open the #voice-ops Teams thread with the drill checklist, the rollback path, the on-call names, and the validation criteria. The drill procedure starts.
3

Capture pre-drill state

Copilot snapshots: active calls per SBC, RADIUS auth rate, carrier-side reachability checks, the current Anycast announcement state. The snapshot is the baseline for validation.
4

Fail SBC to secondary

Through SSH and the routing connector, withdraw the primary SBC’s Anycast announcement. The secondary becomes preferred. Existing calls on the primary continue; new calls land on the secondary within 30 seconds.
5

Fail RADIUS to secondary

Stop the primary RADIUS. The secondary takes over. Auth rate stays inside SLA. The procedure captures the failover transition time.
6

Validate inbound and outbound

Run a TestCallin sweep: inbound on five test DIDs, outbound to five test endpoints, both with and without media. Bidirectional audio confirmed in all cases.
7

Validate 911 routing

The single most important test. Place a 911 test call from a test endpoint with a known address. Confirm it routes to the test PSAP for the address, not to a stale primary-site path.
8

Hold for the soak window

Stay on the secondary for the contracted soak window (often 2 hours). Watch every metric. Anything outside SLA terminates the drill into rollback and is documented.
9

Fail back

Reverse the procedure: re-announce the primary, restart RADIUS primary, validate. Capture the restoration time and the call-continuity status.
10

Compliance report

Generate the drill evidence: timeline, snapshots, validation results, soak-window metrics, 911 evidence, sign-offs. PDF goes into the compliance file and is emailed to the regulator’s contact and the customer’s compliance officer for any tenant who requested evidence.

​
Support outcome

  • The drill record links approvals, infrastructure actions, and the observed validation results.
  • The service owner can distinguish successful recovery testing from a rollback or failed check.
  • The customer receives the agreed report, and remaining defects become assigned remediation work.

Procedures

Build the Voice DR drill once and run it every quarter.

Files and artifacts

The compliance evidence pack lives in the workspace.