Skip to main content
Ubiquiti publishes a critical firmware advisory for EdgeRouter X devices. The ISP operates 14,000 of them across residential subscribers. The rollout has to stage firmware centrally, go out regionally with per-region maintenance windows, notify subscribers before each window, watch reboot success rates, and roll back the handful that don’t come back cleanly. The support agent’s job is to progress the advisory through impact assessment, approved rollout, subscriber communication, and verified completion. Give failed devices an explicit escalation owner and require each region’s results before proceeding. Use the technical walkthrough as the fulfilment plan for this support loop, with agent actions limited to configured tools.

​
Systems involved

​
Technical playbook

1

Verify and stage the image

Copilot downloads the firmware from the vendor advisory, validates the SHA against the published hash, and uploads it to the ACS repository. The image appears in the ACS catalogue with the advisory ID.
2

Plan the rollout

Split the fleet by region — 14 regions, roughly 1,000 CPEs each. Each region gets a 2-hour window spread over ten nights. Business subscribers are scheduled last so any issues surface on residential first.
3

Pre-window subscriber comms

48 hours before each regional window, Twilio sends an SMS to residential subscribers: brief outage, window, self-service URL for status. Business subscribers get a personalised email through Gmail.
4

Publish maintenance

Statuspage publishes all 14 maintenance windows with the affected regions and the advisory reference. The IVR hold message picks up an automated region-aware notice 30 minutes before each window.
5

Execute the first window

The CPE firmware rollout procedure targets Region 1, 10 percent of the CPEs at a time. For each batch, the ACS queues the upgrade. Copilot watches the CPEs come back online against LibreNMS reachability within the expected reboot interval.
6

Watch the success rate

Target threshold is 99.5 percent reboot-and-reauth within the window. Region 1 hits 99.7 percent. Seven CPEs didn’t come back — Copilot flags each one with the last-known state and queues them for individual attention.
7

Handle the stragglers

For each failed CPE, Copilot pulls the RADIUS last-accounting record, the ACS session history, and the LibreNMS last-seen. Five come back on the next day’s reboot. Two are dispatched for field swap.
8

Subsequent regions

Each following night, the rollout procedure runs for the next region. Statuspage and #cpe-fleet maintain a running status board. Residential complaints are near-zero because the comms went out ahead.
9

Close

After all 14 regions, generate the rollout report: fleet coverage, success rate, rollback count, field-swap count, advisory closed. The report is filed to the ISP’s security advisory register and the post-mortem is auto-scheduled.

​
Support outcome

  • Each rollout group has a verified result before the next group begins.
  • Failed devices receive an assigned recovery task with the attempted change and observed result.
  • Subscriber updates and the final advisory report reflect the fleet state actually measured.

Procedures

CPE firmware rollout with advisory ID and region as arguments.

Connectors and MCP

GenieACS, Twilio, and Splynx wired as connectors.