The Second Grid: Why Packet Redundancy Ends at the Utility Feed
Photo: NACA, Public domain, via Wikimedia Commons
Most redundancy conversations in distributed relay networks begin at layer three and end there. Operators count independent fiber paths, verify carrier diversity, confirm that BGP withdrawals propagate cleanly, and take comfort in the fact that no two sites share a metro. All of that work is necessary. None of it describes what happens when the substation two miles away trips a breaker on a 96-degree afternoon in July.
A relay site is not an abstraction. It is a rack in a building with a meter on the wall, and that meter is a single point of attachment to an electrical grid that does not care about your failover policy. The gap between packet-layer redundancy and power-domain redundancy is where a surprising number of relay outages are actually born.
Two Redundancy Budgets That Rarely Match
Engineering teams tend to fund redundancy in proportion to how visible it is. A second carrier circuit shows up in a diagram, gets a monthly invoice, and appears in a vendor review. A second utility service shows up in a lease exhibit and a construction project that nobody on the network team is invited to review.
The result is a recurring pattern: a site with four diverse fiber paths, dual routers, and a sophisticated BGP policy — all of it riding on one utility feed, one automatic transfer switch, and a generator whose last load test happened during the previous administration. The packet layer is genuinely resilient. The facility layer is a chain of single points of failure wearing a costume of redundancy.
The Transfer Window: Three Timers That Must Agree
When utility power fails, three independent timers start running, and the site survives only if the slowest one finishes before the fastest one runs out.
- UPS ride-through. Battery strings are sized in minutes and degrade in years. A bank rated for twelve minutes at commissioning may deliver four under load and elevated temperature, and nobody discovers the difference until the moment it matters.
- Generator start and stabilization. Engine start, voltage and frequency stabilization, and breaker closure are not instantaneous. A generator that reaches rated output in fourteen seconds is doing well; one that hunts for forty seconds is doing something else.
- Transfer switch logic. Automatic transfer switches must decide that the utility is genuinely gone, not merely sagging, and must then sequence the load without tripping the generator on inrush.
Packet-layer failover typically completes in seconds. The electrical sequence routinely takes longer, and the mismatch is invisible in dashboards that only observe network telemetry. A relay node that loses its power domain will go silent without ever emitting a protocol-level signal that explains why.
A Practical Site Classification
Operators do not need a perfect model of the electrical plant. They need a coarse, honest one. Three classes cover most US relay footprints.
Class A — Independent dual service. Two utility services entering the building from separate substations or at minimum separate feeders, each capable of carrying full load, with dual-corded equipment and tested transfer logic. This is rare outside purpose-built facilities and expensive to retrofit.
Class B — Single service with full backup. One utility feed, a generator sized for the critical load, an automatic transfer switch, and fuel arrangements that survive a multi-day regional event. Common in well-run colocation and in carrier hotels with mature facilities teams.
Class C — Ride-through only. One feed and a UPS, with no generator, or with a generator that requires manual intervention. Entirely acceptable for some workloads and entirely unacceptable for a node that anchors quorum in a region.
Classify every site, publish the classification alongside latency and capacity in the internal catalog, and let routing decisions reflect it. A grid that treats Class C sites as interchangeable with Class B sites has already made an assumption it cannot defend.
Power-Aware Failover Testing
Facilities testing and network testing usually run on separate calendars, which means the two halves of the redundancy story are never validated together. A staged sequence closes that gap.
- Static audit. Document every utility service, transfer switch, generator, fuel contract, and UPS string per site. Confirm what is actually dual-corded versus merely dual-named.
- UPS ride-through under live traffic. Push real relay load through the battery window and measure how long the node stays healthy, not how long the batteries stay up.
- Generator transfer with production load attached. The most common finding here is that the generator starts fine and the transfer trips on inrush — a discovery worth making on a Tuesday morning rather than during a storm.
- Extended island operation. Run on generator for several hours, including at least one fuel top-off coordination, and watch thermal behavior in the rack as cooling loads shift.
- Coordinated utility interruption. Where a landlord or utility will schedule a maintenance window, test the full sequence end to end and record how the relay grid reroutes around the site.
Run the sequence at least annually, and repeat step three whenever the facilities vendor changes. Generator maintenance contracts turn over far more often than network hardware does.
Writing It Into the Runbook
Power-domain events produce a distinctive failure signature: multiple nodes at one physical site disappear simultaneously, with no precursor in routing telemetry. If the on-call engineer has no vocabulary for that pattern, they will spend the first twenty minutes chasing a phantom routing bug.
Add a short section to the incident runbook that names the electrical dependencies explicitly, lists the facilities escalation contacts per site, and states the expected recovery time for each site class. Then make sure the classification is a routing input rather than a facilities footnote. Redundancy that exists only in the packet layer is a claim, not a property.