GridRelay47 All articles
Opinion

Blast Radius by Any Other Name: Toward a Failure-Domain Taxonomy for Relay Networks

GridRelay47
Blast Radius by Any Other Name: Toward a Failure-Domain Taxonomy for Relay Networks

Photo: Mike Reyher, CC BY 2.0, via Wikimedia Commons

Every distributed relay network carries an inherited vocabulary: edge nodes, core nodes, relay nodes, aggregation layers, points of presence. The words are useful for drawing diagrams and almost useless during an incident. They describe topology positions, and topology positions do not correspond to the things that actually break together, get fixed by the same team, or share a single authority to act.

This is an argument for replacing some of that vocabulary with something less elegant and more honest: failure domains.

What the Current Labels Conceal

Consider the phrase core relay node. It might mean a well-provisioned site with redundant everything, or it might mean a box in a leased cage that happens to sit in the middle of a diagram. The label tells you nothing about whether the site has a second utility feed, whether its two fiber paths share a vault, or whether the team that can power-cycle it is the same team that can reroute around it.

During an incident this is not a semantic quibble. When a relay node stops responding, the first question on-call asks is what else is affected. Labels like edge and core do not answer it. Failure domains do.

Defining the Domains That Matter

A failure domain is the set of components that share a fate and a controller. Two things belong to the same domain if a single action, fault, or decision can take them both down — or if a single team must act to restore them. For distributed relay networks, five domains cover most of the terrain.

Power domain. Everything fed by the same utility service, transfer switch, or generator. The tell is simultaneous disappearance of multiple nodes at one physical location with no routing precursor.

Control domain. Everything governed by the same orchestration plane, configuration authority, or administrative credential. A bad policy push is a control-domain event even when it manifests on a hundred sites across four time zones.

Path domain. Everything sharing a physical conduit, vault, or carrier handoff. Path domains are where diversity claims go to die, because they are invisible to routing tables and visible on a contractor's excavation map.

Identity and trust domain. Everything relying on the same certificate authority, key material, or authentication service. Rotation failures here do not degrade performance; they sever membership.

Time domain. Everything synchronized from the same source or the same timing chain. Small drift produces strange behavior that looks like application bugs until someone checks the offset.

A sixth, softer domain is worth naming: the operator domain, meaning the boundary of who is authorized to change something. It matters most during handoffs between organizations, which is precisely when nobody wants to discover an unstated boundary.

Naming Failures Instead of Boxes

Adopting the taxonomy changes incident language in a productive way. Instead of declaring that a node is down, an engineer reports a power-domain loss at a named site class, or a control-domain fault affecting a specific configuration group. The statement carries blast radius and likely remediation path in the same breath.

It also makes runbooks testable. A runbook for a path-domain fault should include conduit verification and carrier escalation; a runbook for an identity-domain fault should not begin with checking link utilization. When the taxonomy is explicit, mismatched runbooks become obvious.

Handoffs improve for the same reason. A concise framing of which domain has failed, what is in it, and who controls it lets the next engineer pick up without re-deriving the situation from a dashboard.

The Objections Worth Taking Seriously

There is a real cost to new vocabulary. Engineers under pressure revert to familiar words, and a taxonomy that requires memorization will be quietly abandoned within a quarter. Two design choices help.

First, keep the domain list short and attach it to something already visible. If the asset catalog has a domain field, people will read it and eventually speak it. Second, resist the urge to subdivide. Elaborate taxonomies with nested categories feel rigorous and get ignored.

The second objection is that domains overlap. A single site might sit in one power domain, two path domains, and three control domains. That is not a flaw; it is the reason the model is more truthful than a layer diagram. Ambiguity in real systems should be representable, not smoothed away.

A Modest Adoption Path

Start with the incident review. For every postmortem, require a one-line statement of which domain or domains failed and where their boundaries were. Add the field to the asset catalog for the top twenty sites by traffic. Update the three runbooks that get used most, then watch whether the language sticks in on-call chatter.

None of this requires tooling purchases or re-architecture. It requires a decision that the words used during an incident should describe consequence rather than position. Relay networks are distributed by construction, which means their failures are almost always about shared fate hiding inside apparent separation. Naming that shared fate is the least expensive reliability work available — and the easiest to postpone indefinitely.

All Articles

Related Articles

Clearance Not Consoles: What Relay Operations Keeps Getting Wrong About the Single Pane of Glass

Clearance Not Consoles: What Relay Operations Keeps Getting Wrong About the Single Pane of Glass

Stop Chasing Symmetry: The Case for Deliberately Lopsided Relay Topologies

Stop Chasing Symmetry: The Case for Deliberately Lopsided Relay Topologies

Paper Barriers: Seven Lease Terms That Decide Whether a Relay Site Survives an Outage

Paper Barriers: Seven Lease Terms That Decide Whether a Relay Site Survives an Outage