GridRelay47 All articles
Opinion

Milliseconds Have Causes: Separating Light-Speed Delay From Self-Inflicted Latency in Relay Networks

GridRelay47
Milliseconds Have Causes: Separating Light-Speed Delay From Self-Inflicted Latency in Relay Networks

Photo: network engineer analyzing latency graphs on monitors in operations center, via www.xcitium.com

Latency arguments in relay engineering often collapse into a single number. A dashboard shows 42 milliseconds, a customer asks for 20, and the discussion turns into a negotiation with physics. That is the wrong conversation. Relay latency is a stack of causes, and only some of them are fixed by distance. The rest are engineering choices, including several that operators make badly and then blame on the network.

The Floor Set by Physics

Light in optical fiber travels at roughly two-thirds of its speed in a vacuum, or about 200,000 kilometers per second. That works out to approximately 5 microseconds per kilometer of fiber, one way. A relay path from Ashburn, Virginia, to Chicago with 1,200 kilometers of actual fiber route has a propagation floor of about 6 milliseconds one way, or 12 milliseconds round trip, before any router, switch, or queue is involved. A New York to Los Angeles path with 4,000 kilometers of fiber has a one-way floor near 20 milliseconds and a round-trip floor near 40 milliseconds. No amount of tuning removes that.

Real routes are longer than map distances. Fiber follows rights-of-way, avoids difficult terrain, and sometimes detours through carrier interconnection points. Route stretch of 20 to 50 percent is common. That means the physics floor for a given city pair is often higher than a straight-line calculation suggests. Engineers should calculate the floor using fiber path length, not airline distance, and should publish it alongside any latency target.

Serialization adds a smaller but real cost. A 1,500-byte packet on a 10-gigabit link occupies about 1.2 microseconds of transmission time. At 1 gigabit, the same packet takes about 12 microseconds. These numbers are trivial in isolation, but they compound across many hops and many small packets. The bigger variable is queuing.

Where Self-Inflicted Delay Enters

Queuing delay occurs when packets arrive faster than a link can drain them. A shallow queue creates a few microseconds of waiting. A deep queue creates milliseconds. The difference is not a law of nature; it is a buffer configuration decision. When operators size buffers to absorb bursts without active queue management, they trade loss for delay. In relay networks, that trade is often invisible until an application with real-time constraints arrives.

Bufferbloat is the pathological version of this trade. A router with a large buffer can hold hundreds of milliseconds of traffic, especially on lower-bandwidth links. Interactive sessions and control traffic share that queue with bulk transfers. The result is a network that looks healthy on average throughput while individual relay handshakes stall. The delay is self-inflicted, but it is no less real to the application.

Suboptimal routing is another recurring cause. A relay may choose a path based on BGP policy, cost, or peering relationships rather than latency. A route that crosses the country twice can add 30 milliseconds to a path that geography says should be shorter. Anycast can improve reachability while creating unpredictable path selection. Traffic engineering can fix some of this, but only if operators measure one-way delay and path diversity rather than assuming that the shortest AS path is the fastest.

Bufferbloat Is a Policy Failure

The most useful reframing is that bufferbloat is not a hardware defect. It is a policy failure. Buffers exist to absorb microbursts, not to store seconds of traffic. Active queue management, fair queuing, and explicit congestion notification give operators ways to keep queues short while preserving throughput. Pacing at the sender reduces burstiness. Traffic classification protects control and interactive flows from bulk transfer. None of these techniques is exotic; they are standard practice in networks that care about tail latency.

The hard part is measurement. Average latency hides standing queues because most packets may pass through an empty buffer while a minority wait behind a burst. Percentile latency, especially p95 and p99, exposes the problem. Queue occupancy telemetry exposes the cause. A relay operator who reports only average round-trip time is effectively reporting the weather in a building with no windows.

Measuring the Right Thing

One-way delay is more useful than round-trip time for separating direction-specific problems, but it requires disciplined clock synchronization. Precision Time Protocol can provide the necessary accuracy when deployed correctly. Active probes between relay sites can establish baseline paths and detect route changes. Passive telemetry from switches and routers can show queue depth and drops without adding probe traffic. The combination is more informative than any single dashboard number.

Engineers should also distinguish between cold-path and warm-path behavior. A relay that has just failed over may have empty caches, new routes, and unsettled queues. Its first few seconds of latency are not representative. Stakeholders who see a single post-failover spike may demand improvements that the steady-state network does not need. Baseline separately, and publish both.

Setting Expectations Without Excuses

There is a defensible way to talk about relay latency. State the fiber path, the calculated propagation floor, the measured p95 and p99, and the known queuing contribution. If a stakeholder wants 20 milliseconds between New York and Los Angeles, the answer is not that the network needs tuning. The answer is that the request is below the speed-of-light floor for any realistic fiber route. If the same stakeholder sees 80 milliseconds on a path whose floor is 40, that is an engineering problem worth solving.

The discipline is to separate the physics from the policy. Distance sets the floor. Routing, buffering, and traffic classification determine how far above that floor the network operates. Relay operators who conflate the two will keep promising numbers that physics will not honor, and they will keep missing the self-inflicted milliseconds that are entirely within their control.

All Articles

Related Articles

Blast Radius by Any Other Name: Toward a Failure-Domain Taxonomy for Relay Networks

Blast Radius by Any Other Name: Toward a Failure-Domain Taxonomy for Relay Networks

Clearance Not Consoles: What Relay Operations Keeps Getting Wrong About the Single Pane of Glass

Clearance Not Consoles: What Relay Operations Keeps Getting Wrong About the Single Pane of Glass

Stop Chasing Symmetry: The Case for Deliberately Lopsided Relay Topologies

Stop Chasing Symmetry: The Case for Deliberately Lopsided Relay Topologies