At 2:14 a.m., your phone buzzes. The dashboard is red. Viewers are staring at a spinner. The Slack thread already says “stream down.” Someone pastes a CDN graph that drops off a cliff. Someone else asks, “Is it the origin again?” A third person replies, “No, looks like the encoder dropped.” Twenty minutes later, after checking three dashboards and tailing logs from two regions, you realize it was neither. It was a manifest request storm triggered by a stale origin shield cache. The edge hammered the packager with playlist reloads until the packager’s thread pool saturated. You fix it by purging the shield and restarting the packager. The postmortem gets filed under “CDN issue.”
Six months later, the same failure happens again. Nobody recognizes it, because “CDN issue” is not a diagnosis. It’s a shrug.
This is the cost of not naming your failures precisely. Streaming infrastructure is a chain of interdependent components—ingest, encode, package, origin, edge, player—and when it breaks, the break rarely announces itself with a clean label. The symptoms are generic: buffering, black screen, audio drop. The root cause is specific: an SRT too-late packet collapse, a CMAF fragment alignment break, a DASH MPD update race condition. The difference between a 20-minute incident and a 5-minute incident is often whether the on-call engineer can map the symptom to a named failure pattern in the first 90 seconds.
This article is about building that map. It’s about creating a shared taxonomy of streaming failure modes that your team can use to recognize, escalate, and resolve incidents faster. It draws on practices from other disciplines that have already learned this lesson—and it gives you a concrete framework to start naming the failures that are currently hiding under vague labels in your postmortem tracker.
Why Naming Failures Changes Incident Response
In the Google SRE book, the chapter on effective troubleshooting opens with a deceptively simple principle: “The faster you can identify what is broken, the faster you can fix it.” The book describes a structured approach to incident management that relies on a shared mental model of how systems fail. When an SRE says “cascading failure,” everyone on the call knows the pattern: one component’s overload causes upstream backpressure, which causes more overload, which propagates. The name carries diagnostic weight. It tells you where to look first, what metrics to check, and which mitigations are likely to work. The Google SRE book dedicates entire chapters to naming and categorizing failure patterns—cascading failures, overload, data integrity issues—because the authors understand that a shared vocabulary is not a documentation nicety; it’s an operational tool.
Streaming infrastructure has its own failure patterns, but most teams have not named them. We talk about “latency spikes” and “buffering” and “ingest issues” as if those are diagnoses. They are not. They are symptoms. A latency spike could be a GOP boundary misalignment in the ABR ladder, a TCP BBR congestion control burst that overwhelmed the edge transcoder, or a player-side buffer bloat caused by a Safari-specific heuristic. Each of those has a different root cause, a different fix, and a different set of metrics that would have caught it early. Calling all of them “latency” is like calling every server error a “500.” It’s true, but it’s useless.
The NIST Cybersecurity Framework provides a parallel lesson. NIST’s framework categorizes security failures and risks into a structured taxonomy—Identify, Protect, Detect, Respond, Recover—with subcategories that let organizations communicate about threats with precision. A “ransomware event” is not the same as a “credential stuffing attack,” and the response playbooks differ accordingly. The NIST Cybersecurity Framework works because it forces organizations to name what they’re defending against before they can build defenses. Streaming teams need the same discipline: name the failure mode before you can build a runbook for it.
The Streaming Failure Taxonomy: A Practical Framework
I propose a taxonomy organized around the streaming chain itself. Each link in the chain—ingest, encode, package, origin, edge, player—has a set of failure modes that are specific to that link. Naming them requires understanding the mechanics of that link well enough to describe what broke, not just what the viewer saw. Below is a framework you can adapt for your own infrastructure. I’ve included concrete examples for each category, drawn from real production incidents I’ve diagnosed or heard about from other engineers at 2 a.m.
Ingest Failures
Ingest is where the stream enters your infrastructure. The most common failure here is not a complete disconnect—those are easy to detect. The dangerous failures are partial, intermittent, or protocol-specific.
SRT Too-Late Packet Collapse. SRT’s packet recovery mechanism retransmits lost packets based on a latency budget. If the round-trip time plus the configured latency buffer is insufficient, packets arrive too late to be useful. SRT drops them. The encoder thinks it’s sending a clean stream. The receiver sees gaps. The symptom is video corruption or frozen frames that clear up after a few seconds, then return. The fix is not “increase bandwidth.” It’s tuning the latency parameter against measured RTT jitter, or switching to a listener mode that handles asymmetric paths better.
RTMP Ingest Clock Drift. RTMP carries no explicit timestamp synchronization beyond the stream’s own clock. If the encoder’s clock drifts relative to the ingest server’s clock—common in long-running events—the timestamps embedded in the RTMP chunks become inconsistent. When the packager converts this to HLS, the segment durations wobble. Players see segments that are 1.8 seconds instead of 2.0, or 2.3 instead of 2.0. The ABR algorithm gets confused. The symptom is buffering that gets worse over time, not better. The fix is NTP synchronization on the encoder and ingest server, plus monitoring timestamp monotonicity at the ingest boundary.
TCP BBR Burst Overwhelm. When an encoder uses TCP BBR congestion control on the upload path, BBR probes for available bandwidth by sending bursts. If the ingest server’s receive buffer is too small, those bursts cause packet loss at the server NIC before the application even sees the data. The encoder sees no loss—BBR’s model says the path is clean—but the server drops packets in hardware. The symptom is inexplicable corruption at the start of each new GOP. The fix is increasing the ingest server’s kernel receive buffer and monitoring NIC-level drop counters, not application-level bitrate.
Encoding and Packaging Failures
Encoding failures are rarely about the encoder crashing. They’re about the output not matching the assumptions of the downstream packaging and delivery chain.
CMAF Fragment Alignment Break. CMAF requires that video fragments align across bitrate renditions so that a player can switch seamlessly. If your encoding ladder uses different GOP sizes or different encoder presets across renditions, the fragment boundaries drift. The packager tries to align them and either inserts filler or drops frames. The symptom is a glitch at every ABR switch—exactly the thing CMAF was supposed to prevent. The fix is enforcing identical GOP structure and frame rate across all renditions in the ladder, and validating fragment alignment at the packager output.
Manifest Request Storm. This one deserves its own name because it’s so common and so misdiagnosed. When an origin shield cache expires a media playlist, every edge node that was serving that playlist simultaneously requests a fresh copy from the packager. If the packager’s thread pool is sized for steady-state load, not burst, the concurrent requests saturate it. The packager starts returning 503s. The edges interpret 503s as a signal to retry, which adds more load. The symptom is a stream that was healthy suddenly going black for all viewers simultaneously, then recovering after 30-60 seconds. The fix is not “add more packager capacity.” It’s tuning the origin shield TTL so that playlist expirations are staggered, and implementing request coalescing at the shield so that only one request goes to the packager while others wait.
DASH MPD Update Race Condition. In low-latency DASH, the MPD updates frequently to announce new segments. If the player requests an MPD update while the packager is writing a new one, the player can receive a partial or inconsistent manifest. The player’s ABR logic sees segments that don’t exist yet or have already expired. The symptom is a player that repeatedly requests segments that return 404, then falls back to a lower bitrate, then recovers, then repeats. The fix is atomic MPD writes at the packager and player-side retry logic that validates MPD completeness before acting on it.
Origin and CDN Failures
The origin and CDN layer is where most “stream down” incidents get misclassified. The CDN is often the messenger, not the culprit.
Origin Shield Cache Stampede. Similar to the manifest request storm but for segments. When a popular segment expires from the shield cache—say, the first segment of a live event that millions of viewers are joining—every edge node requests it from the origin simultaneously. The origin’s disk I/O or network bandwidth saturates. The symptom is a spike in origin response time that cascades into edge timeouts, then viewer buffering. The fix is pre-warming the shield cache for the first few segments of a known event, or using a request-coalescing layer at the shield.
Regional DNS Misdirection. A viewer in São Paulo gets resolved to a CDN edge in Miami because the DNS geolocation database is stale. The RTT is 150ms instead of 20ms. The player’s ABR algorithm sees low throughput and downshifts to a lower bitrate. The viewer sees a blurry stream and blames your encoding. The symptom is quality complaints from a specific region that don’t correlate with any server-side metric. The fix is monitoring RTT by region from client-side telemetry, not just server-side CDN logs, and working with your CDN provider to correct geolocation mappings.
Edge Cache TTL Conflict. You set a short TTL on segments to keep latency low. The CDN’s internal caching hierarchy has a minimum TTL that overrides yours. Segments get cached longer than you intended. Viewers near the edge see low latency; viewers behind a mid-tier cache see higher latency. The symptom is inconsistent latency across viewers that doesn’t correlate with geography. The fix is understanding your CDN’s cache hierarchy and TTL enforcement behavior, and testing with actual viewer-facing edge nodes, not just the ones near your office.
Player-Side Failures
Player failures are the hardest to diagnose because you don’t control the player environment. But naming them still helps, because it tells you where to instrument.
Safari Buffer Bloat. Safari’s HLS implementation maintains a larger playback buffer than Chrome or Firefox. On a network with intermittent connectivity—common on mobile—Safari fills that buffer during good periods and then plays from it during bad periods. If the bad period lasts longer than the buffer, playback stalls. But the stall happens minutes after the network degraded, so the viewer doesn’t associate the two. The symptom is “random” buffering on iOS devices that doesn’t appear on Android. The fix is not “reduce buffer size” (you can’t control Safari’s buffer). It’s monitoring buffer health from the player side and alerting when the buffer drain rate exceeds the fill rate, even if playback hasn’t stalled yet.
ABR Oscillation on Variable Mobile Networks. Mobile bandwidth varies second by second. A naive ABR algorithm switches up when it sees a bandwidth spike, then switches down when the spike ends. The player spends more time switching than playing. The symptom is a stream that constantly changes quality, with frequent brief buffering events. The fix is ABR algorithms that use a moving average with hysteresis, or that incorporate buffer level into the switching decision, not just throughput.
Time-to-First-Frame Regression. This is not a failure mode per se, but it’s a named metric that deserves a named failure pattern. When time-to-first-frame increases from 2 seconds to 8 seconds, something changed. It could be a larger GOP size in the encoding ladder, a CDN edge that’s farther away, a manifest that grew because of ad insertion, or a player update that changed the startup logic. Naming the regression pattern—“TTFF regression due to manifest bloat”—tells you where to look. Without the name, you’re just staring at a graph that went up.
Building Your Team’s Failure Taxonomy
You don’t need to adopt my taxonomy wholesale. You need to build your own, based on the failures your team actually sees. Here’s a process that works.
Step 1: Mine your postmortems. Go through the last 12 months of incident reports. For each one, ask: what actually broke? Not “stream was down.” Was it an SRT packet loss pattern? A packager thread pool exhaustion? A CDN origin timeout? Write a one-sentence description that names the specific component and the specific mechanism. If you can’t, the incident wasn’t fully diagnosed, and that’s a signal to investigate further.
Step 2: Group by chain link. Organize the named failures by where they occurred in the chain: ingest, encode, package, origin, edge, player. You’ll start to see patterns. Maybe 60% of your incidents are origin-related. Maybe you have three different failure modes that all manifest as “manifest request storm” but have different triggers. Grouping reveals where your infrastructure is most fragile.
Step 3: Write a one-paragraph description for each named failure. Include the symptoms, the metrics that would catch it early, the diagnostic steps, and the fix. This is your runbook. It doesn’t need to be long. It needs to be specific enough that an on-call engineer who has never seen this failure can recognize it from the description and know the first three things to check.
Step 4: Socialize the taxonomy. Put it in a shared document, a wiki, or a Slack bot that responds to /stream-failure commands. Use the names in incident channels. When someone says “stream is buffering,” ask: “Is this a manifest request storm or an origin shield cache stampede?” The act of asking trains the team to think in terms of named patterns. Over time, the names become shorthand that accelerates diagnosis.
This process is not unlike the work that goes into structuring any complex system of ideas. When you’re trying to impose order on a messy domain—whether it’s streaming failures or something else entirely—the naming is the hard part. The right name captures the mechanism, not just the symptom, and makes the pattern recognizable to others. I’ve seen teams spend hours debating whether a particular failure should be called “SRT Too-Late Packet Collapse” or “SRT Latency Budget Exhaustion,” and that debate is productive because it forces clarity about what actually happened. In a completely different context, writers and editors face a similar challenge when they’re trying to organize a large body of work—they need structures that make the content navigable. Tools that help generate book title ideas that capture the essence of a manuscript are solving a parallel problem: finding a name that is both descriptive and memorable, so that the right audience can find it and understand it quickly. The same principle applies to your failure taxonomy. A good name is a retrieval key for the brain.
Why This Pays Off at 2 a.m.
When you’re on call and the dashboard goes red, you don’t have time to reason from first principles. You need pattern recognition. A named failure pattern is a pre-computed diagnosis. It tells you: check these three metrics, run this diagnostic command, apply this mitigation. It also tells you what it’s not. If you know the pattern for “Origin Shield Cache Stampede,” and the metrics don’t match, you can eliminate that branch and move to the next one. That’s faster than starting from “something is wrong with the CDN.”
Named failures also improve postmortems. A postmortem that says “we had a manifest request storm caused by a stale origin shield cache” is actionable. You can add request coalescing, stagger TTLs, or increase packager thread pool size. A postmortem that says “CDN issue” is not actionable. It goes into the archive and teaches no one anything. The Google SRE book’s chapter on postmortem culture emphasizes that a good postmortem must identify the root cause precisely enough that you can prevent recurrence. “CDN issue” is not a root cause. “Origin shield cache stampede due to synchronized TTL expiry” is.
Finally, a shared taxonomy reduces the cognitive load of incident response across the team. When a junior engineer gets paged, they don’t need to have seen every failure before. They need to be able to map the symptoms to a named pattern and follow the runbook. That’s only possible if the patterns are named, documented, and discussed regularly. It’s the difference between a team that learns from each incident and a team that repeats the same incidents every six months under different vague labels.
Start With the Failures You Already Have
You don’t need a complete taxonomy on day one. Start with the last three incidents your team handled. For each one, write down what actually broke, in specific mechanical terms. Give it a name. Write a one-paragraph description. Share it with the team. The next time an incident happens, ask whether it matches one of your named patterns. If it doesn’t, you’ve discovered a new one. Add it to the list.
Over time, you’ll build a map of your infrastructure’s failure modes that is specific to your encoders, your CDN, your packagers, and your viewers. That map is worth more than any generic monitoring dashboard. It’s the difference between diagnosing a failure in 90 seconds and diagnosing it in 20 minutes while viewers abandon your stream. At 2 a.m., that difference is everything.