14:32:07 UTC. Three seconds into the second commercial break of the second quarter. Eight hundred thousand concurrent viewers — CTV, mobile, web — watched the last frame of the game broadcast hold for 4.2 seconds. The SSAI stitcher was waiting for a splice point that arrived 180 milliseconds earlier than the manifest declared. Then the ad played. No CDN error. No player exception. No encoder dropout. The stream never went offline. It just stopped moving at the exact moment the SCTE-35 cue tone and the underlying PTS timeline disagreed.
What follows is a post-mortem of that incident. The methodology tracks the structured incident analysis framework in the Google SRE book — specifically the chapters on monitoring distributed systems, effective troubleshooting, postmortem culture, and cascading failures. What happened here was not a single component failing. It was a timing misalignment that cascaded through five independent systems, each behaving correctly in isolation.
That same discipline applies to title and framing decisions: before publishing, editors need a way to test a heading promises the same thing the article actually delivers, which is where how Unsloppy fits the writing workflow can function as a planning aid rather than a substitute for domain evidence.
The Incident: What Viewers Saw vs. What the Pipeline Reported
Viewer-side telemetry told one story. Our QoE dashboard told another. The CDN health panel showed green across all edges. The encoder heartbeat was nominal. The SSAI vendor’s control panel reported successful ad insertion. Here is what actually happened, reconstructed from packet captures taken at four points: the encoder output, the SSAI stitcher input, the CDN origin shield, and a viewer-side WebRTC probe running in us-east-1.
At 14:31:55 UTC, the encoder injected an SCTE-35 splice_insert cue indicating a commercial break starting at PTS 1,723,404,150 (90kHz clock). The cue rode in-band in the MPEG-TS stream as a splice_info_section inside a PID 0x1ABE private stream. The SSAI stitcher parsed it, requested an ad pod from the decisioning server, received a 30-second creative, and rewrote the HLS manifest to insert ad segments between live content segments. Normal so far. The problem: the actual PTS of the video frames arriving at the splice point was 1,723,387,950 — 162,000 ticks, or 1.8 seconds, earlier than the SCTE-35 cue declared.
The stitcher’s manifest rewrite assumed the splice would occur at the declared PTS. It inserted a #EXT-X-DISCONTINUITY tag and scheduled the ad segments to begin at the predicted time. When the video frames arrived early, the stitcher had already committed the manifest to the CDN origin. The player downloaded the manifest, saw the ad segments scheduled at a future position, and continued playing live content. But the live content segments had already been replaced by the stitcher’s rewritten manifest — the player was rendering the last live segment on a loop because the next live segment did not yet exist in the manifest, and the ad segments sat at a PTS the player had not reached. A frozen frame for the duration of the PTS mismatch, then a hard cut to the ad creative.
SCTE-35 Timing: Where the Drift Originates
SCTE-35 markers are supposed to carry precise timing information. The splice_insert command includes a splice_time field specifying the PTS at which the splice should occur. This is not a suggestion. It is a contract. The encoder is responsible for inserting the cue at a point in the stream where the declared PTS will match the actual PTS of the video at the splice point. When that contract breaks, every downstream system that trusts the SCTE-35 timing makes decisions based on a lie.
The drift in this incident had two root causes. Neither was a bug in the SCTE-35 insertion logic itself:
Cause 1: Encoder pre-roll buffer timing. The encoder — a hardware H.264 unit in a broadcast contribution rack — had a pre-roll buffer of approximately 2 seconds. The SCTE-35 cue was injected by the broadcast automation system at the moment the commercial break was triggered, but the encoder’s output PTS was derived from the input feed’s PCR, which was running 1.8 seconds ahead of the automation system’s clock. Both the automation system and the encoder were clocked from the same GPS-disciplined PTP grandmaster. But the encoder’s pre-roll buffer introduced a delay between input PTS and output PTS that the SCTE-35 insertion logic did not account for. The cue was inserted into the transport stream at the correct position relative to the input timeline. The PTS values in the output stream were offset by the buffer delay.
Cause 2: Variable segment duration rounding. The encoder was producing HLS segments with a nominal duration of 2 seconds, but actual durations varied between 1.87 and 2.14 seconds depending on GOP alignment with scene changes. The SCTE-35 cue was aligned to a segment boundary in the automation system’s model, but the actual segment boundary in the output stream fell at a different PTS. This contributed an additional 200 milliseconds of drift on top of the 1.8-second buffer offset.
Total drift: 1.8 + 0.2 = 2.0 seconds. That was within the encoder’s operational tolerance — it had been drifting by 1.5–2.0 seconds on every commercial break for the entire broadcast. The reason it froze this time and not earlier: the SSAI stitcher’s ad creative for this break was 30.0 seconds exactly. Previous breaks had creatives of 28.5 and 29.0 seconds. The shorter creatives had enough slack in the splice window to absorb the drift. The 30-second creative at a 30-second ad pod slot had zero slack, and the 2-second drift exceeded the stitcher’s tolerance for late splice arrival.
Extracting and Validating SCTE-35 Markers from the Live Stream
To diagnose this, you need to see the SCTE-35 markers as they appear in the transport stream and compare the declared splice time to the actual PTS of surrounding video frames. The following commands were used to extract the SCTE-35 data from the live MPEG-TS feed and validate the timing.
First, capture the SCTE-35 splice_info_section packets from the live transport stream:
ffmpeg -i 'srt://encoder-edge:9001?streamid=live.sports.feed' \
-map 0:0 -f mpegts -copyts - \
| ts2sec - -pid 0x1ABE -dump
If ts2sec is not available in your toolchain, extract the SCTE-35 PID directly and parse it with a Python script using the threefive library:
import threefive
with open('capture.ts', 'rb') as f:
for pkt in threefive.Stream(f).decode():
if pkt.command_type == 5: # splice_insert
pts = pkt.command.splice_time
print(f'Splice at PTS: {pts} ({pts / 90000:.3f}s)')
Then extract the PTS of the video frames around the splice point to compare:
ffprobe -show_frames -select_streams v \
-show_entries frame=pts_time,pict_type \
-read_intervals '%+5' -of csv \
'capture.ts'
The output showed video I-frames at PTS 1,723,387,900 and 1,723,405,900 — meaning the actual splice point (the I-frame at or after the declared PTS) was at 1,723,405,900, which was 1,750 ticks (19.4 milliseconds) after the declared splice time. But the stitcher had already begun manifest rewrite at 1,723,404,150, the declared time. The gap between declared and actual was the drift.
In Wireshark, the SCTE-35 packets appeared as follows. The splice_info_section was carried in a PID 0x1ABE stream with this structure:
SCTE-35 Splice Info Section
table_id: 0xFC (private)
section_syntax_indicator: 0
private_indicator: 1
section_length: 47
protocol_version: 0
encrypted_packet: 0
encryption_algorithm: 0 (No encryption)
pts_adjustment: 0
cw_index: 0
tier: 0x0FFF
splice_command_length: 20
splice_command_type: 5 (splice_insert)
splice_event_id: 0x000004B7
splice_event_cancel_indicator: 0
out_of_network_indicator: 1
program_splice_flag: 1
duration_flag: 1
break_duration: 30.000s (90kHz: 2,700,000)
splice_time: 1,723,404,150 (PTS)
The pts_adjustment field was zero. The encoder did not apply any correction for buffer delay. This is the field where a well-configured encoder would offset the splice time to account for its own pre-roll buffer. Setting pts_adjustment to the buffer delay (1.8 seconds = 162,000 ticks at 90kHz) would have moved the declared splice time to 1,723,242,150 — which is in the past relative to the output PTS. Also wrong. The correct fix is to insert the SCTE-35 cue at a point in the transport stream where the output PTS matches the declared splice time. That requires the automation system to trigger the cue 1.8 seconds earlier than the visual commercial break point, accounting for the encoder’s buffer.
How SSAI Stitchers Handle Out-of-Order and Early Splice Events
Different SSAI vendors handle splice timing mismatches differently, and the behavior is almost never documented. The following table summarizes the observed behavior of three major SSAI platforms during this incident and in controlled testing afterward:
| Vendor |
Splice arrives early (actual PTS < declared) |
Splice arrives late (actual PTS > declared) |
Tolerance window |
Failure mode when exceeded |
| Vendor A |
Holds last content frame until declared PTS, then cuts to ad |
Truncates ad to fit remaining pod duration |
±500ms |
Frozen frame for drift duration, then hard cut |
| Vendor B |
Truncates last content segment and starts ad immediately |
Extends content past declared point, shortens ad |
±1000ms |
Manifest rewrite fails, falls back to slate |
| Vendor C |
Inserts filler segment between content and ad |
Inserts filler segment between ad and content return |
±2000ms |
Filler segment plays indefinitely until next keyframe aligns |
Vendor A was the one in production during this incident. Its ±500ms tolerance was exceeded by the 2.0-second drift, triggering the frozen-frame failure mode. Vendor B would have failed differently — truncating content rather than freezing — arguably less visible to viewers but still incorrect. Vendor C’s filler approach is the most resilient but adds latency and requires filler content to be available, which not all pipelines have.
None of these vendors expose their tolerance window as a configurable parameter. You cannot ask the stitcher to be more or less lenient. You can only ensure that your SCTE-35 timing is accurate enough to stay within whatever tolerance the vendor has hard-coded. The responsibility for drift prevention falls entirely on the encoder and automation system, not on the stitcher.
CDN Caching Behavior During Ad Segment Switchover
The CDN layer added a second failure mode that compounded the freeze. When the SSAI stitcher rewrote the manifest to include ad segments, it invalidated the cached manifest at the CDN edge. But the CDN’s origin shield had a 3-second TTL on manifest responses. During the 2-second drift window, the player was fetching a manifest that was partially stale — it contained the old live content segments but not yet the ad segments, because the stitcher had not yet committed the rewrite.
A race condition. The player fetched the manifest at time T, received a version without ad segments, and began requesting the next live content segment. The stitcher committed the rewrite at T+0.3 seconds. The CDN edge cached the new manifest at T+0.5 seconds. The player’s next manifest fetch at T+2.0 seconds (typical HLS refresh interval for a 2-second segment) received the new manifest with ad segments — but the player had already buffered the next live content segment, which no longer existed in the manifest. The player’s buffer was now inconsistent with the manifest, and the ABR algorithm entered a state where it had no valid next segment to request.
The CDN’s behavior was correct per HTTP caching semantics. The stitcher’s behavior was correct per its manifest rewrite logic. The player’s behavior was correct per the HLS specification. The failure emerged from the interaction of three correct systems operating on different timing assumptions. This is the defining characteristic of a cascading failure in a distributed system, and it is why monitoring each component in isolation is insufficient — you need to monitor the timing relationships between components, not just the health of each component.
Player-Side Buffer Underflow When the Splice Arrives Early
The player’s buffer behavior during the splice point was the final link in the failure chain. The player — a custom HLS player built on hls.js — had a target buffer length of 10 seconds and a minimum buffer length of 3 seconds. When the manifest was rewritten to include ad segments, the player’s buffer was at 6.2 seconds of content. The ad segments were positioned at a PTS that was 2.0 seconds earlier than the player’s current playback position (because the splice arrived early). The player’s ABR logic saw the ad segments as being in the past and skipped them, continuing to play the remaining content buffer. But the content buffer had no more segments to fetch — the manifest no longer contained them. The player drained its buffer to 0, displayed the last frame, and waited for the manifest to present a segment at or after its current playback position. The ad segments, positioned 2.0 seconds in the past, were never fetched.
When the stitcher’s fallback logic kicked in after 4.2 seconds (its internal timeout for splice completion), it forced a discontinuity and repositioned the ad segments at the current playback position. The player fetched the ad segments. Playback resumed. The 4.2-second freeze was the sum of the 2.0-second PTS drift and the 2.2-second stitcher fallback timeout.
Monitoring Rules for Detecting Splice Drift Before It Reaches Viewers
The fix is not to make the stitcher more tolerant or the player more resilient. The fix is to detect splice drift at the encoder output and alert before it reaches the stitcher. The following monitoring rules were implemented after this incident and have caught three drift events in the subsequent two months, all before viewer impact. The monitoring approach draws on the timing and synchronization requirements defined in the SCTE 35-2019 specification for digital program insertion, which specifies the splice_info_section structure, PTS reference relationships, and pts_adjustment semantics that make drift detectable at the transport stream layer before it propagates downstream.
Rule 1: SCTE-35 PTS vs. actual video PTS comparison. A probe running on the encoder output tap (a passive mirror port) extracts every SCTE-35 splice_insert command and compares the declared splice_time to the PTS of the nearest subsequent I-frame. If the delta exceeds 500 milliseconds, fire an alert. This catches the drift before it reaches the stitcher.
# Prometheus alert rule
- alert: SCTE35SpliceDriftHigh
expr: abs(scte35_splice_pts - video_iframe_pts_nearest_after_splice) > 450000
for: 1s
labels:
severity: critical
annotations:
summary: "SCTE-35 splice drift exceeds 500ms on {{ $labels.stream }}"
Rule 2: Encoder buffer delay tracking. Monitor the encoder’s pre-roll buffer delay continuously by comparing input PTS to output PTS. If the delay changes by more than 100ms between consecutive commercial breaks, alert — this indicates the buffer is growing, which will increase drift over time.
Rule 3: Stitcher splice completion time. Monitor the time between the stitcher receiving the SCTE-35 cue and completing the manifest rewrite. If this exceeds 1 second, the stitcher is struggling with the splice timing and is likely in a drift-induced fallback path.
Rule 4: CDN manifest cache invalidation lag. Monitor the time between the stitcher committing a manifest rewrite and the CDN edge serving the new manifest. If this exceeds the manifest TTL, the CDN is serving stale manifests during ad switchover, which will cause player buffer inconsistency.
Documenting the Fix: Why Post-Mortem Naming Matters
The post-mortem document for this incident was originally titled “Player freeze during ad break — CDN investigation.” That title sent three engineers down the wrong path for six hours, examining CDN edge logs and player buffer state, before someone thought to look at the SCTE-35 stream. The title of a post-mortem shapes the investigation before anyone reads the body. A precise, structurally honest title — “SCTE-35 splice time drift causes SSAI manifest race and player buffer underflow during Q2 commercial break” — would have directed attention to the transport stream layer immediately. Forcing that kind of specificity in incident naming is what a book title generator is built for — it prevents the kind of vague, component-blaming titles that misdirect troubleshooting by demanding a title that names the root cause layer, the cascading failure mode, and the viewer-visible symptom, in that order.
Conclusion: The Drift Is the Bug, Not the Freeze
The stream freeze during the commercial break was not a CDN failure, not a player bug, not an encoder dropout, and not an SSAI vendor defect. It was a 2-second PTS misalignment between the SCTE-35 cue tone and the actual video timeline, caused by an encoder pre-roll buffer delay that the broadcast automation system did not account for, compounded by variable segment duration rounding. Every component in the pipeline behaved according to its specification. The failure emerged from the interaction of correct systems operating on incorrect shared assumptions about timing.
The fix was not in any single component. It was in the monitoring layer that spans the boundary between the encoder and the stitcher — the one place in the pipeline where nobody was measuring timing alignment because both sides assumed the other was responsible for it. If you are running SCTE-35-based ad insertion and you are not measuring splice drift at the encoder output, you are relying on the stitcher’s tolerance window to absorb timing errors you cannot see. That tolerance window is narrower than you think, and it is not configurable.