What is HLS streaming?

The short answer

Asked plainly, what is HLS streaming? It is HTTP Live Streaming, published by Apple in 2009, and it is now the ordinary way video reaches a browser or a phone. It got there by being unremarkable. It is an adaptive bitrate video streaming protocol, which is a long way of saying two things: video is cut into short pieces, and the player chooses how good a version to fetch as it goes.

The trick is good. None of it needs a special server. The pieces are ordinary files fetched over ordinary HTTP. Any web server, any content delivery network, any cache already built for images and stylesheets will serve video without being told it is video.

The problem HTTP live streaming solved

Before it, streaming meant a dedicated protocol and a dedicated server speaking it. RTMP was the common one. It worked. It also carried two costs, and both turned out to be fatal.

Two of them, and they compounded:

• Infrastructure. A streaming server is a separate thing to run, scale and pay for, and it cannot borrow the enormous caching layer the web had already built for everything else.
• Firewalls. Corporate networks routinely blocked the ports those protocols used, so a video that played at home failed at work, for reasons nobody watching could diagnose or fix.

HTTP live streaming sidestepped both by refusing to be special. If the video is just a series of small files fetched over port 443, then every cache on the route already knows how to store it, every firewall already lets it through, and scaling is a problem somebody else solved a decade earlier. That is the whole idea. Everything else is detail.

The manifest and the segments

Two ingredients. Segments are the short pieces of video, typically two to ten seconds each. A manifest is a text file listing them in order. That is the entire protocol, and everything that follows is a consequence of those two sentences rather than an addition to them.

The manifest is the part you can read. Reading one is the fastest route to understanding the protocol, faster than any explanation of it:

#EXTM3U
#EXT-X-VERSION:3
#EXT-X-TARGETDURATION:10
#EXTINF:9.009,
segment0.ts
#EXTINF:9.009,
segment1.ts
#EXT-X-ENDLIST

That is a complete stream. Line by line:

• #EXTM3U must be first, or a strict player rejects the file unread.
• #EXT-X-VERSION says which revision of the specification applies.
• #EXT-X-TARGETDURATION is the longest any segment may be, in seconds.
• #EXTINF gives the duration of the segment named on the next line.
• #EXT-X-ENDLIST means no more segments are coming.

A player fetches the manifest, sees two segments, requests the first, starts playing, and requests the second while the first is still on screen. The buffer is how far ahead it got.

Note what the server is doing during all of this. Nothing at all. It is answering requests for files, the same as it would for a logo, with no notion that a video is in progress and no state to keep about who is watching what.

Adaptive bitrate, and what it actually does

Here the second half of the name earns itself. The same content is encoded several times, at different qualities. A master manifest lists them:

#EXTM3U
#EXT-X-STREAM-INF:BANDWIDTH=800000,RESOLUTION=640x360
360p/index.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=2400000,RESOLUTION=1280x720
720p/index.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=6000000,RESOLUTION=1920x1080
1080p/index.m3u8

The player times how long each segment took to arrive and compares it against how long the segment lasts. Three outcomes:

• Arriving faster than it plays means headroom. Try a better rendition.
• Arriving slower means trouble ahead. Step down now, before the buffer empties.
• Arriving at about the same rate means stay where you are.

The decision is the player's. Not the server's. The server published the options and has no idea which one is being taken. That is why the same stream can serve a phone on a train and a desktop on fibre without anyone configuring anything.

It also explains the most misdiagnosed complaint in streaming. A stream with a single rendition has nothing to step down to, so when the connection dips it cannot degrade. It stalls instead. People report that as buffering and blame the player, and the cause is usually a master manifest with one line in it where there should have been three.

Live and on demand are the same format

One tag. That is the whole difference. A finished recording ends with #EXT-X-ENDLIST, which tells the player every segment that will ever exist is already listed. Total duration is knowable, so a seek bar is possible.

A live stream simply omits it. The player re-requests the same manifest every few seconds, expecting new segments at the bottom and old ones dropped off the top, and the file is a moving window rather than a description of a whole. Nothing else changes. Same syntax, same segments, same server doing nothing clever.

This is why a live manifest saved to disk is useless an hour later. It names segments that have long since rotated away.

Segments are not always transport streams

The original specification used MPEG transport stream for the pieces, which is why the segments you meet most often end in .ts. That was sensible in 2009. Transport stream came from broadcast, it tolerates being joined mid-flow, and the tooling already existed.

It carried a cost. Nobody minded, until later. Transport stream wraps everything in 188-byte packets with their own headers, so a few percent of every byte you pay for is packaging, and the format shares nothing with the containers the rest of the web uses.

So HLS learned a second segment format: fragmented MP4. Same protocol, same manifests, same tags, different pieces. The gain is not really the saved overhead. It is that fragmented MP4 is exactly what the competing DASH protocol uses, so one set of encoded files can serve both, and a publisher who once paid to store and push two complete copies of every video now stores one. That is the practical reason it caught on.

You will see both in the wild. Telling them apart takes a glance:

• Segments ending .ts are transport stream, the original flavour.
• Segments ending .m4s are fragmented MP4.
• An #EXT-X-MAP tag pointing at an initialisation segment also means fragmented MP4, and its absence means transport stream.

Players handle either without being told which.

Subtitles and second audio tracks

Renditions are not only about quality. Far from it. A master manifest can also declare alternative audio and subtitle tracks with #EXT-X-MEDIA, each pointing at its own manifest, which is how one stream offers a second language or captions without duplicating the video.

Worth knowing because of how it fails. Those tracks are separate requests to separate addresses, so a stream can play perfectly while its subtitles never appear, and nothing in the picture tells you why. The manifest does. If a player shows no subtitle option at all, the usual answer is that the master never declared one.

The cost: latency

Nothing is free. The bill here is delay. A player generally wants a few segments buffered before it starts, and segments are several seconds each, so a viewer sits somewhere between fifteen and thirty seconds behind real time on a conventional setup. For a film that is irrelevant. For an auction closing on a clock, or a broadcast where being half a minute behind means hearing the result before you see it, it is the whole problem.

Shorter segments reduce it. They cost more requests. Low-Latency HLS, added later, pushes partial segments to get the delay down to a couple of seconds, at the price of needing more from the server than plain file hosting. Which returns to the trade the protocol has always made: it is simple and universal because it is willing to be a little late.

HLS players, and why the browser decides

Support splits along a line that catches people out:

• Safari and iOS. Native. Hand it a manifest and it plays.
• Smart televisions and set-top boxes. Native, almost universally.
• Android. Native in the platform player, and in most apps built on it.
• Chrome, Firefox and Edge. Not native. They need JavaScript to do it.
• Desktop applications such as VLC. Native, and less constrained than any browser.

Chrome, Firefox and Edge do not, and that catches people out. On the desktop the job falls to JavaScript: a library fetches the manifest, downloads segments, unpacks the video and audio from each one and feeds them to the video element through Media Source Extensions. The browser never learns it is playing HLS. It is handed something it already understands, continuously, by code running on the page.

Which leaves one honest caveat. All of that runs inside the browser's security model, so a stream whose server declines to send the header permitting cross-origin reads will not play on a web page no matter how good the player is. The same address in a desktop application works, because a desktop application is not bound by that rule. It is the single most common reason a stream that is demonstrably alive refuses to play in a browser.