How your smart TV knows what you're watching, even over an antenna (ACR explained)

What automatic content recognition is, and what it is not

Automatic content recognition (ACR) is software built into a smart TV that works out what is on the screen and reports it to a server. It belongs to the TV's operating system, not to any app, which is the whole point: it sees content from an antenna, a cable box, a game console or a laptop on HDMI, a phone mirrored to the screen, and the TV's own streaming service. Northeastern's researchers put it as four steps: the TV captures short snapshots of the audio or the picture, sends those snapshots or fingerprints made from them to the ACR provider, the provider matches them against a library of known content, and the result is used for targeted ads and audience measurement.

It is not a recording of your living room, and the TV does not upload the video you are watching. What leaves the set is small and regular, a few hundred numbers or an audio fingerprint at a time. It is also not the same thing as an app knowing what you watch inside that app. Netflix knows what you watch on Netflix. ACR is how the TV maker knows what you watch everywhere else.

The researchers cite a 2025 estimate that 82% of US TV households have a smart TV that supports ACR.

Why TV makers run it: the Vizio numbers

Vizio is the one maker whose public filings split the TV business from what happens on the TV afterwards. Its 10-K for 2023 reports two segments:

• Device (selling TVs and sound bars): $1,081.8 million in net revenue and a gross loss of $8.6 million.
• Platform+ (SmartCast, its TV operating system with ads and content distribution, plus Inscape, its data business): $598.2 million in net revenue and $364.9 million in gross profit.

Total gross profit was $356.3 million, so Platform+ made more than all of it. The 10-K describes Inscape as "our ACR technology", able to identify what is on screen "regardless of input source", and says the data is used to target ads and is licensed to measurement companies and agencies. ACR revenue on its own is not broken out, so the honest reading is that viewing data is one part of the business that turns a break-even TV into a profitable one, not the whole of it. The same filing says Netflix and Disney+ restrict Vizio from using viewing data from their apps, and that viewers have to opt in at setup and can opt out later. Walmart bought Vizio in December 2024, so 2023 is its last full year as a public company.

How a TV can recognise a show from a few colours

US patent 8,595,781 was filed in 2009 by Cognitive Media Networks, the company Vizio bought and renamed Inscape, and granted in 2013. It describes one way to do the matching. It is a method, not proof of what any TV ships today, and Samsung's algorithm in particular is not public.

On the TV side:

• A system function wakes up periodically, "e.g., every 0.1 second", and reads pixel data from N patches of the frame, with 13 as the example.
• A patch is a small square of the screen. Its pixels are reduced to a single RGB value, a function of the pixels inside it, and "that function may be averaging". The patent calls the set of values for one frame a pixel cue point.
• Cue points for a run of frames, ten in the example, are accumulated and sent to the server roughly every second, with a timestamp.

every 0.1 s   13 patches → 13 average RGB values      (one cue point)
every ~1 s    POST { timestamp, cueData: 10 cue points }
              10 frames × 13 patches × 3 channels = 390 numbers

For scale, one 1920x1080 frame is 6,220,800 RGB values. The cue throws away almost all of it and keeps a coarse colour signature that still changes from moment to moment, which is enough to tell programmes apart when you look at many frames in a row.

An "offline" server does the same thing to live channel feeds, sampling the pixels at the same rate as the TVs and storing the cue points for each programme with their offsets in time, plus metadata such as closed captions.

How the server finds the match: nearest neighbours, buckets and suspects

The server cannot look a cue up exactly. The colours a TV reports never land precisely on the stored ones (compression, scaling and the TV's own picture processing all nudge them; that list is ours, and the patent simply models the gap as noise), so the problem is a nearest-neighbour search: find stored cue points that are close, not equal. The patent's matching algorithm is built for "ambiguous cues".

To keep that search small it uses locality-sensitive hashing (LSH), a family of hash functions designed so that inputs that are close together usually land in the same bucket. The patent hashes each incoming cue point to pick the sector of the database to search, and only that sector is searched. That is the opposite of what a cryptographic hash is for: SHA-256 is built so that changing one bit of the input changes the output completely, which is right for passwords and useless for "close enough".

Inside the sector, every stored cue point within a small radius becomes a suspect: a programme and a time offset that might be what the TV is showing. The patent calls the procedure path pursuit. Each suspect gets a probability. As each new cue point arrives the server updates the list, keeps the suspects whose next stored cues keep lining up with the incoming ones, drops the rest, and discounts older evidence over time. When one suspect's probability passes a threshold, that programme and offset are the answer. The patent's own picture is a bead sliding along one wire in a tangle of wires: the wires are the programmes' paths through cue space, the bead is the TV's stream of cues, and the job is to say which wire it is on. It also notes the algorithm can recognise only seven of ten frames and still find the most likely location.

Recognising sound instead of pictures

Not every system looks at the picture. Audio-based ACR samples the sound and computes an audio fingerprint, in the family of techniques behind Shazam and the Haitsma-Kalker fingerprint. LG's ACR comes from Alphonso, which LG owns, and is audio-based. VIDAA's partner, Unruly, also sends audio fingerprints. This is a fingerprint of the audio the TV is playing; the study measured traffic to ACR servers and did not involve the TV's microphone.

What Northeastern measured, and how

Smart TVs are locked down, so the team could not root them or read their encrypted traffic. Instead they built an automated living room: 12 TVs from ten manufacturers covering the eight operating systems that run about 95% of US smart TVs (Tizen 34%, Roku OS 18%, webOS 15%, SmartCast 11%, Google TV 8%, Fire TV 5%, and VIDAA and Xumo within the remaining 4%). A control machine played the same content on every TV through three routes: their own broadcast channel over the antenna input using HDMI-to-RF modulators, an HDMI feed, and screen mirroring. Every packet each TV sent was captured. Each of ten viewing scenarios ran 30 times with ACR on and 30 times after opting out, in random order, for 5,943 experiments in all.

Finding the ACR servers among everything a TV talks to was a method in itself. Starting from three reference cases whose ACR domains were known (acr-us-prd.samsungcloud.tv for Samsung, tkacrXXX.alphonso.tv for LG and XX.fp.us.acr.unruly.co for VIDAA), they kept candidate endpoints that:

• appeared in at least 24 of 30 opted-in runs,
• were specific to one operating system rather than shared ad or CDN services,
• received more data than they sent back (the reference endpoints receive about 2x, 7x and 7x),
• received traffic on a clock, found with Fourier and autocorrelation analysis: every 7 seconds for VIDAA, every 14 seconds for LG and about every minute for Samsung, up to about 135 seconds for Sony's Google TV.

That left one endpoint per system: scribe.logs.roku.com for Roku, gateway-ink.amazon.com for Fire TV, an address in Samba TV's block for Sony, and an Amazon Kinesis Data Streams endpoint for Vizio. Xumo had no ACR settings, no candidates and, its maker confirmed, no ACR.

Which inputs get watched

A scenario counted as tracked when the ACR endpoint received regular traffic in more than 80% of the 30 opted-in runs, and the results were close to all or nothing:

• Antenna: tracked on every operating system they could analyse. Over-the-air TV has no app to track it, which makes it a natural target.
• HDMI: tracked on five of seven, not on Roku OS or SmartCast.
• The TV's own streaming service: tracked on most; Fire TV and Sony's Google TV were the exceptions.
• Netflix and YouTube: never tracked, which matches the restrictions Vizio's filing describes for Netflix. Tubi, which is free with ads, was tracked on Tizen, Sony's Google TV and VIDAA.
• Screen mirroring: tracked only on VIDAA.
• Idle: Vizio sent ACR-pattern traffic even sitting on its home screen, in every run.

The blank-and-mute test

Because the team controlled their own antenna channel, they could remove one sense at a time: a black picture with normal sound, or moving pictures with no sound. Traffic that disappears when the picture goes black points to video fingerprinting; traffic that disappears when the sound goes points to audio.

• Samsung (Tizen): no traffic to the ACR endpoint in all 30 black-screen runs, in either state, and removing the sound changed nothing. Video. That fits the Samsung Ads guide, which says its ACR takes "glass-level screenshots every 500 milliseconds" on opted-in TVs and converts them into patterns. Samsung's US support page separately says the feature does not record or watch the content; both statements are Samsung's.
• Sony's Google TV (Samba TV's ACR): traffic fell 52.9% on a black screen. Video.
• LG (webOS, Alphonso): traffic fell 97.3% on average with the sound removed, and not at all with the picture removed. Audio.
• VIDAA: audio, confirmed directly, because its messages are unencrypted.
• Fire TV, Roku and Vizio: traffic looked the same either way, so the method cannot tell. Some systems may send fixed-size records whatever is on screen.

Turning it off, brand by brand

Every system with ACR offers a switch, and five of eight name it without the words "automatic content recognition":

• Samsung: Viewing Information Services
• LG: Live Plus
• Sony (Google TV): Samba Interactive TV
• Vizio: Viewing Data
• Roku: Use info from TV inputs
• Fire TV: Automatic Content Recognition
• VIDAA: Enhanced Viewing Options

After opting out, small background traffic under 5% of the opted-in volume was set aside, and anything more was flagged as consistent with ACR. Opting out fully stopped it on four systems: LG, Sony's Google TV, Fire TV and VIDAA. On Samsung, traffic consistent with ACR continued in 41 of 60 antenna runs, 18 of 30 HDMI runs, 6 of 30 Tubi runs and all 30 runs of Samsung's own streaming service, more than 60% of the scenarios evaluated. On Vizio the setting had no observable effect on traffic to the Kinesis endpoint. On Roku the switch does not cover Roku's own streaming service; TCL's settings screen says so and Roku TV's does not.

Two caveats matter. Most of this traffic is encrypted, so the team inferred its purpose from timing, size and destination and cannot say what the data contains or whether it is used after the switch is off. And an earlier study (Anselmi et al., IMC 2024) found Samsung's opt-out did stop ACR traffic, on different firmware and with a different switch. The 2026 study reflects the devices and firmware it tested.

What the video left out: data in the clear

Two findings are about exposure rather than recognition.

VIDAA sends its ACR data to Unruly's servers over MQTT without encryption, so anyone on the network path can read it. Each message carries a permanent device ID that cannot be reset, the time zone, the app name, the input (antenna or HDMI), the antenna channel and the audio fingerprint, and the server replies with a genre, including categories such as Religious, Health and Medicine, and Political. The broker also accepted subscriptions without a login: in an ethically limited three-hour sample the team saw more than 57 million detection results from more than a million TVs. VIDAA added authentication after the disclosure; when the team checked in September 2026 the traffic was still unencrypted.

Separately from ACR, Vizio's free streaming service WatchFree+ puts the name of the show or channel into unencrypted DNS lookups and SNI fields for Amagi, the company that delivers the streams, so an internet provider or whoever runs the network can rebuild a list of what was watched. That leak does not depend on the ACR setting and has no off switch.

Terms

Automatic content recognition (ACR)

Software in a smart TV's operating system that identifies what is on screen by fingerprinting the picture or sound and matching it on a server.

Pixel cue point

The patent's name for one frame's worth of patch colours, the unit the TV sends.

Fingerprint

A compact summary of a piece of audio or video that stays similar when the content is the same, even after compression or noise.

Nearest-neighbour search

Finding the stored items closest to a query rather than an exact match.

Locality-sensitive hashing (LSH)

Hashing designed so that similar inputs usually get the same bucket, used to narrow a nearest-neighbour search.

Path pursuit

The patent's matching procedure, which tracks several candidate programmes and offsets and keeps the one whose cues keep lining up.

Opt-out

The TV setting that is supposed to stop ACR; its name differs by brand.

Platform+

Vizio's reporting segment for its TV operating system, ads, content distribution and Inscape data business.

MQTT

A lightweight publish-and-subscribe messaging protocol, used by VIDAA's ACR without encryption.

Questions

// further reading

• Watching the Watchers, project page ↗: the findings with interactive charts
• The paper (PDF) ↗: methods, endpoint identification and every result
• ACR blocklist ↗: the researchers' list of ACR endpoints to block at the router
• US patent 8,595,781 B2 ↗: Inscape's cue points, LSH and path pursuit
• Anselmi et al., IMC 2024 ↗: the earlier two-TV ACR study
• Vizio 10-K for 2023 ↗: the Device and Platform+ figures