Do you know what’s on TV?

What is TV and where is it headed?

A quick preview.

TV is about the content you see on the screen. It is in two forms: Linear TV (the TV you get from Satellite or Cable) and Non-Linear TV (the TV you get On Demand, largely from the Internet).

Linear TV is still going strong if you consider its large numbers - although in some parts of the world they say Non-Linear TV is eating into TV.

There are some analogies you can draw between them.

Linear TV is like Mumbai Local Train system. It runs on its own. Large masses of people get in and get out.

Non-Linear TV is like an Uber or Ola. You hail for a Taxi. There’s one just for you. You get in and get out at your destination, at your time and at your place.

Now, will Uber and Ola replace the Mumbai Local Train System? Never. It simply wont scale for those numbers. But both will co-exist.

You should expect TV to co-exist, and even better become Hybrid.

Non-Linear TV has merely taken the place of DVD Players. Those DVDs are gone. They are all in the Cloud now, and coming in wires over the Internet.

The Linear TV world was designed for a world devoid of the Internet

You have entire stacks of technology that did not assume any Internet. Broadcasters acquired content licenses, actual tapes, and then transmitted “streaming” content to satellite, that was then collected using dish farms, and transmitted over wires to homes - to be then decrypted and shown on TV.

As a consumer, you’d always consume content through a Set-top Box, that had no return passage, and was not a connected device. There was no reason to.

With Hybrid TV coming in, you’ll soon see Set-top Boxes or TVs themselves allow you to consumer both Linear and Non-Linear TV together. These boxes are going to be “connected” to the Internet, and can send back data about what is being watched.

Imagine, something you always had over the Internet. The ability to find out which unique user just visited a page - has never been available to the largest medium of information delivery. And that is all about to change.

You’ll have millions of boxes ready to transmit data back about what is being watched. But is the TV industry ready to do things with that data?

Connected Linear TV opens many doors

Imagine a system that knew:

The channel you just saw.
The ads you did not switch away from.
The actors you did not miss when you watched a movie on TV.
The songs you waited for, even if you had to go past an ad break.
The news topics you track.

Not at Internet scale, but at TV scale.

Here are some business usecases:

TV Synchronized Advertising: Ads (or follow on ads) that show up on your mobile, as soon as you’ve seen the same ad on TV.

Celebrity Apparel: Ways to shop for apparel worn by celebrities that just came on TV.

Advertising Measurement: The exact people (or their stereotypes) that watched an ad on TV.

To do all this, we first need to know “What exactly is on TV?”

The state of metadata on Linear TV

The EPG (Electronic Program Guide) is as good you get right now. The EPG describes what Show is on which channel. You’ll probably get more metadata about the Show itself, the actors, the duration, and so on. But there’s no available data on:

What ads just played?
Which House Promotion just played?
Are we in an Ad Break?
Which actor is on screen?
What was just said on TV?

You cannot wait for Broadcasters to give you this data. A lot of technology needs to change for that.

The world is about to need them all. So, let’s start reverse engineering this data.

So, what exactly is Linear TV made up of?

Linear TV looks like this (if you allow me to use a regular expression):

(ContentSegment . BreakMarker? . (Ad|HousePromo)+ . BreakMarker? ContentSegment)+

The EPG you get largely tags the Content Segment, but wont tell when the Ad breaks.

But no body is going to tell you about BreakMarker, Ad, HousePromo etc. There is no database to look up. No standards or compliance. No watermarks (at least those that are accessible to third parties).

Goal: Can we figure out ContentSegment . BreakMarker. Ad . HousePromo .

Look at TV as continuous streaming video: 60 fps pictures + aligned audio + EPG metadata.

Are there patterns that we can employ to unearth these segments, and tag them appropriately?

Here are some heuristics:

Break Markers: Signature sequences of video or audio that are specific to a channel, and/or a show. If they signatures that belong to a show, they always come along with the show, and very seldom else where. They are also sentinels that separate the content from the ads.

Ads: Ads are short. Ads are heavy on audio. Ads repeat a lot. Ads occur across channels. Ads always come together.

House Promos: House Promos come with Ads. [This kind of makes them hard to distinguish]. House promos feature actors that are regular in the Show. This is unlike Ads.

So, let’s start mining for clips - that repeat a lot in TV. A clip is a maximal sequence of audio/video that occurs intact, and repeats many times on TV.

This is maximal sequence mining problem. NP(Hard). We need to make it more efficient. Let’s do it in audio first. Audio can be converted to numbers - using fingerprinting algorithms (references).

A temporal reverse index is built, and used to mine for repeating sequences of audio.

Not all audio repetitions are true clips. Eg: Fake laughter. Background music.
Different audio clip candidates could be slight alterations of each other. Eg: Variants of ads.

Continue the processing in video domain, by creating a more constrained set of candidates around audio candidates.

After candidate clips are obtained, “score” them - to label them as break markers, ads or house promos.

Demo of the whole system in action:

AdBreaks.in tags ads in real time on Linear TV. I’ll show the audience how audio candidates are generated, how we generate video candidates from them, and further get clips to the exact frame. Will further show how break markers and house promos are classified.

Alongside, I’ll also demo use cases where AdBreaks.in has been used.

1) TV Ad Research.
2) TV Ad Playout Monitoring. TV Synchronized Advertising.
3) Dynamic Ad Replacement in OTT streams. [A collaboration with Amagi]

The road ahead:

Language-independent Hot keyword detection in news.
Celebrity frames, and detection of clothing.
Linear TV segmentation for Catch Up TV.

The Fifth Elephant 2017