The Fifth Elephant 2017
On data engineering and application of ML in diverse domains
BM
Bharath Mohan
@bharathmohan
Do you know what's on TV?
Submitted May 22, 2017
Section:
Full talk for data engineering track
Technical level:
Intermediate
The mobile has made tremendous progress - but it is still referred as “second screen” to the Television. Television (specifically Linear TV) will continue to be the most efficient way to get high quality content to millions of homes. Even though all the devices around us have gotten smarter - people still watch TV by memorizing channel numbers and move between painful guides. At the root of this problem is the lack of enough “smart” data about what’s happening on TV. Do you know?
- What channels are playing English Action Movies that are not in an Ad Break?
- Do you know which Ad is playing on (say) Star Plus?
- Do you know which celebrities are on screen - right now - across 400 channels?
- Do you know which shows are talking about Trump, right now?
Deeper data about Linear TV not only enables much better experiences for the viewer, but the entire industry.
Bharath Mohan walks you through some amazing patterns in the way Ads, House Promos, and other artifacts are played out on Television, and how some of these were exploited to build algorithms that predict what Ads are playing on TV.
Outline
Full slides are at https://drive.google.com/file/d/0B_ctBNRYUwf9bWRDMlVtbFc3QlU/view?usp=sharing
Speaker bio
Bharath Mohan loves to study how information flows through society - and create products that make the right information get to the right people. He got his PhD on this topic at IISc, Bangalore - mining for nurturers among computer science researchers. He then went on to work at Google News - studying how news starts off from an original source, and is quickly copied or re-hashed by several publishers across the world. He’s been doing startups over the last few years, and the latest one Sensara.TV is about unraveling Television.
Slides
https://drive.google.com/file/d/0B_ctBNRYUwf9bWRDMlVtbFc3QlU/view?usp=sharing
Login to leave a comment
Bharath Mohan
@bharathmohan Submitter
Hi Zainab, the talk is for data scientists, machine learning practitioners and enthusiasts, even businesses and entrepreneurs that are trying to evaluate the potential for machine learning in Television. Very few people in India have looked at Television data mining, and there are truck loads of opportunities there - when all of TV viewing will become "connected".
Zainab Bawa
@zainabbawa
Hi Bharath, we need draft slides or some form of detailed outline of your talk for us to understand what is the content you intend to cover.
Bharath Mohan
@bharathmohan Submitter
Do you know what's on TV?
What is TV and where is it headed?
A quick preview.
TV is about the content you see on the screen. It is in two forms: Linear TV (the TV you get from Satellite or Cable) and Non-Linear TV (the TV you get On Demand, largely from the Internet).
Linear TV is still going strong if you consider its large numbers - although in some parts of the world they say Non-Linear TV is eating into TV.
There are some analogies you can draw between them.
Linear TV is like Mumbai Local Train system. It runs on its own. Large masses of people get in and get out.
Non-Linear TV is like an Uber or Ola. You hail for a Taxi. There's one just for you. You get in and get out at your destination, at your time and at your place.
Now, will Uber and Ola replace the Mumbai Local Train System? Never. It simply wont scale for those numbers. But both will co-exist.
You should expect TV to co-exist, and even better become Hybrid.
Non-Linear TV has merely taken the place of DVD Players. Those DVDs are gone. They are all in the Cloud now, and coming in wires over the Internet.
The Linear TV world was designed for a world devoid of the Internet
You have entire stacks of technology that did not assume any Internet. Broadcasters acquired content licenses, actual tapes, and then transmitted "streaming" content to satellite, that was then collected using dish farms, and transmitted over wires to homes - to be then decrypted and shown on TV.
As a consumer, you'd always consume content through a Set-top Box, that had no return passage, and was not a connected device. There was no reason to.
With Hybrid TV coming in, you'll soon see Set-top Boxes or TVs themselves allow you to consumer both Linear and Non-Linear TV together. These boxes are going to be "connected" to the Internet, and can send back data about what is being watched.
Imagine, something you always had over the Internet. The ability to find out which unique user just visited a page - has never been available to the largest medium of information delivery. And that is all about to change.
You'll have millions of boxes ready to transmit data back about what is being watched. But is the TV industry ready to do things with that data?
Connected Linear TV opens many doors
Imagine a system that knew:
Not at Internet scale, but at TV scale.
Here are some business usecases:
TV Synchronized Advertising: Ads (or follow on ads) that show up on your mobile, as soon as you've seen the same ad on TV.
Celebrity Apparel: Ways to shop for apparel worn by celebrities that just came on TV.
Advertising Measurement: The exact people (or their stereotypes) that watched an ad on TV.
To do all this, we first need to know "What exactly is on TV?"
The state of metadata on Linear TV
The EPG (Electronic Program Guide) is as good you get right now. The EPG describes what Show is on which channel. You'll probably get more metadata about the Show itself, the actors, the duration, and so on. But there's no available data on:
You cannot wait for Broadcasters to give you this data. A lot of technology needs to change for that.
The world is about to need them all. So, let's start reverse engineering this data.
So, what exactly is Linear TV made up of?
Linear TV looks like this (if you allow me to use a regular expression):
(ContentSegment . BreakMarker? . (Ad|HousePromo)+ . BreakMarker? ContentSegment)+
The EPG you get largely tags the Content Segment, but wont tell when the Ad breaks.
But no body is going to tell you about BreakMarker, Ad, HousePromo etc. There is no database to look up. No standards or compliance. No watermarks (at least those that are accessible to third parties).
Goal: Can we figure out ContentSegment . BreakMarker. Ad . HousePromo .
Look at TV as continuous streaming video: 60 fps pictures + aligned audio + EPG metadata.
Are there patterns that we can employ to unearth these segments, and tag them appropriately?
Here are some heuristics:
Break Markers: Signature sequences of video or audio that are specific to a channel, and/or a show. If they signatures that belong to a show, they always come along with the show, and very seldom else where. They are also sentinels that separate the content from the ads.
Ads: Ads are short. Ads are heavy on audio. Ads repeat a lot. Ads occur across channels. Ads always come together.
House Promos: House Promos come with Ads. [This kind of makes them hard to distinguish]. House promos feature actors that are regular in the Show. This is unlike Ads.
So, let's start mining for clips - that repeat a lot in TV. A clip is a maximal sequence of audio/video that occurs intact, and repeats many times on TV.
This is maximal sequence mining problem. NP(Hard). We need to make it more efficient. Let's do it in audio first. Audio can be converted to numbers - using fingerprinting algorithms (references).
A temporal reverse index is built, and used to mine for repeating sequences of audio.
Not all audio repetitions are true clips. Eg: Fake laughter. Background music.
Different audio clip candidates could be slight alterations of each other. Eg: Variants of ads.
Continue the processing in video domain, by creating a more constrained set of candidates around audio candidates.
After candidate clips are obtained, "score" them - to label them as break markers, ads or house promos.
Demo of the whole system in action:
AdBreaks.in tags ads in real time on Linear TV. I'll show the audience how audio candidates are generated, how we generate video candidates from them, and further get clips to the exact frame. Will further show how break markers and house promos are classified.
Alongside, I'll also demo use cases where AdBreaks.in has been used.
The road ahead:
Zainab Bawa
@zainabbawa
Bharath, also elaborate on who is the target audience for this talk?
Zainab Bawa
@zainabbawa
What is in it for folks who are not using TV, but other mediums, to attend your talk?
Abhishek Balaji
@booleanbalaji
Hi Bharath,
Please share draft slides detailing the content you will cover in this talk and what the key takeaway is for participants. We need this information by 29 May to evaluate your proposal.