Submission type
Talk/session proposal (25 min + 5 min Q&A). A 20-minute showcase also works: outline items 3 to 5 and 7.
Describe your session in 2 paragraphs
You are defined by the company you keep. This talk follows that one old saying through three philosophies, three types of algorithm and nine seminal papers from two decades, from PageRank in 1998 to GraphRec in 2019. Take the saying literally in three ways and you can rebuild three algorithms. Important company makes you important: pass the importance around until it settles, and you get PageRank. Your company says what you are like: learn to predict the company, and you get word2vec. You wear a different mask in different company: weigh each neighbour by how well it matches what you look for, and you get attention.
This is how I do it. I start with each philosophy, a plain argument about the world, and only then write it as maths. Before each answer, I ask the room how they would solve it. Then I change the ideas, one detail at a time. Put a weighted average inside word2vec, with each neighbour weighted by how well it matches what the word looks for, and you are close to attention. Keep an algorithm, swap the table under it or change one detail of it, and you get six more of the papers in the grid below. Each change can be a new algorithm. So the audience can make their own, instead of taking what somebody built and extending it minutely. Then I put all three to work. Every meeting note, ticket and failed test creates a task, and someone has to own it. With a startup, I am building an attribution engine on their Jira and GitHub data: PageRank for who owns the code, vectors for who works like whom, attention for whose past work fits this task. Who should own it? Ask the company they keep. I have not built it yet. I will build it and present it. These are only building blocks, so the talk ends on one question: can you do better?
The nine papers, placed on a grid:
|
Pass it around until it settles |
Predict the company |
Weigh the company |
| Pages and graphs |
PageRank, 1998 |
DeepWalk, 2014; node2vec, 2016 |
Graph attention networks, 2018 |
| Words and sentences |
TextRank, 2004 |
word2vec, 2013 |
Transformer, 2017 |
| People |
Who to Follow at Twitter, 2013 |
DeepWalk, 2014, tested on social networks |
GraphRec, 2019, weights learned from the two people’s vectors |
Outline (25 minutes + 5 minutes Q&A)
- 0:00 Why the basics still matter. Someone told me college basics like minimax no longer matter. I disagree: the foundations build the intuition you apply. The whole talk follows one saying: you are defined by the company you keep.
- 1:00 Everything is a table. Put things in the rows and what surrounds them in the columns: pages and the pages that link to them, words and the words around them.
- 2:00 Pages: who vouches for you? A page’s own text can say anything, but the links to it are written by other people. So a page is important if important pages link to it. That is circular, and the room tries to solve it before I show the answer: start all pages equal, pass the scores along the links, spread a little evenly so no loop traps it, and repeat until they settle. That is PageRank. Its limit leads to the next scene: a score says how much a page matters, not what it is like.
- 7:00 Words: known by their neighbours. Describe each word by the words around it. That table is far too big to write down, and the room tries to shrink it first. word2vec gives each word a short vector whose only job is to predict its neighbours, so words with the same neighbours end up close. Its limit: one vector per word, so “bank” sits halfway between the river and the money.
- 12:00 One word, many meanings. People behave differently with different friends, so a word should too. The room proposes a fix first. The answer: rebuild each word’s vector in every sentence, weighing each neighbour by how well it matches what the word is looking for. That is attention, the core of the Transformer. My own false start: I once imagined word2vec with a weighted average of its neighbours, was sure such extensions existed, and missed that this is attention.
- 17:00 Same ideas, new tables. The three ideas work on any table. Walk the grid above: random walks on a graph, read as sentences, turn word2vec into DeepWalk; on people, the same ideas give Twitter’s Who to Follow (who you keep landing on) and GraphRec (how much each friend counts). The room names a table from their own work.
- 21:00 Live experiment: an attribution engine. I take one line from a meeting note, such as “fix the flaky payments test before Friday”, and the room guesses who should own it. Here the rows are people and the columns are the code and tickets they touch, and each idea does one job: PageRank finds who owns the code the line points to, vectors find who works like whom, and attention weighs each person’s past tickets and pull requests against this task. An agent combines them into one owner, with the evidence. Then the scoreboard: on past tasks, frozen at each task’s opening with the assignee and mentions hidden, how often the right person is in the top three, compared with the files’ top editors, the person first assigned, and chance. I also check which people it never suggests. I am building it with a startup on their Jira and GitHub data; not built yet. Results are shared in aggregate, with their approval.
The examples before 21:00 are synthetic: they show how each algorithm behaves, not how well it works.
Mention 1-2 takeaways from your session
- See one idea where you saw three algorithms. PageRank, word2vec and attention each define a thing by the company it keeps.
- Turn an argument into an algorithm. Start from a sentence you would accept about the world, write it as maths, and solve it.
- Apply it to something new. Name the company in your own data, then pick a column of the grid, or change one detail and make your own.
Which audiences will benefit most from your session?
Data scientists, analysts and ML engineers who use rankings, embeddings or LLMs and want the reasoning behind them. Students who wonder whether the basics still matter. Once the intuition is clear, these algorithms become building blocks, like LEGO, and you start to see new problems to apply them to.
Add your bio: who you are, where you work, and any relevant context
Varun Tulsian leads AI at Fermi.ai. He was VP, Head of AI at Needl.ai, where he worked on retrieval and question answering over long finance documents. He spent about eight years on search and recommendations at Walmart Global Tech. He has an ME in Computer Science from IISc. He writes about ML on his blog, wity’ai (https://varun-ml.github.io/).
LinkedIn: https://www.linkedin.com/in/varuntulsian
GitHub: https://github.com/varun-ml
Slides
Not yet. The written version of the talk is here: https://varun-ml.github.io/posts/pagerank-word2vec-attention/
Topic/domain tags
information retrieval, search, recommendations, open source, ranking, PageRank, embeddings, word2vec, attention, transformers, graph ML, NLP, first principles
{{ gettext('Login to leave a comment') }}
{{ gettext('Post a comment…') }}{{ errorMsg }}
{{ gettext('No comments posted yet') }}