Abstract

Glance inspires consumers to make the most of every moment by surfacing relevant experiences for them with its ‘smart lock screen’ innovation. More than 225 million consumers enjoy Glance on their Android smartphones across markets. Glance harnesses the power of Machine Learning (ML) to provide consumers with a highly personalized and engaging user experience featuring top content from both local and global publishers across a diverse range of topics. The key to delivering this experience has been a commitment to rapid and continuous experimentation. Enabling this is Alchemist - our state-of-the-art in-house experimentation platform.

When building Alchemist we had to take several unique design decisions to fulfill our goal of massive concurrent experimentation. For example, we needed to shift away from hash-based experiment targeting to a richer model capable of targeting any combination of user attributes with full experimentation lifecycle management. We had to develop a high-performance config server for experiment delivery across a vast user base that switched between offline and online states. We introduced automatic metric generation tied to the platform for seamless performance analysis. These core features of Alchemist have boosted the rate of ML experimentation, enabling us to test more features for our users and rapidly scale those they love.

In this talk, we will share our insights and lessons learned from developing and running Alchemist and how these can be applied to boost the rate of experimentation in your products and services.

Overview of the talk

This talk navigates the ambitious task of executing rapid experimentation at an immense scale. We explore the challenges that come with such a large scale, from ensuring personalization policies are honoured to maintaining performance, and the innovative solutions we’ve developed to meet these challenges head-on.

Business Implication

The cornerstone of our solution is Alchemist, our in-house experimentation platform. Alchemist features a sophisticated engine capable of targeting any combination of user attributes with full experimentation lifecycle management, a high-performance config server for the user base, and a unified metric bed that ensures verified metrics are accessible in a single place. These features alongside other aspects of Alchemist have increased our speed of experimentation; accelerating the pace of innovation.

Key Learnings

We discuss the business implications of Alchemist, shedding light on how it empowers Product Managers, Engineers, and other stakeholders to conduct massive concurrent experimentation. With Alchemist, the typically time-consuming process of setting up experiments becomes swift and efficient, all consolidated within a single portal, providing out-of-the-box metrics.

Beyond ML Experimentation

The impact of Alchemist transcends beyond ML experimentation, its versatility makes it a powerful tool for experimenting with any aspect of the user experience - be it UX, personalization, or backend modifications.

Join us as we explore the dynamic world of rapid experimentation enabled by Alchemist. Learn how you can apply these core learnings and principles to your own experimentation practices.