The Fifth Elephant 2013

Mozilla tames a rabble of HBase, PostgreSQL, RabbitMQ, elasticsearch, and Python to chew through 50 Firefox crash reports each second. Explore the strengths and weaknesses of these tools and their consequent niches in the greater crash-catching system. Greet the complexities that emerge from the combination, and see how we engineer around them to keep our never-lost-a-crash record pristine.

Outline

Receiving and organizing every Firefox crash in the world is a big job. Concurrency, realtime constraints, and a volume of data 110TB strong all contribute to the challenge of giving Firefox engineers what they need to find and squash browser bugs.

Follow a Firefox crash from its genesis in a collapsing browser process through the dizzying array of collection, storage, and reporting systems that make up Socorro, our open-source crash collector. Enjoy war stories of weird, interlocking failures, and see how we nevertheless continue to fulfill our mandate: “Never lose a crash.”

Speaker bio

Erik Rose coordinates the impact of 108 spring-loaded buttons at Mozilla, venting a byproduct of static analysis, search, and pattern-finding software. His past selves have done realtime fuzzy matching against the corpus of U.S. voters at Votizen, caused the Django community’s tests to run in funny orders, written a book about Zope and Plone, and released a bevy of eclectic Python libraries. When not speaking or coding, Erik retreats to his volcanic fortress in the wilds of North Carolina, where he discusses formal language theory with his dog, Max.

Links

• PyCon 2013 talk explaining elasticsearch’s internal data structures and how they should drive your implementation decisions: https://www.youtube.com/watch?v=lWKEphKIG8U. Slides: http://www.slideshare.net/ErikRose/elasticsearch-the-missing-intr
• Mozilla talk about designing beautiful APIs: https://air.mozilla.org/poetic-apis/. Slides: http://www.slideshare.net/ErikRose/poetic-apis
• PyCon 2012 talk, Parsing Horrible Things: http://www.youtube.com/watch?v=tCUdeLIj4hE
• DjangoCon 2012 talk about making your tests run 5 times as fast: http://www.slideshare.net/ErikRose/djangos-nasal-passage
• Github: https://github.com/erikrose/
• Twitter: http://twitter.com/ErikRose

Slides

https://gist.github.com/erikrose/5603537