Rootconf 2014

On devops and cloud infrastructure

Mesos: Cluster management framework and it's role at Shopify

Submitted by Yagnik (@yagnik) on Friday, 18 April 2014

Section: Full talk Technical level: Intermediate Status: Confirmed & Scheduled

View proposal in schedule


As applications get scaled across multiple server, there is a need to maximize resource utilization, minimize data replication across different applications by allowing applications/frameworks to share data centre. The talk will introduce Apache Mesos, a framework for sharing resources across nodes in a data centre along with details about it’s architecture. It will also cover how Shopify uses Mesos to power its infrastructure and scale on demand.


As infrastructure moves towards service oriented architecture, it has become common to spin up numerous servers which underutilize the resources available to them. Along with that, we tend to replicate our data across various frameworks such as Hadoop and MPI. There is a need for a framework which will allow us to maximize the utilization of fine grained resources such as memory, cpu core, io bandwidth and allow us to share data across different frameworks. This talk will focus on the introduction and architecture of Mesos, a platform for resource sharing among multiple cluster computing frameworks such as Hadoop, Spark, MPI and others. After getting some details of Mesos and its architecture, the talk will dive into how Shopify uses Mesos and use of marathon, a framework built on top of Mesos to scale Shopify.

Speaker bio

I am a software developer at Shopify.



  •   Shivangi Chopra (@shivi555) 5 years ago

    Sounds interesting

  •   Sreekandh Balakrishnan (@gnuyoga) 5 years ago

    do u have a slideshare ?

    •   Yagnik (@yagnik) Proposer 5 years ago

      I’m sorry Sreekanth, I’m still working on the slides.

  •   Sreekandh Balakrishnan (@gnuyoga) 5 years ago (edited 5 years ago)

    Will be great if i can see a sneak peak of the same to get a sense of what we are going to cover. Do you think it will be ready before tomororow or the coming week ?

    •   Yagnik (@yagnik) Proposer 5 years ago

      I could get some rough slides done by the end of the week.

  •   Ranjib Dey (@ranjibd) 5 years ago

    will this be a focused session on mesos itself (containerization, resource offering model, generic HA abstractions etc) or the ecosystem i.e. mesos as resource manager, and the scheduler or other frameworks on top of it?

    •   Yagnik (@yagnik) Proposer 5 years ago

      @ranjibd, initially I was thinking of former where I’ll get into details of mesos but some were interested in knowing more about how Shopify uses mesos, how we scale, some deployment strategies etc. Do you have any opinion on which might be better suited for the conference?

  •   Ranjib Dey (@ranjibd) 5 years ago

    great. mesos a generic resource management framework is worth knowing :-)

  •   Sreekandh Balakrishnan (@gnuyoga) 5 years ago

    Am super excited !! Definetly looking forward to hear your talk. Summary of the call
    1. No of server Shopify is manaing
    2. What problem triggered you to think about using Mesos
    3. What are the options available to solve the problem that you have
    4. Mesos - the good and bad ( can we talk about it being bulky over Core OS ?)
    5. Is it used in Production
    6. After using Mesos what’s the overall impact to internal customers ( can we do a before and after metrics)
    7. How does it failover, how can we rollback ?
    8. What other learning from Shopify we can share ?
    9. Demo of your production setup ? or a basic intro demo ?

    let me know what you think. I hope the suggestions are useful.

    Once the slide is ready let me know we will do round 2 of our discussion.

Login with Twitter or Google to leave a comment