Rootconf 2014

As applications get scaled across multiple server, there is a need to maximize resource utilization, minimize data replication across different applications by allowing applications/frameworks to share data centre. The talk will introduce Apache Mesos, a framework for sharing resources across nodes in a data centre along with details about it’s architecture. It will also cover how Shopify uses Mesos to power its infrastructure and scale on demand.

Outline

As infrastructure moves towards service oriented architecture, it has become common to spin up numerous servers which underutilize the resources available to them. Along with that, we tend to replicate our data across various frameworks such as Hadoop and MPI. There is a need for a framework which will allow us to maximize the utilization of fine grained resources such as memory, cpu core, io bandwidth and allow us to share data across different frameworks. This talk will focus on the introduction and architecture of Mesos, a platform for resource sharing among multiple cluster computing frameworks such as Hadoop, Spark, MPI and others. After getting some details of Mesos and its architecture, the talk will dive into how Shopify uses Mesos and use of marathon, a framework built on top of Mesos to scale Shopify.

Speaker bio

I am a software developer at Shopify.
http://ca.linkedin.com/in/yagnik/

Links

• http://mesos.apache.org/