The Fifth Elephant 2017

On data engineering and application of ML in diverse domains

How Paytm uses k8s for global expansion

Submitted by Pranshu Saxena (@pranshus) on Tuesday, 4 April 2017

videocam_off

Technical level

Intermediate

Section

Full talk for data engineering track

Status

Submitted

Vote on this proposal

Login to vote

Total votes:  +3

Abstract

At Paytm, we are constantly engaged in creating new environments and aligning infrastructure for standard services such as Authentication, Access, Logging/Monitoring etc. There is also the case of dynamic resource allocation, high-availability, scalability, security - then factor in ‘x’ number of environments and you have a fairly complex problem to solve. This is especially the case for big data infrastructure where many of these aspects are dynamic and changes happen very frequently - we’re constantly trying to hit a moving target. Enter Kubernetes: In this talk, we will dive into how we addressed these challenges and how we leveraged Kubernetes and Infrastructure as code to not only deploy big data enviornments with integrated logging/monitoring, dynamic resource allocation and high-availability but also reduce costs and increase efficiency in a secure, automated rollout.

Outline

-Infrastructure as code (Pre-Deployment) -Deployment (on AWS) -Deploying applications/services -Scaling/HA -Monitoring/Logging -Security considerations

Requirements

We can present this subject as in-depth and technical or present it as a mid level, business orientated talk

Speaker bio

We will find out who can travel once selected and then share this info. It will surely one of the people who worked on this project from our office in Toronto.

Comments

  • 1
    Sandhya Ramesh (@sandhyaramesh) Reviewer a year ago (edited a year ago)

    Hi Harinder, will it be only one person giving this talk? Since you’ve mentioned “we will talk”, I want to clarify that as a policy, we allow only one speaker on stage. An assistant (for demos etc) are allowed, but only one person speaks.

  • 1
    Sandhya Ramesh (@sandhyaramesh) Reviewer a year ago

    Hi Harinder, we’re evaluating proposals currently. Could you upload a slide deck and a two minute video of whoever is presenting the talk? Thanks!

  • 1
    Zainab Bawa (@zainabbawa) Reviewer a year ago

    Harinder, we require the speaker details beforehand to discuss comments, and in case the proposal is shortlisted, the proposed speaker will have to do a preliminary rehearsal for final confirmation.

  • 1
    Zainab Bawa (@zainabbawa) Reviewer a year ago

    Couple of other questions for this proposal:
    1. Why is Kubernetes a better choice than other technologies available?
    2. What is the big data problem for Paytm? Need more details about the context and why this problem is crucial.
    3. What are the insights and larger picture for audiences not in the payments domain to take back from this talk?

  • 1
    Abhishek Kumar Singh (@abhishekkrsingh) a year ago
    1. Why is Kubernetes a better choice than other technologies available?
      We choose Kubernetes over Docker Swarm and Marathon because we wanted modular container orchestration engine that doesn’t tie itself to very specific tools for doing networking, service discovery, loadbalancing, monitoring, management etc like Docker Swarm natively supports only docker container while Kuberntes is agnostic to underlying container engine. We looked at the community and their support and we found Kubernetes communitity bigger then Docker Swarm and Marathon and we felt we will get balanced support from the K8s community.

    2. What is the big data problem for Paytm? Need more details about the context and why this problem is crucial.
      Currently, we are not using K8s for big data but we are planning to use soon for running our big data services on top of k8s.

    3. What are the insights and larger picture for audiences not in the payments domain to take back from this talk?
      This talk is for everyone who run or wants to run micro-services architecture on K8s on scale. We will share our insights about how we do Kubernetes Networking with policy, Service Deployment ( rolling updates ), Service Discovery, Load-balancing , Monitoring, Vault, and Security of K8s Cluster.

  • 1
    Pranshu Saxena (@pranshus) Proposer a year ago

Login with Twitter or Google to leave a comment