Rootconf Delhi edition

On network engineering, infrastructure automation and DevOps

Tickets

The good, the bad and the ugly of Kubernetes - A Practitioners View!

Submitted by Vijay Dharap (@dharapvj) on Saturday, 19 October 2019

Section: Full talk (40 mins) Category: Systems engineering Status: Rejected

Abstract

Kubernetes continues to take the world by the storm every passing day. But at the same time, it continues to be fairly complex beast to tame, manage and be beneficial to the businesses.

In this talk, I would like to quickly look at:

  • why I like Kubernetes and what benefits does a k8s based infrastructure can offer (the good),
  • what challanges did I face when I implemented my k8s cluster and how did I overcome them. What are the typical gotchas to look out for when you implement Kubernetes (the bad)
  • And last but not the least – What part of kubernetes perplexes me and what am I doing about it (the ugly)

I hope that I can take the crowd on the journey filled with practical experiences – accompanied by small code snippets – and give them an insight on how they can watch for some pitfalls and have a successful kubernetes implementation.

Outline

  • Small overview kubernetes exposure
  • The Ugly Some of the decision which backfired upon me
    • Choices of EKS as Kubernetes service (replaced it with KOPS later)
    • Grafana based Alerting (Replaced with Prometheus Alerting)
    • kubeadm install - etcd failure in master nodes
  • The Bad – Some issues faced which became apparent little later but were resolved:
    • certificate length and cloudfront
    • Access log monitoring and analysis – ELK / EG
    • EFS starvation issues - separate your monitoring infrastructure from your application infrastructure
  • The Good – What I loved during my journey
    • helm charts - why is it awesome, how can it be even more easier with below tips
      • hub.helm.sh why to use
      • Install vs upgrade –install
      • configmap gotcha
      • locking down the version
    • Resource limit and health of your worker nodes
    • IaC and terraform – a lifesaver… (oh… and also a pain to deal with)
    • KOPS – a lifesaver
    • Prometheus – a highly extensible monitoring tool – ephemeral storage, etcd health, certificates, and other OOTB tools like jmx_exporter, blackbox, etc
    • Nginx-Ingress and automatic ELB vs ALB-nginx-ingress
  • Tips and Tricks
    • K8s specific Productivity enahancements: kube-ps1, kubens, aliases, etc
    • General linux productivity enahancements: git ps1, byobu, putty default values for timeout resolution, dealing with corporate CA issued certificates
    • Secrets are namespace specific. How to copy secret from one namespace to another
    • Ensuring you don’t lose your home directories in IaC
  • There is no silver bullet
    • IaC where to draw the line between terraform vs scripted application
    • Reflection on what should NOT be in kubernetes

Speaker bio

I am a Principal Technology Architect at Infosys in India with broad ranging experience from architecting large enterprise web applications to cloud infrastructure design and deployment in DevOps mode.

I am very passionate about Open Source and am a Developer Advocate within Infosys. I have spoken at conferences in India like GIDS, JSFoo, ReactFoo, etc. I organize technical conferences within Infosys. I enjoy keeping up-to-date with trends in UI and UX world and love discussing technical topics on twitter at my handle @dharapvj

Slides

https://drive.google.com/open?id=1f2hlmVLhyc_ML0HLIeYDqOji_2xj9ik7

Comments

  •   Zainab Bawa (@zainabbawa) Reviewer 4 months ago

    Thanks for this proposal, Vijay. Help us understand who is the target audience for this proposed talk?

    •   Vijay Dharap (@dharapvj) Proposer 4 months ago

      Hi Zainab..

      Everyone is adapting / planning on adapting Kubernetes in 2020. There is serious shortage of experience on setting up and managing kubernetes properly. So, quite a few, like me, would get pushed into setting k8s and managing it without a chance to learn it under an experienced k8s administrator.

      The target audience will be all such Infrastructure Architects and leads who may or may not have great knowledge on kubernetes and can use insights about what works well in kubernets and what are the murky areas and how to deal with them - without going all-in blindsighted.

      I am also thinking that, may be, I can rewrite this with title “Reporting live from Kubernetes Warfront…” and then can segregate topics in which areas we are “winning” (aka good), which areas are under heavy fights (the bad) and which areas we need to really put our best sqadron (the ugly).

      Hope that answers your question.

  •   Zainab Bawa (@zainabbawa) Reviewer 4 months ago

    Also, can you share draft slides to help us understand the content and the scenarios you are trying to cover. This will help us decide faster.

    •   Vijay Dharap (@dharapvj) Proposer 4 months ago

      I would do that in next 3-4 days and would attach them here.

      •   Zainab Bawa (@zainabbawa) Reviewer 4 months ago

        The slides will help us understand what content you are covering and the general problem statement. When we reviewed the proposal in the editorial meeting, our concern was that we don’t want generic prescriptions, but your war story with Kubernetes and your learnings from the process. We look forward to the slides. Add the link to your slides in the slides section.

        •   Vijay Dharap (@dharapvj) Proposer 4 months ago

          Hi Zainanb,

          Good morning! I have prepared some draft slides. https://drive.google.com/open?id=1f2hlmVLhyc_ML0HLIeYDqOji_2xj9ik7. Slides are in pptx format for now. I realized that some of my ppt animation is not working well in google slides. I will fix it in coming days and have a google slides deck ready.

          For now - Please use download button and open in powerpoint and run “presenter view” to see my narration.

          This is a WIP draft. But I have added textual content as much as possible. Also I have polished few slides for you to get an idea of how the slides would be presented. In such slides - you will also see “Narration” in notes section. That’s why viewing the deck in slideshow mode and Presentor view will be most effective.

          Slides which have been polished to 80% readiness are - 1,2,4,5,6,7,9,15.
          Rest slides are WIP but the text should give an idea of the content. As many slides as possible - will have loads of screenshots to make it easy for audience to digest. But it is taking time to get all bad scenarios reproduced and screenshotted.

          Hope for now - this version will suffice.

          Comments, criticism welcome.

  •   Karthik Venkateswaran (@karthikvt26) 4 months ago (edited 4 months ago)

    Hi Vijay,

    The outline looks super packed and impressive :) The first thing that came to my mind based on the above topics is to understand each one of them by imagining the lifecycle of a simple contrived service (Service deployment, monitoring, observability, scaling it, managing the certs, exposing it etc.,) (I know this may not be the best example to evaluate it, but would love to understand in terms of them to see the good, bad and ugly emotions). Here are the questions that popped up in mind.

    1) The problems which encouraged you to use kubernetes and how you set it up and used it (would imagine back in the day there was no managed kubernetes services and probably compare that pain in the current times)
    2) Once you are into kubernetes, the challenges you had faced in terms of managing the services (applications/services managed using helm) and how the k8s/community tools assisted in making it easy. This will probably cover most of the good parts.
    3) The problems which encouraged you to look for monitoring tools and the impact of those tools in terms of good, bad and ugly nature.

    These were the questions at the back of my mind. Would love to see the flow you had imagined and plan to put it up in your slides.

  •   Zainab Bawa (@zainabbawa) Reviewer 4 months ago

    Thanks for the slides, Vijay. My colleague Anwesha Sarkar will review the slides and get back to you with the decision.

    Meanwhile, one question that does come up in the case of Rootconf is that we look at infrastructure from the point of an organization and not so much from the POV of an individual. Given that this work on K8s is your own experience, while working in an organization, the question about what are the learnings, patterns and anti-patterns for organizations rather than for individuals. Have you thought about this?

Login with Twitter or Google to leave a comment