Rootconf Hyderabad edition

Modern development moves fast, with businesses expected to push out multiple iterations and updates in very shorter sprint cycles. And with the current release model, daily deployments are unavoidable. Doing a deployment without impacting the system is being used by 150,000 business across the world, serving 500K requests per minute was a very big challenge. And building test cases to capture all production like scenarios is highly difficult since many issues might occur only during high load with complex business use cases.

Ever thought about testing a code with production traffic? This talk will showcase how we test major changes with live traffic.

The code will become eligible for GoLive only after ensuring the error rates, status codes & performance on the request shadow environment.

What ?
Mirroring portion/full live production traffic to an air-gapped shadow VPC to mimic the actual requests.

Why?
To test major changes in Code/Infra before rolling to Production
■ Docker
■ Rails/Ruby Upgrade
■ OS version Upgrades

Outline

Internals - How it works?

In actual VPC:
Every web request is stamped at HAProxy (by looking at domain, cookies etc)
Stamped requests are mirrored by Envoy sidecar.

In shadow VPC:
Requests are received by NLB and forwarded to HAProxy (similar to Production setup)
All external http requests go via shadow proxy

• Requests can be mocked
• App & all services should be configured to honor the http proxy variable

Read only Infra IAM Role for all machines in Shadow VPC
NAT level ACL rules to control outbound traffic

Tech Stack:

HAProxy
Envoy
Diffy - developed by Lua (https://github.com/twitter/diffy)
Golang
Lua

Speaker bio

Vasanth Pandian is a Sr. Devops Engineer at Freshworks