Scaling databases with a database access layer
Submitted by Gaurav Gupta (@gagupta79) on Saturday, 26 April 2014
Section: Full talk Technical level: Intermediate Status: Confirmed & Scheduled
The objective of this session is to discuss an architecture to scale and manage the database tier by introducing a database abstraction layer. We will discuss various challenges of database scalability and then discuss few solutions.
We will be taking few references from the Facebook's WebscaleSQL project, Google's spanner paper and MySQL proxy.
When an application goes viral it not an easy feet to scale the infrastructure to support it. It requires some crafty deployments, constant improvement in the application stack and a lot of automation in the infrastructure to serve the growing business.
For the past couple of years, I have been looking at the problem of scaling the database tier efficiently. Database (especially relational database) remains an interesting frontier to scale up in the multi-tier architecture. NoSQL offers promise of scale for certain use cases, but there a large ecosystem for SQL that remains and will continue to grow. I propose a database abstraction layer that offers great potential to scale, optimize, perform and manage the traditional database tier. This layer will virtualize the databases from applications and provides a single interface to DBAs, DevOps and Developers to manage, test and develop their database tier.
I will be talking about certain common optimization techniques that can be offered from this layer:
- Instant scale up - transparent connection pooling and multiplexing, transparent caching, surge protection
- Transparent scale out - read/write split, load balancing, query routing, Sharding
- Automatic high availability - automatic failover
- Real-time Analytics
I am currently the VP of Engineering at ScaleArc, where we are building a solution like I described above. For the last 12 years, I have been working on building high-performance sub-systems on Linux in file systems, storage and networking domain. Most recently I was the founding engineer at CloudVelocity, where I helped build a hybrid cloud platform. Before that I worked as a Principal Software Engineer at Symantec, Cisco Systems and NeoPath Networks, and a contributor to OpenStack. The time I spent in these startups was a great experience and I hope I can share some of these learnings with fellow engineers.
In short, I enjoy sharing ideas on innovation, engineering and product development. I am looking forward for all the interesting sessions proposed on this conference.