arrow_back Building the Infrastructure to Handle Big Data
Submitted by Kaushik Paranjape on Thursday, 28 June 2012
Big Data Infrastructure & Processing
Learn about how MySQL database can be scaled up to for your requirements.
Start ups that have a web based product typically start with a database that can fit on a single instance. As the product grows and the client base grows, they start facing scaling issues. As the data grows query time increases, insert time increases, the data grows out of bound. Tech team starts looking for scaling solutions, they look at no-sql databases, hadoop based solutions and what not. We at sokrati have gone through this phase and finally decided to come up with our own sharding solution. Sharding is a simple concept where data on a single instance of mysql (one shard) is kept within limits, once the shard is full or tends to become full you bring up another instance and data is populated on the other shard. This sharding solution solves multiple problems
Data is distributed across multiple shards, making the queries faster.
Fancy dbs like no-sql are not required, hence reducing the re-engineering of all the apps
There is a single point of contact (sharding db) for accessing any database, hence slave credentials can be returned depending on type of access (read only accesses can go the slave).
Multiple slaves can be added to achieve load balancing.
If client has data-secrecy issues and wants its data to be hosted separately, can be easily achieved by having a separate shard for that client.
Kaushik Paranjape is a Co-Founding Geek at Sokrati. Sokrati is the leading Search Marketing platform in India.