Scale Search Infrastructure with Apache Solr and Kubernetes

Submitted May 28, 2020

Kubernetes is fast becoming the operating system for the Cloud and brings a ubiquity that has the potential for massive benefits for technology organizations. Applications/Microservices are moved to orchestration tools like Kubernetes to leverage features like horizontal autoscaling, fault tolerance, CICD and more.

Apache Solr is an open-source search engine platform built on an Apache Lucene library. It offers Apache Lucene’s search capabilities in a user-friendly way. Lucidworks Inc runs over a thousand distributed-mode Apache Solr Clusters spread across several machines for a plethora of use-cases around Search and Analytics. The traffic demands a massive scale which creates scenarios of in-depth micro-management like operating systems upgrade, scaling cluster dynamically, etc, affecting the overall search experience.

This talk is focussed on the journey taken by Lucidworks on addressing scaling clusters horizontally and vertically, on the basis of query traffic load, data ingestion throughput or any other relevant metrics by extending capabilities of Kubernetes and Apache Solr to achieve true physical and logical autoscaling, satisfying modern era SLAs and infrastructure cost. The talk concludes with how the solution adopted opens up the future scope of fine-grained scaling of search clusters.

Outline

Motivation.
- Why we need smart self-scale system for any e-commerce search.
Limitations in open source search engines, like Solr and Elasticsearch at scaling.
Apache Lucnee and Apache Solr.
- Short introduction to Apache Lucene & Solr.
- Which tools of Solr are used for a typical e-commerce search.
- Discuss Autoscaling feature in Apache Solr.
Short Introduction to Kubernetes.
- Discuss Horizontal Pod Autoscaling feature (HPA) of Kubernetes
Discuss the integration of Autoscaling feature of Solr and HPA of Kubernetes to achieve traffic driven search cluster.
- Showcase an example configuration.
- Live Demonstration - simulate live traffic, and see a live search cluster scaling up and down.
Discuss future scope
- Scaling search cluster on fine-grained attributes, instead of overall traffic load
Referneces

Speaker bio

Amrit Sarkar is Cloud Search Reliability Engineer at Lucidworks Inc, California-based enterprise search technology company, with 5+ years experience in search domain and big data, e-commerce and product. He is working primarily on running search-based applications on Kubernetes, and developing and improving core components of Apache Solr.

The Fifth Elephant 2020 edition