Background

With the recent technological advancement in LLM’s, embedding generation and Retrieval Augmented Generation(RAG), there is immense interest in using these technologies to solve problems involving Semantic Search, Chat Bots, Code Graph, Knowlegde Graphs etc.

This has led to the development of Data Management systems which can deal with large scale generation, storage and search capability for these vector embeddings.

This Birds of Feather (BOF) session was held at The Fifth Elephant 2024 conference on 13th July. The BOF was targeted towards a technical audience who were either using Vector Databases in production, or were exploring the possibility of adding VectorDBs to their tech stack.

Outline

What are vector databases? Why should one care?

• What problems are solved by Vector DBs over traditional databases?
• Brief overview about RAG and how Vector DBs fit into the picture.
• Compare with a traditional database based on architecture and based on problems solved.

How are vector DBs built? How do they work under the hood?

• Brief, high-level overview about the inner workings and considersations made while building a Vector Database

What are the main Vector Databases out there in the market? How does one decide which one to go with?

• Compare with popular DBs. Explore their pros & cons.
• Which DBs are good for which speicifc use cases?
• Thoughts of going with something like pg_vector which have full text search search primarily with an additional option to add in vector search.

Performance comparison with some metrics

• Ingestion speed, query speed, Recall.

Any drawbacks with VectorDBs in production

• What are the concerns one might have when using these in production?
• Any specific caveats one should keep in mind while using VectorDBs. E.g. access control.

Discussion summary and key takeaways for audience

1. Recall vs. efficiency trade-off: In the Space Partitioning based Vector indexes achieving very high recall (>95%) in vector search often requires probing additional clusters, leading to diminishing returns in terms of computational resources versus recall improvement.
2. Quantization for memory optimization: Binary quantization techniques can significantly reduce memory usage while maintaining acceptable performance. For example, 36 million Wikipedia paragraphs were stored in just 2GB of RAM using binary quantization.
3. Hybrid search strategies: Combining traditional database filtering with vector search can improve query performance. Filtering on regular data before performing vector search reduces the number of vectors distance calculation if the selectivity is high.
4. Challenges in index updates: Updating large vector indexes while maintaining system performance and recall is a challenge. Updates to index are CPU intensive especially for graph-based indexes and as updates happen maintaining quality of index is also a challenge. Mirroring and rebuilding on separate machines before switching was suggested as a solution for major updates.
5. Embedding quality is crucial: The effectiveness of vector search heavily depends on the quality of the underlying embedding model. Techniques like PCA can be used to visualize embeddings and check for effective clustering.
6. Sequential vs. random access: For certain data sizes, sequential scans can be more efficient than random reads. This is particularly relevant when the data doesn’t fit entirely in memory and graph-based approaches lead to random access patterns.

Moderator and speakers at the BOF

1. Aldrin Jenson moderated the discussion. He is Product Engineer at Athena Intelligence. Aldrin was part of the Indic Subtitler team which was shortlisted for Hack5 2024 edition Demo Day competition.
2. Rajkumar Iyer works on the effort to build Native Vector Database support in Microsoft SQL Server.
3. Chaitanya Chokkareddy is co-founder at Ozonetel.
4. Aditi Ahuja is SWE-2 at Couchbase. She is part of the organizing team at Bengaluru Systems meet-up.
5. Sumesh Meppadath is Product Manager at GalaxEye.