The Fifth Elephant 2023 Monsoon

Background

Large Language Models have demonstrated amazing capability for solving complex problems. But they can’t answer what they haven’t seen, and to take advantage of these amazing models, we need to expose our data to the model. Fine-tuning is not an option, at least not a cheap one. Prompt engineering is a helpful technique to provide context to LLMs, which helps the model restrict its answer to the context.

To use context in real-time queries, we need to process and store them beforehand. This includes vectorizing and storing in a vector database. This works well if we were just querying for similar text, but production application use cases are complex. Your application data is sitting in a structured relational database like Postgres and vector data in a vector store like Pinecone. Data fragmentation is always a hard problem to solve.

Additionally, semantics searches are not a replacement for conditional queries. Prices <$20 is not the same as Price >$20. Yet, it will come out as highly similar because conceptually they are similar, i.e. it is a comparison statement with price value.

In some cases, we may also be required to query vectors with predicate filtering, i.e. filtering similarity search on a subset of documents.

This is not a solved problem; some vector dbs provide additional metadata store to run such queries as if it was structured data, but it isn’t the same as your data model sitting in a relational db like Postgres. And, if data fragmentation wasn’t enough of a problem, we are now talking about data duplication. And what about failures and data states going out of sync?

And, if all of my application data, structured data, is sitting in a relational database and my contextual data in a vector database, how can I securely access my vector data with role-based access control and authorization when all of the data for a role is sitting in relational database.

Comparison study

I will motivate the talk by talking through different queries that one works with vector dbs.

We will also get a sense of popular vector dbs like Weaviate, Pinecone, Milvus, etc., to understand the capabilities they support.

Data APIs to rescue

We will understand how Hasura’s Data API solves these problems by providing an abstraction over different data stores giving the capability to query with remote joins as it was all in the same storage along with authentication and Role-based access control, a big gap in current vector db workflow.

Additionally, we will walk through some of the features like event triggers can help set up auto-vectorization on insert/update/delete events in no time without your vectorization workflow getting tied down to a specific vector db.

The super powers of authentication and RBAC can also be applied to your LLM calls using remote actions in Hasura. This can be extended by creating custom prompts and executing RBAC over them. For example, a LLM query like summarise public data can be open to all but summarising employee data is open to only managers and HR.