Balancing Relevance, LLMs, and Latency with Progressive Retrieval

Abstract

One-line summary

Exploring how to make company discovery smarter and faster when user language doesn’t match unstructured business data, balancing search relevance, LLMs, and latency through progressive retrieval.

What problem are you addressing?

We built a search over a large universe of companies. The main data for each company is one broad business description. There is no granular taxonomy (sector, sub-sector, product, business model or use case) to filter or group by. This gave us four connected problems.

1. The user’s words and the data’s words do not match: A company can be very relevant to a concept and never use the user’s term.

2. The search string is much shorter than the text it is compared with: Raw query embeddings underperformed because short user queries and long business descriptions are fundamentally different text types, causing vector search to favor surface-level word overlap rather than true business relevance.

3. Company descriptions contain multiple aspects. A single description covers a company’s products, customers, geography, business model and other topics. Its embedding mixes these concepts, making it harder to identify companies relevant to a specific search intent.

4. Making the search smarter made it slower.

We introduced additional query processing such as aliases and LLM-based understanding to improve the interpretation of user input. However, every preprocessing step added latency. The system could become better at understanding a query while simultaneously becoming worse at delivering a responsive search experience. This forced us to rethink the assumption that everything needs to happen before retrieval starts.

The core engineering question was: how do we improve search relevance over unstructured company descriptions without making users wait for every improvement to finish before retrieval begins?

Who is the intended audience?

Developers, Platform Engineers, Data/ML Engineers, Engineering Leaders

Practical takeaways

• Make search understand intent, not just keywords. Use query expansion and intent-focused representations to bridge the gap between what users mean and what company descriptions say.
• Improve relevance without making users wait. Start with fast retrieval, then progressively refine results with more expensive processing instead of making every improvement a prerequisite for the first results.

We will share:

• Architecture or design decisions
• Production experience
• Failure modes and debugging
• Operational challenges
• Trade-offs and alternatives
• Before-and-after experience

What is your experience with this problem?

• Production system
• Hard-earned engineering lesson

What approaches failed, disappointed, or created unexpected problems?

• Raw query embeddings. Embedding the user’s short query and comparing it with long descriptions gave weak results. The two texts are too different in length and form.
• Canonical terms without aliases. Descriptions rarely use the canonical name of a category. Without LLM-generated aliases, hybrid recall dropped relevant results.
• Aliases without embeddings. Pure BM25 with aliases rewarded word repetition. A staffing agency and a consultancy ranked in the top 15 for a software query, only because their text repeated a key term. The embedding leg pushed them down.
• One big LLM call before search. We first generated themes, sub-themes, signals, exclusions, and the query rewrite in one call. Output size dominated latency, and the rewrite was often stale by the time the user refined their focus.

Our users describe the companies they want in their own words. Our only data is free-text business descriptions that use different words, come in five mixed templates, and leave out most of what investors search for. Each step that made the search understand the user better added latency. We dissected the data, compared search approaches in Elasticsearch for quality and latency, and found which steps paid off. We now show a fast first result and improve it in stages.

What will you do differently today?

Today, we would first study the data and build a benchmark to measure search quality. Instead of embedding the user’s raw search, we would use an LLM to rewrite it like a company description, improving search quality. We would show results quickly, without waiting for the LLM, and do most of the LLM work when indexing companies, so search doesn’t have to wait.

What trade-offs did you consider?

• Why hybrid search (BM25 + kNN with RRF) over pure vector search? Pure vector search lost the exact terms and acronyms that users depend on. Pure BM25 rewarded keyword repetition over real fit. Hybrid search with Reciprocal Rank Fusion lets aliases drive recall and the embedding drive ranking.

• Why rewrite the query with an LLM instead of embedding the raw query? A short query and a long description are different kinds of text. We ask an LLM to write a pseudo company description from the request, so the query and the indexed text have the same shape. The trade-off: one extra LLM call, and a stylistic bias toward the template the rewrite imitates.

• What did we optimise for? Time to the first result, then relevance. At first, we showed nothing until the LLM had parsed the user’s input into themes, sub-themes, and a query rewrite. The user waited for every LLM step before seeing a single company. Now we search in two passes. The first pass searches with the query exactly as the user entered it, with no LLM step, so results appear almost at once. While the user looks at them, the LLM parses the input. Then the second pass runs the full hybrid search and replaces the first results with better-ranked ones.

What did we knowingly give up?

The search still runs on the original descriptions, which come from five different templates and contain boilerplate and some wrong facts. To clean or regenerate all descriptions with an LLM would cost too much time and money, so we improved the queries instead of the data.

What alternatives did we consider?

• A fixed industry taxonomy. The structured fields were too sparse and sometimes wrong. So a filter on these fields would hide most of the relevant companies, and some results would be wrong.
• Dropping aliases and trusting embeddings. Rejected: about 40% loss in recall.
• Index-time structured extraction. An LLM pass over every description to extract fields such as revenue model and product type. Rejected for now: first we must decide how to divide a description into dimensions, and a pass over many descriptions adds a cost we chose not to take on yet.

How can this help other practitioners?

• A design approach: progressive retrieval — start fast, then refine.
• A mistake to avoid: tuning search before you understand the shape of your data.
• A new way of thinking about the problem: The usual approach puts all query understanding (LLM parsing, aliases, query rewrite) before the search, so the search must wait for all of it. The new approach starts the search immediately with the raw query, and each step of query understanding improves the results when it finishes.

Current state

Production experience. Hybrid search, LLM query rewrite, aliases, and the two-pass search (raw query first, hybrid second) run in production.

Tags: #search #llm #elasticsearch #rag #latency #casestudy #failurestory #databases

Speakers

Pankaj Sharma (pankajs@sahaj.ai)
Sneha Bagri (snehab@sahaj.ai)