The Fifth Elephant - Pune Edition

The ostrich in you war rooms: Business Intelligence (BI) bottlenecks

The BoF session brought together data engineers, product managers, and analytics practitioners to confront what the facilitator called “the uncomfortable truth about analytics” that despite massive investments in compute and infrastructure, analytics remains painfully slow. The room explored whether BI tools are the bottleneck, or if the problem lies elsewhere in organizational workflows, tool lock-in, cost structures, and the gap between what dashboards deliver and what decision-makers actually need.
A candid, practitioner-driven conversation unfolded across several themes: the persistence of legacy dashboards, the promise and pain of LLM-augmented analytics, the cost explosion that comes with opening up ad hoc querying, and whether pushing compute to the edge (local machines) is a viable alternative to centralized cloud analytics.
A recurring theme emerged: legacy dashboards aren’t going anywhere but they’re no longer sufficient. The real bottleneck isn’t a single tool; it’s the gap between fixed dashboards and the dynamic, iterative questioning that modern decision-making demands.

Notable insights and discussion highlights

Dashboards are necessary but insufficient: Everyone agreed that predefined dashboards serve a purpose for known metrics, but they can’t anticipate every question. The real demand is for iterative, conversational analytics.
LLM copilots help, but the semantic layer is critical: Teams deploying copilots found that without a well-maintained semantic layer, hallucinations and wrong SQL generation become serious problems, sometimes even threatening to bring down production clusters.
Cost unpredictability is a first-order concern: One participant shared that their projected token costs went from X to 3x within three weeks, and 10x within a month. The shift from fixed dashboards to open-ended querying fundamentally changes cost profiles.
Caching and pattern reuse are essential: Just like call centres where 200 questions cover most requests, analytics queries follow patterns. Caching known queries and converting conversations into reusable “recipes” reduces cost and latency.
Tool lock-in drives AI adoption: When people switch companies (Qlik to Power BI, for example), they lose productivity. AI/LLM interfaces that accept natural language reduce this friction significantly.

Key topics discussed

1. The “war room” problem: analytics is still too slow

The session opened with a relatable scenario: you’re in a war room meeting and someone says “these numbers don’t look right” or “get me this data by tomorrow.” Despite years of investment in data platforms, data engineers are still fielding ad hoc requests because dashboards built over 2–3 months don’t answer the questions that matter in the moment.

Key insight: The bottleneck isn’t compute, it’s the mismatch between static dashboards and the dynamic, evolving questions that business stakeholders need answered. Product managers know what to ask; they just don’t have the means to get answers without going through the data team.

2. LLM copilots and the semantic layer

One participant described deploying a copilot that allows anyone to ask questions in natural language, with SQL generated and executed against their data warehouse. The critical enabler was a semantic layer, definitions of metrics, entities, and business logic, that provides context to the LLM and reduces hallucinations.

What worked:

• Building a semantic layer on top of existing data definitions to augment LLM context
• Copilot-style interfaces where users ask questions and get SQL-backed answers
• Using the semantic layer to scope what the LLM can access and how it interprets queries

What broke:

• Poorly maintained semantic layers leading to wrong SQL that can overwhelm or crash production clusters
• LLMs hallucinating results — generating numbers that look plausible but aren’t backed by actual data
• Cloud-based BI tools that upload raw data to LLMs (e.g., Julius) with file size limits (5–50 MB), making them impractical for real-world datasets

Tools and alternatives discussed: Participants mentioned Cube for semantic layers, Snowflake’s native capabilities (Cortex), knowledge graphs for cross-domain semantics, and the general pattern of needing a translation layer between natural language and SQL/DSL.

3. Cost explosion — the Pandora’s box of open-ended querying

A significant portion of the discussion centred on cost. With fixed dashboards, you know your compute profile, there’s a bounded set of queries. But once you give users the ability to ask anything via an LLM interface, usage becomes unpredictable.

Real-world examples shared:

• One organisation projected token costs at X, saw 3x in three weeks and 10x within a month
• A participant working with their COO couldn’t reliably forecast costs because token usage depends on what users type and what the model generates
• The pattern: pilot for three months, re-evaluate, adjust budget, essentially treating it as an experiment

Mitigation strategies discussed:

• Caching: Identify the top queries (like the 200 questions in a call centre) and pre-compute answers
• Recipes: Convert successful conversational analytics sessions into reusable, cacheable workflows, “when the conversation is done, you convert it to a recipe”
• Caps and guardrails: Set per-user or per-session limits, though this creates friction in B2B contexts where customers expect unlimited access
• Intent understanding: Don’t pass raw user queries to the LLM. Instead, understand the intent first, then construct an optimised prompt — reducing tokens and improving accuracy

4. Access control and data security

As more people in an organisation get access to LLM-powered analytics, access control becomes critical. Participants raised the concern that role-based data serving must extend to the LLM layer, users should only see data they’re authorised to access.

Key concern: Traditional BI tools enforce access through their own permissions model. When you add an LLM layer on top, you need to ensure the semantic layer and query engine respect the same access controls. Without this, you risk leaking sensitive data through cleverly worded natural language queries.

5. Local compute vs. cloud — the data locality debate

A spirited discussion emerged around whether analytics should move closer to the user’s machine rather than staying centralised in the cloud.

Arguments for local/edge compute:

• Sensitive data (sensor data, customer data) that can’t leave a customer’s premises
• Eliminating round-trip latency to cloud for interactive analysis
• Cost reduction, local compute doesn’t incur cloud charges per query
• One participant noted they had 3–4 edge deployments running computer vision and analytics models on-premises

Arguments against:

• Compliance and security concerns, every data download must be conscious and controlled
• Opening “your entire world” to someone’s laptop isn’t viable for most enterprises
• Even traditional BI tools download response data; the question is about scope, not principle

Consensus: It depends on the use case. For sensitive, on-premises data with dedicated hardware, local compute makes sense. For most cloud-native organisations, centralised compute with proper access controls remains the default, but the option of pushing pre-processing and cleaning to local machines before sending results to the cloud was seen as a pragmatic middle ground.

6. SLMs (Small Language Models) for analytics

The group briefly explored whether smaller, specialised language models could serve analytics use cases, especially for edge deployments where large LLMs can’t run.

Current state: One participant mentioned Number Station (later acquired), which built an SLM on LLaMA for SQL generation, but it didn’t achieve mainstream adoption. For edge deployments, participants are still using traditional computer vision and quantised deep learning models rather than SLMs. The consensus was that SLMs for analytics are not yet production-ready but worth watching.

7. Prompt engineering and model drift

Teams deploying LLM-based analytics face an ongoing maintenance challenge: system prompts need updating, foundational model releases change behaviour (e.g., Gemini 2.5 behaves very differently from Gemini 3), and non-deterministic outputs make traditional golden dataset testing unreliable.

Approaches discussed:

• Building evaluation frameworks first, then running prompt optimisers, rather than manually tuning system prompts
• Keeping system prompts persistent and stable, with optimisers handling model-specific adjustments
• For B2B: each tenant’s schema and business context differs, making per-tenant prompt management complex
• Limiting confirmation questions to one at most, users reject solutions that ask too many clarifying questions before providing answers

Conclusions from the discussion

• Legacy dashboards are here to stay but they need augmentation. The future is a hybrid: static dashboards for known metrics, plus conversational/LLM-powered interfaces for ad hoc analysis.
• The semantic layer is the linchpin. Without well-maintained metric definitions and business logic, LLM copilots hallucinate, generate dangerous SQL, and erode trust.
• Cost is the new frontier. Moving from fixed dashboards to open-ended querying fundamentally changes economics. Caching, recipes, intent-based routing, and usage caps are all necessary.
• Access control must extend to the LLM layer. Role-based permissions that work for dashboards must also gate what the LLM can query and return.
• Local compute has a role especially for sensitive data and edge use cases — but isn’t a universal replacement for cloud analytics.
• Engineers build, but shouldn’t always. The instinct to build custom solutions is strong (“we are all engineers, we will build ourselves”), but exploring existing tools and platforms can save significant effort.

Key takeaways

• Start with the semantic layer, even a minimal one, before deploying LLM copilots. Without it, hallucinations will destroy user trust.
• Treat cost as a first-class design constraint. Cache aggressively, convert conversations to reusable recipes, and set usage guardrails.
• Understand user intent before hitting the LLM. Don’t pass raw queries; interpret first, then construct optimised prompts.
• Invest in access control that works across both traditional dashboards and LLM-powered interfaces.
• Watch the SLM space for analytics, specialised models for SQL generation and data querying are emerging but not yet mainstream.
• Accept that dashboards and AI analytics will coexist. Design for both, not one replacing the other.