The Fifth Elephant 2026 Annual Conference

Description

If you run data workloads, you’ve probably hit the same wall we did. You want compute that stays in your own cloud account, runs fast, and stays cheap — and sooner or later, someone tells you to pick two. This is a hard problem we solved for our own platform, and this talk shares those learnings around the engineering it takes to get all three.

We’ll treat it as a trilemma between ease-of-use, performance, and cost, and use serverless Apache Spark on Kubernetes to illustrate. First, the economics: across different Spark engines, we compare self-managed deployment against using vendor-managed serverless deployment and show how running compute yourself comes out cheaper, and we’ll walk through exactly why. But, that cost efficiency and speed gained can come at the expense of complexities of self-managing workloads.

Then we’ll go deep on the infrastructure challenges we had to build and solve to make a genuinely easy serverless experience run inside your own cloud account — remote upgrades, hardened multi-tenancy, reliable metering, and the numerous specific problems on Kubernetes infrastructure that no vendor we looked at had solved. For each one, we’ll cover the design choices and the tradeoffs, so you can take the same patterns back to your own platform.

I’ll close on why this matters more now. As AI makes software integrations and even maintenance cheap, the value of labelling things “fully managed” is eroding fast. The lasting differentiator is the hard part it can’t fake: running the workload efficiently with less toil. That’s also where we’re headed next — agents that operate the data platform itself.

Takeaways

• What the “serverless tax” really costs you — not just the cloud discount the vendor captures, but the control you give up over where your data runs and how the workload is tuned — and how to weigh it for your own workloads.
• The economic case for keeping compute in your own account, built so it holds even if you assume every engine performs identically.
• The specific infrastructure problems that make in-VPC serverless hard, and how they can be solved.
• A way to weigh Kubernetes deployment models for Spark against the three axes that actually conflict: ease-of-use, performance, and cost.

Who should attend?

Platform and data engineers building or buying managed compute, and the people who sign off on those invoices.

The economics need no deep Kubernetes background to follow; the infrastructure section goes deep enough to reward those who run Kubernetes in production.

Bio

Nilesh Mahajan is a founding engineer and Head of Infrastructure at Onehouse.ai, where he builds the systems that run managed data platforms inside customers’ own cloud accounts across AWS, GCP, and Azure — the deep infrastructure problem at the center of this talk.

He owns infrastructure, security, and compliance for large-scale data workloads, including running Spark and other stateful distributed systems in untrusted, multi-tenant environments.

Before Onehouse, Nilesh spent years building infrastructure at Uber, and alongside engineers who built it at LinkedIn and eBay. The platforms they relied on inside those companies kept data and compute close, under their own control.

He now also leads Onehouse’s AI initiatives focused on making data platforms run efficiently with less operational overhead.

LinkedIn: https://www.linkedin.com/in/nkmahajan/
X: https://x.com/nilesh_mahajan

https://gamma.app/docs/Breaking-the-Trilemma-7y63de2pzr4vepx

{Add the link to 2-min elevator pitch video}