Open weights, real stakes

In this session, we’ll enhance the safety of Sarvam, a sovereign Indian LLM, running on RHOAI / OpenDataHub with NeMo Guardrails through the TrustyAI operator. The idea is simple: Sarvam already knows. When you give it sensitive data, it reasons that the information is private. But knowing is not the same as enforcing. The model behaves differently on different runs, keeps no audit trail, and still lets the data through. Guardrails turn that awareness into a guarantee that works the same way every time. We’ll show rule based checks using Presidio for PII detection and regex for India specific identifiers, applied to both input and output and across English and Indian languages. Sensitive identifiers are blocked outright, while ordinary personal data is masked so the request still works. You’ll see Sarvam served with vLLM, NeMo Guardrails configured against it, and how it protects a live chat conversation end to end.

Takeaways:

• How to use guardrails on an LLM: serve it with vLLM, wire up NeMo Guardrails via the TrustyAI operator, and route every prompt and response through the rails.
• How to block sensitive identifiers and mask ordinary PII, on input and output, in English and Indian languages.
  The model knows what’s sensitive, but knowing isn’t enforcing. Guardrails make that awareness a consistent, auditable guarantee.

Bio:
I am a Senior Software Maintenance Engineer at Red Hat
My work centers on making large language models (LLMs) more efficient, scalable, and production-ready.

Co-speaker: Ritesh Shah rshah@redhat.com
{Add the link to draft slides - PDF/PPT - with comments access}