Introduction

Atlassian’s Jira Service Management (JSM) is dedicated to empowering users by delivering exceptional support. A primary objective for JSM is to empower users through self-service capabilities, enabling them to swiftly address issues with minimal human intervention (a.k.a Resolution rate).

This talk explores how Atlassian’s JSM is leveraging agentic systems to enhance the Resolution rate through JSM’s Virtual Service Agent (VSA). We’ll discuss three key components: a conversational platform that provides accurate responses as the first layer of support, the integration of human oversight (Human-in-the-Loop), and how we are leveraging Agents to drive business impact.

Target Audience

The talk is intended for a wide audience, encompassing AI researchers, developers, engineers, as well as product and business leaders, who are keenly interested in large-scale Gen AI and agentic applications.

Outline

Anatomy of JSM Help

• Roles: Agents and Help Seekers
• Common IT/HR Support Requests
• Knowledge Base and Intents

What is Resolution Rate?

• Resolution Rate
• Knowledge Base Article and Intent Volume
• Content Gaps and Answer Quality

VSA Conversational System

• RAG (Retrieval-Augmented Generation) Overview and Architecture
• Achievements: Increasing resolution rate from 12% to 35%

How Did We Increase Resolution Rate by 2X and Drive Core Business Impact

• Identifying Gaps
• Improving Context Gathering Capabilities in VSA
  • Vague Query Detection
  • Clarifying Questions
  • Smart Routing
• Identifying and Reducing Hallucinations in Answers
• User-Based Personalisation
• Handling User Behaviour Changes

Evaluation

• Human-in-the-Loop for Quality Assurance
• Gap Analysis: Identifying unassisted topics due to knowledge gaps
• Assisting JSM Admins:
  • Creating Knowledge Base (KB) Articles and Workflows
  • Faster Adoption and Improved RAG Performance

Current Performance

• Resolution Rate
• CSAT
• Scale and Latency

Challenges and Next Steps

• Adapting to User Behavior
• UX Experience: Making conversational flow seamless, asking the right questions
• Understanding Usage Patterns and Building Targeted Features
• Balancing Search vs. Q&A
• Building Autonomous Agents

Key Takeaways

• Delivering business impact with Gen AI-powered platforms has challenges that can be tackled with good innovation, adapting user interfaces to agentic systems, and having humans in the loop to build agents that learn over time.
• Ways to gather more context for the agents to ensure high-quality assistance.
• Understanding user behaviour and building targeted applications requires research and data-driven analysis.