The Fifth Elephant 2025 Annual Conference CfP

Introduction

Jira Service Management (JSM) is Atlassian’s platform provided to customers for service management, designed to help teams deliver exceptional service experiences. A core priority for JSM is maximizing the efficiency and effectiveness of support agents, as their productivity directly impacts customer satisfaction and operational costs. To address this, we developed the AI Suggestion Panel, an innovative agentic system built to significantly enhance agent productivity. This presentation introduces the AI Suggestion Panel, exploring how this intelligent co-pilot provides agents with actionable, context-aware suggestions by leveraging historical ticket data and advanced AI techniques, including large language models (LLMs), sophisticated semantic search, and continuous learning, ultimately aiming to dramatically reduce Mean Time To Resolution (MTTR) for support tickets.

Target Audience

This talk is designed for AI practitioners, machine learning engineers, data scientists, product managers, and customer support leaders interested in the practical application of Gen AI and retrieval systems to optimize enterprise support operations and improve agent efficiency.

Outline

The Challenge: Optimizing Support Agent Workflow

• Navigating information overload and repetitive tasks
• The imperative to reduce MTTR and improve customer satisfaction
• Identifying bottlenecks in the ticket resolution lifecycle

Introducing the AI Suggestion Panel: Your Agent’s Intelligent Co-pilot

Core objective: Proactive assistance for faster, more accurate resolutions.

Overview of key functionalities:

• Automated Ticket Summarization: Instantly grasping ticket context
• Predictive Next Best Actions: Guiding agents on effective resolution paths
• Intelligent Escalation Prediction: Proactively identifying tickets needing escalation based on SLA, Time to Resolve, sentiment, etc.
• Contextual Reporter Details & Asset Information: Providing a holistic view of the user and their environment
• AI-Driven Field Suggestions: Recommending optimal assignee, priority, and other relevant fields

Powering Suggestions: The AI Engine Architecture

Leveraging Past Wisdom: Utilizing a vast repository of resolved tickets.

Advanced Ticket Retrieval:

• Hybrid Search: Combining keyword and semantic search for comprehensive matching
• Query Rewriting: Enhancing ticket title understanding for improved search relevancy
• High-Recall Retrieval: Employing bi-encoders to fetch relevant issues from an OpenSearch cluster
• Precision Re-ranking: Utilizing cross-encoders for fine-grained similarity scoring

Intelligent Suggestion Generation:

• Sending current ticket context and top similar resolved tickets to an LLM
• Generating suggestions for next actions, draft replies, and field values
• Escalation & Contextual Logic: Rule-based and model-driven insights for escalation and reporter details

From Concept to Impact: Performance, Accuracy, and Adoption

Latency Optimization Journey: Reducing P99 latency from 21 seconds to 8 seconds for real-time assistance.

Accuracy Enhancements: Achieving and iterating on 70% accuracy for suggested actions.

Measuring Success:

• Current Scale: Over 125,000 daily invocations
• User Adoption: 10,000 Monthly Active Users (MAU) benefiting from the feature
• User feedback and impact on agent productivity and MTTR

The Road Ahead: Evolving Intelligence

• Integrating Knowledge Bases: Transitioning towards a RAG (Retrieval Augmented Generation) architecture for richer “next steps” and “draft reply” features
• Continuous improvement of accuracy and suggestion quality by building project awareness knowlege graph

Key Takeaways

• Insights into designing and deploying a large-scale AI system that directly impacts enterprise support agent workflows
• Practical application of bi-encoders, cross-encoders, and LLMs for complex information retrieval and suggestion generation tasks
• Strategies for optimizing system latency and accuracy in real-world AI applications
• Understanding the iterative process of feature development, from initial concept to widespread adoption and future enhancements using RAG and knowlege graphs
• Extending and enhancing the feature by incorporating human feedback for continuous improvement.