The Fifth Elephant 2024 Annual Conference (12th &13th July)

Introduction

Atlassian’s Jira Service Management (JSM) has consistently strived to empower customers by delivering top-notch assistance to those in need. A primary objective has been to promote self-service within JSM, allowing users to promptly access help while reducing the workload on agents.

The introduction of Generative AI led to the development of JSM AI Answers, enhancing JSM’s search functionality to provide conversational and precise responses. This talk will delve into the transition from conventional search systems to an advanced semantic search system, bolstered by large language models.

Target Audience

The talk is intended for a wide audience, encompassing AI researchers, developers, engineers, as well as product and business leaders, who are keenly interested in large-scale Gen AI applications.

Outline

• Anatomy of JSM Help
  • Involvement of Agents and Help Seekers
  • Support tickets
  • Request Types
  • Knowledge Base
• JSM Search Evolution
  • Keyword-based search
    • Limitations in understanding user queries and intent
    • Lack of personalized results based on user profiles
  • AI-based search
    • RAG architecture (Retrieval Augmented Generation) and its adaptation for this use-case
  • Help Resources - 1P (First Party powered via Confluence) and 3P (Google Drive, Sharepoint, Dropbox, etc.)
• Defining success metrics
  • Number of Distinct Knowledge Base Articles
  • CSAT
  • Assistance and Resolution Rate
• Measuring Quality
  • Evaluation framework - LLM evaluator vs non-LLM evaluation
  • Answer Quality
  • Content Gap Analysis
• Improving Answer Quality - Achieving 94% Accuracy
  • Enhancing query understanding and utilizing multiple-query variations to improve search relevancy
  • Leveraging bi-encoders and cross-encoders for document ranking and re-ranking
  • Personalizing content based on user location
  • Addressing vague queries
  • Enhancing search result consistency by eliminating user-based personalization
  • Identifying and reducing hallucinations using Chain-of-Thought and System-2-Attention
  • Recommendations for request types
• Current performance
  • Customer usage and feedback
  • Latency improvements - Reducing from 23s to 9s
• Next Steps
  • Seamless transition between Virtual Agent and AI Answers
  • Gap Analysis - Assisting customers in identifying knowledge base system gaps
  • AI-powered Knowledge Base Creation - Aiding agents in writing/updating KB articles on content gaps

Key Takeaways

• Application of RAG and LLMs on a large scale and the architectural variances from academic standards
• Challenges and insights in making iterative improvements and efficient evaluation
• Difficulties in objectively measuring the system’s “helpfulness” to users
• Extending and enhancing the feature by incorporating human feedback for continuous improvement.