The Fifth Elephant 2024 Annual Conference (12th &13th July)

Maximising the Potential of Data — Discussions around data science, machine learning & AI

Balasugavaneswaran K

Shashank Rao

@shashankpr Author

Unified Help in Jira Service Management using AI

Submitted May 31, 2024

Introduction

Atlassian’s Jira Service Management (JSM) has consistently strived to empower customers by delivering top-notch assistance to those in need. A primary objective has been to promote self-service within JSM, allowing users to promptly access help while reducing the workload on agents.

The introduction of Generative AI led to the development of JSM AI Answers, enhancing JSM’s search functionality to provide conversational and precise responses. This talk will delve into the transition from conventional search systems to an advanced semantic search system, bolstered by large language models.

Target Audience

The talk is intended for a wide audience, encompassing AI researchers, developers, engineers, as well as product and business leaders, who are keenly interested in large-scale Gen AI applications.

Outline

  • Anatomy of JSM Help
    • Involvement of Agents and Help Seekers
    • Support tickets
    • Request Types
    • Knowledge Base
  • JSM Search Evolution
    • Keyword-based search
      • Limitations in understanding user queries and intent
      • Lack of personalized results based on user profiles
    • AI-based search
      • RAG architecture (Retrieval Augmented Generation) and its adaptation for this use-case
    • Help Resources - 1P (First Party powered via Confluence) and 3P (Google Drive, Sharepoint, Dropbox, etc.)
  • Defining success metrics
    • Number of Distinct Knowledge Base Articles
    • CSAT
    • Assistance and Resolution Rate
  • Measuring Quality
    • Evaluation framework - LLM evaluator vs non-LLM evaluation
    • Answer Quality
    • Content Gap Analysis
  • Improving Answer Quality - Achieving 94% Accuracy
    • Enhancing query understanding and utilizing multiple-query variations to improve search relevancy
    • Leveraging bi-encoders and cross-encoders for document ranking and re-ranking
    • Personalizing content based on user location
    • Addressing vague queries
    • Enhancing search result consistency by eliminating user-based personalization
    • Identifying and reducing hallucinations using Chain-of-Thought and System-2-Attention
    • Recommendations for request types
  • Current performance
    • Customer usage and feedback
    • Latency improvements - Reducing from 23s to 9s
  • Next Steps
    • Seamless transition between Virtual Agent and AI Answers
    • Gap Analysis - Assisting customers in identifying knowledge base system gaps
    • AI-powered Knowledge Base Creation - Aiding agents in writing/updating KB articles on content gaps

Key Takeaways

  • Application of RAG and LLMs on a large scale and the architectural variances from academic standards
  • Challenges and insights in making iterative improvements and efficient evaluation
  • Difficulties in objectively measuring the system’s “helpfulness” to users
  • Extending and enhancing the feature by incorporating human feedback for continuous improvement.

Comments

{{ gettext('Login to leave a comment') }}

{{ gettext('Post a comment…') }}
{{ gettext('New comment') }}
{{ formTitle }}

{{ errorMsg }}

{{ gettext('No comments posted yet') }}

Hosted by

Jump starting better data engineering and AI futures

Supported by

Gold Sponsor

Atlassian unleashes the potential of every team. Our agile & DevOps, IT service management and work management software helps teams organize, discuss, and compl

Silver Sponsor

Together, we can build for everyone.

Workshop sponsor

Datastax, the real-time AI Company.

Lanyard Sponsor

We reimagine the way the world moves for the better.

Sponsor

MonsterAPI is an easy and cost-effective GenAI computing platform designed for developers to quickly fine-tune, evaluate and deploy LLMs for businesses.

Community Partner

FOSS United is a non-profit foundation that aims at promoting and strengthening the Free and Open Source Software (FOSS) ecosystem in India. more

Beverage Partner

BONOMI is a ready to drink beverage brand based out of Bangalore. Our first segment into the beverage category is ready to drink cold brew coffee.