The Fifth Elephant 2024 Annual Conference (12th &13th July)

Maximising the Potential of Data — Discussions around data science, machine learning & AI

Prateek Mandloi

@prateekmandloi

Mayank Sawhney

Enterprise-Ready Data Lifecycle: Powering AI & Analytics at scale

Submitted May 27, 2024

In this session, we discuss Atlassian data architecture to help demystify the complexities around building a real-world scalable Delta Lakehouse meeting data governance and compliance requirements and how we enabled various teams to iterate fast for their data-driven initiatives.

Today companies are increasingly looking to make their data ever so accessible to fuel their analytics and AI ambitions. However, constructing a robust, enterprise-grade data pipeline presents a myriad of challenges that organizations must learn to navigate to harness its full potential. It may seem daunting initially as it requires potentially high upfront investments, often making it challenging to evaluate the necessary steps and ROI.

Intended Audience for the BoF

  • Data Engineers, Scientists, and Architects: Professionals responsible for building scalable, compliant data infrastructures. If you want to learn more about how to help your team build enterprise-grade data-driven products the talk is for you.
  • IT Decision Makers (CTOs, CIOs, Business Leaders): Leaders and executives seeking insights into strategic investments in data technology to maximize ROI leveraging data architecture.

Outline of the BoF

Enterprise-Ready Data at Scale

Core Components of an Enterprise Data Pipeline

  1. Data Modeling: Structuring data in a way that it is usable and efficiently accessible. Creating and exposing internal vs customer-facing data models.

  2. Overcoming Key Challenges:

    • Methodologies to assess initial investments in technology and personnel against the expected returns, emphasizing long-term value over short-term costs.
    • Ensuring data governance for compliance such as DARE, BYOK, and working with UGC/PII data without stifling innovation.
    • Problems of Data Access for AI and Analytics:
      • Data Silos
      • Data Quality Issues
      • Complexity in Integration
      • Access Permissions
  3. Rapid Innovation and Scaling

    • Powering customer data insights using Atlassian Analytics for customers across different Atlassian products.
    • Data pipelines for model training required for AI/ML use cases.

Key Takeaway

Unlock Innovation Ensuring Compliance with High-Quality Advanced Data Management: Real-world experience with tips and tricks from Atlassian in-house end-to-end data lifecycle management. High-quality, accessible data is the cornerstone of successful AI and analytics projects. The talk will help one understand various aspects to focus on while building data-products ground up.

Comments

{{ gettext('Login to leave a comment') }}

{{ gettext('Post a comment…') }}
{{ gettext('New comment') }}
{{ formTitle }}

{{ errorMsg }}

{{ gettext('No comments posted yet') }}

Hosted by

Jump starting better data engineering and AI futures

Supported by

Gold Sponsor

Atlassian unleashes the potential of every team. Our agile & DevOps, IT service management and work management software helps teams organize, discuss, and compl

Silver Sponsor

Together, we can build for everyone.

Workshop sponsor

Datastax, the real-time AI Company.

Lanyard Sponsor

We reimagine the way the world moves for the better.

Sponsor

MonsterAPI is an easy and cost-effective GenAI computing platform designed for developers to quickly fine-tune, evaluate and deploy LLMs for businesses.

Community Partner

FOSS United is a non-profit foundation that aims at promoting and strengthening the Free and Open Source Software (FOSS) ecosystem in India. more

Beverage Partner

BONOMI is a ready to drink beverage brand based out of Bangalore. Our first segment into the beverage category is ready to drink cold brew coffee.