The Fifth Elephant 2024 Annual Conference (12th &13th July)

In this session, we discuss Atlassian data architecture to help demystify the complexities around building a real-world scalable Delta Lakehouse meeting data governance and compliance requirements and how we enabled various teams to iterate fast for their data-driven initiatives.

Today companies are increasingly looking to make their data ever so accessible to fuel their analytics and AI ambitions. However, constructing a robust, enterprise-grade data pipeline presents a myriad of challenges that organizations must learn to navigate to harness its full potential. It may seem daunting initially as it requires potentially high upfront investments, often making it challenging to evaluate the necessary steps and ROI.

Intended Audience for the BoF

• Data Engineers, Scientists, and Architects: Professionals responsible for building scalable, compliant data infrastructures. If you want to learn more about how to help your team build enterprise-grade data-driven products the talk is for you.
• IT Decision Makers (CTOs, CIOs, Business Leaders): Leaders and executives seeking insights into strategic investments in data technology to maximize ROI leveraging data architecture.

Outline of the BoF

Core Components of an Enterprise Data Pipeline

1. Data Modeling: Structuring data in a way that it is usable and efficiently accessible. Creating and exposing internal vs customer-facing data models.

2. Overcoming Key Challenges:

   • Methodologies to assess initial investments in technology and personnel against the expected returns, emphasizing long-term value over short-term costs.
   • Ensuring data governance for compliance such as DARE, BYOK, and working with UGC/PII data without stifling innovation.
   • Problems of Data Access for AI and Analytics:
     • Data Silos
     • Data Quality Issues
     • Complexity in Integration
     • Access Permissions

3. Rapid Innovation and Scaling

   • Powering customer data insights using Atlassian Analytics for customers across different Atlassian products.
   • Data pipelines for model training required for AI/ML use cases.

Key Takeaway

Unlock Innovation Ensuring Compliance with High-Quality Advanced Data Management: Real-world experience with tips and tricks from Atlassian in-house end-to-end data lifecycle management. High-quality, accessible data is the cornerstone of successful AI and analytics projects. The talk will help one understand various aspects to focus on while building data-products ground up.