Rootconf Mini 2024 (on 22nd & 23rd Nov)

Introduction

5C Network is a leading AI-powered platform in the healthcare space, specializing in radiology and medical imaging. We manage and process large volumes of medical data, including over 1 billion DICOM objects, providing critical diagnostic services across India. Our focus on leveraging cutting-edge cloud and AI technologies allows us to deliver high-quality, cost-effective healthcare solutions.

The Challenge

Cloud expenses quickly became a major operational cost and bottleneck in managing vast amounts of data, especially in healthcare, where stringent data retention policies require long-term availability. Initially, all data and processing loads were hosted on AWS, leading to significant costs due to egress charges and storage fees. The challenge was to maintain scalability, compliance, and performance while drastically reducing cloud expenses.

Our Solution

To address this, we architected and adopted a multi-cloud and hybrid cloud strategy, which resulted in an 87% reduction in cloud costs. The approach consisted of several key initiatives:

1. Multi-Cloud Strategy

• Data Migration to Cost-Effective Providers: We transferred large volumes of data from AWS to India-focused Data Centres that don’t charge for egress.

2. Cold Storage Implementation

• We implemented a data-driven cold storage process, minimizing the cost for archival data while maintaining compliance with healthcare data retention policies (7-year availability).

3. Technical Challenges in Migrating Large-Scale Data

• Migrating over 1 billion DICOM objects was a complex operation.
• Custom Python and JavaScript tools were developed to ensure that images were transferred accurately and without data loss.

4. Image Minification

• A typical CT Scan can be as large as 5GB.
• Scan images were automatically minified to 0.5% of their original size after a week in the display.
• The minified data could be retrieved as needed, striking a balance between storage efficiency and data accessibility.

5. Advanced Observability Practices

• We built a custom monitoring dashboard on top of OpenTelemetry to track the status of data transfers, storage latency, and bottlenecks.
• This increased visibility allowed us to optimize processes and further reduce costs through better resource allocation.

6. Final Optimization via Edge Computing

• Frequently accessed data was cached locally within the client’s system and browser, reducing the need for repeated data transfers from the cloud.
• This last-mile optimization significantly minimized latency and data movement costs.

Key Takeaways for the Audience

• Learn how to optimize costs in Complex Systems that also need to adhere to stringent compliance requirements
• Gain insights into how to improve performance at scale for Image-heavy applications
• Familiarize how to handle Large-Scale Data Migration
• Realize the Importance of custom-built Observability metrics
• Understand how to leverage Hybrid Cloud for Last-Mile Performance

This talk will offer practical insights into managing cloud expenses at scale, overcoming technical migration challenges, and deploying resilient, cost-effective multi-cloud systems in India for complex environments like healthcare.

About the Speakers

• Shubham Kumar - Senior Software Engineer, 5C Network