Rootconf 2026 Annual Conference

[WIP] Helios: From an Critical Mobile Banking App Outage to an AI-Driven Resiliency Platform

One-line summary

How a major mobile banking login outage forced us to rethink observability, disaster recovery, incident management, and service prioritisation, ultimately leading to the creation of Helios, an in-house AIOps platform that transformed the way we detect, respond to, and prevent production incidents.

What problem are you addressing? What real engineering challenge, question, or experience is this session about?

This session shares the journey that began with a severe production incident where our Mobile Banking login platform experienced nearly eight hours of downtime.
While restoring service was the immediate priority, the incident exposed deeper systemic challenges:

• Disaster Recovery failover was significantly more complex than anticipated due to multiple dependent systems, a combination of legacy and latest cloud solutions, compliance requirements, and operational constraints.
• Existing monitoring tools provided visibility but limited control over operational data, with a lack of connections between legacy and cloud applications and with our automation capabilities.
• Every service alert was effectively treated as a critical incident, creating alert fatigue and slowing decision-making.
• Engineering teams lacked a structured framework to define business-critical SLAs and prioritise response efforts.
• Extreme pressure on time and a long list of regulatory compliance actions.

These challenges drove us to build Helios, an internal AIOps platform focused on observability, intelligent incident response, service tiering, and operational resilience.
The session explores the technical and organisational decisions that helped us move from reactive monitoring to proactive and predictive platform-driven reliability engineering.

Who is the intended audience?

• Platform Engineering
• SRE
• DevOps
• Infrastructure Engineering
• Engineering Leaders
• Developers building large-scale distributed systems

Level

Intermediate to Advanced

List one or two practical takeaways

• Lessons learned from building an in-house AIOps platform that combines observability, incident intelligence, Infrastructure, release engineering and compliance requirements
• AIOps case study: Bringing a real, true meaning to DevOps. Enabling Operations into the hands of engineering
• A practical framework for defining service tiers and SLAs in large microservice ecosystems.

What will you share?

Incident that changed our observability strategy

• Mobile Banking login outage that lasted close to 8 hours.
• Key learnings from incident response, recovery efforts, and executive escalations.
• How the outage exposed hidden operational gaps.

Disaster Recovery reality check

• Why our first instinct was to switch to DR.
• What prevented successful DR activation.
• Hidden dependencies discovered during failover exercises.
• Balancing compliance requirements, cost effectiveness, and operational reliability.

Building open-source observability at scale

• Why we adopted an open-source observability strategy.
• Gaining greater control and ownership of operational data.
• Integrating metrics, logs, traces, incidents, and automation into a single platform.

Helios: The AIOps platform

How Helios evolved into a centralized platform for:

• Incident intelligence
• Observability
• Operational automation
• Reliability engineering
• Compliance validation
• Security posture management

Defining business-centric reliability

• Identifying Critical User Journeys (CUJs).
• Mapping microservices to business outcomes.
• Tiering services by business impact.
• Defining SLOs and SLAs based on customer experience rather than infrastructure availability.
• Reducing unnecessary P1 incidents by classifying services appropriately.

Compliance and Security Automation

• Automated port-connectivity compliance checks across environments.
• Continuous policy validation.
• Real-time vulnerability scanning using Prisma.
• Automatically generating remediation pull requests for identified vulnerabilities.

Platform Engineering and Infrastructure Standardization

• Importing manually created infrastructure resources into Terraform.
• Standardizing Terraform modules across teams.
• Improving governance and reducing configuration drift.
• Enabling repeatable and auditable infrastructure deployments.

Chaos Engineering

• Introducing controlled failure testing.
• Validating resilience assumptions.
• Lessons learned from chaos experiments.
• Using failures to strengthen recovery preparedness.

What is your experience with this problem?

• Production system
• Real incident and outage
• Internal engineering platform
• Reliability transformation initiative
• Compliance modernization effort
• Security automation project
• Chaos engineering adoption
• Hard-earned operational lessons

What approaches failed, disappointed, or created unexpected problems?

• Assuming DR readiness based on documentation alone.
• Treating every service as mission-critical.
• Alerting based on infrastructure symptoms instead of business impact.
• Manual compliance checks.
• Manual vulnerability remediation processes.
• Infrastructure resources existing outside Infrastructure-as-Code governance.
• Limited visibility into service dependencies during incidents.

What will you do differently today?

• Identify Critical User Journeys before defining SLOs.
• Continuously test Disaster Recovery readiness.
• Build observability platforms that enable action, not just visibility.
• Automate compliance validation wherever possible.
• Integrate security remediation directly into engineering workflows.
• Standardize infrastructure through Terraform from day one.
• Use chaos engineering proactively instead of waiting for production failures.

What trade-offs did you consider?

Why build Helios instead of buying a platform?

Commercial platforms provided monitoring capabilities, but we needed:

• Greater control over operational data
• Custom automation workflows
• Integration with internal incident processes
• Compliance-specific controls
• Long-term platform flexibility

What did you optimize for?

• Reliability
• Faster recovery
• Operational ownership
• Compliance
• Security automation
• Cost efficiency
• Engineering productivity

What did you knowingly give up?

• Faster implementation through off-the-shelf products
• Vendor-managed feature development

What alternatives did you consider?

• Commercial AIOps platforms
• Multiple point solutions
• Traditional monitoring-only approaches
• Managed observability products

How can this help other practitioners?

This talk provides:

• A blueprint for building an internal AIOps platform.
• A framework for service tiering and business-aware reliability.
• Lessons from real-world DR failures.
• Techniques for identifying Critical User Journeys.
• Practical approaches to defining SLOs and SLAs.
• Methods for compliance automation and security integration.
• Guidance on infrastructure standardization using Terraform.
• Ways to adopt chaos engineering incrementally.
• Mistakes to avoid when scaling reliability practices across microservices.

Speakers

4 key members behind Helios

Tags

#aiops #platformengineering #sre #observability #reliability #incidentmanagement #failurestory #disasterrecovery #chaosengineering #terraform #security #prisma #compliance #devops #kubernetes #microservices #slo #sla #cuj #opensource #automation #casestudy #banking #infrastructure