BEGIN:VCALENDAR VERSION:2.0 PRODID:-//HasGeek//NONSGML Funnel//EN DESCRIPTION:Practical case studies from enterprises and startups X-WR-CALDESC:Practical case studies from enterprises and startups NAME:Optimizing costs of cloud infrastructures X-WR-CALNAME:Optimizing costs of cloud infrastructures REFRESH-INTERVAL;VALUE=DURATION:PT12H SUMMARY:Optimizing costs of cloud infrastructures TIMEZONE-ID:Asia/Kolkata X-PUBLISHED-TTL:PT12H X-WR-TIMEZONE:Asia/Kolkata BEGIN:VEVENT SUMMARY:Introduction to the conference\; motivation DTSTART:20211126T063000Z DTEND:20211126T064000Z DTSTAMP:20260421T135608Z UID:session/MEQiCBpjv2j9Ztoq27rYdP@hasgeek.com SEQUENCE:0 CREATED:20210922T035357Z LAST-MODIFIED:20210922T035400Z LOCATION:Online ORGANIZER;CN=Rootconf:MAILTO:no-reply@hasgeek.com BEGIN:VALARM ACTION:display DESCRIPTION:Introduction to the conference\; motivation in 5 minutes TRIGGER:-PT5M END:VALARM END:VEVENT BEGIN:VEVENT SUMMARY:Cloud cost mental models DTSTART:20211126T064000Z DTEND:20211126T072000Z DTSTAMP:20260421T135608Z UID:session/YXMt8R5oAfUAPDWp1rdkgd@hasgeek.com SEQUENCE:1 CREATED:20210922T035456Z DESCRIPTION:1. Dealing with cloud cost (AWS) at Capillary Technologies\n2. Transitioning from Cost Optimization to Cost Sustenance \n3. Cloud Cost Attribution and breakdown\n4. Cost Attributions (By PnLs\, Asset\, Produc t\, Team)\n- Attributing cost to business metrics\n5. Cost Governance\n- C entralized vs De-centralized Governance (Shift-left)\n6. Cost Baseline\, A nomalies and Leakage\n7. Forecasting and Planning cost optimization opport unities LAST-MODIFIED:20230108T103046Z LOCATION:Online ORGANIZER;CN=Rootconf:MAILTO:no-reply@hasgeek.com URL:https://hasgeek.com/rootconf/optimizing-costs-of-cloud-infrastructure/ schedule/cloud-cost-mental-models-YXMt8R5oAfUAPDWp1rdkgd BEGIN:VALARM ACTION:display DESCRIPTION:Cloud cost mental models in 5 minutes TRIGGER:-PT5M END:VALARM END:VEVENT BEGIN:VEVENT SUMMARY:Discussion about cloud cost mental models DTSTART:20211126T072000Z DTEND:20211126T075000Z DTSTAMP:20260421T135608Z UID:session/ETiPvYMHc71nAQ999Nxh2r@hasgeek.com SEQUENCE:1 CREATED:20210922T035906Z DESCRIPTION:This interactive session will focus on cloud cost governance a nd management at intuit scale. LAST-MODIFIED:20230108T103046Z LOCATION:Online ORGANIZER;CN=Rootconf:MAILTO:no-reply@hasgeek.com BEGIN:VALARM ACTION:display DESCRIPTION:Discussion about cloud cost mental models in 5 minutes TRIGGER:-PT5M END:VALARM END:VEVENT BEGIN:VEVENT SUMMARY:Break DTSTART:20211126T075000Z DTEND:20211126T080000Z DTSTAMP:20260421T135608Z UID:session/JN6MDafEhR9Z3fxExZAhCA@hasgeek.com SEQUENCE:0 CREATED:20210922T035641Z LAST-MODIFIED:20210922T035646Z LOCATION:Online ORGANIZER;CN=Rootconf:MAILTO:no-reply@hasgeek.com BEGIN:VALARM ACTION:display DESCRIPTION:Break in 5 minutes TRIGGER:-PT5M END:VALARM END:VEVENT BEGIN:VEVENT SUMMARY:Tips and tricks for visibility\, optimization and staying optimize d by design DTSTART:20211126T080000Z DTEND:20211126T084500Z DTSTAMP:20260421T135608Z UID:session/UBqLqFfcsQkWhWByB8Z7SE@hasgeek.com SEQUENCE:1 CREATED:20210922T035656Z DESCRIPTION:1. Different Instruments of Cost Visibility in AWS\n- AWS Cost explorer\n- Using AWS Budgets as baselines and alerts\n\n2. Machine Reser vations (Discount vs Optionality vs Liability)\n - A decision graph of res ervation\, on-demand\, spot\n - Planning purchase of Savings Plan\n- Bulk or Systematic buying of reservations\n\n3. Tying Cost with Workloads\n- R eal-time workload \n- Batch Workloads\n- Auto-scaling of Real-time Systems \n- Cloud-ready application\n- High Availability vs Cost LAST-MODIFIED:20230108T103046Z LOCATION:Online ORGANIZER;CN=Rootconf:MAILTO:no-reply@hasgeek.com URL:https://hasgeek.com/rootconf/optimizing-costs-of-cloud-infrastructure/ schedule/tips-and-tricks-for-visibility-optimization-and-staying-optimized -by-design-UBqLqFfcsQkWhWByB8Z7SE BEGIN:VALARM ACTION:display DESCRIPTION:Tips and tricks for visibility\, optimization and staying opti mized by design in 5 minutes TRIGGER:-PT5M END:VALARM END:VEVENT BEGIN:VEVENT SUMMARY:More tips and tricks on cloud cost DTSTART:20211126T084500Z DTEND:20211126T091000Z DTSTAMP:20260421T135608Z UID:session/HpXd4m7wvUJB48MHwxdkDm@hasgeek.com SEQUENCE:1 CREATED:20210922T035602Z DESCRIPTION:This interactive sessions will focus on further tips and trick s on cost optimizations. LAST-MODIFIED:20230108T103046Z LOCATION:Online ORGANIZER;CN=Rootconf:MAILTO:no-reply@hasgeek.com BEGIN:VALARM ACTION:display DESCRIPTION:More tips and tricks on cloud cost in 5 minutes TRIGGER:-PT5M END:VALARM END:VEVENT BEGIN:VEVENT SUMMARY:Topics for discussion for future sessions DTSTART:20211126T091000Z DTEND:20211126T092500Z DTSTAMP:20260421T135608Z UID:session/85fZHPNxX8uXUbWPNkWMd2@hasgeek.com SEQUENCE:0 CREATED:20210922T040020Z LAST-MODIFIED:20211112T103759Z LOCATION:Online ORGANIZER;CN=Rootconf:MAILTO:no-reply@hasgeek.com BEGIN:VALARM ACTION:display DESCRIPTION:Topics for discussion for future sessions in 5 minutes TRIGGER:-PT5M END:VALARM END:VEVENT BEGIN:VEVENT SUMMARY:Conclusion - summary and key takeaways DTSTART:20211126T092500Z DTEND:20211126T093000Z DTSTAMP:20260421T135608Z UID:session/4kNnNPPmVbDDU7kmXwmNb7@hasgeek.com SEQUENCE:0 CREATED:20210922T040033Z LAST-MODIFIED:20210922T040033Z LOCATION:Online ORGANIZER;CN=Rootconf:MAILTO:no-reply@hasgeek.com BEGIN:VALARM ACTION:display DESCRIPTION:Conclusion - summary and key takeaways in 5 minutes TRIGGER:-PT5M END:VALARM END:VEVENT BEGIN:VEVENT SUMMARY:Introduction to the Conference DTSTART:20220318T063000Z DTEND:20220318T064000Z DTSTAMP:20260421T135608Z UID:session/Gvp7zcs5DwXR1sGEadbFrY@hasgeek.com SEQUENCE:16 CREATED:20220221T120149Z DESCRIPTION:On data transfer costs and how to handle cloud costs for batch processing and big data workloads LAST-MODIFIED:20230108T103046Z LOCATION:Online ORGANIZER;CN=Rootconf:MAILTO:no-reply@hasgeek.com BEGIN:VALARM ACTION:display DESCRIPTION:Introduction to the Conference in 5 minutes TRIGGER:-PT5M END:VALARM END:VEVENT BEGIN:VEVENT SUMMARY:Every Penny Counts: Optimizing Google Cloud Platform Bills DTSTART:20220318T064000Z DTEND:20220318T071500Z DTSTAMP:20260421T135608Z UID:session/JxPf7Jn64LPuzvvqc87a15@hasgeek.com SEQUENCE:7 CREATED:20220223T091943Z DESCRIPTION:While it may seem obvious\, companies overlook their cloud cos ts and lengthy cloud invoices. Whether you're just starting out or looking to optimize and get more of a bang for your buck\, this talk will be of i nterest to you. We'll discuss the key points that affect your bill\, commo n pitfalls and open-source tooling to optimize your GCP bills like a power user. GCP offers a well-rounded set of services that can address your bus iness needs\, but remember cloud is billed pay-as-you-go. This talk outlin es practical steps you can take in the cloud today to ensure cloud cost su ccess tomorrow.\n\nhttps://app.pitch.com/app/public/player/02235184-5bdb-4 d55-82eb-1ae0eeca8f13 LAST-MODIFIED:20230108T103046Z LOCATION:Online ORGANIZER;CN=Rootconf:MAILTO:no-reply@hasgeek.com URL:https://hasgeek.com/rootconf/optimizing-costs-of-cloud-infrastructure/ schedule/every-penny-counts-optimizing-google-cloud-platform-bills-JxPf7Jn 64LPuzvvqc87a15 BEGIN:VALARM ACTION:display DESCRIPTION:Every Penny Counts: Optimizing Google Cloud Platform Bills in 5 minutes TRIGGER:-PT5M END:VALARM END:VEVENT BEGIN:VEVENT SUMMARY:Founding a FinOps/Cost Optimization Function DTSTART:20220318T071500Z DTEND:20220318T075000Z DTSTAMP:20260421T135608Z UID:session/WF52WN8Nyyg7wLSqxtQQwx@hasgeek.com SEQUENCE:9 CREATED:20220223T092006Z DESCRIPTION:High-level agenda of a talk on founding a FinOps/Cost Optimiza tion function within your organization.\n\nKnowing Your Organization:\nWha t is important to your company? \nGrowth/Innovation vs. Security vs. Opera tions vs. Efficiency\nHow this guides the focus and position of your FinOp s team for success\n\nKnowing When To Start:\nWhat is the right level of s pend?\n\nThe Three Ways FinOps adds Value:\nSaving with prepay\nAnswering the question: What's going on?\nRealizing technical savings opportunities\ n\n--Jason Rhoades\nFinOps Leader\, Intuit\n LAST-MODIFIED:20230108T103046Z LOCATION:Online ORGANIZER;CN=Rootconf:MAILTO:no-reply@hasgeek.com URL:https://hasgeek.com/rootconf/optimizing-costs-of-cloud-infrastructure/ schedule/founding-a-finops-cost-optimization-function-WF52WN8Nyyg7wLSqxtQQ wx BEGIN:VALARM ACTION:display DESCRIPTION:Founding a FinOps/Cost Optimization Function in 5 minutes TRIGGER:-PT5M END:VALARM END:VEVENT BEGIN:VEVENT SUMMARY:Break DTSTART:20220318T075000Z DTEND:20220318T075500Z DTSTAMP:20260421T135608Z UID:session/8exxcNh6sg8udDJGwD3wJh@hasgeek.com SEQUENCE:0 CREATED:20220315T094058Z LAST-MODIFIED:20220315T094058Z LOCATION:Online ORGANIZER;CN=Rootconf:MAILTO:no-reply@hasgeek.com BEGIN:VALARM ACTION:display DESCRIPTION:Break in 5 minutes TRIGGER:-PT5M END:VALARM END:VEVENT BEGIN:VEVENT SUMMARY:Optimizing Cost of Data Platform Workloads DTSTART:20220318T075500Z DTEND:20220318T084000Z DTSTAMP:20260421T135608Z UID:session/DcN3obKFj1jVnq7vNVDpYT@hasgeek.com SEQUENCE:6 CREATED:20220223T092030Z DESCRIPTION:Today\, data platforms workloads constitute a major portion of the cloud spend. With every company increasingly using data driven decisi ons\, this share of cost can wildly go out-of-hand if not governed and opt imized effectively. \n\nAt Capillary\, we have been building data driven p roducts since the last 12 years. Over the years\, our data platform has ev olved through many big data systems to a domain-centric\, multi-tenant dat a lake powered by Spark running on EMR and Databricks. The data lake is de ep embedded inside our Engagement platform\, Loyalty platform\, Insights a nd AI/ML products. \n\nThis talk will focus on how we do data platform cos t governance and manage the cost with growing adoption of more and more da ta related feature requirements. \n\n**Key takeaways**\n\nParticipants wil l learn\n\nCo-relating data platforms metrics and cloud cost metrics to de rive insights \nTuning Data Engineering pipelines to reduce wastage (Query Optimizations)\nFleet design for ETL pipeline with cost considerations (I nstance Selection\, On-demand/Spot management)\nArchitectural patterns whi le designing for interactive workload (Reports / Dashboards)\nCost Governa nce around Ad Hoc Analytics (Notebooks)\n\nPrimary focus of the talk will be on **Apache Spark** based systems. \n\nPrakhar Verma is the Principal A rchitect with Capillary Technologies. He has over 12 years of experience i n building data-driven products. \n\n\n LAST-MODIFIED:20230108T103046Z LOCATION:Online ORGANIZER;CN=Rootconf:MAILTO:no-reply@hasgeek.com URL:https://hasgeek.com/rootconf/optimizing-costs-of-cloud-infrastructure/ schedule/optimizing-cost-of-data-platform-workloads-DcN3obKFj1jVnq7vNVDpYT BEGIN:VALARM ACTION:display DESCRIPTION:Optimizing Cost of Data Platform Workloads in 5 minutes TRIGGER:-PT5M END:VALARM END:VEVENT BEGIN:VEVENT SUMMARY:Data transfer cost optimization. DTSTART:20220318T084000Z DTEND:20220318T091000Z DTSTAMP:20260421T135608Z UID:session/WqmjuoYpvpywSqEzfi1hA6@hasgeek.com SEQUENCE:8 CREATED:20220223T092050Z DESCRIPTION:**Need for Data Transfer Cost Visibility**\n\nData transfer is a vital part of the functioning of an organization\, when an organization is spread over multiple regions it becomes even more important. Data tran sfer cost contributes to a large chunk of overall cloud costs for most lar ge-scale AWS architectures. \n\nSolutions Architects and DevOps engineers\ , while optimizing cloud spending on AWS\, find it time-consuming and hard to reduce data transfer costs. This is due to the inadequate and distribu ted visibility of the source & destination of the resources or services th at are exchanging data in the form of traffic within or outside the cloud environment which results in constant surges in the overall cloud costs as we are unable to effectively pinpoint the root cause of the issues and ac tivities leading to high data transfer costs. This would also help Securit y Engineers to identify any resources using public IPs and perform public data transfer so that I can find data leaks and fix any security vulnerabi lities.\n\nControlling & streamlining data transfer costs need changes in the architectures. To do that\, we not only need to develop context around data transfer pricing but also need to have granular visibility into the traffic or data exchange between services or resources\, both of which - a re very time-consuming and hard to build & maintain.\n\nWe set out to buil d a system that provides DevOps engineers with pre-built and near real-tim e granular visibility into data transfer costs so that they do not have to spend any time and effort in building the same. \nThis should be able to additionally help engineers to reduce the data transfer costs by providing them insights & suggestive actions so that they don't have to spend time & effort in doing a continuous literature review of data transfer pricing which in most cases is not a very exciting work to do regularly.\n\nEngine ers start to solve this problem by using cloud-native cost-control tools w hich at the max\, provide information about the infrastructure API operati ons that are causing data transfer costs\, but no information about the tr affic between the resources or services that invoke those API operations.\ n**From CUR to VPC flow logs**\n\nThere are many columns in the AWS Cost a nd Usage Report. Which helps build an understanding of your data transfer cost and see which resources contribute to this cost. AWS CUR has an Opera tion line item\, which provides Generic costs like Interzone-In or VPCPeer ing-In can be filtered by Operation\, enabling you to find resources that might generate unnecessary Operation costs. But lacks the granular visibil ity which resources and/or Services are communicating with each other.\n\n To obtain this visibility we can use VPC flow logs which are records of tr affic flow within an environment\, between applications\, and services. Th ese are raw network logs providing details about the source and destinatio n of the request\, the ports that the resources are communicating on\, the total size and total packets\, the attached network interface\, and much more. But this data in itself does not provide much information.\n\nBelow is an example of a vpc flow logs\n\n![image](https://drive.google.com/uc?e xport=view&id=1N9Yi0u944SoFy9VQwbtSLxU7KFo7lAgG)\n\nVPC flow logs can be e nabled for a VPC\, a subnet\, or a network interface. This also includes r esources for which the network interface is created by AWS services. This acts as a central source of truth for analysing the data transfers within an environment. The volume of VPC flow logs generated is humongous\, as th is is the log of each and every network request that happens within an env ironment\n\nWe use a 3 step ETL process(Extract\, Transform\, and Load) to analyze Data Transfer Cost\n\n**Prerequisites**\n\nWe at Opslyft have an internal tool that polls AWS APIs and other services to collect resource\, service mapping to their network interfaces along with a list of IP mappi ngs for all AWS global services. And a separate API which powers up the pr icing data for data transfers.\n\n**Step 1. Extract**\n\nVPC flow logs are enabled on all VPC'\;s across all regions and child accounts. These fl ow logs are delivered to a central S3 bucket either in an hourly or daily partition. Along with extracting VPC Flow logs this step also involves our internal tool which polls AWS API and other services to collect informati on related to the IP address to resource mapping. This also allows us to l ook at the IPs mapping to a resource with the function of time (ie. At a g iven time which resource was using the requested IP address).\n\n**Step 2. Transform**\n\nA lambda is triggered on an hourly basis once the flow log s start being delivered to a Centralised S3 bucket. Here the lambda functi ons reads the flow logs and enrichs them with the resource metadata that i s collected by the mapping created by IP to Resources. This Step also take s into account the many conditions that are posed as a challenge while ana lysing flow logs. To mention a few \nLet'\;s assume that Instance **A** is communicating with instance **B** the flow logs are generated by the n etwork interfaces attached to these instances which means each network int erface logs 2 flow records one which is outbound another which is inbound\ , resulting in 4 flow log records per data transfer. This complexity incre ases when we introduce a NAT gateway between transactions resulting in 12 logs for one transaction.\n\n![image](https://drive.google.com/uc?export=v iew&id=1KG2XtrihpkaXxVjyZ3A8QklhKZJyPOlK)\n\n**Step 3. Load**\n\nThe enric hed flow logs are then stored in S3. Which is then further aggregated to b e used by the dashboard that contains data grouped on account\, service\, region\, az and traffic flow direction. This output is stored in an RDS da tabase\, where a separate lambda populates our aggregations with the trans fer cost we incurred using the AWS pricing API. This in turn powers our da shboards with different views which would help you understand and plan the next steps in your path to reduce data transfer costs.\n\n![](https://dri ve.google.com/uc?export=view&id=1sjEBQouppuXaUiT9riBV7LDZaZyuXI9q)\n\n**An alyzing the data**\n\nFrom the dashboards below\, we can get a holistic as well as granular view of the data transfer operations in our AWS VPC. Add itionally\, we have an aggregation browser\, if we want to manually browse \, filter and group the flow logs to get a better understanding of the dat a flow within your VPCs.\n\n![](https://drive.google.com/uc?export=view&id =1KnTxdy7XLP6IIHl7oJyumT6Le3Rq4Ipe)\n\nThe above aggregations provide us w ith an overview of the data flow and the costs that we have incurred. As y ou can see we get a clear visibility over where our transfer costs are com ing from\, this can be further drilled down to figure out the source and d estination regions or services of these transfers (as shown below).\n\n![] (https://drive.google.com/uc?export=view&id=1QypamT4kp5z2g3ZrvgFdDRHuWgacY VsS)\n\nThe drilled down version of this data will aid you in figuring out the necessary changes that we need to incorporate so we can reduce our da ta transfer costs\, for example here we can see a sizeable amount of data is being transferred from EC2 instances to us-east-1 region through a NAT gateway\, empowered with this information we can decide if we can shift th ese instances into a subnet with an internet gateway\, which would remove our transfer costs as outbound transfer through an internet gateway is not charged.\n\n![](https://drive.google.com/uc?export=view&id=10kDJqbPIXZSbL GyZ1PUbojsXKdUcpbPl)\n\nAdditionally the aggregation browser can help us i n aggregating the data further using more filters and time ranges from whi ch we can plan out actionable items to reduce our transfer costs.\n\n**Key Takeaways to help you reduce your data transfer costs**\n\n1. This has al lowed us to gain meaningful insights on which of the NAT Gateways can be r eplaced with Internet gateway to reduce both data processing costs and the NAT Gateway running cost. ![](https://drive.google.com/uc?export=view&id= 117_z4fO-z0xScb1NNVCW7f1fWn7EBYe8)\n2. Use Amazon CloudFront to serve stat ic content which reduces transfer costs by introducing caching at your edg e locations as well as providing you with higher availability. \n ![](http s://drive.google.com/uc?export=view&id=10a8K-xrsOhYj52vmL3rs1Ma2zbblglB6)\ n \n3. Keep Infrastructural components like RDS and Redshift in the same r egion to save on Inter-region transfer costs. Transfer costs vary between source and destination regions.\n![](https://drive.google.com/uc?export=vi ew&id=1w63zfyV7qxxfpU725KHVNMx4T36fRbmA)\n\n\n4. This allows us to monitor traffic between resources in different regions or multiple accounts and r ecommend using VPC Peering or VPC Sharing to further optimize the data tra nsfer cost.\n![](https://drive.google.com/uc?export=view&id=10a8K-xrsOhYj5 2vmL3rs1Ma2zbblglB6)\n\n5. Avoid using public IP addresses for internal da ta transfers within the same Availability Zone. Intra-Availability Zone da ta transfers are free\, provided you use private IP addresses\n\n\nHave a look at the slidedeck here: https://docs.google.com/presentation/d/1dphLNK K6qBjKjv3K2AJzhhkA2kIAaduxerrPi8tSwOI/edit?usp=sharing LAST-MODIFIED:20230108T103046Z LOCATION:Online ORGANIZER;CN=Rootconf:MAILTO:no-reply@hasgeek.com URL:https://hasgeek.com/rootconf/optimizing-costs-of-cloud-infrastructure/ schedule/data-transfer-cost-optimization-WqmjuoYpvpywSqEzfi1hA6 BEGIN:VALARM ACTION:display DESCRIPTION:Data transfer cost optimization. in 5 minutes TRIGGER:-PT5M END:VALARM END:VEVENT BEGIN:VEVENT SUMMARY:Conclusions and way forward\, Facets.cloud DTSTART:20220318T091000Z DTEND:20220318T092000Z DTSTAMP:20260421T135608Z UID:session/8x1JhmyGSA1p4swyjYBXLE@hasgeek.com SEQUENCE:3 CREATED:20220315T065250Z LAST-MODIFIED:20220316T103829Z LOCATION:Online ORGANIZER;CN=Rootconf:MAILTO:no-reply@hasgeek.com BEGIN:VALARM ACTION:display DESCRIPTION:Conclusions and way forward\, Facets.cloud in 5 minutes TRIGGER:-PT5M END:VALARM END:VEVENT END:VCALENDAR