Continuous Delivery Foundation cdCon 2020, 20 Oct 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Lightning Talk - Speeding up your CI/CD Approvals and Risk assessments with Machine... Gopal Dommety

Description

Lightning Talk - Speeding up your CI/CD Approvals and Risk assessments with Machine Learning: An Overview - Gopal Dommety, OpsMx

As Enterprises are implementing Continuous Delivery, it has been observed that Approval Gates and Risk Assessments are mostly manual processes. In this talk, we will discuss the time it takes for risk assessment and approvals, and how the use of data & machine learning can greatly reduce these times.
We will introduce OpsMx Autopilot- an intelligent continuous verification platform that works with your CI/CD Platforms, such as Jenkins, Spinnaker, ArgoCD, and Kubernetes. Autopilot will help CD implementers to gain control of their production deployments and help answer the above questions.
about: Gopal Dommety is the CEO of OpsMx. OpsMx was founded with the vision of “delivering applications with no human intervention". OpsMx Enterprise for Spinnaker, a multi-cloud, and cloud-native Continuous Delivery Platform, is trusted by many Fortune-100 enterprises.Previously he was the CEO of N42 and held senior roles at Cisco.Gopal studied Computer Science at IIT (Kharagpur), Finance & Public Policy at Stanford University, and obtained Ph.D in Large Scale Distributed Systems from Ohio State University. He was awarded 61 US Patents and designed widely deployed and popular Internet Protocols.

A

Good afternoon, thank you for joining me at cdcon. Today, I'm going to share with you my perspective on how to speed up approvals and risk assessment in your ci cd process.

A

I'll start this talk with describing customer problems that we see and present a solution called autopilot and end by sharing customer stories. Let me introduce myself. I am gopal demeti ceo of opsamex.

A

We started op smx three years ago with a simple vision: to deliver software without human intervention and to make every engineer's life happy.

A

I think uh in this conference we all share this common vision. Today, opsimax is a dedicated team of 95 people. We offer two products. uh The first is ops mx enterprise for spinnaker. This is an enterprise radius, spinnaker distribution.

A

We also support open source spinnaker. Second product is autopilot a continuous verification platform. We are grateful to have some of the largest global enterprises use our products at.

A

Scale now I was talking to a cio of a large enterprise, and this is what he told me he told me: we implemented a modern, ci and cd and still our process of application delivery is slow and costly.

A

uh He leads tens of thousands of engineers with thousands of applications and about 400 000 deployments every year. So why is their delivery so slow.

A

If you look at the delivery pipeline, they have.

A

Dev qa product prod approvals, the approvals take anywhere from 3 to 12 hours per release and before the final approval in every stage, risk assessment takes two to three hours for them, for example, in deployment risk assessment is about determining if the canary, blue, green or rolling update is going fine test assessment is about looking at the test results and determining if this can go to the next stage.

A

If approvals or risk assessment is not done right, the cost of production failures is about 35 000 dollars per minute. For this customer and their cost of compliance violations is very high too.

A

Let's look at why the risk assessment take time takes so much time. um Risk assessment is hard for three reasons.

A

First, risk assessment is very state specific. That means the risk assessment that you do for deployment stage is different than the test stage, and also risk assessment is very app specific.

A

So, during deployment you are looking at logs metrics events from either your canary blue, green or deployment strategies to assess whether the deployment is good to go to large scale production in production, you're trying to figure out if there's a problem, because of which you need to roll back in the test stage, you're trying to figure out if one of my thousand tests failed, can I push forward or not.

A

The second reason is that these processes are mainly mostly manual. A lot of contextual domain. Knowledge is needed, for example, for this specific application, a specific log error is critical, whereas another another type of log error is not and that somebody needs to know.

A

The third reason is that the data that is required to make these assessments is not all in one place. For example, if you're looking at test stage, you have the test darnest data, you have the logs from the application itself, that is they're all in two different places and they not all need to get correlated.

A

um Now. If you look at why approvals are time consuming once the risk assessment is done, we need to roll up the risk assessment data um to do the approval, as shown below like from test data or cannery data, etc. Then we also need to add the devsecops issues from sas and dashed uh policy checks, app version, dependencies, etc. All of these data is typically not in one place. It's in silos and the approver typically lacks the visibility overall visibility to um make quick approval decisions.

A

um The second reason is: they lack the fine grain policy controls to be able to scale to thousands of applications, thousands tens of thousands of users or hundreds of thousands of deployments, and the third reason is that approval process needs to be flexible and typically is very complex, with multiple stakeholders.

A

The solution is what we call autopilot and it is a machine learning system that sys sits on all of the data, including the data that comes out of the individual stages. It aggregates all this information um and then applies machine learning to this data for the various assessment risk assessment in the various stages, once the risk assessments in various stages happens, which then it applies.

A

Fine grain policy checks across these stages and then either presents the overall visibility in one place, for the individuals to make decisions, approvals, etc or even automate. The decisions when that decision is very clear.

A

um Autopilot currently works with several ci cd platforms, including spinnaker jenkins, tecton, argo, cd, etc.

A

Let me go through how autopilot is used in three customer case studies.

A

In this deployment assessment case study, the customer had three problems. It was taking them about 3.6 hours to look into logs and metrics of deployment across multiple services whenever they were doing deployment.

A

There are too many on-call engineering resources involved in this process.

A

When, when releasing fast, they had error-prone releases auto in this case, autopilot is being used in staging environment, to observe every dependent service during deployment and analyze, logs metrics um and events.

A

um They use jenkins for the ci cd and data dog and elastic search for their data sources, with autopilot they're able to reduce um the verification from three point: six man hours to five minutes um and also reduce deployment failures by 70 percent.

A

Next, I'm going to cover a production assessment case study in this case study bad deployments uh take place um when a bad deployment takes place. It takes about an hour for the dependent service to get impacted.

A

This is a very large data center deployment and in this case the rollback was being triggered after the sls were sufficiently degraded because they were using production monitoring tools for their sla and degradation. Identification.

A

Now autopilot is deployed and is used to monitor services after deployment continuously, not just services, but also all the dependent services, and, in this case autopilot is able to trigger roll back uh 30 minutes sooner, uh thereby saving about 1 million dollars per incident.

A

Let me describe how a large insurance company is using autopilot to speed up approvals um in for this customer. The challenge was that approval was taking multiple days and time sometimes weeks, and the reason is that it's a complex environment with multiple groups, lots of application, dependencies and different tools, um so autopilot today enables them to approve in hours as it provides overall visibility and also automates through fine-grained policy.

A

um Now let me um summarize uh in most companies uh approvals and risk assessment is taking much longer than it should and more expensive than it needs to be. This is really slowing down. Releases autopilot can enhance your continuous delivery by speeding up approvals and risk assessments, and you can find more information about autopilot at optionx.com.

A

I'm happy to take any questions that you have.