Cloud Native Computing Foundation ArgoCon 2022, 21 Sep 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Argo CD Observability with AIOps - Detect Incident Fast - Amit Kalamkar & Vigith Maurice, Intuit

Description

Argo CD Observability with AIOps - Detect Incident Fast - Amit Kalamkar & Vigith Maurice, Intuit

Intuit runs 2000+ services on Kubernetes and uses Argo CD to implement GitOps workflows and progressive delivery with Argo Rollouts to minimize risks associated with service changes. However, about one third of all incidents at Intuit are caused by rolling out those changes. We have a need to surface operational insights in real-time to make it easier for service operators to detect and resolve change-induced incidents. In this session, we will discuss how we extended Argo CD with in-cluster observability and how we integrated an open source, Kubernetes-native operational analytics platform (Numaproj) with Argo CD to provide AI-driven anomaly detection. We will demonstrate the ease of configuring in-cluster observability with AIOps in Argo CD, how these new capabilities help service operators detect incidents and take corrective actions faster, and how this effort is helping Intuit reduce incident MTTD/MTTR.

A

Good afternoon, everyone is everyone enjoying argocon so far,.

A

So today we wanted to talk about how we incorporated observability powered by our aios platform into Argo CD. My name is Amit kulmker I, lead, observability and analytics at Intuit and I have with me widget Morris. He is a principal engineer at Intuit, I am representing pneuma flow, he is representing Argo. We have all our bases covered.

A

So here's our agenda for today we will talk about the problem you are having, which is change, induced incident and how we solved it. Then we will have a live demo, Wireless guard permitting. Otherwise we have a backup, then widget will get into Numa praj and how it's powering these experiences, and then we will talk about what roadmap we have in further.

A

So let me let me start with into it. Most of you should be familiar with Intuit with our products like TurboTax QuickBooks. We are one of the largest fintech company out there. We are also one of the largest SAS company and we are pretty much committed to open source. We are not only using for internally, but we have a lot of Maintenance as well as contributors uh in a lot of projects. Our goal is to name name few.

A

We are also one of the top end user company for cncf and with these new capabilities which we are building, we want to make sure we continue this collaboration with other end user companies out there. Let me first ground everybody with the tech ecosystem which we have. We started our modernization journey in 2018., we pretty much modern modernize everything the front end back end platform infrastructure. Some of the things which we did as part of this is moved all our container payroll into kubernetes.

A

We also created paved roads so that develop it becomes easy for our developers to deploy. So the github's automation is based on Argo. As part of this, we deliberately ensure that all the stock, the platform infrastructures is instrumented, so we get real-time events into our operational data Lake, and then we build the iops platform which we'll talk about little bit later, to derive actionable insights from them.

A

Let me also touch base little bit on the scale at which we operate. We have more than 2500 Services, currently running in production on kubernetes. As far as our data lake is instant, we generate more than 2.5 billion real-time events per day and the aiops platform is already generating more than 16 million predictions per day, and it is increasing day by day, as we add more use cases.

A

One of the core principles of Intuit is innovation. We want to make sure we know it fast, and these Innovations goes to our customers, so it's only possible because of the investment which we have made in our github's automations, as well as Argo CD Argo rollout. As many of you know, Argo CD came out of into it and now it's used by hundreds of company across the world.

A

So now let me talk about the problem which we are talking pratik mentioned in the today's game. Node, the operational excellence is one of the priorities at Intuit.

A

We want to ensure that all our core capabilities are always available and we want to also ensure that if we have incidents we, hopefully we don't that our MTD and mtdr is manageable.

A

So when we looked into the data, we figured out that one third of our incidence were caused because of the change and which we wanted to address that. So we did a little more deep dive into that and to figure out why? So there were two main reasons: one we found out that there was a disjoint experience between the deployment, which is the Argo cdux, which most people use and the observability experiences. We have built two for developer.

A

It was a little bit harder to figure out the quality of change and we wanted to address that because it was increasing our mttd.

A

So what we did was we brought our observability experiences into Argo CD, so it gives gave us two things: one: the new capability where the Matrix is already incorporated into Argo CD, so the developers can look at Argo CD itself and look at any metrics. They want whether it's a golden signal Matrix any custom Matrix at the place where they are deploying to for the quality of change.

A

We gave them one signal using the new map Rod, which widget will talk about where they can look at the one signal and say that is the change good or bad, and if it is bad, you can revert it back so now. Let me hand it over to vijeth to do the demo. Thank.

B

You Amit so before the demo. Let me talk about the demo plan. um What what we are going to do is: let's. Let me pretend that I am a service developer, who wants to make a change and the change I have has a bug inside and we will see how the new changes to Argo CD, that is the metrics available, and also the AI powered anomalies score how we can use it to quantify the quality of the change. Okay, so demo time before the demo, let me introduce to a small demo app.

B

What you see is that- and there are some yellow features moving across the screen. These fishes represent a request. This browser is generating uh every second, so you can imagine say around one to five requests per second and the angry redfish represents an error. It is coming from the back end. The change I'm gonna make is a change to the back end, not to the front end, and we will see the experience which the customer sees because of the change and how we can detect the problem.

B

Looking at the arbor City observability, so it starts by a git PR. We are gonna merge. This pull request in this is the change we have and Let me refresh the screen. Here, it's a little slow yeah. We got it. Let me sync in the changes, so there is nothing new in here. So far right, it's the same. Marvel CD, you are, you are seeing. Videos represent Health here as a heart, the green icon, but it doesn't represent the application that's running on the resource. Rather, it shows the health of the resource.

B

For this reason, what people do is ah traditionally they will switch out from here. They will go to the Matrix providers UA, for example, at indeed we use wavefront. It could be Prometheus data over what not that is the disjoint experience. Amit was talking about to solve that problem. What you have done is we have integrated a matrix tab here. This Matrix tab will show The Matrix.

B

As you see here, okay now, let me I will talk about in detail how we are getting the metrics, but for the demo just talk about what we are seeing right. These are the kpis which you think is very relevant for your application. The configuration you see here is the default. We have deployed it into it, the reason being at Intuit. Every application that is deployed has a concept of golden signals.

B

Golden signal is nothing but a family of metrics from the red meaning red error, request, errors and duration, slash utilize use, use, standing for utilization, saturation and errors, and we call it golden signal popularized by Google's term. Actually, so what we see here is latency on the x axis is one minute, so everything you see in this Dash, this Matrix view is one minute and we show the score for that. For example, latency the way you interpret this September 20th 1340 I have a latency for the deployment.

B

59 FF has a latency around 0.2 milliseconds, okay, the latency here is in seconds on the left hand, side what we see the key thing here, I told is average latency. It does not mean that we can only show average, it could be P99 or p75. The only thing you need to change is the configuration for getting the metrics from the backend from your Matrix provider. It is very configurable, but the whole page is configurable. In fact, you can configure it like this see.

B

If you see here right, we see that this is the golden signal uh View and in here we have multiple metrics, that's coming in okay, so you change your expression. You get the metrics and when the UI is rendered now now that the deployment is going through, let's see what the user experience is, the purple fishes represents, let's say, more, enlightened fishes, but that's the in. We see a lot of Errors way more Angry fishes in play, meaning there is something going wrong in the deployment you could see this using the metrics.

B

That's the point of Matrix right. So, if you were to see here it takes around, we scrape the Matrix around 30 seconds. So let me see whether I can just switch between and see whether yeah you can see that hey I, see a latency going up. I also see my 200s going: okay, a spike in 500, the total number of Records. This is the total number of Records. That's coming per minute is around 990.

B

That remains a constant and also we show more metrics, like utilization metrics right here we are color coding, each port and each deployment ID to that Port. We see nothing much different from there, but we- we have put a pie chart here, just to show the power of the charting Library available in Argo City right.

B

So you see that okay, now you you clearly see there's a problem right now. The question is: is it enough right? Is it enough to make the decision to roll back if I am an uh an experienced developer in this project? I could make that decision by looking at the key metrics, which I know about right, but that's not enough right. We, the concept, is everybody should be able to make that decision very easily and to make that you should look at the system holistically rather than an individual metric.

B

It boils down the fact that how can we provide a single metric that can represent the quality of your deployment? That is where the AI Ops platform comes into the play. So if you see here right, okay, it's clearly so there's a problem. It depends on how we interpret this data, so at 1349 right after the deployment deployment is shown by this flash icon. You see that that normally has gone up right. It's showing around 9.97.

B

What to interpret the score. You need to understand how anomal is generated, meaning there is a three segment score. The segments are ah you you? Could you get meaning it depend on your use case, but at into it we have zero, the three meaning the application is behaving within the normal operating pattern between three to seven meaning the most of the metrics or the patterns are deviating from the normal operating pattern. Above 7 means that it's totally deviated right, meaning the deployment is bad at this point, one could come back and roll back meaning.

B

This is the single score that can tell you that whether your deployment is good, bad or ugly. This score is generated by analyzing many different metrics. That's the whole point is a multivariant system. We earlier we talked about holistically analyzing the system right, so it's a multivariant score, which is also configurable. For example, let's say if I were to do an anomaly on the number of fishes versus the golden signal along with latency and error.

B

You could do it, so all you need to do is change the way the configuration is put in place, so it can learn about the metrics and then give you the anomaly score based on the system.

B

This model, in fact we use a auto encoder model. This has been at Intuit for more than three to four years. We have been using the system and it's very powerful. It create incidence at Intuit. So it's that powerful and that's the reason we thought hey. We have such a good great system that can detect problems. Why not integrate it to Argo CD and make it available for the community so that everybody can make use of this cool feature we we are using at scale now that I am done with the demo.

B

Let's go back to the architecture, how how how things play behind the scene.

B

So what happens here is when the user loads, the UI right right. It talks to our city, server, good old way. Argo CD server through rver CD extension framework knows that observability has turned on is on and then it talks to something called argosity metric server. This is something we developed for this project. Okay, so the rocity metric server has both the configurations of all the metrics View and so forth, and also it. It also acts as a proxy to the metrics provider. We use Prometheus, but it could be others like wavefront data dog.

B

What not right this makes the data available to the UI right. Ui gets the data through this request, request flow in the data flow. This is the decoupled architecture, meaning to make any metrics available in our CD. All you need to do is make sure that the metrics available in your metrics provider. This is how a of system is able to give you the normal score totally decoupled, meaning it can write the score back to the provider Prometheus and you see it in the UI.

B

Now let me get little more into the anomaly system, because that's very interesting system right, um so let me I'll go ahead in small small batches here, because there's a lot here: how are we getting the metrics right? This is a real-time system. This anomaly score is not computed when you open the page. It's always there right it starts by prometh is having a feature, meaning it can stream in through a HTTP, remote writer protocol to our anomaly system, so every metric parameters get it will forward strains in is the right word.

B

Streams to our pipeline, which does feature engineering feature engineering is something like you can imagine some scalar standard, scalars and logarithm transformations to make sure that it is compatible with our inference system, and it also does some grouping Group by for the anomaly detection system to understand the multivariant aspect of it once that is done, it forwards to the ml inference system where an score is assigned to it.

B

Now this core is send it to a post processing system that normalizes the score between 0 to 10, so that the score is intuitive. Otherwise it will be all over. The place we want to make sure that the user can assign and make a decision based on the score. It should be Universal, and this is a happy path, meaning uh the feature engineering happens.

B

The ml inference assumes that the model is in memory so that it can do the post processing, but this is an aop system, meaning it has to understand based on how the things are changing in the backside. For example, there will be new application coming in. There is a new application, an application, existing application, exposing a new endpoint. It means that we are seeing the system for this metric for the first time right, so we do not have a model in place and there should be an online system that should be able to compute those.

B

So this is what happens in case. It does not have a model, it triggers the online triggering. This happens through Argo workflows. We trigger the model, the provider is again Prometheus, it reads the data trace. The model puts it into a model storage we use ml from here, and this is which the ml inference system will to get the model in place, so meaning once it is running, is always running. You can change your application.

B

It detects the changes it will still assign a normally score and within our system is good enough, such that, if you have 180 points it can give you anomaly score right and it won't scale if you were to change your code or a configuration every time. You add an end point because then during RCS will say that hey only if I added that configuration that problem is not there right, it's always. We always detect the changes and do an online training.

B

This architecture has been in place um again at into it for a while right. We use this to create incidents. That means this should be a very proven architecture. We can wake up half off into it, because our anomaly score is bad. It does not I'm not talking about alerts here. It literally create incidence. Right means it should work like talk like it's like clockwork. You read it's very beautiful system.

B

If you were to look at this at a microscopic level, each component has some subtlesies right. That becomes barriers to build a strain processing system and I will talk about this barriers in a very high level. There are many, but I'm I'm just coping to few. So what are the barriers for building an AOC system?

B

First is: if you were to look into every component, what we do is we. Everybody should do no streaming right. Most of our email. Engineers are experts in streaming too at one point, because they all need to know how to read from Kafka right to Kafka. It sounds easy, but if you were to use Kafka Kafka is a dumb broker. Smart client architecture that meaning the client should be very, very competent right, and there are a lot more with the streaming in having streaming reliable streaming in each component, but that that is a problem.

B

So each water, each component should be lightweight rather right. That's a barrier. Second is auto code. Here, hot dog actually means non-standard by not standard what we mean is we need to look at the streaming platform as an end-to-end system, not as a component at a component level. Think about it that in the nth vertex nth component, you have a backlog, The Source should know about it. Otherwise you will end up in data loss. The Source will still push it through and the last component is backlogging.

B

It will never catch up and what will happen is if you were to use any streaming system, they will plunger the whole data, and you will end up in data loss and mostly what happens? Is people try to deploy this connected system as a micro service, the worst right? It only ignores about a thing at a time, but you need to look at holistically the entire pipeline to make it work like a torque.

B

Also is very difficult to extend the platform, because if you were to use streaming, if I want to say fork out a data for a b testing, you need to create a new Kafka topic or a streaming topic. Then you need to have your own consumer group. It gets quite complex right, so what in fact happens? Is people try to put more code onto the same component?

B

Thinking that okay, it's easier to do it for the time being, because we are just plugging and playing or experimenting, but soon the experiment will get into production and there we will see issues right. So it's very important that in aops we get to experiment and play with reliably and um operational excellent way. It does not mean there are no data engineering Frameworks that can do streaming. There are excellent systems, for example, we ourselves use a passive link uh based on Apache beam, abstraction, then samsa. Is there a spark?

B

Many many players are there, but they are built for data engineering. They are not built for AI, Ops or simple data processing. Their aim is like 2 billion TPS right and think about it. If I were to show this demo with a spelling cluster near application cluster, who will be eating more cost right, your fling cluster will eat a lot of money because it's very complex and the barrier to get into streaming is very high.

B

The knowledge of how to really do streaming can get very, very complex, but it comes with the cost too in case, um but if you were to do data engineering and centralized system, what you do is you need filling your things, but if you were to move the problem at the source level like today, what you are seeing is mttd. As soon as a problem happens, we need to have a lightweight system sitting along with your source and able to detect the problems and, of course the solution is not cheap either.

B

So how does this ideal air of system look like right? I think we all can agree to that. It should be very easy to use. One would say it should be running in less than minutes and possibly even 11 in less than five minutes. It should be truly language. Agnostic I should be able to write each component in any language I like without knowing about how the streaming itself is done behind the scene. It should be lightweight, meaning I should be able to move it all the way to its system right.

B

It should be native to kubernetes, because kubernetes has a lot of light edge. Schedulers cost efficiency is very, very important because, as the dam moves through, you will have input coming in different um throughputs, and if there is no input, your system should be able to detect that and scale down to zero meaning you are not incurring any cost at that point.

B

Separation of concerns is Paramount here. When you build a complex systems like streaming systems, you need to make sure that each component worries about what exactly they have to do. They don't have to worry about how streaming Works, how do I make sure that my data is forwarded only once, because exactly one semantics is also tricky. It is like the parent of horse semantics right. It's like Grandad of all, but here what we mean is we can support lower odd semantics like at most once or at least once the whole point is.

B

The platform should be able to support this out of the box and then candidacing streaming this. What it means is it doesn't matter whether we read from Kafka or HTTP streaming, endpoint or TCP endpoint. The rest of the ml internal system should work, and that is the true story where today I shot Prometheus with https and remote writer at Intuit at scale. What we use Kafka as the input, because we have a centralized way of doing uh AI Ops, but when you do it per cluster, you do HTTP but rest of the anomaly detection system.

B

Nothing changes if I were to move. The sync, for example, writing. Instead of writing the parameters I can write a wavefront for all I care. All we need to change is the sync. The anomaly detection system with the ml Engineers Rod, still works, as is right. There's a small Point called Watermark is very, very important it.

B

It helps you do completeness: okay, I'm, just moving on just by saying that, because it's a very tricky but very interesting topic, anything that is built on an Ideal system should be operationally excellent, meaning it's fire and forget, think about it. Your input is an Infinite Source. It never ends right and it's not like something that comes up instant. Then you have something you can look. This is like fire and forget system. You move the source, it auto scales. It can do retrace board migration, node migration, it shouldn't matter.

B

In fact, we look at the system as a queuing theory at each vertex right, and that makes it beautiful, I would say, and it should be operationally excellent if you write something on this ideal air of system. With this, let me introduce pneuma flow. This is the platform we built based on our learnings for from four to five years. Pneuma flow is one could say summarize something like is a kubernetes native, dag based DAC meeting director, a cyclic graph based a distributed stream.

B

Crossing platform developed in mind with the mindset that is stream processing should be made easy for every application developer and every ml engineer. Okay,.

B

To go in just one slide, but let me try to do some justice to it first step. Is you read from the source? Okay, you have an infinite bound problem. You are going to read from the source: infinite unbounded stream is a subset of boundary stream. Okay, so it implicitly means we can support boundary. But anyways you read from the source. You did see like see, assign event time and Watermark. This is to track the progress of time.

B

Time is very important in an unbounded system, once you read from the source, you have the next vertex called UDF. Every message goes to a user defined function, which is nothing but a container. The input to the UDF is a key and a value. Okay key is a string. Value is a bite error. It could be anything the output of that is also a k and a value, but it could be either 0 or nothing, meaning you think about.

B

Like a filtering system, you are dropping the data because you are not happy with the data or it could be one one output that is a good old map and, lastly, we could have more than one that is called a flat map that is a superset of all those zero one or many okay, so the UDF can have input is key value. Output is zero one or more values key value pairs.

B

Once the output has been computed, you can make a decision which path the UDF. Should the message should take a good analogy, for this is if you are doing experimentation, you want the same data to go to two path. You want a Model A to run on it and also model B to run on it. There are some cases where the UDF could be a decision. Udf. What it happens is I, am analyzing the data and found out okay. This is the data that belongs to a locality system.

B

The data is passed, not dense, I'm, gonna use, something like Som model versus. If it was dense, I would have used a auto encoder model right, so you can make decisions on the message and forward it the right place that is called conditional forwarding.

B

Once we have conditionally forwarded the message, you could have n number of these steps. Okay for the Simplicity I just put ah to there. You could purchase to sync the the last two things of things, meaning the sync is different from a UDF meaning. You need to do some batch writing and things like that. You should be efficient and be able to scale right, so the sinks are taken separately and pneuma flow considers a pipeline between a source. We call it terminals a source and a string, whatever happens in between the source.

B

The moment we raid it to the sink. The moment we write to it to a external source is follows all the exactly one semantics and all the goodies which we talked about in the previous slide.

B

um Please come to the booth. We can talk a lot more about pneuma flow and how Argo CD is using it, because this is just one slide, trying to summarize a very interesting topic and a very cool feature in so please feel free to stop by with this I will hand it out. Amit everybody like.

A

The demo yeah, so we did went into a lot of details. I want you guys to remember one thing: Miss pneuma flow takes care of everything. What we talked about. You should be up and running in five to ten minutes. If you are not, please let me know we have a problem.

A

So what's coming ahead right, so let me tell you that we want to make sure that this Argo CD observability. We want to roll out to all our clusters within into it. Second, the next thing is, we want to integrate with Argo rollout so that we remove the human from the equation. So I'll go rollouts, look at this anomaly, one signal and automatically rolls backs, and we feel that pneuma flow has a lot of interesting use cases which we are already working on in in cluster analytics.

A

That includes automatic scaling that will directly affect cost as well as automatic, alerting so means. Please go ahead and look at our git reposed Nima proj Argo proj. If you like, it start it, and always we are hiring. So if anybody is interested, if you like to work on this awesome stuff, please reach out to one of us. Thank you.

B

Yes, you can scan the code, you will get the GitHub robo.

A

And there are a lot of examples you can go to new uh the GitHub and try it out.

B

Sorry I couldn't.

C

Hey that the charts at the beginning, where you showed you know, response time, for instance, for the old deployment and the new deployment and also the configuration where you get the data sources. It looked. It looked a little bit like hayanta.

C

um Also the configuration like how is it related is that uh Kayenta, a part that works with pneuma flow and it's interconnected, or how about these? Do these pieces fit together?.

B

Yeah, so the metrics that we get in the dashboard, the configuration we showed was for the um the Matrix view configuration but down the lane. We also have configuration for what metrics is should be injected to the anomaly detection system. What are the multivariant features you would like to have, and that are the metric name? You just need to provide so um I would say it's in one place you could from. We are trying to keep it along with the deployment of your application, where you could say: okay, these are the metrics.

B

I am interested in the view, because these are my application kpis, and these are the metrics that should be inputted to the multivariant anomaly system and.

A

I think to your second part of your question, everything is real time. Yeah.

B

Or latency yeah, so the it is, ah the end-to-end latency of the pneuma flow per size in milliseconds. The only latency that could be added is how complex the UDF is in between in case. If you are using a very complex model which is running on a very cheap Hardware, it could take like, for example, like around a 0.5 seconds, but everything is real time. So, if you were to use a GPU based system, then that should not be a problem, but 0.5 seconds is the end to end for the anomaly detection system asterisk.

B

Is that how often do you scrape the metric from the application right? If you were to scrape it every 15 seconds? That is actually the delaying factor? It's not the end to end inference system.

C

Okay, thank you.

A

Okay, thank you.