Cloud Native Computing Foundation EnvoyCon 2019 (San Diego), 21 Nov 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Graph-based ML Anomaly Detection and Insights for Envoy Systems - Anoop Koloth & Hanzhang Wang, eBay

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

Graph-based ML Anomaly Detection and Insights for Envoy Systems - Anoop Koloth & Hanzhang Wang, eBay

At eBay, the services have been moving in close to end-users for faster and better experiences. SLB powered by envoy today helps us to address dynamic caching of content on edge across the globe. Monitoring the reliability and availability to the customers is always a top ask. The data and insights generated by Envoy are in-depth and detailed for granular and powerful anomaly detection.

In this talk, the speakers will present on how they managed to build a monitoring system and leveraged data generated from envoy clusters:
(1) Processing billions of hits served from different platforms from worldwide in real-time.
(2) Key Performance Indicators from Envoy ecosystem.
(3) Effective ML solution for proactive monitoring diversified eBay systems.
(4) Graph-based modeling and algorithms to deal with system complexity.
(5) Symbiosis and enhancement with existing SRE solution.

A

Bring us here, I'm a noob Koloff I work at traffic engineering team at eBay, and this is my friend Han Zhang Wang, who is a senior researcher at eBay, so what we would be covering today is like eBay on why ecosystem? What data and traffic insights we generate from our anomaly? Data for anomaly, detection and Han would actually cover the graph based anomaly: detection, okay, so traffic engineering at eBay. The primary goal we have is to reduce latency for our end user and improve our availability to business.

A

So we serve about 12 billion requests per day across three different platform desktop mobile web and native and last year we explored the capabilities of on why to break the stream in encode and decode path and Stitch dynamic caching experience to our end user, thereby reducing our above-the-fold response time from 1.5 seconds to 700 millisecond.

A

Okay, moving into the details of our existing ecosystem, we have users accessing ebay, using leveraging, browsers or native app and all the connect. All the requests are currently terminated. Tls terminated at software load balancers, which is powered by Envoy, and it is spread across 13 different locations across Gio's and we have a dedicated backbone MTB to our data center and the response is given back to the end user. Why? This is happening?

A

We have log stash agents in the Envoy cluster ecosystems to act, to collect the collate the access logs and send the bay sent that same back to data center and from the data center. It's through the Kafka streaming. We tried to do dual rights to the elasticsearch cluster and currently we have been exploring highly with regard to click house, because we have hit issues with the volume elastic search cluster, but for the machine, learning and anomaly detection.

A

What we currently do is like we have dual rights, one to the machine learning cluster and one to the normal cluster and any insights and alerts which we receive from the machine clusters are notified to our SEC team, DeWine team and our core. You FES ecosystem and, interestingly, when we did. This all was good at getting all the request, level, information and everything.

A

But when we were trying to do the root cause analysis to exactly find out what happened to the user requests, we found out that there was a necessity to have all the data into a centralized place. That's where you could find other lines like latency related metrics, coming from site, speed native service data, and we also managed to send scripts through envoy to actually piggyback on client level. Insights like lines have any error or and other stuff to have more root, cause analysis capabilities, okay at a very high level.

A

This is how we started at eBay, but now the system is supported by contour and it has way at once, but this is how we started so in the Envoy container in the Kuban, it is world we do have file bit running and which practically pauses the access logs into our ecosystem, and we have status tea for all control, plane, related stats and exported over the Prometheus back took rough on our dashboards for DevOps to manage and yeah I'll move on to the next slide. Yeah.

A

Okay, as I pointed out, these are the different data sets which we currently capture. In our traffic engineering ecosystem, which is greater than 12 billion records per day, and we are spread across 120 dimensions and 60, different metrics, critical, metrics and based on this data, we currently today do anomaly: detection get traffic insights, we use the same for ramp up decisions, and we also try to figure out the bot attacks and possible remediations, and we are on the verge of exploring security and experimentation support on average.

A

Ok, looking at the raw record. So ideally, if you would embed envoy software load balancer and try to get the access log pretty much, you would get all this information like and what we do is like. We do. Some data and reach met on other sites to do geo, IP, mapping and figure out additional dimensions of interest, but I will try to shed a light on the response flag part so, which is very interesting for people like yeah. Oh no has this response flag attribute wherein, like you, have flags like DC, where.

B

Your downstream.

A

Connection got terminated like UT, which is more on upstream timeout UO for upstream overflow and a not for no routes configured. So, oh as we were getting more insights, we found these flags were more interested to. Actually, let us understand the system mode and figure out that their way upstream issues which we had with our database data center servers and we were having bought attacks which are from the downstream, so it actually helped us to detect anomalies on those flags to get conclusions of possible attacks and how we could actually mitigate the same.

A

So ok, given that we have observability today we have access to the information like we. This is just a sample data of a production traffic.

A

This is it's very low percentage, but we have access to information like we have knowledge about which region what is P, what latency is they are performing, and how is the traffic pattern with regard to path refer our business, insights and other stuff, and though we have this observability and we are in a position to well data over ISP peering with regard to any caste like this is for a London pop. We just wanted to validate like or if all the requests are served, close by sequester served by the London pop or not.

A

So even when we have this observability with extensive matrix, it was very manual and tedious effort to actually run the ecosystem. That's where we thought like this is not going to scale. Let us build anomaly detection on these metrics and how? How would we actually tackle this problem? So we went ahead and explored X pack, which is a last elasticsearch solutions, and we had this principle like we as I pointed out.

A

We have many dimensions and might so if you are trying to do jobs for each and every dimension and matrix, you are into this curse of dimensionality. You would find anomalies every time, but you will not exactly get into an actionable anomaly or insights, so we went ahead and we did a leveraged. Our knowledge engineering skill set where we categorizes the dimensions to be primary and secondary dimensions and secondary dimensions are pretty much the influence for influencer for the primary dimensions.

A

So what we do is we, like we've tried to find anomaly detection on primary dimensions, and only then we try to find out if there are anomalies patterns on our influencer. That way, we try to stitch, at least for the metric part. What would what could be the possible reasons for the anomaly detection and when we did this and when we so?

A

This is one such example of anomaly on our Dussel drop and frankfurt pop by increase in the duration time and upstream time, when we did this, we realize that we need something in-house like this. This is not scalable, because we cannot custom tailor for exponential smoothing and we couldn't train our trend detector. So we came up with something called pi UNAM. This is our simplified version of our journey. With regard to anomaly detection, we started very simple. Two and half years back, it was very dynamic alerts.

A

Everything at eBay was static, we moved into dynamic alerts and then we leveraged into anomaly detection. Where expect and a dot- and we have in our solution called PI UNAM, so PI enum is Python base module to actually detect trend deviations and we have fine-tuned to actually address our concerns. For example, a traffic drop and a latency drop for us is different, not for a system because for us, ladies, a drop is okay, but that's not the case with traffic drop.

A

So then, when we were successful to address anomaly detection on one specific data set, then we ran into the other problem. As I pointed out, we were collecting data set from different different resources. Like client, error, client interactions say latency traffic. We were having this gap like this is too much on us. How would you address this correlation altogether, like? How would you actually come up with the RCA?

A

Can we come up with a system where RCA is explicitly addressed and you could actually have an API to pull out the stats and send to the dev ops that oh, this issue has happened because of so-and-so reason. Would you can you please take care of it? So that's where we got into grano and Han will talk about Carano.

B

Hi everyone, my name, is Han Jung, Hwan, I'm applied researcher at the Korea of eBay, so grano is a solution. We come up together to try to address anomaly detection across data set metrics or different machinery models and, let's recap, the vision of a non tection right. So in this work we want to leverage ml and graph based solutions to reduce mean time to detect of production incidents in distributed system. So on this empty PD is stands for the time between the incidence and somebody is knowing it.

B

So somebody here is either uncle personnel or it's an automated system. For example, we trying to achieve auto remediation. You know where data platform and etc and knowing about doesn't count. If there's you get too many alerts and or you get flooded by alerts and also, if you have too high false positive in machine learning models that doesn't work, because we cannot confuse people with too many signals and knowing about it, so it actually means the incident or the root cause cause of the incident.

B

And so grano is an end-to-end graph based on our detection system and root cause any system for eBay distributed cloud native systems at first we're trying to enable this for our distributed data systems, and now it's in production and our edge services and ufs system is our second customer and we work together on a new solutions.

B

So there are three high level layers of grano, the first ones. We want to do machine, learning, time, series, anomaly, detection: we is all possible masters that we think suitable. The second layer is we adaptively endless all the health signals on a real time, property graph, which is connected metrics events or it can be also project on a system topology.

B

Based on your use case and a goal here is we want to leverage the grant graph connectivity and high understandability to identify the root cause of your production issues, and then we have lots of applications from there. We can do automated, alerting sense, like message emails, or we can have our UI to explore and understand the challenges. One main reason behind this is actually when we try to go in towards a ops, which is the trend. Machine learning model is really hard to be understand. You can send the score say: hey.

B

We believe this is 99 percent of chance and the uncoffined so confused. What is 99 means, and this is score. I, don't read this I read what I want to see. I would read the dashboards, so there's a gap between this, so we're trying to leverage the graph to help bringing more translatable machine learning knowledge here.

B

So I want a quick do a quick introduction on the knowledge, graph or heterogeneous graph in our case now, in general, knowledge graph used to represent the knowledge domain and is introduced by Google and also it's connections in different way in a semantic way. In our case, we're trying to represent anomaly detection to me, which is alerts and metrics we care and we connect and buy different type, for example, sub-matrix of or alerts of, and is actually enable or encode knowledge within a network with as a node and edges rather than tables and rows.

B

So you can do much faster traverse, graph, algorithm or it's easier to visualize or understand, and why we use graph or anomaly detection I. Think there's many benefits at the high level. I think first of all, graph is one way we is a native master for us to understand unknown things at the beginning of things like.

B

If, when we go to a meeting, we easily use a whiteboard and then crossings in the graph and also comparing two machine earnings is a white box approach and it's very easy to visualize and searching graph algorithm engine is expensive, but now I think we are ready. We have super-powerful hardware's and also very convenient graph database, for example in this work we use neo4j and the first thing I think it's very friendly to AI, as we everyone doing trying to do.

B

Expandable, AI, I believe graph is one good solution to do that, because you can do two ways. Actually the first thing is you use a graph, and then you embed it.

B

You do a graph embedding and then you can pass it in to your models and all the other ways you can use it as a translator of AI, where you project your machine learning results on the graph that easier to understand and you can easily get the context and last thing is very suitable for doing recommendations or trying to refactoring or trying to find the connection as a RC a so given that I want to recap the motivation again.

B

So this is a example of we're trying to improve at eBay before so we receive alerts or from our invoices and x-pac.

B

Actually, we receive alerts to say at upstream time is too high, and over uncor personnel actually have to go in through more informations, more sub, metrics and spend time, and also try to do the route cosine SS Pat himself and pinpoint, where is doing the triage basically and then she has also to have to explain it with a graph with lots of information, and then still this may not be understand by the other people, and it's not really actionable from the machine learning side.

B

So we leveraging lots of human knowledge here and it's a very manual so as a quick walkthrough for granel, because we have limited time. I just want to show you the high level design approach. Hopefully this can be suitable for your use cases, and so at the beginning, right we have data pre-processed from what you've just introduced.

B

We have all those insights and we have machinery in jobs from X packs and also we build panel where there's different type of models in there, for example, at the addictive decomposition module, is that we trying to find the trend and seasonality of a matrix. We remove that and the rest office noise when you noise, is to hide that's anomaly, but that's all technical details and you can find lots of easy to use much engineering detection models there and you can easily unboard it to your system.

B

If you don't want to choose a commercial solution and then we have all, these results are actually stored in our back-end and we use a service to project it.

B

So, as a first step of granda, we generate the graph anomaly graph automatically in real-time, so we have a sensible graph schema where you can, except you can introduce the components you want to be the monitor or the dimensions you want to monitor at the graph, and we also introduce the links, different types of links and also you can have different types of alerts from different models and then the alerts can have different time range of different severity and etc. So, but anyway, we will project all this in a graph DB in real time.

B

So this I'll start giving you some high-level, simple examples. For example, we have at the dimension level, we have different sites, flows and pops, so for sizes can be our US size, Europe, size and flows. We have checkout flow, buying flow and etc and pubs. We have actually certain different pubs across the globe: that's all high-level dimensions and for the high-level dimensions we can make it combinatory and we can gain low-level dimension metrics.

B

We want to monitor them as well, because they could be a root cause and then they will be served as also as a supportive signals. So in this case we have us check out at San Jose as a aggregated dimension, and then we can build a link as a matrix off of those three high-level dimensions. So we construct this matrix map in real time and then from the some metrics to the metrics. We also build correlations between the metrics, so we observe the longtime behaviors of the metrics.

B

We use a simple Pearson correlation and we try to detect who are the metrics moving together, so they are correlating. So, for example, in this case a two car is correlated with a checkout, because you tend to add car first and then move to check out when you have a spike in a two car. You also have a speck in checkout. That's the business logic, so we're trying to detect this kind of logic automatically by algorithm, and then we have alarms from different places.

B

Duration, p50 alarms is we, for example, we can get it from expect or over models for each alarm. We also embed the alert knowledge raw knowledge on the edges. For example, what's the criticality of the alerts was the expected range of death metrics and what is the actual metrics and other informations?

B

We use that later to present to the uncle and also to power over origin, but that's a schema right. We can have a duration alerts on the sub dimensions or high level dimensions. That's the graph schema, and so we have this anomaly graph.

B

What we do next, so at a high level, we have algorithms to support more accurate anomaly detection so again, back to our example: we're receiving this duration, P 50 alerts and on Pacific dimension, which is us I, took RSA Jose and we paste we will score H based on two things, mainly the first one is the severity of the alert, so how many hosts of frequent this alarm is and for houses survey G for each other, because you will get a confidence from your machine learning model.

B

We leverage that using exponential smoothing and the second thing is alarm connectivities. We believe for the important metrics, if you were spreading out, if you're, connecting to more dimensions means this is you should get more interest on this and that's the two cred here with scoring on edge? After that we will have each edge scored and we want to move the score from the alerts to edge and then to the actual dimensions. So in this case we want to learn how much attention we need to pay for the u.s.

B

at to Cara San Jose, so it may receive more alerts. So we are talking about how to aggregate alerts at this step, so each edge has a score now and the way we aggregate them is one that we have knowledge engineering all wrong. How actually manually detect a define who are the more important ones or the other ways we can observe for the one that unknown right unlabeled, we observe the historical trend of that of the frequency of that alarm.

B

So if you are having a high frequency over the past time, then we believe is likely to be unuseful and some of like the arrow, related or latency related, which is hard to see that we will be getting more attention if we don't label that manually, so that is for ultimate over knowledge, engineering step but I, believe human knowledge is more important in this step as well, and then we use that weight to aggregate all the edge scores.

B

So after this step, this dimension will have a confidence score, so that will be aggregated and then actually that's the one last step, but the most important one we will propagate across these dimensions. So I think a simple reason behind this, like if I connect to all the people here and I actually I have a various and everyone have that. After this talk, then probably I'm the root cause I spread it out. So we need to build edge right. Of course, I don't have the various, and so this is the principle behind this.

B

So we want to propagate in different ways, for example, for sub matrix. We will propagate from sub matrix to the main metrics and the aggregates to boost the signals and also within the metrics that we have defined that dependencies either manually or automatically. We want to propagate between them, because, if I took our shows the issue, then probably it will impact the checkout. So that's also another way of replication.

B

So that's actually also one one of the main reason why we want to leverage graph, because it's easier to do this kind of calculation, it's hard to find them across tables and and so on. So after this all, every node, actually every dimension will have a updated score, will includes their initial score and the propagated score. We believe that will be the final attention you uncle need to pay because we adhere. We leverage all the different signals in the way that we like. So after that we have different use cases of this.

B

We build a knowledge API, as you can Curie in different ways, or we can build a graph to our layers if we want everyone can get in size as they like and also for now. We try and you sent us luck, slackers and email alerts to notify the uncle personnel with a more detailed information, because it's connected by the graph right and also we want we're building a UI to interactive, allow them interactively do the trash and understanding of the anomaly.

B

So this is an example right, so either we will run in real time. What's the others that you're receiving and what's the attention you need to pay and also when you talk about what alerts week number, this is a duration and upstream serve time alerts to the Australian side and we can see the sub dimensions if he's having an issue and then, if you'd, up double click on there, it will move to that dimension and to show you if Sydney at Australia and I took our house, their house signals.

B

So the uncle can going using leveraging this Messer to see more insights from our you know huge data set, so they don't have to trust only on one score of machine learning, which is hey. This is going wrong check it yourself and that's our principle behind it.

B

So I want to conclude with couple takeaways, so I think Anoop give us an introduction on how in what system is powerful for us, it enables lots of critical features for eBay, and today we also able to serve better customer experience with a high APB powered by our US team and anomaly detection system and, as a second part of our talk, we think we should leverage machine learning models, graph, knowledge, engineering, I think I want to stop here for a second I think human knowledge is not replaceable, especially when trying to do a ops, I, don't believe in, like we are looking at the fully automated redemption anytime soon, because it's it's highly positive I think there should be always people involved in a certain level.

B

So I think knowledge. Engineering is very important, at least at the early stage of the a ops and rather than relying on one single method or metrics. So we're looking at a single point. Failure at here so at last I think graph is very powerful to understand, connect and make sense out of heterogeneous data set in different use cases. Anomaly, detection is just one you can use it for many purposes and yeah I believe it's very powerful and you can look up. There's lots of application online to explain it. Thank you very much.