Cloud Native Computing Foundation KubeCon + CloudNativeCon China 2019 (Shanghai), 5 Jul 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Tune Your Microservices by Learning from Traces - Zhang Wentao & Yang Yang, IBM

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

Tune Your Microservices by Learning from Traces - Zhang Wentao & Yang Yang, IBM

Tracing plays a more and more important role in the world of microservices, to help with trouble shooting and bottle neck analysis. But within huge amount of traces generated everyday, what can we learn from them, and make full use of these valuable amount of data? In this session, we will discuss how to use Kubeflow to train tracing data generated by Istio, to reveal the patterns you may never discovered by naked eyes, like seasonality performance downgrade, biggest influencer of system failure, correlation between microservices, etc.. It will help with root cause analysis, and eventually tune your microservices with optimized performance.

https://sched.co/Nrne

A

Maybe this Sunday, she has already been appeared for many times for from the presentations, even by different speakers. The Cooper flow can help us to making.

B

A

Of the machine, learning workflow and kusanagi, simple port bow and elbow from the name, you can tell that at the very beginning it will do the tensile load and if you already attend.

C

The previous session of this.

A

The tank flow and Cooper flow has, or it means elaborate everything detail without fruit flow. If we only have the sense of flow we'll have the machine learning job, which is communicated first I need to understand that principle event. 10 machines intend knows I need to know their words, I need to know their resources and when I submit a new disability which has the PS no and which one is no oneno. So for the good flow.

A

It is like picture on the right hand, side, and it can also help us to show the sonicated complexity of the environments free summer in a motor survey, as well as in the monitoring it provided behind the parking tools to show this memory capacity sold us to help people focus on the key machine learning code.

D

A

Our midst, and today we use that in the flow without.

C

B

A

Each year we'll collect a lot of traces. The amount is about 700,000 each day and.

D

Later I would like to.

A

Share with you, architecture.

D

B

On the left-hand side on the.

A

Blue, it is a kinetic question in first it is the space it is nice pace and another once it has the appointment of the micro surveys in the East Hill. So on each of all, there's we have the steel.

C

They also have the eager and.

A

Elastic moving in space, but we do use the steel people eager because it is all in one and it doesn't. It cannot store a lot of cell traces. They cannot support a lot of paralyzation. So that's. Why does you need to introduce a.

A

Architecture, which.

C

Can support a large amount.

B

Of the foundation but.

C

A

We can trace the trace both the.

C

A

Eco-Fashion in.

C

A

A

Concentra a lot research and finally, we chose in that last exert it will be stored in the.

A

Cement part would be the right-hand side, which is for the machine learning. We have the data preparation and during this tab the data will be pulled from the es and then we'll use the data for planning, because when we class these chemical traces they can be buried in different layers.

A

So that's why we prepare some data is here and we only slash the build that we pay attention to and then generate a document and put it into the service and all this kind of the file will be consumed by the follow and train via a motive. After iterations, the model will be stored in the object, storage.

A

By the Koopa flow run on the banana cluster, this is the overview of the architecture and now I'd like to give the floor to my coworker mr. junkman pal.

D

A

Thank you and later I would like to share with you.

D

A

Machine learning, in our case.

D

And just now, I just.

A

Introduced that in the trays there are a lot of information among them. There are some.

A

D

A

That in each phrase, I know.

D

A

Which service name it is from and where it goes to.

D

A

Source, the origin and the address and.

A

In the service.

D

A

Can we use these days and today.

D

A

The target of the machine learning.

C

The first stop is ETA in PDA. It is a concept.

A

On the statistic, oh, that I dedicate, which means that, in order to process the data you need to perceive.

D

D

A

Futures of the data in different.

C

A

And different I mentioned after we have a perception of the data. We can decide what kind of the model.

C

A

D

A

Use the service duration as the object of the data process.

C

And we can observe it from different time range in time.

A

Break because in different time frame we can see that the features the data show our friends. So let's have a look at.

D

A

Picture he's not.

D

A

Here so each frame lasts for about 30 minutes and we can see that within the lifecycle of the 30 minutes, the duration watch wait back and forth. So the requesting.

D

Time is much waiting and in.

A

This way, we can know that it is related to the.

D

A

The requests people we don't need to analyse that. What are the most of the fluctuation and, let's.

C

Have another dimension.

C

A

D

Has already been.

A

Which means that, within the range of.

D

The 30 minutes the.

A

Fluctuation of the duration is really violence, but when the expanded range 12 hours is.

C

Still 40 and also.

D

Show an features just like.

A

D

B

Can see that there.

D

B

B

Differently, right and I, like this mo doing a D time, welcome, is south, more requests.

B

Any show perception data for machining.

B

The duration is a time serial data, so we're going to pick.

B

C

Here we use ohdf because.

D

I staff its traditional art in.

B

Cameron arias takes time Weiss.

D

B

There is periodic graduation.

B

Transmitted then the network can learn this character.

B

Is a structure, maybe the layers are may disappear, then is DM can come to play because.

D

B

For control, it will have us to decide whether these features should be carried forward or neglected.

D

B

D

B

Crispy we will use natto to generate Nero sequence based on existing sample. We do with language.

B

Maybe to speak.

C

For us, in our.

B

Case based on the previous example, then we will predict the upcoming generation.

D

B

Terms of tracing the machining, let me use Tracy.

D

Before of nominee detection.

B

Time: sequence: prediction based on the previous time: sequence.

C

B

Can generate for future time sequence and I will use this estimate days. I will compare these estimated sequence with the previous one. If there is a tolerance, an aviation and I.

A

Can tell whether.

B

D

We have 1990.

B

Data points in the test, dataset.

B

1996 dealer.

D

Morning, which.

C

B

Ain't no money in.

B

Which is indicated in the report.

D

B

How can we tell whether it is anomaly or not.

C

B

We have used some specific algorithm and also based on our experience.

B

If we use different time frames,.

B

Characteristics.

D

B

D

B

Value is the same. The number is the same.

D

C

D

Displayed are different. It's.

B

Like a no story in China, blind for blind people touch, T touched anything it when they reach to reach to.

C

Laugh elephants.

B

At lag they say it's a.

C

B

They touch anybody, it says it's our own. So what is the animal actually add? The elements elephant that has the.

D

B

We're trying to get these ideas in order to do so. We need to.

C

B

A bigger picture: this is.

D

B

View the data in short time.

D

B

See huge penetration.

D

B

Unacceptable? Which will.

D

Be defined as nominal, but.

B

If we extend the time frame, the.

D

B

Is still there, however,.

D

B

Abrupt then the motor will define it as a normal phenomena.

D

B

Has trained for a while.

B

So for us we want the model to capture, to capture these a nominal anomaly in the.

D

Short duration.

D

B

Also hope its mortal won't mistreat a nominal anomaly as a.

D

B

Train the model we can't just put it aside, bro. The purpose is.

B

D

B

Application example is this.

B

Time reference we can use this. This 10 points to estimate future time sequence. Our time window we can estimate, is to duration.

B

B

Example, I use the real time to read to do. The institutions relies on completed sequence example.

C

Future time sequence, these.

B

Are the previously so? The first approach is more accurate.

B

We need we need real time or accuracy, then or estimation. True, if you just want to know the.

C

B

The time sequence based approach would be better.

B

Three minutes or one hour.

D

B

I use this as the input or kaboom, but then I.

C

Will get a small timeframe.

D

Which we call it 401?

D

B

With this morning, I can estimate what happens in the last 30 minutes is normal for the last 30 minutes, but if we can also extend the timeframe.

D

B

C

A break their motto.

B

D

Characteristics.

B

A refuge over the last 7 days.

D

B

Would indicate that.

D

B

A result, the duration, frequent.

B

Predict that over the long term.

C

B

B

B

What our training and deployment.

B

He is the con job.

B

D

B

Bigger timeframe, we need more, your data set that was.

C

A longer training.

B

But or it's more training are more.

C

B

30 minute model don't be smart.

D

The training very short time that.

B

Means we get iterates the model.

B

We use the is do it's to help us to do publishing in different version.

B

Tracy Tracy is younger and do the kill flow.

D

B

To identify the anomaly services, the anomaly is not a girl, they can notify the operation team.

D

B

B

Auto skating, so the.

C

Micro services can.

D

Provide one service.

B

So this is what we want to share it. If you have any questions there is your hand, please use the microphone.

D

B

When you do, when you do the anomaly detection.

B

Are you mentioned upon the classification.

B

D

B

Here, in this time, for any a fluctuating.

B

If I just use the long term tomorrow, it will treat.

C

B

Value as normal, because it predicts characteristics, will happen here after.

C

B

It will treat it as a normal phenomenon. In this case, I will use the cluster every certain to identify the points.

B

For there was a normal point.

C

B

It why don't you use some Engineer to.

B

Use some traditional machine learning like characteristic engineering to identify and characteristics. First, wouldn't have be better. Maybe here we will think about out. We will give it a try. First question is that.

B

Well, you try to use this in truth, together with the scheduler. Yes,.

D

B

This will involve the distributed motor.

D

B

A virtual dimension: that's a good job to train the data afterwards, this can be traumatic. Therefore, we have a lot of job.

C

B

Was one to use that you say it's.

D

Distributed approach.

B

Traditionally, scheduler has.

B

Faced some challenges when it comes to a paralyzing but later we'll think about use a cube edge as a.

C

D

A

Proposal asking this question class: if you try I.

D

A

Like to know that girdle and the others which one who are fast, the usability so we deploy it, needs time at the time cannot be pasted. It depends on the time and a network for test and why I hope that I can use that as soon as possible right. So it requires the the good performance. So I would like to know that. Is there an imbalance between these two episode as it can be, I can.

D

A

So when you before the model, can you still use the theory and another kind of the model of the optimization method? Well, so far we have encountered this kind of problems so far all right. Thank you.

C

A

Have a question.

C

A

How can I ensure that what I train.

C

A

The one I will re use in practice, so, in other words, do you think that the training is only available in a campus in one areas or it is incomparable? Well, actually what we train? It is the best service. You can understand that it's a related note. Ok, so after I train it I can only use it in one service right here. This right is not a common.

D

A

In the final, it will have the autos going right. Yes, so I would like to.

C

A

C

Scully and I'd like to know that when it is decided to scale.

A

And what, if will decide.

C

A

D

What are the criteria.

A

But I would like to.

C

A

Example and actually I can I need a tea requires.

C

Menagerie I can only small, it's 50.

A

Requests, but if it is go up from them, teach 80 and it will fill in a 52 and what would happen and we'll try to do well, Ashley. In our case, we don't have such a large amount of requests. Yes, this part of the problems but may happen, but we won't go from the extreme like your own, as you described well, Ashley is there.

C

D

C

Doesn't when you dude.

C

Yes, the auscultation can.

A

Be happen, so how can you ensure that this kind of information can be avoided? Well, actually, so these kind of the auscultation with a scale up and the number will be increased, so the process ability ability increase in this way. The duration.

C

Will be reduced.

A

Right, yes, that's right! So that's why that's if we have to.

D

Use at least two models.

D

A

The or discovery I use it for the long term model. So in this way, I can avoid that and will notice the violent graduation.

C

A

That would help what.

D

Does that mean I, don't know.

A

C

Model is responsible for judging.

A

Not the same service right now, so this one is for compensation right. Yes, then this one is for compensation.

D

A

Right down to the notation, thank you very much for your attention and if you still have any questions, please go home to talk to us later.