Cloud Native Computing Foundation KubeCon + CloudNativeCon China 2019 (Shanghai), 5 Jul 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Right-Sizing and Auto-Scaling of MySQL Containers in Kubernetes - Yuan Chen & Min Li, JD.com

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

Right-Sizing and Auto-Scaling of MySQL Containers in Kubernetes - Yuan Chen & Min Li, JD.com

JD.com runs large scale MySQL databases with Vitess on its Kubernetes platform in support of its internet scale e-commerce services. Right sizing and scaling of container resources is critical but difficult due to high uncertainty and variations in workloads. This talk will present how JD develops optimized sizing and scaling techniques for improving performance and resource efficiency of MySQL clusters in Kubernetes. It will describe a system that combines statistical analysis, forecast and optimization algorithms to dynamically adjust containers' resource request&limit values and reschedule containers through Kubernetes and Vitess APIs to minimize resource usage while meeting QoS requirement. This system enables JD to manage its MySQL cluster resources much more flexibly and efficiently, and helps JD dramatically reduce operation and hardware costs of running MySQL in Kubernetes.

https://sched.co/Nrlu

A

A

A

System in your business majority.

B

Automatic data like this.

A

Business type and we can have a business but worrying thing about you, usually war and Joseph calculus tix of that I've. Taken a long time when we came.

B

Even if they said no, that's not, it does one instantly its.

A

Need to guarantee, in.

B

The business of the.

A

Allocated each jobs this was here before you was are now most of traditional focuses all the requests about it earlier about the lowest about. Do.

B

You see between.

A

B

A

Each other will not be achieved at the same time when.

B

A

Any the book, in particular.

A

Will be here are others with us and.

B

A

A

C

You one thing that I want to mention.

B

Is you can adjust it.

C

According to your or.

B

C

To be higher or if you need to be 95%, so that you can see this quite and.

B

C

To make is this: actually, this is already improved and recurrent.

C

For example, the proportion.

A

1 to 2 or 1 to.

C

4, so the faster we can save so many resources. This is quite conservative or sometimes if it is my aggressive thing to all be problems in its performance. So on the one hand we need to save resources resources, but on the other hand we also need to guarantee the performance, but occasionally sometimes there will be a gap, but this is just occasionally.

A

C

The second one I'm trying to mention is how to manage our database first is we need to give it a right sizing and another point to make is I want to emphasize there's just not just one time, because it's a cycle, a life cycle, and it is not real time. So it is a life cycle and we will observe its changes if we find that there is an incremental trend that we will adjust it about.

B

The limit and its requesters you're constantly adjustment.

C

First, maybe on the one localhost and we have some practices in your kingdom, for example, it's CPU memory or its request, and we do not need to restart to contain that because for some questions that need to be restarted. But all that.

B

There will be a high cost of this day if we have the increments of.

A

C

Know that the database, when it is increased it to a certain degree- and there will be not enough space in your disk- oh you're, I, hope and we'll- have some problems so that you cannot resizing through your localhost or.

B

C

You need to bring scheduling, to migrate or to.

B

Show you an ad hoc way so under.

C

That circumstances we need to have such betas and this image.

A

Can show you and.

C

You can divide it into one or two or three, or you can just simply migrate it on this platform and to combine it with. Another challenge is just as I have mentioned. The database has States, it is aesthetical I'll.

A

Be showing during a migration first.

C

Maybe there will be some impact, but the impact we will create.

A

C

First, you need to have you need to build a.

B

C

On your host, but during this process the data- sometimes it may take several hours or it needs some configuration. So we need to optimize this process and we need to be very careful.

B

About in solution, usually so during our auto.

C

Scaling, for example, this turn is contained at my circle. It does not have enough space, maybe you don't not need to migrate. It is also available.

A

So here mainly, we.

B

Can't do these three.

C

Things first is migrational because sometimes when the database is too large, so it.

B

C

Merging about their hostess election, optimization like this addition or removal other containers, so how to do it or whether to do it, how to migrate or which you to migrate so mainly.

B

C

Will take some three.

A

Aspects into consideration.

C

How, where do you migrated to, for example, the affinity and about your maximum value? Maybe you will have high resource code, I, dunno and the.

B

Second, is the migration.

C

Cost as you currently or won't, take the size of the data into consideration.

B

Or you go and among students willing to.

C

Make useful after your microchips, we should also think about the objective host or a match between their association. So we will take these three aspects into consideration and until optimize, the local.

B

Selection, whether.

C

To add or remove and was.

B

Designed talking.

C

About the addition, this is another key aspect for optimization.

B

And I think information on the slide. So during the migration.

C

Of resources, we have a metric that is a multi resource balance. What is good? Because you want to avoid.

C

Avoid the situation out there into no space of the memory of the CPU, so you need to balance the Monte resource. You know and them were closed. Currently.

A

It is open source and.

C

A

C

You put it on the China.

B

To opium European ranch.

A

C

You put this container on this physical host. How.

B

C

B

C

Example, 20% CPU out of 20 percent memory and you and so that you get their.

B

C

The first once, if you remaining affection.

B

C

Remaining fraction that exists Sarah, but it does not take the future.

A

Into consideration.

A

C

You get 5 percent to CPU and five percent of memory remaining friction button. However, the what does not work like this, because sometimes the memory may be minus.

A

B

C

The memory and 5% of the CPU she saw that under such.

B

Circumstance it does not come.

C

By this one come up with an optimization, it is a.

A

B

C

Is a multi recess and present we.

B

Have superior memory.

C

C

So we have 70% simeon.

B

To you a sense memory, the.

C

Second is 40% and.

C

B

Want to migrate the container water that we can relearn.

C

B

Patience of you and 80%.

C

Of memory lane so traditionally, maybe you can put it on post one. It will be better woman.

B

As your more balanced, but our.

C

Idea is that if the remaining fraction can match you, the request is your main inflection. You.

B

Want this, it will be better what does mean by matching your example.

C

B

C

And 40% everything.

B

That can use completely.

A

B

A

To take these to practice,.

B

A

And sake you, your choker.

C

Situation and how similar they are, no similarity, so you can see the equation on the slide and we need to evaluate the similarity between the request, source and available resources, even to three-dimensional.

A

Four-Dimensional.

C

That it can also.

A

C

So they cosign. If it is 1, then it.

C

Sometimes the CPU remaining fraction, it's.

B

Not sometimes the memory meaning friction.

C

Is much so? This is according to the feature of the workload to schedule and to scale, and that is why we need to take the workload into consideration, and this is quite suitable, no matter its performance or its functions.

B

Can work quite well.

A

C

If it is good enough, then it is.

A

Also, ok, and this.

C

Is the result of the experimental evaluation.

A

C

Can see our suffer, it is about 5000 service. I am doing. Take you train this just.

A

Get your gonna be gonna, be you and then we don't pay. So the two results. If we.

C

Cannot work if we cannot schedule it anymore? That means.

B

C

Might be some fortunate, and this is the current of version that came out as Akane. This is.

B

C

Four hours, it's kind, maybe 20%, and it if we cannot be scheduled anymore wait, but we can still schedule about 20%.

A

C

If this is the result,.

A

C

Our way is to schedule and undo.

A

It cannot be scheduled any more about.

C

The 25,000.

A

Containments some.

C

Being financed and get success success, so you can see the success rate on the right side, so in practice, I think this results. It's of great significance to.

B

C

20% improvement, so this is you what we want to check? My comment is because it is a comments, as shown, so you can adjust it. According to your businesses, we.

A

Got to optimize.

C

Another host selection know when to migrate or scheduling, and that's our magic restaurant is his romantics.

B

C

Will put this resources on the container.

B

C

B

Their ability so.

C

I pretend it's that is, we should have this infection. Wait.

B

C

B

Some more memory, a memory function.

C

B

Message is similar to microfiche.

C

Idea why we need to have the.

A

Weighted samosa.

B

Memory, if our application is secure.

C

Intensive so that we can give in to a higher priority. But if it is memory, then we're working from memory.

B

Aiya priority, so we all will you.

C

Can see them, they should be set up. Let.

A

Us say workload.

C

B

Can see the equations on.

C

The slide so after a long time scheduling this database is questionable. We're.

A

Now very what kind of containers brings.

C

Them on 16g.

A

B

C

This we can calculate balance the cpu fraction and its propulsion.

A

C

Weighted sum of CPU.

B

C

Total weight of all the way from of the memory and the memory of fraction so we'll take these factors into consideration, and this is one way for us to improve.

C

Another way of optimization and.

A

You will consider.

C

During optimization of scheduling of the container, what fundamental thing is how to design your.

C

Association come on you suppose.

A

C

A way I put it on the container, you do not know which can be improved.

B

And maybe it is.

C

Predictable so.

B

C

Work better, this kind of also been taken and some metric, the median absolute deviation, so we can calculate the m18 so.

B

That you evaluate the.

C

Results: usage of the.

B

Dynamic system.

C

If the resources usage changes so that we can, we can we.

A

Should not do it more? Your.

C

Clothes on that on that control.

A

B

Is another factor to.

C

B

C

Is about affinity.

A

C

You base, it, is, you know, rolling, and it does not change very frequently, so we can.

C

A

B

C

The physical host they can reach the maximum value. At the same time, there is a simple way this.

B

You already have a container to.

C

Predict the increments, it has high correlation and it can calculate it. This.

A

C

And just now, I have you already talked about this baseline, that is the vibration first, how to sign my circle, the idea it's funny you're likely to it's at.

B

Stage we do not have enough.

C

Space on the localhost how to migrate over to schedule it and how to optimize it, we need to consider cost also.

A

How to locate it.

C

In size scale, you guys it should be conducted step by step and then.

A

Traditionally, the summarization is for the stains. This was stick to management.

A

Actually, we have documented.

A

Usually the best updatable Natomas contracts.