Cloud Native Computing Foundation KubeCon + CloudNativeCon China 2018 (Shanghai), 21 Apr 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Large-Scale K8s Cluster Operation and Management - Lv Jiangzhao, JD

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

Large-Scale K8s Cluster Operation and Management - Lv Jiangzhao, JD

JDOS(JD Datacenter Operation System) is the very large-scale container cluster system that running in JD's datacenters across the world. It was designed and developed based on Kubernetes. Today, almost all the JD's business has been deployed and running on JDOS. At present, the number of containers in JD's production environment has been millions. How to manage such large-scale clusters is a challenging issue for JDOS developers and operators. However, JD have only 2 full-time SREs to manage the clusters. This presentation will share some of the following experiences: 1.Node Component's detection and management; 2.Master Component's fault detection and failure recovery, especially for the etcd nodes; 3.How to significantly reduce apiserver requests, in order to build a much larger k8s cluster.

To learn more click here: https://sched.co/FuKA

A

Today, I'm going to talk about large-scale cave cabinet s, clusters and management and operation of JD.

A

Oxidation so current status offenses your.

B

Coj, do you know more second,.

A

Out on to my signal, surprise: it's the operation.

A

Starting from 2014.

C

A

A

B

A

Achieve the full containerization in 2015 to two major promotion festival, we achieved the full implementation of Jada's 4.0 and accumulates many experience.

C

A

We achieve the container distribution second skål skål ation, combining with docker I, think you might all know that we have scaled containers in 2016, since we investigate hypnotics technologies operator fatigue in.

C

A

Have a whole new resources platform combining with kinematics.

C

A

We began to establish our ecosystem, combining the network, storage monitoring, log collection, kernel technology.

C

A

The whole ecosystem was started by our departments, and some of them are tailor-made using open.

C

Source technology.

A

And some of them are independently developed. During this period we also built a large scale calculation perform and the image center in 2017, the J.

C

Dos 2.0, using.

A

C

Antics technology was launched.

A

And it was the world's largest proven artist clusters and we achieve a second level Scout scaleable of these platforms, and it was process achieved on a platform from source coding to encoding and to change from the hours level. 2 minutes level.

B

Second level and.

A

In 2018 we have J dos 3.0 and it was a pass platform. The resources can be allocated, it's more intelligent.

A

This is the self-developed components.

C

A

Je dos 2.0, including.

B

A

It was based on T, V, DP TK, as informants load. Balancer was one of the best solutions in.

B

The industry and.

C

Also container DNS, it's.

A

A high performance DNS service developed by us and also application renew services. We offers quick online scaling up and down grayscale release, etc.

C

A

Also, the rollback of version.

C

A

The same time, there are also monitoring service from resource monitoring methods, monitoring.

A

Among the service based on monitoring index.

C

A

Then container locks flex locks. These system is developed by us. It can quickly search or setting, alarms or long time, storage.

C

A

Service by using a CDN, acceleration and other technologies to quickly distribute.

C

Images in multiple.

A

Cities multiple data, centers image, scan technology can safely scan images and.

C

The most important.

A

Part is resource service for the bottom level of kinetics earth resources. That is the overall management of container resources. Our allocates quarters at different level, and we calculate and manage resources for different departments of uses and applications on different levels. We will send the bills to our customers and how they use the resources to help them enhance the.

C

Utilization rates and also.

A

See ICD service.

C

A

Icd service, it was an independent service based on Jenkins. You can't supports horizontals, spelling and encoding service for thousands drops at the same time and it was introduced by my colleagues recently.

A

Currently, this is the status quo of clusters. We have around millions of containers. Online single cabinet is cluster, have over 9000 notes over 20,000 of my services and applications, all the services shopping malls, logistics, Big, Data, etc. All those services they are fully containerized Connect containerized.

A

Operation and management status quo of foster the daily delivery achieved from 10,000 each departments, operation and maintenance. They can reach 10,000 and special operators. Two of them clearance of the warnings can be done daily.

A

About the optimization of cabinet investors, Vanitas.

C

A

Are using the version 1.6.

A

We have tailor-mades based on version 1.6. We investigate 2015, why we stopped updating them following the community, since there is newer version like a one point, ten or one point 11 after reaching such a larger scale, it's an it's impossible that we keep following the updating and speed of the community. What we do is.

C

A

On the tailored version 1.6, we will keep tracking the.

C

Update cuban antics community.

A

If there's functional status, highly related to our services,.

B

A

We will go to update that to our production, environment.

A

Configure map optimization.

C

A

Use this computer map.

C

Kinetics, well, you will be familiar that with.

A

Posters girl reaching such a level each note will tens of containers if each containers amount of different from fear maps that is perforation file. There.

C

A

Constant problems, each containment zone in default, setting every few seconds it will ask for a ps4, dynamic multi to mount it to the party. This will give API server a very hard pressure during a high-pressure take a thousands of nodes posture, as example. If there's 25,000 containers each part, each part will have 10 configured Maps.

A

There will be 250, KBS, QBs and every five seconds there will be a request. Then the API, server or ABCD will have a large pressure. What we do is in terms of the dynamic modes computer map we will divide them based on our business, need business demands for hard stand, doesn't need dynamic, Mountain or remove that function. If is a demand for dynamic months, we can click on our module and choose to use that function by analysing what our lives.

C

A

This operation, the request, will be reduced by 99%.

C

A

The scale of the cluster can be increased. Of course, they're more optimizations can be done.

A

And also our fully use labels.

A

Can be used computers or in the production we are using these labels first is.

A

Nodes, it's property.

C

Including on city model, disk.

A

C

A

C

The labels to the.

A

Part where it belongs to it's, not it's the part of the data center related to the network infrastructure resources related. We have zones, we need to buy different zones, different IDC and the status of those and its image status to.

C

The network status of.

A

C

A

These labels, we can have a small range of management and the allocation of resources.

A

We also have a resource for tribute I.

C

A

This is quite leading function in a community.

A

C

A

Have a dimension.

C

A

The resources when there's no sources left in the resources pool for updating deployments it will destroy first and the rebuilt one. If at such current circumstances, other resources is not sufficient and you need to launch a lot of applications. Then the part will be in pending volume. Status prioritized part will not be able to guarantee its resources and will be listed in the scheduling.

C

Of view, and even.

A

If we prioritized parts, we cannot satisfy our customers demands. That's why we have heart review function when updating deployment it will not Reese reschedule instead, based on the original status, it will.

C

Upgrade in place, there are several.

A

Benefits brought by this function. It can prolong the network.

C

Sources, lifecycle.

C

A

It will not okay reallocate our hosts, since it also only updates the background of image, then the repute speech.

C

A

Be increased and we reuse their rotors routers.

C

A

The part we were released around the.

C

A

Always has to container.

A

The part do not need to be release the router, so you will have a better focus.

C

A

Thing to share is that we need to carefully use the automated recovery it involves proven and takes several functions that is soft, evacuation and compile and heart battery. It's.

C

A

Personalized to function that we can use, but it's not the optimized function in the production.

C

B

Groups will be established automatically and based on their functions. The decisions will be made whether to join them.

C

B

Is not required on the host.

C

B

We will look at the priority and drain them accordingly,.

C

B

Addition we have thousands tens of thousands of vacations online, so we need to know their requirements. Immersive scenarios, the developers don't need self-recovery. What they need is not self dreaming. They need to maintain the scene at the site.

C

B

About the scheduler optimization, as you know, the other stream scheduling has some problems. The algorithm is not optimized a just a solution.

C

B

Based on the scheduler, we have done some customization, for instance, for the every set we have done. The random.

C

B

Random sampling to find to the local optimum, we don't need a global optimum once our cluster has reached a scale like 500 or 1,000 nodes. When we do to schedule a pout, then we don't need to turn to all the knots, because there will be a wisdom of resources. Based on this, we can think whether.

C

B

The scheduling we can reduce the number of nodes like every time we just.

C

B

At a 500 dance to find their scheduling, optimum.

C

B

After we done that, we have reduced a time from minutes to seconds.

B

However, the problem still remains with the algorithm when the resources are running out, the local optimum.

C

B

C

B

Would be a waste of resources and efficiency, but thesis of this is various type II. Distribution of containers. Kubernetes is based on C and M.

C

B

Using these architecture in our production environment, we found.

C

B

Terms of distributing the containers IP, this architecture seems to be better.

C

B

What we have done is we have developed our original scan data server.

C

B

Have replaced it a TC, d-wave, Mike SQL. As a result, our.

C

B

Has reached 2,000 per type, sorry.

C

B

20,000 per minute- and that is about our daily maintenance and operation of communities in JD's, to ensure the security of components. We have done lots of things. First. Is we try to decouple the components?

B

Every component must be of high availability and extreme cases, even.

C

B

Is like three or four replicas is unavailable? There will still be one or two available, and also we have the monitoring of performance.

B

The stages of components are monitored every per second, and we have warnings if things come up, if, like the stages is not good, there will be according the warnings and.

C

B

Our daily operation and maintenance, we have established some warnings and we also look at the infant and also abnormal parts. This event and part of enormity and also indicators of our clusters.

C

B

There are several types of powder abnormal ease, one is out of the CPU and out of memory, and the reasons behind that is the scheduling bug.

C

B

When we reboot the scheduler smen note.

C

B

They will not select up fresh note. It will restart the old note.

C

B

There is no information.

C

B

The resources are zero.

C

B

Are no resources are on the old note. As a result, there will be a wrong distribution and the goodness word attackers.

B

It well set the status of not as out of CPU or out of memory and the power to have just be dead.

C

B

Abnormal T's crash rule back off.

B

Normally it's because there are some failures of the butene of applications, not for these kind of containers. We think it's mainly happens in online and it's not because of the kubernetes status and third abnormally is terminated.

B

Some containers may be the stages at a facility for a long time. This is also an abnormal status for the part, and the reason may be the host is of life or it's broken, or the terminating of status of the only host is broken. For example, the volume of mountain is occupied by another application, so the whole status is terminated if this happens and we need to get back to the root cause and fix about the fourth. One is all.

C

C

B

This is mainly because the parts walk, so it's over straightened. We have a picture here and it shows to real-time information about the status of parts in kubernetes and also you can see the members of parts and thirties graph. You can see that whether the part is healthy or.

C

B

Wire for older clusters, we have an all-round inspection. Inspection is very important in the production environment and is based on the accumulated parameters.

C

B

These included some very common corner parameters and systemic parameters. Here is a list of parameters here. Lots of em and the second kind of parameter is the version of the modules to maintain the clusters.

B

We have lots of versions, so it's very important to keep these information and the third is the process, inspection and this inspection is to look at each machine to see whether it's healthy and it's, whether its operating properly and we keep operating the inspection system we also use serve here serve, can significantly improve the efficiency of inspection. It can converge within a second the.

C

B

Part is etcd data security.

B

Kubernetes, which a certain scale is very easy to run into this issue. So here we look at how we backup at the data and when there is a failure or a disaster. How do we retrieve the data? And here are three main methods? One is read: data just read the raw data. We have comparator program which can read the rotator directly and then we'll write into the distributed file system and second approach is to use the snapshot, and this happened on very fixed basis, and the third approach is very straightforward: it just copy the directory.

C

B

Another route or another directory and backup in local environment here, is about the problem. A deist version upgrades- and this is also based on our experience.

C

B

Using a computer controller.

C

B

Are several automate automate ATAR functions when there is a function, not ready or offline.

C

B

Will be some operations in the and controllers and the point controllers? Some functions may be removed and as a result, there may be some failures. So we have customized the controller and for just for.

C

Your reference, if.

B

You want to upgrade a classroom or a version, so first you need to stop the controller.

C

B

C

B

Unexpected situation, second, is to stop the scheduler, and the third step is to stop the.

C

Crew Blatty.

C

B

You operate a coolant or other components.

C

B

This is also to avoid the mismatch of the data.

C

B

To avoid some abnormalities of controllers hears about the clusters, data visualization I can skip this because my colleague has also, as already shown these in the flash speech.

B

And here's about how to handle the huge amount of warnings when our cluster have raged like tens of thousands and lots of classes. A hoster may be taken down, and it can happen every day and it can happen with more than one host.

C

B

How can we deal.

C

With it, every.

B

Day and when a coaster is broken, there will be one is of.

C

A

The planet Earth data would see if they belong to the same rack or the same machine.

A

These parts and these containers are, they belong to the same occupation or in the same host.

B

After the analysis.

A

C

A

A huge amount of warnings.

C

P, heaven warnings it.

A

Was caused by certain.

B

A

Alright, a vacation was.

C

Not fully tested.

C

C

A

Some ran into problems and the containers other forces will send our warnings. If we do analysis, they will relocate to C, which is the core reason for without warning, and then we can do.

C

A

Then friendly plenty of prediction.

C

A

Its many for too promotion festivals.

C

A

It's the RNC stuff working on it most of time. They don't know how large container shipping after analyzing the application to see historically during the eighteenth truth with more.

B

Eleven of November.

C

A

Was the container used during that two days and based on that analysis, we.

C

Can make a prediction.

A

On how many and large.

C

Containers should be, and.

A

After launching it online, we can monitor its utilization rates of resources.

A

And all the CPU utilization or discs on utilization and loans, etc. We'll monitor them and generate a reports periodically, just as you can see on the right hand,.

C

A

To comprehensively show that how the resources are being used in that application of, thank you very much. If you have any question, feel free to ask me.

A

Good morning, when you upgrade given a taste monster.

C

A

Stop the controller manager first and then, if.

B

C

A

Speech delivered by people from Google, saying that they should stop the API server first and the scheduler and the controller manager will be waiting.

C

A

That the same yes, we skipped a part of API. First, you can, you should stop API if there, if you upload your business platform in here, you should stop that port and then you work on the couplets controller manager and the scheduler etc.

A

C

A

A quick question if the Google community foster is very large. If you stop the scheduler and the controller manager, and you will be needing a very long process when you need to upgrade a grayscale. No.

C

A

Will upgrade to grayscale first.

C

A

Test environments and then without in the production environments, so it also released some in the grayscale and for the upcoming I know today will be upgrades in matches, so it won't take a long time so first in grayscale and then in badges enlarge. Yes, of course, we will fully test a version. We will not wait until we finish the update. We, you buuuut a test environment to test the online version. Yes, we have a flight testing environments and also online testing environments.

A

For grayscale, because there will be no flow of a newer version and older version will not be any problems, we will ensure the compatibility among them. Normally they are compatible to each other for the faults and the container site. How we do that then. Second question: after maintaining that sites, how will you release the resources? Otherwise they will take off resources as well. After dropping those kubernetes itself has a mechanism.

A

They will keep the container for advertising as long as you don't clean them manually that will exist after leaving the part.

C

A

For recovering parts, the corresponding part, the container will be found out and was gradually.

A

And we also find out the parts that only starts at one time. We will clean them after recovery. We will start the container how you restart sites I mean leaving the FEL status. You can check the log Noby documents for checking for restarted when it overwrites no.

C

A

The data in the container and a lot in default, we said to the proportion that partition that we set for them for controller manager and the scheduler. If you stop them, we without influence your service request.

C

A

New service request, without.

C

A

We stopped the API as well, and what about users they want to use it. Do we stop users from using them for.

C

A

And a scheduler, we will stop the service. How long will you stop? The service depends on how you revise your version. Normally upgrades will be done in a short time, so it's exceptional for users, yes, and it will choose a corresponding time to avoid time pressure. The clashes final picture saying that the prediction of the resources allocation I want to know that before some resources will that be wrong solely occupied by certain node, without the any quarters or with prediction, how will you manage the resources allocation?

A

What do you mean by while not sharing the resources, putting some quota online or CPU occupying the certain resources? How do you manage the resources allocation? Well, we can choose the scale before launching its corresponding to the limits of kubernetes and will not exceed our limits.

A

When many parts are ready, they will season for resources right. So maybe you can separate for different partitions, separate different parts. We saw that in darker some of the resources will be solely sent to you mean Sam. You said.

C

A

Already support CBS s online rely.

C

A

Mechanism right based on special scenario, have a structure departments on application. We will have CBO set, but for most of the time we will use CPU share our.

C

C

A

To view a desk, this caste will be divided based on labels right. Yes, if you using labels, it's not it's not.

C

A

We will not delete them, we will put it as our shadow war status and another question over cells.

C

A

Is no certain scale for that.

A