Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2020 - Virtual, 4 Sep 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Help! Please Rescue Not-ready Nodes Immediately - Xiaoyu Zhang, Alibaba & Di Xu, Ant Financial

Description

Don’t miss out! Join us at our upcoming events: EnvoyCon Virtual on October 15 and KubeCon + CloudNativeCon North America 2020 Virtual from November 17-20. Learn more at https://kubecon.io. The conferences feature presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Help! Please Rescue Not-ready Nodes Immediately - Xiaoyu Zhang, Alibaba & Di Xu, Ant Financial

For a Kubernetes cluster, nodes are crucial to make pods running properly. So it is indispensable to monitor nodes status and detect node problems. Node problem detector (NPD), an open source project in Kubernetes community, is a good answer to address this issue. Nowadays NPD has already been well accepted and widely used in production environments. Actually identifying the problem is only the first step. What we need to do next is to handle those problems and rescue the nodes. In this talk, we will list common problems and share how we establish rules to decide whether a node is ready or not and how to fix them if recoverable. Moreover, we will introduce some use scenarios on how we make a 99.9% uptime guarantee with ten thousand nodes in a single cluster. We will share some experience on how to recover the nodes within 10 minutes as well.

https://sched.co/Zek3

A

Hello, everyone nice to come here to share some experience about the kubernetes operation. As you see from the topic, we will bring you some cases about what we should do when vocaloid is done and in summary, we conclude from how to recovery. Here's a node this topic, partly based on community projector and a party based on our further advanced improvement.

A

Now I want to introduce us firstly, my partner d, hu from and group I'm sure, jung from tencent cloud. Both of us are kubernetes members and very active in community contribution. We are always trying to use the latest clutter native technology in our work.

A

Here is our agent. Firstly, we will introduce a common case which is not ready. Node then introduce a open source projector to find the resistance. Its name is mpd short for another problem detector. Besides, we also introduce a new system. We called not self-healing by using those tools. We would want to share some lesson. We learned and start some discuss at last draw a conclusion of this topic.

A

We will discovery the terminal shoe. The knot is not really like this picture. What should we do next to deal with this? We should firstly make the reason of another problem, visible by some ways like checking node status logs matrix and then try to remedy those beta nodes and then make the node ready again.

A

But how to make the problem visible quickly here we introduced a community project mpd in the deploy, as demo in every node and report detected. The result periodically. Another problem is reported as another condition or event. Another condition is the permit problem mark the node available, such as kinetic, lock, the corn, better disk.

A

Event is temporary problem but informative, such as om, killed.

A

Only detecting problem cannot meet our satisfication. We want to fix a healthy node as soon as possible. We need other operators, alongside with mpd, to help discovery and fix problem. Look at the picture we design, we add some components.

A

Not extension is the customer source definition which takes place of the natural node resource.

A

Mpd will report the result to not extension. Config is a map of repair, measured and problem conditions. Another pair agent has many built-in repair shirts.

A

They rewatch the not retention conditions are here. It will start self-healing by running specified scripts.

A

But which kind of a problem should be attention? Let's do she share his experience.

B

Normally, the problems could be varied from hardware to software. Here we have divided them into below three groups: hardware system, software, runtime hardware issues really include physical disk problem network issues, this car, etc, etc. Here we have not only list of few common problems for system software. Kernel bug is usually the most frequent one and logger that will be another one. That requires completely stop working due to disk 4. in our company about 6 percent of the problems are caused by runtime such as stockholm means runtime circuit.

B

Etc, this is a configuration yamaha for our non-problem detector. Actually, this yaml is quite long. I only take part of it here. We have divide, defines two groups of policies. One is for monitoring docker. The other is monitoring kernel.

B

Let's deep dive to see some sample plugins here we have used prepared check the check fd binary, to periodically check the amount of the being used. If it exists present a condition will be updated to another status.

B

Let's see the right picture, another simple plugin is about inode. This will be quite useful before the system might not get exhausted, especially when your pod uses host path volume and creates too many folders by externally.

B

Actually, we need to write some revenue scripts to fix these issues, we're detecting them.

B

But mostly those ram descripts are written in shell. Two mining coin classes need to be considered. It is very hard to maintain these scripts and keep every fixer attempted. The most difficult part will be the complex interdependency.

B

It is hard to split the issues apart and you have to find out the author, which one is the root cause and which one should be solved. First.

B

Here I want to have some discussions for the deployment of the detector under the repairing agent. Normally, there are two ways: one is through ipm or dabbing packages. The other is using demon set.

B

We can't say which one is best, but in our practice we use stem cells by considering the upgrading effort since the repair scripts and the detecting plugin will be it frequently. It is hard to have a static strategy to manage this, and we use config map to start our strategies as well.

B

Last one is a resource all has about mpd. If you have included too many plugins in your agent, please limit the results of your daemon set port.

B

A

B

The community planet labs, have open source, the project name, the drainage, which will audit, which will automatically condom and join kubernetes nodes based on node conditions.

B

You can set a configurable jump buffer time and the channel will automatically transfer eviction posts one by one. It is very safe. You can see the below.

B

Events here is what we want to address in conversation last. Actually, the repairing scripts are not inversely valid in any infrastructure. That's why we have not taken too much time on how we fix the issues.

B

There is no silver bullets to fix the notes. As you know, you'll need to have your own secret manual. The most important rule we have got is kiss keep your infrastructure simple and standard.

B

It will be much more convenient to fix your.

B

B

Thanks for coming to our session feel free to pin us on slack.