Cloud Native Computing Foundation KubeCon + CloudNativeCon China 2019 (Shanghai), 5 Jul 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Linux Kernel Live Patching - Haishuang Yan, China Mobile

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Linux Kernel Live Patching - Haishuang Yan, China Mobile

This presentation is about a methodology which keep linux kernels live patched and running without interruptions, its technical details, limitations as well as kpatch tools.

https://sched.co/NruH

A

Good morning I'm from china mobile mr. yen, I Xiang so today we're gonna talk about Linux kernel life matching.

A

Also I'm gonna talk about what is life magic life patching first, it's on I bug fixing and can dynamically fix, bugs functions for a kernel, isn't, for example, replaced a function online, for example, if we detect that there's a bug in the function or cluster function, and we can replace this old function with a new one or if you.

B

A

The bugs the only thing we.

B

A

To do is that we replace the whole function with new one, so this is what we do so during this process, we.

A

Will not crash the system or cast a panic?

A

If we don't need to reboot the system, then the risk will be minimized.

A

Why would do this life bug fixing.

B

A

We have to first detect some minor security bugs, for example, change the code and change the structure of the data and leave that minor bug fixing, and we don't need to do a live patching and also we cannot upgrade the generic version of kernel with it and the version of the kernel. If we want to change it, that means lots of patching multiple patching, not.

C

All of them are.

A

Applicable for life patching if we want to add new features, we cannot do it with live patching.

A

So why we're using live patch first so know that the a lot of codes in kernel- the quantities are large and some minor bugs is navigable, and there are some loopholes as well and, for example, mostly, the online systems are using the lower old version kernel. So you need to fix a lot of bugs.

A

Option one is to upgrade the version of the kernel. That means reboot the system.

D

A

You don't have high availability, high availability devices online them. That is pretty difficult.

A

B

A

Try to reboot the system directly in a hard way, then we have to.

B

A

B

A

Is a risk of failure in rebooting, then we need help from professional maintainer in operation managers that will cause further latency in the progress. So.

A

Reboot, it's gonna cost a lot of downtime and their cause is pretty high and especially afraid of.

B

A

This business sections didn't have a high availability mechanism. If we shut down our online business, then that's not good, that's very damaging, and some of the parts cannot accept downtown longer than 100. Ms, so we have to recalculate the reboot time needed and we had to rescheduled and for traditional cloud computing. We have hundreds of physical devices and we have tens of thousands of virtual machines. Then, if we want to reboot them all that it's not realistic and.

B

A

Have to reschedule the reboot time.

B

A

Reboot it in different patches gradually.

A

So had to guarantee that there's no risks for the reboot of physical or virtual devices, so.

B

B

A

Live patching for kernel! That's why.

B

A

All of this system then reboot. If we have live magic life patching.

B

A

Is the reason behind life patching so first, let's talk about trace.

A

Well, we are using this half trace mecanim in life, patch f, trace.

B

A

Is it it's a tracing tracer for kernel function and if we're using F trays, then we are analyzing. Certain events for kernel, for example, scheduling and interrupt.

A

You can help us to dynamically trace the functions of kernel and, for example, the informations about the stack and tracing latency. For example. If the interrupt is occupied.

B

B

A

For example, yeah.

B

A

Will have to calculate the time needed for a reboot.

B

A

We have to find a kernel function, for example the schedule you can see on the screen that actually at each entry they will have system code.

B

A

A fan tree is just a triple line, and a fan tree is actually functioning as.

B

A

Add this trump line, then it will track on the functions on the time needed for implementation and when it will be implemented so.

B

A

Can add this coke you as a scheduling at the beginning of the function, and then we can achieve this.

D

Progress: what is it if.

A

You're following the given message and add Q at the beginning of the function. That means a lot of workload for the memory and also.

A

That means additional several millions of functions that means lot of overhead, so based on our calculations, it's going to affect up to 30% of the performance, so we alter our path, which shows other solutions. For example,.

B

A

Alter the other orders to notice on reboot, so we minimize the impact on the performance, so this.

B

Is what we do so.

A

If we replace their command from you to notice, then this is what happened.

A

For example, this is schedule function.

B

A

We're doing the counter a compelling or counter assembly koku has been replaced with knobs and if we want to start the reboot.

B

A

That's the only thing we need to do this, the only change we won't want to make and if we said I have trace here and if you want to do the schedule for the function and we need to track it down and.

B

A

Pretty similar with the function, filter and later we're gonna go through some more information about a kernel, and this is the only two commands when we need to give so as to open.

B

A

This is the dynamic, so this is the principle for the dynamic back trace and how can we achieve the online patching for kernel using this F trace, so we have to.

B

A

B

A

Actress, entry and the other life patching solutions, for example, k-swiss and okay graph, so there are pretty or they're pretty similar. So first we.

B

A

To Cape match.

B

A

Doing the patching we.

B

B

A

Scheduling in stop machine.

B

A

Are using the stop machine, canada in linux,.

B

A

And we are setting limitations for their version of other cpus, only the executor allowed to operate, and we are also doing interrupt for security checking.

B

A

There is a conflict between the existing content and life patching we need to exit otherwise Thursday. There might be a panic in the system or a crash in the system, so the life patching starts suitable for all platforms. For example, if some of the platforms are using scheduling, then obviously it's not suitable for live matching.

B

A

Using the dynamic F trace mechanism.

A

We need to see how it's functioning.

B

A

At the entries of each function, we add this flag, which is Kaku, Kaku, essentially scheduling and the logic behind it is that we save some of the stack information at the entry of the function and then have a registered handler handler change. The location of the.

B

Function if we.

A

Used to have learned they will therapy run the new function, not the old ones. So we are basically replacing the out function with the new functions and then.

B

A

Is how we do the patching between all the new versions of functions.

B

A

Before patching.

B

A

We need to do a scheduling for F entry, then F trace.

A

We need the code and save all the relevant information and we need to get a registry handler to replace the location, the address and return to the new function.

A

So this is how the patch is functioning after the patching process.

B

A

Now, let's talk about security check.

B

A

We have to guarantee.

B

A

B

A

Wrong address contained.

B

A

The principle is focused on different processes, different thread, and we ensure none of the to be patched. Functions are on the stack of any tasks and we have to guarantee the function. It's code right start machine context and we have to walk through all thread and check all functions on stacks and avoid any failure, and we also need to verify the back back trace address on the stack, for example,.

B

A

You run this thread in function, one and replace function, 2 in with a new version.

A

A

First scheduled function, 2 in the function, 3.

A

And pass the security check at all.

A

We have two kids.

B

A

Do it through dynamic passion process.

A

We have three modules.

B

A

Basically, we are altering all these codes in the patching into life. Patching.

B

A

All the patched codes will be.

C

A

D

A

And differentiated the functions and it's patching and we will produce modules of the patching and the core team in accordance with.

E

Information about some functions, it includes the replacement functions when.

E

By patching is active, then we were just scheduled as new functions and also the commercial core module. They means the point okay. Well, it's just provide a service interface for the patch module to register new functions for replacement.

E

John the new functions either want to be converted to so we can achieve all my replacement.

E

B

E

E

Is to generate a matching first, we want to compile the original kernel code. It were market with this function, sections and.

E

It will rebuild a dream clinic to attach the kernel and it watch for.

D

Change objects, objects.

E

Like what kind of function has been replaced and what kind of function is being added, it is so and to put aside and to link old output object and in generate the patch mode.

B

E

Is the clip out how it can generate the life patching? This is the whole process, so to just a review. The key page builder build a patch module under the patch.

E

It can generate a kibosh test, the module- if you want to apply this like patching, we can refer to this one and.

E

It was, it is today able in a boat, so, firstly, we need to compile the origin of SRC, and then we have this output objects and we do the comparison and if there's some abnormal ones, that we were abnormal functions, we will want to put this abnormal ones in this object.

E

Output objects and for the kernel modules we put this assistant to be able in a boat.

E

So I still ensure that my passion is enabled I like.

C

E

For this demo question here, I like to share with you.

B

E

This create a sub site, for we are generating this sub system.

E

There is no space left on device, the error, but we have this community patches and.

B

E

E

With cuba, creating containers and destroying containers there's a buck.

E

The technology it's.

B

Not decreasing.

E

A

Quickly reaches the limit.

E

The 0 y 593 limit, if we want to create, then there.

B

Is the kind of pot.

E

Here, so if we want to apply for our car, no, it's not practical practical.

E

If we want to apply for.

B

E

A lot of notes and if you need a play date, all the notes there are life cycle can be very long. So we encourage to use the my.

D

Passion technology and.

E

Any concerns the song.

B

E

In the data structure, we can use Copic macro and the shadow variable function to avoid that this change in a data structure.

B

E

I can do it demo for you to this demo, for you.

E

So I for this passion, I disabled, it I.

E

Can monitor the state group? These are the resources for the C group and.

B

E

Clip producing the containers.

E

E

Remember a secret was resources, I have created like over 65,000 and every time I created. The containers. I will always exert very quickly exit quickly. So in theory I'm not reaching this limit of all 65,000, then and 300.

B

E

Why am I running this containers?

E

The building of this container will fail. We can see that number is increasing when it reaches its limits and if I still keep building these two containers.

E

The container will be disabled.

E

In the beginning, you can see we have successfully built container.

B

E

Sir, my passion is disabled, I.

E

Can't build over thousands of them, but it has not exceeded this limit, but it failed and then you can see we can realized with this deck.

E

With the light passing.

E

We can start this disabled like like patching once we started it, you can see. The container is the being built successfully.

A

E

It's actually your life patching, it's the one renovated and we can do the online repair. We don't need to reboot the system.

E

If we just remove this location, then you can see the task would fail instantly.

C

This presentation.

B

Today is not my demo.

E

My team can beat Nevada.

B

E

Stop some issues.

E

I chase, we should notify. Initial functions are not supported.

E

Function, detection and it doesn't support the detection.

E

Initial functions.

E

When the car is enabled, they were being loaded, but they won't be used for later was.

E

Purchased which modify functions, amazing adventure who are not supported.

B

E

80% of all civilians, the currently supported.

B

E

Is very complicated, it's not supported and when you are building this.

B

E

E

Operations, they have higher requirements of financing into a greater risk, and we should take into consideration of this. Thank you.

E

I'm a question: if there is a data structure, changes in the kernel, can you just repeat the how you build.

B

E

App and all the way the structure.

B

E

B

E

Use the shadow variant to access this address, it's kind of the amount of together as the keypad. You can provide this navigation.

E

It is very core structure.

E

B

E

B

E

If we add the LP ID, it's also acceptable.

E

E

Have a question for.

A

This live patchy, so.

E

We can realize the function level variants, the level. How can we do the decoupling over it? If we did it.

B

Depend on the box.

E

B

Should be careful with it.

E

Because if you use like patching meter across the deadlock.

B

E

Need to analyze large venous function.

B

E

You cannot use them like patchy. So can we do the nemesis manually without the tool to perform this analysis? Yes, we can adjust the charger annually manually. Yes,.

E

Intervention, it is a very complicated. You know it's very difficult, yes, I mean.

B

B

E

It's also okay, but.

B

E

It's very complicated for some complicated deadlocks. I think we still need to have human intervention.

B

E

The crashing call analysis. Thank you.

B

E

B

E

E

D

E

To know if we have some tools to automate this process to make it more convenient for the light patching.

B

E

B

E

Not sure you have this tool, the key patch. Actually we do the packaging. If the patching is the powder, then we want to try it. Yes, that's the core technology, but in terms of the OEM, you need to do another packaging.

B

Yourself, yeah.

E

If the patching fails, we need to do another packaging, but for batching I think you should use the RPM package.

E