Cloud Native Computing Foundation ServiceMeshCon 2019 (San Diego), 17 Dec 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Collaboration is Important for Admission Webhooks in Service Meshes - Alejandro Pedraza, Buoyant

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

Collaboration is Important for Admission Webhooks in Service Meshes - Alejandro Pedraza, Buoyant

Admission controllers provide a way to tap into Kubernetes’ object persistence workflow by modifying manifests. The Kubernetes API server provides a set of default admission controllers, but users can also provide their own through webhooks. This is a common pattern used in service meshes to inject a sidecar proxy. Ideally, in-tree admission controllers and webhooks should be able to cooperate, in the sense that mutations made by some controller should be able to be taken into account by other controllers regardless of the invocation sequence. Up until 1.15 this cooperation has had its problems. Now, through a clever reinvocation mechanism this has been addressed. Alejandro Pedraza, software engineer at Buoyant, will give an overview of admission webhooks reinvocation policy and what it looks like now to operate in the particular case of the sidecar proxy provided by Linkerd.

A

Okay, hi everyone. My name is Alejandra pedasi I work for buoyant and I'm here. To tell you why collaboration is important for a mission webhooks in service meshes a mission web hooks are great, they can be used to add validation, apply policy or somehow enhance the resource being persisted in kubernetes in service meshes like linker d. We use them as the default mechanism to inject our sidecar Network proxy, which results in great user experience, because services are created, they get added automatically into the mesh without the need of any manual intervention.

A

A mission web groups have had. However, there are caveats that are described in this presentation, but I'll also let you know how their support guide improved as they went GA earlier this year. Hopefully these will shed some light on a couple of edge cases that people might fall into and using service meshes.

A

One thing we do every day as kubernetes practitioners, a million tanks, is calling coop control apply what happens under the hood, when we do that. This is the workflow that QAPI server follows every time a request is sent after a couple of preliminary steps.

A

The mutating admission phase starts, which triggers a series of plugins in sequence, where each one might mutate somehow the resource, then we go through schema, validation and then to validation, admission, which also triggers a series of plugins this time in parallel, some of the same plugins that got triggered during the mutating phase might also be triggered using during the validation phase, but naturally only their validation. Logic gets called this time if the validation goes through, the resource gets persisted into add city and the request is returned successfully.

A

The point that concerns us today is mutating a mission webhook, which runs the custom controllers that allow us to tap into and expand this workflow. You may add, as many admission web hooks as you want, you need to be careful how they would interact with one another.

A

Now let me illustrate how linker D uses a mutating emission webhook to inject this sidecar Network proxy and some of the challenges we've had in doing so. Next to the sidecar injector I'm, going to use lynnie Granger, which is a mutating admission plugin provided by kubernetes. That will allow us to add a memory limit into each container that doesn't have it and resource quota, which is a validating admission, plugin also provided by kubernetes. That will check that that memory limit has indeed been added into all the containers.

A

So let's say we just want to persist a very simple part. With one container when the mutating admission fall starts, it calls Limit Ranger, which adds the memory limit to that container, and then the mutating emission webhook triggers the sidecar injector, which just adds the sidecar container, and then the resource quota validation is triggered in kubernetes versions. Previous to 1:15. This validation will fail because the limit ranger didn't get a chance to affect the sidecar because it was added afterwards. 1:15 is a version that we still need to support.

A

So in the particular case of limit ranger linker D provides a series of flags and defaults so that the sidecar injector can do the job of the limit. Ranger plugin. Another very interesting case is part security policy, where the workaround consists on capturing the effect that the plugin had on the main container and apply the same effect into the sidecar as it gets injected. But this can only take us so far. Thankfully, after kubernetes 1:15, whenever a mutation is detected during this workflow, the admission work form runs the second time.

A

So we go back to the sidecar injection step. The sidecar gets injected that is detected as a mutation, and then we go back to women Ranger, which runs a second time, and this time it has the opportunity to add the memory limit into the side bar and this time the resource quota validation goes through.

A

It's important to note that during this second run, the sidecar ejector didn't get triggered because by default only the default plugins provided by kubernetes do participate during the revocation run. Now, let me talk, let me take a small detour and talk about a limitation that sidecar proxies have related to the pods life cycle. Whenever a pod that has been injected with the sidecar proxy shuts down, it is very likely that the proxy finishes before the main container and if the main container requires hitting the network during its shutdown process, this is going to fail.

A

Kubernetes doesn't provide currently a clean way to handle this, although very likely a solution is coming in. 1:18 didn't quite make it to 1:17 last week, where you'll be able to flag a container as being a sidecar, in which case the shutdown process of the sidecar won't start until the main container has finished. In the meantime, we can provide workarounds, for example, using a pre stop life cycle hook added into the sidecar. That will allow us to add an arbitrary number of seconds to wait for the sidecar before it should.

A

It starts its shutdown process so as to give enough time for the main container to finish its business. Now, let me say segue back into our main topic by implementing the addition of that life cycle hook using a new mutating admission web hook. That will sit next to linker this sidecar container and sidecar injector, and we will see how they interact with one another. So I have here a kubernetes cluster version 115 that already has link ready installed and this life cycle who I just mentioned, is already there.

A

So, let's try persisting a very simple pod that just contains one nginx container. The only special thing here is this annotation that will trigger the linker D sidecar injector. So let's apply it.

A

Okay and let's see how the pot ends ends up. Looking like I'm going to use here, a oops.

A

When I you see Jake, you filter that we all allow us to focus on the lifecycle property of the linker, a proxy okay. So this means that the linker D sidecar was indeed injected, but the lifecycle hook didn't get added. This is a similar situation to what we just saw with the limit arranger example where the this new lifecycle webhook got added, sorry and got executed first and then the the sidecar proxy got added, so the first webhook didn't get a chance to act on the second one on the effects of the second one.

A

We are on kubernetes 115, so a relocation was actually triggered, but, like it said, these custom web hooks don't participate by default in the reading, vocation run, but that's easily fixable. We just edit the the configuration of this lifecycle web hook, there's a new property towards the bottom called reinvasion policy. So if we change this default value from never to even need it, these will just flag it so that it does participate during the invocation run.

A

Now, let's try the leading the nginx pod so that it gets regenerated by the deployment.

A

A

Okay, so this time the lifecycle hood got added so again. What happened here is during the first run of the animation workflow the proxy got injected and during the second run, the life cycle got added into it.

A

Okay, as a summary, so after kubernetes 115, we should no longer see weird behaviors when trying to integrate our custom web hooks with the rest of the kubernetes controllers. We should leverage this newly invocation policy properly, that God introduced into the web hooks configs and after 118 the sidecars. The sidecar story should be much more complete. Thank you very much. That's all I have for you today.