Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2019 (San Diego), 22 Nov 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Admission Webhooks: Configuration and Debugging Best Practices - Haowei Cai, Google

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

Admission Webhooks: Configuration and Debugging Best Practices - Haowei Cai, Google

Admission (mutating and validating) webhooks have become popular mechanisms for extending Kubernetes API request admission. The admission webhook API is graduating GA in Kubernetes 1.16, where new features are introduced and debuggability improvements are made. In this talk, the audience will learn common pitfalls in admission webhook development, best practices in webhook configuration, and how to identify and debug failures caused by misconfigured or buggy admission webhooks.

https://sched.co/UaVt

A

Hello: everyone. Today we are going to talk about the configuration and debugging best practice for admission web hooks. My name is Howard hi I'm, a software engineer at Google cloud and I am a member of the kinetic API machinery.

A

Here is the agenda of what we will cover today. I will walk you through what you are the admission approach and why do I want to use them? What are the best practice for configuring, my admission webhooks and what are the existing debugging methods that helps me to park? My admission webhooks in the end I will give a short demo about debugging misconfigured app hook.

A

So what are admission web hooks? First of all, we need to understand what the admission procedure in kubernetes the kubernetes api server rounds with some admission controllers by default. A mission controller is basically a piece of code that intercepts requests to cube API server prior to proceed to the if resistance of the object, but after the request was authenticated and authorized.

A

If we take a closer look, the admission controllers can be categorized into two types: mutating admission controllers and variability and mission controllers. Mutating controllers may modify the object, they admit and validation controllers may not if any of the controllers in either of the face rejects the request. For example, let's see the resource quota controller rejected a request because the quota was exceeded, the entire request will get rejected and an arrow will be returned to the end-user immediately other than using the building and mission controllers.

A

Criminal's allows you to extend the admission control your customer with admission webhooks. You can configure the true building and mission controllers called mill hitting admission web hooks and validating admission web hooks to talk to your own web hook. Back-End mutating web hooks are calling sequence while the that evasion web postcard caught in parallel the. Why do I want you? You saved admission webhooks. This is basically like why we have the Beauty admission controllers.

A

The admission web folks can help with enforcing security policy, for example, only allow pulling images from certain registry that are known to the enterprise and which web book can help with carving the adherence to certain practice like having good labels annotations for object, animation, webhook can help with automating and validating the configuration of the objects in the cluster like detecting and the fixing images deployed without some semantic tags.

A

Okay, so we understand why we may want to use a omission of hoax. Let's see how to configure a mesh webhooks here is a example. Validation web for configuration I won't go through the fields, all the fields in detail. Basically, this is a web configuration called pod policy at example.com.

A

The configuration has one web hook with the rule that trigger some area, name: space, pod creation, requests the client configured config, chelsea api server, how to talk to the web hook. Back-End the web hook doesn't have any side effect and it has a five second timeout for the api server jihad web hook. There are some other optional fields for you to customize your local configuration, which are not sure here. I will explain them more in more detail when we come across them in the best practice.

A

Okay, so, let's see some of the configuration as practice while you show up hooks here is a list of since we will talk about, let's walk through them, one by one item Potence, it is natural to expect the cubic answer will invoke your admission. Your mail hitting web work once if a to match the emission webhook configuration, but do you know that the customer may call your clicking web hook twice for the same request?

A

The reason for that is basically ordering mutating webhook is hard. Imagine you have to shoe admission to militating Web books. The first web hooks is called inject sidecar, which injects a static a container to your pod, for example. Here I'm creating a part, the web hook append a side hug container to the list of containers in the part ii web poke is called always pro image. It enforced a security policy in both eternity scenario by setting the image Pro quality, to be always for all the containers in a pod.

A

Now we have two containers in pod: both has their image or policy set to be always, which is great.

A

Imagine if the API server orders C to web hooks differently. It has a reversed call the always go image web hook, which sets the image Pro policy to be always for my country, my container, then the API server called the inject sidecar container web hook, which attends recycle container to the pod. Now the party ends up with a container without the image properties that which violates our security policy to.

A

Solve this problem, which clearly plasterer does is basically a best-effort re invocation to make sure everyone every web book in the latest state of an object, for example, for this instance, if a server noticed that the object was mutated by the inject sidecar web hook, so it invokes the always Pro image will hook again, which at this time sets the image for policy for the sidecar container and we come to the desired state of our web hooks.

A

Now we understand that previous error make halt by mutating webhook twice. The best practice is for my mutation. Well, folks to the idempotent and mine, an idempotent military web hook is able to successfully process, and now jacket has already admitted and moo potentially mutated, the always Pony meat web. Pork is a good example of an item order his web hook because applying the web folk song. It's reason I mean it's output, won't change the result.

A

To give you an example of a now I'd the importance of a poke. Imagine we still have a inject site calculate web hook, but this time the web hook append the cycle continue to the continue list, with name service by the current timestamp.

A

This Bible is not idempotent, because if you applied again on the article of itself, you end up with two containers site heart containers with slightly different name. This is probably not what we wanted.

A

Another best practice is intercepting all versions of an object. We know that kubernetes support multiple API versions, for example, cubic a server allows creating and updating deployments by extension, V 1 beta 1 ab c on beta 1 p.m. we won't be able to absolutely 1. Ap is if a record is configured to only intercept deployment request seen apps few one. It will miss the request in the other versions, so it's recommended to set the match policy to be equivalent, which this configuration ensures that they were web.

A

Work will get request that on hundred equivalent from api server, and we will also ensure, if a future, it has a wrap grade, happens. The ideas and the class are introduced, a new version for the deployment applied, for example. Let's say we have a absolute to it, but the webhook will also get the API request.

A

Availability, it is recommended that visual approach should evaluate API requests as soon as possible, because every web poking location latency as to the total API request latency, we recommend you set a small timeout for your web book.

A

This example sets a two-second timeout for my web hook. If the web hosts slow and didn't evaluate the request in 3 seconds, the plasterer will either reject tape, API request or ignore the web hook. Invocation failure and I'll admit the requests based on the federal policy of the web hook.

A

Guarantee the final state of the object is seen since we know that another web book that is called after your military macaque may change the object. It is a good practice to have a valuation web hook that ensures your beautician is made.

A

If you are say you are using the auto image web hook to enforce the security policy, it is recommended that you add and evaluate always Pro image by poking, which make sure the image Pro policy is set for all the containers in your path side-effects to some admission web hook, maybe mutates related resource as part of the request processing. For example, the resource quote on a resource called Howard hook, may increment the quota usage when it admits it can request that consumes the quota.

A

The best practice is to avoid satified effective possible if a celebrex is required for admission web hook make sure to have a reconsideration McCadden, for example, a controller that decrease the quota usage in case the request didn't mix through because it was rejected by something Allison get admission chain and also don't trigger the side effects. If a admission review has a dry run, sad and the web hook should the web whole configuration should specify the its side effect to be non-viral.

A

Avoiding operating on the crib system namespace, this is basically for safety purpose. The cube system, namespace hosts system, critical components, for example, the cubic a server itself, has some host art hooks that will straps the internal controllers. If your admission web poke rejects the request in cube system, APIs namespace, it may cause the cube API server to be not healthy, also, the namespace host objects or control plane components, for example, the service account for all the entrepreneur components and the pod like cube DNS.

A

If your editor up hook is not intend to modify the cube system, creates control. Plane. Behavior is recommended that you exclude the cube system namespace using a namespace lecture or a object. Selector, for example. Here is a object selector that only matches the namespace only matches to requesting a namespace that doesn't have a long time round level, with value 0 or 1. Also excluding cube system.

A

Namespace makes post API server in the web hooks back and more efficiency because, as we mentioned before, every web hook, invocation latency builds up the entire API request, latency now by excluding the name cubed cube system namespace the web. For communication is scoped to only what is needed.

A

Now, we've look at some of the configuration best practice to admission unhooks how to read about our commission webhooks before looking at some of the existing debugging methods. Let's first take a look at the ways that admission were provoking vocation made Val.

A

This is the table of types of of weblog failure or rejection. I'd enter sink them in his really different categories. The first category is a bad action, actually a valid web of rejection. For example, the raddest api server sends a admission review through the web hook back end and to back-end determines the request is invalid and responded with a 400 or 4/3. In this case, the youkai server surfacy responded HP codes to the client so, for example, the end user, who are generally send the request work at a forestry for bidden.

A

Another valid web for the rejection is actually a 500 internal error to the web hook. Backhand. If the web hook back end hits an internal error but is still able to compose a mission response, it will sense the plasterer and 500 internal error. In this case the oppressor will also service the arrow to the end user. So the answer rivers will get a 500 and specifying the web hook denied the request.

A

The second category of web hock failure is our calling web hook. For example, if the web hope bank back-end is too slow and timed out or the API server just felt to inebriation action to talk to the web hook, backend or if the web hook sent badly admission response, but the UK server felt to parse it because it the response was malformed. In this case the API server will return to the end user, a 500 internal error and a reason explain the error.

A

The last category is a actual API server internal error. This should really or never happen in reality, but I'm. Just listening to it here in case that we have a programming error in kubernetes that castle panic. In this case, the API server will also return a 500 to the end user.

A

To just make the table a little bit more compact in the weapon configuration for each webhook. It can configure a failure policy in v1, API valor is default to be reject, but you can contribute to be ignore when you start webhook. In that case, if an arrow column webhook occurred, for example, timeout connection failure welcome to response the offender will ignore the Republic vocation, failure and to return no arrow to the end user.

A

So you may notice that all three categories may end up with a 500 response to the end user for a single webhook failure. You may be able to tell what actually happened from reading the reason, indeed, with return the error. But imagine you are monitoring the auto traffic in the cluster and you see a search of 500 when you are developing your webhook. How would you be able to tell what kind of failure happened to what web hooks? We have some metrics for. Also the caption eclipse, API server serves promises metrics on it.

A

/ metrics endpoint, which reports the status about itself. Among all metrics reported, there are two metrics are related to admission: webhook invocations wise, called a mission webhook admission duration, which is a histogram metrics that basically tells you how long a mission web hope invocation typically take.

A

Another metrics called a mission web Hoku rejection town, which is a control matrix that helps you how many rejection or ad-hoc failure haven't grouped by the webhook name. The API request, operation type, the type of see web hook itself, so rather picking on mutating the alt tag. Basically, the three categories we mentioned before and the rejection code, which is if a rapport did evaluate the HTTP that is called rejection. It will be reported here so just give you a quick example of how the matrix may look like here's.

A

A metrics output I got from running some basic kubernetes III test. Basically, it shows you that there were three error. Column webhook occurs from the three line with a highlight in the bottom. There are some webhook rejection with no arrow, so that's a valid web protocol rejection and you can tell from the test codes that the web hook, which actually read request with 400.

A

Sometimes you may want to keep up some num obvious web hook, behavior other than basic monitoring the cluster. For example, you can tell that your web hook back and worse processing the mission review and mutating the object, but it seems to be ignored from if that's our point of view and is the result. If object, we never got mutated or you find that your web for back-end was processing the review and the Asura did mutate the object, but it was in a different way that you really expected.

A

Could it be a back in your web hope back end? Could it be about in the UK server or could be some other really web hooks that change the object after your mutating web hook?

A

We have auto flocking to answer this kind of question based on the documentation. Kubernetes audit logging helps it cluster admin answer the following questions: what happened when did it happen, who initiated it, etc?

A

To just give you a quick view of how rubinettes audit logging works. Here is a audio policy that sets the kubernetes observer to log path, change at request, response level and log pod, slash log and path, slash left status, tab, resource requests at metadata level and I just put some random order. Log record from my cluster here is a record. The record is actually called an event, its own defined under an audit that case dot, IO v1, API version.

A

This is a metadata level audit record, so you won't be able to see what was actually in the request. But basically this tell us someone was creating a kubernetes band in the cube system. Namespace the Creator was someone called cube scheduler, and we have some other metadata level information, for example, the time stamps.

A

Now, let's take a look at how Krueger has several leverage, the add, walk audit logging to record activities related to mutating WebWorks?

A

Basically, there's a field in the audit record called annotation, which you will imagine, is basically a map of key value pairs that allows plug-ins in Cuba cancer, for example, authentication authorization mission to customize the audit records with their own information that we want to record the admission. Webhooks use the two types of annotation Kish to record mutating web for events. I will go through them, the definition of them in more detail, but the prefix, the an occasional basically differs in the prefix one starts with mutation and one starts with patch.

A

Here's a example of a nutrition audit event that has this annotation you can see. There is a key called mutation that were poked. Are the mission Dell KACE that I? Oh, that's wrong! One index to this is logging, an activity religiously imitating what poke.

A

So, let's take a look at how this annotation are defined. The first annotation key records a mutating webhook currents. The key starts with the world mutation, followed by webhook definition of kissed on Io, remember in index number, for example, from zero and index 1. The numbers are zero indexed this. So this number means this is a a activity record for a mutiny, webhooks that happens in the first web hooking vocation round, and it was the second why poke in the list of beauty web hooks?

A

This type of annotation is recorded if the audit level is said to be metadata or higher.

A

The value of the annotation is a JSON block with three L's, the configuration the web configuration name, the web hook name and the mutated, which is boolean that house, if the hatch sent by the mutating madhawk actually changed the object or not, the outer annotation key starts with the word patch, followed by the same thing and just give you another example round 1 index 3 means this is a but hook activity that happens in the second one round, which is the real invocation round, and this is the force built-in web poking the between web hook list.

A

This annotation is recorded if the other level in the cluster is said to be request or higher. The value of the annotation is a JSON JSON blob of with for back for fields, the web hook, configuration name, the web hook name the past hype, which is JSON patch in this case, and the actual jism hatched and from the mutiny Red Hook. As you can see, this mutiny web Polk, who was trying to set the value under slash latest national team's h-2b.

A

Yes, just to give you a sample of how this annotation would look like this is the actual annotation actual other events that I captured from my cluster. As you can see, there are a lot of annotation with this prefix. This is from a also from a quarantine three tests. So there were a lot of web folks that trying to do mutation- and you can see both the mutation and the patch annotation- will record it.

A

That's the but not list you can always debug your own web book back in the previous methods like metrics and dialog helps huge. If you are suspecting something was wrong on the cube, API server side in a resultant, it could be apply. Several feels more like a black box too, but you could always have good logging in the admission by Polk backhand that you deployed, so you can have a better view of say what admission review the backhand got from the observer and what a mission response it responded to the API server.

A

Okay, have you give a quick demo about how to debug admission a misconfigured initial approach using the audit log? The demo is pretty single, so I'll just capture some screenshot say I was developing. Let me show a polemic faster and I was happy off sudden. The castle breaks I cannot apply apart anymore, the if hazard just responded with the part. My part is invalid. It tells me that there is a duplicate value. My side, car in the container list, which so basically this is a duplicate name in the list of container.

A

The response is a valid widget kubernetes rejection because you create a server, tries to make sure in the path. All the container names are unique. So I recognize this. My sidecar name. This is, should be injected by my inject sidecar webhook. So I just take a look at which are the mutant web hooks. They are in the classroom. I found that the injected sidecar up hook has been working for a week, but it suddenly breaks after I deployed the other web hook.

A

Luckily, I have the audit log enabled for my local cluster, so here I configure it to log activity at request level and I, basically try to grab the latest request to the part creation endpoint since I know there is no one else doing pod creation. My culture as I was developing myself. I just grabbed the latest one and try to see what was there. The actual audit event is pretty long, so I split into some pieces first, this is giving me the metadata of this event.

A

Basically, you would say: I was creating the pod using cube Cairo. The parties called my pod. This is are all expected and nature. It shows the API server responded, 8422, which is also alleged rejection from a customer point of view.

A

Later on, I see the original request sent from queue Cairo, which only includes a container called mine container. So there is no injected site container, yet these are still all expected request from Cuba kado. But after that we see there are some annotation that recorded by the cubic a server. So here we can see that there are three web hooks, mutated, the object.

A

If you look at the mutation web hook admission round 0 index 0, we can see my jack side cover hook tried to inject today that continues with part, and it was it successfully related web hook and after that at position. 1 there said the other web hook inject a mutated. The part again and later on the castle tried to revoke revoke the inject psychical web cook and the Jack's Itokawa hook, mutated the object for the second time, and we can also see what the actual patch is from the patch annotation.

A

Basically, from this point I see, the problem is that my inject sidecar web hook was not I, don't Holden. It was pending a sidecar container to the pod, regardless of if there's a set a car container in the pot. Already so I turned to fix you I poke back and buy first detect. If there is a cycle container already new pod, if not injected car, otherwise just skipped the admission response and send a empty patch. So now my wife Jorge is idempotent and everything is happy.

A

Ok, ok, so the key takeaways from this session is that configure additional folks following best practice. We also have an occupation described in this past practice on the website and use the metrics and other logging to monitor and debug your web hooks. Thank.

A

B

So much away any questions in the audience. There's an open mic.

C

Could you elaborate a little bit on what is the system requirement in order to configure webbook and the alien situation control.

A

Admission webhook was brought to be one in 1/16 and also the visual projection count and the audit logging are introduced in 1/16 as well, but.

C

My question is more specific: on I'm installing a new cluster from scratch and I'd like to start configuring it. So what would that consist of.

A

You need to so. The question was: how do you bring up a new schwab? Poke class were starting from scratch. You basically need to deploy a web poke back end. It can be deployed in a party can be a URL that from your own server back-end and you need to deploy admission web for configuration either in for the mutating one for the battery. If you want should help the a pleasure to be able to talk to your web hook again and also there are some three circuits that we need.

A

You think you need to enable the built-in admission web television controllers, which are enabled by default to so those to the machine control is called medium. Admission were cocked and very big machine rocks.

C

And from in our back perspective, do you need to have system cluster and mean role or something like that or yes,.

A

Enabling admission controllers, which, even though they are able, by default for those two controllers at a time, this action is a quasi fermi-level action, because it's at the time that we bring up the cube advisor, creating the admission configuration. The mission of all configurations is more based on the our back row, but it's also tend to be at cost in animation mo so.

C

So, is it fair to assume that this use case is not or main developer but more like cluster administration, that trying to provide infrastructure, yeah.

A

It you can only deploy in the shrub book if the customer may grant you the are back rule to be able to create those configuration. Thank you.

B

One more short question who has a short question.

D

Thank you. Is it safe to run a web hook on the cluster that it's going to apply to, or is there a danger that you'll lock yourself out? If you get that wrong.

A

There are potential risk that, if you're a poke yeah- yes, it's it's okay to deploy a web hook, backend in your classroom by pod, but there are potential risk that you can deadlock your web hook. If you do something wrong like say, if your web hook is admitting how to relate to the deployment related requests- and you found a bug in itself and you are trying to update the backend.

A

Basically, you are trying to update the deployment of your web of action, but it's already up there and the power caused it to reject the hot or deforming updates request to itself. Then you end up with a deadlock, so it's recommended that you use namespace lecture or objects lecture to try to make your web hook to not admit requests regarding itself to a point. This kind of deadlock.

B

Once again, thank you so much wrong way and remember to rate the session at Shechem. Thank you so much.