Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2020 - Virtual, 4 Dec 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Into the Deep Waters of API Machinery - Federico Bongiovanni & Daniel Smith, Google, & David Eads

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2021 Virtual from May 4–7, 2021. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Into the Deep Waters of API Machinery - Federico Bongiovanni & Daniel Smith, Google, & David Eads, Stefan Schimanski, Red Hat

We'll cover 3 common icebergs with lightning talks: 1) My namespace won't delete, help! Why does Kubernetes sometimes refuse to delete a namespace? How to diagnose and resolve the root cause. 2) What is REST mappinp? Ever wonder how you're supposed to go from singular to plural, why we did it, and how to deal with it? What happens when two CRDs choose the same name? 3) Taking out the trash: Garbage Collection in Kubernetes. We'll explain how OwnerRefs, BlockOwnerDeletion, Foreground, Background, and GracePeriod work.

https://sched.co/ekGp

A

Hello, kubernetes community, welcome to our sig api machinery deep dive session. My name is federico von giovanni and I am the co-chair of seek api machinery. Our agenda today will have three topics and a small bonus information. At the end, our first topic is going to be namespace termination and daniel smith from google is going to walk us through it using his signature, handmade presentation style.

A

Our second topic is going to be rest. Mappings and stefan shimansky from red card is going to unveil all the secrets on how it works and how to deal with special situations in our third topic, david is from red hat will explain for us the mysteries of garbage collection and how does it work in all the special cases?

A

Finally, I will give a farewell with some quick info about the sig and our meetings, so without any further delay, let's get to it and go to the deep waters of sig api machinery.

B

Hi everyone today, I'd like to tell you a story: a story is called 60807, I'm daniel smith. I co-tl the api machinery sig we're working on kubernetes since before was open source.

B

So I'm wobbling up on github original lava lamp on twitter.

B

I work for google software engineer, I'm not speaking on behalf of google other than they support my presence in open source.

B

um I wrote daniel drawing my slides today turns out. I can't draw and talk at the same time so we're doing this in two steps.

B

Story begins with a user.

B

You won't be surprised that there's also a kubernetes cluster in this story and every cluster needs an api server.

B

And of concern particular concern in our story. Today is the namespace which I'm going to draw in a big fat line.

B

B

To give this namespace some sort of name.

B

We'll call it s2020.

B

Because this user would like to delete this namespace.

B

So here's the action user says delete api server, says sure, that's the technical http response code. These are posts to recreate, maybe ir server says no, it still exists. There's a conflict.

B

Api server is a villain.

B

How could this happen to us? This makes the user sad and angry.

B

How could we do this to the user.

B

Let's talk a little bit about how resources in kubernetes are deleted, all objects and kubernetes are deleted the same way.

B

So, although we're talking about a namespace right now, there's some background information about deletion. That might be useful.

B

As with many of much of what I'm saying, this is a simplification.

B

Because there's more to it for some particular objects, but, roughly speaking, all kubernetes objects have metadata and inside this metadata, there's two fields relevant to deletion. First, one is deletion timestamp.

B

If that has a non-empty value, then the deletion process has started for that resource and there's also a list of finalizers and the deletion is not final until all of those finalizers have been removed, slightly confusing, because the literal name of the finalizer that we care about today is kubernetes, and this is the finalizer that the that is removed when the namespace object is empty.

B

Okay, so until that is removed, the namespace is not deleted from the system. I'm leaving out some details about namespaces have special places. They keep their finalizers. Roughly speaking, all objects work like this.

B

So, let's look at our scenario again.

B

The component in kubernetes responsible for removing this finalizer is in controller manager. It's the namespace lifecycle, controller.

B

Its job is to watch namespaces that have begun. The deletion process ensure that they are empty, that all the resources inside them have been deleted and then remove that finalizer.

B

The space lifecycle controller is the villain now to list the resources inside of a namespace. You need to know which types of resources there are to be listed. The process of figuring that out is called discovery.

B

And at this point you might be asking yourself: how does discovery work?

B

Well, I'm about to tell you all api server. All api requests in kubernetes uh have a url path. This is this identifies what exactly you're, making the request about right and again, I'm this is a simplification, but for our purposes all api requests start with the word apis and the next thing that comes is a group, the api group.

B

The next thing is a version, and after that is the resource.

B

The resource is the thing that goes in the url path. The kind is the is the the the type right in a type system sense.

B

um So if you make an api request- and you just say apis, if you stop there, then you get a list of group versions that api server knows about.

B

And if you list the group, you get a list of versions that are inside that group right, a list of versions and if you list a group and a version, then api server will tell you which kinds are inside. That group version.

B

And if you say more than that, then you're making an actual api request, which will do you some good, like list objects or just you know something useful.

B

So how does the api server fulfill this contract to do that? We should talk a little bit about api server. Api server is actually three api servers in a trench code. The first one is the aggregator. The second one is the built-ins.

B

It serves pods and services and stuff, and the last one is the extensions api server more commonly known as crds, and if a request can't be served by the aggregator, it goes to the built-ins.

B

If that one doesn't want it, it goes to the extensions, and if that one doesn't want it, it gets a 404 which I've conveniently drawn off the bottom of the screen. For you.

B

Now the aggregator may know what to do with the request and it may be for a external api server. This means that that request must be proxied. It is not served from the same process as the kubernetes api server. The aggregator is the villain.

B

The canonical example is the metrics api server, which kubernetes ships by default. This means that there's various things that could go wrong. Maybe your network is not working um at least that link of your network, or maybe the metrics api server, is co-located with a process. That's hogging the node, or maybe it just has the wrong resource requirements.

B

In any case, the aggregator needs to get the kinds from the target api server right. Only the groups and versions are registered with the system.

B

So uh if that network, if that external api server can't be reached, then there's no way for the aggregator to tell you which kinds are in the group version that you're asking about the metrics api server is the villain.

B

That's kind of unfortunate, but it is also unavoidable because the extension api server is the only thing that knows what it serves. Aggregator cannot know that it would be very heavy-handed to require api authors to list that to pre-register that with the aggregator. It also wouldn't help the problem, because, even if you know what kinds, if you can't actually go out and list those things, then you also can't confirm that they're deleted.

B

If the lifecycle controller can't discover the the kinds that it should be deleting, it is never certain that the namespace is empty, and if it's not certain that the namespace is empty, then it can't remove that finalizer so whose job is it to make sure that this thing is working anyway?

B

I think it is the job of the system administrator, the cluster administrator rather- and I think the cluster administrator should be performing this job by using some monitoring.

B

The metrics api server is served by a regular kubernetes pod, so it should be possible to monitor it another way you can monitor the overall health and like not just the specific metrics api server, but if you just run the command cube control api resources once in a while it does this discovery process and it'll tell you if some group is not working and then you can look at the api service object for that group and figure out what component is not happy.

B

So thank you for listening to my talk and watching robo daniel draw these.

B

If you think that we could do better I'd encourage you to come help us improve our error messages or our design.

B

B

B

The system administrator is the villain.

C

Welcome to the second part of the seek api machinery deep live today. I want to talk about a concept which is pretty central in kubernetes, but it's not well known to many people and maybe, if they use client go and build some non-trivial controllers, they have read it, but they they might have question marks around that that mapping action is very simple, just say, cue, cuddle get pots in your terminal and get parts is a command which is generic, so generic means that it doesn't know anything about pots.

C

It has to work with any every resource that is available and known to the cluster and to make that uh happen, it has to query the discovery information of the cube api server. So it has to ask it: um what is the part and to do that? If you increase the velocity to at least six, you will see what it does. It can be slash api to get the versions of the legacy. Api is queries apis to get api groups and their versions, and then it continues in queries like more than 30 api group versions.

C

So api, slash, v1 api, slash, apps, search, b1 and 30 more and after doing that, it has all this information too to find out what the port is, and it sees a part this resource which shows up in the api, slash v1. So it's a legacy core api version, v1 resource and from this information. It also knows that it's namespace, so part is namespace resource and from this information it can construct the url to to actually list the ports in the system, and it knows its name space.

C

So it adds the namespaces default default is code, one the namespace. So it knows the url just by discovering information how to get supports, that's what it does in the types of kubernetes. There are kinds and resources kinds are the upper case, singular words which you find in manifest usually and the resources they are components of urls, so the group version and the resource name plus information of the scope, that's all needed to build a url and the mapping between those two worlds.

C

It's called rest mapping and that's our topic today: google resources and google kind. We call them fully qualified or complete if the group, the version and the name are provided. If the group is empty, it's a legal secret black for pots, we have the same for clients and we can also talk about partial resources and partial kinds. So in the first case the version is missing. So abstractica said it's a partial kind. V1 replica sets lower case. It's a partial resource.

C

A group is missing and if both information, those kinds of information are missing, it's also partial, so replica sets. The loan has. No group has no version and that's what we know. Everybody knows from qatar, and usually you do it on the command line. You can get replica sets it's a partial resource and the last mapper is used to to fill the missing information and to get a fully qualified resource, which then can be used to query.

C

The replica sets there's a syntax to couple syntax to say this is group apps, so dot apps tell us q carter and the group is given. You can even just use a prefix, and this is completed by the last mapper, so dot a is completed to x, v1 and, of course, you can have complete group version, resources.v1 dot apps.

C

If you try this prefix matching, if you have a version, it will fail, because the raspberry pi doesn't support that if you try dot v1 without a group, two cutter parsing doesn't work, so it's also rejected everything I talked about here works for singular and plural, so you can give a singular word here: a singular partial resource. It's completed to a complete pure resource.

C

Where are these members used? So we saw two cattle that we have server itself has some use of rest mappers. So there are some gc garbage collection, related admission plugin here, but the main consumer of those class mappers, especially discovery based ones, has controllers controller manager.

C

Horizontal port, auto scaling can work for all the resources that have a scale subresource. So this is a polymorphic or generic use case. That's why asthmatic is involved in port disruption. Budget is also involved, and the garbage collectors may be the most interesting case here and then we will spend several slides talking about the garbage collector and this uses. Mapping at this core um quota doesn't use last map being surprisingly, but it uses discovery. So it's not that surprising, because discovery and last nothing are deeply connected.

C

uh Quick, look on the respect, interface and go. There are three kinds: the yellow functions here. They map partial resources to kinds. They map the right ones, partial resources to complete resources for the qualified resources and the green ones. Take group kinds complete so who kind of cannot be admitted, complete group, clients and versions to rest mappings and last mappings.

C

If you look here, what ransomware is is actually the resource, the kind fully qualified and scope so basically, the last one is kind to resource and the green one is most interesting in the context of garbage collection, because garbage creation uses that owner perforances are group kinds, and so everything which we talk about here makes a difference or has influence on coverage collision so take. Let's take a look on the yellow and the right ones. Where we can have partial input, so partial input is completed if we already pass complete information, so fully qualified resources.

C

Of course, we get back fully fully qualified resources, again, not surprising what especially here, if you have all three components and they're not empty, there's no prefix matching. So if you have just a but you pass a version as well, you get an error.

C

This prefix matching only works if the version is submitted so yeah for the case reversion submitted, but the group is complete. Obviously it's it looks through its formation of group versions and finds replica sets here in this case mb1 and v1 beta1 of the apps group. If you just have the a is the prefix matching which, which you already have seen happen so absolute one replica set, is found and f v, one beta one replica sets. The raspberry also looks an authentication k or s.

C

I o and probably some other groups with a, but there is no replica set, so those are not returned by the last number. The last one, the group, the group is missing. It's filled in as expected and if group and version are missing, that's the usual case on the command line, then group and version are added. The order of the results depends on the preferred group version order. You saw that in the discovery information already and other than that resource names are ordered, alphabetically all right, so discovery and rest mappings are connected.

C

Let's take a quick look on discovery, information on the slash, api's endpoint. You get all api groups and save versions and the preferred versions on slash apis apps. You get basically the same, but just for the apps scoop. In both cases you don't get resources. So if you want to know about resources, you have to go to one level deeper, apis apps you want, and then you get everything about oil resources in the case, replica sets.

C

Lowercase prover is the name of the resource, singular name it's empty, so the rest method will fill in the lowercase kind. The scope of this is the namespace, so we have to add the namespace to the ul. The kind skeptica set uppercase as we expected the verbs are given. So we know what we can do with the resource. Short names are returned for crew cutters or aliases on the command line and cut categories. So you can list all resources and replica sets are among that all right. So keep in mind to get resources.

C

You have to have the third column as well from discovery info, so you can use the discovery, client and you can grab it with a caching layer. So there's a disk cache and a memcache discovery wrapper memcaches for controllers, because they are one long running and this cache is used by cubecutter, for example, and if you have a cached discovery client, you can pass that to the constructor for a discovery. Rest mapper and you get the last mapper. As you expect implementing the interface.

C

We have just seen, there's a reset method on the lastmapper, so you can evaluate the cache manually if you like. This also happens when you have a typo and a typo means the cache is the cache miss and the cache is invalidated. This means you get. Those 30 plus discovery calls again and again so, depending on your use case, if you have a controller, there's a risk of hot looping. So keep this in mind.

C

This might be important so where it's used for resources here, what I want to highlight is that the controller manager calls reset every 30 seconds, and this is a way it gets to know about new resources. So if you create a cid in the cluster, it takes max 30 seconds until garbage collection knows about the new cid.

C

We talked about that. We need this versioned discovery endpoint to get resources. The consequence is, if you have aggregated api servers in the system. You need this red arrow here to to know about the aggregated resources. If this breaks down for networking issues or because api server, redeploys or something like that discovery, clients and rest members won't see the resources in aggregated vi. Everyone.

C

This means that the discovery cloud will have an error with return error and the rest. Mapper will have incomplete information, and this might have consequences. So keep this in mind. The discovery client is kind of graceful, so it it returns. Errors directly, but it also gives you the partial information it was able to gather from the cluster so always expect a nominal first result, even though the error is also bundled and co-present whatever.

C

This means in your use case, the discovery rest mapper is very graceful, so it just continues with partial results and ignores errors, but obviously incomplete information might have consequences for your use case. So think about your controller and where you use the last mapper try to fail gracefully, because you have to expect errors, they happen and stay consistent in a way which matches your use case. So we saw a namespace controller which blocks rather than doing uh stupid things. It blocks the the work.

C

Maybe this is much more important to stay consistent like that than continuing working, gracefully garbage collection, david, we'll talk about it in a second, that's also important to cutter. You can just print a warning or something like that. That's completely fine, but depending on use cases, behavior must be different, and with that I pass over to david.

D

Now, let's move on to garbage collection of kubernetes api resources.

D

The first thing to consider is which binaries are involved and we have a cube api server, which does basic crud on the resource and tracks. State like is the resource present. Has it been removed or does it have a deletion time stamp set, but finalizers that prevent it from being removed? And then we have the cube controller manager, which runs a garbage, collector garbage, collector controller, and it looks for resources with absentee or missing owners and takes action on them and we'll get to the particular actions later on.

D

In some examples to describe the relationship between the resources we have owner references on children, children list their parents as an owner ref, you can have more than one and a namespace child can only refer to parents in the same name, space or in cluster scope, name spaces having the owner. References on the child allows permissions to be listed on the child and not on the parent, which makes it safer to express them via the api.

D

Looking at an owner reference. Here's an example in a resource manifest that shows a single owner reference. There is a section that refers to coordinates where to find the parent, in this case we're looking for a config map named i3 you'll notice. There is no namespace, so this means that it has to be in the same namespace or clusterscope.

D

We have a block model deletion field that only does something with foreground deletion of parents. It has no effect in the default case. There is a controller field which doesn't actually affect gc behavior at all. It's used by higher order logic to control ownership uh of items and then there's a euid. The uid refers to the uid of the parent and it's necessary to handle name, reuse of parents.

D

If you rapidly delete and recreate parents, we need to know if it's the same parent, in which case you need to be preserved or in a delete and recreate case. It's different and the owner ref is now invalid and the resource needs to be collected.

D

It does mean that you can't hard code them into manifests, which is a pain point, but we don't know of another way to solve the delete and recreate use case, so the mechanics of actual deletion. There is an option to delete in the background. This is the default case. It means that resources are removed immediately, assuming you have no finalizers and the garbage collection controller finds child resources that have no additional owners in the background and deletes them. This is what happens when you run queue.

D

Control delete a particular config map, so in this example, we are going to delete i3. The delete command sends a propagation policy of background to the cube. Api server and i3 is immediately removed from the api. The garbage collection, controller notices and then deletes k3 because it no longer has a valid owner reference.

D

Then the garbage collection, controller notices that o2 and o3 have no owner references and they get deleted in any order.

D

Another option for deletion is to orphan it you might use this means that resources are going to be deleted, but instead of cascading through garbage collection and deleting the children it'll simply remove the owner references to the children. This is something you might do if you want to replace a parent for some reason, like maybe there's an immutable field, and you need to change what that immutable field. Is you have to delete and recreate?

D

But you don't want to cascade through everything to do this, you would use a cue control, delete with a cascade equals false and it would send a propagation policy of orphan. So if we work through the same example only this time we're going to delete i3 with an orphan policy, then i3 is march relations, deletion timestamp is set and an orphan finalizer is added to the list. The garbage collection, controller notices this finds the children and removes their owner references once the owner reference is removed, then the orphan finalizer is removed and i3 is deleted.

D

So you can see here we end up k3 and the other owner. References are still intact, and now we can recreate i3 and re-link it if we wish foreground the deletion of the foreground is one of the more complicated ones, and this is what block owner deletion is for. It allows for more control over the ordering in which resources are deleted.

D

This means that we're able to have parents wait for children to be removed before they themselves are removed. It's important to note that this only works. If the parent is deleted with foreground deletion, if it isn't, then your block or deletion doesn't actually do anything.

D

So it's more of a hint there's actually no command to trigger this, so instead we have an example of using a delete, dash dash raw to send a propagation policy of foreground to the cube api server. So you can see here we have our same example. We have owner references with block owner deletion, set for a couple of the cases and we're going to walk this through.

D

The first thing that happens is that i3 is marked for deletion and a foreground deletion is added to finalizer s. The garbage collection controller notices this and goes to the next level and marks k3 the same way, but k3 cannot be removed from the api yet because there's a block owner deletion from o2.

D

Instead, we actually have to go and remove o2 and o3. First, once o2 is removed, it becomes possible to delete k3 and once k3 is removed, we can delete i3. You can see here that the ordering is actually the reverse of the background deletion, but remember it's optional. If you deleted i3 with background deletion, you wouldn't get this order. So if you need to preserve your resource you're going to want to set a finalizer so bugs we do have bugs one of them is with block under deletion.

D

If you have two parents of a child, then it doesn't behave right in fact I'll. If you delete i3 in this case I'll just skip through the pieces, we already know we're, marked relation and now we're at the point where k3 should not be deleted until o2 has been removed, o2 should not be removed early, but the garbage collection controller removes the owner reference from o2 to k3.

D

By doing this, it becomes a valid target for lesion and k3 can be deleted. This is a bug we end up deleting i3, k3 and o3, because the owner reference from o2 no longer exists. We don't currently have a pr to fix this bug.

D

It is less than ideal, but I hit it again while creating this demo figured I'd, mention it there's another case where clustered namespace references can cause deletion of of resources with valid owners. So in this case uh we have a cluster scope, resource n2, which has which has taken an owner ref against the name space resource, and you recall, I said you should never do this. It's not allowed.

D

The api doesn't prevent it, and sometimes it appears to work. Because what will happen is i3 exists in the cache in the garbage, collector and n2. Does an existence check uh with a ui and it matches i3, so it appears to have a valid owner, but on every restart the cube controller manager effectively has a race. It races to see whether it observes i3 before it observes n2. If you lose the race n2 is observed. First, there's an i3 existence check, but there is no i3 at the cluster scope.

D

Remember, there's no namespace, so the cluster scoped resource can't depend on a namespace scoped, one that causes n2 to be deleted, and that's probably okay, because it has an invalid owner reference. But k3 was deleted as well, even though it was well formed because we saw the uid disappear.

D

This is a bug. We have a pr to fix it that we'll be looking at doing in 120. in a related problem.

D

The same thing can happen across namespace, where, if you race and i3 exists before you see k2, then k2 won't be deleted, but if the race goes, the other way the existence check fails because there's no i3 in namespace one and so k2 and k3 are both deleted.

D

There's one more common case of seeing this and it's inside of a single namespace where the kind is set incorrectly. When this happens, you end up with a deletion and it causes k2 and k3 in the namespace to be deleted. As I said, we have a pr fixing this in 120 and it will prevent the good owner references from having their resources removed.

D

It was a whirlwind tour through garbage collection, and if you want to ask more questions about it and you don't get them in the session, you can find us in slack and then in the community meeting and mailing list. I've got links here all right.

A

Hello again, I hope you enjoyed our did that topics from our presenters. Personally, I really did before ending this session. I wanted to remind you that we have regular sick meetings every two weeks and twice a week. We do our bag and pull request three ashes for 30 minutes. It's a great way to get involved, join our mailing list to get the invites.

A

We also have cube builder and api expression working groups as part of our c, and they have their own respective meetings twice a month. We are all always in our slack channels.

A

Finally, I will leave you this slide with contact and material information in case you want to know more or do more with us in the name of the entire siege. Thank you for attending our session and we wish you a good and enjoyable crypto. Bye, bye,.