Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2019 (San Diego), 22 Nov 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Deep Dive Into API Machinery - Antoine Pelisse, Google & Stefan Schimanski, Red Hat

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

Deep Dive Into API Machinery - Antoine Pelisse, Google & Stefan Schimanski, Red Hat

SIG API Machinery is responsible for all generic API topics in Kubernetes, i.e. for the generic API server implementation, API CRUD semantics, discovery, the admission control mechanism, conversion, defaulting, persistence with etcd, general controllers like garbage collection, Go client libraries, code generation and extension points like CustomResourceDefinitions, aggregation & admission. This session will have two parts: A deep dive into a selection for API Machinery topics, probably: defining API types in Golang, groups+versions+kinds+resources, tags, code-generation, schemes, different variants of codecs – and how to use all this with CustomResourceDefinitions and a custom client-go client. time for general discussion and opportunity for API machinery questions. This session is targetted especially at: People using the Kubernetes APIs with client-go and wanting to understand what is going on behind the scenes People extending Kubernetes with APIs using aggregated API servers or CustomResourceDefinitions

https://sched.co/Uahb

A

Hello: okay: let's get started hello. Everyone welcome to capi machinery deep dive. My name is Antoine Felice.

A

This is Stefan shemanski and we have Mike spread so from IBM and Stefan is from right at we're going to talk about apiary machinery and Stefan is going to stop talking about the CRTs and some of the new things we are working on. Then I'll talk about suicide, apply and then Mikey is gonna. Talk about peyote and fairness. So.

B

One topic we are talking in the city at the moment about this immutability, so marking things in ser, Diaz unchangable. As you know, from pot specs for example, prospects cannot be changed, similar things we want for CDs and yeah. I would start with that. So it's not finished. Everything here is working progress as a kept as a PR against the cap to update it. I just want to see to show ya. What's ideas here are so the idea is that we can mark fields, immutable, mutability immutable, and then this is a the slice.

B

Here's what's defining an array in open API cannot be changed and our question was: what does it mean? Obviously, if you don't change anything a B to a B, it's fine. If you change the order yeah, it looks like some change in the in the array. So you don't want to allow that, but we quickly came to cases like empty slice. There's as null should be food it'd be the same or not.

B

It's not so clear for many native types it is but for C edges here these are usually handled strictly, so those differences are preserved. So probably it's not the same same thing: if you, if you're just slice not at all, so the slice, not you find it all in the in the surrounding object. Also in here in the CID world, it looks different, there's a second way to to put immutability on something gets put on the items so I'm, not only the whole array is immutable, but only the items individually.

B

So every item would be unchangeable and if you do that, so we get similar cases first order changes, not a good idea. We won't reject that, but maybe removal is okay because yeah.

B

The second element, the second second item here in this in the slice: it's not changed because it doesn't exist anymore in the result, so one could say: yeah might be the same and adding something could also be allowed, because the array can be changed, but not so items. But if an item doesn't exist before it's not really changing so yeah we're discussing those things. If you want to look for details, the cap is there and update of the cap. Recipe are at the moment when looking at null and empty yeah.

B

There were different opinions what to do, and in native types we usually identify them so now and MD is the same thing or undefined is the same thing, and but we have done some choices in in NC, IDs, so defaulting exists will be GA and the next version and defaulting is strict, so and defaulting only undefined. This default it. If you have a null in an object or an empty list, it's never defaulted, so defaulting is strict.

B

So we made a choice not to identify those values, validation we have set for ages and everybody who use validation relation you can easily distinguish between those special educators. So I would say. Validation is also a strict mechanism. We have implemented for a long time so yeah. The consequence of that is probably we want also equality to be strict. So always when objects are compared and you have null you have empty, you distinguish it's never the same thing.

B

The rule of thumb, which we have in mind here is if we define objects, are yeah when objects are equal, when are they equal on the client side, when you have an object in your hands to manifest, for example, and say you consider them equal, then, if you do a request with this object as a server, you would also expect as a server. That's the same thing so I'm really hand waving here at the moment. So it's really really a really rough idea, a rule of thumb or something like that.

B

So if objects on the client side and in ED CDI considered equal and I, do something with objects: I, better get the same results or the same response and I get the same reside also in LCD. So we want something like that. So the consequence is probably that we want a strict equality, so we don't want to unify null MD and undefined.

B

Negative types are different, everybody know was that if you use go long types and you're unmarshal into them certain things yeah they identified by omit empty. Everybody knows on with empty in goal encode, so we have something which normalizes in native native types already and see at least we don't have set at all, and the question is maybe we should have set so for many use cases. We don't want to distinguish between those three special educators. So maybe we need something similar in series as well.

B

We are looking for puto buff or some variation of binary encoding and Porter. Buff is even more like that. There's no distinction between not defining something in a CID and empty.

B

So maybe we need something like normalization as well, and we are thinking about that and if you, if you look on the request pipeline, we have in kubernetes 4c IDs looks like that all the green bubbles- here's green circles- they are strict. So we have makers. Mechanisms like defaulting I showed that validation and service logic, so patches and so on. Everything is tweaked, and maybe we need something which normalizes source values in this pipeline, basically yeah repeating what we do with native types with us, so : and marshalling.

B

We want to replicate that for CID, so you get something like that. Once is reputable here. There may be one for to normalize and we are discussing that at the moment, and actually there are more places where we have to take on a more fuzzy look on our objects so actually identified. Those here, I think they're more in admission. Even so, this might become something in this year. Deep processing immutability also sent us to this topic, so we have something new.

B

Since 116, you can define this type for fields in CDs and, let's type traditionally it was just atomic. The whole list was either unchanged or replaced by an update. We have two more cases. Now we can say a list. An array in a CID is a set. Then you expect that every element is unique, so there should be some unique reading, a check in the CID poster and the siapa setting, and we also have a map type where you have a subset of the fields which are keys.

B

So everybody knows containers and ports there as they are lists, so this type is map and the name is identifier for each element, and we also support that in 116 for CRS yeah and where's the same thing for maps. If you have a map, you also want atomic as an additional kind or different additional type to Maps everybody who knows strategy merge patch. We had similar things already before for native types, but it could say how things are merged.

B

So this is all about merging when you, when you update, for example, here on the left side, we have a map, a has a value, X and I update it with be 42, then by default those are merged. So granula is a keyword here as a type those are merged. If you change it to atomic, you want an atomic map, it's a replacement. Then, if, if a map is atomic, it behaves like a slice like an array: arrays are by default atomic and replaced.

B

So if you have a set to X and you update with B 2:42 and it's an atomic map, then just be 42. Yes, wizard a is just dropped and yeah, so this is all introduced for which no reason was servers had to play, and Antoine will continue with this topic now.

A

Okay, server side apply. So when we started implementing this feature, we spent a fair amount of time trying to understand how configuration and objects worked in kubernetes, and so we went back to the roots and we realized that kubernetes is about declarative configurations. What does this mean? It means that multiple people in the system have an opinion about the resources and they're going to send what they care about, and this is going to be merged together into the object.

A

The object is changed, my multiple axles and they have to collaborate together to have a good, reasonable value. The resources typically specify the intent and multiple actors are changing it. So the idea behind cube call apply initially is to a little clients and by clients. I mean mostly users like human beings, to send our intern to the server and say hey. This is what I care about this is what I want the system to look like. Please make it so the poem we studies, it's it doesn't work with.

A

You have multiple actors, which means that controllers can't have intent. At least they can't use client-side apply. They have to either change the object, fetch the object modified the way they want it to look like and then update it, or they have to build a complicated badge. Sometimes it's very hard, because if they want to do a strategic knowledge patch, it just doesn't work. So it's very hard to write a controller that is going to specify an intent.

A

So we consider these use cases. We can look at some other drawbacks of client-side apply. If you're writing your library that is going to send client-side apply, it's very difficult. Constructing the SMP patch is very hard. The protocol is also extremely how to update this. In google, organically and anytime. We update the server to implemented a new feature. We have to make sure the client also has the feature, and we need to make sure that you are talking the same language. It's very how you also have mostly to use the API machine reliable.

A

If you want to do that, which means you have to do it and go. There is no implementation in Python or anything else. The only other option is to actually use cube, coral and other poems'. We have is that there is no multi key associative least.

A

There is no unions, you can have multiple versions, don't even try to apply an objection, a different version, that's not going to work, so we decided to try and fix it from very far away. Southside is about tracking people's intent, and what we've done is that we tracked fields that we believe people care about when a client apply their intent.

A

They send us the fields that they care about, and so we're going to track these and we're going to merge everything on the server so that the resources look the way we think you want it to look.

A

Because so, clients and the entire internships in the server side apply, but existing controllers can't use all side apply because they've been built before server side apply existed. So what they do is that they keep using the update or the patch and we're trying to guess their intent based on what they changed. So if, if a controller changes a field, we're going to assume it has an opinion about this value and we're going to track it. So we have two different workflows.

A

Now we have the source side apply, workflow, where you specify exactly what you want and we have the update workflow where you change the resource and we guess what it is pretty much. The update workflow is used for anything that exists today. That is post for creation boot for update that this is going to use the the update workflow and if you send an apply, we're going to use all the fields for your intent. The way you supplies. Interestingly, it's a patch, so you just send the object. It's a yellow object.

A

You send the object with all the things that you care about the values that you want and we're going to keep track of that. So how do we keep track of what you own?

A

We've built some sort of tree of all the fields that you care about, and we move move that inside the metadata of the of the object. As we can see, it's not just the name of the field. It's also F colon, which is actually useful to understand the structure of the object. We can see f means fields and we also have a que name static sidecar. This is useful to understand that containers least, but the least are actually maps, and so we can track the object by looking at at the key.

A

So for each person that replies or modify the object and for each version we have these set of objects that they are. We can we we can have the some of the fields can be owned by multiple people. So if two persons apply, if two person apply the same value for the same fields, it's going to be tracked by the two persons. If any of the person tries to change the value, then it's going to cause a conflict.

A

You can first the convicts we're gonna, see that next and yeah so a conflicts when you do an update. So if you modify the objects or send a patch, which is the case for most controllers, we are going to automatically grab the ownership of the field. That is because we don't want to break backwards-compatibility. So if the controller is used to update or send the patch is going to have to succeed, it can fail now. So we are always going to change the ownership of the field.

A

So if I'm a d HPA and I'm trying to change the replica of an object, anytime, I change, this replica I'm going to own that that field apply a small use cases. So, when I'm trying now with apply to object to update this number of replicas, if it's owned by someone else, I'm going to end up with a conflict.

A

Because- and that's that's actually a feature- we don't want people to override values that they don't own by accident. If the HPA has decided that the value should be least- and you should either decide that you have the ownership of the field- and you have the opinion or you have to give it up.

A

Merging merging is extremely simple: we just take the current object and we apply the new configuration on top of that. What I'm saying simple, of course, I'm lying it's very, very complicated.

A

What we have to do is that we want to remove the field, the the items in the maps that you don't care about. So, for example, if you had a container and you decide to change the container for another container, we need to realize that you remove the container container, so we can't just apply blindly. We have to find that you have you had this container before you don't have it anymore and nobody cares about this container and we also have to do that across versions which is pretty complicated.

A

There are quite a few features that we are missing on our futures, the drugs. Also we have so we have a lot of problems with performance, it's keeping us pretty busy, since we are definitely on the critical path, because we have every single update per batch and apply operations to track. That's a lot of work for the API server, so we're trying to reduce the performance impact also, since we are out tracking the fields for every single modifier of the object. This is a lot of data that we have to add to the object.

A

So we are trying to make sure that these field series as small as possible, we're also working on unions, which is not completely well defined. Yet another interesting feature that we want to implement is the ability to remove fields where I said that you declare the opinion and the things that you care about. You have absolutely zero way to say I care about this, not existing. This is just the nature of the object.

A

We don't have a good way to say: I don't want this, so we would like to implement a way to say: I want these to be removed or I want these to not exist. This would be extremely convenient for controllers, and ideally we would like to track mutation from mutating webhooks, but again because of we can't perform all these operations for every single mutating webhooks.

A

C

So we don't have the declines correct. Well, when you look at it at home, you'll find the names I'm working with a few people, aren Prindle and daniel Smith, from Google Bruce, ma and and Tony hey from and minjin from, and financial and myself. Okay, um can we go down a little.

C

C

Okay, that's better all right, so we're working on adding a new feature. There's a cap for it about API priority. In fairness, the concern is getting some consume. Self-Protection in the control plane- currently buggy controllers, buggy or aggressive clients- can can take down the control plane. Just by overloading it. We want to provide some self protection, so the basic idea you first think of is rate, limiting. It occurs to us that it matters that some R in some requests take longer to serve than others.

C

So we want to really look at the product of the RIC request rate and how long it takes to serve each request and if you think about it, a minute that averages out to number of requests executing. So what we're going to do is regulate request executing and really. This is just a more sophisticated version of the max and flight handler. That's there today, the existing one makes a simple distinction between mutating and non mutating requests, which we think is actually not very interesting anymore, we're more interested in distinguishing where these requests are coming from.

C

What they're doing oh there, it is okay, all right. So the basic idea is: this is a filter. It goes into or a handler. It goes into existing handler chain, I've zoomed way in on the new or there's some earlier handlers and some later handlers, and we can debate about whether it's in the right place we're still working on that. But the basic idea is: there's two concepts here: there's priority levels, we're going to classify the traffic and assign it to priority levels. So the first stage is classification.

C

We have a series of flow schemas that the each request is matched against. The flow schema is in turn, the first one. That matches is the one that determines it. A flow schema is, as the name suggests, the description of a collection of flows for each flow schema. Its flows are mapped to a priority level and then a priority level disguise describes what to do with it. Each priority level has a request, concurrency limit and then the choice for how to handle things that exceed the limit.

C

You can either drop them or queue them also some requests. Some priority levels are exempt from this limiting limitation at all and that's important. All right, and if it's queueing there's a set of cues that are configured, we use a technique called shuffle charting which lets you actually get a lot of independence with a relative. If you few number of queues so away, we go. Let's look at some example configurations so for a priority level. Maybe you start in the bottom right here. The simplest kind of priority level is one that is exempt from controls.

C

It just says: hey I'm exempt next level up in sophistication top right hand, side top right hand, side. The slide is a request. Priority level that says: okay, I'm gonna have concurrency limits, but I'm not going to do any queuing. So you see there there's a parameter called assured concurrency shares. So what we do is at the API servers options, there's already parameters to configure a concurrency limit, that's used by the the maximum flight handler.

C

We repurpose that okay, we divided up proportionally amongst the priority levels that are not exempt, and this shares is just their proportion. Okay, so that kind of logic. So the shares, together with the overall limit, determines the absolute number limit for this priority level, and that's all there is if there's no queueing, if a request, if again, if request comes in for a prairie level, that's already at its max it we'll get dropped or rejected more precise.

C

Now, let's look left or left-hand side of the screen here, a more complex example where we doing cueing, okay, we still got the same parameters for the concurrency limit, but we've now it's also got queuing parameters. We've got a number of queues, we had a hand size, that's the shuffle shorting. The metaphor: is you deal up a shuffle a deck and delisted in certain number cards, which is a number of cues?

C

Then we're going to pick the cue that has the one of the cues that has the shortest length to put the new request in and then we also have some limits on how think long things can sit in the queue a one is, there's a time limit and that's a a PR server, a parameter and there's also a length limit for each queue which Prairie level has its individual queue length limit all right. So that's priorities on the flow schemas.

C

We've borrowed a lot of concepts here from our Beck and adapted them, and and we recast them. We realized that in our back there's actually a lot of Union structure. That's not done in the usual way. We do unions in in our API models, so we revised it to follow the usual patterns for unions. Okay, so there's a couple of concepts here, there's like in our back. There are subjects, verbs and objects or targets. We say because everything's an object, so subjects.

C

There are three kinds of subjects we can match on, like in our back general user or use by matching find user name user groups and service accounts. For then for the then there's four targets or yes, there's either resource style target or non resource style target. You see an example here with the non resource style, where there's just a verb in a URL pattern for the resource style. We have the usual terms in our API. We have API groups, we have the resources and we have namespaces.

C

Let's look at another example to bring a couple more things. We can talk about resources that are not clust that are not names based. We could also talk about requests that are for all namespaces and that's matched with this cluster scope. Boolean.

C

Here's also an example of a service account from where we talk about the namespace in the name of the service account so.

A

D

Gonna take questions.

A

We have five minutes right. Ten minutes.

B

Not sure is a computation to sent to different SD clusters.

C

B

That's actually working focus PR fixing that at the moment technically is possible. We had discussion to to build something like a storage, computation layer, so something more abstract and more official, but I haven't really really early. So if you're interested that in that yeah Joyce discussion speak to the people from the sake, but no at the moment it's not possible.

A

Okay, so the question is: if there is a conflict, how do you know what the conflict is and would you ever conflict with, so we actually keep track of who has made the change, our ones, the field? We keep track of the time when it happens so pretty much if you have a conflict and you try and change the number of it because it's gonna send you an L saying that you have a conflict with the HPA that change the value at this given time.

A

So we know exactly what Easter causes of the conflict. Does that answer your question.

A

So we're not keeping track of the entire history for size reason, but we keep on the track of the last change made by a specific controller use. Only.

C

For size, but it's logic logically, the there's, this takeover of ownership yeah.

A

B

They will be attached and updated on the scheduler, so I think.

C

There's an older.

B

No, that's not maybe at the moment I'm not sure we will attach them.

B

Thank you very much. Thank you.