Cloud Native Computing Foundation Kubernetes Community Days Bengaluru 2021, 10 Jul 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Understanding the Kubernetes API Internals

Description

Kubernetes Community Days Bengaluru'21

While many people write operators, very few understand how the Kubernetes API actually works and what goes on behind the scenes. Part of the difficulty in understanding controllers and the API -- and implementing them -- is that the tasks are broken up and performed at many different times by different pieces of code. One of the strengths of this talk is to integrate the pieces and reveal the relationships between far-flung interfaces and methods. This session is targeted especially at: - people using the Kubernetes APIs with client-go and other related frameworks - people extending Kubernetes with APIs using aggregated API servers or CustomResourceDefinitions

Slides: http://bit.ly/kcdblr-api-internals-slides

A

Hey everyone welcome to the second talk of the day after dim's amazing keynote. We will be taking you through.

B

A

To the internals of the kubernetes api, I.

B

Am navarro- and I am nikita.

A

The kubernetes api server lies at the center stage of all components. Each component of a kubernetes cluster, for example, scheduler cubelet, keep controller manager all talk to the kubernetes epa. Server for operating in objects be creating them, reading them, updating them or deleting them. Take a server leads, manipulate state and restore security.

A

Api groups in kubernetes are logical collections of related functionality. For example, workload objects like deployment stateful set replica set, are in the apps. There are about 20, plus ebay groups in the kubernetes api ranges.

A

When we see the api endpoints, we will see that the core group is a bit special. We will see that later. When we discuss about resources um going ahead, the kubernetes apis are versioned and each api group can have one or more versions. They are usually of the form b1 alpha. One then v1 alpha two, then, if it graduates to beta it's b1, beta1 and so on, eventually going up to v1 v2.

A

The code group at this point in time has only single version, but groups like apps has versions like b1 v1, beta, 1, v1, beta 2 and v1. Any group can have more than one or just one api versions.

A

When present in multiple versions. Kubernetes api server needs to perform conversion between them, such that there is no loss of data. Now you have to keep in mind that versioned objects are just different representations of the same stored object.

A

Types in kubernetes api usually graduate from alpha to beta but general availability with alpha being the most volatile as in they can change during the life.

B

A

Beta being closer to stable and.

B

General availability.

A

Means they will be there for certain number of releases unless a replication has been called on.

A

Next up kind is the type of an object in the kubernetes api universe, and you might have.

B

A

Like each object that is returned by the kubernetes api has a field called kind pod deployment stateful set service are examples of kind.

A

Each of these kinds exist in a version api group. This very relationship is called a group version kind or gvk. We will use this abbreviation a lot in future slides now. You might wonder how are these go types which we define in code, mapped to blue version kind?

A

We will look at something: called a scheme scheme maps go types to all possible group version kind combinations scheme, also stores a list of conversion and defaulting functions. We will explain what converters and defaults are in a later part of this talk.

A

In the kubernetes world, there is also a concept called resource. A resource identifies an http endpoint, exposing the current semantics of the associated object. Remember I told that each of these apis returns you a representation of the object that you are accessing, which is stored on hcd.

A

A resource is always part of a version api group other than a few special endpoints them being healthy, which are mostly used for clustered administration, and basically, this combination of a group version and resource is used to identify an http endpoint and each of these endpoints receive and return in unique kind, for example, if if I do get to deploy, get slash deployments, uh as in the first example, tp server will return a deployment list object. If I get on a specific deployment, it will return a deployment object, similar case with the later example.

A

Coming to the later example, you notice that the structure is a bit different for core core group types. This. This is a really special case. In that case, we use the slash api prefix and, in case of all other api groups, we use slash apis.

A

There is also something called a sub resource. Some of these resources may have a subresource like port forward exec locks, which are used in case of pods in case of resources, which can be scaled. There is sub resource called scale which is used by the horizontal part of scalar.

A

The mapping of a gvk to a gvr is called rust. Rest mapping and it's there are helpers called rest mapper, which perform the same, uh which gives the gives the users the flexibility to map between these two concepts.

A

A gvk is essentially served by the api server under gvr http endpoints. That we saw previously do note that agvk can map to many group version resources. Most of the gbk is mapped to a single gvr, but objects like scale can exist in multiple gbrs because by virtue of them can be present in others.

A

External types are what you see in the staging src kts dot, io, slash api directory uh in the kubernetes code base. You may check out the code base uh from github.com slash, kubernetes communities. um These external types are what are supposed to be consumed by the users, and these are externally consumable now coming to internal types. They are also known as hub types and they are not exposed to the users. It is not a good practice to use them as a dependency in your code. They result for usage internally in the api server.

A

Now this of this internal types give the code base flexibility to version objects by the consumers as external types, but then at the same time provide a baseline for internal computation as hub types for objects having multiple versions, the internal types acts as a conversion, medium.

A

Now, having told you about the concepts that are using the kubernetes ap servers, we will dive into the code base and we will be using here a vs code extension code tool and you can go to the gist and download the json file to essentially replay the code to your on your own computer.

A

Now, let's deep deep dive into the code, so number one thing.

A

So if you, if you have checked out the kubernetes code base, you can see that inside the staging src caters dot, io slash api directory, you can find all of the external types which can be imported by the users.

A

The external types directory holds all of the api groups. As you can see here, we have apps authentication authorization, auto scaling bad certificates. These are all the api groups in the kubernetes api. Now inside all of these directories, we have api versions.

A

Inside that, uh inside inside those uh inside those uh apps v1 directories, basically the versioned epa directories. You will find the types.com, which contains all the kinds that are present in this specific group version combination.

A

Now all of these external types have tags to generate some repetitive code: open api, spec and code documentation.

A

Along with times dot co, there is a dot go inside. Each version, ebay, group directory, which contains helpers for registering a type gvk mapping in what is called a scheme.

A

You can see uh in register.gov, there is a helper function, called add, node types which essentially calls scheme, dot, add-on types and provides it with a list of all kinds that you want to register for that group version. Here we are passing in the group name apps and a version called v1.

A

Hopping onto internal types, the pkg api directory in the kubernetes code base has all the internal types.

A

For example, we saw deployment.

B

As an external type here, we.

A

See deployment as an internal type notice that there are no json or prototypes to annotate this code, because all of those repetitive stuff is generated from the external types.

A

We do register the internal types as well, but they are registered with a special internal version. Constant and this types are not considered stable or serial serialized.

A

The install function in pkg apis, apps install.go registers, the external and internal api.

B

A

Whenever this package is initialized, so you can see here that we are registering our internal types, we're registering all of our version external types.

A

Now we have talked a lot about scheme. We have seen some preview about scheme now. You can see here that scheme stores, information to map between the registered go types and the corresponding gv case. It also stores any conversion.

B

A

And default functions there are there. Are there, are information in this structure to give you all of the information when you, when you have a gv cave, and you want to find the code type for the same.

A

Now that we understand like how this api types are registered into the runtime scheme, we have to understand like how an api request is handled. So whenever an apa request comes to the api server, they go through a set of filters which are defined in default field 100 chain in a site called in the pkg server directory in the staged api server.

A

Now this filters, in sequence, run for run some checks and if any of these checks fail, uh an appropriate http return code gets generated. Response code gets generated along with an error message um and if any checks, if all the checks succeed, it goes on to the next step. We will see in the next part of our talk.

A

The chain has filters for panic, recovery, api priority and fairness, authentication authorization. It also has filters for adding course headers.

A

There are things for auditing.

A

So now, when the request passes through all of them, we have to see like which logic gets called. Now we have to understand, like how is the multiplexer structured so, depending on the http path of the request, we have some rules which will request to the appropriate handler. Now, where are this defined?

A

So when, when you instantiate or set up an ap server process, it gets cascadingly installed using an install rest method which basically checks the storage in your runtime and based on that it enumerates through all the epa groups and installs uh the corresponding endpoints, which which map to each of those gbks which are exposed in the current kubernetes version.

A

Now, with this, uh we have completed part one of this session. I will hand it over to nikita we, who will give you a overview of conversion defaulting and how we handle this, handle the process of admission and mutation and I'm going to remove myself and then continue giving you pause or something.

B

Thanks for the wonderful code base walkthrough, uh now that we have seen how the kubernetes handler chain works, let's understand how resources are serialized and persisted in hcd and the most important concept to understand about serialization and kubernetes. Is this thing called conversion? So what are we talking about? What are we converting? So the first thing to understand about conversion is that each api resource has something called as a storage api version. What is the storage repair version?

B

Each resource needs to be stored somewhere, so it's stored in fcd, but what version is it stored in? So this version that it's stored in is called the storage api version and it is usually the most newest and stable version. uh Let's take an example by considering the storage version to be v1.

B

uh If you write a yaml uh file or a if you create a resource using cube, ctl apply and you specify the api version as v1 beta1 in your yaml file, then this v1 beta1 version would be called the input version.

B

Your input version could be even something like v1, beta2 or v1 beta3, and in all of these cases, since v1 is the version of the resource that will be stored in lcd, we will need to convert them to v1, but since this means that there could be several of such cases of conversion because we want beta 2 to v1, we want beta 3 to v1 beta 1 to v1, to make it easier. Kubernetes uses something called as an internal version, or it's also called a hub version.

B

This is equivalent to the internal and external types that nabarron was talking about.

B

Similarly, v 1 beta 2 would be converted to v 1 by the internal version, and the same goes for v 1 beta 3., uh let's just consider v, v1 beta1 in our subsequent examples, to keep things simple.

B

Another important thing to understand here is that the process of moving from v1 beta1 to internal and so on also involves something called defaulting. For example. What if there are additional fields in we were that are not present in v, one beta one. So how would we ensure that, like these additional fields are defaulted or present in v1, when we convert the object to ensure that the v1 version object eventually contains valid fields? Defaulting is done, that is, fields are defaulted, whichever ones are desired uh after the conversion. So please.

A

Remember like whenever we usually talking.

B

About conversion, we're also talking about defaulting so now that we're familiar familiar with conversion, uh let's understand how the overall resource handling works. So the first conversion that we talked about was the input version to the hub version, so the for the first conversion from p1 beta 1 to the internal version. This process is called decoding and our decoding involves three things. First is called the serialization or basically you have you submit a yaml or a json data.

B

So you convert this data from yaml form to the golang type and second is called defaulting that it applies the default values that we talked about. And finally, the third step is conversion to the internal version. So just to recap a little bit: serialization convert the data in the column, form apply the default values and finally convert it to the internal version.

B

So we have the internal version and we want to convert it to the storage version. This process also undergoes three steps, so the first step is admission. So if you, uh where mutating and validating my books are applied and if you're not familiar with uh web hooks or admission, you can think of them as like. Mutating webhooks can modify the object and validating webhooks can reject a request. So first step is admission. The second step uh involves this construct called as a registry.

B

What it basically means it prepares the object for getting stored into lcd by doing some additional operations on top of it like removing the status part. So if you've noticed, whenever you create an object and if you specify the status, it doesn't get persisted in that cd. Why? Because the status part is removed here in the registry, uh validation is also done here. So so, like the name is a valid uh like follows: the proper characters.

B

uh It's not a. uh There are specific constraints that we use in kubernetes. All of those validations are done over here and in the third step. Finally, the object is encoded before it can be stored in ncd.

B

Remember that we talked about this thing called as a storage version, so encoding here means conversion to the storage version, applying the defaults and deserializing it into protobuf or whatever the specific storage media type that's been provided uh and, finally, the object gets stored in the v1 version in xcd, uh so to recap very quickly, so you have the internal version and the storage version and the conversion between them takes place in three steps. So first you have admission so mutating and validating web hooks.

B

There's a thing called as a registry which we look in the code, what it exactly means and it does things uh it does some operations before creation, and it also does validation. And finally, it gets converted to the storage version before it gets up. It gets processed into fcd, let's quickly look through the code and what it looks like like navarro, I will also be using the vs code extension code through code tool and if you want to run the code in your vs code, you can do that by using the gist in the slide.

B

Let's quickly walk through what the code actually looks. Like don't worry, even though this looks scary and the api server code feels like it could be scary, it really follows some basic principles that we just saw. uh So I'm going to start the quarter right now. So the first thing we're going to be looking at is the api sr directory. uh It's in the.

B

If you go to the kubernetes kubernetes github source code, it's in the stagings, uh src or source k, stud io, api server directory and in specific uh today over here, we're going to look at rest handlers. So what we're going to do is we're going to look at the workflow for creating a resource and the rest handlers or the creating handlers are specified in the handlers.

B

Directory so before we look at how it requests like a resource gets created. Let's look at what happens when you actually send a request to the api server when you send a request. There's this thing called a request scope which contains common fields that are required for all these handlers, so it contains conversion defaulting and such methods.

B

For instance, this field is called a serializer or a negotiated serializer.

B

It's basically an interface for obtaining encoders and recorders for multiple media supported types, for instance, if you specify a json uh format or if you specify yaml format, both of them have separate encoders and recorders, but all of them can be combined together in this thing called a negotiated serializer.

B

Next, you have this thing called as an object converter, which is used for conversion, so converting one object of just from one version to another, so from v1 to the internal version, and then you have a defaulter which does defaulting now, let's look at how a resource actually gets created. So there's this function in the aps, our uh package called so there's this file called create.go, and then you have. This function called create name resource so like the command says it will handle a resource creation with a name.

B

So it's creates a resource with a name now create name resource. Basically, it just calls create handler. So the main fund is in the create handler function and we will look at how that's done so, let's look at what create handler does create handler. Firstly, you want you're creating a name resource. So first you want the name and the namespace of the resource that you're creating, so that is derived from the request.

B

Then uh serialization is struck. I remember that we talked about first, we had that v1 beta1 resource and it had to be converted to the internal version. So now we're going to do that conversion. First, so from the input version to the internal version and for this conversion there were three steps: one was serialization so from data to the golang, uh so like a yamaha or json data to the golang type, and then the second step was defaulting and conversion. So all of this is called decoding or serialization.

B

For that uh you need to know which data you have select. Do you have a yammer data, or do you have a json data? So all of this output media type, which media type are we looking at camelot? Jason is also derived from the request using the negotiated, serializer and negotiate output. Meter type method in this.

B

Next, we want to perform recording for that. We need a decoder, so this decoder is also created using the serializer uh using the decoder to version method. uh So what does this? Do it serializes? It does defaulting and it does converting we'll see that a little later to serialize something you need something to serialize, so uh limited, read body basically reads the request body. It just does a read all method on it. So it's not something fancy. It's just a simple read all on the request body, so you get the data.

B

Finally, uh you to decode the data, so you also do defaulting and conversion with this uh decoding is done by the simple method called decoder: dot, decode, that's literally it after recording. We have your. We have the internal version now we're gonna do a few more things on the internal version. That is uh first, we talked about admission, so we had mutating webhooks and uh validating webhooks. So in this case mutation mutating admission.admit, the mutating webhooks are applied and in this request, func the validating webhooks are applied.

B

So we'll just quickly see what request func means uh so request. Funcs calls are create. It basically means that it's trying to create the resource, uh but it also passes this thing called admission to validate object, func, which uh in turn calls validating admission to validate. So it's calling the validating, webhooks and then creates the object.

B

Now we also learned that the next step after admission was doing some more operations on the resource before storing it. uh So, let's see where that happens, so that happens in this directory called registry. So registry is basically a directory and it's also go along type, but then we will not go too much into that right now. So there's this directory called registry and it prepares the object for storage and does some validations on it too. So, let's look at it since we're looking at creation.

B

It also has this create method, uh so we'll just look at the create method for now uh in the create method. There's this thing called before create it prepares the object for creation by removing things like status, part of the resource or doing some more uh mutations, if needed.

B

uh Finally, we're going to validate it, so we validate the object before storing it to lcd and that's the main parts. uh The main function of the registry: now we want to finally store it to lcd, so we have to convert it to the storage version. So, let's see where and how that's done so this is directory called lcd3 because you want to store it in a cd. So, let's see how that works. uh There is also a very convenient create method here.

B

So we just look at that and there we have this thing called runtime.encode. So previously we looked at the decode method. uh Now we are looking at n code, so we are deserializing it back to pipes. So we converted to the storage version this. What all of this happens under the hood, so it converts to the storage versions and encoded back to bytes and finally, that's done so we call the uh xcd client and then store the object and put it in a cd, so that's kind of it for the code tour.

B

Now, let's go back to the slides for a bit, so we looked at the code base through right now um and let's very quickly, touch base on this to be very clear: we're going to be covering the city there. So you might have noticed that we did not discuss about crds at all. Crd is an aggregated api servers are not part of the generic kubernetes api server. They are uh aggregated with it like the name suggests, but they're not part of the community's api server directly. They follow the same pattern, though.

B

So once you understand this, you can also understand the crd and the aggregated api server code, uh but one thing I will.

A

B

Out is that crd, the code for crd is stored in the directory called ap extensions, api server and for aggregated api servers and uh directory called cube aggregator, but covering them in detail results its own talk, so we will not be talking about them in this session and that's kind of it uh for now, thanks so much for attending. Even though this is virtual, uh we have a feedback link.

B

So if you have some constructive feedback that that you would like to share with us, please do leave us a message there uh and yeah thanks for attending be safe, bye.