Cloud Native Computing Foundation Kubernetes Community Days Bangalore 2022, 11 May 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: How to write API conversions for Kubernetes CRDs?

Description

In all open source projects like Cluster API and its providers we initially start with alpha releases, however, eventually, most of these projects will need to move to a more stable API and that's why they made multiple releases and introduced some changes in the existing APIs. Since now we support different versions, and users can request either version, so we need to define a way to convert between our versions. These API changes can include addition, deletion and renaming of some existing and new fields in the existing CRDs, but while introducing these changes we need to make sure we should not break and continue the support for older versions. So to attain this stability we need to write some conversion functions and to understand the logic behind these, we first need to understand Hubs, Spokes and other wheel metaphors that are heavily used in writing those conversion functions.
A simple approach to defining conversion might be to define conversion functions to convert between each of our versions. Then, whenever we need to convert, we’d look up the appropriate function, and call it to run the conversion. This approach works fine when we just have two versions, but what if we had 4 types? 8 types? That’d be a lot of conversion functions. So to understand this problem and unblock all the new contributors, we want to present this talk where we can brief the concepts used to write the conversion function and also give a demonstration of how you can update any API and extend the support for older versions.

A

Hello: everyone in this video we are going to talk about how to write api conversions for mobility cid throughout this video. We'll cover several topics like what problem do api conversion exactly solve? We'll also demonstrate the use of conversion gen tool and briefly give an introduction about the concepts that we extensively used during api conversions like hub and scope, and at last, we'll give a demonstration where we'll show you how you can write the api conversions for converting between different apis.

A

So let me introduce myself. I am shivani single mts3 at vmware and I am a contributor to cluster api lifecycle provider for aws.

B

I am madur agarwal working at vmware and I am a member of kubernetes working in cluster livestock.

A

Before understanding how to write conversions, let's first understand what problem do: api conversions are.

B

We know that kubernetes clusters have multiple apis and each api starts with alpha version and then gradually evolves over time and as apis evolve. The cluster needs to update to the latest version.

B

The clients of the api server, like controllers and users, can also upgrade to the new versions, but it is not necessary that all the clients might upgrade to the new version.

B

So if the cluster has upgraded to the latest version, then what will happen to the clients who are querying the api server and the clients are having the old version.

A

So to solve this issue, we have api conversions for rescue. Let's understand this with an example suppose we have a google d server that store data in etc, with latest version that is v2. Our client, daniel, who is on v2 version, tries to write a test object on the server, then ap server stores the test object, aetc with version b2, then another client bar, which is on v1 version, tries to fetch the same test object.

A

But how will the api server convert the test object from version b2 in dtcd to the version d1 requested by the client? So to solve this, we will write api conversions. These functions converts the adcd data to the requested version.

A

Another use case is when client on older version updates data in adcd, which is on new version. This scenario might lead to a case where we can use the data.

B

Didn't get how data can be lost, let's try to understand it with an example, so I'll try to explain the example first, so we have.

A

An application which kubernetes application, which will have a crd, called registration. This crt represents the vaccination of an individual.

B

A

Have a v1 api of our registration, which is containing the registration.

B

Details of the course, like name, verified id and registration date later, multiple vaccines came in the market, so we updated our version to we do which added a new field called vaccination name and after that we found that one person can be vaccinated with.

A

B

A

And multiple times.

B

So we added a new.

A

V3 api, which has a list of vaccination details, so we will assume for for this example, that the storage version of our atcd.

B

A

So now we will demonstrate a case where a client tries to create an object of v3 version and stored it in etcd in v3 form. Then another client tries to fetch the same data in v1 version, update it and store it back into etcd. So we'll check due to the schema difference between v1 and v3 apis. Is there any data loss but before jumping to code, I will quickly walk you through about our projects, folder structure, so q vaccine is a cube builder project. Following the standard queue builder directory structure.

A

We have api config controller packages under api package. We have defined our v1 v2 v3 apis. So now let's start and apply a sample of v3 version. So here, if you go to sample, we have this covalent v3 registration yaml. So we will apply it and create an object of v3 version.

A

So we can see our registration sample v3 example is created. So if you see it in json, you can see the spec on the spec. We have name vaccine details and verified id. Let's fetch the same object in v1 version.

A

And store it in another yaml.

B

A

So if you see or the v1 conversion of our v3 example, you can see in the spec we have name and verify id and our api version is v1. So, let's try to update it.

A

In the name field we can add single and let's save it so now we will apply this yml.

A

And see our v3 object has been configured. So if you try to fetch this object again in v1 version.

A

You will see it's same as what we have just created. The spec name is updated. Now, let's see what happen if we try to access this object in v3 version, so we'll check. Is there any data loss or not.

A

A

So so we can see the spec here. We don't have the vaccine detail. But if you look at the previous example that we originally applied, we had the mixing detail. So this conversion and updation leads to a data loss.

B

A

We saw that data was lost by.

B

Conversion from v3 to v1 and back to v3, so the data was lost while conversion from v3 to v1, initially because.

A

V1 is missing the field vaccination details, which is present in v3.

B

A

When we convert the.

B

Data from v3 to v1, all the common fields are copied, but vaccination details is lost and when we convert back to v3, we see that the final.

A

B

Has lost information as compared to the initial vp? So how can we solve this problem, so this problem can be solved using annotations, so annotation is nothing but a key value pair which is stored inside an object. So, as you can see in our middle uh slide that we have annotations and we have stored the key value pair. So we solve our problem of data loss by storing the entire v3 object, as in the annotations.

A

B

A

Basically, you can see here that we have added an annotation.

B

Conversion data, where the entire v3 object would be present now.

A

B

Will convert from e1 to v3.

A

We can refer this annotation.

B

And fill the vaccine details and this way we won't lose the data. So this logic of storing data in annotations and restoring data from annotations would be a part of convergence. So we now sort of understand the logic of retaining data and converting from one version to another. But let's say: if there are multiple versions, then.

A

Do we have to write this logic.

B

For each pair of versions,.

A

No, we will use hubspoke model for this. Let's try to understand this with the help of diagram. If you see, on the left hand, side, we have several blue dots and each dot represent one api version and a.

B

A

Connecting these dots represent a function that we use to convert from one version to another, so there are lot of lines which represent. We have to write lot of conversion function, to convert between pair of api versions.

A

So to solve this, we move to the right hand, side and choose any one version as a hub and all other versions as I spoke. So in our example, we have we choose v3 as a hub and v1 and v2 as a spoke. So so in this model we will write conversions between the hub and all spokes. So in this way, let's suppose we want to convert from v1 to v2, so we will first use conversion from v1 to v3 and then from v3 to v2.

A

Here we can clearly see that in this model we reduces the number of conversion functions. So now, let's write the conversion for our example, but before that, is there any tool that we can use to write the conversions.

B

Yes, there is a tool called conversion gen, which can be used to write conversion functions automatically for us. So now we will use convergent end to write conversions for our project. Before doing that, we need to specify the hub. So in our project we have assumed v3 as the hub, so we have added a method hub in conversion.go in v3 and for the spokes that is v1 and v2. We have added a file called dock.go which contains a conversion, then tag and the hub.

B

So this conversion, then tag signifies to convergent gen, that it has to write conversions from v1 to v3 and vice versa. Similarly, we have added this for v2, so convergent end will write convergent functions for conversions between v2 and v3. Now we will run the conversion tool. So, as you can see, this is the conversion.

A

B

Which contains input directories as v1 and v2, so conversion gen will scan these packages for these stacks and write conversions for us in our output file called z, generated dot conversion. So this will take a few minutes.

B

Okay, so now conversion gen has completed and we can see that it has created conversions files in packages v1 and v2, and also it is giving us some warnings.

B

So so these bondings mean that conversion then could not write conversion for some of the fields and we need manual conversion for them. For.

A

Example, while writing.

B

Conversion from v1 to v3 it could not write the conversion for registration date and also for the conversion from v3 to v1. uh It could not write conversion for vaccine details, so let's check out the conversion.go file in v1 package.

B

Okay, so here we can see that uh the there is a function which converts from v1 to v3 and the input is v1 and output is v3. So conversion, then, is copying the common fields from v3 to v1, but for certain fields which are missing uh in v3 it is. It requires us to write the manual conversion, so it means conversion, then, can help us to an extent, but for complex cases we need to write convergence manually ourselves. So now we are going to write them.

B

We will handle the conversions for uncommon fields which could not be converted by con convergence. We have added two methods here: convert to and convert from, convert to convert spoke to hub that is v1 to v3, convert from converts hub to spoke that is v3 to v1 inside these methods. For conversion of common fields, we will reuse the auto-generated methods created by convergentin for the conversion of uncommon fields. We will use the annotations that we discussed earlier.

B

Let's go through.

A

B

From method here we will store the v3 object in annotations of v1 and we will restore the data from these annotations, while converting back from v1 to v3. As we.

A

Can see in convert to method.

B

If there is a annotation present in v1, then we will restore the vaccine details from it and in this way we preserve the data during conversion.

B

A

B

More implementation, details in the github and we will link the github at the end of this presentation. Let's go through an example to understand the effect of this conversion.

A

So now we will rerun our previous example, but this time using conversion. So let's apply uh our config, our yammer for v3.

A

So we can see, we have a v3 object known as registration v3 has been created. Now, let's take it.

A

In a form of v1 and store it in some file.

A

So if you see in this file this time we have this annotation piece field introduced in the v1 version, so we can see the annotation is a key value map here the key is covin dot, juv dot in slash conversion fund data and in the value we have the whole v3 object, uh yaml that we applied earlier.

A

So now, let's again update it.

A

And let me add the name now I'll apply it again.

A

And we can see that this object has been configured now. Let's try to fetch this object in a form of v3 and we'll see whether we still have the data or we lost.

A

So we can see now in this spec. We have the vaccine details as it is that we applied in our original v3 object. So if you check the ml, then here also, we have the vaccine details like co-vaccine schooling. Similarly, here we have so we can say that using conversion we preserve the data during the conversions to access our code. You can refer to this link and if you want to reach out and reach out to us, then you can refer this handlers.

A

Now we are open for any q, a.

A