Cloud Native Computing Foundation Kubernetes Community Days Bangalore 2022, 11 May 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Cluster API 101 :How the APIs works?

Description

Cluster API is a Kubernetes sigs projects that provides a declarative way to create and manage Kubernetes clusters on many providers like Azure, AWS, GCP etc. In this talk, we will help understand how Cluster API is a consistent API across providers which not only enables broader use cases like hybrid and multi-cloud, but it also provides standardised tooling that can talk to this interface by providing blueprint of it’s components on top of which the provider components are built-up. They will touch upon the various involved controllers in cluster API and how the cross communication happens with the hooked in provider controllers in order to provision a Kubernetes cluster with a demo.
The audience should walk away with a good understanding of Cluster API projects, it's involved apis, machinery and how infrastructure provider gets integrated with Cluster API. This talk should be different that other existing Cluster API talks as it talks mostly around the APIs and how different involved controllers communicate and perform various tasks.

A

Hi everyone- I am ankita and I am an engineer at vmware, I'm working on cluster life cycles. I have recently got introduced to cluster api community and have started contributing actively to kappa. We are going to do a one-on-one session of cluster api with the demo over to you, ashitosh.

B

Hi everyone, I'm ashitos and I'm an engineer at vmware working on kubernetes lifecycle team. I too got recently introduced to the cluster ap community and have started contributing to capg. Prior to this you know I was working with hop anybody's project.

B

This is the agenda that you can see on the screen and I would like to invite ankita to start this. All.

A

Thanks ashitosh, so, as we know, kubernetes is a complex piece of software. Provisioning and managing kubernetes cluster have always been a challenging job installing and configuring all the components such as cubelet, scheduler, etc, generating certificates etc is very time consuming. When done manually.

A

Imagine the situation when you have to manage hundreds of clusters, provisioning production grade clusters upgrades and day-to-day cluster operations can be tedious. This situation gets more worsened when you have to operate kubernetes clusters on different infrastructure providers, for example, aws, azure, gcp, vsphere, etc. Though cubadiem is a tool that solves the problem of installation.

A

It does not help with the complete life cycle and management. Cluster api is a kubernetes 6 project that addresses all these problems. It introduces declarative apis that can be used to manage the life cycle of one or more kubernetes clusters.

A

Before we get a little deeper into cluster api, I would like to invite asha those to help us to do a recap of how kubernetes controllers work.

B

Thanks akita kubernetes controllers are kind of an infinite loop that keeps watching the current state of system and tries its best to convert the state to the desired state that is captured in the manifest, for example, aml file. You can see a sample code on the screen that tries to explain this statement.

B

This is based on a declarative api pattern where users just specify how they want the system to look like and provide instructions to make sure the system looks like as they want. Reconciles and controllers are the loops that receive the desired state and work to get the system to desired state.

B

This enables a self-healing feature and make sure that the state is not desynchronized, unlike the imperative apis, where we actually specify instructions which can be missed due to a networking thing or something else.

B

Humanities allows us to extend its declarative apis and use it for other custom operations by using crds, we can specify api and write a controller that will act upon the api to do operations.

B

Often there are certain tasks and operations that can be automated via this controllers, and these are sometimes termed as operators too.

B

Cluster api is one search operator, uh that is, that automates, the installation upgrades and other cluster operations, and all that too declaratively now we'd actually specify kubernetes cluster by writing couple of configuration yaml, which should always yield the same. Kubernetes cluster agrees to immutable principle. This can be of great help in ci, siri and other systems.

B

Clustered api offers some declarative apis that are pluggable to a different component, known as provider which actually performs the operations, for example, creation of vms networking etc. Around the infrastructure, if I can just uh reiterate, uh capi, is just a two tool. You know that will run as a pod in your kubernetes cluster and then provider, for example, cloud api, pro cloud, cluster, api provider, azure or cluster api provider aws. You know these are also some of the tools that will run as a part in a kubernetes cluster.

B

Now this pods or the tools works in conjunction to help us provision kubernetes cluster or manage the life cycle of this things.

B

As you can see on the screen, the small left box, it is a simple kind cluster where cappy and providers are installed. The middle one is similar just that it is a production grade cluster with hj, which has capi and cap providers component installed.

B

The idea of bootstrap cluster is to help onboard the users easily use the cluster api itself to create a kubernetes cluster and then the created cluster can be further used to create and manage multiple cluster which can be used to deploy workloads. Hence this is strummed as a management cluster once the bootstrap cluster is created. Here is what happens.

B

A production grade, kubernetes cluster is created by using the bootstrap cluster. All the cap and required components are migrated to this newly created cluster from the bootstrap cluster.

B

Now this newly created cluster is termed as management cluster, which can be used to create workload clusters on different cloud providers. As you can see on the screen screen, the top box is like the azure cloud, and the bottom is aws. Where we have couple of workload clusters technically, we could create multiple clusters from the bootstrap test itself, but this exercise is done to make sure that the management cluster is production, grid scalable and meets the enterprise needs.

B

Now I would like to invite ankita to go little deeper into the core, kp and other apis.

A

Thanks ashotosh, so we have corecappy, uh which consists of some apis and crs, that is, cluster machines. Machine sets machine deployments that are reconciled by their respective controllers. Similarly, cab providers also has apis and crs, for example, in case of cluster api provider azure, it is azure cluster azure machine which is again reconciled by the respective controllers, which actually perform the operations on the infrastructure, for example, creating or deleting the virtual machines, creating networking, subnets, etc.

A

There are other apis also offered by capian providers, but we are just excluding it here for the sake of simplicity, the crs have some cross reference across each other, and also the status is updated on them by their respective controllers.

A

This sets the wires across various controllers, either of the core corecappy or the providers to take required actions at appropriate time. So this coordination is kind of enabled by status and conditions that are updated on the cr objects.

A

Also, this cross referencing of cr objects is what gives the plug-in capability and provider works in conjunction in conjunction with the core capping.

A

This can sound a little complicated, so we will use an example to understand how this works, but before diving into the example, I would like to mention some high level things cappy tool hooks in with providers such as cabsie and kappa and others to help provision and manage lifecycle.

A

The providers are responsible to carrying out the infrastructure actions and updating the appropriate status and conditions.

A

Capi has also in-built providers such as cluster api bootstrap queue bm, whose job is to bootstrap a kubernetes node, though capi is flexible and can be used with bootstrap provider other than cubadium capi also has the control plane providers which are responsible for managing control, plane machines managing can mean upgrading control, plane machines, making sure kubernetes components such as hcd schedulers are ready.

A

Now let us go through the example to understand the above. We'll try to understand this example with the reference of cluster api azure, though similar workflow exists for other providers too. So let us say we want to create a simple kubernetes cluster on azure provider. So, to do that, we need to apply few of the yammers at the minimum on the kubernetes clusters so that it contains both the cappy and cabzi providers uh already, which is already installed on them. So these are the following crs that has to be applied.

A

That is, the cluster object comes from cappy azure cluster comes from cabsie cubadium control plane, uh that is, the default control plane provider is inbuilt to capi azure machine template also comes from gabzi machine deployment comes from cappy and cubadium. Config template comes from the uh cab bk, that is the bootstrap provider, which is also in the default provider used for bootstrapping in gaapi, and we have azure cluster identity, which is uh coming from the capsi provider, which is used for the authentication purposes.

A

So a cluster is defined using the cluster crd by default, cluster resource spec defines how networking is going to be set up. What siders we are you going to use and what domain name to use for services and cluster resource status saves the endpoint ip address and port cluster controller takes care of reconciling the cluster crd config note that it is referencing azure, cluster infrastructure provider, api and cubarium control, plane the control plane provider api, it waits for the infrastructure creation to complete and then sync the cluster status. Accordingly.

A

Note that azure cluster is also created which actually deals with the cluster created on azure infrastructure. It makes sure that control, plane, nodes and machines are ready to use and overall, cluster status is healthy. In addition, it creates a cube config secret for workload clusters so that users can have access to the workload clusters.

A

Machine deployment, machine deployment like the cluster is created by user. It creates and provides declarative updates to machine sets which, in turn, creates or deletes required machines. Machine deployment works similar to a kubernetes deployment.

A

It reconciles the changes to a machine spec by rolling out changes to two machine sets the old and the newly updated it takes care of unowned and unmanaged machine sets in the cluster and manages the scaling up or down of the machine sets. Finally, the machine controller creates a provider-specific machine by using a template. In this case, it is azure machine template. You can notice the reference on of the ammo on the screen.

A

Cubadium control plane it is responsible for managing a set of control, plane machines, kuberium control, plane provider is a default control, plane provider controller used in the cluster api, which takes care of bootstrapping scaling deletion of control, plane nodes without disturbing the whole cluster forum.

A

Unlike worker nodes, cluster control, plane machines do not have machine deployment, as you can see on the screen. The api has the capability to ingest configuration flags for control, plane, kubernetes components such as api server, controller manager, etc.

A

It is responsible to instantiate a kubernetes control plane with services like fcd, cube, kubernetes, api server, kubernetes controller manager, scheduler cloud controller manager, cue proxy and code dns.

A

The q8m config template defines how the machine should be bootstrapped and joined to the cluster, using kuberium from a bootstrap provider notice that this configuration was referred back in machine deployment and for all the machines of that machine deployment. This bootstrap config will be used. I would like uh to invite ashutosh now to continue further.

B

Thanks ankita, now you can see on the screen the resource hierarchy. uh We now understand how machine deployment manages worker machines using similar constructs like a deployment to part management in humanities and how cappy uses templates to create the actual azure machines by using azure machine templates, which is finally reconciled by the captcha.

B

As ankita has already mentioned. The cluster object helps set up the pod, sider and other cloud agnostics settings setting up the honor references and statuses. It basically coordinates the overall cluster operations. Now let us see how capture objects looks like.

B

You can see the azure cluster yaml on the screen, which should be also created by the user. This controller is responsible for provisioning, the azure infrastructure, which will provision networking security, grips the load, balancer and some other components with as our best practices baked in it, references one or more cabbage specific objects, for example azure cluster identity, and this is for authentication purpose to the azure cloud.

B

You can again see some yamls flashing on the screen. This is azure machine template the left side, one is for control plane and the right one is for worker node. The actual azure machine is created by using this template by the capi machine controller.

B

To summarize, the provider-specific controllers actually reconcile their crs by creating or updating resources on the infrastructure. Now let us jump into a demo and see all of this in action.

B

For the sake of time, I've already created a kind cluster and installed capi and capsi using cluster ctl, just to mention that cluster ctl and cli tool, which is very helpful while trying to operate with capi I've, also set up required environment variables and created a secret to authenticate to azure cloud. This is very straightforward and can be found on the cluster api.

B

Talk link I'll share the relevant links in the next slide, but let me just pull out the parts here so this these are the pause in the kind cluster you know you can see. This is the captcha controller manager, and this is the cafe controller manager. So this is actually cloud provider azure, and this is the control plane provider that we were talking about in previous slide, and this is the q8m bootstrap provider.

B

uh Let me show you the secret that we just created.

B

So this is the secret that I created. uh This is used, for you know, authentication purpose to hazard cloud and now let me just show you the yaml that I generated using cluster ctl.

B

This is the clustering ammo and then we have azure cluster and you can see the cuban control plane where we have arguments for api server, controller manager and we can see the machine template reference for control, plane machines and we want one replica and the version should be 1.22.0.

B

This is the azure machine template for control, plane machine, and this is the machine deployment where we have replica count as one and the version is 1.22.0.

B

I'll just apply this one.

B

Got applied, cube, ctl, get clusters and we can see that the phase is provisioning. We can also use cluster ctl to describe this one cluster ctl describe cluster kcd demo cluster and we can see what phase the target cluster is in. So you know right now we have like cluster infrastructure, which does not have a ready state to true, so it's still in the provisioning state. So let us pull cube, ctl get azure machines and it is waiting for cluster infrastructure, so it will take some while to get this vms created on the azure cloud.

B

Well, it takes a little while to get this created, so we have again fast forwarded the video for the sake of time and now cube ctl get azure machines.

B

We can see that the machines are succeeded, let's also run cluster ctl describe command and we can see that the status the status true for control, click for cluster infrastructure control, plane and worker uh yeah. We can see a warning here, and this is because we don't have a cni provider installed yet so before we do that. Let us pull the cube config for this target cluster, so cluster ctl get cube config and the name of the cluster is kcd cluster and we will do demo dot, cube config.

B

And let us use this cube config to see the nodes, cube, ctl, get nodes, hyphen knife and cube config demo, dot config.

B

Config, yes, we can see two nodes, one is control, plane, node and one is worker node and the version is 1.22.0.

B

uh Let us install a cni provider here, so I've already ran this command sometime back, so I'm going to search for it. Cube ctl, apply.

B

Hyphen hyphen cube config.

B

Demo, dot cube config, and this should apply the calico cni yaml on the target cluster.

B

Cool, let us save the nodes already yet.

B

It can again take some while to get the status updated to ready, but this should be quick.

B

Yes, the notes are in ready straight now and now I'll just quickly try to scale up the worker machine I'll, add one more virtual machine to do that. Cube, ctl, get machine deployment and I'll edit this machine deployment. Let me just copy this one and cube ctl, edit md and I'll search for replicas.

B

I'll make it two, it is getting edited I'll again run cluster ctl describe command and, as you can see, you know it is waiting for available machines, because now we want two replicas.

B

So, let's see, if that got created, cube city will get machines.

B

We can see there is one machine that is in provisioning state from last 21. Second, cube city will get azure machines.

B

And this is the new machine that is being created.

B

Cube ctl get azure machines, so we can see that the bootstrap is in progress.

B

B

Let us see yes, so the azure machines have succeeded now and we'll do once more cluster ctl describe.

B

And yes, everyone is true. uh Now let us see that we can actually see the nodes.

B

We can see this new node that got it at 93 seconds back and now we will try to upgrade this qr test cluster to 1.23 cube ct will get kcp. Let me pull the kcp I'll edit. This cube city related kcp.

B

B

First, we will.

B

I missed kcp here will edit jcp, so we are trying to update control plane first and we will make it to 1.23.

B

We'll save it and we can cube ctl get azure machines. We can see that there is one new azure machine that is coming up, so it actually creates a new azure machine with 1.23 version.

B

We can see that this one succeeded now cube, ctl get machines and the other one. The other machine is in deleting phase, so the older one should get deleted in some time. Let us see what is the state of node here.

B

um Yeah, so the output we can see on the screen, so the older node has scheduling disabled and after some time this node will disappear and we can see the newer node that has come came up has version as 1.23.

B

So this is about upgrading the control plane machine. Now we could actually upgrade the virtual machines uh to do that. Let us just see if that node got deleted.

B

So, yes, the node got disappeared, and now we have a new control plane node, which is in 1.2 3.0. Let us now upgrade the worker machines version, cube, ctl, get machine deployment and we'll do cube ctl. It did machine deployment.

B

B

You mark the version as 1.23 you save it.

B

B

And we can see that there is a new node, that is, there is a new machine that is coming up with 1.23, which is in the provisioning state. It really takes a while, and the newer worker machines will come up with 1.23 version and the older one will be decommissioned.

B

uh We can see that we have two new machines that came into running state with version 1.2.3, and there is one machine that already got deleted and one that is being deleting state. uh Let us see, cube city will get no siphon, hyphen cube, config demo, dot, cube config.

B

And we can see that we have two vertical machines with 1.23.0. So now we have like every machines with 1.2 3.0 version. So that's how we upgraded the target cluster 2 1.23, and it is so simple we can simply upgrade by editing ml or we can create uh scale up the worker machines by editing the yaml.

B

So you know this is the declarative pattern that we were talking in the slides uh where you just do cube, ctl commands to operate, or you can use cluster ctl command to operate with this cappy tooling, and cluster ctl command is more cappy aware, so it is very helpful to debug or to analyze, what's happening in your kubernetes cluster, so that is it for the demo and I will hand over to ankita.

A

Thanks ashitosh, so now, if you want to get involved and contribute to cluster api and its providers, so here are some links that could be helpful to get started with and explore. What actually uh is in around the cluster api and its providers and feel free to join the community slacks for different cluster api providers, including the cluster api core channel.

A

Also, you can suggest some of the ideas and report issues bugs on the respective github repos if you have any, and we hope that this session was helpful for you and now we'll move to the q a session. Thank you.

A