Cloud Native Computing Foundation KubeDay Japan 2022, 9 Jan 2023

Previous Meeting

Next Meeting

⏯

youtube image

►

From YouTube: Autoscale Your on-Premises Bare Metal Kubernetes Clusters - Shukun Song, Fujitsu Limited

Description

Don't miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe in Amsterdam, The Netherlands from 18 - 21 April, 2023. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Autoscale Your on-Premises Bare Metal Kubernetes Clusters - Shukun Song, Fujitsu Limited

On an on-premises bare metal Kubernetes cluster, autoscaling is a difficult task but important as it can increase the efficiency of resource usage to help organizations reduce cost, for example power consumption. In this talk, Shukun will introduce the idea of combining 3 projects, Cluster-API, Metal3-io and Cluster Autoscaler, to achieve autoscaling on a bare metal cluster. Metal3-io is a CNCF sandbox project which aims to provide a Kubernetes-style way to manage bare metal servers and Cluster-API, as a sub-project of Kubernetes, is developed to manage Kubernetes cluster lifecycle. Combining these projects, it is easy to manage bare metal infrastructure and build Kubernetes clusters on it. Machines and clusters are managed as Kubernetes resources and all the provisioning and deprovisioning work can be done automatically with just declaring or configuring those resources. Also, it is possible to combine Cluster Autoscaler as well to increase the efficiency of the physical resources, for example, concentrate applications deployment to some servers if the total usage of the cluster is low, to spare the unused resources. Shukun will introduce the detail of these projects, explain how they are interacting with each other and show how to use them.

A

In this session, I will introduce how to use cluster API metal, cube and classical scalar to scale on-premise the motor kubernetes clusters. um This the contents of this presentation, a pretty repeated with the previous one. So you know okay, my name is shukun I work for fujits and currently focus on biometric provisioning on kubernetes and today's contents first is some background and then I will introduce how the three-way projects work, how to use them and to finally show you a demo, okay, the background.

A

Nowadays, we still need on-premise for some readings like a need for for control of the entire entire environment, or need for some specific level of security and the privacy or maybe the compliance requirements. But we all know that managing and on-prem with on-premise environment can be very tedious because we need to do everything by ourselves. So how could we reduce the operation cost and as here we are at qubit day?

A

The solution I want to introduce today is a kubernetes way, combining cluster API metal, cubed IO and the clustered scalar to automatically create and manage kubernetes clusters, which are provisioned by biometer servers. So kubernetes we we are all very familiar with, is a great tool for deploying applications and the rest. Three projects are, can be used to manage the infrastructure.

A

Well, first is clust API, it extends the kubernetes API to create and manage the kubernetes Clusters, and it is a kubernetes community, subproject created by the cluster lifecycle. It also runs on kubernetes cluster and that cluster is called a management cluster. The cluster is created by and managed by. Cluster API is called the workload cluster then, and the infrastructure can be a cloud or bare metal, but no matter what infrastructure infrastructure is used. There must exists in infrastructure provider component that Provisions machines and networks for our clusters and for bare metal cube.

A

Diode can be that provider.

A

It is a barometer provision interval for kubernetes with it we can install OS to a barometer server in the kubernetes way and the last one is cluster or scalar which adjusts the size of a cluster of a cluster according to the workload and using this cluster API, it watches a workload cluster and the sender scale request cluster API when necessary and combine the the three projects. We can automatically create a manage and scale a metal kubernetes clusters.

A

Okay, now before talking about the details of the three projects, there's a little bit background knowledge or, although I believe that everyone's very familiar with, which is the way in which kubernetes works. Okay, kubernetes use these objects to describe the state of the system and controllers to watch those objects and make the results that happen.

A

So this is the fundamental mechanism of kubernetes, and we also know that kubernetes provides a easy way to extend its API, which is crd and the custom resource definition with crd kubernetes can understand the new kinds of objects and then combine with custom controllers to control those new objects. We can make kubernetes do what we want.

A

So the projects we are talking about today also use this mechanism and okay, we're keeping this in mind. Let's go further about how the three projects work. The first is cluster API again cluster API extends kubernetes API to create and manage kubernetes cluster and to do this job.

A

Cluster API first defines two new crds, the cluster and the Machine. The cluster, which represents a workload cluster and machine, represents a infrastructure component hosting a kubernetes node and in that component, can be a virtual machine by a metal or even the container, which works like a machine.

A

With these two objects, we can describe the basic state of our cluster, and next we need some controllers to watch these objects and the request, resources from infrastructure and put machines into kubernetes nodes and to decouple from infrastructure specific Logic. The cluster API uses the idea of provider.

A

There are three providers defined here: the infrastructure provider, control plan provider and bootstrap provider. The infrastructure provider is responsible for requesting resources from infrastructure and the metal cubed IO is an infrastructure provider, and the bootstrap provider provides data to a machine for bootstrapping it into a kubernetes node and the control plan provider is used to create an encryption plan for our cluster.

A

Now each provider needs to Define some theories and Implement related controllers, so that the cluster and the Machine object just need to interact with the those infrastructure, those provider 30s and do not need to worry about any detailed provider. Specific logic by the way the A blue arrow in the image here means the reference referring to another object and the black one. That means the creating or modifying an object.

A

The cluster object contains the inter-cluster network configuration like Port, cider or serviceider, and also references to the infrastructure, cluster and the control plan objects and then the infrastructure cluster object contains information about the infrastructure Network, which the cluster runs on and also some other configuration for our cluster. And here the example here I used is a very specific to mat. Cube dial foreign object contains templates of infrastructure, machine and bootstrap config objects, and those templates will be used to create those objects.

A

And an infrastructure machine object represents a machine on the specific infrastructure. It contains information for the controller, to request the machine from the specific infrastructure and then provisioning and bootstrapping it into a kubernetes node. The bootstrap part information is actually from a secret generated from the bootstrap config objects.

A

Okay, the bootstrap config object contains information about how to bootstrap a server into kubernetes node, usually through cloudy neat and a secret. Storing those data will be generated and referred by the infrastructure, machine objects and the last is the machine objects. It contains the references to the infrastructure, machine object and bootstrap config objects, and also it has it knows the name of the cluster which it belongs to, and there are many other theories defined in this architecture, but to create a kubernetes control plan.

A

We mainly need three objects: the cluster infrastructure cluster and the control plane object after the three objects are ready. The control plan provider will start to create the infrastructure, machine objects and bootstrap config objects and then create a machine object which has reference to the previous two objects.

A

And after that, the Machine controller will set only reference to the infrastructure machine and the bootstrap configure object to notify that to notify each provider that these objects are consumed by some machine and then the bootstrap provider will know. Okay, I need to generate a bootstrap data for that machine and the infrastructure. Machine controller will know, start to use those data and request a machine and bootstrap that machine into a kubernetes node.

A

Here are the sample implementation implemented by the class API Community, which are using cubeadm to bootstrap a machine and when creating worker nodes, cluster API has defined another two crds machine deployment and machine set. Their logic is familiar with the control plan provider and the relationship between them is just like Port replicate and the deployment in kubernetes foreign.

A

Provisioning tool for kubernetes and also is in the infrastructure provider for cluster API.

A

There are two main components in beta cubes, the cluster API provider, meta cubed, or we can call it a cap M3 and the permit operator. The captain, 3 is the infrastructure provider, and the government operator is the Bermuda provisioning tool and the crd defined by them are actually with cubed cluster and made equipment machine and the permit host a permit host object represents a barometer server.

A

It contains information such as BMC details, boot mode, OS, image, OS image, user data and so on. The metal operator will use this data to access the barometer servers, control the power and do the provisioning work.

A

Now, to use the metal cube along with cluster API, we first need to create Bermuda host objects for each parameter servers. This barometer host objects are the server inventory ready for provisioning class API machines.

A

Then we need to create a cluster API object that will generate machines for us here. I use machine set for the instrument example, and also we need to create related metal, cube machine templates and the middle Cube data template object.

A

The medical machine template is used to generate a medical machine which is a consumer of the barometer host object and The Meta Cube data template is used to generate network data and the metadata used for cloud they need for when provisioning our parameter server and like we talked before the machine set controller, will use the those templates to create a medical machine and cubadium configure object and also after that, a machine object will be created and referring to the previous object, two objects, then the Machine controller set on the reference and each provider will start to start the provisioning work, which is generating the secret secrets that storing our bootstrap data and after all, the secrets are ready.

A

The medical machine controller will choose a suitable permit host set part of its spec, including image and reference for user data metadata and network data, and also the consumer, and the next. The metal operator will detect those changes and start the provisioning work you this data.

A

About about provisioning, Biomet operator uses openstack ironic project as the back end. There is an ironic server running along with the barometer operator, accepting requests from the permit operator and it will access to the barometer server change. The boot option and Power on the server through its BMC start a ram disk and runs an agent called ironic python, agent or EPA, and then the IFA will talk with ironic, inspect hardware and write data to the hardware disk.

A

Okay, this is the professional workflow. It shows how the image is written to the server when using pixel boots and after right after written to the disk, the server will reboot and if we set the character, bootstrap data, then a kubernetes node will be initiated.

A

Okay, the last one is cluster or scalar.

A

You can adjust the size of a kubernetes cluster according to its workload, to implement this.

A

The core mechanism and the idea it uses is node group. The node group is a set of nodes nodes that have the same capacity and the set of labels.

A

Cluster or scalar defines an interface also called anode group to control a node group, though this interface has methods like increase size or digital nodes and so on, and it is implemented by each cloud provider and cluster API is one of them, and the cluster scalar will check the parts deployed on the target cluster periodically if there exists a port whose status is pending because of lacking resources.

A

The cluster scalar will then choose a node group and calculate if scaling up that node group, whether the new node could host the pending ports or not. If so, how many new nodes are needed, and after the calculation, the cluster scalar will call increase science method to scale up that node group and requests will be sent to the related cloud and the new machines will be created and boot into kubernetes. Nodes scaling down is similar with this Logic.

A

The cluster scaler checks the resources usage of each node to see if there are nodes that are not needed anymore. If we exist, it will start scaling down by calling the delete nodes method.

A

Foreign okay, cluster API is one of the cloud provider. So if you're using cluster API, a node group is actually a machine set or machine deployment object and the scaling job is done by changing the The Reflex field in their spec.

A

Now, because the auto scaler needs to not only check ports running on the workload cluster, but also modifying a machine set or machine deployment on the management cluster, so the cluster of scalar needs to need access to both of the two clusters.

A

Okay, the next part is about how to use the three projects.

A

Cluster API Community created a tour called cluster carto, which can deploy related, crds and controllers and also generate number files for objects like cluster. We can use cluster control in each command to create a cluster, API, core components and each provider to deploy a metal operator and ironic. The medical community provides a script deploy.sh.

A

This script can deploy by multiple Creator and ironic in cluster as a part, and we can also deploy ironic outside of the cluster using another script around local ionic.sh, but to run both of them. There are some variables need to be configured here are a part of them and if using deployed data sh, the these variables are defined through one of these two files and for around local ionic.sh.

A

They are defined through system environment or we can use default values defined in the script itself.

A

Okay, after all, the components are ready. We just need to create objects, they are cluster, kubadium control, plane, machine deployment, kubadium Cube, config, template and medical cluster medical machine, template meta, Cube data template and also by murder hosts now and with all the needed objects created. The provision job will start automatically about cluster Auto scaler.

A

It can be deployed using Helm and the chart can be downloaded from the git from GitHub and when installing, we also need to set some specific variables to specify cluster API as the cloud provider and also specify how the cluster or scalar could access to uh workload. Cluster and management cluster.

A

The cluster API mode here defines the how the cluster of scalar could access to the Clusters and in this example, I am using cube cubeconfig in cluster mode, which means the class. The old scalar will run in the management cluster and use cubeconfig to talk to the workload cluster and use in-cluster authentication to access the management cluster and the cube config for workload. Cluster is specified through cluster API, cable, config Secret.

A

The value is the name of the secret stored in the management cluster.

A

To control which machine sets or machine deployments should be scaled. We need to add these two annotations to those targets objects. The clustered scalar will monitor any machine set or machine deployment that contains both of these two annotations.

A

Okay, finally, the demo part in the demo I will first create a one Master One worker cluster, using two metal servers and then scale up that cluster.

A

This image shows the network of the entire entire environment, with the management cluster needs to access to The, BMC and also access to provisioning network, which is which are which, in which ironic rounds and also the barometer Network, which is in which the kubernetes the workload cluster kubernetes runs.

A

A

Well, because I'm using bare metal and the provisioning of the build metal server is very time consuming. So actually this is a video record and and and I have already created uh all the components you can see here. The cap incaping copy system namespace, there are cluster API controller manager and there are there.

A

There exists, uh queradium controller airplane, a control, player controller and Cuba ADM, bootstrap controller and also cap, M3 controller manager and permit operator, and ironic and I have created three Bermuda hosts the their state is available, and if, uh if you, if I get in the detail of the barometer host, they can find.

A

The hardware information in their status, for example, here the leaks and the CPU and the storage.

A

A

The next we need to create our control cluster and the control plan. The first is the cluster object. It looks like this: it has cluster network inter-cluster network configuration and the control reference and infrastructure reference, and the next is the cubadium control plan object.

A

Well, it's a very it's a little long and you can see here. We we have a infrastructure reference to a metal, cube machine template that will be used to generate a meter, cubic machine objects and also we have template for uh for both simple data, and it is in the format of cloud init. We have I, have defined a lot of commands to run and also a lot of files to uh be created in the server.

A

Okay, the next one is the machine, medical machine template and in the template we have the data template to use to generate, to use, to generate the network, data and metadata, and also the image information for the server and, as you can see here, the here are the bitcube data template and it contains information about uh metadata and the network data.

A

And last one is MailChimp cluster. It only contains the control plane, endpoint and also above value, indicated that we, our cluster, has no cloud provider and after created those objects, or can we can get the cluster that says the phase is provisioning and uh because they are, the controller is checking whether the infrastructure cluster is ready or not, and if we get cubadium control plane object here. There are nothing in it, not because the cluster is not provisioned yet and a few minutes later, a few seconds later.

A

Okay, the because the metal cube cluster, our infrastructure is ready. Now, so the cluster is going to the provision phase and we can see the cubadium control. Plane has replicas 1 and updated one and and it is unavailable. So it is in the initialized phase and initializing phase and we can see a new machine is created and in the provisioning phase and the metal cube machine is also be created about 30 seconds ago and bare metal host.

A

One of the metal hostly is chosen and it is provisioning and the power is online and the consumer is our medical machine object and almost after 15 minutes in my environment, the data, the OS image, will be raising into the the metal host the remote host. So.

A

The state yeah here the state after 50 minutes, the state of the permit host is provisioned, which means the all the data has been written to the server and and it. uh But but if we get the medical machine, uh it is not ready not because the server gets rebooted and the old command is running now and kubernetes is not a up yet and after another, two almost another two minutes.

A

Okay, here you can see another two minutes after another two minutes.

A

The machine has a provider id set and the face is running and the metal cube machine also has the same provider id and the red is true: the provider it is generated by metacube automatically and the and will be copied to the machine objects, and if that is, and if that provide is that the machine will consumed be considered as runny and ready, and we can see here, the control plane is now initialized and the API server available, and now we can access to our server to get and you can get a node and also the cube config for our workload.

A

Cluster is stored in the management cluster as a secret, so the my cluster can be config. We can get this use this to access our cluster.

A

Okay I have stored the cube, config and get a note and also get pulled.

A

Now the next is I need to create a machine deployment to create our create a worker for our cluster. The machine deployment has the bootstrap template to create a bootstrap config and a template to create a machine, a medical machine.

A

Okay, the there are similar ways. The so I was similar with the control plane. So I skipped this and after these objects are created and.

A

Here the machine deployment is in scaling up, phase and a new machine is created and the new medical machine is created also.

A

And after another, two okay, the machine object has a rep has a reference to the bootstrap config and medical machine object, and after another and yeah and a new another bmh has been chosen and it is in provisioning phase.

A

Okay, after another to 20 minutes, it will become worker, well, ready, okay, the worker is ready now and they have provider id set and the Machine deployment is in running phase and or get note. There are two notes in our workload cluster.

A

The next I will use this script to create auto scaler, okay, the autoscaler is created, and you can see here the port is running and to trigger the scaling up. I need to ensure that our machine deployment has the right annotations here.

A

Okay and now next, to trigger the scaling up, I will create a deployment. The deployment the port deployed by this deployment does nothing but a request for for CPU course, and because the worker has 20 CPUs, so, okay, the port is running, and so, if we, if I add this to six- and there will be one portal impending status, impending phase- okay.

A

So here the report they're append Import in the workload cluster and you can see the machine deployment is scaling up again and the new machine is recreated and the last moment host is chosen for provisioning. Okay, that's all.

A

Okay, the last is some information about the community.

A