Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2020 - Virtual, 4 Dec 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: High Performance KubeVirt in Action - Huamin Chen, Red Hat & Marcin Franczyk, Kubermatic

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2021 Virtual from May 4–7, 2021. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

High Performance KubeVirt in Action - Huamin Chen, Red Hat & Marcin Franczyk, Kubermatic

This talk details a real world solution design of a high performance KubeVirt for running mission critical enterprise workload. KubeVirt brings Cloud Native Virtual Machine management to Kubernetes. It unifies workload orchestration across Containers, Virtual Machine, as well as Serverless. Solution designs around KubeVirt, however, are sparse as of date, especially in networking and storage. On the other hand, workloads running on Virtual Machines often demand high performance and isolation enhancement. In this talk, Red Hat and Kubermatic jointly share with the community their customer engagement experiences of building a high performance KubeVirt environment by integrating with Gardener and multiple CNCF projects, including Kubernetes, KubeVirt, Rook, and Kubernetes Network Plumbing Working Group.

https://sched.co/ekDs

A

Greetings, it is our pleasure to meet each other at this cubecon 2000 virtual event. My name is juan chen. I work as a office of cto as a redhead.

B

And my name is martin francik, where I work as a software engineer at kubernetes.

A

This talk is organized in this way. First, we go through the cube verse project. Kuvroz is the cncf project that manages the virtual machines the same way as kubernetes, thus for containers, then we introduce gardener projects. Gardener is the open source project, that's managed, kubernetes clusters, life cycle and our offers they run an a2 operation.

A

It runs on a number of infrastructure platforms, including pure words. We explain how kubrick's features can help gardeners to deploy high performance and highly secured kubernetes clusters.

A

We focus mostly on the networking and storage features that we, how to use motors cni for high performance and fully isolated networking configuration and how to use data volume and clone to accelerate virtual machine deployment at scale.

A

Ship first joined the cncf sandbox over a year ago in 2019. It enables workloads that run instead of vms to be deployed on the same kubernetes cluster. As containers are running.

A

It's used some of the kubernetes native objects, such as positions, volume, claims and resources claims. It provides a convenient way to describe virtual machine configurations and their state. In this example, a virtual machine that consists of virtual natural hemispheres and the virtual disks are described as devices api inside of the virtual machine.

A

Specifically, the interfaces api describes how the networks are configured which network they are attached to, whether that is the kubernetes part network or additional networks thus are defined. The network attachment definition.

A

The virtual disks are described in a diff api.

A

The information includes such as the size of the disk, whether the disk is the data disk or cloudiness disk, and specifically the disk can be refers to our existing virtual persistent uh volume claims.

A

In addition to the virtual machine, apis qrs also provide apis to manage the virtual machine templates, but data volumes and virtual machine states queue first can be conveniently installed on openshift and kubernetes as operator that is available as operator hub.

B

Gardner is an open source project that delivers fully managed kubernetes clusters at scale. In simple words, image that you can create new kubernetes instances as spots, we can call it kubernetes as a service. Let's take a closer look at architecture. Gardener runs on kubernetes. We can split the diagram you see into three separate clusters.

B

The first square on the very left side is the garden cluster, where all core components run like gardener, api server, gardener controller manager and gardener scheduler, the second one in the middle, the seed cluster. As you remember, I said, gardener is capable of creating kubernetes clusters as spots. Thus, the role of the seed is to run control, planes of freshly created, kubernetes instances.

B

Each seat must have a running component called garden, led, as you can observe. There is an analogy to kubernetes core components: garden led to cubelet gardener, api server, controller manager and scheduler to kubernetes ones. Last but not least, the shoot cluster on the right side, where all worker nodes run the cluster itself must be created on one of the following cloud providers listed below each provider has a gardener extension that knows how to start a new worker node.

B

To summarize this, a newly created kubernetes instance is stretched between seat where control plane is present and shoot where worker nodes are available. Communication between them takes place via load, balancer services and vpn connection.

B

Recently, we added the keyword extension that allows customers to use any bare metal environment that supports linux kvm and allows to start many kubernetes clusters. On premise, I must admit that I find this use case very interesting as we run cube vert vms as worker nodes. It brings some challenges and part of them. We will describe in the following slides.

A

Mouse's cli is a meta cri that allows kubernetes parts or cube versus vms to attach to a different network other than the default spot network. In order to use modules, you have to install the necessary dimmer set and crds in openshift. These are all pre-installed for you. So in this example, we create a linux bridge based um ipvlan, enabled network attachments definition.

A

So there are multiple types of autos water sunrise. So in this example we use bridge. In that example, you can see macvillan and an ipv, so the bridge has to exist first, if you do not have it, you have to do using some other automation mechanisms to create the bridge on the nodes. As you want, the attachments happen, the linux bridge uses ipvlan tag1234 on environment, especially when you are running on public clouds.

A

You may not have access to native vlan, so you can create tunneling mechanisms and on top of the tunneling device like, for example, vxlan you create the bridge and then using the vlans on top of the tunneling device.

A

The ip address management used in this example is a well about which allows you to manage ip addresses across the nodes globally. There are other types of ipl address management, such as static, which only manage the specific ipl addresses along cost local, which is limited to per host that vms or parts are running.

A

Converse use motors for a number of use cases, so in this use case we want to separate the networks, so the virtual machines can use the defaults, a part network to access, kubernetes native or services such as api server and using another other network for data management.

A

So by separating the two networks, the vms can ensure that the traffic from each of the network will not interfere with each other for such configuration. We first defined a network attachment definition that describes how the network interface is constructed and how ip management is done.

A

The network attachments definition is referenced in you know: virtual machines or networks api on the right side of the yaml. The network apis supplies a number of types. The part type is a kubernetes port network. The modus type is a reference to the network. Attachments definition uses the format as the namespace slash the name.

A

This also supports a number of attachment mechanisms, such as bridge as well as the srlv.

A

Once this yammer is applied, the virtual machine general is applied in the background. The first launcher part will create a necessary networking and accurate new environments to configure the vm, so you should adjust runs as desired, allowing the vm to access both the plus network and other networks is great for the vms to access to access all types of services provided by kubernetes and other networks, but for security, sensitive environments.

A

You do not want the vms to be exposed on the part network, because, if you can pee in or just attack of vms, without proper authorization, network policy is surely going to help, but it's not efficient.

A

So we have another configuration for full isolation, while the vms is completely disconnected from the pod network. So if you look at the yellows and the configurations, the only difference is that in the virtual machine networks, api, the path network is completely removed. Only the modus network stays in this configuration when the vm starts. It loses access to the part network.

A

So the any network traffic from the cubenet spot network will not reach the vm. This ensures that the vm is in full isolation using its own view length, and this also saves the ip address from the part network that is also efficient.

A

We were going to see the both configurations used in different use cases in gardner, with last information in mind. Now, look at some of the converse. Networking configuration use cases. The first is high-performance networking configuration.

A

As you know, the virtual machines are often worked with cloud native storage. That's a provisioned by rook rook is another cncf project has recently graduated it's a storage backend operator. That's provisions, saf hfs nfs, as well as apache cassandra.

A

One of the recent features in rook is that it allows a safe cluster to use surface network defined by network attachments definition so that the front-end, the public network, that the south clients are interacting with, has the best possible bandwidth and while the uh the cluster network, where the osds are communicating with each other, to share for the data to balance the data they are separated from the public network.

A

So, in this use case the vms and the staff public network reference, the same network attachment definition such as their traffic will be isolated in the same network.

A

In this configuration they can assure that the traffic is not interacting with other networks and the performance is guaranteed on a separate occasion. What a full isolation configuration the vms are protected in their own vlans.

A

In this configuration we use linux bridge as the connectivity mechanisms in network attachments definition reach the vlan number embedded in the definition again for use for environments. Let's do not support native vlan text.

A

You can build up a tunneling ip service underneath using vxlens and creates a bridge device on top of the vxlan, so the vxlan will emulate a switch trunk that will just tunnel all the vlan between different endpoints. So this is uh one of the configurations you can reference for network isolation.

A

Now, let's put our learnings into practice, we will use gardener and the kubernetes integration as a case study, to show that we how we can use the two configurations of motorcycling line to provide the best performance, as well as the best isolation, so recorders garden project provides kubernetes as a service.

A

It's a provisions, kubernetes clusters on a number of infrastructure platforms, but it does so by abstracting the underlying resources, including the cloud environment, the infrastructure, environment, operating system network and worknose management and such and such this abstraction, programmatically and administratively allows government to be extended to different platforms.

A

So the kubernetes cluster work now created by gardener is called shoot. So we have two shoes clusters. In this example, the blue shoes cluster consists of infrastructure config. That has two networks. One is the shield storage washer network. In this case, the shell networks refers to the network attachments definition created by rook for steph's public network, so the reference here will be the self reference use, uh use the name, space, rook staff and name staff. We are refers to our existing network attachment definition.

A

There are vms that will use this uh rough network attachment definition to attach to the same uh cluster staff. The tenants network, on the other hand, provides a set of information that will be used to create a network attachment definition on the fly, so this is its own privacy network. Attachment definition thus will must be used by other clusters to have another layer for isolation.

A

The nasa attach depth also uses a vlan one, two three four one, two three four to ensure that the wheel and the vms are running in isolated vlans that are not accessible by other vms on the different attachments.

A

The second excuse, cluster, the green cluster of the gracious cluster also refers to the same theft storage network for high performance purposes. This creates its own nice tennis network by providing a different set of configuration for network attachments definition.

A

The vlan configuration in this case is different. It's two three four five. So by having two network attachment definition and two vlans, the green network and blue networks are separated physically and symmetrically as a different kubernetes cluster.

A

So this just shows the case that the multi-modal configuration used by q verse can provide a flexibility for different use cases.

B

Before I start talking about data volume and clone, I must say a few words about the project behind the containerized data importer. It's among one of my favorite add-ons in kubernetes. The primary goal is to provide a declarative way to build virtual machine disks. However, you can use it as well to initialize kubernetes volumes with some data out of the keyword context.

B

Cdi includes a custom resource definition that provides a data volume object type. Those objects are an abstraction on top of persistent volume claims and are very helpful in terms of data imports and uploads onto pvc. This is a way to automate virtual machine disks management.

B

Without that you would have to prepare a pvc that contains the disk image yourself. One of the good outcomes of cdi integration with cubeverd is that you can specify a concrete data volume which is tied to the virtual machine lifecycle. You can do it over data volume template in the vm spec. When you delete the virtual machine, the data volume would be destroyed too, together with the provision storage. So there is no requirement for the user to take care of the cleanup on the right side of the slide.

B

We can see a sample data volume defined there. We have two interesting parts source and pvc the source determines if data is supposed to be cloned from another persistent volume claim or download it from some address.

B

In that case, in the source we would have http options with url value below there is a pvc section, as the name indicates it's about pvc settings like what kind of storage class you want to use and how big storage you want to have. So having all these details in place, data volume would create a new persistent volume clone for our disk. The add-on makes our lives much easier. I encourage everybody to check it themselves if you haven't yet I'm going to show the possible user strategies in the next slide virtual machines images management.

B

Here we can see two diagrams that show two different strategies of data volume, usage, ad-hoc and pre-allocated data volumes.

B

For now, let's focus on the first one, the ad-hoc approach on the left side, as I mentioned in the previous slide, we can download data directly from the internet by changing source from persistent volume claim to http, with an appropriate url.

B

So now considering the scenario where we type data volume to virtual machine by using data volume, template in the spec means each time we create a new virtual machine each time we download the disk. This can cause many problems. If you want to schedule hundreds or thousands of vms first, you can have performance issues, you might have quite slow network connection or some red limitation, even heavy network workload.

B

Let's say you must download four gigabyte disks for each vm and you run many of those second, the endpoint you are using might disappear, causing reliability issues. Somebody could say that, instead of downloading stuff from the internet, you could create- let's say a local http server. I don't know maybe using nginx and serve images from there. Yes, you could, but that doesn't solve all problems I mentioned in 100 percent, and you still can cause network latency on your cluster.

B

Now, let's focus on the pre-located data volume as it helps to get rid of problems. I spoke the diagram on the right slide. The preallocated data volume that is visible in the middle is a source for other data volumes. You download the disk image only once and you use the volume as a source for other virtual machines. You want to bring to life this strategy. We follow in the gardener, cube extension implementation.

B

We get rid of problems I described previously. We reduce network latency data volume, supports two types of cloning and I'm going to describe this in the next slide. We have two ways of data volume, cloning host, assisted and smart cloning.

B

The first split on the left side shows a diagram of the traditional cloning of persistent volume claim that is host assisted. It means there is a data stream from source pvc to target pvc. The entire volume is being copied. It is a traditional heavyweight approach. It is not necessarily needed to waste storage space and put some additional load on disks just to copy the whole image.

B

On the right side, I show another strategy of volume cloning. It is a feature called smart clone. This is being executed when it's possible in order to improve the performance of the cloning process. The containerized data importer team introduced smart cloning that uses volume snapshot, of course, to use such a feature. The csi plugin, which you use, must support snapshots.

B

The yam structure of the data volume is still the same. The data volume automatically checks if smart cloning is possible. If yes, then create a snapshot of the source pvc next, the snapshot is being used to bring a new persistent volume claim. Finally, the snapshot itself is going to be deleted.

B

Additionally, if you want to use the feature, the requirement is to make sure that source pvc is in the same name space as the target pvc. The source and target pvcs are in the same storage class and finally, there must be a snapshot class associated with the storage class depending on the storage backend. You can really benefit from this, especially when snapshots are implemented as copy on right, which is often on the next slide. I'm going to show you in detail how we deal with disks and volumes management in the gardener cube, vert extension.

B

Each shoot cluster that I was talking about at the beginning is defined as a custom resource object. You can see an example on the left side. The should yaml definition. As you can see, there is a worker section where users can define volumes the first one, the volume object spec. It defines a root disk of virtual machine and the second one data volumes which are blank additional data volumes disks.

B

The worker pool definition, is processed by the worker controller in the keyword extension based on that an appropriate specification of virtual machine is prepared. The size parameter is an obvious indication of the storage size and the type determines which storage class should be used. Somebody could ask why didn't you name the parameter storage class instead of type? We didn't because gardener supports many cloud providers, and this is the api.

B

Then, following the arrows, we can see that volumes are translated to data volume templates, so data volume is closely related to the virtual machine lifecycle by default. The extension uses a prelocated data volume approach, which means that the first virtual machine created will download the image and other virtual machines from the worker pool will clone the volume we use blank data volumes to provide additional disk and storage space.

B

These are completely blank disks and it's up to the user to handle them. One way would be to run cloud init and format the disk according to the to their needs, although I must admit that this is not yet possible to pass users cloud init scripts in the extension.

B

The reason is that we are using cloud init to join nodes and we still have to decide how we want to deal with the custom part of cloud unit, but for sure at some point this feature will land in the extension on that slide, I'm going to talk about highly isolated clusters and how to upload disk images there. It happens that customers demand highly isolated environments, not so long ago, at cubermatik we had such a case and environment where we could not pull keyword images directly from the internet.

B

There was only a bastion host where we could access a cluster and expose a few services outside, but still those services were only visible to the bastion host each bot couldn't reach the internet. Somehow we had to provide virtual machine images to the cluster. We could create a local image repository.

B

However, with containerized data importer, it is quite easy to upload image. We can benefit from that and prepare data volume which will be a source for others. The upload is done over cd8 upload proxy first users have to expose upload proxy service and it must be accessible from outside the cluster.

B

Next, it is required to create a data volume with source set to upload similar like on the slide, and finally, we must request an upload token that can be done over upload token request: custom resource, the crd of the object, comes from the project itself.

B

When all dimension steps are complete, it's time to upload an image you can obtain the generated token from the status section of the cr. Then you have to pass the token to an http request with a disk image. There are two ways of upload, synchronous and asynchronous.

B

The synchronous connection can be closed unexpectedly because of the conversion or resizing process that can take some time with an unsynchronous approach. The connection will be closed as soon as the disk image has been transmitted.

B

However, you have to check yourself if the process of resizing and conversion is finished for more details and examples, I encourage you to visit containerized data importer, github page it's a very nice project which solves many issues regarding volumes and disks management.

A

This concludes our talk. We are so excited to share with you, the optimizations in converse, using monitors and beta volumes for high performance and high security configurations. There are even more features in cubic verse. That's including the cpu memory, storage and networks. We are looking forward to share with you in the future opportunities.