Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2020 - Virtual, 4 Dec 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Tune It Up! Enabling Low Latency in Kubernetes Clusters - Francesco Romani & Yanir Quinn, Red Hat

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2021 Virtual from May 4–7, 2021. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Tune It Up! Enabling Low Latency in Kubernetes Clusters - Francesco Romani & Yanir Quinn, Red Hat

Now that Kubernetes has conquered the cloud it’s time for it to move on other domains, like CNF (Cloud-native Network Functions), the cornerstone of 5G deployments! However, to make that happen, we need to address an important limitation: ensuring low latency. In this talk we will first and foremost give an overview of the most important setting relevant to optimal low-latency behaviour, demonstrating the level of detail we can achieve in fine-tuning a cluster. We will explore the kubelet settings (topology and cpu manager), the kernel settings on the node (using tuned) and how they interact to deliver the workload guarantees. The talk will help in gaining a better understanding of what Kubernetes provides for this specific and demanding class of workload, considering the planned enhancements coming in the next releases, like the kubelet memory manager.

https://sched.co/ekDp

A

Hello, my name is yanil quinn and I'm going to give this session alongside francesco omani we're both engineers at red hat and today's session is going to be about how to enable low latency workloads in kubernetes clusters. So, let's start now that kubernetes has conquered the cloud it's time for it to move on to other domains for and foremost conquering cloud native network functions the cornerstone of 5g deployments.

A

To make that happen, we need to address an important limitation, ensuring.

B

A

B

Of running cloud.

A

Native network functions such as workloads designed for telcos stalls, interesting challenges to overcome. It's very.

B

A

Requiring fine tuning of the node behavior to ensure low latency processing of network packets, where special hardware and software are needed, kubernetes container platform can be augmented and tuned to to become a great platform for some demanding class of cnf workloads, and this talk will demonstrate how we will describe the features available alongside upstream kubernetes, to provide real-time guarantees and to handle low, latency and high performance workloads.

A

Let's have a look of.

B

A

Use case taken from the telco world, a cluster admin handling a deployment that should provide low, latency, real-time or dpdk purposes will want to configure it without deep understanding of all the interactions of the kernel and operating system, components that are in the mix.

A

You should be able to do so in an easy, declarative manner that will encompass all the moving parts to achieve this goal, such as isolating workloads for interruptions for gaining better latency results for his telco company and on the web scale and edge fronts.

A

So how do we do it and how can we guarantee performance and low latency? Let's have a look.

A

The ecosystem that will provide the way to achieve this is built from these existing and further developed components.

A

We will explore the cubelet settings, topology and cpu manager, the kernel settings on the node using tunde and how they interact to deliver the workload guarantees kubernetes tunings that can be achieved. Harnessing and manipulating kubelet settings entangled with cpu and device managers, as well as applied topology structures.

A

And doing that on the side, we can be using operators and we'll be using operators to apply machine configurations to the operating system itself, including installation of real-time kernel and other configurations between the kernel and cubelet.

A

We will be setting node.

B

Level tunings relayed by.

A

Tundi, this also will be done by an operator at the top. We have an operator performance, add-on operator. We will see it later on. It will orchestrate everything to achieve optimized clusters for applications sensitive to cpu and network latency.

A

If you're asking yourself what is an operator so an operator is a kubernetes controller, specif specified to operating an application.

A

It's a kubernetes pattern that is extending the kubernetes control plane with a custom controller and a custom resource definition that add additional operational knowledge of an application.

A

We will see some of the operators in actions, as I mentioned, combined to tailor a tuning suit for our.

A

Cluster to optimize kubernetes clusters for applications sensitive to latency, installing a real-time kernel and configuring. The operating system to run as a real-time system is one of the key aspects.

A

Linux can be considered as the best choice for high performance computing. Due to the years of linux, kernel optimizations focused on delivering high average throughput for a vast number of different workloads. Real-Time is not necessarily superior or better than a standard kernel. Instead, it meets different business or system requirements.

A

It is an optimized kernel designed to maintain low latency, consistent response time and determinism all resources managed by the linux. Any linux kernel cpu time is a principal resource tuning. A cpu towards better performance aspects should be according to scheduling, algorithms, that use priorities and real-time policies to control execution.

A

Favoring high priority tasks, while interrupts and preemptions, are enabled for a much wider range of time than on traditional kernel.

A

Let's see how we reach and touch the nodes in our cluster to make them real stars using operators.

A

B

A

Machine config operator, one of the operators we mentioned earlier on and will see following the slides.

A

The machine config operator, manages the operating system and keeps the cluster up to date and configured using this operator platform administrators can configure and update systemd, cryo or cubelet kernel network manager and more on the nodes to do so. The machine config operator creates a statically rendered machine config file, which includes the machine configs for each node. It then applies that configuration to each node with the machine, config server and the machine config controller really relaying this configuration in the following example.

A

We see how a machine configuration can set a kernel type to real time on a specific node in declarative way. So we have this configuration called of kind machine config. We set real time kernel as an argument, and this will apply using the operator on the nodes. Let's see it in a micro demo, now.

A

Before the micro demo we'll see another example here of a machine, config capabilities which sets kublet's configuration such as cpu memory and topology manager, policies on the node, the configuration goes through the operator and finally, lands on cubelet conf files for matching nodes and now we'll get to the demo. So we'll see how we apply. Real-Time qnl and cubelet arguments on nodes using the operator.

A

First, we'll see how we change the kernel nodes to real-time kernel.

A

So here we label the node with a we have an alti worker label.

A

We create the machine config, as we saw earlier on, to set the node as real-time kernel.

A

We create a machine config pull a poll where the machine configurations we create will apply to. So. If we apply the machine, the pool, all the machines configs associated to it will be applied on the nodes. According to the labels, the enables the label we used earlier on was the walk lt.

A

B

Get the nodes we see the label.

A

And now digging into the node, we can see that the linux has changed to real time. That's one example: I'm using here oc, but you can consider it as a cli, very similar to cube ctl with additional capabilities.

A

Next example, we will see how we change cubelet configuration on desired nodes, so we click create this machine config and fit machine configuration with cubelet config settings, then that will sync into cubelet conf files on the node. So we have here cpu system reserved.

B

A

And cube results, for example, and also manager policy and reconcile period.

A

B

Create the configuration.

A

The pool already existed so changes has applied and then we go into the specific node we labeled and.

B

A

See go into the cubeletcon file. That values has actually changed according to to the configuration we we added.

A

So after seeing how we use machine configuration, let's talk about tunity tuned is an open source system tuning service for linux. It monitors, connect connected devices using the udev device manager, tune system settings according to the to a selected profile.

A

It supports various types of configuration like syscatal, csfs or kernel boot command line parameters which are integrated in a plug-in architecture. Many of the plugins are supported by another operator, node tuning or portal, and include the plugins we see below cisco cpu bootloader, for example.

A

Tund also supports hot plugin of devices and stores and also stores. All of all its configurations cleanly in one place tune the profiles that that manifests, the required tunings we desire, can be defined hierarchically, as we can see in the diagram which reduces duplication and simplifies maintenance.

A

Tundi also supports full wall back where the system can be easily returned to the state before the profile was applied, and it also includes a number of predefined profiles for common use cases. For example, presets for high throughput, low latency or power saving are distributed.

A

Ultimately, we will want to ship a profile that best suits the tuning requirements. We have the profile can be composed from one or more presets to have our desired settings, but more parts are needed to compile to complete the picture, as we will see soon.

A

Here we meet yet.

B

Another operator.

A

After the machine config operator and no tuning operator, it is designed for node tunings that provide us a centralized way to customize node level tuning for cluster administrators, abstract, os version level, dependent, tunings details, away, enable modularity and tuned profile. Inheritance provides same defaults for control, plane and worker nodes and also provide dynamic tuning with wallback, without the need for node reboots.

A

The node tuning operator manages tundy as a containerized kubernetes demon set. It ensures custom tuning specification is passed to all containers tuned to demons running in the cluster, in the format that the demons understand the daemons run on.

B

A

Nodes in the cluster one per node deployment of custom tunings is done by creating a tuned dcl which includes profile data recommendation, logic and machine configuration labels and selectors to apply these tunings on desired nodes.

A

So, let's see a short demo of how the node tuning operator applies, our tune d profiles.

A

First, we label the node. We want changes to apply to as we did before. So in our case, we see the label here we create the pool to be associated with the notes, as we mentioned earlier on.

A

And here we see a tuned declarative cl associated with the no tuning operator where we set several variables, such as kernel arguments for isolated calls, but we also see it includes a real time and openshift nodes apparent profiles in our example. So, let's see the parent profile.

A

B

Parent profile also.

A

Sets several variables, for example, under cisco canal hang task, timeout 6 to 600.

A

B

Get into the nodes.

A

Again and we debug our specific node, we can see the kernel arguments has changed to isolated, cpus equals one and also check the cisco value for kernel, hung task, timeout sex, which is indeed.

B

600., let's look at the cubelet and let's see how we can tune it and then the work nodes for telco workloads.

B

Let's start with the most basic computer resources, the cpu cores to meet the latency requirements. We need to mitigate cpu contention and context switches to do so, we can use the cpu manager, which is built in in the cubelet. The cpu manager supports different policies. The static policy allows exclusive allocation of cpus to the workload exclusive locations. Allow us to avoid the context, switches and reduce and to reduce cache misses, because no other workload can run on the exclusively allocated cpus.

B

However, system activities can still run on isolated cpus for full insulation. We need other settings like, for example, handling the error processing on housekeeping cpus housekeeping cpus can be selected using, for example, other cablet options like the reserve, cpus option, command, write option and the tech workloads very often require access to survey virtual functions or to hardware devices in general.

B

Kubernetes provides a device plug-in framework which works together with the device manager, another component built in in the cubelet, and this and this system can be used to expose other resources to the cubelet hardware supported by the device. Plugins includes the aforementioned reserve virtual functions, but also other devices like gpus or hardware accelerators in general. The device plugins report topology information for the order they manage, including the new affinity.

B

A

B

Is another component built in in the cubelet it orchestrates the other resource manager like, for example, cpu and device manager? It allows to align the resources that a workload requires using the topology information provided by those managers. For example, we can do normal cell alignment.

B

The topology manager is configurable using policies with the most restrictive policy enabled called single numanode all the resources. A workload required must be allocated with the same new affinity. If this requirement is not met under this strict policy, the workload won't be admitted. Other policies allow less strict behavior.

B

The topology manager can do, for example, best effort location if it is not possible to allocate the required resources with the same new affinity, the workload will still be admitted.

B

Let's see a few examples of new aligned and numa unaligned resource allocation in order to see how the topology manager works in general, we should not ex not expect renumera cell to be symmetric, or we should not expect the numa cell to all er the new cells to all have cpu cores memory and pci buses, because in general, numa cells may have different combination of those resources.

B

For simplicity's sake, however, we will consider we will consider a system with two symmetric numa cells. Each of them will have cpu cores local memory and a pci bus attached to them. This is the simplest scenario and we use it for illustration purposes.

B

Our first example is a numa unaware resource allocation. The workload is allocated regardless and innova cell alignment for the required resources, so those required resources required by the workload may span across different luma cells.

B

This still allows the workload to run, but the performance is not optimal because, for example, of the of the traffic required between the pneuma cells, for example, if our workload gets core number three and the cervical function for numerous zero on the left and core number six in the memory from muscle number one picture on the right. This is a legal allocation.

B

Unaware allocation does not prevent the workload run, but if, for example, core number six on the right needs to access, there's a referred virtual function on the normal cell number zero on the left it it must pass across the cross, pneuma link and this causes a performance penalty.

B

This is our second example, as it is a numa aware, resource location we can do. We can obtain this allocation, for example using the single luma node topology manager policy. In this case, all the resources. The workload requires are aligned on the same numa cell. In this case, the number cell 0 on the left, the performance is optimal because no cross number link traffic is graphics, is ever required.

B

Let's see a demo of how the topology manner works on a real kubernetes.

B

Cluster, okay: we are now seeing the demo, so let's run a workload which requires a line of resources.

B

For workload perspective, the only requirement is to obtain in the guaranteed class and quality of service class. So this is the example workload which we are going to run, which is very very similar to the workload we seen previously in the in the example slide, so two chords are required and one is through virtual function.

B

So, from the user perspective with topology manager enabled there is no difference in the flow we just run. The pod.

B

And it is running okay yeah, it is running so, which is more more interesting, is how the resource looks like from inside the port.

B

So to inspect the resources from cdepot, we can run a debug workload, but first, let's see how the definition workload definition actually looks like so again we record the same resources and we just run a different command because we want to run our debug tool, so we create the port as usual.

B

We wait for it to be running.

B

And here it is no not yet here it is so now we can jump inside the board and check the resource, location.

B

And okay, there it is so two cpus core two cpu cores are required and this workload got two of them and both of them are bound to numerator cell one. We required one srv virtual function, which is as a pcr devices again bounded to numadon one. So everything is aligned on normal node. One.

B

We have seen so far how many moving parts like operator components, cubelet settings. We need to configure to ensure optimal performance for low latency workloads. Let's see how we can bridge together everything, since this is also a quite complex task. Let's see how we can it could be automated and to automate it. We can use, for example, another operator which is the performance operator to configure a cluster. We can use this operator, which simplifies the process, exposing a high level profile which allows us to configure the pers formats behavior of the cluster.

B

We use another higher level of abstraction. The performance operator will orchestrate the configuration of all the components like the operator we see in the cubelet settings and we it will drive them instead of user, so the performance operator acts as orchestrator for all the cluster components.

B

So in this slide we see an example of all of how the performance operator will drive all the other components and, for example, some settings are still done almost directly by the operator itself, like, for example, for now huge pages configuration.

B

This is how the performance profile could look like. We, for example, we can request like huge pages, we can select the reserved housekeeping cpus and we can. We can add command line, kernel command line arguments and we will see now how the performance operator works in practice in a cluster.

B

Okay, we have a cluster with few nodes, a few worker nodes and we selected the role to make sure the performance operator can work, can change the compression of work or not. So we see the performance separator is running and we how the spec look likes so again, some camera arguments, some youtube edges for illustration purposes: the single number node topology manager policy, and in this case we also we require the real-time kernel, and for that we need a cluster capability to support real-time current.

B

So, for example, we see again the huge pages, and now we see how the performance operator works with the all the other cluster components. For example, we are now checking the tunde the tuning node tuning settings we seen previously in this session.

B

Okay, we are seeing how the 2d profile is translated in practice and now we're checking. We are checking we're going to check the huge pages, so this is how it should look like, but now, let's jump on a node and check how the all those settings actually translated to node tunings, for example, again with proper clustered support. We see in the rt kernel button.

B

We also seen the kernel arguments we required and the huge pages we requested in just a moment, and here they are.

B

Here it is huge, page is configured, so this is how so this is how the performance operator drives the configuration of all the cluster.

B

If you want to try the performance operator, we can check the github page of the project on which you'll find, of course, the search code. Everything is open, source, the documentation, the installing instructions we have pre-built container image and very soon, we'll have all the operator available on operatorapp.io to ease your consumption.

B

So this concludes our session. Thank you very much for attending.