Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2020 - Virtual, 4 Dec 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Prescriptively Benchmarking Kubernetes System and Application Using K-Bench - Yong Li, Facebook

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2021 Virtual from May 4–7, 2021. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Prescriptively Benchmarking Kubernetes System and Application Using K-Bench - Yong Li, Facebook & Karthik Ganesan, VMware

This session gives an introduction of K-Bench, a framework designed to benchmark Kubernetes infrastructure from various dimensions in a highly configurable manner. K-Bench accepts a rich set of configuration options and yaml specifications, and parses them into a workload execution plan to provision, manipulate, and coordinate Kubernetes resources. It produces SIG complaint latency and API responsiveness metrics with improved accuracy by adopting a novel timing model. K-Bench also provides a container interface to easily port legacy infrastructure benchmarks into k8s, includes several pre-integrated benchmarks and blueprints to evaluate infrastructure performance at scale like aggregate benchmark throughput across pods, transaction latencies, infrastructure elasticity and resource efficiency in k8s.

https://sched.co/ekDd

A

Hi there welcome to our talk. Kubernetes is experiencing an exponential growth, but the benchmark tools to evaluate kubernetes infrastructures are still catching up. Do you wonder how to evaluate and compare the performance of two kubernetes platforms for a target use case?

A

Do you wish there was one tool that can you know, point and shoot at a kubernetes cluster and get you all the key performance metrics summarized in a nutshell? Well, then, you are at the right place in this talk. We present k bench, which is a framework to prescriptively benchmark a kubernetes platform.

A

I am karthik ganeshan and my cooper center is yongli.

A

So when it comes to performance, there are multiple aspects to look at some aspects related to the control plane include responsiveness, meaning how long does a target platform take to respond to my change request next is scalability. How well does the control plane of this particular platform scale say I want to deploy thousand parts concurrently. Can this particular platform handle that and then how resilient is this platform to require from failures?

A

On the other hand, when it comes to the data plane or application performance, there are basic things like the core infrastructure capabilities like how powerful are the cpus or you know what are the I o and the network capabilities of the underlying infrastructure.

A

As these things can have a significant impact on the application performance say, one has a network intensive application and want to know how it is expected to perform on a particular platform or compare multiple platforms for this particular case to choose one and then comes aspects like resource efficiency. Right, hey. Can I pack more more pods on one platform over another without having a performance impact and, lastly, performance isolation? Characteristics are becoming more and more important. How will my performance be affected when there are noisy neighbors?

A

There are a multitude of small tools to evaluate some of these aspects. Specifically, some are reliable. Some are not. Some are accurate. Some only provide a very coarse grained picture and you can get quite tedious if one wants to get the full performance picture. You know for a particular platform having gone through. You know some of these pain points. We rolled up our sleeves and started, putting a versatile framework together so that we can get all we can benchmark all these various aspects you know with ease. That is how we got to k bench.

A

So what is k bench cabbage is a configurable framework to prescriptively, deploy and manipulate kubernetes objects and in this process it lets us benchmark both the control and the data plane performance aspects with ease example, on the control plane side, you can ask k bench to deploy 1000 engine x, pods, concurrently and observe part startup latencies at millisecond, ground library, say for data plane. You can deploy a persistent volume and record asynchronous rewrite bandwidth to it from a pod.

A

Kbench has an extensible design and provides a multitude of configuration options for the end user. So using these options, one can orchestrate pretty much any workflow with ease using your target workloads.

A

Each use case in kbench is represented by a simple and a prescriptive configuration file at the end of your runs. Kbench provides an intuitive set of performance metrics and a lot of detailed diagnostic data that can be very useful to you know, go ahead and resolve some of these performance issues.

A

So, let's take a closer look at the features offered by kbench separately for the control and the data plane aspects again for the control pin, k bench provides accurate and fine-grained critical path. Latencies for your workloads.

A

You know when you think about some of these latencies you can think of. Like you know, hey what is the part scheduling, latency body, initialization latency, you know startup, latencies, etc, but if you think about it, right, kubernetes is a declarative system where you specify the target state of the cluster, and the cluster eventually gets to this target state via a bunch of asynchronous events that happen in the system.

A

So what is the latency for my trigger change? Well, it is certainly not how long coupe cuddle took to you know say, submit a request right. It is actually how long the cluster took to get to this final state that I desired.

A

So when it comes to measuring these latencies of triggered changes, we need to keep track of the various events in the system until the cluster reaches the target state in k-bench, we use a novel methodology to keep track of both the client and the server side. Timestamps of all these different events to tackle this problem and provide meaningful control path. Latencies for you.

A

For the data plane, one can evaluate application performance using real containerized benchmarks in kubernetes parts, while one can use their own workloads, they can also leverage built-in workloads provided by kbench to stress specific infrastructure resources.

A

For instance, you can use redis memphia that comes pre-built in k-bench to stress the compute and the memory aspects or say the fio benchmark to you know, stress the I o aspects, etc.

A

So, using these artifacts one can you know, scale up or scale out resource usage to study infrastructure performance scale up, meaning, you know, use a single part to increasingly stress a particular resource category or scale out, meaning, you know, add more parts to the system to consume resources. You know in a particular category, k-bench also includes built-in blueprints of workloads that take advantage of these benchmarks to evaluate you know different aspects of data plane performance.

A

These aspects can include things like hey. uh Can I you know, can I get a uh and for some information on what is the part density that this particular um you know framework, and I mean this particular platform can achieve, etc.

A

So now that I know having given a overview of k, bench yong will discuss the different elements of k bench to construct a workload to evaluate the different control plane aspects to you. Young.

B

I'm happy to be here presenting k bench control plane. First, let's start with some basic terminology when we're talking about cable interactions, we're referring to the kubernetes object, lifecycle and devops operations such as create delete list scale.

B

Actions can be specific to resource types. For example, you can create any type of resource, but you can only scale certain type of object like deep line actions run with action and resource type specific options. For example, if you are creating a service, you can specify service type and ports as options.

B

On the other hand, if you are skating a deployment, you can specify a number of replica as an option.

B

Multiple actions run one after another for the same resource. Object from an action chain. Action chain is useful when you want to define a list of actions that depend on each other. For example, you want to create a list of pod before updating and deleting them.

B

Based on actions, we define operation, an operation in k, bench contain a collection of action chains and each action chain is executed for one particular resource type.

B

Action chains for different resource types in one operation will run in parallel.

B

We also define predicates in k bench, which are conditions under which a particular operation will be triggered. There are many different type of predicates. For example, you can define a particle that check status of your running object.

B

You can also define a predicate that wrong command and check the return status of the command and the only wrong operation. Under certain scenario, you can even define predicates that are examine a runtime environment inside a.

B

Container people define a set of metadata before running your workload and attach operation thread id that manipulate a certain set of object, and later you can refer those metadata labels for filtering and selection purpose.

B

Of course, you can pass your customized defined user labels and use your use for selection purpose later now. Let's look at the overall design framework.

B

Here is a running kubernetes cluster with a bunch of objects capable to accept a k-bench config file along with standard kubernetes, yummy files, the kubernetes config, password and dispatcher logic is responsible for parsing those config files. It depends on what type of resources you configure to measure and benchmark. It generates a list of resource managers.

B

Each resource manager is responsible for managing the life cycle and events and resource metric collection for a particular resource type. For example, a pod will be responsible for managing pod resources.

B

And you can specify for each resource type what actions and operations you want to run here, for example, for pod, you define a list of I actions to create a part first and then update and run command inside a pod and for other type of resources. You can define totally different action list.

B

You can put all them together in one operation and those actions for different resource type will run in parallel, but action chain for one particular resource type will run sequentially and you can specify concurrency and resource manager will track your concurrency configuration and maintain a thread pool of appropriate size for that resource type.

B

So you can use the key labels, keybench pre-attached labels and your customized defined labels for resource selection purpose.

B

You can define predicates to guard your operations, so all of those come together to form a very flexible execution plan where you can define your workflow to be executed uh on certain type of resource at a certain concurrency under a given condition against a pre-selected uh resources.

B

In addition, k bench exposed container interface to all side users, so users can run command micro benchmark inside the container and mail performance. Accordingly,.

B

We also integrate promises and real front for resource monitoring and triage purpose.

B

The framework is based on client goal and can run on different platforms such as vsphere, openshift and gke kubernetes offerings.

B

Let's look at a list of example, resources, actions and configuration options for pod. You can create list, get run copy, update and delete, perform those actions and for different actions. As we mentioned, you can configure to run with different options, for example, all the options actions you can specify concurrency and the sleep time, which is the time in seconds.

B

You want to sleep after each action for create. You can specify image, point policy, image to use and yummy spec and run for run action. You can specify the command to run and for copy. You can specify the passes, local paths on container paths etc and for other actions you can specify you know labels label, key and label values, etc for deployment.

B

In addition to the all the above actions, it also supports skill action. Skill uh and a cr is create. Action has some additional options like a number of replicas.

B

For replication, controller and state set will provide a similar set of actions and configuration options cabinet also support some other type of resources, like name spaces, service, config map, etc.

B

Here's a use case example layout as a configuration file. You want to define your workflow to run no more than six seconds. That's a timeout on the top, and here you define a list of operations, and your first operation will be guarded by a predicate that checks the running status of pods initializer.

B

And after that, you define the body of your first operation you want to here. You want to create deployment using this, given yummy file at a concurrency four, and in parallel with that, you want to create parts and run a search script and attach these labels. At a concurrency of two and after your first operation, you can define more operations which will run after one after another.

B

Cabbage produce c compliant api matrix. It reports a 50, 90, 99 api, latency, uh latencies and uh api responsible time for certain operations, also breakdown report, fine grain, critical pass components, for example for paul creation. It will report scheduling, initialization image, point and startup or latency on the node.

B

It has improved accuracy compared to some existing benchmark.

B

Current currently uh kubernetes server side only give where course grained timing. Information runs to second. So we want to study some metrics, it's mid-second level. This is not surfacing, so we rely both server and client-side timing information and we use even the callback mechanism and it checks the triggering reason and condition under which the inventory trigger and measure and record timing information. Accordingly,.

B

So, for example, here we want to start a uh the pod initialization scheduling latency at a different concurrency using an existing benchmark. It just gives us all zeros, because the pod scheduling is very short running operation.

B

On the other hand, using kbench allows us to report the detailed client side metrics as millisecond level uh for different concurrency.

B

This is pretty much about the keyboard control point features and now I'm going to hand it over to carsick talk about data point features. Thank you.

A

Thanks john, let's take a closer look at benchmarking: that data plane using kbench with the rich container interface that kbench offers one can orchestrate pretty much any workflow say you have a use case in mind, and you want to evaluate how your use case will perform on a target. Kubernetes cluster k bench can help you with that. You can start with the create operation using which you can deploy your kubernetes resources with your target containers and assign them some labels.

A

You can just reuse the yaml spec of any kubernetes artifacts that you might have labeling. These resources helps us filter and select specific resources on which we want particular actions to be performed.

A

That way, we can orchestrate different workflows with ease. Next, you can copy any workload, artifacts or workload specific configuration files for current run into your containers. Again, you can use these labels to selectively. You know: choose those kubernetes or objects to which you want to apply this action.

A

Then you can run commands inside the pause to trigger your workflows. One of my favorite features of k bench is the ability to trigger actions using conditional predicates.

A

These predicates can be kubernetes system based or something that is evaluated inside a container say you have a particular server plot deployed and want to generate load from a client part against this particular server part. You can wait until the server pod gets to running or even better. You can wait until the server process inside the container gets to a particular state using these predicates.

A

Now, maybe again you want to copy. You know your results out of these containers and to enter the client from which you are orchestrating all this, and then you can go ahead and use the delete action to clean up your cluster.

A

Now, having looked at, you know how to orchestrate a workflow using your workloads in kbench. Let's take a look at some of the pre-integrated workloads that come with kbench to stress different infrastructure dimensions. These pre-integrated workloads are ready to be used out of the box.

A

As you can see on this particular table, you know you have workloads redis memphia that can be used to stress cpu and the memory aspects. You can get aggregate transaction throughput across all the different parts in a particular cluster, and you can also get transaction latencies.

A

For you know your deployments and with fio you can get read, write bandwidth for various data ratios block sizes, on ephemeral and for system volumes.

A

Io ping is integrated with k bench to get io latency on fml and persistent volumes. You have iperf3, which can provide you, interpod tcp, udp, bandwidth information, and there are blueprints with varying pot placements on nodes zones, regions, etc. To get you, you know more specific information, uh and that is q, puff integrated into k bench and it can provide. You know, interpod network latency and again there are blueprints with varying port placement that can orchestrate you know keeper, for you typically the end results or you know the generated performance metrics only paint.

A

You know the final picture right if you notice an anomaly or a performance issue. Just this data is not enough analysis and iterative improvements need deep infrastructure. Diagnostics, kbench provides support to inject performance and diagnostic data to dashboarding services like wavefront and rafana. These are done by using distributed telegraph data collectors and they can be configured with. You know, supported output, plug-in to view results on your dashboarding platform of choice.

A

These telegraph collectors are fed with thousands of handcrafted performance metrics that can be monitored for linux and esx hosts. This data can be valuable when correlated with actual performance results to deep dive performance issues and resolve them.

A

Having talked about the data plane features of cabins, let's take a look at some example. Use cases where we have put k bench to good use. K bench was used extensively in evaluating and improving vmware's kubernetes products. In this use case, we deploy a standard java benchmark inside multiple kubernetes parts.

A

We use kbench to find the maximum cluster level aggregate transaction throughput for these java workload that can be achieved on you know different kubernetes clusters, but with the same hardware, resources cabin showed how a virtualized kubernetes cluster can actually beat the performance of a bare metal cluster, both using the same hardware. It also enabled us to get deep insights into what was the root cause for some of these performance differences.

A

Some of the results generated by k bench were so impactful that they were used by vmware ceo, pat gelsinger, for kubernetes product announcements in the opening keynote of our annual conference.

A

Here is another example of a data plane, pre-integrated blueprint called dp internode. This blueprint automatically deploys two parts on two nodes using anti-affinity rules. It uses a headless service to run iperf3 across the parts to provide tcp, udp bandwidth, information and qperf3 to provide the network latency information.

A

These blueprints can be run all these different blueprints for both control and the data plane can be run as a suite to get all these key metrics. Just in a nutshell,.

A

In summary, we presented k bench, which is a highly configurable and easy to use. Benchmark framework to evaluate kubernetes performance can be valuable for competitive benchmarking across multiple platforms to identify performance issues, root, cause them using diagnostics and iteratively improve your platform. Performance k bench is open source and we would like to welcome everyone to use the tool if you have a need, provide us feedback and to contribute to the project.

A

We thank you for your time and we are happy to answer any questions you might have.