Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2022, 11 Nov 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Lightning Talk: Back To Basics: How To Measure Etcd Performance And Not To... David Perez Rodriguez

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe in Amsterdam, The Netherlands from April 17-21, 2023. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Lightning Talk: Back To Basics: How To Measure Etcd Performance And Not To Die Trying - David Perez Rodriguez, Gorilla Logic

Everybody either knows what Kubernetes is or has heard it. It’s a critical component to scalable, high availability and distributed design of most cloud based productions systems. Why would I bother understanding how it behaves outside the cloud provider I commonly use? Well, that was the case of this project, which aimed to build an IoT system that handles Terabytes of data, entirely on-prem due to business needs. As expected, things were not behaving the same as in the cloud provider: lots of kube-api errors, missed heartbeats, database operators started rolling restarting deployments because of it; but the main reason was well hidden from the sight: etcd performance was not great on prem. etcd has an extremely and sustained high performance that is based on two factors: latency and throughput. But in this on-prem environment, latency was affected by the hardware’s initial design. How do you measure etcd performance? Benchmarks to the rescue! Learn about this experience, what is benchmark, what is latency, what is throughput and how to effectively measure etcd performance through benchmarks to correctly test your infrastructure when a brand new kubernetes cluster is created, particularly on-prem, and take advantage of the full potential of the Kubernetes environment.

A

Hello, everybody good evening, thanks for coming to this activity, my name is David Perez I'm from Costa Rica, first time in Detroit. So thank you all. So what I'm going to talk today is a little bit about etcd or ATV, a couple of Concepts that are important to understand the performance of hcd through Portland latency, how to Benchmark at city with a benchmarking tool that comes with an hcd and some remediation and tips in case we face that problem.

A

So, first of all, what's that CD is a distributed, Value store for storing critical information, critical data for a distributed system. You can use it for anything, but it's probably most known for being core component of kubernetes, and you don't actually talk to at city right with your applications, but that is part of the cube. Api server functions to talk to etcd, to store and retrieve the kubernetes cluster estate.

A

So, even though you don't directly speak or talk to the hcd, you can still see issues there, and these are two examples that I saw one from the etcd log. That says that a request took too long to complete and some other logs in the Q API server, saying that there were some habits that were missed, indicating that their server is likely overloaded.

A

But what is exactly to be overloaded in at CV terms, so we need to understand two concepts, throughput and latency, and we understand latency as the time taken for an operation to be completed and throughput the number of operations that are completed in a period of time. So in single terms, you can see latency as how fast an operation is completed and throughput how much data or how many operations are completed in that period of time so about the Benchmark tool that I was talking about.

A

Hcd has a known Benchmark tool as you're, seeing here in the screen that that you can actually easily configure. You can give a many arguments to the CLI and do different kind of testing against hcd and to to make a quick demonstration here, I put on the top side of the screen, some requirements and the part above it is the the information from etcd documentation. They did.

A

They did a benchmark on those requirements and I took them, but I changed the the disk I'm using a hard drive disk in the part below and when we do the Benchmark. We can notice that we don't have that much difference when we are using one connection and one client. But when we go distributed with a thousand of clients and hundreds of connections, we start seeing a lot of difference in the throughput.

A

The request per second and also in the latency also for the rate information, will see a similar thing right, a similar case that, with one single connection, there's not much big deal. But when we go um with multiple clients and multiple connections, we start seeing problems in throughput and latency, especially on-prem right. So when you face this, what what can you do to remediate this? So, first of all, when, when you're designing a system make sure that, if, if you can design and have a head-on time, the number of requests that your application is expecting.

A

So you don't overload a CD, also try to avoid HDD at all. But if you cannot do it, because you cannot afford it, because the cluster is already there make sure that the HDD is at least 15K RPM and also there's something you can do. You can configure with tools like ionize the disk priority, so it is, it takes priority right into disks and also turns some Etc settings like the Harvard timeout, the election timeout.

A

So you don't overload the server and also there are some more tools like the this Benchmark fio and also the node conformance from kubernetes. That will help you address more issues rather than just the Benchmark for Head CD.

A

So I went ahead and did a change just changing the HDD to SSD, and we can see here the big difference that we have now the same cluster, but using solid state drives. We can see that the requests per second that throughput goes actually double, which is good, so doing a quick recap. If you want to know more about benchmarking at the CD, especially on-prem, you can see the operations guide in the documentation and the performance section. Also there's a guest there with some scripts.

A

You can use to create your own three-member, hcd server, and it will also give you some commands to do Benchmark if you want to test it more. So that's my information. Thank you very much. I appreciate it and have a nice day.