Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2021, 14 May 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Graduated Project Lightning Talk: Vitess, are we Fast Yet? - Akilan Selvacoumar, Florent Poinsard

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon North America 2021 in Los Angeles, CA from October 12-15. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Graduated Project Lightning Talk: Vitess, are we Fast Yet? - Akilan Selvacoumar, Florent Poinsard, Planetscale

Vitess is a database clustering system for horizontal scaling of MySQL. Nowadays, performance is a considerable market factor, and things have to go fast and right. With over 23 000 commits, Vitess has to ensure it’s delivering flawless performance to its users. In order to meet this need, we created AreWeFastYet, a benchmarking monitoring tool for Vitess. This talk covers how we track the performance of Vitess and a high-level overview of the benchmarking tools we created.

A

So today's talk is on betas rv, fast shape by mia keelan and florent. um A bit of a background with this with this is a database solution for deploying scaling and managing large clusters of open source database. Instances with this is based on mysql slash more adb, it's massively scalable. It is a cncf graduated project with this runs both on public and private infrastructure and works very well with dedicated hardware.

A

Bethes has over 24 000 comments, it's being used widely by companies like youtube, slack, jd.com, github and pinterest, and is pretty popular among the dbas for horizontally scaling, mysql databases, I'd like to give it over to florent talk more about today's talk.

B

Great, thank you. Yes, um just like you said uh it is being widely used more and more in bigger and bigger systems, but this comes down to two problems. um We have to ensure that vita cisco base is reliable um and also it has a good performance. This is too big attribute that we want.

B

So how do we ensure reliability? Well, we have test unit test end-to-end test. um Everything goes through cisd pipelines and et cetera, but how about performances?

B

Well, we have benchmarks, and today we're going to cover how to benchmark a big project. Just like venus. We come up with five pilots on how to benchmark a new principles project such as vitesse.

B

First, we want it to be easy. uh We want to foster a culture of benchmarking, we want to encourage people to do more and more benchmarks. Secondly, we also want it to be automated just like unit test. We want to avoid human error. We want to spend more time on important things and so on.

B

Then we also want it to be reliable. uh You know, benchmarks result is not a boolean, uh it's not like unit test pass or failed. uh Everything is measured in nanoseconds, so it has to be reliable.

B

Then we also want it to be reproducible we want to resolve. I mean we want to allow engineers to debug a benchmark if there's an issue or a spike in performance. We also want it to be observable. um You know we want to see results. We want to see reports, we want them to be accessible by anyone in the community. So how did we achieve that? Well, we've created. Are we fast? Yet?

B

It is an automated benchmarking monitoring tool for vtest. uh It is open sourced, of course, and version number two is being developed in golden at the moment.

B

um We execute different types of benchmark in our fast, yet we have macro benchmarks and micro benchmarks, the first one uh we have otp and tpcc, which are two different types of micro benchmarks that are widely used.

B

This type of of benchmarks macro benchmark, uh usually run between 30 and 60 minutes, just like end-to-end tests almost, and for that reason we execute them only after merge or master or after release or before.

B

Secondly, we also have micro benchmark. um Vtest's code base is coded in googling, so for that reason we we execute micro benchmark with go test bench. uh It focuses on a single function, very tiny piece of code. uh It is very short running between one and ten minutes, and this is why we execute them after every commit, then how to execute uh these two types of benchmarks with rfis yet well, rfi state has a cli, so we can execute.

B

You know benchmark individually through the cli or we execution is being triggered by commits pr from vts's code base and from there we use a configuration file where every single attribute of the benchmark is defined and declared. um This enhance the reproducibility of benchmarks. We just have to copy the configuration file to someone else's laptop and we can execute the same test once we have that the configuration file we execute terraform inside our fast kit again to provision the infrastructure which relies on equinix that provide us with bar metal server.

B

Permutal server is a good way to improve reliability. We know we have the same hardware specs. um This is a good, a good way. um Then we apply some configuration within civil want to configure the the hardware that we provisioned and then we execute the benchmark uh either micro benchmark macro benchmark anyway, we execute it and then we get results. We save them in mysql database and then we send the results in a slack channel and we broadcast them to our website, uh rendering graphs and and etc.

B

Talking about this uh sharing and observing uh I've talked about the slack and the website. um Well, we also have a third way of uh of doing so of sharing and observing, which is uh through github status. We want to validate or invalidate uh fetuses pr and commits based on those results.

B

This is still in progress, but we hope to get that merged soon. um Now, acclimate is going to talk to you about the different dashboards that we have in the web. Ui.

A

So the major dashboards we have is the micro benchmarks dashboard, um as you can see here, with the comment, hash, 925 and so on. So on, we compare this micro benchmark, run against with s9 and, as you can see, we have different different metrics and different color coded to show the difference if it's an improvement or or it's a warning, and if you click on the blue links, it actually points to the benchmark.

A

The code itself on the official test repository we also have is the qps, which is the queries per second.

A

um These are the results we actually get from sysbench and we and this as fluent mentioned previously, we run this off every merge and these are the merges for the past 30 days which occurred in the tess master repository, and we have- and these are the following- metrics, which we get from sysbench and we have two major runs, as one mentioned: otp and tpcc runs and both get the following metrics and also we get transactions per second for otp and tpcc, and this is exactly same, as I mentioned in the previous slide, I'd like to give it over to florian for the conclusion of restock.

B

Great thank you. So, let's go back to our five pilots that that we that we saw before um we say easy well. Rfisd makes benchmarking easy, because you know we have cli commands everything is uh executed by a cli or uh commits npr.

B

Additionally, it is very easy to add micro benchmark in in micro benchmark, because we have a very centralized configuration file. It is also automated, uh just like we saw we have terraform and in symbol configuration everything is executed after a commit uh using github, webhook or release. It is also reliable. uh We have permittal server uh using equinix and we also have hist and history of all the previous benchmarks results stored in mysql database. That way, we can aggregate the results and provide a more reliable result.

B

It is also produceable I just like just like. I said we have a centralized configuration file, just copy it to someone else's laptop and execute it. And finally, it is observable: just like iclin showed us, we have web uis dashboards and we send slack messages and github stages, comment and etc.

B

This is the url of the project on github. uh Don't hesitate to go check it out and uh thank you.