Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2018 (Seattle), 15 Dec 2018

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Lightning Talk: How Fast Was My Database? - Josh Berkus, Red Hat

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

Lightning Talk: How Fast Was My Database? - Josh Berkus, Red Hat

I know my database will be slower on Kubernetes and cloud native storage, but HOW MUCH slower? This is a question everyone thinking of moving traditionally hosted stateful services like database to Kubernetes ask all the time. And until now, we haven't had good answers. This presentation will detail a series of microbenchmarks on PostgreSQL running on and off Kubernetes in a variety of configurations, including bare metal, local storage, gluster, and rook. You'll get a solid idea of what the cost in latency and throughput is for abstracting away your storage problems, and be able to make platform decisions for yourself.

To Learn More: https://sched.co/GrVS

A

A

So when kubernetes 1.0 came out over three years ago, it was like everything else in casino and stateless great for web servers great for app servers, not so great for databases, but then in 1.4 we got stateful set both a component and a method for deploying stateful micro services, running them in a kubernetes cluster, and since then, we've gotten a whole host of technologies that help us run things with state using kubernetes to the point where just about any application, no matter how stateful can be running kubernetes and somebody's released a tool to help you run it there.

A

However, my former colleagues actually want to know a little bit more than can we run databases on kubernetes, but should we run them there in one question they want to know how fast is it?

A

What's the performance overhead, what am I looking at sacrificing to get this container cloud, so I decided to run some simple performance tests using a simple transaction processing test with two different database sizes, one in memory, one bigger than memory do some tests, where we do a large data load for a large right and a large data read for a large read to basically get the general parameters of how performance works on these systems and so I'm going to actually give you a series of stats organized into quadrants like this.

A

These are the meaning of the different quadrants in terms of that, the stuff on top higher is bigger stuff in the bottom. Lower is bigger, is better so first we need our control here right, which is running it on quote/unquote, bare metal.

A

Now I'm actually running this in a public cloud running this in AWS, it's not really bare metal and that can become important later on when we look at some of the different storage configurations, but that's our base sort of numbers running it without kubernetes, so our first architecture, one runner and what kubernetes calls ephemeral storage and that's actually, where you just let the container runtime store things in its local file system. It's completely untrustworthy, so you'd better have a good replication setup to make sure that you don't lose your data when we failover containers.

A

Now, of course, a federal storage in this case means that it's actually putting stuff on EBS.

A

So the ephemeral storage numbers look pretty good as a matter of fact for the in-memory database case, they are indistinguishable from not running on kubernetes at all. The overhead of other components is higher than any kind of network or redirection overhead kubernetes might be adding. Now these numbers don't look quite as good. We have a little bit of slowdown here, particularly for the larger than memory database transaction processing case. What's going on here.

A

Well, kubernetes does not like you to run a pod that uses 100% of the resources of the system it likes to reserve some for the system, components and everything else so I have to dial back and throttle that pod to use slightly less resources in the system I had and when the database is much larger than memory. That dialing back has a strong performance effect. Now, let's look at some other storage configuration, so one is for every one of the kubernetes cloud providers of the AWS cloud provider, OpenStack cloud provider or whatever.

A

There is a native storage component there to use the cloud provider storage directly through kubernetes, and so let's see what the performance is like here. Well, unsurprisingly, since we are still using EBS, the performance is pretty much identical to the previous case, even though we're going through a different code path by the way, this particular performance test is a lot of jurors, so things that are less than a difference of 5%, really don't matter so we're getting the same performance there. So two really good performance cases. Now, let's do something more sophisticated.

A

Let's do container eyes. Cluster storage, now what that is, is I actually run a bunch of storage daemons in containers like luster, SF and then I network them together to create clustered redundant storage running on top of kubernetes. That is also supplying storage to kubernetes. Now this is a redundant file system that is making multiple copies of everything.

A

So we don't expect it to be as fast as using a plain file system and in fact, a cluster FS, which is the first one I tried, which specifically says it is not optimized for databases we find out. Well, it's not optimized for databases running on kubernetes or not definitely much I big cut here. Rook and staff did a little bit better in my tests that much better, particularly for the in-memory database case, because it's optimized a lot more for slow write for small writes.

A

So some conclusions that actually came to from here was, first of all. If you're not going to be using container as cluster I storage, we can be using some form of straight storage. Don't worry about the overhead imposed by kubernetes for the database case? It won't matter. It's not a significant amount and you'll have some other options that allow you to trade off for dumbed nc4 performance, but keep in mind that each type is optimized for a different workload. So follow me further test in my blog thanks.