Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2020 - Virtual, 4 Sep 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Speed Racer: Local Persistent Volumes in Production - Matt Schallert, Chronosphere

Description

Don’t miss out! Join us at our upcoming events: EnvoyCon Virtual on October 15 and KubeCon + CloudNativeCon North America 2020 Virtual from November 17-20. Learn more at https://kubecon.io. The conferences feature presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Speed Racer: Local Persistent Volumes in Production - Matt Schallert, Chronosphere

Local Persistent Volumes are a powerful feature of Kubernetes, enabling users to build and operate high-performance stateful systems in their clusters and bridging the storage gap for on-premise users. However, their usage comes with subtle pitfalls that users may not be aware of, and understanding of these aspects is key to a seamless production experience. In this talk, Matt will share best practices and how-to’s for local volumes based on experience running them in production since their alpha release. He will cover topics such as in-cluster volume provisioning, best practices not obvious from Kubernetes documentation, and cloud provider-specific local disk behavior. The audience will walk away with a better understanding of the tradeoffs of local volumes and how to leverage them in their workloads, concrete tips for avoiding common mistakes, and day-two operational best practices.

https://sched.co/ZenO

A

Hi everyone, my name- is matt shalart and today, I'm here to talk to you about using local persistent volumes in production. This talk is kind of supposed to be everything I wish I knew about local volumes when getting started with them. I found that there's a lot of great information out there about local volumes, but it's kind of spread throughout various places such as the kubernetes, docs, blog posts or external repos, and so I'm hoping that this can kind of be a helpful holistic overview.

A

A little bit about myself. I'm an infrastructure engineer at chronosphere chronosphere is a hosted metrics and monitoring platform, particularly built for large-scale high-throughput use cases. So you can imagine that we care a lot about disk performance performance, cost trade-offs and similar that we'll talk about later chronosphere is built on top of m3, which is an open source, metrics and monitoring platform that we are also maintainers of and before working at chronosphere. I was an sre at uber, where I worked on the in-house metrics team there.

A

So let's jump right into it. What are local, persistent volumes? Local persistent volumes are a feature of kubernetes that combine two potentially familiar topics and they're both kind of right there. In the name, the local part of local persistent volumes refers to local disks.

A

Local disks are something that are offered in some form or another on most major cloud providers.

A

The general idea is that they are fast disks that are attached directly to the host the server hosting your virtual machine. This is in comparison to typical remote storage offerings, which are usually offered as a service. They go out over the network in this case, since local disks are directly attached to the node, you get significantly better performance and as well as more consistent performance, so the tail latencies you observe will be more predictable than with something like remote disks that have to go out over the network.

A

However, since local disks are attached directly to the node, the data only persists for the lifetime of the instance. So this means that if your instance is terminated or there's some sort of host maintenance, you may you might lose all the data on that local disk.

A

However, although there's this downside of the data only persisting for the lifetime of the instance, the benefit is that, because the disks are attached directly to the host, you get this significantly better performance and at a reduced cost and we'll take a look at kind of what exactly that performance and cost differences are, but because of all of this local disks are better suited for kind of a narrower range of use cases than the more general remote disks and we'll also discuss that in more detail. But first, let's just take a look.

A

How much better is the performance that we're talking about so these numbers examine gcp's, local ssds, op, less local ssd offerings, since they publish their numbers pretty clearly, and it's easy to take a look at them. Aws and azure both have similar, if not better relative performance and cost numbers that we're talking about here. So the same benefits kind of apply across multiple providers and if you're curious, the cockroachdb cloud report kind of breaks down these storage characteristics across a variety, more dimensions than we're just looking at here, but even still so on.

A

Gcp local ssds can offer between 7 and 12 times, better read and write iop. I o operations per second compared to the fastest equivalent for most remote disk, offering the throughput stats are also dramatically better, so you can get over twice as much read throughput and roughly 1.7 times right throughput compared to any given persistent disk. So because of this balance of I o performance and right throughput, local ssds typically are suited for more kind of random, read and write use cases, but in any k, in any case, these numbers are, you know.

A

Obviously, the performance here is significantly better than just using remote disks, but the real benefit of all this is that, in addition to being more performant, local disks are also much cheaper than uh than remote storage. So, in gcp's case, local ssd storage will cost eight cents per gigabyte, whereas uh remote disks, the fastest remote disk, offering, will cost 17 cents per gigabyte.

A

The persistent volume as uh part of local persistent volumes refers to persistent volumes and kubernetes that you might be familiar with uh so local local, persistent volumes kind of combine these two topics, but first, let's take a look at uh kind of some of the behavior characteristics of persistent volumes in kubernetes.

A

So in kubernetes a persistent volume is a resource that represents some sort of remote store or some sort of storage resource.

A

You know, storage that can be concretely mounted and forked with a pod and red or red, read from or written too, and a persistent volume claim is the claim that binds a persistent volume to a given pod.

A

So a pod will be a persistent volume claim will have a pod that can uniquely use it that persistent volume claim will bind it to some persistent volume. Resource kubernetes ensures that your pod is mounted with the store with the storage backing its persistent volume claim. This typically happens by first attaching your persistent volume to a node that your pod is running on and then making uh that attached persistent volume available to the pod itself.

A

However, for a long time with persistent volumes, there was kind of this assumption that storage could just move around with pods. So say your pod was running on node a and it had some persistent volume attached to it. uh If node a failed, kubernetes would take care of rescheduling your pod onto say, node b, and it would be able to attach that same persistent storage to the new pod or to the same pod on a new node, but local volumes, kind of uh local disks, rather kind of violate this assumption.

A

So you know, there's this assumption that storage could move along with a pod from node to node, just like kubernetes can kind of schedule your pod on any node.

A

So this is kind of in conflict with some of these properties that we've talked about about local volumes, where they're tied to the lifetime of an instance. So, in this case, if your pod is running on some node and it has local storage attached to it and that node fails, kubernetes can't just move that disk to a new node, as you can't just detach a physical disk to attach directly to a host machine and move it somewhere else.

A

Similarly, you wouldn't actually want kubernetes just rescheduling your pod, even if it could, if it was using the local volume. uh So this is, you know, outside of the case of node failures, just in normal operations.

A

If your pod was running on node a and had some local volume attached to it and your pod was suddenly rescheduled onto node b, your pod would actually have lost all of its data as all that, as all that data is stored on the local volume attached to node a so if kubernetes rescheduled, your pods, suddenly you've lost all your data.

A

So how can local volumes work um rather than solving just for the use case of local volumes? The kubernetes community solved a more general problem and or came up with a more general solution and that's called topology aware volume provisioning.

A

So what this means is, whereas previously, when a persistent volume claim was created, a persistent volume would be attached or would be provisioned pretty immediately. After now, when a persistent volume claim is created, it actually waits for uh the first pod. That's going to use that uh use that persistent volume claim before uh deciding where to schedule both the pod and the persistent volume.

A

So this means that the scheduler can take into account both the pods affinity requirements, as well as the storage's requirements, and this helped not only with the problem of local volumes, but also with the problem of multi-zone storage in kubernetes, even with remote storage. So previously there was a behavior. Where say you were running a regional kubernetes cluster and you scheduled a pod in zone. A your storage could have actually gotten provisioned in zone b and then suddenly, your pod wouldn't be able to use that storage because they were in different zones.

A

So, by solving for topology aware volume, provisioning kubernetes solved for both this use case of multi-zone storage as well as local disks, and this is kind of something we frequently see in the kubernetes communities rather than taking an immediate problem. Solving just that narrow use case kind of figuring out a more general solution that works for multiple problems, so under the hood persistent volumes are, are now have a field called node affinity on them, and this is similar to the node affinity terms.

A

You might see on a pod, so this allows the persistent volume to express um requirements for where it's scheduled. So in the case of a local volume, maybe it can only be scheduled on a certain node or in the case of zonal storage. It can only be scheduled on nodes in the given zone.

A

So how do you get started using local volumes? What are kind of some of those day? One operations well to get up and running with local volumes in order to provision them. There's a tool called the local static provisioner.

A

So with the local static provisioner, you take care of mounting disks at a certain path on your node you'll point the provisioner at that path, and the provisioner will pick up all of the disks that are mounted and turn them into kubernetes persistent volume resources.

A

Additionally, the provisioner will take care of life cycle tasks such as wiping the disk before and after it's mounted uh or when the persistent volume or sorry wiping the disc um when the persistent volume claim is deleted or the disc is first provisioned kind of some of those uh life cycle. Tasks that mimic the behavior of you know: new storage being attached in order to first use local storage you'll create a storage class dedicated to local storage.

A

So this just pretty much specifies a type of storage that you'll attach that you'll use with your workloads and then in your various workloads that you're using disks for in the persistent volume claim templates uh rather than just or rather than specifying a remote storage or not specifying a storage class you'll just specify the storage class that you created dedicated for local disks and that's it. You create a storage class. You move all of your workloads to it and you have 10x faster storage at half the cost, if only wouldn't that be nice.

A

Unfortunately, because of some of these properties that we've discussed local disks do behave very differently than regular persistent volumes that you might be used to so safely. Using them requires more planning and kind of operational experience.

A

So what are some of these uh day? Two operations that you know you might have to take into account these differences and behaviors the first one that we'll look at is what happens during node failures.

A

So one of the promises of kubernetes is that nodes can fail and kubernetes will help your applications be more resilient by rescheduling your workloads on different nodes and kind of abstracting away. These failures for you, additionally kubernetes will not just go randomly deleting your data as though that wouldn't really be helpful uh from like an orchestration perspective or any perspective.

A

However, both of these behaviors combined means that, if a node fails and that local persistent volume no longer exists, the pod using it will be stuck. So this is because you know, as kubernetes, won't just delete your data if a pod was using local storage, that local storage is no longer available, kubernetes won't reschedule it because that would mean bringing up that same pod uh with local with without any data or sorry won't reschedule it without manual intervention.

A

So, in order to allow this pod to be rescheduled, you have to delete the persistent volume claim and then that pod will be rescheduled with a new persistent volume attached to it, um and this means that unless you automate it, this is a manual operation. However, at the same time, automating it means that you're automating kind of the deletion of data, so you kind of have to take some care with this.

A

uh Another kind of day-to-day operation that you'll have to take into account is how you handle backups and restore of your data. So with remote disks you might be used to snapshot and restore, which is a pretty popular kind of disaster recovery mechanism offered on most cloud providers.

A

So with snapshot and restore, you can just take point in time snapshots of your remote disks at any time and uh bring up new copies the remote disks, um with uh with all the data from that snapshot- and this is kind of helpful- you know if you were to accidentally corrupt your data or delete something kind of just being able to go back to any point in time and recover that data.

A

However, there is no snapshot functionality for local disks on any cloud provider. um Additionally, there's kind of, as we talked about earlier, since uh local disks are tied to the lifetime of an instance. There are few to no guarantees about the availability of that local disk. So your most cloud provider, documentation it'll talk about some of the caveats of using local storage, but the bottom line is that they all kind of say.

A

You know you can't really depend on that data always being there, because if there is a node maintenance or a maintenance event on the host machine or some wider maintenance event or a node failure um that local disk might be uh might no longer be available.

A

So this means that you are responsible for uh backing up any data. That's stored on local disks.

A

One pattern that can help here is a sidecar pattern with you know for backups, so you can have some pod that runs alongside your app and copies all of your data up to an object store on you know some sort of regular interval or some other durable remote storage.

A

And uh finally, it's also worth talking about cluster upgrades, so uh hopefully you're keeping your clusters up to date, but the way that uh cluster upgrades typically work on most cloud offerings is that they're done by replacing nodes one by one, with a node with the new version. So this is compared to say replacing uh where, rather than upgrading a single node in place, your cloud provider will just iterate over your nodes and kind of replace them with nodes. With the new version.

A

However, this would be problematic to do with local disks, so you can imagine if you were using local volumes with your nodes and your cloud providers started just swapping out those nodes one by one. uh You would also be deleting all of your local all of your local data one by one.

A

Additionally, even if you didn't really care about the data on disk, say you were using local volumes for purely ephemeral use cases, you would still have the problem where these pods were stuck and couldn't be rescheduled, because, as we mentioned previously, unless you delete that volume claim binding that pod to that storage, uh kubernetes won't reschedule the pod.

A

So a safer solution here is to instead kind of preemptively evacuate your node pools. So, for example, if you were going to upgrade, you know, node a you might first manually or through some automation, delete that volume claim allow that pod to be rescheduled on a new node and then actually allow that node to be upgraded and again, this kind of requires both some planning and some automation, um but once you've migrated pot, your workloads that are using local volumes away from your nodes that are you're going to upgrade.

A

Then you can either safely upgrade them or just tear them down.

A

So we've kind of talked about some of the things to consider when using local volumes and kind of these subtleties, but that doesn't mean you should just avoid them because, as we mentioned earlier, if used properly, you can get these dramatically uh improved, store those dramatically improve storage performance at a reduced cost, um so we're kind of going to look at some use cases or patterns for using local volumes that are a better fit given these given their characteristics.

A

So, first, let's talk about something that might not be a good fit.

A

Local volumes are not a good fit for things like single primary instances of a database or some other application. For example, say you were only running uh one replica or one instance of my sequel on your in your cluster using uh local volumes, which arguably you wouldn't want to do with remote storage anyway. But let's take a look at that with local volumes.

A

So say there was some sort of node failure um and the now suddenly, your my sequel pod would be unavailable, and since it's a single instance, you would, uh you know, be failing all reads and rights to it.

A

Additionally, even if you were to delete that volume claim and allow that mysql pod to be rescheduled to a different node, suddenly it would come up with a new empty disk and you would have lost all of your data.

A

So a better fit for using local volumes would be workloads that already have kind of like fault, tolerance and replication built in an example of this would be kafka which allows you to write or to set a replication factor and then handles distributing your data. So in this case uh say you had a client running that was sending rights to kafka.

A

Those rights would be replicated across three instances on three different nodes with three different local volumes.

A

This means that, even if one of your instances fail, your overall availability isn't impacted, as there are still two replicas to store the data.

A

But, more importantly, since when this instance fails, you lose the data on local disk when a new node or when a new pod comes up to replace it on a new node, it will have an empty disk, but since the or since you're using an application that can handle copying, data kafka will stream the data that it's supposed to own from the nodes that remained available and reconstructed state.

A

So, overall, you wouldn't have lost any data, although you you know temporarily, it was unavailable, but overall you wouldn't have lost any data and your overall application availability wouldn't have been impacted.

A

So, generally speaking, these workloads that are fault, tolerant and handle replication for you will make it easier and safer will be easier and safer to run on local volumes and again, especially if they handle this. uh You know some of this replication and other behavior under the hood. uh You won't have to build as much operational tooling. On top of it, there are a few other use cases that can take advantage of local disks, pretty well outside of just databases, so some data processing pipelines can use disk as kind of a spillover cache for data.

A

um So you know these cases where the data on disk is purely ephemeral um can be really helpful because, again you know, if you're, using it for some sort of local cache and like data processing, you don't really need that storage like stored on a remote store and replicated for you, because you can lose it at any time.

A

Additionally, there are some databases that can use. You know certain paths for just incoming rights or buffering incoming rights and then have a or use a different storage path for uh long-term data storage. So, for example, cassandra can do this um this, and so in this case, you could do something like have incoming rights buffered to a local disk and use a remote disk for long term storage.

A

Again, you wouldn't want to do this unless you were replicating the data across multiple instances, but this would mean that for that more performance, sensitive use case of kind of committing incoming rights, you can just use the local disk and even if that instance, it fails and is rescheduled.

A

You only lost from that one in since a short period of time, so a short period of data. So this is kind of a best of both world scenario.

A

So, to recap, some of the themes we've talked about you know, since local volumes present kind of a departure from typical operations that you're used to you really do have to consider the impact it will have on both your workloads and your day-to-day operations and you'll likely have a smoother experience, getting started and running further in production, if you at least start with these workloads that are kind of like already fault, tolerant or purely ephemeral as they'll require you to let build less operational tooling.

A

On top of that, your cloud provider will likely document the behaviors that you should expect with local disks and this knowledge, combined with how local persistent volumes and kubernetes work is going to be pretty key to painting a better picture of what to expect in day-to-day operations.

A

So um those are both to or both your cloud providers documentation and some of the the kubernetes resources are good things to kind of absorb um and, in addition to some of the practices we've talked about here, there's a really helpful best practices, dock in the local static, provisioner repo that gets into kind of more of the nitty-gritty details of mounting disks and everything.

A

uh These are just some helpful resources that I've referenced so far and I'll post these in these are. These will be in the slides that are posted afterwards and that's about all. I have thank you all for joining, and I hope that this was helpful. um Like I said we, I work at chronosphere and we have a virtual booth here where we would love to talk further with you um and you can come see a demo of the platform that I mentioned and if anyone is available or sorry I'll be available later.

A

If anyone wants to talk about uh just local disks or any of the topics, we've talked about or kind of have any further questions, um and with that I will get into the q a.