Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2018 (Copenhagen), 14 Nov 2015

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Using Kubernetes Local Storage for Scale-Out Storage Services - Michelle Au & Ian Chakeres

Description

Using Kubernetes Local Storage for Scale-Out Storage Services in Production - Michelle Au, Google & Ian Chakeres, Salesforce (Intermediate Skill Level)

Companies and their customers continue to generate and store data with rapid growth. Kubernetes with local persistent storage enables high-performance workloads and scale-out storage software services to keep pace with demand both on-premise and in the cloud. We'll show how we leverage the Kubernetes local persistent storage feature to run multiple distributed storage services in production clusters. Topics discussed and demoed include node and local disk preparation, local PV provisioning with multiple storage classes, workload scaling, recovery, and future improvements.

"About Michelle
Michelle Au is a software engineer at Google and is leading the local persistent storage project in Kubernetes. Prior to Google, she was at EMC, working on cluster management and cluster communication protocols. She received a BS in Electrical Engineering and Computer Science from UC Berkeley.

About Ian
Dr. Ian Chakeres enjoys building cloud software teams, products, and technology. At Salesforce, Ian is constructing software infrastructure optimized for scale-out stateful services at Salesforce using containers and Kubernetes orchestration. Ian holds an Executive MBA from the University of North Carolina KFBS, a PhD in Electrical Engineering from the University of California Santa Barbara, a MS in Electrical Engineering from the Ohio State University, and a BS in Electrical and Computer Engineering from the Ohio State University."

Want to view more sessions and keep the conversations going? Join us for KubeCon + CloudNativeCon North America in Seattle, December 11 - 13, 2018 (http://bit.ly/KCCNCNA18) or in Shanghai, November 14-15 (http://bit.ly/kccncchina18).
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.

A

All right, everybody we're gonna, go ahead and get started today. We're gonna talk about kubernetes, local, persistent volumes in production, and if you sat through Sods talk, it's a great introduction to kubernetes storage and we're really going to kind of build on that and talk a lot about local persistent volumes. Michelle is the primary developer and really the lead when it comes to local persistent volumes and kubernetes she's been working on it for more than two years and I've spent more than a year working on it as well.

A

So today, we're gonna actually tell you some of the things about how you will leverage or how you can leverage and some of the tricks for leveraging local persistent volumes in kubernetes. The agenda is I'm gonna start out by talking about why kubernetes, and also why local storage and some of the work that we've done at Salesforce over the last year, then we're gonna dive into local persistent volumes and give you more information about when you should leverage them. When you should not what is good about them.

A

What some of the caveats are and then the bulk of the talk is going to be about the local volume life cycle, so part of the life cycle saag talked about when it comes to persistent volumes and persistent volume claims, but with local volumes. There's some additional work that administrators need to do to set up and handle the life cycle of these local volumes and we're going to talk a lot about that and something about node preparation, essentially preparing these local volumes for use in kubernetes.

A

Then I'm going to give a brief demo about some of our node preparation at Salesforce and then we're gonna close out by talking about the future roadmap as well as summarizing the talk.

A

So why are we using kubernetes and local volumes of salesforce at salesforce? We are seeing an extreme growth in the demand for storage, and we ourselves at salesforce, are consuming that storage, as well as we're storing things on behalf of all of our customers, and really this isn't looking like a linear, Kirk graph.

A

You know this is going or going growing like a hockey stick and it's the responsibility of the infrastructure teams and one of the teams that I was on to kind of keep ahead of this growth curve and just to give you an example of the type of growth that we're seeing we're seeing a growth rate that's higher than our stateless workloads, especially for resources for storage, and so we have more than doubled our kubernetes production fleet over the last year and we've actually really energized more than 10 petabytes of storage capacity that is made available through kubernetes and the storage services themselves are running on kubernetes.

A

So why did we choose kubernetes and why are we leveraging it for running storage services? So last year, Steve Sankey from Salesforce came and talked about our work with stateless workloads, and so we have had experience running stateless workloads and the developers love it. We love the whole workflow for kubernetes and its deployment of services.

A

The way that you develop the way that you manage them as you go forward and the scalability associated with them, and so things like a manifest check in to production in tens of minutes are really powerful when you compare them with legacy imperative mechanisms related to deployment and management of these services, including the build phases as well, and so we wanted to provide that same type of experience to our storage services.

A

As I mentioned, we have a lot of different storage services that store the data that arrives into Salesforce, and so these storage service owners want that same type of experience and some of the key things that we needed to do. There were to leverage our local storage.

A

We are all in the cloud much in our first party data centers, as well as across many of the different public clouds, and so leveraging that local storage was really key for providing these storage services and some of the other things we loved about kubernetes were things like immutable containers, declarative manifests and the active reconciliation that's happening, and these are really key for that scale aspect. So when we want to increase the cluster size, it's a manifest change when we want when a note goes down, active reconciliation takes care of it.

A

So kubernetes has been a really an enabler for our storage services.

B

All right, so, if you are, if you attended a sods talked just before this, you can see that it's really easy in kubernetes today to access remote storage by using persistent volume, claims in storage classes and dynamic provisioning. But there are some reasons why you might still want to use local storage in your environment.

B

For one reason, if you have high performance workloads that need direct access to SSDs to bypass the network, another reason is that you might want to save some costs, because local disks are generally cheaper than their remote counterparts and especially in bare metal environments. You might have a lot of spare disks lying around that you want to use for more efficient and better utilization, but there are definitely major trade-offs.

B

You need to consider if you're going to use local storage in your pods first is now your pods will become less flexible in terms of placement, because now your pod you're tying your pod to the availability of your node and your disk. So if your node or disc becomes unavailable or fails, your pod is also going to become unavailable until you can resolve that issue, so that potentially results in lower availability for your pod and in addition, local disks are generally not replicated, so you might encounter lower data durability as well.

B

In many cloud environments, the local disk offerings are basically offer very low data durability guarantees. If the VM dies, then all the data on the local SSD is wiped out. So you need to be really careful if you want to consider using local, persistent storage and you have to really understand the trade-offs and make sure that your workloads can handle all of these trade-offs. So for that reason, local storage is definitely not a general purpose solution and it is really only suitable for very specific use cases.

B

One such use case is those for distributed data stores. These workloads typically shard and replicate their data across multiple notes so that they can handle potential, node failure or data loss on any of those notes. There are quite a few applications that fall into this category, including SEF, Cassandra, cluster, HDFS, etc.

B

Another suitable class of applications are those that require high performance on disk caches, typically for those because it's because it's caching it it can survive handling losing that data or having the node fail, because you can always restore that data from its source, but they still want some semi persistence of the data so that they can avoid the cold. We start issue and avoid having to reload its cache from the source every time a POC restarts some examples of these include data analytics.

B

So those are some examples of applications that are suitable for using local storage. It's not a comprehensive list, but those are the major ones that we have been targeting as part of this feature.

B

So the main problem is is that before we develop the local persistent volumes feature, it was very difficult to actually run these types of workloads on top of local storage in kubernetes. To give some background on that previously, the only way to access local volumes in kubernetes was to use something called a host path volume. Here you are specifying the path to your filesystem correctly in your pod spec, and this has a lot of problems first. This is not secure because you are allowing your pod aka your user, to specify any path in your cluster.

B

So you need to really trust that your user doesn't make a mistake and also trust that your user, you know you have to you- have to trust your user to choose the right path.

B

This is also not portable, because you've encoded your node specific paths into your pods back. So if you wanted to take your pod and run it on a different cloud or you know a different cluster that has its node setup, node and disk setup, all completely different, you would have to go and manually change all the pod specs to conform to the disk layout on your new environment.

B

So if you imagine having to manage like thousands of these pod specs and you want it like, and then you want to port your workload over to a new cluster, you have to go in and manually change all those thousand specs to conform to your new environment.

B

Similarly, post path volumes, doesn't it doesn't really provide any way to do scalable disk accounting from here? You can't really tell what disks are available in your cluster and it's really hard to tell what disks are being used by other pods in your system.

B

So all of these problems put together make needed really hard to use local storage in production at any sort of scale.

B

What many workloads have had to do is to write very complex operators that basically implement a disk reservation and management system and also have to implement manual scheduling of pods to those individual nodes, and when you start having to manage individual pods, the major downside there is, you can't take advantage of existing high level controllers and kubernetes such as stateful sets.

B

So if you wanted to, if you wanted to do, if you wanted to handle scaling or upgrades or health checks or any of the roll backs and any of those other stateful set features, you would have to end up. We implementing those in your in your custom operator, so the local persistent volumes feature aims to fix all of those issues by leveraging the existing persistent volumes interface that we have in kubernetes today.

B

By doing this, you now get security because only administrators can create and define the content of the persistent volumes you get portability, because the persistent volume claims will add a level of indirection in between the pod and the underlying storage implementation. So if you look at this speck here, you've noticed now. So if I go back to the hosts path, spec I was encoding specific paths in my cluster in my pod. But now, if I use a persistent volume, claim, it's completely portable there's nothing in here. That gives any hints of any particular.

B

You know environment or node setup. In addition, persistent volumes and persistent volume claims gives a better way to do disk accounting in kubernetes, because every single disk is explicitly exposed as a persistent volume and those persistent volumes have statuses of available or bound. So you can easily tell which disks are available in your system and which disks are being used by exactly which claims and which pots so by and by using the persistent volume claims objects. Now you can really take advantage of the powerful features built into stateful sets.

B

The stateful sets include volume, claim templates built-in, which gives you one volume per stateful set instance, and it automatically handles things like scale up and scale down. So if there are existing operators or helm, charts using stateful sets that were previously just tested against remote storage, now you can easily leverage those same stateful sets and just switch to local storage just by changing the storage class name.

B

So this feature this local persistent volumes feature, is now available in beta as of 1:10. There are a few limitations for this initial beta phase. First, you must format and pre mount the disks first on the notes and we don't support dynamic provisioning yet, but those those two items are under active development currently and also as part of this feature, we made significant enhancements to the scheduler first to provide data gravity. So what this means is that if you have a pod, that's using a persistent volume claim that points to a local volume.

B

The scheduler is smart enough to understand that this local volume is tied to a specific node and will always schedule your pod to that correct node, and on the other side, we've also made the pod scheduling and initial volume binding operations more tightly coupled together, so that the scheduler and the volume controller can coordinate better when scheduling pods. So now, whenever we we are, whenever the volume controller is trying to find a volume to bind the claim to it's, not only going to look at all the volumes available on which nodes.

B

It's also going to look at your pod requirements such as CPU memory affinity and high affinity, nodes, tints and Toleration. It's going to look at everything, along with all the local volumes that are available and be able to choose an appropriate node that can access the storage. You need and also run your pod.

B

That same logic also handles a case where, if you specify multiple claims in a pod, we make sure that we will bind to local volumes. That are all on the same note.

B

So that's some general back background on the future. Now, for the rest of the talk, we're going to focus on the local volume life cycle from when you initially set up your cluster and your disks to win your pods end up consuming and releasing the persistent volumes.

B

We've split up the phase into the life cycle into two main phases. The first phase is the node and disk preparation parts. This is specific. This is specific to your environment and it involves things like partitioning or rating the disks formatting and mounting them once the environment has been prepared. Now it's ready for use in kubernetes and it's going to enter the second phase, which is the persistent volume life cycle.

B

We've written a open source controller that can automate this part of the life cycle for you and it's going to handle things like creating the persistent volumes and then, when the pods end up consuming and releasing these volumes, then it's going to clean them up, delete them and then put them back into the kubernetes pool for use, so we're going to deep dive now into each of these phases. So going back to the first phase, which is the node and disk preparation. This is so you might be asking.

B

Why is this part environment specific and why can't kubernetes? Do this part? For me, the main challenge here is that there are so many different choices in methods that you could use to configure your local disks. You can partition them. You can, you know, do any of these raid configurations. You can use LEM, there might be proprietary methods and setup that you need to do and it would be very difficult to encode all of this into the API kubernetes api.

B

So we chose to the API very simple and just have a path to the mounted volume passed in and then leave all the steps before that on how you actually get to that path up to the environment.

B

So, going back to all of these choices, you might have what what which choice should you choose or which you know which one or number of combination of these should you choose? The answer really depends on what workloads you intend to run and support using local storage, for example, if your workload requires higher data durability, you might consider rating the disks together, but if you are just if the workload is just using just once the fastest disk available, then you'll give it the whole disk.

B

A lot of it also depends on the platform and ops requirements in your environment. So if say, your workload requires more capacity, that's bigger than any disk capacity you have. Then. You may also want to consider aggregating the disks together into a larger logical volume, to use you might be limited by the number of disks available on your nodes.

B

The capacity of those disks and also the speeds of those disks there might be there might be special reserved disks on your hosts that you don't want kubernetes to touch, and you might be bound by existing processes and tools in your company for the node setup such as ansible or puppet. You might need to integrate. Consider integrating all this all these steps into that or potentially not so putting all of these factors together is really what's going to determine what is the best way to set up the local disks in your environment.

B

So now we're going to go we're going to give some specific examples of how this node preparation step is done in some real environments here, so I'm gonna start by talking about how note preparation is done in gke. I would consider this to be a very simple use case, because gke is a cloud provider platform.

B

The disk naming of the local SSDs are very well defined across all machine types. So what we've done here is we've written a very simple shell script that runs whenever that runs whenever the VM boots up and before kubernetes actually starts, and it just looks for disks at predefined locations and then it will go ahead format and mount them into directories that kubernetes can understand later and because also this is a cloud provider service.

B

We've actually included this node preparation step into the whole cluster, bring up so as a gke end user. You don't actually have to do the node prep. You just have to create a cluster with some local disks and by the time, the clusters up and you can log into it all. The disks have been prepared and formatted into the correct locations.

B

So again, I would consider this a pretty simple example on the spectrum now I'm gonna hand it over to Ian who's, going to give an example of a more complicated environment.

C

A

I'm gonna talk about a more complicated environment, I'm gonna talk about first party cloud data, centers for Salesforce and some of the things we encountered and also some of the design decisions that we made I'd mentioned earlier about the scale at which were operating and that's very important, but also we have a very heterogeneous set of environments, so one size does not fit all as well as a whole suite of processes and tools around provisioning around accounting, around financial items, the different disk, layouts and configurations.

A

So we have a lot of different things to consider and it's much more complex than a simple shell script. One of the initial decisions that we made, which is kind of interesting, is we chose to perform the node preparation after kubernetes is already running.

A

One of the reasons for that was, we didn't want to become embedded with other processes, tools and configuration, and our teams already working very actively on kubernetes and we loved some of the features I'll talk about some of the features that we really leverage specifically daemon sets in this slide and in the demo. So as an example of some of the challenges and some of the things we need to do, each of the environments may be different.

A

Each of the different server technologies may be different, and so we actually describe in a claret of manner both the configuration that we expect, as well as the configuration or what we want.

A

This node preparation phase to do, and so we have a daemon set that consumes, that particular configuration and information and that node prep daemon set is watching all the time for whenever new servers roll in and our provisioned and our setup by kubernetes, so that it can be provisioned and part of this node preparation, so that those local disks and local storage resources that we want to be exposed to kubernetes for storage services can be used in later parts of the lifecycle, so that node, prep daemon set is doing that operation and it's doing things like partitioning the disks.

A

Cleaning them up or mounting them can deal with accounting as well and inventory and really at the end of it. It's preparing things for that local volume provisioner that Michelle mentioned, and so what the preparation there is to place these resources in a directory that can be discovered by the local volume provision and using some of the daemon set magic, which I'll demonstrate. We actually kind of do a very clean handoff between the node preparation phase and the later phase of the lifecycle, which includes and is managed by the local volume provision.

A

So here's some of the manifests from our daemon sets so I mentioned. First, we have the node prep daemon, set on the left hand side, and then we have the local volume provision or daemon set on the right hand, side and on the left hand side. We say hey. If there's no label for node preparation, no preparation has not been completed on this. Node then run this daemon set.

A

And so what will happen is if a new rack or a new servers enter this cluster, then the node prep daemon set will run automatically on those nodes and on the right hand, side. We say wait until this label exists and the value of that label is mounted to indicate that node preparation has already completed and what you'll see is actually the node preparation labels, the node at the end of completing its job and when that label appears.

A

Kubernetes will actually remove the instance of the node preparation and it will start up an instance of the local volume provisioner.

A

So we talked a lot about phase one and that node preparation, which is really environment, specific and now I'm, going to cover the second part. The second phase here, which is not environment, specific.

A

We are leveraging the open source, local volume provisioner the same one, that Michelle mentioned that gke uses and other people can use for the second phase of this lifecycle that is all occurring within kubernetes and what the local volume provisioner does is it's looking for certain it's looking within certain discovery directories and any of the directories that it finds the local discs that it finds in those directories.

A

It will create persistent volumes for those once those persistent volumes are created in kubernetes, they become consumable, so you can write a persistent volume claim or a volume claim template in your stateful set and when that pod and that workload is scheduled, it'll actually be allocated and bound with persistent volumes, and so this is kind of the same life cycle as normal persistent volumes, even if they were non-local, the difference being that these are going to actually leverage the local discs instead of a network-attached disc.

A

And so it follows that same life cycle, where you can claim them, they become bound eventually at the end of their life, they can be released and when they're released the local volume provisioner takes care of cleaning up the kubernetes resources, as well as the local discs, and then this life cycle can repeat in terms of those persistent volumes of any discs that are not represented in kubernetes will be reintroduced as persistent volumes that can be claimed so now I'm going to give a demo of some of that and daemon SAP magic for the node preparation, as well as for the local volume provision.

A

Now in this demo, we have nineteen notes. We do not have the node preparation running because as soon as the node preparation day, misete runs it's going to prepare everything we do have the local volume provisioner running, though, and so what you'll see here is that we we have the local volume provision or running, but there are no instances of it running because none of the nodes have the node prep label. So what we would expect is it would be nineteen if it all the nodes were prepared, and the other thing you can see here.

A

There are no persistent volumes configured in this cluster, so here I'm, showing the manifest for the local volume provisioner and showing you that, yes, it says right here. If you see the node prep label with mounted, then I want you to start running. The daemon set an instance of that daemon set on this node and it's not running yet.

A

What we're gonna do we're gonna start up the node prep soon, so here's the node prep configuration and as Michelle mentioned, this is specific to our environment, to a set of machines, a specific data center or a specific cluster. And what we're showing here is. We want to expose several different discs at these well-known locations and we want to expose those for discovery in a different location. That's going to be configured and managed by the local volume provision and what you see on the bottom. There is the SSDs. Actually, there is no data zero.

A

So just keep that in mind as we go forward, because maybe this disk is being leveraged by another system process and we don't want that exposed through kubernetes. Now what I'm, showing here is the FS tab or what is mounted on this machine, because this is important. These are the disks and the drives that are on this machine before node preparation and what you see is there are three hdds and two SSDs, so the data and the date fast data there and what we're gonna do now is we're gonna, actually start that node prep.

A

So when the node prep starts, what we expect to see is instances of the node prep daemon set to run on every single one of the notes and then, when it completes its job, the local volume provisioner will start running on each and what you see is very quickly. The node prep goes to nineteen instances, one for each node and then it actually completes its job and goes to zero. And what you see is the local volume provisioner. It started at zero and then went to 19.

A

So now the local volume provisioner continues to run the node prep is all done, so all that work has done. The local volume provisioner discovers all those resources based upon them being placed in the discovery, directories and you'll see here we have now 95 different local persistent volumes. Several of them are HD DS and some are SSDs now.

A

The final I'm going to show here is what happened to our FS tab right, our mount points, because we needed those to be remapped and mounted so that the local volume provisioner could take over and also we want this to be persistent right. Every time this node boots up or every time this machine is being used. We want those mount points to persist, and what you see here is that the HDD s and SSDs have been vine mounted into the location expected by the local volume.

A

Provisioner and you'll also notice that fast data 0 isn't there. So this is kind of that example of you may need to do things for your environment and you may not want to expose all the local resources that are available.

B

Alright thanks Ian, so yeah does a great example of a very complex environment where you have different machine types with different number of discs on each node, and it was good that you guys are able to leverage able to automate all of that.

B

So to summarize, hopefully, you guys now have some good ideas about how you can leverage or how can you can do local disk administration in your own clusters? There is some work that you need to do it's not as magical as cloud providers, storage where you can just click a button and it disk shows up. But hopefully we showed you today that the process can be automated and made made manageable.

B

And then, if you step back a little bit, you can see that this node preparation step is just a relatively short part of the entire local volume life cycle. Once those disks are prepared. Now you can hand it over to the local volume static provisioner to manage the remaining part of the kubernetes life cycle, as as pods can now consume them, and they end up eventually getting released.

B

And then this provisioner can now manage the cleanup and reuse of those volumes, and perhaps the the most powerful part is that from the users from the users perspective, their workflow has not changed at all. Most of the work is hidden underneath in the admin flow, but to the user. They can still use reuse. The existing persistent volume claims that they have been already using in their saiful sets and to just use local storage. All you have to do is change the storage class name in your persistent volume claim you don't have to.

B

Actually, you know include all the complex logic in your operators that you used to have to do. If you wanted to use local storage so looking forward, there are still plenty of enhancements planned in this area. One enhancement is to enable raw block raw block volumes. This is now available in alpha in kubernetes 110 and it's very useful for those use cases that can get higher performance by bypassing the file system layer so, for example, at Salesforce. They intend to use this feature to store these self metadata on.

B

Another very popular ask has been to support dynamic provisioning with LVM. This is very useful for those environments where you want to have simplified disk management and better disk utilization, as long as your workloads can handle the performance penalty of potentially sharing discs with other workloads and finally, the last major improvement.

B

We're looking at is to do the filesystem formatting and mounting steps in the kubernetes plugin itself right now you have to do this in the node prep step, but once we can do this in kubernetes now you can remove that from the node preparation, and so that will really simplify those simple environments where you just want to expose the whole disk to two kubernetes. So then you can just instead of having to partition and format and mount them. All you have to do is just tell kubernetes the paths directly.

B

So if you're interested in using this feature, we have a lot of documentation available, both the official kubernetes documentation and ian, and I have also done a few blog posts on all of the stuff we've discussed here today.

B

You can find these slides on my speaker, deck file, speaker comm, / NSA, you 42, so you'll be able to download the PDF to these slides and get these links can give you time yeah, and so, if you have any further feedback or questions or you want to get involved in helping to develop and test these features, please join us in the storage cig, we hold meetings every two weeks on Thursdays and we are also active on the six storage slack Channel as well, and you can also contact Ian or me directly.

B

Our handles github handles are here so yeah. We can open it up for questions. Thank you.

D

All right thanks great talk, any hooks to extend the anti affinity policies for for scheduling. My stateful sets to kind of take into account. For example, I could I could read from my matrix collection, how are the disks being utilized and encounter? Basically, the performance penalties with the dynamic provisioning.

B

So the question was, you want to dynamically, create anti affinity policies based off of metrics? Yes,.

D

So, basically, to avoid having the performance pay, no penalties, then.

B

D

Multiple disk intensive applications, okay.

B

You can potentially, let's see so if you want to dynamically, generate anti affinity policies, you probably need to have like an operator, an operator that will read the custom metrics and then generate the anti affinity. I mean you can always try to. You can put anti affinity on pods or sorry on nodes into your stateful set and that will spread your pods across nodes to spread the IO that way, but you're asking about spreading.

D

B

O----, so you can actually do pod anti affinity. In that case, right, you can say if you have two different io intensive workloads. Oh.

D

B

They're, your users pods, oh, that's harder, but yeah that could be something yeah. The suggestion was to write an admission, a dynamic admission webhook to be able to dynamically generate those pod palettes and high affinity, pod policies, but I think I. Think pod anti affinity is what you or look for to be able to say two different workloads that are both I/o intensive. You want them to go on different notes. Yeah.

E

Hey thanks really enjoyed the talk for the local provision, or can it say, detect disk failure and then move a pod like reschedule pod off of a node. If say off, like an operator that handles like changing, replication and stuff great.

B

Question so right now we don't have health metrics built in right now, but it's currently I think some people are experimenting with that. To add, be able to add health checks and to be able to put some sort of metadata in the persistent volume object to denote the health of the volume and then from there. People can build higher level controllers and operators to be able to react to those conditions, so you can say like for your application.

B

You can say if this volume becomes unhealthy, then, and my application can handle the failover of that data.

B

Then you can, you know, start the failover process to release to delete your persistent volume claim which will delete the disk and delete the data, and then that will free up your pod from that and from that node binding and then allow your pod to reschedule and get a blank new disk, and you know, do the reconstruction or whatever it needs to.

C

Yes, so I have a question for you, so you mentioned, salesforce is managing like morning PBS of data and then even the lists are local storage. Is the operator feature so I wonder how much of data are actually using this feature? And how long have you guys been using.

A

So I can't tell you about the data side in terms of how exactly we're using it, but I can say: we've spent multiple years using kubernetes in production at Salesforce now, and that has given us the confidence just running our storage services there, as well as I, would say our storage services or replicated data stores.

A

So we do retain multiple copies as well as application owners that we talk to for leveraging those storage services, often leverage another storage service for really high durability situations, blob and object stores, just like you would in any other cloud environment.

F

Again great talk, one of the most useful and applicable things to my personal issues, I've seen all week. So thank you. Two questions. First of all, I should imagine this room is gonna, go and start like some open-source pre provisioned their daemon sets now probably like 10 or 12 of them. Is there any chance you can just open source the one you have is like a base for the community, and the second question is you said that when a pod reschedules, it will always get the same volume it had in that stateful set.

F

What underlying kind of kubernetes primitives? Is it using like? What is that in excellent.

B

Question so as part of making local storage work in kubernetes, we completely revamped how the scheduler works in its understanding of volumes, so we've added a node affinity field to the persistent volume. So this is just like how in the pod, you can specify node affinity to force your pod to go to specific nodes. Now you can put it on a volume instead, so you no longer your pod no longer has to be aware of the node constraints of your volumes. Your pod doesn't need node affinity.

B

Now, if it's using a volume that has no affinity on it, the scheduler will still be able to schedule it correctly. Based on that, so this actually works for more than just local storage. It works for zonal storage and it also works for really crazy distributed systems that might only have disks available in like three out of ten nodes so and it's based all off of labels, so you can define your topology for your volume exactly you want, and.

A

I'll follow up with open sourcing, so Salesforce is pretty big in open sourcing things and sharing information. I don't have a particular timeline, but I will say in addition to things like node preparation, we have higher level abstractions that help with distributed data stores, and we do intend to open-source those so keep an eye out. I think it's Salesforce github dot and that's where it would end up.

G

Great talk just a follow up question, um so you mentioned you made. Some changes to scheduler is aware of local storage, and also you had a slide about. Local storage is not prime for general purpose. Storage. Instead use distributed file systems like safe HDFS. So in scenarios like that, where you have HDFS, let's say doing 3-way replication, should there be a mechanism for this type of the sugar for systems to communicate with kubernetes, to say where the replicas are. You know, which knows there are yeah.

B

You can do that exactly using the persistent volume, node affinity so say. If HDFS you have replicas on node, 1, 5 and 10 in the volume for your HDFS volume, you can say you could put a node affinity to it on nodes, 1, 5 and 10, so we will always schedule your pod to one of those three nodes and it you need to. You need to represent it as a volume. An HDFS volume.

A

I'll add to that a little bit kubernetes. One of the really powerful features is all its labeling capabilities, and so as an example, if you care about racks right, you can label your objects and make your system knowledgeable about that type of topology information and leverage it. When doing things like scheduling and in the reverse, let's say: you're a storage service. You can add labels and things to pods and nodes and racks and persistent volumes that allow other controllers to watch and observe and reacts to those labels.

A

So kind of labels can provide a communication mechanism that is asynchronous between different processes.

B

Yeah, so the whatever the SEF provisioner or cluster provisioner is in you need to somehow encode the topology information into the provisioner. So when the provisioner creates those volumes, it can fill in the node affinity.

B

The other cool thing about node affinity is it can support requirements and preferences. So you could do things like my volume is globally accessible on all nodes, but I prefer your schedule on these nodes. When my replicas are so I can get better performance.

A

Alright, thank you. Everyone.