Rook Rook Presentations, 14 Nov 2015

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Kubernetes Runs Anywhere, but Does your Data? - Jared Watts, Upbound (Beginner Skill Level)

Description

Want to view more sessions and keep the conversations going? Join us for KubeCon + CloudNativeCon North America in Seattle, December 11 - 13, 2018 (http://bit.ly/KCCNCNA18) or in Shanghai, November 14-15 (http://bit.ly/kccncchina18).

Kubernetes Runs Anywhere, but Does your Data? - Jared Watts, Upbound (Beginner Skill Level)

Kubernetes is now the defacto container orchestrator and it is able to run basically anywhere, from cloud providers to bare metal clusters. With this ubiquitous ability to run your applications comes a need for persistent storage that also runs anywhere Kubernetes runs. One of the most powerful features of Kubernetes is the ability to deploy your applications consistently no matter what the environment is. The same should be true for your storage if you want to realize the full potential of Kubernetes. We will walk through the architecture of Rook (https://rook.io/), a distributed storage orchestrator for Kubernetes that provides storage to your application pods anywhere Kubernetes runs. We’ll explore how to use extension mechanisms such as CRDs as well as patterns like controllers/operators, resource management, and rolling upgrades to build a portable Kubernetes storage solution.

About Jared
Jared Watts is a Founding Engineer at Upbound, where he is working on advancing cloud-native computing by ""freeing the cloud"" and is also a maintainer for the open source Rook project (https://rook.io/). Before Upbound, he worked on innovative storage projects at Quantum Corp. and helped found Rook. Prior to Quantum, Jared worked on P2P storage distributed across the public internet at Symform, a startup later acquired by Quantum. Before that, Jared began his career at Microsoft working on applications for Windows Server setup and management.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.

A

All right so welcome to my talk here. Guys, thanks for showing up in the afternoon, appreciate it on the last day of the conference, especially so. The title of the talk here is that kubernetes runs anywhere, but does your data and my name is Jared watts maintainer on our project called Brook and I'm? Also a founding engineer and I start up called up bound.

A

So you know, as you've already got from the title of the talk. Kubernetes runs everywhere and it's pretty amazing. Actually what got accomplished to make that happen. So you know in terms of container orchestrators. Kubernetes is the minute it's the de facto CEO, it's the lingua franca right. It's really really is supported by everyone. Now it runs in all the cloud providers. You know Google Amazon, Microsoft, Digital, OSHA and all those guys you can run it on bare metal, on-premise, hybrid environments.

A

Kubernetes really runs everywhere, even by you know, a developer laptop, which is what I'm going to be using showing some demos later on here, so even more important than that, though, that kubernetes itself runs anywhere. It also allows all of your apps to run anywhere to you can deploy your apps to any of these environments that we just talked about, and so what that means is that applications can be portable, and so why is portability important though so? A portable solution, as we've kind of talked about, can be run in diverse environments.

A

It can be run anywhere, but what that gives us, though, is the power of choice. So if you have an application, that's portable can run anywhere, then at deployment time, when you want to get that application running somewhere, you're able to take advantage of the best environment for the job. At that point, so you can do things like for costs.

A

You can optimize, go the cheapest route or you can go for you know the best quality of service, maybe some environments or have some particular features that you're depending on and you have to go to certain environments. That way, you know, multi-cloud stuff is getting pretty interesting as well, where you know you can have some resiliency. If you deploy your app across multiple environments, multiple clouds, you know, if one major outage happens, it happens and say Amazon, but you're also deployed in Google.

A

Then you know you have resiliency against there's also interesting things like compliance, you know certain governments have certain requirements for where data can be stored, and so that can influence. You know your choice of where you're going to deploy your application.

A

So you know how did kubernetes do that, though, how did kubernetes enable this incredibly portable application deployment ability? So what it did is it defined a common set of abstractions and primitives for the kind of capture application deployment concepts, so things like pods services deployments, although what those really are doing are capturing in an abstract way, what it means to deploy an application, so a pot is a pot. Is a pod anywhere you're gonna go right.

A

If you want to deploy your application somewhere like in one of the cloud providers that pod you can you have the same expectation that your app is going to run the same way, no matter where it's being deployed, so it brings up this concept of you know, write one white excuse me write once run mini where you write a single, manifest single resource.

A

The yam will file, let's say, and that one n will file unchanged should be able to run any worker, but Nettie's runs, and then that gives us the developers a bit more freedom to focus only on their application and not worry about the the details of the environment that that application is going to get deployed into they've written the manifest once and it will run many times many places.

A

So, in addition to the abstractions for application deployment, concepts, there's also some storage abstractions. That kubernetes is defined because real some real world applications, they're stateful, they need to participate. You know the abstractions. The acrimony this is defined are incredibly useful for helping you know kind of abstract these sources, these concepts of storage- and we have things like persistent volumes and purses volume claims. Storage classes is another one. You know, there's plugins I think the talk just before this one was about CSI.

A

So if you're there for that, you've learned a little bit about the evolution of where volume plugins are going industry-wide. That is, but basically all of these different storage abstractions help support the portability of stateful applications.

A

So let's go into a little bit more detail about what those abstractions are right. So a persistent volume, PV and PVCs storage for applications is normalized on this concept of a volume. So if you have a pod- and it has a storage requirement, it's done in a abstract way through a volume and that volume is independent of the backing storage right aware that a storage is actually coming from, so a pod could have a volume that is maybe being backed by a googol, persistent disk or maybe by a SEF Redis block device.

A

It's the volume is the abstract concept, that's very powerful, because then that enables the application to not have to worry about where the storage is coming from. It says: I need storage. Give me a volume, I, don't care where it comes from. Necessarily that definitely enables portability. So storage classes, that's another abstraction in kubernetes for storage, and you can think of those as blueprints that abstract away the details of how a particular storage request is going to get fulfilled.

A

So these are normally defined by administrators, so administrator, who has setup details of the storage in the cluster? You know they'll kind of fill in the details about how storage of requests will be fulfilled. We know where this so we're just going to come from and there's different sort of parameters that you can get from those as well, such as a we know the provider of this storage. Where is it going to come from? But then you have other things too, like the quality of service, different policies for backup.

A

You know various parameters for this storage, that kind of define where it's going to be fulfilled from, and the really really good thing about, storage classes, in particular, is that those lay the foundation for dynamic volume, provisioning and kubernetes. So I'm not sure you know how the experience in this room is with kubernetes versions going a little bit further back. But if any of you all remember before dynamic provisioning was a thing and you could only do static. Provisioning of volumes.

A

Dynamic provisioning goes very far beyond that, because with static provisioning, only basically, you ended up with an administrator having to manually create some storage, we're from wherever the backing storage is probably somewhere external to the kubernetes cluster right and then they would have to go in kubernetes and manually, create a persistent volume that mirrors or represents that storage that they manually created, which is kind of a pain in the butt right. You have a human getting in the way or having to come into the process of providing storage to your application pods.

A

So with dynamic provisioning. You know it's that takes away that step where on demand, you can have an application or a pod. That's saying: I need storage and then programmatically from running code in the cluster. That storage request will be provisioned fulfilled.

A

You know mounted attached all those sort of things so that no human has to get in the way of an application, getting its storage and then the final abstraction here is volume plugins, which is the the means by which external storage solutions can be integrated in throwing stickers around. Oh that reminds we do have a bunch of stickers up here. If you want any at the end, so you know that's the means by which these external storage solutions can get integrated into kubernetes.

A

The interesting thing, though, is that these volume plugins may enable that integration, but it doesn't solve the deploying that storage solution are managing that storage solution. That's kind of an exercise left up to the admin and you also get into the situations here where the applications are going to end up relying on provider particular cloud provider, specific managed services like Google, persistent disk and Amazon EBS, and things like that. So let's talk a bit real, quick about these abstractions that exist in kubernetes that and you know, have some aspects of a name in abling portability.

A

Let's talk about where they fall short, so one thing is that the focus is really on the consumption side of storage. It's not on the provider side of storage. So it's really dealing with when an application needs storage needs to consume. It lets get that request fulfilled, and it's doing a great job of that. But it doesn't really have a strong story for what's happening on getting that storage to exist. In the first place, say you have a blank kubernetes cluster with nothing in it.

A

You know, where's, the storage gonna come from well, something's got to get set up for that, and there's nothing really that kind of enables that very very easily in kubernetes right now. So these are the external storage solutions that applications are relying on. They need to be accessible, somebody's got to deploy them somebody's got to manage them. That creates a burden.

A

Now also, we talked earlier about how storage and kubernetes is normalized on the concept of a volume, but not all data behaves like a volume. Now there are many data, intensive workloads that you may want to run in kubernetes, and they have needs for a higher level of storage than just a volume things like databases, key value stores, caches message, queues all these different types of storage. The applications rely on there's no real abstraction of those right. Now that defines or describes the need for that type of storage, it's pretty basic Eligius volumes.

A

So what you end up with once again is that a lot of applications end up relying on proprietary cloud provider, managed services. A good example here is, you know: Amazon's relational database service, RDS, and so when you're, taking a specific dependency on a specific storage solution. Instead of having an abstraction, then you're going to end up with vendor lock-in right.

A

So a portable storage solution. That's what this talk is supposed to be about right. So, let's start, let's start talking about what that might look like, so we need something that is going to help us make stateful apps be more portable and be is just as environment agnostic as kubernetes itself, which has done a fantastic job of that. So a couple ideas: what about storage that itself runs inside at the curb radius cluster?

A

So if you're deploying a storage solution inside kubernetes, you can start taking advantage of some of those built-in powerful abstractions that it has like pods, and you know replica sets. Daemon sets things like that and another idea. You know what about expanding, making it a set of Sorge objects, abstractions broader, they can Center to kind of start describing these data needs that applications have that don't just fit into a volume.

A

So this is kind of a hypothetical here of what a stateful application may look like if it was truly portable. So we're gonna take the example of wordpress, so WordPress is stateful in them in the sense that it needs a bad database to store its data. So you know all the cloud providers they have their version of a managed database service, as we just talked about amazon has RDS and Google has cloud sequel, so there's solutions out there.

A

You know so WordPress describing the application of WordPress in the top right here and then describing its need for data and the bottom right that could be done in one single llamó file, and so with that one yellow file we're basically describing that our application and how it needs to be deployed for, while also describing its need for data and how that might be deployed as well, in an environment agnostic portable way. So it's in a situation like this.

A

What would happen is that, whatever environment you're running in the need for a database, the general need for a database will be fulfilled by whatever environment. That is, if it's the cloud sequel or if it's RDS that'll happen, WordPress gets its database or Bres is portable. Now, ok, so now we're gonna get into how one could build this. So, let's go for a couple concepts quickly that I've heard people talking about this week, so they're probably not brand new concepts to a lot of people, but let's go over them in many ways.

A

So when we're talking about extending kubernetes, there's a couple different ways to do that, one of them, that is a very powerful concept, is custom resource definitions, and so what those enable you to do is to basically, you know, teach kubernetes about a new first class object type. So you know we have things like pods and config maps and things like that, but see RDS allow us to define our own arbitrary types that look just like a pot or you know just like a daemon set, and you know just like any other.

A

First class API object now, since their API objects, they get persisted in at CD also. So what this does is this these CR DS they allow a native management expiry. It's through cube control, I've been calling it cube cuddle for the longest time and to speaker lounge, I overheard. Somebody saying that only noobs call it that and cube control is the right way to call it so I'm trying to correct myself now. I will assimilate.

A

So you know when, when you get to be able to start managing, you know arbitrary types or user-defined types in your kubernetes cluster with the built-in kubernetes tools it starts. Looking, like you know, a first-class citizen of kubernetes. You know that whole very consistent, cohesive experience is a powerful thing, so these CR DS they don't have any functionality by themselves, they're. Basically a type declaration of you know.

A

This is a type and these are the type of the properties that belong on it, but it doesn't have logic or features inside of it in itself, but with those types we can define these abstractions we've been talking about here. So this is what a CRT declaration looks like. You know in yam all you're, basically saying that this is a customer research definition, you give it your API group information, you know what group it belongs to the version of it and that what we're looking at here is a kind of a data base.

A

This is what it would look like to define a sort of an abstract portable concept of a data base. Now the other way that we extend kubernetes is through a concept called operators, core OS I think it was yesterday morning talked about their new operator, SDK and they're.

A

Actually it's at least to my knowledge, the ones that first came up with this operator pattern or started evangelizing it, and it's really really really powerful we're you know an operator is a special type of controller, a kubernetes controller that were all familiar with like a node controller, pod controller things like that, but what it does is that it codifies the domain, expertise or subject matter expertise that it takes to run a complicated application, especially over time to you know not just the initial deployments of it, but also the management of it too.

A

So, basically, you know any actions that a human would need to do to manage a cluster. You can automate that with operators so that you know the software will do this. For you and one of the examples that really made me personally see the power of operators was the Prometheus operator, because I wanted I had a need, for you know, monitoring in my cluster and I. Don't personally know how to set up for me theose, but with a cube control, create the operator and a few control create Prometheus without really no asthma, myself.

A

Having no idea of how the software is, gonna, get managed and deployed underneath the covers the permeate. This operator did it for me, so it allows users to start using or consuming software systems that they don't need. You know, without the operational experience to manage it, so let's offer do the job right, so the real way that operators work is through a control loop, just like any other controller.

A

So what they do is they sit in the loop and they'll watch for events on the CR DS that you've defined like a create a database or update a database, and they also watch for changes to the cluster itself. So the core part here is that they'll reconcile the users desired state with what the actual state in the cluster is.

A

So, for instance, you know you have a actual state is no database, but the users desired state is yes, please database it'll reconciled it to and make that you know a reality and it does that through a kind of a three-step process. One is observe. You know, first discover what the actual state of the cluster is and then analyze determine the differences between the actual state of the cluster and what the users desired.

A

State is and then act on that you know perform operations or do we know whatever calls you need to do to make sure that the desired state is made a reality so continuously in this loop, the operator will be driving towards that desired. State.

A

And as we kind of alluded to earlier, you know once the software system gets deployed, it's got to be managed. Think about day. Two operations you've deployed it, but on day two you've got to support it right. So there's a whole bunch of things that operators can do that. We can teach them to do and automate with software like ensuring they're, healthy scaling it.

A

You know making sure that if there's a failover or sorry failure, you can, you know, fail components over he'll, the the system backups restorations, one of them, that's particularly interesting to me- is upgrades or across versions of a piece of software. You may have breaking changes or you may need to do some sort of migration right and the operator could do that for you as opposed to having to have a human go in there, and you know, do things manually, let's make software, do it.

A

So just let the operator manage it, it's very powerful to have automation, doing things for you now a key aspect when we're getting down into the code of what an operator would look like. Is that to be able to talk to the kerbin ADIS api? You can't do much in a kubernetes cluster without being able to talk to the api server and you know making requests performing operations getting information.

A

So the way that's done in an operators often is with the client set, and so a client set is is a collection of various clients that can talk to the crew brandies api, to get work done, and so from your controller, your operator and your go code. You don't have a client set there that you can start making these calls with. So you can see some examples at the bottom here where we have a client set and then we're asking for you know the core: the core API v1 of the core API.

A

You know, I want to do something with pods, so I call the pods function and the default namespace and I want to list these pods. So programmatically with code U and a client set here, you're getting all the pods in the system and that's part of that observing step of operators of I need to know what the system is doing before. I can analyze it and start making changes. You know. Similarly, we can do things like in the storage, the v1 of the storage API. We can create storage classes.

A

So all this we can do programmatically from our operators. So it's not only important, for you know discovering system State. It's also important, for you know, performing operations to drive towards that desired State in the cluster.

A

Another key aspect of of operators here is Informer's, so an informed. So we once we have the need in our operators to be able to tell when an event happens. So the way we can do that one way, a naive way would just be to keep asking the API server hey.

A

Is there an object, I care about ask again ask again, but if you're polling the API server like that, if everyone's doing that in the cluster, especially considering how many components can run in a kubernetes cluster at once, that gets expensive really quickly, it's incredibly wasteful, so an informer is a way to efficiently watch for events from the API server and get a notification about it.

A

Informers off also have a very nice property of caching objects locally, so that, if you have a client, that's asking for hey give me the pods give me the pods give me the pods. It won't continue to hit the API server with that request, it'll fulfill that request from its local in-memory cache of objects to be even more performance and there's also a concept of shared performers. Where, if you have multiple controllers all running in the same process, they can share information in an efficient way.

A

So it's gonna move through this pretty quickly. So you know in your operator, you know to start implementing these systems here to be creating an informer. You need a source of where the information is going to be, or you know, that's the kubernetes api server. You say hey.

A

I want to I'm going to learn about pods every 30 seconds, so I want you to refresh that cache for me, so it doesn't go too stale, but here's the important part to me when you're asking for events on objects and the kubernetes api server you're, also giving it these function.

A

Handlers these function, pointers to handle errors to say, okay, here's a function, I want you to call when some a pod has been added, here's one for update and here's one for delete, so basically we're defining behavior that we want to have happen when events happen in the system that we care about, and the last part here is about event triggers. So you want your when you're watching for events you want it to be a level driven, not edge driven.

A

So what that means is that you can't always assume that you're going to see when a transition happens, you have to make decisions on level driven, not edge, driven for when you see a state in the system, you may have missed when that transition happened, but perform the reconciliation. Do the operation anyways.

A

When you see the state of the system in a state that you care about all right so back to the beginning here finally running your data anywhere, so we've kind of talked through a bunch of concepts here, and this is pulling it all together. So by extending kubernetes we can start defining these new, useful, portable abstractions. So a you know, users and apps.

A

They can describe what they're, not only how do they want to be deployed, but also what their storage needs are as well and as soon as we've described, both the storage needs and the deployment needs of an application. What we're getting there is true portability of I can now take this application, even though it's stateful and has a requirement for storage, it can go anywhere now and then at the final part, there is how operators can make that desire for data. It just happened because of their. You know automated domain knowledge of how to operate.

A

Excuse me how to run a cluster, and it does that in two ways, with either deploying storage solutions into your kubernetes cluster or having the smarts to talk to whatever cloud provider. You're running in and getting the manage services up and running and provisioned all right, so now I'm going to show a few things here, so we were in this talk here. We were talking about.

A

You know how we can have portable storage solutions or how we can bring storage into pure vanity, the cluster itself, so it can run a bare metal, hybrid solutions, cloud providers, all that sort of stuff, so we're going to show a few operators from the Rooke project that I mentioned I'm a maintainer on and we're gonna show how a lot of different storage can come right on into our kubernetes cluster, and it's all my laptop right now, but this could be bare metal.

A

This could be cloud provider doesn't matter so I'm gonna use some aliases when I'm typing and I apologize for that, but it makes for less mistakes, I hope so right now, SEF is a distributed. Storage system that Brooks supports and I'm creating an operator for it.

A

I'm also going to do the same thing for cockroach, DB and I'm, also going to do the same thing for Mineo, and so what we're going to see here is hope. That's big enough, but there's a lot of information to kind of fit on the screen here, so we'll make it make it work, but we're showing here is did I three different types of operators out to this kubernetes cluster, one from Mineo, cockroach DB and one force F, and so these operators are sitting there.

A

They've defined their own custom resources that capture the concepts of what asset cluster is and how its deployed what a cockroach DB database is how its deployed all those sort of things are captured by these operators and they're waiting there for us to do something useful. So now that we have created these operate, these three different operators, let's go ahead and start making some storage too.

A

So I'm asking the Ceph operator now to create a cluster, a SEF cluster. Specifically. So when I created that using cube control, it created an API, an instance of our customer resource definition, SEF cluster, in with the kubernetes api server, which triggered an event to our SEF operator, which says oh cool, somebody's added a set cluster, so I'm gonna go ahead and I know the specifics. I've got the domain expertise of what it means to be a safe cluster with distributed storage, so I'm gonna make that happen.

A

So we see here the multiple pods start coming up and you know start the operators going through the steps that it takes to build a scepter for us directly in kubernetes oops. We are also going to do the same thing or similar thing for cockroach DB and we're gonna do the same thing for Mineo and we'll go back over to this view here so very similar to SEF. We asked for an instance of a cockroach DB database.

A

We asked for an instance of a Mineo objects server and those operators knew what we're asking for because they got an event on it. To add so add that to the cluster and then they're using client sets and the kubernetes api to build these storage solutions for us directly in our cluster.

A

So it looks like most all that's running now, so we have SEF, we've got cockroach, we've got mini. Oh let's use this stuff. Now that we've actually got a storage solution deployed in a portable way and our kubernetes cluster, let's, let's use it I'm gonna start do some copy thing here. Sorry, this is not meant to be readable.

A

What do I want to do.

A

So I've defined a storage class for for the SEF provisioner, so that when an application requests storage, then that can be fulfilled because the storage class is the blueprint right fir that defines how a storage request can be fulfilled and then I've also gone ahead and deployed to other as a stateful application.

A

I've deployed WordPress and I've deployed a data base underneath WordPress to serve as this persistent storage that WordPress sorry, the my sequel database, that's going to sort of handle, the storage for WordPress is actually going to be served underneath by that SEF distributed storage system that we deployed out to the cluster. So it looks like those are running now and let's do something similar for cockroach, DB and Mineo. Also.

A

Oh yeah, this guy.

A

Yes, so let's do something useful for cockroach.

A

So I've got what I've done here now is I've gone ahead and created a kind of a load generation app for cockroach. It's gonna, do a lot of random, writes and kind of you know benchmark and things like that.

A

So we're kind of looking at it's logs here of all the rights that it's doing to the database to the cockroach DB database. So we know that the the cockroach DB instance that we deployed is up and running and functional, and you can do rights and things on it. We can also do a select to look at the dead, random data that it's writing to it also, but let's also take a look at Mineo.

A

So this is just asking for you know what URL you know. Ip address. Did the Mineo object server end up on its yeah? There we go and, and so now we're seeing the the Mineo object server. You know log in page up and running.

A

And now we have, you know an s3 compatible object, server up and running in our kubernetes cluster. So let's just create a bucket and let's add some files to it from the desktop I. Don't want those PDFs.

A

And so you know in our cluster we have object, storage, that's up and running in a portable way, and we can start using it, for you know, storing files in our object, server, we've in addition to that we've got. You know a database instance up and running that we could consume your cockroach DB. We also have a distributed storage cluster of Ceph running that also services, block storage, file, storage, object, storage. So basically, what we have here is, you know, in a portable way.

A

We have a set of operators and a set of you know: custom resources that have extended kubernetes to be able to bring storage that your applications require into kubernetes in a portable way anywhere. You deploy cloud providers, on-premise laptop whatever it's. Let's go back to the sides, real quick.

A

Alright- and that is all so thank you very much for attending this- talk to dead any questions.

A

It's a question: oh yeah I have the guy one here, I, don't know if it's not yeah.

B

There you go, could you maybe show us real, quick how you do the configuration for where the data stored, especially in safe, you have the possibility to have an SSD cache on the very top of it. So what's, how does look, how do I configure in work that should be saved on SSD and how do I configure the flash times and things like that? Yeah.

A

Sure so you know you were clearly familiar with Seth you've used it before. Obviously you know and as you're mentioning there you know, Seth has this ability to kind of define the topology of your storage cluster. Of of you know where data lives, you know classes of it. To put you know, certain information on you know faster in me and me SSD devices things like that right. So.

A

In the example demo, we had here just running on my laptop, you know: I, don't have a set of SSDs and in VMI's and stuff to use like that. So I'll make this bigger, don't worry and that's try to bring it up as quick as possible.

A

A

So in this is this is what is this safe cluster CRD is right here defined, and so this is where you get some of those configuration options to start saying how you want the operator to deploy the store, the Ceph storage for you. So this particular field here, like metadata advice. This is where you would say hey. This is my SSD that I want you to use and then you can start defining. You know other aspects of the cluster like you know what particular nodes you want it running on what devices those are.

A

You know the spinning platter devices where you want data to end up, if you want it to use file, store a boost, or you know a bunch of various constructs of how you want to deploy the storage. Now, since these are some, some of these are abstractions over what Seth provides.

A

There's gonna be some simplicity or some things that staff may do that you know isn't necessarily directly supported by rook, but you do have the ability availability to specify how the storage resources in your cluster, you know what hard drives and devices you have in your cluster should be used by SEF, and now it's important to note too that with rook, since you know, there's more support than just force F.

A

This can be done in a common way now, because all storage providers need to know what storage they can use in the cluster to make something useful right, like even mini oh cockroach DB. So in a consistent way, you know using rook and these operators, you can say: hey I, want you to use these devices for what the specific storage solution on top of it is be that cockroach or next Center staff or whatever it may be.

A

A

Yeah, so the question was you know? What do you do if you have an existing SEF cluster and you want to use it in different SEF clusters? So there isn't there isn't a specific. You know migration path or you know full support with rook to bring it into kubernetes. If you already have it existing.

A

There are things you can do, such as you know, set up replication with the SEF cluster that you've created in kubernetes and start replicating the data over or snapshots and move them over things like that, but there's not a specific way that that's supported. So it's you know mostly for starting with new SEF clusters.

A

Not sure if I have found that, if oh.

A

It like in the demo I, was doing here Oh in your hyper.

A

Yeah I believe like a decade since our you know, Federation between the Cooper, dandies Custer's and that's kind of a hot topic right now. Actually, with you know this whole multi-cloud story too, and so there's actually some some thing. You know thinking right now, that's being done on that to how to make that management story of multi clusters and multiple clouds look more like managing a single cluster, and so you know they don't have a story for their eggs.

A

You know a solution for that right now, that's actually something that we're thinking up and working on right now to support that better. That's a good question! You see it's time, check I! Think! That's all the time we have here if it so. Thank you very much. Everyone appreciate it.