Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2018 (Copenhagen), 14 Nov 2015

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Storage WG Deep Dive – Ben Hindman, Mesosphere, Clint Kitson, VMware & Quinton Hoole, Huawei

Description

Want to view more sessions and keep the conversations going? Join us for KubeCon + CloudNativeCon North America in Seattle, December 11 - 13, 2018 (http://bit.ly/KCCNCNA18) or in Shanghai, November 14-15 (http://bit.ly/kccncchina18).

Storage WG Deep Dive – Ben Hindman, Mesosphere, Clint Kitson, VMware & Quinton Hoole, Huawei (Intermediate Skill Level)

The storage working group has some guidance from the ToC. This session is of roundtable format where we will be able to discuss the structure and goals to help support the ToC.

About Quinton
Quinton is currently Technical Vice President of Cloud Computing at Huawei. Previously he spend five years at Google, where he was an Engineering Lead on the Kubernetes team, and Technical Lead and Manager of Ads Serving SRE. He was also the founding engineer of the Amazon EC2 cloud computing project (2005-2009) where he lead much of the early design, implementation and operation. As a lead engineer at Nimbula (a startup that was sold to Oracle in 2012), lead the design and development of their Cloud IAAS product (both private and public cloud). His experience prior to EC2 includes telecommunications (call center switches, CTI systems, mobile billing systems), financial (data exchanges, point-of-sale systems), travel (internet booking systems, airline and hotel reservation systems) and mobile (aviation GPS navigation, banking etc). Quinton is also part of the 15 member kubernetes steering committee.

About Clint
He is the Cloud Native Engineering Director at VMware.

About Ben
Ben Hindman is one of the creators of Apache Mesos, a platform for building and running resource-efficient distributed systems at scale. Ben started working on Mesos as a PhD student at Berkeley before he brought it to Twitter where it runs on thousands of machines. An academic at heart, his research in programming languages and distributed systems has been published in leading academic conferences.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.

A

All right welcome everybody to the advance session for the storage working group in terms of an an intro I just wanted to make sure that we're setting the table for what you guys should expect for this session.

A

So this is a quote: unquote round table it's not exactly round in the room, so the chairs are set up for presentation, but the intent of this session was, for you know, existing and new storage working group members, so that we could talk about the Charter and the mandate, and you know, discuss like what we're gonna be working on over the next few months. So we're not going to go through a bunch of slides and do a deep dive. It's just gonna be a conversation among the SWG members to talk about.

A

What's going on, so that's kind of the expectation. So if you guys are interested in you want to participate, please hang around. We're glad to have everybody here is that can I can I also get a kind of a show of hands? Just so we understand you know who, in the room, has actually attended an swg meeting before the storage working group, so who's new.

A

A

What's that, oh sorry, yeah so who's new, a lot okay! So it's about a little bit more than half as new okay.

A

So the first thing that we wanted to cover is the the ask of the CNC F. Like you know, what is storage at the CNC F? If you guys attended the the intro session, which we had already we kind of covered, you know the landscape that we've been discussing. You know, why is storage important?

A

You know what a storage look like you know: how does it work one of the different components that we find in the architectures, but I think a lot of that is captured. You know from the intention of the CNC F for storage in this sentence right here and then the scenes you have in tends to enable a thriving storage ecosystem that is vendor and platform neutral and interoperable for applications right, so that I think that last piece is really the key.

A

When it comes to the cloud native environments at all times, when we think about you know, what's what's happening or when we think about different components. Working together, it's all about the user experience, it's all about making sure that these applications can consume storage services really transparently and an abstract of what cloud they're in whether they're on Prem off Prem storage platform. It really shouldn't matter so that's kind of the goal with a lot of the work that we're doing is just enable these.

A

You know neutral environments that enable a cloud native experience for for storage and applications.

A

These are the three priorities that we really wanted to talk about today and describe what they mean.

A

We've actually got toc members here today to help us with these and to clarify and to dig in deeper we've got Alexis in the back of the room, we'll be running the mic back there when we have questions for him and then we've got Ben, of course, up in the front and I think that is Quintin in the room.

A

Now, if we don't have Quintin right now, but he may join us in a few minutes as well.

A

So the the first priority that we're talking about here is a group is, is a stop one, so clarify the terminology and landscape.

B

Let's do one more poll how many folks like work in storage today, if you could raise your hand like they work at a storage company, are they okay? How many folks are here just because they're interested in using storage in their clusters or doing stateful stuff? Okay, so.

B

Hopefully, some how you can use that to decide which ones you want to spend the roundtable on I'd.

A

Actually like to probably ask a question of someone in the audience: maybe you can answer the top one for us who here has been actively looking at the storage ecosystem and cloud native and been confused, seeing mixed messages not really knowing where to go. Anybody want to describe their their challenges in the cloud native storage, space.

C

So I mean this is more a general point, I think so. I spent a long time in the open set community, and you know I used to work in storage development at at IBM and and operated very large, safe clusters for a living, and it does strike me that there seems to be less of a pull for standardization and there was in in OpenStack. You know OpenStack cinder. That was it.

C

You know, that's the mechanism by which that's gonna happen, and you know it feels like things are not yet at least that's my impression over the last year that the things haven't quite got to that state. Yeah and.

D

C

May not be true right.

A

I I think that that's a fair comment: there's been two projects that have come into the CN CF foundation or seen CF, that our storage related the tests and rook right and if you look at those projects like rook, is an orchestration tool for storage platforms. So you can run you know, different persistent applications on top of kubernetes, essentially and then you've got the test, which is a sharding technology for my sequel.

A

So you can make my sequel horizontally scale underneath those things you may consider that there's like specifications, which I think is what causes the ecosystem to not splinter anymore, like people focus on the specs and things get better. The one spec which is ongoing right now is CSI right and that's addressing like the block volume support in the ecosystem, and that I mean that's been a great thing for everybody so far, and so I think that you know bringing the ecosystem together and rallying around something. It just takes time like.

A

Sometimes you have to innovate with all these higher-level projects and then that drives down into creating specs I. Don't that.

C

Definitely felt like in the early days of OpenStack that there was a big push from the storage vendor community to to move to that position, because they were all desperate to to provide.

C

You know that support to their customers, who wanted to use OpenStack right and you.

D

Know in perhaps.

C

There are perhaps there are less folks wanting to use sans and stuff these days. You know in this kind of environment, you know.

A

Does anybody else have any any other stories about doing the research and cloud native storage being confused by it? Having a lack of you know, common knowledge, do you think? Do we think that that's a problem like is this? Is this something that is gonna be valuable? If the storage working group tackles an idea of creating a white paper that describes the ecosystem and the components and how things work together, is that gonna help bring people together?

A

Anybody want to comment on that, because that has that worked in other areas. Any examples of that I think the serverless working group did that recently and I think that that's been people have kind of widely accepted. That was a great thing to do, to help educate and from a storage working group perspective. This is one of the things that we've wanted to do for a while.

A

I think that, in order to move forward right, it's the white papers and the specs that allow us to build this, like common knowledge and common way of working together- and you know, I- definitely feel pretty strongly that if we can get this white paper done, that we'll be able to kind of move on to more interesting topics and storage in the future.

B

Why don't we dig into the first one I think? That's probably the easiest one to dig into I mean I think we can just use this as a working session. Since that's what the original plan was.

B

Why don't we dive into the things that we know that we want to make sure we've got clear definitions for we've got a great crowd sourced audience here that can help provide some insight into whether or not these definitions make sense, as well as I think it could make sense to share what we currently think about the landscape and and get some feedback from folks about again whether or not this makes sense, I I think again.

B

If we could just get a raise of hands of the folks that have joined the working group and probably will continue to join the working group in the future. I think it's a pretty small number we have today. We've got like okay, eight or so so. For those folks, this is actually a really great opportunity to get feedback from everybody else in the room and if there's anything else, you guys would love to get feedback on when it comes to what the working groups trying to do in storage, I think we can do that.

A

So we we presented this slide at the intercession and I'll, give you guys the quick elevator pitch on it and we can chat about it. The purpose of this is basically saying that you know.

A

Storage comes in many forms, you've got kind of the bottom most part, which is the north-south activity, where you have the kernel and it's issuing storage requests, etc down to disks and things like that, maybe it's remote storage, maybe it's local storage, but you're, essentially accessing LBA, logical blocks or you're accessing files, and that's what we have the volume functionality to to participate in.

A

That's where the CSI functionality actually works for and then you've got the East and West activity, which is also storage and those may be key value stores and maybe no sequel, databases right. Those are all you know: you're, storing data there as well, but they're, more structured, but when it comes to the the Charter of the group, we're looking at both of these things and the idea is that we want to kind of focus on it from a top-down perspective.

A

We want the user experience to be consistent, so, whether you're using volumes or whether you're using databases east-west right, the users shouldn't really care they just they they run their app and then, under the covers, the app connects to the storage. That's relevant.

E

Though it seems like there's a pretty fundamental difference, though, in the way this access actually happens and the way people are actually consuming it, since all network bakes serve based, services generally are using stem API. Is the application is aware right that it's not it's not making direct POSIX calls it's not doing everything mediated through the kernel, so in some ways like these are just generic network services that happen to store state right, so I'm not sure that it really fits the same pattern.

F

Hi how's it going Jesse I'm, not sure the distinction holds though NFS like it's a standard file protocol, but it goes over. Networking doesn't have to there's user space clients right and s3. It's a pretty standard protocol. It goes over network right, I mean like it or not. S3 is effectively the de-facto object file protocol right, so they just it's not the distinction that I would have made. I didn't even know. Was you talking Jesse, Jesse and I'd love to disagree anyway? That was all that was all I want to say, like I.

F

Think it's more nuanced than just network or not network.

A

So what do I think? Is this a a decent way to to talk about storage and cloud native? You know from a top-down, you know: you've got a user experience you're trying to provide there's a couple, different ways that so we're just being accessed but from a working group right. This is the picture we're trying to tackle any likes-dislikes.

A

G

Thanks, so sorry, this is probably naive, seems I'm, not storage expert, but in terms of networking, when you are working from an application perspective and deploying your containers, you don't actually care at all about the network. Why can't this be done for the girls, towage aspects, so I would not actually have to specify anything in my deployment. Will it related to the storage classes? For example, I mean storage classes are an improvement but they're not actually.

G

When you want to go across different storage, you actually have to change your classes, and that means that you also change the deployment file which makes your deployment not portable. So why can't the storage group actually address it in a way so that you really feel like you have a storage and but you don't actually have to care about how this is specified. Somebody else will actually give you that.

G

Then it just becomes available from the container perspective very similar to how you actually add blocks in in the VM on Amazon, for example, I, don't maybe the stupid thing I'm not really sure what you think about that I.

F

Think in that case, the difference is whether or not you can actually provide a sort of universal API to something right. I, don't think the storage working groups job is or should be, to try to define new api's that they didn't capsulate. Other api is right for maybe to take a slight spin on the slide on the board.

F

There's things that are obviously networked, and there are things that are not obviously networked right. Nfs is not obviously networked. It pretends to be a local filesystem right block devices is, they can be local or they can be networked and they're, not obviously so sequel, databases, they're pretty, obviously networked right, and maybe that distinction sort of holds more. It also holds more with the the API sense like as an application. If you're using a file system, you probably don't care how that file system is implemented.

F

If you are on very rare cases right and if you're using a sequel database, you actually probably care what characteristics of sequel you're going to be using.

E

And I think to your point: Tim like it is the whether or not the way you're accessing it is like this already a standardized agreed-upon API right, so is it like F open? Are you going through making like direct kernel syscalls, or is it a library that you load it in that's then constructing packets or restful api or whatever right.

C

So do you guys see all that stuff on the right hand, side about other data storage types? Do you see that, as in scope for the storage working group, because I mean you know to me looking at that, you know the the kind of distinction there is between this stuff is kind of that. The operating system layer right this stuff isn't the application layer.

C

Your application needs to know about these things that are on the right- and that's kind of you know to me- is the differentiator between those two things, not necessarily whether it's never I mean networked, is kind of you know irrelevant or relevant right I mean some of that stuff. You could connect to it locally on some socket or you know it doesn't necessarily need to be over in network I.

A

Think I mean feedback from me on. The slide is I. Think networking probably doesn't matter because it can be one way or another, and so maybe that's a I think just the idea of east-west activity, north-south activity, maybe clarifying enough and adding kernel and networking in here may just confuse it.

A

Does anybody want to respond to his question about? You know whether it should be in scope or out of scope.

D

I was gonna, say, I agree with Jessie. The difference is not networking between north-south and east-west I. Think the difference is whether your application actually needs to be aware of what it's talking to or not in the case of the things that are north-south here, that the the operating system takes care of that so that the application doesn't need to, and we since we're not in the data path.

D

We can do magic here with CSI, whereas everything over here doesn't necessarily have a standard data API, which makes it much more difficult for us to build a standardized system on top of. So it does lead to the natural question whether this should or should not be part of the the purview of this group. I.

H

F

I I agree with it, except I, don't think it's static, I think the definition shifts over time right, I would say five years ago. This would have been a fairly accurate representation and today, even though I work at a place that doesn't make s3 I would say, s3 is the de facto standard and like we just have to suck that up, and so maybe that one starts to shift to the other side and like no.

F

We don't really have a an effective fuse sort of virtualization over objects, but you could sort of imagine that and it and it seems like the reasonable and it shifts from the obviously network category, to the not obviously network category or maybe abstracted and not abstracted categories right.

I

And I think from generalisation point of view it it does make sense to address documents as networking, because if you look from I want to access persistent data, it should be an abstract layer where I can access objects underneath whether it's an s3 bucket or it's a file should shouldn't be relevant from standard point of view in the storage plain sense, if that makes it makes sense,.

J

I'd argue that one of the coolest things about kubernetes was that adding a volume, abstraction, persistent volume and adding dynamic volume provisioning around it, enabled the deployment of kubernetes and opening a number of scenarios across many hosted solutions of kubernetes and deployments.

J

Imagine if we had to Tim's point a persistent bucket that spoke as three or maybe even a persistent database, and that gets a little more complicated but- and we even had you know, dynamic bucket provisioning and dynamic database. Provisioning right I'd argue that the same value, we're extracting out of volumes today in kubernetes would apply to bucket and to databases and other forms of storage in the same way, regardless of how whether it's you're going over a network or yurts.

J

Those are implementation details at some level, but you know enabling a somebody who's writing an app to say, I'm, not only declared it deployments and replica sets and other primitives, but also to be able to say that I've got a database or I. Have a volume or I have other constructs that are stateful is is quite helpful from a user experience. Standpoint.

H

So as a follow-up comma, to what you just said, I think what reason persistent women's work fun is because kubernetes is primarily control playing concept for data plane.

H

Politics is basically the API that applications used to deal with Falls or blocks, whereas with documents log streams, time series, if you have a persistent bucket or a persistent database that can help in provisioning, maybe but not with data plane operations, which are quite different across different time. Series, different log streams, different data bases and that's where the problem comes.

K

My name is Wei I'm from VMware and I. Think I came here really just to listen actually but I'm. Really, it's kind of my opening for me to attend a working workgroup and so I'm very grateful. I should came to this session and so I think when I came here, I really think we only talk about the CSI.

K

So to me on the right hand, side, it really seems to me the community connecting to the services service provider, but I, guess that's what data service it is meant for so I think to me, I'm very new to this group. So I think we should be focusing all on the left-hand side. I think that's the definition. We should target it for that's my opinion. I.

L

Kind of disagree, they're, all storage, they we should consider them, but in the same time you have like I also agree because I think that's the big split right. You have a general purpose, regardless of what that is. You have your general purpose and then you have your specialized types where you have your specialized and you can only store documents in it. You cannot just draw a random, binary and I. Think that's the big distinction between the two types that we need to make.

M

More here, in fact, it's another question that I have I wonder.

L

M

It's possible to virtualize the storage in a way that it would be possible to define a single LAN that would be managed by the by the container ecosystem to provide smaller chains to individual containers. A little bit like I providers are doing.

A

Anybody have any follow up on that sure.

N

Yes, all the in my mind, at least and I'm, a storage idiot in the sense that I can't do much anything else, but all the things on the right side what they have in common is they need the piece on the left side. So that's to me is the common denominator, and whenever you have a database or whatever database, it is eventually it will need blocks or a file system right and so talking about documents, log streams and whatever to me.

N

That's there's not relevant for the storage aspect itself, so I'm not saying that's necessarily a bad idea, but I think it conflicts to subject a lot.

H

This is just a comment, so a lot of technologies for document stores, log stream, time series they've been around before containers and kubernetes and the way they're consumed, wouldn't really change. If you're deploying them inside containers and pause. You know it's the same. I can't see containers or technology. The coffee ideas that way work works can help it deploying these technologies, but in terms of consumption, are not see really much difference between how they're used outside of the cloud native landscape versus how they're, using inside the cloud any landscape.

H

Maybe can't some I can comment on that.

F

I'm glad I get a third try, I I'm concerned with it, so so to your question.

F

Should this be CSI that, like this group, this group is specifically the CNC F storage working group, not just kubernetes, sig storage right so community? If this were a community, sig storage, meeting, we'd, probably all be talking about how awesome CSI is going to be, and it will be right side.

F

But this this group has a broader Charter, well, I mean presumably I, think we're sort of still defining the Charter really but I see the Charter as forward-looking. I see the Charter as defining what we think the best practices are. The best technologies are for cloud native operation, I mean that is really. The mission of the cloud native compute foundation is to to help people figure out what their best techniques and the best technologies are and I think another way to distinguish your north, south and east west would be legacy.

F

Cloud native I think that for me, if I can wax philosophical for just a second cloud, nativeness is about decoupling. It's about having less tie-ins to less things. You mount a file system. You're tied to the OS you're tied to the file system, you're tied to the block device, you go a block device, but at least you get rid of the file system right, you go to an object store. Well, we now you're not even tied to a physical machine.

F

Like you, don't have hardware, you don't have you can run from any machine in the network and the more decoupled you get. The more cloud native. You are right, the more freedom you have to move, the more expandability and scalability you can capture in your application. To me, that's the heart of cloud native is it's also about elasticity and the ability to get more of something when you need it. Just like that right.

C

F

But not, but not to you right, it's like service! Well, there's no such thing as serverless. It's serverless to you! There's a server somewhere right, but because of SLA Xand guarantees that you make for these API driven services. You get to pretend that that doesn't exist right and that's the power of cloud native and that's the thing that CN CF is trying to capture and so I see the the purpose of a white paper here, not necessarily to standardize we're not going to sit here and define a new API that encompasses all all true sequels.

F

We're not empowered are endowed with that bed authority right. What we can do is characterize categorize, texana Mize, the various storage technologies that we have and make recommendations for people about how these things fit with a cloud native ethos and when they're appropriate when they're not appropriate and how they're likely to be implemented and what the likely failure modes are and how those things fit into different categories and I. Think that's where the information comes in, where we get to take people who are not here at cube con right.

F

This is sort of the elite of the elite. We go to the rest of the world and we say this is what we think is the best way to consume cloud native stuff, right, I think we're already getting there with compute, because we've got things like containers and kubernetes. The next obvious thing is data and I. Think that that's the Charter here, thanks for my random just.

O

To follow up I guess what you could say is you should have usage patterns. It's for workloads using data is to your point. So if you can tax on demise that that would be helpful for the larger audience.

C

About the consumption side of it, or is it about the operations and management side of those kind of things right so I mean if I go back a little bit and look at something like Seth, for example, from an architectural perspective, is you know, on a certain paradigm, is a fantastic way of managing storage right, because you end up with all these things that are pretty much exactly the saying you scale them just by adding more of them, you manage it all in exactly the same way, which was a completely different thing from how we did storage before those kinds of distributed.

C

Storage systems came in so I guess. What my question is: do you see that side of how do you operate and manage those kind of storage systems as being as important as how do you consume that storage I.

F

Think it's a really great distinction and I personally think we should be focusing on how people consume right. Now, it's great that we can take systems like rook likes F. Like all these. You know, storage OS, like port works, like all these other things that are going to turn the management of storage into a cloud native problem also right, but at the end of the day, there's a turtle at the bottom.

F

The turtle at the bottom is not cloud native right, even inside Google, at the bottom of the stack there's a bunch of physical machines that are in physical racks that have real wires plugged into them and there's a bunch of poor people who run around with carts all day and fix them right and that's not cloud native, and that's fine like that's it. We accept reality right.

F

They're, not poor, but but but somebody somewhere is, is carrying the load so that the rest of us can build more elegant systems on top and for now a lot of storage is. Is that right? So you take a system like SEF, which turns non cloud native storage into cloud native storage. Right awesome and you know you don't get one-to-one. You buy ten terabytes of physical disk. You don't get 10 terabytes of Ceph right I, don't know what the ratio exactly is with Ceph, but you pay something for the privilege of cloud native izing it.

F

You heard that word here. First.

F

And so that's my feeling, I really think what we owe to our users is to talk about how to consume storage in the safest, most scalable, most elastic least coupled ways.

B

Ok, we'll go to behind me, but then I do think. This is a good opportunity to hear folks about the value of this discussion versus I. Think there was another comment back there around. You know this all rolls down to. At the end of the day, it's just file systems and disks and and the value of that, coupled with the operation side of those pieces.

P

Ok, if you look, let's say the normal data files assume you have to think about. How do you make let's say the file system layout, this I knows and all the stuff. If you look at object, you don't have this anymore. You can't talk. You can scale the billions of things you have systems which have this eventual consistency. So you have your HTTP point Vigo for your as free traffic and yes systems with strong console Stanzi. So you can we get there from multiple places. You have note buttons in front of it.

P

If metadata metadata is consistent with the objects, so you don't need to build some, let's say no SQL database and probably if the file changes, the link from the no SQL to safe is not got it any more. You have problems with that. So we have seen customers. If you look, let's say for Oracle: yes, let's Airmen can go to s3, so a lot of applications have to as free as a target there and as its IP based I think it's reclad native. In this way you can deal with that.

P

You have some people also building search things in so, if you want to transfer something from this bucket, you can decide only give me those things with this attributes and then it over it's a totally different approach and I feel. This is something which makes value.

C

Hard requirements, for you know I, ups and stuff, like that, you know, and and people still want to do that stuff in containers. That's why people want to add volumes to containers right. If nobody cared about that stuff, we wouldn't even have to worry about volumes, because everybody could just use object, storage, and you know that would be fine.

F

If our, if our mission is to cloud native as everybody, then we start somewhere, we started with compute and I'd say we're winning that war storage. If that's the next topic, it's a it's a transition right. We we can't snap our fingers and have overnight everybody rewrite their applications to use object store if we could they'd all be better for it probably. But we can't do that right and I think there. The real value is in the the decoupling and the the less stuff.

F

You have to worry about to make things happen that you have to worry about not saying nobody has to worry about right, but that you have to worry about or that you're willing to put money up for for the privilege to not worry about, and if I drew my own spectrum of cloud nativeness right, I'd, probably put files pretty far on the left of not cloud native and I'd, put object pretty far on the right of pretty cloud native. That doesn't mean that there aren't billions of applications out there, that you read files I.

E

Think whether questions, though especially when we start talking about a lot of these network services, are there common elements? What are those common elements? Is a taxonomy or is literally every single solution, different enough that there's very little common elements here. There's we can't come up with a standard API for these things, because they're just different enough and everybody uses them differently enough that you can't swap them like sequel, for instance, right like it, takes a pretty trivial implementation to be able to swap between my sequel, Postgres and Oracle, and have it actually work. I.

D

Think it's also worth going back to what is the purpose of this group. We can sit here and philosophize about storage Hall today, but ultimately we need to make sure that we understand what the deliverables for this group are and what the purpose behind those deliverables are. It sounds like quinton. You might be able to explain this more. What what the TFC wants out of this group. It sounds like it might be: taxonomy a landscape and then the TOC uses that to inform decisions for the TOC I.

Q

Think, as Tim mentioned earlier, we do ultimately want to get to some kind of recommendation saying you know this. Is this stuff works well in cloud native environments and this stuff or my personal opinion, is that there are actually many use cases and you it's it's very difficult to say this is better way of doing storage, and this is a worse way. It's very application dependent.

Q

So I would like to just understand the strengths and weaknesses of different approaches in for different applications, but I think before we even get there. We just need to come up with a common set of terminology. We need to understand how people are doing things today. I mean people are doing stuff in cloud native land today, with NetApp filers, EMC boxes, local discs, Ceph. You know that there are many different ways of doing this and I think just getting an understanding of what that landscape looks like and from the people who are actually doing it.

Q

What were the? Why did you do it that way and not the other way, because it was better for our application and what problems you run into I. Think that information is very useful as a sort of baseline to have the subsequent discussions about in general. What categories exist? What are they good for? What are they bad for, etc? I.

C

Was just gonna say, do we do we have any any numbers on those kind of questions about you know, what's the penetration into the community of things like now, filers or EMC arrays or whatever, from from any of the user surveys, I.

A

Think the best thing that I've seen so far, the survey is this. The CNC F has done so. They've got surveys about. They use the different types of usage of storage platforms, something specific to kubernetes right now, but I think Mesa sphere probably has some of this information as well, but there's some information out there. Okay, we gotta close it out like that, was actually really fantastic. So thank you guys, all for participating.

A

You know the working group talks about the stuff all the time, so you'll find our meetings every other Wednesday we're either hearing from technology within the industry, so projects or products come to present to us about what they're doing and how they're thinking about cloud native or we're gonna be talking about philosophizing about storage. Just like this. So thank you guys very much enjoy the rest of your show and looking forward to seeing you at our our meetings. Thank you.