VMware TGI Kubernetes, 1 Oct 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: TGI Kubernetes 169: Exploring CSI idempotency and leaky resources

Description

Join xing and jay as we explore how CSI providers and Kubernetes controllers deal with idempotency and resource leaking for volumes and snapshots.

- 00:00:01 welcome to TGIK with xing ! episode 169
- 00:01:00 News of the week
- 00:10:00 CSI controller service RPC
- 00:15:00 CreateVolume should be idempotent, so should most other things
- 00:17:00 Create volume names returns an ID
- 00:24:00 Volume UIDs and handles, AWS, Vsphere
- 00:27:00 DeleteVolume succeeds, even if a volume isnt there
- 00:36:00 AWS EBS csi driver idempotency support and parameter mismatches
- 00:51:00 Demo of deleting PVs, deletion timestamps, vSan leaked storage example

A

Hi everybody, it's uh jay, it's another tgik episode today, it's the october, the first and I have my friend zing she's, a csi maintainer, um so she's deep in this stuff and uh the storage stuff, so she's, gonna, she's gonna, show us some cool stuff today. So um we'll give people a minute to roll in make sure the stream is running. Okay and then um yeah cool. It's working all right!

A

So martin's here and let me see here there we go yeah martin is here: martin borgman, hello, it's good to see you from the netherlands. We see you every week, I'm so glad um and I'm I'm I'm having a beer right now, because I don't have to do the show today, zing's doing the show. So I could I could just watch her do stuff. So what's up uh balzas from hungary and we've got folks from finland yucca, I just feel like I'm just becoming friends with everybody.

A

Now I just know all of you all so well- and I don't know if you all came last week, but we had a special guest join from the show we had.

A

We had uh one of our friends over from uh from red hat, join us miss massimiliano um giovagni from italy, cool so um good. We got everybody in here and ricardo's here, okay ricardo's here now we can start so. What's going on, let's see news, let's start out with news of the week.

A

It's been a long week for me, we had a lot going on. I've been doing a bunch of stuff. I've been hanging out with customers. I've been hanging out with my team, different parts of the country, I'm ready for the I'm ready to learn about some random upstream stuff. Going on. Let's see, we've got policy g, rpc, retries, okay, solicity 2.1, and I think they have some network policy stuff that they put in here too. So this is kind of related to one of our recent shows.

A

I don't know whether they implement the network policy api or not. um So.

A

Let's see here, requests retries for http request, container startup ordering work around smaller, so link rd has always been lightweight and easy to use compared to you know. Whatever other service messages, that's always been their claim to fame, and I guess maybe it's even smaller, now um automatic mutual tls. These policies are built directly with I mean I think, that's been around for a while, so um to, conversely, set a default authorization policy, so this seems to be like it's um yeah.

A

This is all there are existing features, but now they have some new features in linker d server and server auth, which allow fine grain policies to be applied so yeah they have like pod level policies. I guess um who else is joining in okay, martin's, drinking whiskey, okay, so everybody's having fun right now, cool balls? Ask: can you bring the message pop-up window?

A

I can't move the message pop-up, unfortunately, or at least I don't know how to um is it in the way I can. I could definitely just not have it around as much what's up ymo from canada and okay. We got a garage from india, nagaraj and choco happy tgik and ameem he's on my team. Ameem is here.

A

Thank you. Amin for coming to support me waleed cnj walid. Where are you from again ameem is from brazil lucas from kansas? Okay, now the americans are showing up all right, so server up and server authorization. So let me turn this off.

A

I know it's annoying when I keep these on all the time, so, okay, uh which together allow fine-grained policies to be applied across arbitrary sets of pods, so server can select across admin ports on all pods in a namespace and server authorization can allow a health check connection from the kublet okay, so they're ad, adding some notion of specific network policy type stuff, that's component specific in linker d. What else is going on kubecon?

A

I don't think it's even worth spending too much time on this, because I don't even know what am I going to say. Everybody knows kubernetes coming up so, let's see, um does anybody have any talks? They're su especially excited about from kubecon. Is anybody giving a talk? Pakistan? What's up aga?

A

Oh okay, well, leed you're from saudi arabia cool ago, you're from uh pakistan cool! So is anybody excited about anything? um I know my friend arun is giving he he made a tool called kate's netlook, I'm excited about that and he's gonna be presenting that over at kubecon k8's net. Look! Where is it here?

A

um K8 net look github, so this is he's got yeah. This is my friend arun. We used to work together and well. We work together again now because he just came to vmware he's going to be presenting this tool. I'll, add this to the show notes if folks want to play with it, it's a way to debug pod networking in real time. So he's given a coupon talk. I don't know seeing anything interesting in the storage space going on at kubecon that you know about.

B

uh So yeah, so I will be uh giving an introduction and type of uh kubernetes storage.

A

Cool so zing's given another cause. So this isn't your only talk this month.

B

uh Well, yeah, I actually have also had some talks in the the colos, the collocated events.

B

A

You all right so you've got tgik and then you've got kubecon, so you got a zing's busy. So thank you, zing for coming to hang out with us. You got a lot going on all right.

A

It's beefy, spire, okay! This is big for governments and stuff like that. People who want to be safe uh carmen said. I don't know what this is, though, so this is oh, this is like a holiday, so there it's like a holiday to to support spiffy inspire or to celebrate it or whatever.

A

um So this is all about, I guess application identity and in the enterprise and all that there's.

B

A

Tgik episode on this that joe went over all this stuff and I think we've been- I was talking to lori the other day about doing another tgik a new one. On the same thing, I think someone's going to do that, um so what else is going on building and sustaining open source communities? This is important I feel like this is important at work and in open source like um so. Let's see all these different things. Let's see all these. These are all the different ones. All the different uh linux conferences.

A

um There's ospo europe envoy con okay. So that's what contour and all those other things are based off of. I think that's what istio and and maybe even linker d is based off of I don't know um so we got a lot of other conferences check those out valero.

A

Okay, so valero is valero, 1.7.0 he's out, okay, so congratulations to the valero team. That's our backup and restore solution for k8s. um It used to be called arc, uh so the latest version is out. I don't know if they have a chain lock yeah, they have the releases. So let's see what happened valero.

A

This is a storage thing, I don't know zing do you know anything about valero? Do you ever use it.

B

Yeah yeah, I uh actually I'm the tech lead for the uh plugin for vsphere.

A

Oh cool, so, okay, any updates there that are worth discussing here.

B

uh Well, so we are going to do a release of 1.3 soon, uh so uh yeah cool completely. For that I'll just be testing it's the most of that.

A

This says 1.7, so what's one three, this.

B

Is okay, so lara has a plug-in framework. Oh.

A

B

So what do we are working on is the plugin that supports the vsphere.

A

Okay cool, so you use this.

B

Here so it basically uh well uh backup the vsphere volumes by taking a snapshot.

A

Oh okay, so you have like so you have different, so you have vsphere, you have aws and you have all these different and then you have. You run the valero plug-in like in a cluster and then that thing talks to the native storage and does all the backup and recovery in here.

B

Lara will be uh basically think of the backing up the kubernetes metadata.

A

B

It calls those plugins to backup the person volumes.

A

And then those get uploaded.

B

To like for the vsphere plugin, it's got getting uploaded to uh s3 bucket.

A

Oh okay, so there's s3, so vsphere uses s3 with valero. So, okay.

B

And then compatible.

A

Cool all right interesting, I didn't know that so vsphere doesn't use its own native storage. It just uses s3, that's interesting.

B

Well, because it has to store this as a different place right, so it's not just uh we'll take a snapshot.

A

So that when your data center goes down, you can.

B

Yeah we want to upload it somewhere. That is durable.

A

All right now, okay, so I guess there's a lot of stuff in here: they they went. They went over to the distro list, so it's safe for now and now we're gonna get into one more and then one more news thing and then we're gonna, zing's gonna take over and I'm just gonna hang out and hang out and like drink a beer and watch her watch. Her teach watch her like just teach me a bunch of csi stuff. I'm excited this is an easy day for me.

A

Kate's node, flag, command line option on falco falco is like a security tool right. What is falco? It's like some kind of a security tool for cloud native security, project defect or kubernetes yeah threat. Detection thing this is from uh who is it systig? Or I don't know it's one of these? I forgot the company behind it.

A

um One of these security companies, um anyways kate's node command line, option which allows filtering by node name when requesting pod nated data. Typically, it should be set okay. So this is. These. Are the new falco.3.0.30.0.

A

Features um all right, something.

A

What's your representative okay, so this is just a performance, improvement and then proposal for plug-in system, so, okay, falco, plug-ins, okay and then they've got a new falco release schedule. I guess falco is 100 open source. I thought it was like a closed source thing, but it's an open source thing. I guess they have a whole community thing and it's not directly associated with the company. So that's cool. I think it's like a continuous monitoring solution um anyway.

A

So the new falco's out, if you're a security person, you probably know more about it than I do. um Let me see check back in the stream and see what's going on what's up vlad.

A

When are you going to do another show um me and vlad are supposed to do a show. One of these days, uh falco rules filters the syscalls from containers krk4. What's up, and carlos is here carlos. What's up, I had fun with you last week.

A

What did you give? You gave a demo recently with falco.

A

Okay, well, maybe you'll have to give me a demo with falco sometime because I don't know how it works. Think ids, yes, cystic.

A

Hey hey carlos sorry, I was late learning about web 3, a great panel of experts. What what's web 3?

A

Well lee's excited about security day. What's web 3 carlos and then we're going to take it away with zing, go ahead and let me know, and then we're going gonna go so zing is gonna, show us the csi stuff um really quickly. uh She's gonna be sort of digging into like some of the api server, kube controller manager, parts and some of the csi driver stuff, and how all this gives us item potency and so on and so forth. There's a lot of idiosyncrasies in terms of how resources are managed and stuff.

A

So I don't know zing, I'm just going to turn your screen share on and you can sort of just get us started. I okay, I'm happy to I'm happy to just uh sit back and watch. This is exciting. So let me add you to the stream here. Let me know when you're ready to share your screen and I'll turn it on.

B

Yeah, I'm ready now.

A

Okay, cool there we go cool all right. You.

B

A

Yes, we do all right, thank you and thank you so much for coming exciting cool.

B

B

uh So I will talk about item potency in csi, um so let's first look at a csi spec. Let's look at how csi spec defines item potency, um so here it talks about item potency requirements. This is under timeouts. So if a call, if a csi call times out the ceo may uh retry uh in that case, um the caesar driver needs to make sure that it will continue where it either left off when it gets retried.

B

uh Actually, let's look at the potency. Let's look at the create volume. Yeah look at this one, so for create volume.

B

um This call has to be adam potent that means that when uh you gotta create a welding call and if, uh if that has the same set of parameters, the same name name, uh the same accessibility requirements, capacity, volume, capabilities and parameters, then it should just create the same volume.

B

So if the volume already exists, then it should just return. This return. Okay with this uh create volume response uh so think about it like when you call create when, when the clipboard is called for the second time, uh if it's already there, then the second time when it's called it's almost like a get volume, call because you're supposed to you get the same volume back so so.

A

Are all of the things in csi? Are they all item potent or is it just most of.

B

Those are uh only the like the list volumes- those we usually don't say they are that important, because you know every time you call that.

A

Okay, one sec one. Second, we have a question from nagaraj. I can't tell whether this is csi related or not. Can you explain, coop ctl get deployment engine x, windows command, prompt is.

B

That um that's not really! I.

A

Don't think it's yeah, so I just asked if it's csi related, but I just wanted to let's make sure and and hi again alex so go ahead and keep going. um You may not want to share this screen because it'll show that recursive, like anyways yeah, so go ahead. Keep going.

B

Sorry yeah most of the call, no no yeah go ahead and ask questions most of the calls right, create volume, delete, uh quick snapshot, delete snapshot uh like controller publish volume or no property volume. All of those are item potent. um So one thing uh I want to mention is right. So if you look at here, as it says, the input is a name, um and maybe I should let you go.

B

Let's see if it's uh go to the next one yeah. So this is a creative volume request right, so in creative volume requests you. uh This is co well provider name uh and at when you, when it sees the driver, give a response. It actually will return a id. So it's here right. So this is the this volume will be in the credit volume response, so this will give a volume id back. So these two are actually not the well, not necessarily the same.

B

um I can actually show you what is passed in here. uh So if we go to here right, this is the external provisioner.

A

Good answer, wales: that was a good answer. I agree with you yeah. It's definitely not a csi thing, but it's hey. Any question is valid. This is a we're happy for any anybody to ask any random question during the next hour, so go for it. Yeah keep good, go ahead and go ahead and see.

B

Okay, so um right, so, if you look at here right, this is uh this: is this? Is the water name prefix? So by default? It is pvc, uh and this is followed by a euid, so this you can actually uh specify if, if your driver does not, cannot take uh a volume name that long, then you can actually truncate it, but by default tdc-uid is the warning name that the css driver will get in the create volume request?

B

uh And then you know after the season driver created a volume, it will return a volume id so uh that may or may not be maybe the same or maybe different from this name right depends on the driver. Most of the time I I see they are different. I actually have not seen a driver use the the same url name and the endpoint id, and then uh once you get that id uh in subsequent volume operations like delete volume, uh so you all give that boring id. That's the unique identifier.

B

Okay, actually see.

A

uh So can I have two volumes with the same name? I can't do that right. You cannot yeah.

B

So actually so.

A

We really do have two two unique ids for a volume. Then right.

B

Oh, you mean a name and a id yeah, so this is the. This is the input right. So when you give, this is a this is the input that the csr driver will get at crevolent time so uh when, when it is done, it will give you a warning id that is actually from the storage system. So I'm going to show you how that looks like right. So.

A

B

um If you look at here right, so this is a pvc, so I have this one created. uh So that's the this is the pvc name and this this pvc. Of course this has a uuid right. uh So uh the input to the create volume uh request call that name will be the pvc dash and the uid of this pvc and then um I think, that's I think. That's this one.

A

Okay, so that's the uuid from.

B

A

To two okay, adc.

B

Whatever, okay abc, I think, that's the, I think, that's the uid, let's see.

B

Right yeah so and then.

A

um Well, waleed has an interesting comment here. He said.

B

A

A volume name issue between open shift and v sphere with the max length is 75.. Have you ever heard of that before.

B

Oh 75, the name or which field, because for the forwarding name uh right now, the default actually should just be 40, because uh maybe you are talking about a different field.

B

Yeah- and I don't- I don't know which field you're talking about, so I know that we actually have some problem uh with the name with the like the length requirements. We actually um increase that.

B

So, if you have a real, you still have a problem. Maybe it's words actually bring it up. So there are a csi communication at community meetings. Monthly. You can actually bring this up because I believe that that problem that should not be problem anymore,.

A

He's saying the data store volume max length is 75., datastore, plus.

B

Volume, oh that's you're talking is, is, are you talking about a vsphere specific thing, not really a csi.

A

I guess it's open shift, specific sorry. Okay, sorry.

B

About that, I guess we're talking about openshift. uh I know that in csi we actually. uh uh Actually you can actually check this out. I think see. That's.

A

B

But I think it's a different one though I I know that in I forgot which release we actually increases the.

A

Oh, it's on the.

B

Integration look at here right. If you look at here, it says uh you look at this one increase from 192 to 256. This is the node id field, so I think there are some other places. We also uh increase relax the the size limitation like that.

A

B

Yeah so, but I don't know that when you are talking about, maybe that's not a csi limitation. That maybe sounds like data store. That's yeah! That's probably something else. Sorry, I'm not sure exactly which one you're talking about. But if it's css specific you're welcome to bring that up to csi community meeting.

A

So I guess we're going to see waleed at the next csi community meeting. That's great! The community, the csi community just grew by one from you coming to hang out with us today thing so all right. So what do we got? What kind of clusters is this a v sphere cluster or what? What is this.

B

This is a yeah. This is a uh right, it's a cluster running on vsphere, um so I have this one. I just want to show you right. The I was educated to show you the volume id part. So this is the the part.

A

Let's check it out.

B

um And uh okay, actually, I think I want to show you this.

B

All right, so, if you look at here right now, the volume handle so this is a this. Is the unique volume id returned by the csr driver, but of course your driver may return something different that I'm just showing you this example, but this is the the warning. Handle field is the one where we're looking we're looking for um so uh so this volume I can show you, uh okay,.

A

So the volume handle is a uuid returned from csi driver.

B

Driver underlying.

A

B

Driver when you create a new vehicle, this is actually the like this, because this is, though this is actually the fcd id.

A

B

Here thing yeah: so uh if we but.

A

If I do this in amazon, I'll get the same type of thing right.

B

A

B

And so, if I look at here right, so I should refresh this one. I just want to show you um there's one volume created here right right, so you see that this is uh our. This is uh the list here ui. It shows this one volume. uh So uh when I create when I uh created this uh pvc right, so this is my pvc demo right.

B

So if I'm making this call um when I'm making this call uh when I actually, when user requested this ppc, I should say that and then, if the external provisioner uh make the create volume call multiple times with the same request, then it should just you know, create this one one item here: one one volume, yeah.

A

So that volume name it's a uuid with the pvc in front of it. If I made this in a different cloud, would the does the csi interface require you to return a uuid, or can it just be any huge string.

B

It can be, it can be anything. It's just suv, a unique identifier.

A

It could be anything okay,.

B

This is actually the input if you look at this one, the the pbc right. So what do you see here? This is actually the input. This is the pvc dash uuid of the pvc itself. This is actually the input to the created volume. Call right.

A

B

Request and the return voting id is actually this field. This is the fcd id.

A

Cool okay, yeah.

B

um Yeah, so if you are.

A

Fcd is a vsphere specific thing: fc.

B

Yeah yeah, that's it uh he's here so waleed has.

A

One other walid has one other question: how can I delete stale volumes from the vcenter console that you are in right.

B

Now, oh, so, actually there's a there's a way. Now I just. I think this is actually a relatively new feature right. So if you look at here, if you select this so so uh of course you of course you watch. Why don't you try to delete that from kubernetes right? You don't want to delete from here, but I actually understand another example. I'm going to show you later. In that case I actually have to go clean up.

B

So let's say, for example, for some reason you have leaked volumes which I'm going to you know look at later, then. In that case uh you want to use this to clean up. So that is, if you don't have any you, don't have a pv uh api object in kubernetes anymore, that is associated with this volume. Then in that case you have to manually clean it up, so you can actually come here and uh you know you can click this and delete it.

A

B

A

And that's actually, what we're going to talk about anyways right, the whole resources and leaky resources.

B

And stuff yeah right right, very.

A

Appropriate question walid.

B

So let's go back to go back.

A

Friedrich friedrich is here: sorry: okay, keep going.

B

What, where did that miss friedrich.

A

Showed up never mind just keep going drink. Where is.

B

A

B

Okay, uh all right so um and then for I, oh it's the dog, okay, it's this one's back! It's here! So, let's see, let's go back to this item. Potency thing, that's the create and then so for delete the same thing right. So this must be item potent.

B

If so, if the, if the volume id does not exist, then uh if you can't find it then actually the see the driver should just return. Okay, it's already deleted because you know it's already: it's it's! You don't you're not supposed to return a not found arrow in this case, so that's actually very important yeah. So that's this one. Let's take a look of a few uh drivers and see what they do.

A

So delete succeeds even if the volume isn't there. Okay,.

B

Right yeah, so that's to support item adam potency.

A

Yep, that's okay, cool.

B

uh Okay, so let's see gc.

A

This whole item, potency thing is really weird like it sounds.

B

A

Great thing, but it's like when you think about it: it's counter-intuitive in certain cases right.

B

But you also don't want to create multiple volumes right.

B

If it you know, if it times out, and you don't want to say uh you trying to create one volume but ended up having 10 and then you don't have a kubernetes api object associated with that, then you know. Oh.

A

B

And that's a problem yeah, and so I just want to take a look of the tcpd driver and see what they do: create volume.

A

No noelle, I didn't even see you when you came here and now you're leaving us. Where are you from and where are you going noel keep going sorry I was just saying hi to noel. I don't. I think.

B

Noel's new here go.

A

B

Okay, um so yeah so the great volume. uh This is the gcp the driver. So first it gets the input parameters like the capacities. It gets. The name brand then based on it based on this name um it. I think it will try to get a volume key so basically, based on the replication.

A

Okay, so we're inside of the google, the gce csi driver, okay, cool.

B

Yeah and and then it gets a volume id here um and then you see here it tries to acquire a lock. So basically, if there is already a credit volume operation going on with the same morning id then you know it's going to return this saying: okay, it's already there, so it's not going to allow two operations going on and then also, if you look here right, it's going to do it's going to validate the disk. It's going to say: okay, let's look at this.

B

If this disk is already is existing and then it checks the input parameters and see if there is any uh incompatibility so just to see if there's any existing disk, but it was created with some different parameters, so that's incompatible yeah. So those are the things it's. It's checking to make sure that uh it's actually that important.

B

So that's uh on there create what inside uh and then let's say, delete volume.

A

Would it be worth comparing the the vsphere and the google one side by side like? Could we put up the code for both of those together and see how they're different.

B

uh It's actually quite different. I can actually.

A

Show you yeah, that's that I'm interested I'm curious so.

B

Like it's actually so uh I need to open another.

A

B

Another window.

A

Yeah waleed, please do add that to the notes.

B

People are different, so actually I'm going to show.

B

So the the main logic for handling item potency is actually in this library here.

A

There's divian is divian still working on windows. Csi driver support for vsphere. I think so.

A

B

A

Things, ricardo.

B

So uh yeah! So if you look at here right so create volume actually go to.

B

So this is the create volume. Here again the vcs is a driver. Oh you want it to show side by side. Well, actually, it's probably doesn't really make much sense, it's very different. So let's actually look at this one.

B

First you'll see it's very different, uh so this actually initially it's implemented using a using a a map, it's in-memory map, so it basically saves a volume id along with the task info and to keep tracks of that and making sure that it's, uh it's not going to you know, go create another task for the same boarding id, but then I think we found some problems because when you, um so this solves the problem when when this is called multiple times by the um by the external provisioner, however, if the csi driver crashes in the middle of the gray volume or if see, the driver cannot communicate with this field for some reason for a long time, then this this one does not work.

B

uh So that's why actually uh right now! This is something actually uh kind of ongoing. So this this is a very new. So that's why it's actually still uh controllable.

A

Create volume with improved item potentially.

B

Yeah this is revenue. This is going to be it's not released here. This is going to be a alpha feature in the coming 2.4 release uh yeah. So this just used a different approach. It's basically what persists that uh know in this data store. So um it's going to check it's going to check whenever it gets requested, you're going to check to see if it's already there's retrieves this from the store and if it's already created successfully and returns this voting id here right.

B

So that's the only idea here and if it's still ongoing, then you know it will just retrieve this task information and then, if it's a new request, it will make this call it's seen as api called to create volume, and after that, it's going to retrieve that information and then and then either it will return failure right so that stat as a status, error or return success and then updated the information in the store. So so it's going to be a little different from the the previous.

B

uh You know the the map approach yeah, so this is still still on ongoing. I think that this is uh merged, but I think there's still some um bug fixing on going. It's been tested, yeah, yeah,.

A

B

So it's different! That's why I'm like, if you look at them side by sides actually and not not very so, okay.

A

Yeah so moz we're going back to google again, so moz is teasing us about showing vsphere stuff, but moz don't worry about it. We're going back to the google csi driver and.

B

We actually we actually look at the the aws why it's actually more similar to the google one. I think maybe.

A

Yeah, that might be interesting. I think it might be worth having them side by side, even maybe zing like having one on one side and one on the other, so we can um yeah. This is really cool, so I I never have looked into this code in detail, so this is cool. I'm appreciating it aws csi, okay, cool right.

B

So um if you look at the so, if you okay, I think it's this one controller right, so you see here they actually recently. This is july. They recently made a change to ensure credible ad potency utilized latest, go sdk.

A

So now we're in the okay. So now we're in the aws ebs provider.

B

Yeah, so if you look at this right to just check and see what what changes they made here, and so it actually added this uh added this arrow uh basically uh to check if the parameters mismatch, so let's actually see yeah. So, basically, when it, when crit warning um returns an error, it checks to see if, uh if it's this mismatch or basically, if making us making a call, we can create a volume call with uh the same name but different set of parameters. So that's a mismatch. Then you return.

A

B

That's they added this checklist, so I think this is similar.

A

So that's how they implemented that same item, potency, okay, yeah,.

B

So that's actually similar right. You actually still remember. We were actually looking at uh some code earlier uh if it's existing, then yeah right. It's similar here validate this is the google one.

B

A

Cool, so we got google on the left and we got amazon on the right and okay cool.

B

Yeah, so you see here, yeah, basically, checks validate and then it says: okay, it's already there. It's just incompatible, that's the same as this check here. Actually: okay yeah. So I think yeah uh and then I think, for for delete.

B

I think we were actually looking at oh yeah here so for the lead. I think it will just a check to see if the if the volume is already existing and it's going to.

A

Oh, we got a good question from carlos, so where is all this state managed? Is it in the is it in the actual pv object? Can I coupe cti at the pv and see that or.

B

Oh so this is the drive, so we are looking at the driver right, so the driver maintains its own state.

A

How does like for any given one? How does it do that like what, if I restart the driver, where does that data go? Is it a local file or is it on.

B

The api server, I think it depends yeah, I think it depends. I think I actually don't know how the uh you know the gcpd driver persistence. I think it's because they retrieve this from their cloud provider. So.

A

B

Stored somewhere, I don't know where that is so. It really depends right, so it I think it varies. If you look at the the very simple like hostpass drive with a sample driver, it just stores that in a file.

A

B

Like the vsphere driver, I think initially it was in an in-memory map, but that one, you know, goes away right. So now, actually in the new approach it's going to be saved in the api server. Okay,.

A

Cool but actually.

B

I don't know if that's the best either, but but that's better than what we have before. At least you know it will stay there.

A

Well he's saying I think, she's saying that uh carlos uh he's carlos is saying why not store it in net cd, so I think he's I think yeah yeah.

B

He's saying the same thing as.

A

B

Yeah yeah, so that's better, but I think um there is one slight problem, but that's probably you know maybe something we need to look into in the future, which is uh it's actually it's actually better for csr driver or not. It depends on kubernetes, because if you look at the, if you look at the css back okay, I think this screen is too small I'll go back to the full screen, so uh so the css back right, supposedly it's set or actually should go to the beginning of that right.

B

It actually says that uh I don't know where it says that somewhere it says that supposedly you are, you are oh okay. Here is the objective right, so the storage vendor right one plug-in, and then that should work across a number of container orchestration systems. But now I think, because kubernetes is, you know very, very dominant, so I think it's mainly kubernetes now and that's why we're not not thinking about that uh very much, but um it's actually, you know, should be just you write this driver.

B

It actually should just work for other content occupation systems.

B

That's why it's actually um yeah! It's fine that you want to save that in aps, server seems convenient, but then there's a drawback. Then it's not going to work for a non kubernetes. Ceo.

B

Okay, so yeah, so that's I think so. That's that for item potency.

A

It's funny because you know it's like you said other systems and it's like now. I can make my my misos joke right, because I've been waiting, yeah.

B

A

It wouldn't work in me. That's why.

B

Yeah yeah yeah, it's just they're kind of disappeared, but there's actually a one there's a nomad. Actually they actually complained that our driver does not.

A

B

With the you know, with them.

A

Yeah there you go so yeah. If you, if you started in the api server, you wouldn't be able to run your csi driver nomad.

B

Yeah yeah, so I mean that's something that we, you know still need to.

A

And you know who uses nomad roblox, and so, if you like roblox, then you should make your csi drivers kubernetes, independent.

B

B

So, uh okay, I will talk about the second part, which is you know, avoiding leaky resources, um so we actually, uh you know, run into um several bugs in kubernetes above us. So this this uh this is a bug fix this one was actually fixed quite quite some time ago, um just to go there yeah. So you can fix this. Since I said you wanna if you're 2019, uh so this is about fixing external provisioner, because uh what happens is uh before this fix. uh When create volume comes out.

B

Basically it sees that driver takes a long time to create a volume right takes the times out, and then we ended up having those uh volumes created on the sword system, but they are not really recognized by external provisioners. They are kind of lost, uh so there's a.

A

Okay, so he wants it to retry, create after a transient error. But I thought, if create, is item, potent wouldn't a higher level thing, be the thing that retries the create yeah.

B

The whole point of this item: it's not provisionary. It's the high level thing right. It's not provision.

A

B

We're talking about right so right now we're talking about the co side, not the csr driver.

A

Okay, okay, so this is the higher level thing: yeah, okay,.

B

A

B

Driver will also need to do something and then co also need to do something to make sure, and there is a nothing that leaks so um yeah, so you see that he actually made this change uh so so now there are three status. If it's finished, then that's you know now we know that driver is completely done. It could be failed, it could be successful, but it's finished.

B

No change is a temporary error and then in background, meaning that it's still ongoing. So we can actually look at what are the transient arrows there? It's a final error yeah. So if you look at here, this is a credit volume right. So do you create volume and then, if this, uh if there's an arrow, and then we check, if it's the final error, if the final error, then we return finished.

B

Otherwise we return in the background. So it's still ongoing. So so here there are a few error code here, so if it's either the client application console or timeout. So actually this is the one that I see most, which is this time out. The csr driver comes out while creating volume uh and then um or if it's a server, shutter or temporary, auto resource of all of those- and this is a csi uh css back defined error code aborted, which is operation pending forwarding. So all of this will say it's john go in so return.

B

This is not final return force yeah. So so that's the that's. The bug fix there.

A

B

And and then after that right we still discover uh stations. Well, we have leaky volumes, so this is an issue. uh This actually is an issue discovered by a customer. Actually, so they found out that, depending on you know how you delete the pvc, you know whether you delete pvc first or whether you delete pv. First, you get different results, uh so this one was actually, but I was actually surprised when I found.

A

B

This one, so I can't just show you how this one is like right. So uh so I have this.

A

Focus, oh so carlos is asking which specific volume takes a long driver takes a long time to create a volume which one were you referring to.

B

I think it depends right, sometimes it just because it's busy, then it could take a long time, I'm not really picking up on any specific drivers. I think it could happen.

A

Yeah, like I mean you could imagine a quota situation, maybe right in a particular cloud where you have some quota and it's.

B

uh In that case, I think it should fail.

A

Would it fail instantly okay, so, in that.

B

Situation, it would fail, so could also have like network issues right. So if you just some, there are some problems. Your css driver cannot communicate with your system. For some reason you know some temporary issues, but maybe it lasts like an hour or you know you could also get timeout.

A

I mean: is there a guaranteed timeout on a volume create or is there a default timeout.

B

Yeah there is a default. Actually you can actually set that so the external provisioner. Actually uh it actually has the timeout actually operation time yeah. I see this one right so in the in the external provisioner, you can actually set this okay there's a default here 10 seconds, but then you can actually, if you so.

A

A vendor would set that or.

B

A vendor yeah yeah, because the this this is a side card- that's deployed with your plugin right. If you just use the driver, so you can actually set that.

B

So um so so I I I showed earlier that I have this. uh I have this volume here right um you, if you look at the reclaim policy here, it says, delete delete means when I delete the pvc it should just go ahead and delete the pv and also the volume associated with this right. So I can actually go ahead and do that.

B

So this is a yeah. Okay, the normal operation right.

A

Yeah all right, so we won't let you yeah and.

B

Is there a question.

A

B

A

Wait, there's always questions okay, but but it does keep retrying correct.

B

Yes, it should it will retry yes, so it should retry forever yeah um yeah. So uh if you look at here right, so I deleted that right because I delete the pvc you see that deleted and then you can go check here.

A

This is the normal situation.

B

This is normal. I just want to show you what is normal, what is not normal right.

A

B

So now I should not see anything anymore. It's deleted.

A

B

If I, let's I'll, just create another one.

A

Are you going to show us the weird thing now the thing that doesn't.

B

Happen: okay, cool! I I thought yeah this is. This is really weird that I did not expect right. So it's crazy already um see it's here.

A

I don't think you can always recover the data. I think there's different policies for different volumes right like there's some policies you can set where volumes will not delete the actually get deleted, like from the underlying storage, but they'll get deleted from the kubernetes api of pvs, but then I think there's other volume controllers that will actually- and I think it's normal like if I'm in the cloud and I delete a volume- I think it's normal- that I lose the persistent disk associated with that volume.

B

Well, so you can set this a reclaim policy here right, so this reclaimed policy can be deleted or retained. If you want to retain it, then yeah.

A

I set it to retain.

B

Then you can yeah, then this will be there.

A

Right but that not all csis implement that retain right.

B

This is the this is the kubernetes side right, so kubernetes had the uh the pv pv controller and the um and also as well.

A

But do all are all reclaimed policies implemented by all csi drivers.

B

This is another csi driver side thing: this is a it's.

A

Not we need his.

B

Level thing yeah, so this one this is not depending on your css driver, so this policy will always be there, because if you set this reclaim policy to be retained, then the external provisioner is not going to call your css driver to delete the volume. So it's not going to be deleted at all. It's.

A

B

Up to your driver, yeah.

A

Joe thompson says: checking in from wv. I don't know where wv is nice to see nods to the container orchestrators. My current and previous employers run yes, joe cool. I like your hat by the way joe, you know, I had one of those when I joined red hat like five years ago. um Okay,.

B

Okay- so I just want to show this to this one right so this, but if you delete so I just you know last time I delete by just depicting the pvc directory now, if I wanted, if I believe pv first.

B

Right so it's a hanging because there's a finalizer there, so it's not going to be deleted, but the time stamp the deletion timestamp will be added to pv. uh So so it's actually the pv right. Now it's actually still still there, but if you get it, it's still there right. So uh now, let's now, let's go ahead and delete the pvc.

B

So now you see that tv is gone and then so they're gone here right. But now, let's check here: okay.

B

So this guy's still here, oh yeah, so this is so this is a leaking right. So basically what happens is uh even though the deletion is delete, but if you try to ddpv first, then you run into this problem. Your thought is deleted, but then it's actually not it's still there right. You know all your kubernetes.

A

B

A

Just there wow.

B

Yeah yeah, so this is a this is a bug, uh so uh we actually we're trying to.

A

Fix this, it's a bug at the eight at the kubernetes level or at the.

B

Start of that, what this is level yeah, so we are trying to so we actually have a. We actually have a cap for this.

B

A

Carlos I see what you're saying about the pvs I'm just like wondering like how is it that at the kubernetes api level retain recycle delete? How is it that kubernetes can guarantee those semantics across different storage providers.

B

Oh so, basically, depending on the so depending on this policy right. So if okay, maybe I can actually show you, I can just show you the code, maybe a little bit. I was just uh yeah. We can actually.

A

Feel free to show us the cap, and then you can show us that later.

B

Okay, so the cap.

B

B

That's what the cabinet says owner pv, reclaim policy, um so I I will just jump into the this is the uh how we are going to fix this so the way to fix this uh is well. We will be adding a feature gate because the since this this uh behavior, it has been there for sin since the beginning right, so we don't know if actually anyone is depending on this behavior for some reason, uh so we don't want to just uh you know, uh change it without a feature.

B

Gate give people some time so so this is actually uh we're. Adding this feature. Okay, so there is a finalizer. This actually is a it's actually already defined in this. uh This external provisioner.

A

So: stick: storage, lib, external provisioner: that's the thing that the csis vendor as the basic logic for how they're implemented.

B

So yeah, so this is actually the this is the provisioner, and then this will be the one who actually calls the uh external provision external, provisional sidecar. So this this provision is also used by other provisioners. I believe so that's why this is. This is a a lib.

A

Says he loves the way I do tgik, I feel so appreciated. Right now, I feel like zing is making it so fun for me because she's doing all the hard work so all right, cool, currently on only nfs and host pat support recycling policy. Okay, yeah.

B

A

What I was wondering I feel like it can't possibly always work: okay, cool.

B

Keep going zing this is. That should be the same though I mean it should not really matter. I mean with this reclaimed policy part right. If you're talking about item potency, then there could be bugs in the drivers or you know some someplace else, but but the reclaimed policy should work. But in this case you see that the clean policy is not owner right, so you can say it does not work because there's this bug so we're trying to fix right.

B

So um so this finalizer is actually already it's already there, but it's always it's actually just not uh not added by default because there's a flag there, it's always false. uh So what we're going to do is just to you know with this feature gate if the feature gate is enabled, then we are going to add this finalizer to the pv. So every time the the pv is after the volume is created, uh we will when we create the pv, we'll add this finalizer there and then on a deletion time.

B

We will delete the volume first and then remove the finalizer so yeah, so that's the that's! The uh fix propose the fix. Oh, this is cool okay, cool yeah. So that's this one! Yes, I.

A

Think this is really cool. You actually brought us. You showed us a problem, it's the first time on tgak that anybody's actually showed us a problem and then showed us the kept that solves it all. In the same thing, um that's really cool, or at least the first one that I've been involved with.

A

Usually mine are just complete mess, I'm just like just trying to do stuff and it doesn't work. So this is cool yeah.

B

Yeah so trying to get this one in one or twenty three and and then there's and then there's another bug so just still yeah, it's like we have already. You know done so much, but then there's still some cases. Then there is a potential leaking there. So this one you know, found this one black open by patrick here. uh So if we start to create volume, we basically cause etcetera decree volume in external provisioner.

B

Now before this is complete, the exact permission is stopped. So at this time we don't, we don't have a pv yet uh and then pvc is deleted and now external provision started now we actually lost that handle to that volume because that's still going on in the background, we don't even know that exists.

A

Okay, so it has to create it, and then you kill the provisioner before it's created right and then you get rid of the pvc.

B

A

External precision is so so then you restart the thing that was supposed to be creating it, but.

B

The thing that was yeah does not really have any does not have knowledge, because pvc deleted doesn't really know that the csi driver is still going. It's still working on that. So.

A

Nobody can reconcile that original thing, so this would only happen if, if so, there's not some kind of an acknowledgement at the end of the creation of like.

B

No, this is a yeah, so this is the also back. So uh this is also you uh so fabio is working on this. I think he's going to send me the cap for this one as well. I think the proposal is also I had a finalizer uh on the before.

A

B

A

How do you add a finalizer before something I don't know.

B

This is the crea. This is the I definitely on the pvc object. The program means this. Is the csi call right, so you have the pvc object. First,.

A

B

Yeah, that's it cool.

A

B

A

Okay, so you're going to put a finalizer on the pvcs and right now, pvcs don't have them. Okay,.

B

Yeah, so uh I I think that.

A

Requires an api: what is that? What does that change that doesn't require to change the api? That's just a change.

B

It does it's going to add a finalizer so similar to our previous fix, the you know, the previous problem. I was showing yeah. This is the proposing to idle finalize on the pvc itself, so the one that I showed earlier this one it's a this finalizer will be added on pv cool, so this is protecting the volume and then this one is basically, but this is still uh relatively new because you know, I think, he's still doing some prototyping um yeah.

B

So I I I can go to the so I should I want to show you the.

B

Provisioner when it actually creates that pv object right so if to provision right, so uh it will creep up provision then close, create volume here and then after the volume is created, then it will go, go ahead and create this present volume pv right. So but then that's the thing here. uh Then there is a possibility. Then something could happen in between you know. When you you talk, you you call it create volume. You could make the csi call uh it's not back yet and then so you don't really have your pd yet so.

A

B

Is created after the warning is created, so I just want to show you the difference, because I'm going to talk about the snapshots, that's a little different from this. We implemented that differently. I.

A

Have a I have a random question before you go to the snapshots. I have a random question: do you all use those golang contexts for this stuff?

A

Because you know with the golang context, there's that whole like cascading thing of like, like I haven't, used them much but like I've seen them used in different places like where you you can have a golang context and then that context can have several things that happen and there's like a whole automatic rollback that can happen.

A

I'm just wondering whether that stuff applies to this kind of stuff.

B

This is probably a little bit more messy. It's.

A

B

I don't know if I can use that.

A

Never mind keep going.

B

A

Just shooting from the hip, okay, yeah.

B

Yeah, so uh so this right, so that's this one. I thought, if there's a question earlier, that I thought are we trying to answer.

A

We have to say we got hundreds of questions.

B

Right is that the reclaim thing is that.

A

We were talking about reclaim and then you dug into that and you were yeah, but oh okay, but I I think you can keep going down this road.

B

Okay, you can maybe talk about that one later yeah, it's fine! So so that's this one! I I actually want to. uh I actually want to show show this, so the so snapshot right snapshot. This one was added later. So the support for snapshots added later uh it's actually modeled after the pvpc.

B

So you can see a lot of similarities, but we also made some changes because you know when we see some problems, we also made some enhancements. uh So one thing one thing that is different is actually exactly this place like to show you.

A

Carlos is excited about snapshots.

B

So snap shutter so so snapchatter we also have two objects. We have a volume snapshot that is similar to pvc, that's a request to a snapshot, and then we also have a volume snapshot, content uh that is referencing a physical snapshot. Just like you know, pv is referencing a volume system so, um and we also have uh two controllers right. So for the for the for the volumes, we have intrigue. Pvc controller uh that is handling you know, they're doing a lot of stuff handling. The binding of the pvc.

A

B

Similarly, here we have a common controller. The snapshot controller is a com controller that handles the binding of the volume snapshot and women's natural content, and then we also have a side car. So this is like the external provisioner, um so uh when as soon as one difference that we uh that only show here.

A

B

Right, so, if you look at here um when we, when we get a volume snapshot, requires a new, a new volume snapshot, so we are going to try to create a snapshot array so uh very early on. We will, in this common snapshot, controller, will be creating a snapshot content. So that's the volume snapshot content api object, so you see that the sequence is different from the external provisioner, because it's the provisional, actually we'll be creating the volume first after the voting is created, then create a create pv.

B

But in that case, if, if you know something happens here, we don't really have a pv that controls that that volume right so here we're doing things differently. We actually create the world snapshot content first, uh so this uh is to avoid leaking. So we always have this handle first.

A

B

Yeah and then this this volume snapshot volume snapshot. Content um will be uh so so this sidecar controller. So this is the seaside snapshot of sidecar. This is going to you know, watching the volume snapshot content. So this sidecar is pretty small. uh It only communicates with the csi driver, so it basically will only watch the volume snapshot content. It doesn't even watch the one snapshot if you object.

A

When you say it's a sidecar controller, it means this is a separate container in the same pod. Normally.

B

So meaning that this will be deployed together with the with the csr driver, so it will be watching the api uh object, like the voting system.

A

So it's another controller inside of the snap shutter yeah, okay, gotcha.

B

Yeah, so this one will be communicating when we talk about sidecar, meaning this will be communicating with the css driver, so this guy will be making call to the csr driver.

A

Well, the common controller.

B

Does not does not make a direct call common controller only communicate with the uh the api server.

A

B

Yeah, so this uh so.

A

How many separate controller watch things? Are you running inside of the snapshotter.

B

uh So for snapshotter we have a common snapshot, controller and then sidecut deploy with the css driver. We also have a validation web hook so yeah, but for the controller part it's just two, the sidecar and the common controller.

A

Okay, two and then a web hook, so you've got three processes that you start: okay,.

B

And okay, so if I go to quick snapshot right so so this basically will be watching the watching the snap one etcetera content right as we look at here right. So this one is getting a one snapshot, content, api object and then there's some checks. uh It checks. Well. This is the for the delete, part checks. If it's going to be well actually this this actually is similar to what the uh what you know the previous question. We can actually just look at this one. uh I think the question previously is about the reclaim policy.

B

We actually have similar policies for volume snapshot.

B

um So if you look at here right, so we're actually checking to see if the the policy deletion policy is a delete right, so it's either retain or delete only if it is delete, we will go ahead and make a csi call to delete snapshot.

B

So you see that's how we can guarantee uh how this can work for all the drivers, because if deletion policy is retained, we're not even going to call the csi daily snapshot is not going to be deleted. You see what I'm saying um yeah so same. This is the same thing or similar. I say similar to the you know how this is handled for the for the reclaimed policy for the for the volumes right similar. Basically, you check the policy. If it's retained, then it does not even go make the call.

B

It does not call the driver. So it's not going to be deleted. Yeah.

A

B

Yeah, so so this is the deletion part, basically uh so for uh I so for the for the for the create um yeah. So basically this one will be watching the volume snapshot, content right, and so so now it's going to make a csi call to create the create a snapshot right. So basically it's in the common controller.

B

We created content. First, we have that handle basically have that api object uh and then uh come to the side. Car we're going to do really making the call to the seaside driver right. So that's one thing that we uh we're doing and uh another thing yes, and there is another change that we made so initially when the snapshot support was introduced, um because we think that snapshot is normally time sensitive. So when people take a snapshot, they want snapshot to be taken at that time. So we actually don't do re-twice.

B

So we thought you know that's important, but then we got leaking snapshots because if it times out, then you know you get snapshots created. Then you know we're not really taking care of those, so we actually make me make the change uh so now. This is actually the same. Very similar to uh what is done in the external provisioner, which is we always retry chris snapchat when it fails, we always retry um so.

A

uh So now we have some more comments: rolling in um t t g, I k dot. I o slash notes so uh lucas, um I I don't. I didn't watch it, but it's good that there's vmware and red hat stuff going on. I don't know what this bgfs csi driver is. What is do you know what b b e e g f s is uh using.

B

uh Let's see if, let's see we have a dog, let's see if it's in a dog I don't know which one they are talking about. Maybe they can tell me if I show the dog here so so, there's a csr driver dog that uh so any csr driver can add an entry to the stock and put the driver here right. So the drivers.

A

So so yeah and you can add it to the notes too lucas by the way so feel free to just jump into tgik dot, io, slash, notes and um and add that to the notes- and you know like.

B

Yeah, so what is the driver name that we can search and see if it's here, then? I because I don't know which one you're talking about. If you add that to your notes, maybe you can you can search to see if that driver is already added to the stock?

B

Is that your driver was the driver you're using.

A

B

So the driver maintainer can just submit a.

A

B

Are in the you know in the in the you know, you are. If you are a driver developer, you can. Actually uh you know you can come here. You can submit a pr and at your drive and see, let's see if any new you don't have a new any new job. You can see right. There are some pr's here, uh yes and there's a there's, a pr ibm blog. So I think this is the update, but you can submit you can submit apr to add an entry here. You see a lot of drivers here.

B

You can add an entry here. If you have a driver, that's not here yeah and then you'll show up in the stock yeah.

A

B

Think joe's.

A

Red hat because he has a red hat on wallet, joe walid is asking if jose from hashicorp.

B

A

B

So yeah uh uh well, okay! So we're talking, oh so we're talking about this uh this actually, so I just want to show you this. We actually have something similar. I remember we actually look at this one in the external provisions. We actually made a fix here because uh we found there were leaking snapshots right. So we actually also added this check here uh when we create snapshot.

B

We also want to check to see if the final arrow, so, if it's a time out, then you know that's still going on in the background. So that's not a final error. um In that case, uh we will still, you know, keep going um so uh we also, we also actually added a annotation.

B

So when we create when we get a creator, um create a snapshot request before we go ahead and create a snapshot. We'll add this uh annotation uh one in snapshot being deleted. I've been created, and so we know that this is ongoing and then we will not remove this until we get the final error so now see if it says the final error, then we will remove this annotation we allow. Then we allow this uh snapshot to be deleted.

A

So it sounds like the job of a csi engineer is like you just spend a lot of time, trying to figure out what things you need to remember.

A

Right, like I guess, that's everybody that right that does anything not here as an official hashicorp rep just participating walid. I was looking into the ibm gpfs csi driver this week and it's now called spectrum scale. Okay, carlos, oh.

B

I think that's there. I remember that spectrum yeah this one.

A

Yeah that one ah okay container storage interface for that.

B

That's there so the driver yeah. So if you click link there they'll go to that. uh Oh it's not a! I thought it was a github guitar. So, okay.

A

Ibm is just all load. Tgik is like you know it's like every it's this time. Every week it's like that last 10 minutes. We always do something ibm related nowadays.

B

uh I yeah, I think I think that's all I want to show here. uh Are there any more questions about this, or did I miss any questions? I have.

A

Like a million questions, and so I'm hoping that we can get you back here to do a deep dive into the csi api and how it works at some point, but uh I mean this is really cool, because you've kind of showed me all the stuff that I never actually think about um that. I need to learn about.

B

I hope it's it's not too much deals. Details.

A

And then um carlos is saying something he might he's getting a bonus in the fourth quarter, so he's gonna he's gonna, take us out all out for like for donuts at the next kubecon. So I'm excited about that he's. Gonna buy me something with his bonus joe thompson. We have horror stories about gpfs.

A

um Okay, so zing, I have one question for you: unless anybody else has any other questions, what's your favorite csi driver and you can't say vsphere.

B

I cannot say this: fair, okay, gcpd, that's the first year.

A

Okay, all right, you heard it from the best the her favorite csi driver is gcpd.

A

Okay, cool there's, never there's, never there's never too much code in tgik.

A

That's true: let me go deep here: don't we all right, web3 csi driver on the making walid all right? Well, you all have been great everybody's been great walid and carlos and lucas, and everybody else has been jumping in and sharing stuff with us and thanks everybody for welcoming zing onto the show. That was that was, and thank you so much of course, zing um for showing us all this stuff.

A

uh We not not very often that we get to go deep with an expert on things, so this is cool um blockchain csi. Can we make a blockchain csi driver zing? Can we do a startup after this.

B

Yeah, if you write it a column, colosseu.

A

But I think we'll make it in secret and you know where we're going to put it we're going to put it in that. What is that town? That's doing all the blockchain stuff nowadays, that's it's like I forgot, there's some town, they have a bunch of rivers and, like all the blockchain stuff, is going to that town. Now, in some town in asia or some country, I don't know laos they're doing all the bitcoin mining nowadays, hadoop csi, yeah ricardo.

A

Thank you, martin. Thank you, martin, for.

B

A

All right um cool, so I'll I mean, I guess, yeah, let's, we can just start closing up here. If anybody has any other questions, carlos rust, yes, we'll be rust and deploy webassembly we're getting fancy now. So all right, if anybody has any other questions for zing, well lead absolutely anytime and waleed. If you have any other ideas, feel free to like ping me, you know I'm on kubernetes slack, I'm and you can find me on twitter. uh You know so whatever like. If you got something you want me to dig into on an episode.

A

Just let me know I'm happy to do one um and hopefully we'll get zing back again and um seek open ebs in the future. All alex, I don't know yeah open vbs. I don't see why not. If we haven't done one already, um what do you think about open ebs thing? Have you ever used? It obvious.

B

So that's actually a sandbox project in cncf. They are actually trying to apply for incubation. They actually had a meeting with them this morning.

A

Okay, what what did you? What what? What? What sort of what can you tell us um why it's a cncf project, I'm confused.

B

It's a storage project; basically it also it has. I think it can provide local storage. It also have a replicated storage. It has a few engines. Why.

A

Would somebody not just do this in amazon.

B

Yeah there are well, if you look at the the driver, you know that dog that showed you earlier right. We have, uh I don't know how many I think we have more than 100 drivers, yeah, so yeah. We have a lot of choices.

A

But open abs must be doing things that aws doesn't do right.

B

uh Well, I think it's just a you know: it's just different. It's like you know. You also have rook right. If you look at you know the storage projects in cf, you have rook, you have longhorn. That was um also, uh I think it's actually also applying for, but isn't.

A

Ebs specific, oh ebs, is not specific to aws.

B

No, no, it's not! Oh. You saw this. Oh.

A

B

Is a different project.

A

Amazon, elastic block store is not the same as okay. I didn't know that all right, so all right, cool, so open ebs is not amazon. It's a generic elastic block store that can work on any technology under under the hood right.

B

Yeah, okay, it also has you know, other oxygenation part, not just.

A

So I can run it on any cloud. Okay, I didn't.

B

A

A

Oh, this is okay, cool.

A

Well, that's your reason, then, because you want evs like functionality and you don't want it in amazon.

A

Cool anything else before we lose zing.

A

Amazon have ep abs, evfs and luster people still use luster. I didn't know that, but do you you cannot do across regions. Can you you need to use portworx, storage, os or something similar? Okay,.

A

Cool I mean we should just do a whole storage tgik one of these days, but unfortunately, today is the show's over, and I guess it's it's time for the weekend, so I I don't know it's great hanging out with you zing. Thank you so much again for coming to coming to um join us. Carlos says, thank you zing. So you are, you are our favorite and we are so glad to have had you today. Thank you. This is great.

B

Thank you jay and thank you. Everyone.

A

Yeah this is cool all right, so let's dig into this stuff again next time, lester's the one from oracle- I don't know, I don't know if there's lester and lustra- and I don't know you know- I used to work on gluster and I don't know whether that actually came from there. But um so you know what I used to work on with me and brad child used to work on this lester. F, that's hadoop! Back in the day, oh.

B

A

Connector um anyways take care everybody, everybody's, making, sun micro systems, jokes now gluster is red hat, so I guess all right take care and thanks everybody, uh you remember it. I thought um we'll see you next time. Thank you. Come for coming to tgik, thanks for hanging out bye, everybody.