Kubernetes Storage SIG Miscellaneous, 3 Mar 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: CSI Community Sync 20210303

Description

Community Sync up Meeting for Container-Storage-Interface (CSI) - 03 March 2021

Meeting Notes/Agenda: -

Find out more about the CSI here:
https://github.com/container-storage-interface/spec

Moderator: Michelle Au (Google)

A

I think in the past, okay, sad.

B

All right today is march 3rd uh 2021. This is the container storage interface meeting. um I think we will just go down our agenda for today. uh First topic is yawn.

C

Well, it's not a real topic like this pr is open for quite some time. All the questions have been answered. I'm just curious if it can be.

B

Merged yeah, I think, if there's no outstanding items, we should be good to go. We just need.

B

A

Otherwise, it's fine.

C

So, who, whom should I think to merge it.

C

D

Right should we also talk about like briefly like what we we discovered from talking with xfs engineers, people.

C

All right, there is a pull request in kubernetes that implements necessary functionality for the csi drivers and it could be like exported to staging in the mount utilities, and we talked to xfs engineering in red hat.

C

There is no easy way how to get physical uh size of the file system like how much the file system thinks, how big it. So we are scraping uh x, you in dump dump efs e2fs utility, uh which is nice well, nice. It is possible and we were parsing is xfs. Information. Ugly and xfs engineering told us that there is a special io ctl for xfs to get the size and there is a simpler command line utility or including the pr to get the value without too much parsing.

C

So for xfs we have ioctl for x3. We don't have any like api, but we have sort of nice, two parse command utility, so this is good, but as if somebody wants to add any other fire system, then this needs to be added to the mount utils in kubernetes.

C

I will find the full request. I.

B

Guess one one question is: is that um in kubernetes we only have things for xfs and ext4 and we don't really support other file systems. So if a driver is supporting a different file system, does that mean they need to um sort of come up with their own utility library.

C

Well, yeah, it depends on the csi vendor will fire system they want to support, and I haven't heard anything everybody's happy with xfs and x4.

D

I guess the the one concern that that I was having was like like what, if the file system that they're using doesn't have a way to determine file system geometry in the way like we were expecting like there's no api for that provided by kernel or or the the the file is terminated so and then, like the xfs engineers suggested that we could always do research and every amount.

C

And um well, but then we didn't get answer if this is safe and how how safe it is.

D

Yeah, we didn't get answer for that, so.

D

D

So, at the restore time we know the file system on the snapshot. Do we? I don't remember.

C

At the restore- yes, we know the snapshot, we know the original pvc do, we know the size, we just don't have a way how to pass it to keyboard, and we were talking back and forth about this and we decided it's a driver problem.

D

But the driver knows in the control plane the file system when it creates the.

C

I don't know actually.

D

I know the pvc, but but just the csi driver knows because.

E

Fs type is passed in as a volume in the volume capability for mount volumes.

C

Yeah, but not the size, not the original size.

E

Oh, I'm sorry yeah. I thought we were talking about a different subject, never mind.

E

That's nice spaced out.

B

Okay, um I guess so do we have a conclusion on do we need a conclusion on this one, if there's some other file system that can't actually uh get this information.

C

um I think, uh as far as it comes to csi spec, then we are fine uh implementation, wise on kubernetes.

E

I actually implemented this in a couple of my drivers, and I noticed, while I was implementing it, there's a few different options, but like things like xfs, grow, fs and resize e2fs um are both item potent like you can just call them over and over again, and if the volume is already the size of the underlying block storage, like just nothing happens so like, I would imagine that any other file system would conform to those constraints.

C

Oh yeah, but then there is a huge warning uh in the month page, but he used to be ornate. The nightmare pages for xfs then be very careful and call through fs or it was somewhere. I don't know, I didn't already have documentation.

E

Well, I was gonna say that that's that's one implementation! You can also call xfsgrofs in such a way that it just outputs the current size of it, and then you can compare that to what you think it should be and and then do something um so like xfs, has a bunch of commands that dump the current file system size without trying to change anything yeah.

D

But xfs info was the one, but it's very very ugly actually, and I'm not I'm not like we. I think. For example, we really have to use the io ctos.

E

Which just dumps the information.

B

Does directly calling the ioctals potentially put a.

B

Dependency on the kernel version.

C

E

C

These things are stable for years. I hope.

E

I think the downside to calling eye octals is is uh porting them across architectures.

E

It can be a little funky like if you only support amd64. That's one thing, but when you start supporting arm and other platforms, you know doing low level stuff like that gets a little more hairy in your code.

E

Because you start to touch, you know like the the syscall layer, and I can't I can't actually remember if if sego gets involved and you have to have glib c involved or or if go can do it all directly, I think I don't think you need sigo to call iactals.

D

E

D

Draft is the prince same thing that xfs info prints. It's the same same format, same thing, so it's same ugly as xfs info.

E

Oh you just don't like parsing the text does that complain. No.

D

It's the kind of exception. You just told it: it's not stable, it's not meant to be consumed, programmatically actually.

E

Right but but if you, if you ship the xfs drawfs binary in your container, then it will be stable for the life of that container, which is what we do.

E

So, like yeah, the binary could change from version to version, but if you ship a version of the binary with your csi plug-in, it's guaranteed to be stable within that csi plugin.

E

So you can do the ugly parsing thing and just deal with it. It hasn't been that bad I mean yeah. I too would prefer something nicer, but I I don't see it as a practical problem.

D

No, I was, I was like just my I was thinking like if, if like, if you put it in css spec, that the the driver is responsible for the size and driver doesn't know the size, and it has to query the size by reading the file system, geometry and it's not possible for certain file system types or you know like as then.

E

D

To it should throw out of range error, maybe that's that's! Okay and.

E

Do we have an example of when they can't do this, because that would be helpful?

E

I I think it's pretty reasonable to expect that every file system on earth can just resize to fill the current block device.

D

That is one and then so. What are we agreeing on okay? That may be part of kubernetes implementation. Detail like how we're gonna resize it. We could always always expand on every every amount, but maybe not for this call.

B

All right um yeah, I think that sounds good, given that right now we mostly know about xfs and ext4, and we have solutions for those.

B

All right anything else on this topic.

A

So it's the plan to have this merged and just say driver need to do this and then in the future. uh We are thinking about uh adding this in kubernetes to expand volume with something in the future.

E

I don't think there's.

A

E

There's nothing to add to kubernetes.

C

Like well yeah.

E

C

Mine utilities imported by by many csi drivers, so uh the resize property size should be there soon-ish. I hope in the next.

C

A

This is why this is for, like 1.22.

C

Well, it's 121.! Oh.

A

This one, I think we can do that, it's a quote freezing. Then it's coming right.

C

Yeah still, four days.

E

We're driving and getting a csi release out before the code freeze so that we can get a bunch of stuff fixed in kubernetes, 121.

E

I don't know if we're going to make that deadline. I was.

A

Just yeah: that's what I'm is they're.

E

A

B

I mean even with even if say like this is a spec changes aren't able to make it. I think we can still add it to the mount utilities right.

D

Yeah yeah we we may not. You may not want to call it all the time on mount, but we can at least merge it to the monitor so that it will be used externally by.

D

B

Okay, um next topic.

B

Is volume mount group.

D

Okay, so james had a comment yesterday I think about the I.

D

There were two things. One was that the for.

D

For intent of mount what should happen like the new files, the like originally I'm? I was thinking that my concern was that it is too specific to just call new files that new files should be readable. Writable by the volume amount group and uh the it's in uh it does prevent like ch on chmod use because clearly, like you, cannot just see each one and expect the new files to be group writeable. They will be generally user writable.

D

So you must even use mount. If your mount option is working properly, then it will be group writable and you cannot just fake it with ch on chmod.

D

It does seem to fulfill what we want for azure file and cfs mounts. So I kind of included that wording back in like what james suggested in the in the spec, and I think he hasn't had any like yeah. So I kind of went back on that and I thought okay, fine, it's just! I just put it there and I think in future. If there are, there are volume drivers, it's that that can support like that concept that apply volume.

D

Group identifier, like volume, mod group, but they only on mount basically basically like there- could be driver that behaves. They apply mount option same as like how ch own behaves internally like like, where on each mount it the permissions are mapped, but new files are written. When you write a new file, it doesn't automatically get get group uh lead, write permissions actually.

E

Wait is that a real thing or is it a hypothetical thing? It's.

D

A hypothetical thing it doesn't happen.

E

D

E

Yeah, so if that happens, it's not good enough and you, you can't use this capability right. Yeah.

D

Yeah, if that happens, it's not good enough. You cannot use that capability because the capillary is limited to like the new files should be group writable and readable. So that was my main concern, but it's proposal being alpha. Maybe it's fine too. I don't know what other folks think, but it's fine to have that wording. It prevents ch on ch.

E

Isn't the real solution here to just like have some tests that exercise this functionality? And then anyone who wants to pretend that anyone who wants to claim support for just runs the tests and sees if they can pass them.

D

Yeah sure, like insanity, tests or something, but we still have to capture in the wording what it, what is expected right.

E

Yeah but okay, I guess it's clear to me.

B

I get so the current warning does talk about the new files right, so yeah. That should be fine right. The word current wording captures that case.

D

Yeah, but it it doesn't support the hypothetical use case that I was talking like if, if there is a there's, a the volume type or file system type that that applies the permissions on mount, so that you, when you mount a volume, you get group ownership, group and group readable writable. But if you create a new, if a workload creates a new file, the new file doesn't automatically inherit.

D

You know, like group ownership or group, read, write permission.

E

So so you're worried that the wording isn't clear enough that it should prohibit that case because, like the wording can only be so clear.

D

No, no, the wording is clear, it could prove, but the wording will prevent such a volume driver from using this monster right.

E

And that and that's what we want so I don't see what the problem is.

D

Why would we want it to prevent on purpose if it works like? If, if say, if a driver supports that and then, if a driver like.

E

If the mount facility doesn't apply permissions correctly to newly created files, then it violates what what the goal of this is, which is that you can run workloads that depend on that on every file and the volume being accessible to the workload yeah.

D

But right, but the workload will be able to read and write files, because new file that it is creating will have user permissions by default like that's how in unix or linux.

D

So it will be able to read, write files because it's getting user-based permissions on on new files and directories and.

E

Okay, so so so you're saying that it's too strict now and that we need to loosen it to say that that's actually, okay.

D

E

D

Yeah, I think it's too strict, but but we'll see I mean it's alpha proposal, so maybe it's fine. So that was my.

E

Concern I I say I say: let's get it merged, you know. If and if there's questions about it, maybe we can specifically address those questions somewhere or I I like the idea of having a test that basically says this is what we expect to work. If you pass the test, you've cleared the bar um and then because tests are much more harder, they're much harder to misunderstand than english words right. You either passed the test. You didn't.

D

Yeah, I think the another thing is like a casual reader of spec may not understand what new file mean in this context like for.

E

Example, but I mean I- I don't think that we can solve that problem right yeah. This requires deeply understanding. What's going on.

D

Yeah, okay, so I have updated the wording for that and I think it's fine for for the implementation that we're posing there. One more thing that I have made change and james was looking into us had some concern was like okay. So originally we wanted, uh like my original proposal, had like node stage. If you publish, if you note, stays the volume with one group, then you cannot note publish with different group, and I put put- must wording there.

D

If you remember ben and like you, uh the stu must use the same uh group that was used during the stage I changed relaxed. To must to me actually like, like, I know, not relaxing, but I.

E

Should should is better than me right.

D

No, I dropped that thing kind of altogether, because I realized that many drivers do not may not actually perform a mount on on the stage like I know chris was chris's driver. His driver does not perform any mount on on on stage. So if, if the actual mount happens on the at the publish time, then that means that multiple volumes, like the same volume, cannot be note published with the different gid. If this so.

E

I think I'm agreeing with you, but I I just I'm confused, because I thought that what that we had agreed to when we last met was we want to the only thing we want to guarantee will work is if it's the same right. If you use the same one for node stage and no publish, we will guarantee that will work if you use different ones it, whether it works or not, is up to the driver and yeah. So the way I would communicate. That is like that.

E

You should use the same one you don't have to, and and here's and here's what you give up, if you don't, you know.

D

Which is read, read the updated wording michelle like. Can you pull it up or I don't know?

D

uh I don't know if there's a way, yeah? Okay, thank you. So if you go.

B

D

Note publish notes, no yeah note publish will have like yeah yep. This is the one, so this updated wording says if plugin does not support, note, publish volume with different volume monitor than the one used during node stage, while the plug-in may return field, reconditioned error.

C

D

C

D

If sp does not support not publishing same volume, one same node with different volume on group, it may return fair precondition. So this I have relaxed.

E

D

Wording: yeah yeah. This sounds great.

B

By may then, what is the other alternative that they can do.

D

They can succeed, yeah.

B

But here it says if it doesn't support.

D

So if it does not support would publish volume with one with different volume mark room than used during the stage, then it it may return. Otherwise.

D

E

So so, if you don't support it, there's no other option than to return fail. Preconditioner. Is that what you're getting at michelle yeah that? What we're saying is, if you do support it, you should succeed. If you don't support it, the way you communicate is by returning failed precondition, error and- and I guess the may emphasizes the fact that you don't have to support it, but you can um but yeah. I don't know it's pretty subtle wording issue.

E

I'm fine leaving it this way. Okay, I.

B

Guess, as a plug-in like as a plugin, an author I'll, read this and I say: okay, I I could return fail precondition. But what are my other choices of what I should? I can do.

D

Okay, so it sounds like we should put that if plugins like, if it's supported, then it should like, if plugins just a plugin may support or or should support. I don't know like.

E

Yeah, I guess you want to move the may to the the supporting of the different volume groups and then say: if you don't support it, then you return failed precondition. This is your only option. If you don't support it.

D

Yeah, I think I originally had more like what say more verbose, putting wording there, but I kind of reduced it down to so we are saying that, uh okay, if like just I want to, if plugin we could say if plugin supports note, publish.

F

D

Sorry, okay, so so we are saying that if again.

D

Plugin may support.

D

I think I have to think and find the right wording for to express the successful condition I mean.

F

You could say something like.

B

A plug-in may support this scenario and if they do or or if they don't, then return failed precondition.

B

F

Should the ceo do something with that error, the failed precondition.

B

D

Yeah, the in this case, like cu, cannot really do anything like the error. The recovery says the the recovery says. The caller must not retry until like. If you read it says, color must not retry until no publish volume uses the same value as the one that was used during note stage, but but in real world. This just basically means that user have to co, will try with exponential backup but user.

D

It just user will have to delete and recreate the workload and make sure that the second part uses the same fs group as the first one right.

F

So not redrive.

D

Yeah in kubernetes, we don't have code path where we don't read: try operations mostly.

D

A plugin may support node publish volume with different volume. Mount group.

D

Then what was used during note stage volume and.

D

So a plugin may support note polish. Following with a different volume, art group, then what was used do you know stage by room, but if it does not.

D

Support uh is that sounds spare michelle like, um or did you take a note in the.

B

Yeah yeah, I took it, though I think that sounds fine.

D

Okay, yeah, if that's not okay, so similar thing we have to put for node, publish, node, publish also like uh a plugin, may support uh node publish volume with the one that was used different, so there's a node. It's just not pollution. There's a note stage and not published not published. So a plug-in may support not publish volume with different volume amount group, then that was used during.

D

D

Okay, I could update the wording. James always said he said that he needs to review the error conditions like the last comment was that we need to review the error conditions and I need to think error cases a bit more. I'm a bit concerned about the complexity either here I would rather err on the side of caution, meaning more conservative, sacrificing flexibility or simplicity.

D

Okay, so I think it's gonna take time to him to review the the error conditions is. Could we simplify it further? Could we make it more strict.

D

I don't know actually.

D

We have two distinct errors, one right now in the in the spec as proposed it one for when staging note stage and note published, use different volume out group and one has one note published not published. They use different. One group.

B

I guess this is james. Referring to these two, um these two clauses here about having the different noun groups.

D

I think he's referring to the error error section. Actually, if you go down.

D

In the spec it will.

B

B

Is it this this section.

D

No, no! No keep going down a little bit more sorry.

D

Yeah yep yep, hey stop here.

D

Above little bit up the basically this those the table, that's.

C

D

B

I think I mean this is just this is the same as what the comment is right, but it's put into the errors table.

D

Actually, you're right, yeah yeah the same as the the description there, except put it in error table because yeah except like here, you have to document recovery and all the stuff which was kind of not described in the.

B

Okay, yeah, I think once like um I think, once you change the wording on in the description, then we can reflect it here. I don't think it will make it any simpler than I think what james was maybe asking for is that was he asking for if we can simplify the error conditions.

D

No, I think he is asking for, like.

D

So I don't know like his intent, because it was very brief- and I haven't talked to him after that. So so the thing is like, but but looking at the message, it can only interpret that when we say note publish could use different mod group as note stage or multiple node publishers could use different mod groups from each other. We are being basically like very flexible about how this is done. Like maybe he's saying that, could we make it more strict.

D

Like could we just say, okay, note publish, should you same as note stage or multiple note, publishers has to use same as they all have to use same same amount book group.

B

I guess if, but if you look at something that, like a plug-in that doesn't support node stage, um then theoretically, they should be able to do different. Mount groups on node, publish.

D

Yeah yeah it works for cifs for sure you can mount same volume multiple times on same note, with different mod groups.

G

It can work, but should we require that it must work like that was. The whole thing was that we wanted to leave the door open to if people are able to implement it sure go ahead and implement it. But if we don't want to, we only wanted to require supporting a single group at a time to be able to claim the capability.

D

To claim the capability, you don't have to support it multiple times right, you could just return failure.

G

But if I well that that's where it gets tricky is we want to make it clear that you know, even if you don't have node stage, if all you can do, is one one group that it's okay to fail here, and that was that's. That's what that's? What brings in all the extra error handling the complexity that we may or may not want us. I mean I'm fine with supporting it, but I don't deny that it's a little more complicated.

D

So we're saying that up so basically uh we are saying that for same volume and uh so there's like a bunch of permutation combinations when we are looking into note, publish like there are certain operations that is supported so the target path if the target path changes. So if the target part is same, then obviously it is supported like like you could use not different, but you could use the same point group on same target. Part, that's fine, but if you change the target path, then you got to use.

D

You got to still use the same amount group as the one that was used previously.

D

um We, if we restrict the wording that all note, publish and notes note publish, should use the same mount group. We still have to kind of.

D

Update the error handling like there's, a error already exists, error case like where say we say that.

D

But already exist error case like like: how do we I'm just trying to think like? If we restrict the amount group thing, then how do we like? Are we reducing the complexity of the error handling?

D

Then the volume already exists error says indicates that the volume corresponding to the volume id has already been published at the specified target path, but is incompatible with specified volume, capability or read-only flag. So we cannot use already exists error for publishing the volume on a different target path. As far as I understood.

D

D

So that basically means even if we restrict the node all node publishes to use same command group. We still have to have a failed precondition, error or maybe return. Another variant of already exist.

D

Does it really simplify the error handling.

D

Is what I'm asking, but there's that error handling, I think, is not simplified by doing that. Maybe what is what happens? What simply? What gets easier is like in future. If you want to restrict.

H

D

Then we cannot do it because you know like.

H

We already started with have more broader, flexible options, so just to be clear, I'm okay with it as it is, with the more complicated areas.

C

H

Think it's more important.

C

That we get it into alpha.

H

D

I I yeah, I don't know, I think I will make the wording change that we discussed today to say if supported, but I don't know the error handling. What james means by that, and how do we address that? Like the the comment that I'm referring to is the last one michelle if you pull that one and this thing, so um it's it's just at the bottom of the pull request.

B

Okay, I I think this one we might need to follow up with james on to sort of have him clarify um what he means by this?

B

um I'm not yeah, I'm not sure what which part he's referring to like, if or in terms of um what kind of changes he's thinking of.

D

Yeah I'll ping him offline, although like I'm, not sure if he'll respond in time, so yeah still in the think zone where today's already what wednesday.

D

But it could actually miss the future. But.

B

Yeah, this will be very tight.

B

All right, um any other topics, anything like.

A

This open pr by patrick right, that's also all right in uh I don't know if we got everything resolved yet.

B

Yeah, I think that one's ready to go.

B

It just needs approval.

A

Yeah, my only question is that um that existing field available capacity are we not going to do anything with it? Just leave it. It seems to be pretty weird.

B

Yeah, I think that's the conclusion. People will be confused, we we leave it and I, I guess in kubernetes we don't even use the field, um it's just kind of a.

A

Dead field or something.

B

A

Who's it now, what is user currently he's going to change it right? Yeah, yeah yeah! It is yes, basically just after this. Well, he still needs to use it right because he only added the word. The maximum minimum size right. Oh you're, saying that he's a code depending on that does not even depend.

D

Yeah, so the available capacity, we still need it right, because I thought this maximum volume side that he's adding it just to to indicate that how big a volume could be. It's like basically.

C

D

Like like, when you fit the boxes, how do you how big a box can be it's not about total capacity available in the in the back end and.

A

Also like options, if he's code previous code uses available capacity now, what has to my.

B

My understanding is that he plans to replace the use of available capacity with maximum volume size.

G

B

I I think right now so right now, like the implementation or the design, can only process one volume at a time it can't process multiple volumes. So that's why using maximum volume size or replacing it can work.

D

A

Maximum volume size can change.

D

Based on how many volumes are already created, and things like that right.

B

A

uh But then that was just basically that you have to be dependent on the newer c size back and then we are moving to beta.

A

I would probably.

B

I mean, since this is a new alpha field. I think I would probably still leave this in alpha for one more release, so we can. Let me test this out. Yeah.

A

I was just wondering if.

I

A

Sorry about that! Oh I'm sorry.

I

Hey I just uh joined it. Sorry.

A

I

A

I was just wondering if every driver can report this much more in size if they don't know how to report just report available capacity. I guess uh let's just have some questions I mean now we are if we are switching to this, but if we are still staying in afar, I think that well yeah, maybe that's bad. I think that's that gives us some time to figure out.

B

Yeah, I guess we need to see um we'll need to see on the kubernetes side um if he intends to still use this field. It might be that he might fall back to it because I think it's an optional field, so you know he'll, look at he'll, look at the maximum volume size first and if it's not set, then he'll fall back to the other one.

A

Maybe better, I think I was just wondering if actually but.

B

I think this yeah yeah, but I think so that would be like the the better way to um maintain backwards compatibility. But I think the issue still stands that the two fields are going to be interpreted the same way in kubernetes. There's no change in how kubernetes interprets those two.

D

D

By the way, the okay are we done with uh sorry. Sorry, sorry, I don't want to change this yeah.

B

I think we're done.

D

So I had this this pr, one pr like node secret for node expands volume. Let me see.

D

Let me add it or somebody's editing, it.

B

Oh go ahead, I'm I was about to type it.

D

Okay, so this was open for a while as well actually- and I think it's pretty straightforward- I mean we have to obviously do some work in kubernetes to consume it or make it accessible and we have. But as far as csi specs is concerned, it should be okay to do it right, I mean unless, like like. I think it's it's fine.

D

The problem is going to be in kubernetes like copying a new secret to pvs if it is different than the control plane, expand secrets.

B

Okay, I guess this one seems um pretty straightforward right. It's just adding another secret for a call.

B

Okay, so I think this one just needs some approval. um I don't think there would be. Are there any like outstanding items on this? I think it looks like. The only thing is that we need to figure out on the kubernetes side how to plumb it through, but I think that can be resolved separately.

I

Yeah, that should be pretty straightforward.

B

All right um so yeah we have these four pr's that I think um most of them are ready to go. I think the only one that is needs a little bit of more discussion is the volume out group one, but the other three, I think, should be ready.

I

And uh do we want to hold off uh for the volume mount group one or proceed.

B

I guess I think we're mainly waiting on um james to sort of clarify his comment that he had, um I think ma. If you can try to ping him today. That would be great, but let's set do we want to set a sort of deadline for when we need to get all these pr's merged, and so we can do a cut.

B

I

B

I

Yeah, I think it takes at least. uh Let me check, I think it's at least three days, uh so we gotta do it like tomorrow. If we want to be on track for a uh getting this ready by monday.

B

Three three business days or three days old,.

B

D

ah Well, I think, james by the way and the one more thing was like the vr. Our bills will be broken on on golang 1.16 by the way, but I think that's not important right now. We have to some point in future: fix that.

B

As long as we stay on 115 for now, or we should be fine right.

D

I

Yeah, uh okay, so it's uh two business days. So if we do it on this afternoon, we can cut it by friday afternoon.

I

If we do it by end of day today we can cut it by friday end of day. If we do it tomorrow, then it'll have to wait till.

B

Monday, please is on the.

I

Tuesday, yeah tuesday yeah.

B

That's so I think if.

I

G

I

I say: let's uh get these pr's merged today, get the rc cut end of day today, target the release friday, and then you got monday tuesday to see if we can make the updates and in the meantime uh we can uh use the rc build for testing and getting the kubernetes changes ready.

B

I

And I guess that means a month. Just shoot me an email by end of day on kind of the status of the mat, one: okay,.

B

All right sounds good. um Any other topics.

B

All right, then, that'll be it for today, thanks.

I