Kubernetes Storage SIG, 18 Jun 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Kubernetes SIG Storage - Bi-Weekly Meeting 20200618

Description

Meeting of Kubernetes Storage Special-Interest-Group (SIG) Bi-Weekly Meeting - 18 June 2020

Meeting Notes/Agenda: -

Find out more about the Storage SIG here: https://github.com/kubernetes/community/tree/master/sig-storage

Moderator: Xing Yang (VMware)

A

Today is June 18th 2020. This is the kubernetes six doors meeting. This meeting will be recorded and uploaded to YouTube.

A

First, we'll do one da 19 planning so the first item. This is an online offline resizing. Six issues is her money online.

A

A

A

So I think come on, so we had several meetings talking about those issues and hammam is working on fixes for for them.

A

The next one is recovering from resize failures: again himand see there, there is a cap that is merged and he is working on a PR right now. So.

B

It's PR is out I'm helping review that one okay.

A

Right so I think that's that's in progress and on schedule. So if I think he said he's going to post a PR for the talk which is required for her as Friday, you think, but why should I think I need to add this sorry I forgot I mean at this dad a new in column here.

A

Next one support can analyze the CSI note plugins in Windows. Do you have deep poor Jing here?

A

Let me show you know the status.

A

Yeah, okay and I know anything about this. I don't know. I was in update.

A

Snapshot: okay, so we are still working on back fixing and so that those are in progress.

A

Next one is now recursive warning ownership come on young watching.

C

A

Anything about this, where it.

C

Just say no okay Emily's, not here so where the status is that we are waiting for the next item for the selinux okay review, because maybe it's going to be merged better.

A

Also expect you energy soon.

D

C

Like the fs group thing, the recursive ownership, it is merged this alpha in 118 in 119 yeah waiting for SELinux. If it's going to be merged. If F has a book change or not,.

E

Okay, little okay.

C

Okay, this is the maximum I can do. Okay,.

A

A

So this one should not be this, so this one should be started well.

C

It's locked, it's blocked by the rest, knocks. Oh, oh.

A

Sorry I think I guess this part was not blocked. I.

C

See rocks I have API review, I have AP IPR I, don't have API review I'm great I'm talking cynics ready, okay,.

A

C

A

Already right: okay, yes,.

C

A

C

I have implementation, for a couple of voting begins just to prove that the API make sense and how the implementation will look like.

E

A

A

Next one is a Linux recursive permission Thanks. What.

C

I told you that's they play of you. Okay,.

A

And this one five-nation has only four windows: is this still not started.

E

F

E

It's not started.

A

Next one, this one says: dump file Commission being projected source; okay, okay, so I think this guy's already merged sit down. We don't need anything.

A

Seaside entry would only handling is that is anyone working of this humble side? Do you know anything about this.

B

I'm sorry, what was like them this.

A

Is the CSI entry read-only handling? Do we have any update or.

B

A

Extra I said issues related to issuing assuming volumes are Mont points and the Jean.

A

Anyone have any update on this Jean Andy.

F

So I haven't need to understand the. What exactly this shoe is. There is a PR and working on to relate to the CSI amount point mm-hmm, so I think I should be able to finish it so I finished, but I just need to actually address the comments, but I don't think it's exactly the same thing, but.

F

A

I'll just put down your take a look: okay, sure: okay, thanks.

A

Next, one is the storage capacity tracking okay, so this is one that is magic here.

A

Okay, yes, Michelle here is just not hear anything. I think.

C

That's a cap, open, yeah, yeah I'm, not sure if you yes, Phoebe I, think.

A

The P R is there that I know that I actually reviewed this once so PR is PR is loud.

C

Margaret, oh.

A

C

You I think I.

A

Think Michelle says she scheduled a just scheduled a review meeting. I believe that's what she said. If you had a meeting.

A

So I know that's what she said.

A

Next one is PVC inline aphromoo water, so this one is the one that young you or helping the being rain. Is this one? Yes,.

C

There is API review in progress and Clayton is looking at it. However, Crichton is very busy, so he's.

E

C

To catch and this one may be very complicated to pass through API review because.

C

They want something more generic.

C

A

Everything to be combined the the existing one and this new one, it's great.

B

A

B

C

Put yet off until beta.

C

Doesn't want to look something: that's throwaway, API, okay,.

B

It's a good point. That's just done work. They.

A

Don't like the like the want to do this in steps they don't want. They wanted to be done in whatever.

C

The credit yeah I that Patrick has is pretty extensible yeah anything there. So if we convince Clayton or Jordan or this it has the it moves the right direction, then it's fine. Okay,.

E

A

Thanks, hopefully, I will be persuaded, oh okay, so this one's writing this so I have actually have not really so I thought about this. Don't have anything concrete yet, and so it's like you know, I was in progress.

A

Or in group API, so this one we just had a title review me design review for this meeting. So we will have another meeting, a follow-up.

A

Next one is the csro tree move fast driver, so this is one finish down. This one is down. Okay, so I think sit down. Then the next one is to see how the tree move. I, scuzzy driver.

A

Startup I need to do yeah.

G

I need to follow up with I need to follow up with folks who, who were reaching out to me about helping work on this driver. Okay, yeah I, think. Basically, the code has been updated, but we need to do to fit and finish with actually like building and releasing images for it. Okay,.

A

Thanks sure next one is to CS a fibre channel driver episode, a brigade and then the flash adapter. Those bows are going to be deprecated. Yes,.

G

It's on me to just sort of archive the repos and mark all the Reed knees and stuff as deprecated. Okay,.

A

Thanks Nisha next wise move out a glass surface, parishioner humble humble there or young. You know the status of this. This is a cluster cluster EFS provisioner, no.

H

Don't know okay, this is Karen here, I. Think the initial PR caught moist, okay.

A

Okay thanks, but there's more than more work to do is now completely array, so there are more.

H

Like it's not complete, I think this okay Automation for images being set up.

A

Okay, so the I think the next two is you currently. A first name was yeah, so.

H

The for the NFS provisioner, we have the initial P already I. Think this one hiccup with the CLA commit I'm working with the intra-group to get that sorted out.

H

We'll get that moist, okay,.

A

And in a second one say, is the same or different. Your client same.

H

It's a different one, so I raised it tuft PR right now, working on it. Okay,.

A

Okay, thank you. Next one is the application of kubernetes incubator, external storage,.

A

Matt is not here, Michelle. Do you know the status of this one? What I'm waiting for other repos to be move around first before deprecating yeah.

G

I think we're waiting for the NFS wants to finish movement, and then we can also clean up all the issues in the old repo move them over to the new repo and such I do see. There are like there are still lots of issues and he ours being opened against this repo I, don't know. If Kieran you can have a chance to like go in and sort of start cleaning them up.

G

I guess it would be hard because, like until we have the new code available and merged, it will be hard for people to like to tell people to say here. Make the PRS new bebo yeah sounds.

H

Good Michelle I'll keep it watch on them.

A

Okay thanks next one warum snapshot, namespace transfer. Do we have Mike here.

A

Mm sad: do you know any update about this? One.

B

I have not seen anything new here. No okay,.

A

A

Okay next thing is watering house, so we do have a walk driver PR, that is interview and then the avoiding order house, the controller agent.

A

A

Next, one: okay, so this one, we moved out thence genetic data, popular object, storage, API, chair for John.

B

I

They're holding.

B

Regular meetings, including stand up I, think the latest statuses they were going to start working on like a 0.1 version of the spec. So on track.

A

A

Okay, so this one- probably we don't have any update this one. It's a CSI, Ephrem or walling beta, fixed box. This is awesome. Patrick young do you know, is this one? Is there anything going on with this or we're just waiting for the new one to be done? First,.

C

A

You know one thing: next one is FS groups for in CSI, no.

J

Question there's some open PR for this. There aren't compatibility test values that I'm still addressing okay.

A

Thanks Christian, obviously it's a migration TV only there. Yes,.

K

Yeah, so we have much the PR for the community said changes. Currently, it is much with teeth which are get alpha after the testing is complete, along with the CSI driver, will move to or bump it up to the beta right. Now we have one concern from the community regarding the way we are asking our customer to upgrade vSphere 27.0, you one, so we have reached out to like the PM's in the VMware, and we cannot back port the api's that we have added in the 7.0 u1r to the 6.5 and the 7.0.

K

So we customer need to a crate or 27.0 you one. So for that we will publish the message for all of our customer.

K

That is the overall status.

A

Okay, so sad do you, we should probably take a look of the message that we want to send out. So should that, should we put that on the /n right talk about that later or how do you want to yeah.

B

We could do it offline, okay,.

A

Thank you, yeah.

G

A

Also, probably loop.

G

In folks, from unsick cloud provider too, because I think this has like broader implications and storage, yeah.

A

We already talked to you and you he is in the cloud provider side. So yeah we will. You know we can talk talking about this here, so he he's aware of this issue at least okay yeah. Let's talk about this more offline. Next one is a GSM aberration. Do you have any dealer nd here we showed you know any status for.

G

This there's, you um he's been sending out PRS, but I haven't um I, haven't been syncing up on the on the latest status, but I think he is actively working on this. I can try to follow up with him.

A

Thanks sure, okay, so this one is crossed out mmm and then we have a WCS emigration, not started. You have there anyone working on this side and have you heard it anything from Mehta but have.

B

Not been keeping an eye on this I'm, not sure /, okay,.

E

L

Someone to tell me what's going on because I've not been paying attention what.

A

Is it Matt yeah.

L

A

You don't know this, it obviously is a migration I think it's a you want to bring this to you beta in shares this one saying that it's.

G

Already data I think the ask here is to see you know continue working on fixing bugs and stuff so that we can get to GA in the next couple of releases.

G

I guess Matt: will you be able to sort of um follow up and and see if you someone on your team or you can own the AWS migration feature.

L

Yeah I have I'm supposed to own it. I just have not caught up.

L

G

Sounds good feel free to sync up. You can sync up with me offline and I can sort of like it. You updated on the current status of stuff, but basically we're still we're still going to overall stay in beta. But, like we're gonna try to see, you know, continue stabilizing the feature and and hopefully trying to get some cloud providers to go to GA and that the next please do.

A

Okay, thank you, madam Michelle. The next one is almost Axios emigration. John you want to do is focus on this.

D

I have not been able to get any update on that one. At this point, okay,.

A

Next one is Oh, South FSF are BDCs emigration, humble it's probably not here. Do you have any update on this one? It's no update, I.

B

Haven't seen anything now.

G

Um-Hum ball was talking to me earlier, I think he's actively working on it, but he's not sure he'll be able to do an alpha by dis release that he started work on it.

A

Thanks Michelle next is CSS scalability, immutable secrets and fake coughing. Okay, this one in system, the next one is CSI. Sick, apps, PVC, created by stiff aside, will not be Auto removed. Its KK here, yeah.

I

So we had a discussion in see, gaps me and Matt with it. There are a few things we are discussing offline on this and we will come back with a proposal on what direction will move and probably raise a cap so I'm, hoping that in a week's time we'll have more updates.

A

A

Next, one is for extension for superset yep.

I

So yeah so we're like a basically riester kept for this ken from see gaps reviewed it. He.

I

There are few updates to the cape itself which I'm working on, but there were concerns regarding how the user errors are propagated, with respect to volumes shrink and how we handle it when we do a shrink from stateful set- and you know like related stuff. So it's not.

A

Supportive right because I think yes,.

I

So we need a way in which we can. You know express this properly when a stateful set is updated back to the user. Okay,.

A

I

The idea is that once code freeze is done me and him and we'll have more discussions on this and I'll update the Cape as well and then work out with Ken and move forward with it. That's the current okay.

A

And you are aware of this other cap that hung on this working array. Yes,.

I

Yes, a lot of food.

A

I

Yes, yes, we talked about that and he said, like after code freeze, we'll have more time to discuss. Okay.

A

Thanks: okay, next one execution hook for became snapshot, so this one we have a container notify proposal. We discuss about that in with the signal they have some concerns for adding the cingulate so we're trying to address their comments, but I think there are enough. Ceo concerns, though, still still being discussed.

A

Next one is this: you dose months, leading to new ripple. Do we have any owner of this start? It is a started.

A

It has Sade young, so it says there was the issue to create a new seating issue.

A

B

What was the Oh.

A

Utah's mountains, you and you repo, oh yeah,.

G

That's some progress: there's a the staging repo has been created and there's a PR out to move move the mount stuff to it. I think it just needs approval from someone top-level at this point, so, okay, I am pinging Tim. To help approve this one. Okay,.

A

Thanks, okay, I think: that's all we have in the splashy. We do have a few items here. Oh.

M

Yeah, so we recently got someone from Google working on the one snapshot Andy. He is also in the core. Okay, he started to if you've started working on items to help a woman start try to to g8. Okay,.

A

M

A

Yeah we well so this.

M

Is it this is all.

A

In the same well, so those are tracked here: snapshot fix issues so and then for individual ones. We have that we have that tracked on the project board under external snapshot. It's time snapshot her.

B

M

B

So the individual issues you can track on your own. The purpose for this meeting is to kind of track the high level for the next kubernetes release and in this coming release we're not moving to GA. So this tracking bug fixes is fine, so for the next release, what we'll do is have a separate standalone item for GA gotcha.

M

Alright cool, then our just used in these natural go to track. It. Ok.

A

M

A

So we have there's a design review here. First Shahar see the driver needs a way to get parts service account token Shihong.

A

N

A

N

Ok, so for security reasons, the sis driver it's a way to get it. The patch service count. Token and I already have received many comments from general and Michelle and the Steve and the on this cap there are there are. There are two remaining issues.

N

One is the 22 weeks and days to right now this only at the top can only pass to the node publish volume, whether we need to extend it to the two other CSI cost, like controller page kind of stuff. Am I appealing I, don't think this is related to past operations right if I understand this, the, for example, the notice stage stage volume call that that calling amongst the warning to the notes, global power system right I mean I'm, not quite sure how it's related to the parts.

N

So this is one thing and the other is the altitude with the failed free month.

N

A

Michelle, since you have been reading this.

A

A

C

Doing no publish only okay I would understand this to be passed in CSI in line volumes when we kind of cop a secret or config maps, or things like that, but once AWS jumped in and they want to authenticate mount using.

C

This concept, like, what's the difference to other volume, plugins that they may need note stage I.

G

Guess the thing, though, is that the service we're talking about the service account of a pod and at the node stage time we don't necessarily have a pod right.

B

B

So you could end up with multiple pods, so you wouldn't want to get into a position where you have to pick one.

C

Stage is not supported and will not be supported because of multiple ports can have interests in the volume and so on.

G

I also asked Michael from the fort to clarify from like the ef-s side. If, if having this at node publishes sufficient or would they also need it at a volume, create volume level.

L

G

L

Represent him but I, don't know the answer so I'm going to ask him I, don't think we care about pay volume, yeah.

C

L

C

L

The controller I don't think we care about people and we only care about having the application mount its own secret. We don't care about great long.

J

O

Is there no work around for this, involving just writing a grenade controller that gets the information I I realized that may not be the right long-term solution, but in the short term like does it even work, and if not, why.

N

Went to the question sorry, the.

O

Question was: can this be worked around in the short term by just writing a controller that gets the required information at the required time, because.

N

There is a short term: one dead out already been put him in the in the sacred stores: si driver data, the ground, the token request permission to the service candle of the driver pod and that they can create token there, but this requires some other changes on the automation. Clocking is to validate that.

N

The token request is requested by the sister on behalf of the pod that the sales driver is mounting the warnings to so it's kind of messy to implement a kind of rules like policies to do the validation and also there are many other sis drivers requests this requests. The service can token so I think it's better to implement in the kool-aid.

N

O

I only asked the question because if you were gonna say you can't do it all.

L

The same problems.

O

Just for the the court as well okay, so it's just messy.

G

Okay, there's. The second question you had was about remounts yeah.

N

If it failed as that by Jim, so so I look at the code with the greyman failed, the arrow will only be logged in the reconciler right and but it's the same as the first mountain right. The first man also I mean the first amount that the same thing as the very mound.

B

Yes, I doesn't have remounts enable to that so.

G

As part of this proposal, it's to enable it for say.

C

Okay, well, if the first mount fails, then the pod is not started, but once the put is running and Ramon fails, then we just log it which is fine. As long is it safe to do like I'm, not sure I fully understand this token expression in this stuff. Does it mean if the token expires, then the port should not have access to the secret data and we should block it somehow.

N

So so the so, for example, the what the water has the secret or which requires an authentication, and so so the sister EVER will request the pod service can token so that at the subscriber can use it to do the authentication and if the the token expires, so basically, the secret also has the detail it will expire. If that expires, it work will require a token and the token will also expiry sometime like 10 minutes. So this it means if the secrets expire, they will require the token again, but.

G

I guess does the is it the CSI driver that like removes the files or something when it's me.

N

I, don't know yeah I, don't know this I asked questions in one of the thread of the comments. I think Steve has a best understandable because he's the one that requests this victory and in another PR the issue.

M

C

We can either leave it to the CSI driver to clean the expired secret, which is I, don't know, doesn't help much because the port can have it cached anywhere or we can kill the port and if killing, the pod is the right way to go and need some changes in cubelet.

C

N

I mean if the city I think, if the secrets expired, there should I mean there should be. Some error returned if the Pats try to access that ones right, otherwise that there's no but.

G

I guess that would be that that would have to be enforced by the CSI driver I, think, because we don't really have control over the IO to the volume yeah. Yes,.

G

I don't know if it would be helpful to like if the the remount fails to add an event or something I don't know if that's gonna cause all of a sudden cause a bunch of events spam. But that's the consideration.

N

Yeah, how does the current Ramona works for the the Dumber API? If that a dream on a field? It's the same thing right, there's I,.

G

Don't think we have an event for it. I think we just log on message: yeah and basically we're data is still yeah.

N

If we do something yeah so busily for the downright guy still, but for the sister overdue definite things, what do you make sense to keep this constant.

G

What do you mean by constant like.

N

In how we deal with the Riemann failures, because for the dollar.

O

N

Yeah we allowed it to be stale um because the ribbon failed but I wake you the pad with the Rimmel fielder for the driver.

G

You mean you want to keep the behavior consistent between like downward API and the CSI driver. Yeah.

N

Or we implement, like, like Jen, said, maybe cute code, should we do this? The cocooning pad strategy to the tumbler API in the Raven appeal.

C

There is a pretty high chance that the data available into the volume are still correct, while in door open case, you are sure that they are wrong. That's the difference. Oh.

N

C

Don't insist on killing the port like I'm, just curious if from security perspective, is it correct that the Lord has access to the data me know that they are expired.

C

N

Yeah, this pin whether the six which are accessible if it expires today for their water, the water, so crystal yeah I think this can be full up for you for Steve. Coming out my comment: there yeah.

G

I think it would be good for her to answer this to follow up with all the different CSI drivers. We know about that. Want this feature so like the Secret stores, I, don't know if the manager has something similar, so just yeah.

O

G

Up with all of them and seeing what their desired behavior would be. Okay,.

N

Yeah, thank you.

A

So this would be a 1.20 item right because this is the cap is just an idiot over the past.

C

Often we expect that this token needs to be renewed, because I think that the scalability team should be involved.

N

How often the token will be.

N

It's like 10, so I would say.

N

So the so the expiration time for the Fotolia units at 10 minutes and the completed will will refresh the token like in the background like at the 80% over the lifetime. Basically, it's a minutes. It will refresh.

C

N

C

Uses the same framework as the tokens that we already have for yes,.

N

G

Csi drivers need to make any API calls.

G

Api calls like um I, think you were mentioning somewhere that the CSI driver needs to make like a API called to validate the audience. This.

N

Is Trevor were not the the storage provider will.

A

Api call you anyone who needs a vehicle yeah.

G

I mean kubernetes api call yeah domain. The main the concern is that, because this, the the current remount thing gets called very frequently and if every remount request will invoke a API call, then we can hit API limits pretty quickly.

N

The IP animate on the safe driver.

I

um Yeah but wouldn't climb caching prevent.

G

I

G

Mean I guess so like I'm, not so that's what I'm not sure about like will this on every node publish like is the CSI driver going to end up making an API call or doesn't.

N

G

N

N

Think it's I, don't know who sent the token review to my understand. It's the.

N

I'm, not quite sure, sorry like, or this driver or the.

N

The the storage provider I look I would look at the documentation of the what they say. The the token reveal will be sent. I'm not quite didn't know like. Oh, isn't that a foundation.

N

Yeah, if there's a API name it under this driver, I think I need to research on this, as well as part of the cap.

A

This requires a CSS fat change.

N

Yes, sir sighs driver, spectrum's I mean.

A

I'm talking about the directors so t we need to add like a new RPC in the CSS bag itself. Well,.

C

That's a question because we call CSI spec like assume that we cannot publish only ones when we need to mount. But there is nothing about periodic calling.

C

This is kubernetes specific anyway,.

G

E

Guess is a question.

G

I have one random question format for, like the.

B

Esri I think there is one more item on the calendar on the agenda.

G

I want to make sure that.

B

A

Has time maybe.

B

We could do a follow-up meeting yeah.

A

We have things.

A

So I know sad: you have this uh flexlearning driver. You want to talk about that. So you should we go to the data set like psycho yeah.

B

We can skip that folks can click on the link if they're interested- and we can talk about it next time. I wanted to give give time to the data side, life cycle framework, I'm, not sure they'll have enough time, but if not.

N

We can move.

B

Them to the next meeting.

P

Yeah sure that's Yanis! It's it's fine by me. Okay,.

A

So you want to talk about the lesson and then maybe we can talk more. You can keep like a for review next time. You can just.

P

Specialization next time, yeah.

A

Do you want to give her like a overview of this? The science.

P

I can look like for few minutes. So yes, so basically it's a framework that we created recently IBM in research in Europe.

P

So basically, what we're trying to do is to introduce a higher level abstraction of from the CSI and we're looking to introduce the data set concept which basically, a new CFD that points to a remote data source, whether it's s3 or NFS, and you know we're adding more protocols as well, and it's we're basically targeting to improve the usability so for users who are not familiar with CSI and the different storage classes, we're looking to give them a high level of suction, so they can use data sets in their pods.

P

So in this, in the same, at the same time, we look to improve performance, because if we introduced this this right on using this abstraction, we can offer transport casting, meaning that the user would be declaring one endpoint, but actually on the under poles. That would be using the cast version of it and yeah we're looking to get more feedback. You know have more discussions with the open source, community and yeah. Any any feedback is welcome and maybe next time I can give a demo as well. I, don't know.

P

What's the time schedule for this yeah.

A

Sorry so so we do you with like schedule several meetings to review this, or do you want to have this as part of the next meeting? What do you think.

B

We can probably do it as a part of the next meeting. If you do 15 20 minute presentation, I think that'll be enough time wise to be able to fit it in okay,.

P

So if you want to have a look on the wrapper before that until until then to cover more questions, maybe that you know you get offline, it would be helpful as well so yeah, okay,.

A

So what projects whoa look at this one? First in the next meeting, all right, the status sounds.

B

A

There was also so I think we do have this a object bucket probationary proposal, I'm, not sure if we.

O

A

Following that, and then we do, we have weekly meetings. You may want to participate in that and see you mean you know you can get input, that's actually coming right after this one.

P

A

All right, thank you.

D

Ding, do you have a second to go back to the planning spreadsheet.

A

D

I just found a reference on the cloud provider OpenStack, the guitar provider, OpenStack. There's a note there that says that the cinder standalone provision of manila provision, a cinder flex volume driver were removed since 1:18.

A

That's I think they're talking about in the entry entry winery that'll be removed. Yes,.

D

That's what I'm reading it is and cinder CSI driver is provided by project called ember. Are you aware of that? Yeah.

A

D

A

D

That there's anything else outstanding there. Oh, this is line 35, okay,.

A

You can you provide a link. Well, what is that link ready? Are you looking at this because ember embrace this? That's like the CSI driver, but we we don't know if they about the I think they already have the translation, far I'm, just not sure. What's the next step, I think they already. This is already beta. So we don't know who is working on finishing this and Breen is Gigi I, guess that's the question so I know I know who is will come amber but not I. Suppose.

D

The best links in the chat, okay,.

A

Thank you alright. So.

A

Yeah so okay well yeah this one here and then take a look later.

A

Two minutes left I. Think TV I already give an update on that. I could see a celebration, promise, fear and so KK has this I think we won't have time I see you have some slides. They are a thank-you thing. Oh you think thank you John KK, so you want to talk about this being the next meeting. You would just want to give a quick overview of this. You have a proposal yep.

I

So right now we don't have a way in which we can tune the performance of persistent volume at runtime, and this presents some scenarios for users which we could potentially improve by providing a genetic framework with kubernetes, and the proposal is for community input. Us are there other. You know, folks, who are also interested in this kind of approach, and there are a couple of approaches which we have put forward to see.

I

If we can, you know like come up with the consensus and work out something where dynamically these performance parameters like I opsin throughput, can be tuned without really having to recreate the PvE and.

A

Basically, it's making war like a sighs. It's just like yes,.

I

So I'll put it up in the next agenda item also, if you know that that presentation is that I'll write, put it up right after that and I'll send the slides to the sip storage as well. So if offline people can discuss that'd be really great.

A

Okay, thank you. Thank you.

A

Uh-Huh, okay, so I think that's we have one minute. So sorry you want to talk about in the Flex, folio or or.

B

Okay, I think we already we can. We can just move it to the next meeting. Okay,.

A

Sure all right, let's see for today, then thanks everyone bye. Thank you.

A