Kubernetes SIG Node, 26 Sep 2023

Previous Meeting

Next Meeting

⏯

youtube image

►

From YouTube: Kubernetes SIG Node 20230926

Description

SIG Node weekly meeting. Agenda and notes: https://docs.google.com/document/d/1Ne57gvidMEWXR70OxxnRkYquAoMpt56o75oZtg-OeBg/edit#heading=h.adoto8roitwq

GMT20230926-170638_Recording_4096x2304.mp4

A

Hello, hello uh today is September 26 2023. This is a signaled weekly meeting. Welcome everybody. We have very long agenda today, uh let's jump right into it. uh First is Philip.

B

Okay, hi everyone, yeah I, will start sharing with the second.

C

D

B

D

B

uh Okay, uh so I'll press give an introduction into the program we are trying to solve here and then I will showcase the.

E

B

Ask for a question or ask for your opinion. uh So today the state of uh the no draining is that we have this like top-down approach, where you are usually run Cube skill drain or if you have some other.

D

B

Or of for example, some note autoscaler or something you can have other options as well, but we just uh we have this top-down approach where we ask all the Bots to to be elected and like without any regard for the underlying application.

B

So we just hope that the Bots get evicted and we can do dream successfully and remove all the pods. Unfortunately, this does not work in every case. We have also vdbs and some applications may wish to stop the drain and block it under some circumstances, and some applications live with with the disruption, but are not happy about it, which you can also see uh I in the motivation power section.

B

There are linked issues where people have problems that, for example, they're single replica application needs to compare with the disruption, uh and since they don't want to um block the train for the administrator, then they are not using pdbs and and their application is disrupted and they they do not wish to use high available applications with multiple replicas and for applications that do not want to be disrupted.

B

They use their pdb and they hard book the eviction. So they leave the decision up to the admin. To actually will first try the addiction we are talking about the addiction by the drag cattle drain, which uses API initiated eviction, and if the admin sees that uh not all Bots are.

E

B

Then it has to dig down uh into the application to see if it can disrupt the application. Even though the pdb is specified, which is a manual process, and they.

E

B

Like to avoid that, so with this in mind, we had some discussions in the background and we are proposing to uh to have the new process for the Deep cattle drain or any like type of drain, uh and this we would like to be this more declarative and give more control to the application.

B

uh So we are proposing to create a new API called node maintenance, uh which uh would usually a Target or not like we started for now. uh We have uh just this simple API, where you specify a node, which he wants to drain, specify a reason and uh I will talk about the stickers later, but usually you start drawing as usual. You might do not unschedulable, and then you create the non-maintenance object. uh You would like to implement this in keep cattle drain first and other, uh like drainers, can implement this as well.

B

By creating this object, this will get picked up uh by the maintenance controller, which will, according to the maintenance object, will uh select number of pods which are on.

F

B

And uh will set a new condition uh calls targeted by non-maintenance on the on on each board in this in this node, and then it is up to the application. uh We would like to first start with a deployment controller and the application can decide how to delete these parts from the from the node uh or how to move them. So in case of a deployment control, we have a Max search field.

B

This controller could actually create a new pods on another node and once the spot become available, then it would uh Delete uh to replica set. It would delete the pods that are on the maintain node, and the advantage is that we can. uh You use the wish of the application which specifies the max search, how many Max, how many search posts we want to create.

B

So this can work also for single replica application, for example, once that is done, that you you can battle drain, would work as normal, but this time, if we see the support for the node maintenance uh objects in the cluster, we will try not to evict the boss, which are supporting the search and we will evict or delete the post. As usual,.

B

So this this is the proposal, and also one of the advantages is that we can add other features in the future. So, for example, if we would like to uh this, this node maintenance controller, put, for example, in the future, select also bots of demon, sets or According to some priority schedule or uh for some priority list and be good to do a like, gradually remove the posts according to the user's wishes.

F

How does Cube CTL drain know how long it has to wait for the controller to to do its thing like, for example, create search, pods and remove pods, and so on.

B

uh So in this scenario we would uh just we would we.

B

We would expect that the Bots uh or the controllers of the pods with resolve this without our our interaction I suppose later we can add some Flags to modify this Behavior, but it would be up to the application first, so you're in wait for.

F

Some time or how will you know that the application is done basically before you proceed to the next steps, where you say: Okay filtering, mirror, pods and PODS of workloads and support search. So when do you go to that stage in the process, so.

B

At first, like the the drain would work same as today, but we would skip these search pods. So we would just in parallel, evict other ports which are not covered by Max search.

F

So my question is like how do you detect that the loop in the middle is done and it's it's okay for you to proceed to the next steps.

B

uh Okay, so this Loop uh runs in parallel with this Loop, so this these are not uh dependent on each other and we also have uh the status, which should be resolved by the Knight by the maintenance controller, which would strike the targeted posts which have the condition and the admin could actually query this. This node maintenance objects and see how many Bots are remaining uh for until the drain is complete.

B

Yeah so also I would, like probably people might be interested why we chose to create a new object. We are discussing this in the alternative section.

B

We also thought about introducing uh like this API into the node object itself, uh which could be also done, but since we are seeing the possibility for extending this API in the future and moving uh skip cattle logic into the controller it, uh it seems best to us to create a new object and do not put this internet out object.

B

um So uh so I can ask for what are your opinions on on this proposal and, if seek notice? Okay, with this approach.

C

I had a quick question: if, um if does each controller need to integrate with this, like, for example, let's say I'm building my own controller, that manages pods, not a deployment or an event set, or something like that? How would I integrate with this do I need to integrate with it or what, if I'm, just creating bear pods or something like that.

B

C

B

So, uh like you do not need to integrate with this, but you will not get the benefits of that and.

E

B

Are also proposing that, oh sorry, uh we are also proposing uh to create a search API uh which could have this shape or another. It was not discussed yet, but uh this this could mean that uh other, like CRVs, could uh implement this API and advertise if they are, if they support the search and they could actually uh migrate or delete the posts themselves.

C

But if the controller did not uh integrate with that, would it just kind of perform like it did today, where it just uses termination increase period and just kills the pods uh there.

B

Or it could behave the same as today, so that means uh the the couple drink would would try to evict the boss and if you are covered by a pdb uh it would not uh evict the boss.

B

E

In this process, we are also like.

B

Proposing to make small changes to the disruption controller and uh to change the counting of the pdbs, so uh these applications, which are supporting the search, uh would actually the pdb would also could cover uh the search Bots, which is not possible. Today.

G

I have a couple questions: um what's the proposed rbac model for creating node maintenance resources, and is there anything you're proposing to do to expand this API? To um do things like only these numbers of concurrent nodes can be under upgrade or only nodes in a single zone can be upgraded. No parallel, Zone upgrade no parallel. Zone maintenance, um one thing that just uh maybe that the Arbok question is probably my first question like um and then the second then afterwards was like.

G

What's the life cycle of this resource, when is it purged that was a garbage collected.

B

uh Yeah, so the arbuck model uh I'm, not sure from what direction you are asking, but this uh like the the are back for for the no maintenance uh could be uh like, should be accessible also only for the administrator. So it's actually the creation on the deletion of the non-maintance object and since we are uh uh putting the maintenance condition on the on each part, only uh the controllers, which you would like to support the search on and have to check their posts.

B

So they need to not don't need to watch for the node maintenance, but this also means that uh some other processes are controllers uh that want to detect the general maintenance can get. Air back, are back for New maintenance and can observe it and can react to this in in, uh like in.

E

B

Way they desire.

G

Yes, um today, if I want to call keep control drain, node I need privileges to get that node resource um change, the schedule attribute of that node resource and then I need privileges to delete all pods, probably independent of a node like we have no way to my knowledge right now on an orback level to to restrict my delete, pods rights to just pods on that node and that node online um I'm curious.

G

Then, in order to have privileges to create a node maintenance record, do I that's kind of like an indirect privilege escalation action today. So would we map that traditionally to you, if you can create a node maintenance resource? It means you should have read rights on nodes, and then you should have delete rights on all pods in a cluster or is there anything we could do if, if we were to pursue this to make it that you could restrict your rights to that which is resonant to that node and no other.

B

Yeah, that's a very interesting question uh like today, uh since all of the actions are done by Cube control drain, you need to have the Privileges uh to actually evict or delete and to uh schedule.

B

And we are also exploring the possibility so that the unschedulability could be set by the non-maintenance.

E

B

So we would have to explore this direction if we could, uh because we could also move the pivotal drain logic. You could.

E

B

Everything to the non-maintenance control and have it all declarative, so we could try to see if we can map the are about privileges to the, not maintenance object. If we would like to select only samples on some notes, I.

G

Guess some notes.

B

G

That Spirit, like it's a client-side choice right now, when you do the drain action and in some ways it's then left up to like an operator practice. But the moment you, you would Advocate to present an API in the cluster space.

G

um I start to worry about like privilege, escalation flows or like the disruptions one can cause, and so that the thing that would be really appealing to me personally on this is: if I could um restrict the set of light rights. You need to give out to actors in order to do maintenance even on a node by node basis um and uh absent.

G

That I'd be a little a little cautious about this, just because I'm unsure what type of chaos I can then cause in environments with this like where you might want to restrict the ability to do node maintenance by node role or by no tank or or the security domain of the node generally um so either.

E

G

I'm just I'm wondering if we can explore some of that before making a a final decision on this proposal, because right now, I could see um maybe some unintended consequences.

B

Yeah, that's a very good remark and we should certainly explore this uh before we are. We have the final state of the cut.

A

Yeah in interest of time, unless there is any other immediate questions we can move forward. As I said, there are many agenda items today. So next items I guess review is a cap as a tradition.

D

B

So you just want to add that uh that it seems that nobody had many big objections against this API. So I would like to ask people uh to do or if uh to do some reviews if they have time- and we will also be uh bringing this to other six uh sick, apps and six CLI and six scheduling to for to ask for the feedback. So this is work in progress and it's still a way to go. Thank.

G

You my only time I I, think my objection is a is, is a little stronger in the sense that um maybe a follow-on activity is to like um see what, if we introduce this API, what is the least restrictive, R backs, we can do and that be an argument for this. Api I worry about the presence of this API, causing um very bad potential actions and clusters.

G

If I can't restrict the creation of these node maintenance resources or or what happens if they're acted on concurrently or but, and so maybe one thing to Think Through is like, could you do some activity to like intersect? This API, with the node restriction, plug-in that we have today um or just generally, around the idea of like um treating this as a way of like doing scoped, barback interaction.

B

Yeah yeah I've also certainly look into that and we'll update the.

A

Another thing is whether node itself can put itself into maintenance, uh for instance, for grade school termination uh based on like it can be implemented for supportences and such uh so. You Progressive determination is configured. Do you want to follow the same approach and not put itself into maintenance, but then it will be human's carrier and not itself can yeah.

A

It will be interesting.

A

Okay, as I said, let's move to the next topic. um Philip, you can step, stop sharing, um Adrian s, yeah.

H

Yeah hi so um since, since uh checkpoint support, forensic container checkpointing was introduced, I I talked to a couple of people who are using it, and one of the main questions was what happens if there are too many checkpoints and are filling up the disk and So. Based on this discussion, um I made the pull request uh linked here in in the document and um and during.

E

H

Review of the pull request, there was a comment from Mike to to bring this um to sick node, so um basically I just want to um let everybody know that I'm working on it and and and and Mike suggested some some different um implementations than than what I did here, but um before we continue uh discussing it on the on on the pull request.

H

um Maybe someone from sick node was once also to have a look at the pull request and maybe um have a comment if and how this code should be be implemented and just bringing it um to to the attention of the group here.

H

I guess we don't need to discuss it in details here, just uh when we see how full the agenda is, but if, if anyone wants to look at it, please take a look at the pull request and maybe um give some feedback if and how this should be implemented or.

D

G

H

Know how signal would want to handle this if we want to discuss this here more in detail or let me know, I.

F

Think maybe a couple of minutes discussion may be worth it so Mike I think uh the main uh point of contention was whether this should be done automatically or it's a responsibility of an operator or an external entity that is actually triggering the checkpoint and moving something away immediately.

I

Right right, the the original cap was restricted to the forensics cases, um but obviously the API can be used for additional cases and clearly they were doing additional cases with additional tools outside of our purview and I. Didn't know. If you wanted to reopen the cap and start talking about ways to manage damages.

I

For example, the the fixed Adrian is putting in I think it's more isolated to you're, making a bunch of backups, and you only want it to be 10, for example, um but right off the top you know you could you could see a policy that would make allow for a backup every month, a backup every week right? What what kind of backup system are we enabling do we want to have policies?

I

Do we want to talk about that or, as Moreno said, do we want to just give up to let somebody else inject plugins or some some other way to decide how to collect these additional checkpoints um and where should? How? Should we name them? What should the graph look like for all the checkpoints? You know we reuse um the Pod IDs.

I

Unfortunately, so if you make a graph with pod IDs, you might be landing people in the directories or you know then putting listeners in directories where they maybe not don't want to be uh it just it just seemed like we were going into some of the use cases that had stopped us from extending the cap to cover them.

I

um So I wanted to Signal people to know, and if they're going to be involved, they want to probably take a look at what this is because it you know it it it. It reads: sort of like a how we handle logs today, where you do, you have 10, and then you start it over and that's and that's fine I think um you know for certain situations, but I'm not sure if that is the right case for checkpointing containers in the memory that's stored in those containers um for other uses outside of forensics.

H

I

H

I'm, not I'm I'm, not sure myself, so um I just I talked to some people, and this was something they always bring up. So I thought I'll I'll try to bring a solution to that, but yeah you're, right, I and I I, don't know if, um if that's the correct thing or if we should do something else,.

I

We've been running into this kind of issue, quite a lot Adrian, which is you know, running out of storage, space and overuse of storage, and this is the heavy storage use case right. It might be more apt for us to be able to point to another Drive where we're storing these checkpoints- and maybe that's a secured drive, for example. Right and that's the kind of thing I would expect to come out of the kept discussion. Yeah.

F

I

I think the body.

F

Is like start with a single number that will evolve into oh, how many per pod, how many per namespace? So how much of this JC are we implementing in the cubelet versus we just let some other entity manage.

C

A

Yeah in general, I feel that we start liking this health Checkers on a node like uh we have like health check to that monitors if Google is still alive and restarts it. uh We have this log like checkpoints, problem uh and uh now we're trying to introduce logs causer here through the Kublai. So there is no nothing else running Gonna Know to assume that can collect logs even unrelated to kubernetes itself.

A

So we have a cab that collects logs through the couplet, so I feel that we put maybe put too much on a cubit shoulders, but at the same time we don't have anything else running a node guaranteed so maybe like. If, if we can extend npg, for instance, to take actions rather than being just a reader functionalities- and we can maybe put it there.

I

Can you post that that comment to yeah thanks.

A

Okay, I will comment on that.

I

Yeah I agree with Sergey those are they're aligned.

I

On the other thing Adrian I wanted to consider is, if you run out of space, we should probably be using diffs or a layer. You know a layer of us instead of just making a copy of the you know container each time.

D

um But it also depends because the checkpoints are created on a single node and it's not necessarily the case that all nodes in the cluster will be the same. So, for example, if a port that is consuming large amount of memory is being checkpointed several times, this may fill the space on one node, but not on another. So there are these cases as well.

A

Yep yeah I was talking about. Do we need a garbage collection for that how it will be collected like chicken for space.

A

um Okay, so I think action item here is to continue discussion on the pr, but I don't think there isn't any agreement right now on what's right direction to go. Okay, I, don't see! Okay! Thank you. Thank you. I don't see! Kevin and Nicole um I think the request is just to approve couple PRS on enhancements uh to move for CGI forward.

F

I I can take a look at that. One I have done that one before yeah.

A

There's also a link to some applying document, I'm, not sure what this is about. um Oh okay, it's applying that okay, applying okay um makes sense. So thank you now, uh Marcus.

J

Yes, uh we have 25 minutes, yo, I I, don't want to take too much of the Sun, so we can get to the end of the agenda but uh kind of a quick introduction and kind of heads up and put more thought and initial comments about the cap that I've been preparing with zwanko from Nvidia. It's about passing down resource for resource information to CRI.

J

So this Gap is about a lot of better visibility of like all pod resources and early on to the Sierra runtime and meet us a lot of two goals. Separate goals in mind. The first one is, would be passing down all resources of all containers already at Port, sandbox creation time on the on a primary uh use.

J

This scenario for for this, this goal is it's like VM, based or confidential container applications, where the kind of VM is prepared at the point, sandbox creation and, and that, on the reason we investment I, would really need need uh information of all the resources like devices mounts mounts and and everything else so that it it can prepare the VM correctly.

J

And then there is a another goal that would be passing down: the like informational as an informational data to the runtime, passing down the kubernetes uh resource requests and limits to the to the runtime in in sort of on office. Obfuscated form.

J

uh I could quickly uh show the okay if I can, if you can make that give me the sharing sharing rights so just show the so. The current content there.

A

J

J

J

Okay, let's just say anything,.

D

J

Yeah so yeah, this is the current okay in this, in its current current state broken progress, but basically the summaries about this kind of two two uh changes that we are. We would like to do so pass down all or resources of the plug, sandbox creation and then second of the important resources, resource requests and limits in in the container. Config information and motivation will list list like two two uh different usage scenarios.

J

First, one is the one we embarrassed and confidential container fun times and and then the other other kind of class of usage scenarios is, is then like custom, resource uh placement or optimization On, The, Run, runtime side and uh the goals. Well, there are these two to the two things currently uh unknown goals. We don't want to change any kubernet resource management or change existing Behavior or CRI, and so you, some user stories, uh then about the Sierra API changes and quickly show what we currently have.

J

So in container config, we would have like kubernetes resources, which is just the resources, resource requests and limits, remix of limits of resources and the same thing for update container request and then for pulse sandbox.

J

There will be in this proposal now: New Field called pole, resource config, and that would that would include resource information of all unit, containers and containers and then for each of these they would list this. Basically, the name pauldenda for resource requests and limits amounts, devices and CDI CDI devices and, and then- and we have like some content about the implementation. Basically cubelet would be the only component affected by this and some refactoring work. There would be needed physically.

J

Affecting the some some aspects of the resource, all the resources are are prepared, some more stuff before even before, so both sandbox creation, but anyway, that's the that's the kip in its current form, and we are working on that um and hopefully being able to showcase because it it is in a you know like into a demo in in upcoming weeks, so you can get uh a practical view. Public would and could work and what are the benefits in Practical music scenarios.

J

Anyway, if you're interested, please please comment and of course right now: people have a quick thoughts on the on this purpose. Also, let us know.

F

I think one thing to make sure here is like: how confident are we that we are covering all possible information we are sending here or we'll have to keep updating it, and maybe we reuse some of the existing structures and one more is: what is the overhead of uh sending this all the time over CRI? Should it be made optional based on the runtime class, or it should be done for run series as well?

F

D

E

J

F

That come to mind yeah.

J

Yeah yeah good questions, uh I have a proof concept, implementation about the cubelet, and there are these a bit uncertain that how certain we are that we can pass pass down everything- and there are some kind of corner cases- amounts that are being added like the plug error logs, and there was some some other case, so that needs to be kind of sorted out. But generally it looks like everything should be because the CDI devices volumes everything is kind of prepared in the managers.

J

Based on the quick kind of proof of concept, looks like it should be doable but yeah. So there are like Corner cases and questions and good for questions about that.

J

But anyway, this was a quick intro and we'll come back back to this one. We have more more flesh on the implementation implementation this inside and like better understanding ourselves, how it would be how it would look like actually in practice.

A

So do you have a specific implementation in mind that will be immediately possible with that.

J

So this one probably can comment on that, so I I think especially in the contact containers they they would be really interested to have this.

K

Hey, can you hear me okay, perfect, can you repeat the question I'm just in between two meetings, so sorry for missing that.

A

K

J

Go ahead um so so yeah Sergey was asking the kind of. Would there be any immediate uh kind of use cases that could be him? Oh.

K

Yeah, okay, yeah right, so the immediate use case is um Kata containers and confidential containers. So uh just for a heads up, uh Kata for computational containers, which is based on Kata, creates a VM and in this VM the Pod is run or the container is run so before.

K

For now, what we're doing is hot plugging all the devices that are coming in, not at the sandbox creation time, but at the container creation time. um So we are knowing okay. This is a CDI device coming there's a PC adivas vfio device coming into the Container, so we are mounting all the things um after the VM is created um in the create container phase, but for code Plugin or for direct attachment of devices like let's say, gpus, dpus or other PCI devices.

K

We need to create a correct, PCI topology inside of the VM, to have all this information correctly available at sandbox creation time. We need to pass through all the devices that are allocated for the Pod or the container um right now. The cubelet is accumulating memory and CPUs and creates two annotations CRI annotations to pass through, let's say to containerly or the CRI interface.

K

So what we have right now is the accumulated CPUs and memory sizes for the VM, but we need also all the devices that are allocated, be it a CDI device or some other devices that are allocated in the resource section of the of the podium or it which is especially for for kubernetes here are relevant.

K

So the immediate use case is correct: sizing of the VM at the sandbox creation time and not at the container creation time so that when the container comes in the VM is sized already with all the PCI topologies set up, um I mean if you need to plug in a PC Express device, you need to have a PCR root, port or PCI switch virtualized inside of VM. So we need to know up front uh what we're dealing with and in the confidential containers use case.

K

We don't want to do hot plugging, because, because this creates an additional attack surface, if we are reconfiguring DVM during runtime, what we want to prevent. That's, why we're saying? Okay, we want all things directly attached at the sandbox creation time and that's where we need all the information that's been allocated. All the resources that are in the yaml in the pod.

A

And today do you just query about the spot or.

K

Today, what we're doing is, we are what I'm doing is I'm creating CDI annotations and we have a fork container d. uh So this four container D is reading the CDI annotations and creating ad sandbox creation time the right PCI Express topology, but we would like to see that it's coming like officially from the kubernetes, the CRI uh and not a hex containerdy. So that's how we're doing it right now, uh CDI annotations, and but this also only works if we have CDI devices.

K

But we want to extend this interface that we can use like all the resources that are located um by the Pod, um be it storage, be it PCI, Express devices, speeds, vfio devices or or something else that we are passing through, so that we can size the VM correctly and one restriction we are going maybe to make in the cap is for the beginning that we are only.

K

um Supporting devices or as a start, that we are only going to support devices that can be described by a CDI, so it would be easier for for the cap to to have a version which, which is working because, as Marcus said, there's a lot of corner cases that we cannot cover immediately.

K

um So I think to get to the question of munal. If the interface is going to be extended or not depends on, if we let's say can say as a community that all the devices they're going to allocate, it can be described as CDI, then I. Don't think that we need to extend interface, because if we understand CDI, we can all pass through all the CDI devices.

K

But if CDI is not the right tool to describe all resources or all devices that are going to be passed for the container, then we might need to extend the interface. So what we're currently doing to is looking at all the use cases that we can find and people are commenting on various PRS in Kata, confidential containers and on the cap.

K

So, let's get a full big picture of all devices that we need to pass through through the CRI.

F

Things don't go, that makes sense.

L

And I would like actually like to add a few more things so right now, like most of the Qatar implementation and similar above VM based implementation, So based on the idea of what block but uh like down the line when TDX VMS will be more uh well I'll take my mainstream. uh The problem is, but those VMS are static after creation, so you cannot really hot plug anything afterwards.

L

So, at the time when they are creating the sandbox, uh we need to pass everything. What is going to be used like approximate amount of oh well, detailed information about resources, detailed information about what volume amounts devices and so on.

A

Okay in inches of time, I think we can finish the discussion here, continuous customer cap, okay, so we have a couple copics from Kevin.

A

Hey Kevin: do you want to talk about your topics.

M

uh Can you hear me.

A

I can hear it now yeah.

M

Okay, sorry about that um yeah, so I just have two uh cups up and just as part of the uh I think there was in the 1.28 Retro. We were talking about trying to get reviews earlier for the kept cycle. So I have excuse me the Pod, ready to start containers cap.

M

The beta promotion is ready for uh review and approval and then I also have uh the Split Image disc one we are hoping to I know there was some interest around especially I, think from Don and others around uh some of those support for this for their use case we're hoping to get hopefully some more feedback on that because we have at Red Hat.

M

We have one use case in mind, but we would be interested to hear about others and part of that cup, so I'm hoping to get feedback on that early, uh but otherwise both of those uh cups are open and ready for reviews.

M

And uh I think that's that's really it just.

A

Thank you, yeah and also in a planning document. We decided on the reviewers. You can pinch those people as well, uh where we decided and finally, we have uh Rob and katarzina do we have any over here.

E

They may not hear I add there, since they are paying us right. So that's why I added it. This is basically, they just want to ask a full review same as the previous one yeah.

A

Yeah for this company to make a very important decision like what's the shape of API we're going forward with and I was we wanted to use pod resources API and once this decision is made, everything else is just mechanics so yeah.

A

Yeah, the problem is, we have too many arguments to uh in any direction like we, like. Every option has advantage and disadvantages. There is no clear winner or loser so quite a significant industries that needs some.

A

And we need to just uh uh not like toss a coin but like uh commit on something altogether.

A

Thank you right now, I think we reached the end of agenda um unless there's any anything anything else. People want to talk about.

A

Okay, uh thank you very much uh have a good rest of your day. Bye.

D

B