Kubernetes SIG Cluster Lifecycle, 24 Jun 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: kubeadm office hours 2020-06-24

Description

https://docs.google.com/document/d/1ONcoy8bOw8SWPUwXxnKeRZST3lnUxYpUv4Y6466h9Ek/edit#

A

You know this is the cranium of showers. Today is the 24th of June 2020 I'm going to share the general comedian chat? Please have your names to the list of participants and your agenda topics just second.

A

And let me share my screen.

A

A

I'll see new meeting participants, I mean except for Josh, but he is presented other areas of qualities already.

A

So he says we have a recent breakage. Basically, the kubernetes release.

A

Started excluding the kubernetes, CNI rpm package, and basically this means that we combines the contents of this package into the complete package and also we did the same for I guess they were packages but for our peer packages in particular. This caused a bit of an issue. So if you have an existing setup, where you have the kubernetes eni package installed and you try to install this new Oobleck package, there is a conflict, so they try to resolve the conflict, but they I think they decided that the solutions are not very good.

A

We also saw a number of reports about this. It is a ticket. We were basically closing tickets over the place and rallying people to the same ticket and the outcome is that we ended up reverting the change. So now the community CNI package exists again and I think there is a plan to remove it in the future, but they need to figure out how to do this less harm, harmfully and maybe like how to update the docs Canadian Doc's are affected, and so on.

A

So any comments of that.

B

No comment from my side at the end of the era after they change, so we are okay.

A

Yeah overall, the existence of the Cadiz CNI package was a bit questionable because to buy the say, people really don't want to install the CLI packages standalone.

A

Sorry, the CNI plugins stand below users, mostly care about the coolant combined with the packages, so I think the decision was overall correct, but these breakages were not expected. I.

B

Guess yeah yeah I'm not against the idea, because at the end you have always to instruct both, and so it will simplify properly things from a user perspective. But given that yeah, basically given to the problem, I'm just commenting that the from a command mean side, we are fine because they are dark and they change.

A

Yeah I don't know how this is going to be resolved in the future, but I'm going to keep this group updated on this topic.

C

A

So you know with the recent events in the United States all over the world with anti racial protests and Seoul in the software world. We ended up with an initiative to remove some of the offensive terminology from big project.

A

This is the next agenda item by the way- and you know back in the day in Cuba diem, we followed what the kubernetes project is doing. So we had a few instances of the master word I just checked today, and we don't have instances of slave whitelist and blacklist, which are words that are also considered offensive, but we still have instances of master which are most related to something and I'm going to talk about later.

A

Before that, there's a formation of a new working group called working group naming and basically the working group is going to be responsible for cleaning the language from a quarry.

A

This project, similar Bustos, is in charge a lot of people volunteer to join the group, I'm, not exactly sure what decisions are going to be made or how the working group is going to process the tasks but as a part of the ecosystem have the responsibility to clean the present instances of the word master in qadian, so I just did some research found some links, I didn't manage to post them in the agenda.

A

First I wanted to show you don't see my screen but I'm going to share a link so.

A

Here, so this link right here explains first for what we are doing with the node role label. That is master.

A

So, like brief explanation, cube ADM, we have a label that we applied to the nodes to the control, plane, nodes and it it is called Lord, ro, kubernetes, io c / master and it originated sorry.

A

Originally it was in kubernetes kubernetes, but then we started using it in cube ADM and at some point kubernetes who already said like there is no such thing as a control, plane, node, there's no such thing as a master mode. You are you can't dictate your architecture, there's no such concept. So the fact that qadian deploys all the components from the same node is like implementation. Little cube ADM, and this paragraph this section here explains like you know, we had a discussion about this lick.

A

Is it appropriate for the pores to set this label whether if it's called master control paint something else, and the outcome of the discussion is that corporates at the third party you're, creating conformal Kuwaitis deployments or even have a special case for cube ATM here, which is considered first party component, may still apply a custom node roll label, and this is also a wider Seaquest lifecycle topic because I believe cops quasi API like keep spray like you know the deployer syndicate of sequel, a cycle everybody's using this, so this obviously implies that we need some sort of kept if we make changes.

A

But the general principle here is that we, like third party and first party, the pores and a lot to use this custom otro, and this is about the label. We, if we renamed the mastered, able to control blame. You know obviously they're changes, but we also use it as a taint and I.

A

The problem with the tape is that if you apply it ain't to a note, this means you have to modify all the deployments to tolerate the tank exactly much in the same key, especially if the effect is not scheduled. You know we are breaking users who want to schedule critical adults on our control a mode.

A

So at this point, I guess I want to get some comments. Feedback from the group and I think like in this meeting we're not obviously we can solve the problem. I need to like get some first steps for the plan of how we are going to transition, because this is not optional. We have to do it and like what is the timeline so yeah open for comments.

D

What I'm thinking here is that we should probably introduce a placement ain't say no draw dot when it is at io, /, control, plane and just deprecated, the old one, but probably will have to duplicate it for at least three cycles over here and for the time being, while the old one is still deprecated, we should paint with both things. I.

E

Agree also, we always a case we really on the label as well, because we have so many no deselect to make for this round a control brain. So I would like to have the duplication cycle for the label.

F

So it's the thought here that will support workloads with the note selector, because it's opt-in and then we have the cycles for deprecation for which the nodes will be labeled with both neutrals and then we can expect the ecosystem to move with their renamed yeah.

F

Is there anything that's not possible like where we, where the breakage occurs, because we have two labels, some combination, selector states or something else.

A

So the labels, the labels in particular, we can have multiple labels. You know deprecated, the old one have the new one, but for the tails we cannot do it because if you apply change to the node, if all the deployments have to tolerate the taint, if you had a couple of things all the deployments have to tolerate both yeah.

F

So means we can add the label, but it would need to be like a feature flag or a post processor or something.

E

But if we need to change things, we put a duplication cycle again at it's, no make sense to do a duplication of cycle to the neighbor, because people you need to read a to delete their primary.

A

The problem is the moment we introduce the new taint we already breaking the existing warp out. Yes,.

F

We can attempt to write a pre-flight checklist for cluster upgrades that warns users and has to be explicitly ignored unless the taints are migrated. Wait. It would be tough to write a migration tool because you would need to have the node speed label with both the control, plane and mastered label, and then you would want the users in that time period for which the labels are present on all nodes to be.

F

Moving their taints and then at some point like say we use one kopitiam upgrade cycle to label notes with both labels. We can at that point, get everything deployed to where those notes are targetable. In the next upgrade cycle. We could add a pre-flight check that looks for workloads that are painting on the deprecated label.

G

One potential thing that needs to be taken account to is the Toleration x' that we're setting if any as well. So we probably want to start setting the Toleration x' before we update the tain as well.

F

Yes, the taints and Toleration x'. I think I was actually talking about looking for workloads with toleration that tanked but yeah as soon as we add the taint middle of it. So those need to be deployed like in sequence, unless we do something like modify the scheduler.

D

C

We can refine this.

D

Multiplying the scheduler is like a little bit of an overkill, but I think that we can probably pull up something like modifying admission controller and just for a few cycles, modify those operations and basically switch over to the new ones and shoot some Logs with four warnings for the users.

F

Mutating admission controller is a interesting idea. My only concern here is that I believe with mutating admission controllers. You need objects to be applied through them for the event to catch the object. So if the things already exist inside of the cluster, then there is still an action. We're like we need to shut these objects through the API server in order for them to be fixed because it's an edge triggered system. I.

A

Think is what we have to convince the maintainers of this controller. To add this foreground I can't drive it I'm.

G

Not giving any guarantees so I think if we start with adding both labels, and then we update any of the Toleration that we're setting in anything in cube ATM that we're deploying and then give the warning that folks should plan on the change. And then we can we'll have everything in our house at least in order to go ahead and switch over the taints. And then you know we expect people do their external work in that amount of time as well.

G

And then at that point we switch the taint and then we start working on removing the additional label and the additional toleration.

F

So he's suggesting jason that we would say for this immediate cycle.

F

We could move to double up on toleration yep so that the workloads that we manage are prepared for a tank that does not exist yet exactly and then in a future cycle after we've communicated the deprecation to the ecosystem that we would expect people to also update their Toleration once we are reasonably confident that we've notified people that the Toleration or that the that this taint is being deprecated and moved people have their televisions, moved in the ecosystem for, say things like CNI, say, calico, flannel and etc.

F

Then we can use an additional kuba diem cycle at a later point to migrate, the taint exactly and- and it would just be a full move from there. We can migrate the tame we could label and at that point, that's when people would expect the breakage. Yes, that sounds pretty reasonable to me. If we did that over you know a number of cycles.

A

I think this is the only reasonable way to.

B

Ask today, if they ever, if they agree with this plan and they, if they they want to set an agenda for this plan, because this has impacted for the users. So and if we go with the normal deprecation cycle, it will take one year to to basically resolve the problem and I think that this is a special case. Where would the community agree to make this as soon as possible?

B

So if we have the sponsorship of the working group I'm following this approach, but for shortening the one year cycle due to the topic, we will do the little departments of the topic. I.

A

Can ask the group is not formed yet I saw the PR for the creation of the group today, but once the group is created, I can join a meeting.

B

A

This question do what this faster or should we follow the g8 policy, because not a tainted label argue the GA features at this point.

B

And the day we have a better picture of all the problems that the project is facing, and so they would, they will tell us okay to go faster because you are the only one that has the proper hold or go slowly, because at the end of the community needs with pneumonia, so that I don't have such visited.

A

Yeah, okay, so for 119 I! Guess what should we start right in the camera? It's already past the deadline for capes for 119, so I guess we need to have a careful this anyway, but I guess it should march 120.

A

This is my understanding here.

B

So in one age and I I think so because also, what is the replacement for master control display? Okay, it is up to us to decide on it, as it is up to the new working group.

F

I would imagine that the new working group will be informed by the previous verbage choices that projects like Cuvier mankind have chosen. This kind uses control plane as well.

A

Right uses master yeah.

F

They use the label.

E

F

In the API, they use control, plane, control, blaming worker yeah.

A

Of course there's the bigger argument: there is going also related to this group, but it was spawned by a different topic like how do we name? You know the Marcel knows how we name. Is it the control plane? It is it something else. I honestly think that we're pretty much set out on the control plane is the you know the sequester logic of qadian to the side, but I I, don't think we can pick something else here. No.

B

Okay, I talked with the working group telling the name that we think that we want to use and the timing that that we more or less agreed I have an additional question because I don't remember if he it is in the Cooperman repository or in the custody repository. But we have. We have people that is asking to apply a label to the workers node to identify the war arsenal with a boat. Is it? Is it a.

A

Note roll label or Islamic.

B

Label, he is even our know. Another neighbor.

A

So node labels have permitted, but admission is going to walk load to roll labels. I think the remembered issue. It was about note row labels, those bought Wed mission and you you can only apply them with a you know, with the clients that has sufficient permissions yeah.

B

Basically, one of the main complaint- the starting point is always that people use proven main, create a cluster one control plane three workers, then the user executes cutter get note and every see avocado get not you. There is a column which is called the role and only the control plane as a role set to master at this moment, but their role for the other mode is empty.

B

Remember so people asked okay, can I have a roll specify, the four discs for the other nodes and and then of course here it is really if you go to and if you look at the list on top.

B

Ok, the Argento or the workers are ever roll known, so.

F

Yeah I mean I feel like this is not the right way like this. This problems need to be solved at the same time, but it is a good thing to think about. I. Do think that this is related as well to the need for things, note, groups and group grouping or labeling notes, so that they can share Kubla configuration that kind of thing resolver configuration.

A

You know this topic in particular: I see the control plane modes being labeled by cube idea, because supposedly cube idea miss what manages them, and it is this cube ADM that defines the rows for this lot, but for the workers, I mean at least saying people can have different groups of worker types like something is handling heavy workloads.

A

Something is got, you know, handling something else and I feel that cube ATM, pretty labeling those it's not that I think to do, because there are so many role variations that the users might want to apply to them. I.

F

Don't see a strong argument for why the information is necessary for rubidium to do its job so to bidding. You know, should not add the information unless it's necessary.

F

But it's a different problem. You know and trying to migrate the master node role to something that's named more appropriately, yeah.

A

I, particularly the user story here is missing. I would like to see the Lord Rose but like. Why should they be gratified for the workers I.

F

Mean the user can do this on their own; nothing stops them from doing that.

B

F

B

The problem separated me.

F

Good to bring up things, I didn't know that somebody had talked about that before.

A

Yeah I mean there was previous disagreement about this.

A

Yeah, so what is y-scoo Sergei, not labeling? The notes like optionally, for instance, I.

B

Think the same problem: do we discuss it now? It was just a topic that to come on top of my mind, while talking about the label.

A

Okay, so, okay, okay, start writing the cap. It's not probably not going to Marchesa debatable for 119 I can talk to the group that is forming it. Touch softer, like I, want to get back to this, the actual plan to know. To summarize it again, we are going to include two narrations in our more thoughts, for you know, add another key and effect here to tolerate the new. You know control plane, but should we also let double label? What's the initial step?

A

Yes, okay, that's one, and once we have that running, for you know unknown releases, we can remove the old politicians and paints and labels. Add.

F

Annotate at that.

B

And in the first declaration, we have to deprecate the label and issue a warning or action item for the result and.

F

The action item will be for.

F

Workloads to to either select on the new label or to add toleration for both labels, yes, which is a little awkward but it's gonna, be necessary and then the those people don't have to change their toleration after the labor was deprecated.

A

Yeah they can leave them like that is.

F

Yes, but eventually there will be a time where it's no longer to sing so I. Imagine that that's probably just gonna stay there.

F

B

The end when we remove finally the label, we will raise annual extra item. That is, okay, you remove the second. They all the want, polarizer on.

F

The ecosystem will likely still be at a point where, when we release kubernetes, you know, CNI providers are still using. Then the minimum number of manifests necessary to support older kubernetes versions in addition to the new ones, so they will likely leave them in place even after the label is deprecated.

F

G

At some point, they'll eventually, you know update their minimum required version and unfortunately, it'll just have to be until their minimum required version is the one where we switch. The paint yeah.

H

I'm just like to fight one of the CLI manifest what domination? Yes, that's the the only driving reason.

F

Why I can think of potentially yet I? Think? That's really the only reason why you potentially want to modify the coop Navy's components to equate this taint inside of the scheduler.

F

Is just so that the action item would be would not leak that label further into the community manifests snowmeow well.

G

The problem there is, you would have to back port that fix to enable the community to adopt control plane over master and that to work for the older versions of kubernetes and yeah.

F

You know we start talking.

G

F

Only gets you earlier basically.

G

Yeah and and once you're talking about that you're also talking about like vendor distributions, you know that are releasing updates for unsupported versions and all that weirdness, so I, don't think. That's really a feasible path forward for us. Yeah.

F

And the only benefit that comes from it is that it just moves the window from which the both Toleration are not required to an earlier date of time, because that you eventually do reach that deprecation point. And it's just a lot further out.

A

So among the myths of the setting, the tension to Mauritius a little bit, but this is the conical manifest Vanitas calico, and it only includes this. So it's like operator mist exists. The effect.

B

A

Not scheduled and it can still schedule the control plane modes that could be.

F

Yeah, maybe we have a misconception that the labels are being used in toleration.

A

So maybe this suggests.

E

It is the case if we can double label Andy. He moved the only change the tank name, I.

A

Think this is a redundant I, don't understand if this is the case. This is the observation from the Calico installation. Then this is redundant.

F

Can you look at the Calico toleration one more time, so it's just tolerating in general- and this is for calico note no scheduled and critical add-ons only and this would go for every every node in the cluster regardless. So it's not control plane, not specific, but it needs to tolerate control, plane, label.

C

Exactly I'm saying, but.

F

This would this is actually different than.

F

Tolerating the note law, which has to do with me, if not all specific, instead of these effects, you could combine them together to actually only tolerate these effects on.

F

G

I think seeing eyes a little bit. Probably the counter example to because see and I in general has to tolerate no schedule based on networking not being available for the cubelet as well.

F

Can we think of the other kinds of add-ons that might be using a different toleration.

G

One that I can think of is what are we doing for the manifests that we're laying down like for the cube proxy deployment or Core DNS.

A

A

Think we're doing the same us the coordinates case actually.

B

A

This is what we drink.

A

So I'm very confused by this one. In particular, we just have an operator with a missing key and effect.

A

So Parker Fusco is decision, scheduling.

G

I guess it would depend on what the defaults do.

A

So I go to another experiment because we like we need some testing here. I think that we might be able to just remove this and users can remove it too, and I think this is sufficient for able to be able to schedule I this I've confused I. Don't think this is needed that I'm going to try it.

A

Yeah, so I guess no changes for 1:19 anyway, Russell you have a comment and.

D

Just spotted the communication, this appears to be some sort of special case, so.

A

An empty key with operator exists much so keys key values, effects, which means that will tolerate everything and, at the effect matches no effects with key value. Well,.

A

So this means that we, this confirms that we don't have too much the mode ro explicitly I, guess.

G

Which pretty much makes sense for cube proxy, but core DNS is probably gonna, be the more tricky one, because what we're trying to say is that it should be able to be deployed on the control planes. But you know we don't necessarily want it running on user-defined chains.

A

Yeah, the Preferences physical coordinates or the control plane nodes. If the control plane does go down, I've seen coordinate smooth to the workers as well.

G

Which is possible yeah and we definitely wouldn't want to deploy it anywhere, where CNI isn't available as well. So we can't just do a blank, no schedule, one.

A

Yeah I guess this is still these. This is still needed.

G

But the good thing is that, since we control this one, it's easier to go through that deprecation cycle, it's just a matter of we can find and identify any external users that are relying on it.

A

Yeah from experience I've said messages through communities, Delfino pastas or the cube idioms like channel reddit.

B

A

If I get more responses and then the issues at some point start appearing like having problems so I would expect this new working group that is being created to also handle some of the hate mail that we are going to receive for breakages. So I'm going to ask them about this topic as.

G

Well, so it might be good in particular reaching out to like the cops folks to keep spree folks. The mini cube folks obviously been for kind I know in close to API we're doing some interaction with the label, but it would be relatively easy for us to add support for backward and forward compatibility for whatever period we needed. So from our perspective, I'm not worried about it about the change, but at least the other six sponsored projects. It would be good to see where they're currently relying on it. If at all,.

A

Yes, that would to put this topic on the agenda for the legacy meeting. Hopefully people words who have disability comments on this, so yep.

A

Okay, any anything else for this topic.

A

I, don't have any other gen titles. If somebody has anything they want to discuss.

F

It's the only additional value-added comment on is that alternative to control plane is manager. It's what stalkers swarm uses and it's the manager worker analogy. It's just shorter to say, I. Think I've found myself not saying control plan, sometimes because it's just long.

F

We we were able to suss out several reasons explicitly why changing components like the scheduler is an unattractive deprecation option for this label. So if that conversation comes up in the future, I don't know how many notes we took on. Why we shouldn't do that, but we could probably all remember that we decided not to modify control, main components.

G

So I think in the case, when you're talking about the scheduler and of the general kind of overall thing, I would agree that trying to lump all of the components as one control plane doesn't necessarily make sense, especially if you're looking at like pod based control, planes and and all of that stuff, because you know they may not actually be centered on the same type of hardware.

G

I do worry that if we try to change terminology, we're talking about a lot bigger change, because we've already done a lot of the documentation, rewrites and we've already gotten a lot of kind of community coalescence around the idea of calling these things kind of control, plane nodes. So I'm not saying we shouldn't do that. But if we do it's a much broader discussion, I think yeah.

F

And I've sat down even with like contributors who have been confused with other deployment. Topologies a good exam like Ubuntu's charm to kubernetes does not deploy the control point named pods and the control plane does not run in the same Alex II containers as the couplets there's no couplets, and they were trying to deploy some security and auditing features that relied on scheduling a pod to the same place where they could inspect. You know PS flags and things for control, plane, components and I.

F

Just was not possible, you know so there's definitely confusion and the fact that, like control plane is part of kubernetes, but it's frequently not yeah.

G

And I think the difficulty comes when you start talking about different kind of opinionated deployment models like the default cube, ATM model or what we were generally doing in cluster API or cops or some of these other ones. You know some of this terminology makes sense, because you know you need an abstract way to talk about, like the bundle of things that we're deploying as a control, plane machine, but then in other cases it doesn't.

G

So that's probably a good place to make sure that we're differentiating the usage and the discussions that were usages of these words in the appropriate places for the appropriate concepts to.

A

Control plane is very widely agreed from the perspective of cube.

B

A

Least, we have it everywhere in source code in the commutation. At this point, the same way we used to have mastered. We renamed it and moved to everything to control pain, I still in favor of using control plane. But if we have, you know, maybe on a city level, if we agree that manager.

B

A

The better term I guess cube ATM should comply with the consensus.

F

Doing a github code search of this label and the add-on that I'm, seeing most often of being using this kind of toleration, is actually like ingress controllers and service load balancers and the kind of thing people pinning those things to.

C

F

Point notes using a.

C

Master label technically.

A

Pro plane originated in British and still use this distinguishing core components controlling conflict they. This is the.

F

A

Is what we're using a control plane, both in the case of comedia.

C

At least signifies the presence of these corporates, something that runs parts of the control plane.

G

So that is interesting that the ingress controllers seem to be the ones latching onto it and at least from somebody who's been involved with multiple kubernetes installers. That makes sense, because, generally you know, you already have to put a load balancer in front of those hosts, so you also generally have you know static? You know, Ian asked for access, so there's either DMS or IP addresses. So it makes sense that you might also want to target that for ingress as well, but.

F

Also for things like load average, the load average on control clay notice, you know likely going to vary eight less than workers which are running high compute. So it can be an attractive place to put things if you don't want dedicated nodes.

A

Anymore, comments of this topic.

A

We have ten minutes. I wanted to do this efficiently as quickly or rather PR triage see what we have.

A

Suppose you have a new corporate clean up I guess they should March after your other peers.

D

It's technically dependent on them, so we basically consists of two parts: the first part three factors: the component config tests and basically just use the QP names coastal configuration component config so that we don't rely on Cube proxies or any other external to cube a damn component convicts to to our testing. So in case, some of these external or topics, change or a new field is introduced in them.

D

The impact on Q beta is really small and you don't have to go our unit tests and change stuff, and the second part of this is basically just cleans up the old yeah.

D

This is the only test of joint operation compiled that was like extended and they used as a partial replacement back in the day of the puzzle test when it was disabled in the output to another three days, so basically I just narrow down the scope for this unit test and remove the wall of testing files, and usually these testing password like a PR targets by anyone who was actually changing, something on the Q proxy or cuboids component config, like adding a new field. There.

A

Thank you. This is a great refactor. The thing is: is this lawlessness to March for 119, but we can still play it.

D

Yeah we can do whatever you like. So basically, it's not kosher after a time that just tests but yeah.

A

I'm going to apply Tabasco, you know if we think it's Lots possible we're going to remove it.

A

One downside of this change is that we are no longer going to be able to see potential changes in the couplet, which is a fairly active API that changes all the time, but still it's fine that we don't have to accept these modifications by contributors, Thank, You, medium.

D

Yet they usually like the cute little changes, don't break something for us. They usually just start a new food. It has same default so.

A

Okay, I'm going to have a look at this, possibly after we watch the previous peers or in parallel. I can try alright, so yeah.

D

There is only one big SPL: that's the great one table yeah. So.

A

So they may have lost my link just a second.

A

Or maybe more than one secret debatable github speed, so we are grading it city server, two three, four, nine, this cycle, I think the problem he right now is just we don't have the images the GCM packet yet and it's some complications are out switching the there's, a domain flip at Google which took three months or something like that. It's not clear why it's happening the problem there, but hopefully we can get this new version of HD.

A

The Clyde is already updated to my knowledge, so we now did in the server. These are Bart ports waiting for approval by the patch team. It's about the regarding the member. Don't reading the HT member. If it's already exist, I like Fabrizio I'd, say that I don't know if this even works. I have not disabled.

A

So we are only like. We have sanity of the logic, but I haven't seen it in practice at all.

A

So if HDD, if you have a if the member is already there, but the storage is not good, for instance, the storage for this member is not good. If you try to add the member in or exists in a cluster, so new member is still going to be on it, but the storage is broken. So I don't see how this helps anything.

B

But I think the problem here was different. The problem is that we want to make faces independent, so the idea was to or Cooper mean, join independent, so it means that we want to being able to rerun the Commodore, the world command or the single phase is in and necrosha where the common was revolution from so it is not meant to to fix. Some to recover is just to mated to a low to run the comma.

F

This basically just is defensive coding that results in idempotent behavior right, because it would fail if it was already there, because.

B

We saw people that that basically to does come, enjoy it if it fails, run again come at me join, but the common was not it up on that. So this was becoming an infinite loop for these people, so we want David wallow to duties.

A

Okay, you know. Ideally, we should get some testing that the DS fixes we are doing, or even doing anything for reporters.

A

Maybe we should ask not here if he can level the tests.

A

Okay, this is working progress very shirts, disappear, I haven't reviewed it, yet we can get to this. Hopefully, next.

D

Week is it actually because I think it used to cover with marker.

A

You say that I saw the work in progress anymore, I.

D

Think it had with marketing the like the name of the issue like on the name of the PR, so I think you probably renamed it. Can you scroll down a little bit to check.

A

Yes, it's like I guess we have. We are pretty low on time. So if you have anybody on the call that wants to help to do a review on this pier yeah.

D

I'll take a look at it tomorrow here.

E

A

A

A

Robustness of cuba, DMHC operation, we have one minute.

A

This by the way or bits of this PR is waiting on you. If you can have a look at it to support, it has been open for 10 days.

A

It's horrifying it city client to have the same little I want watching for the dial and everything about.

B

A week and some time in meantime and next week, not before.

A

Okay, do you want me to pick you again on this world.

B

Notification on our panda butter and I'm, currently beautiful, okay.

A

A

I would read the beatings because we had almost died. Takes everybody see you again next week right.