KubeVirt Community, 21 Apr 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: KubeVirt Community Meeting 2021-04-21

Description

Meeting notes: https://groups.google.com/g/kubevirt-dev/c/Te8M5vCiav0

A

Hello, everyone I'm chris caligari and I'm your coovert project organizer- and this is our weekly community meeting to discuss important issues in front of the community and I will share my screen. So we can watch our our.

A

A

Okay, uh if you could fill in your your name at that in that attendees list, that will be excellent and if you have a topic you'd like to discuss, please add it to the agenda or the open floor and ken kevin: hey, kevin, sorry, yeah, yeah,.

B

Kevin it would have been.

A

A

B

Still getting uh used to all of the names.

A

B

A

B

A

The first topic please proceed.

B

Yeah, I I wanted to uh bring us up mainly after uh I don't know. If it's roman here yeah.

C

B

Yeah uh yeah, because I I don't know if it's like tell me if we should discuss it in a different manner but like uh yeah, how we would handle different kind of probes in keyboard like this one is about the exit probe. But we uh have the qemu ping probe and I think it's like a general topic on.

C

Yeah, so I can tell you a little bit about the history about how we are implementing right now, probes and uh and why we only support right now, the probes which we have so, if you're still creating liveness and readiness probes, your free aims. You can do that right now, only for networking, probes and not for exec probes, and one reason for that is that for networking probes we can simply mostly reuse the pot probes.

C

It has some limitations like uh if you pause the vm liveness probes with failings on, so you can't use them for all, but it works kind of okay, but not perfect, and the the step for me was always so when we add more probes that we would redesign this to be more suited for our virtualization needs and exec probes is one of the things where I would have wanted to see a redesign of our probe architecture before we add them mostly because exit probes have some uh nice side effects on various versions of container run times, and and we have we burned us- we burned assault already with exit probes when we use them for other purposes, and uh I made sure that they will remove all exec probes in the whole project.

C

Due to that and yeah that's the background, and I think your question is now on how to move forward with this. How such a redesign would look like right.

B

Yeah, I I I already spent some time before using the um like going through the pod probes and, like the only other way I saw was like using those um uh redness conditions.

B

Yes, that would allow us to do different custom stuff, but of course,.

C

So I I see uh other ways too, like one which can potentially collide with istio, maybe is using the localhost one: two: seven zero zero one as as endpoint for the probes, you can configure that um and basically word launcher, would listen there and then proxy to return the very appropriate and you could not only use it for, and that means for yeah, and you could not only use it for http probes and you could also use that for exit probes by then on the part still configuring a network probe, but at the end then doing the exec probe or the queue exit probe or whatever we want to do there.

C

The potential issue I see here is that istio and other mesh networking solutions may want to you talk through these ports on localhost to the actual workload and not to word launcher, and uh for that I would still see see that already as an improvement to what we have now.

D

um We already have sorry for speaking over here, um but we already have like a set of ports that we cannot use for the um vmware cloud. Some of them are used for the live migration proxy. Some of them are used for istio, so, okay, so it would be.

C

Just just use one more yes, yeah, okay, okay, it would make much difference still. There would be one additional possibility and that would be just creating another local interface which listens on one two: seven, zero, zero, two or something and use that for all that stuff.

B

uh Just to understand correctly, what you mean is, then you would define your exit probe in um in your vmi spec and it would be translated into in an api call on a http probe on the pod.

C

Laundry exactly and vert launcher would have.

E

C

Spec and know what actually to do and when not to do it and for networking.

E

This means that.

C

There is another in general, it means there is not too much which needs to be reimplemented, because it's mostly acting as a proxy and sometimes returning something instead of the workload like when the game is paused and we don't want to get it killed or something.

B

Okay, the only thing that would have to be done is translating a axic pro command into a proper http query, which I don't know depends on how complicated the exit pro is.

C

Yeah so yeah, so you would create a so. We would translate on the pods back the exec probe into a http query for localhost on a specif, specific port, which launcher would listen there and then just do the exec, which is defined to be my spec, and uh I would suggest to on the long run, move all over. Also all existing probes to that pattern, which we already.

B

Have I it doesn't, it doesn't feel uh intuitive at.

F

B

To to I don't know, what would we do like that? Base64 encode, the shell command and the exit probe and send it forward to avoid escape problems.

C

No, no, no you you would just uh you would really just so you on exit probes and so on. You can cannot really check. What's getting back right, you just see. Okay, the exit probe succeeds or fails right, yeah.

G

A

Example, probes.

C

You do not care about the return, so word launcher sees the vmi spec, so word launcher can literally just oh.

H

C

Yeah with launcher gets from redtember will be my spec, so you would just really create you would just for all probes. You would basically just take over the frequency and retry counter and everything, but you would for all probes, just always ping. The same local host word launcher port and with launcher would be.

B

C

So we can still leverage the retry counting and all those mechanisms from yeah from the cubelet, but we can we have more control than on what to actually do and how to do it.

B

And that wouldn't mean more overhead as well.

C

Yeah, it would mean almost no more overhead yeah.

B

B

Fair enough yeah other questions like yeah. Well, can we just no? It wouldn't really be an api change either right, because it's.

E

Just yeah that's.

B

Good, okay yeah! I can get behind that.

G

That sounds great. That would be great.

B

Yeah I I was like I was brain lagging on the, but I need to encode the exit command into path for the http probe and that's like it felt weird but yeah that was that was completely off yeah. Okay,.

C

Yeah I mean could have been, but luckily it can be done much easier.

B

Yeah all right, okay, then I'll I'll. Look at that see also how I can get the network networking right. I don't. I.

A

B

About that yet like if I need a new interface or.

C

Yeah, okay, you could yeah, I mean you can start with the exit pro also and then you we can move them over afterwards. It's also fun yeah.

B

Yeah sounds good thanks that helped.

C

Thanks for asking bring it up.

A

Okay sounds good: um are we all set with this topic, then?

A

I think so all right next topic, discussion of non-root implementation of.

I

Launcher hi, everyone.

I

Hello, can you hear me yeah, yeah yeah there you go right, sorry, double muted, uh so I want to bring up the discussion, the non-root implementation of launcher. uh I have a few topics uh mainly. I would like to discuss how we want to test this feature.

I

It would require for some lanes to also run the implementation of route and non-route, or we would also have the opportunity to say that we can default to the non-root, and that is basically the first question.

C

Okay, so one question here: what exactly would prevent you today from running from just or let me rephrase it, what exactly is not encapsulated in your non-root pr, which would have to be considered outside.

I

So uh currently, the v2fs is not supported by the session mode and also the huge pages are not included in the pr, because there is some race condition which I would like to address. After that.

C

Did you consider moving everything over gradually so by first? First, in your pro request, starting all the features which work and still use more privileges for the cases where it does not work or is going back and forth? Yeah.

I

Yeah sure this is how it works right now. So if you have the huge pages or video fs, you will run the root implementation. So it says it is a gradually uh virtual implementation.

C

Okay, so what ex is there really something which you would need from ci interlanes.

I

uh So I mean we need to test the download right because we we can, for example,.

I

I had a three base and there was a br from growth sec which changed some permissions and it chips out the non-roots. So.

C

Yeah um so, but you have your pr.

A

C

I mean I would expect we at some point either come to the conclusion that what you have right now is good enough and safe enough to take in so that we.

I

C

Have any regressions and just merge it or and then from there on you, add huge pages and other features and just remove more and more the privileges or.

I

Maybe I will ask the second question, and the second question is that there is an no a easy way to integrate and unroot in users for in the kubernetes.

I

So if you use some device plugins that in the upstream is not that we are not testing them, then you can.

I

The plugin will give you the device with the root root uh ownership and you will not have the access to it and that will break the download implementation so that this one consideration where I would be more required to not default for non-root.

I

I'm not sure how it is used in the community to run some additional plugins.

C

Yeah, as far as I know there, most of them are always running this route.

C

I see multiple ways. Yeah I mean I would just expect that we start conservative and where we know that nothing is added which would need root like devices extra devices or something.

C

That we would use the non-root one another about to use the one, the more privileged one deciding invert.

C

Controller is that reasonable.

I

um Sure yeah sure, but because I.

C

Mean we have that already for other things like vladik, did a lot of device stuff and so on, and here's a lot of cases where he adds more and probabilities for certain requests and address for everyone.

I

Yeah sure what I'm trying to point out is that you can have a device plug-in and I think we we are. We have the option to add the device to vm, and then you can alter the xml flipbeard through the for the sidecar right.

A

I

This cases won't be easily detectable.

C

Okay, so so we would need.

J

Yeah, but I guess in that, in that case the user would know that um he cannot run non non-root.

J

Yeah questions: do we do I guess what you're trying to ask? I think whether what do we do in rci do we test both non-root and root? Do we test all the features with root and non-root, or do we just default to non-root and then special cases will be handled differently.

C

I guess in ci we we cover pretty. We recover more things which we support in general, so I think you should just go with the long route case where possible and the rest would anyway just played in ci, which means it's not ready yet and there you could fall back to privileged mode.

I

Okay, so we would like or the ways to to go with default of the unknown in the ci and don't double the lines right.

C

Yeah, what's not clear to me is so so I don't expect that we will have to two code paths merged where we, in the sense of just in channel, to use non-root.

C

And then somehow switch to the root version I would have at least I always had it in my mind that, with with your work, we are gradually moving kubernetes over to lesser privileges and not having parallel implementations, but maybe that's not how it is done.

I

I was thinking about it about the case. When you have a cluster, we introduced the non-root. We still want to to test the root path, because there are vms that are running the root implementation right.

C

I

C

I mean we test, we test devices and all this kind of things, so I would expect that they are failing, we just huge pages and so on. So we test the cases here we would need root. I agree we could think about if the sidecar is enabled and the site car notation is there.

C

We, you may also want to go with root for now.

I

Okay, no, that was not that's not my point. My point was that the testing so for the testing. uh If we switch to test the non-root in uh that in the cases we can run it and root only for cases where the known is not possible to run, then we might miss some code paths the for non-root for root, sorry for forward implementation and that can break some existing workloads right.

C

Okay, that makes more sense. No, no! I think you get it.

I

Right. Sorry, if I wasn't clear.

G

David radick.

J

G

J

Need another lane.

H

C

Why not so we would have one lane, then, where we start test, where we run as many tests as possible is non-root.

C

Yeah, it will give us a password, I think, and then it would basically be a feature gate or something which decide to decide the conflict. What you want.

J

C

J

Did we tested with the srav and the some kind of future.

I

Yeah so currently I'm forcing the number to be tested for all lanes. It's passing all all cases. I think there is one test which is failing um yeah. I lost some some part of the implementation by rebasing, so I can fix that.

I

So currently it's passing all the tests on all the lines with manual implementation.

H

Nice, how did you do the capsis resource capability or make it work so.

I

H

I

Don't need the capability.

I

J

Yeah, you don't need this, because it's running with the same user.

I

Yeah exactly so, we don't need.

H

A

Okay sounds good. That's going to be an important feature for our version. 1.0 correct.

I

It was mentioned in the v1 spec. What we want to do.

K

Yeah yeah, it's.

I

K

It's a major one for sure. Oh david, you are here, I am I've, just been kind of lurking.

A

You're uh you show up as a very strange uh username.

A

Okay, uh we spent uh a bit of time on that one. So, uh let's move on to uh arm 64 cross, compiling roman.

C

ah Yeah I just wanted to give I I gave a few status updates in the past already just wanted to highlight that the the arm engineer continued with cross-compilation support and I made the job the pre-submit job required today. So whenever you do a pull request, you're now also doing cross compiles for arm 64 and works. Pretty nice and yeah. Really everything from rpm installation from container builds uh c compilation go compilation. Everything is cross compiled, so we really get the full keyword thing cross-compiled there natively, which is pretty nice.

C

That's a huge big amount of work and I think a good role model for ppc 64. everything is. It can follow the same pattern without too much effort.

A

That's awesome, uh see all things open demo way down at the bottom. Talk about that more um kevin back to you for feature gate, deprecation.

B

Yeah, I I forgot to add that one earlier um yeah, that's also follow up on a pr. I continue that link right now, um so with note labeler as a part of kubert. Now um we figured out that uh cpu uh no discovery and the uh hyper-v enlightenment feature gate, hyper, restrict check, feature gates could be default and um yeah. We didn't know how to do that until we were shown. I think, let's do, that. There is a way to default feature gates now, which I honestly still don't really understand.

B

If it's actually a default, because it doesn't look like one, it feels like it just defaults for generation and I like to bring up general discussion if we want or should be able to uh change or deprecate the feature get api and to make it fields that can be properly defaulted in our part of the schema and yeah.

B

I'm sure somebody has an opinion about that. Well,.

K

What it sounds like um okay, so we have a feature gate for some of this node labeler stuff, because node labeler was previously in a different project. Now it's in keyboard, it's always enabled, so we always just want to use it. If it's there that that's the case right.

B

Yeah, we want to use it if it's there, but people might have to disable it if they use emulation and uh cpu features on their vmis because then their bmi, so it gets scheduled. Okay,.

K

So it sounds like uh it's still a feature gate then, like. I guess I'm not understanding why we would need to change anything. It sounds like it's working as intended, so we have a feature gate where people can enable. I.

C

Guess that they, I guess.

K

The point is that you, but.

C

We cannot enable that by default, and then you just disable it that's not something which you can do right now. Yeah.

B

It is a special case to disable it not a special case to enable it anymore, like that switched.

C

One solution would be, for instance, that for every feature, gate, a disable feature gate associated and only one of the two can exist, and then you could have defaults or something like this or a map. I think it's really confusing.

K

Because then, what yeah.

C

What's the default behavior.

K

Like if you didn't specify the feature gate, what do you get if we have both enable and disable.

C

Yeah yeah, it's just, we don't have a way to graduate things out since the feature gate is also used for uh you know you have an alpha feature. You want to allow people to play with it, but it and it should be okay, but you don't want it to be any by default. So you add the feature gate it's not enabled, but at least in kubernetes, then you have the the step where they enable it by default and you can still disable it.

B

Yeah feature yeah, I'm using future gates having a a life cycle most of the time, and I I I I actually. I was surprised to see that future gates are like the string array in keyboard, and I I know in hco, for example, it's like fields that can have defaults that aren't a schema and I would like to discuss and propose that futuregate could become an actual object with fields that we can describe the document and also default.

B

And if that's like completely off as an idea, because it doesn't match what we want to do with kubert or if it's just, I don't know.

K

So the history here is um it's kind of long. We had a complete map before we had the keeper custom resource and yeah.

E

K

Where the list of strings came from because we had a config map, you had a feature gate uh key uh and then the values were just a string of commas.

K

Whatever word, I can't.

E

K

Values when we were translating that to the keeper cr we essentially just took that and directly. I pulled it into the key for cr for better for worse, um so that might have been an opportunity to to kind of change things here. um So now I don't know what there's not I mean we can come up with some sort of scheme where we add a new field.

K

I guess, and we have to maintain compatibility- is the problem. Yeah.

E

K

Have now so I don't um it's unclear, it's not clear to me what the correct path is. I kind of wonder if, if there's a path that we should be taking, that deprecates feature gates and uh moves them into well-defined fields as they become like a mature feature. So let's take live migration, for example, law immigration at this point uh is probably mature enough that it could be enabled by default.

K

Yet if somebody wants to disable it cluster-wide, then that could be like a toggle and a well-defined api.

K

So maybe there's a graduation where at futuregate once it's no longer an experimental or alpha feature or something like that uh gets removed from the list gets enabled by default and then graduates to another field where it becomes uh something you can toggle on and off.

K

I don't know I'm just yeah.

B

It's not the topic, that's kind of what I'm what I'm looking for, or I'm hoping go towards that because, from what I've heard in mental discussions that feature gates should be like developer, try something out, but that also implies that it should be gone as affinity at some point.

E

There is a nice pattern of uh feature object that I will I see I saw in the hyper-v area in the web book. I curvy, so it's like a feature object has a boolean of enable disable it. It also has a dependency if it depends on other feature.

E

So if you would like to, I don't know, refactor this area to become more more scalable object, I mean so. I think this would be nice to adopt from there.

K

Yeah that sounds way more expressive as well. um That might that might make a lot of sense.

B

And it also means we can attach documentation to the fields and we can, as I said, have defaulting, but also it's part of the schema that can be seen in documentation that I really like, because right now, I don't know where feature gates are documented. But if, in the worst case, I look at the code which ones exist and that's like not nice. So.

K

One thing: here's a thought: we can preserve the existing feature, gate behavior, that string uh and create this, this new way more expressive structure and it's kind of like um other parts in our api, where you pick one so you either pick the more expressive way of enabling feature gates or use our old one in the more expressive way has a way of enabling and disabling and clear api.

E

K

What the default behavior is, so it gives people a transition path to the new thing. Yeah.

B

What what is the general practice or stance and keyboard around generally deprecating fields hasn't been done before like if we want to get rid of the old feature gates at some point? Is there a way for us to do that.

C

We always say: if we want to do it, we should probably follow the one from kubernetes where we use you another certification and 12 months.

B

C

Gone but we never did it so.

B

K

Basically, it's it's just. We need to agree on a policy for it yeah, so we've we've kind of just kicked the can down the road for years, which is I mean that kind of makes sense to a degree like let's tackle.

E

It once it becomes.

K

K

It's like a forcing function for us.

B

All right so, okay, um how about I wouldn't mind starting a a document about the specific case which could serve as our first example for that deprecating stuff, like laying down a road map and applying on how to do the new feature, gate object and.

K

Ooh use the new design process, so your user.

A

B

Thinking about that.

K

B

K

Can say I want to add this thing, or this happened yeah you got it so.

B

It could be an example for the design process and for the duplication process and yeah. I'm.

K

B

K

Creating a design document- probably this week as well, so you won't be the only one.

A

Awesome we have an idea of how many future gates we have.

B

I

Seven, I think I don't know.

K

Probably around ten something like.

B

The list of variable constants we have here are 13, I don't know if any all of them are used in the code anymore.

B

B

Featuregates.Com.

C

And it's right now also a mixture of features which are made. Some features are also major, but are disabled due to security implications like sidecar, for instance,.

B

Like and like one one part of this came from like the current cpu feature, like node labeler feature gates, there are two of them and we would actually like to merge them because there is no reason to keep them separate anymore. So yeah, one of those.

B

All right thanks and I I know what you do now- that always helps.

A

Thanks kevin okay, um next topic, maya.

L

Okay, so we wanted to remove some walk-arounds that we added for the kindling, and someone proposed that perhaps someone is relying on this.

L

So this is another heads up that they're being removed, and please pick up if you are running a lane in the setup and if it's better like just if you are running it at all, that's it.

H

What exactly do you want to remove.

L

There was some tests being skipped if running on kubernetes in docker, they weren't passing.

H

So are there any other tests other than srv tests that uh skipping on a kinda setup.

L

uh Sorry, I missed some of that, I'm having connection issues, but I think it's not a problem from the srv lane. It's uh it's running a very small subset of tests and not these tests.

H

A

Thanks great ci improvements are always welcome. Thank you, maya.

A

um I have the next topic um david, your favorite one. um What are we moving liberty away from raul downstream.

K

Yeah, I don't have any more information.

F

Anyone else easy reply.

A

K

Anyone else have any information on, uh I think we're we were going to transition to centos streams.

C

As far as I know, that still takes time.

C

But I have no clue when it's done, but people are talking about at least half a year or more goodness, just tell me that uh do we ever get another issue.

A

C

I think so what the.

J

Magic and just the regular um center stream doesn't include what rail av has.

C

Right the machine types yeah, but there is this plan from uh from from the red hat people on liverpool to do all this railway stuff in center stream, make it always a little available in an extra stream on center stream.

C

That's interesting, and that could be something which you could pick up and it would be more transparent for your.

A

Community, okay! Well, if you could, uh if we don't have to wait, while you find that that issue, I can plug it in uh after the meeting or or create one. If there's nothing, not one, there.

C

A

Okay, I'll create one I'll create that issue then so we at least have something.

A

A

Okay um time for the open floor, um I have just a few event updates.

A

uh So, let's uh roll through this pretty quick um red hat summit for everybody who has volunteered uh if you have not gotten your calendar, invite for your time blocks. Please see me those those invites went out yesterday and we will be testing the platform very soon.

A

All things opened um stu, and I have had a couple meetings now uh regarding what we want to do with all things open. um The issue is is in the bullet. Point uh looks like we're going to come up with a internet distributed um edge, computing, kubernetes, cluster, running on uh raspberry, pi 4bs, and probably some intel's thrown in also so mixed architecture, mixed workload.

A

So that circles back to the arm 64 cross, compiling which I'm so happy to hear, because that was a a big question. I know whether we could even run coovert on on arm 64s.

A

I spoke with uh my my nasa colleague last night and uh asked him if what he would like to see in uh in a mixed workload demo that would uh that would perk his interest.

A

His number one issue with uh with his openstack implementation is, uh is uh dealing with cellulometer and getting metrics out of his virtual machines.

A

So if, in this demo, we can provide a a quick, uh a quick view on how the node exporters work with a graph on a dashboard showing metrics he's uh really going to be interested in. Seeing that.

A

So also, if you're interested in participating, please chime in on the issue or contact me. We have about six people now that are gonna provide hardware and uh to build this spiderweb across the internet.

A

It's really starting to sound, pretty cool um kvm forum um popped up uh in the news yesterday and if, uh if, if you're interested in participating in kvm forum, please talk to me, I didn't really push this one, because uh right now the the wording on the event page is uh just it didn't sit. Well with me um being that it was it's an in-person event and we have a global pandemic going on right now, uh I'm uncomfortable with uh pushing people into traveling um in in this situation.

A

But if, uh if, if you're comfortable with it- and you want to go to ireland, um let me know and uh we'll see what we can do. Maybe you've got. Maybe you live in ireland and you can just walk down the street and attend the conference.

A

All right, so that is the updates for events. Does anybody else have any any top general topics that they want to talk about.

E

Hi, I would like to touch a topic, it's more like on the project level and keyboard um about our policy of cherry picking, prs to stable branches, and so far as I'm aware, we are cherry picking, urgent bug fixes which are usually filed in baxilla, and our recent efforts to reduce the flakiness in the upstream ci will have resulted in a product called prs which may fix some area of code to stabilize the test.

E

And so my question is whether it's rational to um such prs as well, which I don't know, may affect this stability of the release branch, but still they. It may, on the other hand, reduce the flackiness on the lanes when we run them on the stable branches.

E

K

Do they fix a bug or I guess, if they've reduced flakiness, then.

K

They need to address a production issue. I would, I would like to think it was product if it's texturing the actual production code. I think we're okay, with back porting um test fixes the adult touch production code. When it actually touches production code, we want to make sure we can justify it somehow. So how can we justify.

E

Yeah, okay, so for the sake for formality, we can just create an issue and link the issue to the pr and engineers justify the dpr and saying that we are actually fixing this issue because taking this fix backwards to stable release for the purpose of ci stability and does it make.

K

Are we fixing defects in our.

E

K

I, like I'd, have to actually look.

E

K

Every example of this uh it's kind of a case-by-case basis, like it's really difficult to say without looking at the the pr and what it addresses.

A

David, I hate to say um we're going to have to define this officially because it's going to be part of due diligence.

K

I think we have we've said that we um backport bug fixes explicitly said like defects here. Let me find sorry. I've got some background noise because there's construction, I'll I'll post the link to the document real, quick and see if the verbiage in there makes sense to everyone. Okay,.

E

E

K

So that's the link I posted in the chat and can you get like.

E

How bad is my background? Noise.

K

Right now is it like on? Can you hear me, or is it just.

A

It gets bad, then it goes away: okay, well, yeah. So, third, okay, sorry.

L

Your voice is significantly louder than the background noise either way so you're good.

K

Okay, that that's what I wanted to know I was to me it sounds really really loud, but maybe it's not as bad for you all uh yeah. So that's our our policy and I think we just have to justify the fits within that policy.

K

So that's where the argument needs to be made and if we're not comfortable with that policy, then that's a different discussion.

E

Wow well, thank you for the heads up I didn't realize we have such poses so yeah. I will go over it and I have some concerns.

A

With important documents like this being hidden, um so it goes for all the documents from licensing to contributing guidelines, code of conduct, release policy, membership policy and uh part of uh part of my discussions with the red hat opens office of open source um program.

A

They uh they really want us to organize these documents. um It's it's a big plus for us to to do this. To, for when we uh petition the cncf for graduation, we don't want to be zigzagging around the different repos and through directory structures to find policies.

A

So peter don't give me a hard time about my uh my prs for broken links.

D

Sorry, I can't help.

A

Myself, I'm trying- and I don't want to like just like slam in a whole bunch of different uh pr's- I'm just trying to do baby steps to organize our our library of stuff.

A

And I'm also trying to do like 100 other things.

A

A

Okay, any other topics.

A

um We're at 750 um how about we, uh we skip bug scrub this week, since we had a really nice one last week and uh you guys get 10 minutes back of your morning.

K

Sounds great appreciate that thank you.

A

All right, thank you, sounds good everybody thank you for joining us and we will see you around the project or or see you next.

A

H

H