Kubernetes SIG Network, 6 Jul 2023

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: [SIG-Network] Ingress NGINX meeting for 20230706

Description

[SIG-Network] Ingress NGINX meeting for 20230706

A

All right, friends.

A

Hello: everyone, it's uh July, 6 2023, and this is the Sig networking Ingress nginx sub-project meeting, which means that it is here's to the cncf code of conduct, which means be awesome to each other. If you have any issues, please report those to myself or Ricardo or to the Sig networking um lead with that we'll go ahead and get started. We always try to welcome new members. I do see a username on here that I don't recognize I'm wondering if that is Michael. But if not, if you'd like to introduce yourself, please go ahead.

B

Hey I am Brandon. uh This is from Microsoft here to check out what you guys are here. Working on.

A

Microsoft I hear those folks like Ingress nginx over there under sure land. Oh, they sure do.

A

um Okay, well welcome. Brendan. Thanks for uh coming and hanging out with us. uh I don't see anybody else. I did introduce. I did ask someone to join. They were asking about the mod security issue. You guys thought you remember seeing that one, the one with the performance issues once they flip it. It just shuts everything down in production.

A

I asked them to join, um but I don't see them on so we'll go ahead and roll through um old action items, probably a bunch of things that I didn't look at The annotation validations, that's the stuff you're working on. You said that that's ready to go Ricardo to review.

C

I, have it I, have it at a topic on that.

A

Okay, um I didn't put anything in the two hour release stuff here, I guess we could move this to your um to the permanent issue that you just put in. So we can start having a discussion there instead of being in docs and showing dots around okay, yeah I.

C

Have I have another another item for that as well to discuss.

A

This yeah I'm, just I'm just doing a quick review of this stuff, make sure that we're closing.

C

Out, no, no, that's! Okay!.

A

Oh, this is the issue that I was talking about. Let me pull this one up, so we can discuss it again. This is the mod Security One mod security issue.

A

We can review it, even though the person asking about it's not here.

A

We won't discuss the community survey unless Dylan comes back. I know we need to. There are some asks that we put out. We don't see any responses to the tweets, the dev mailing list or the notifications.

A

um We can continue to try to ask folks, but we just keep putting information out there and they're not reading it they're not reading it. That's the unfortunate part, especially with this 2o, all the deprecations in 2o, coming out.

A

Okay, that's the review the action items. um Ricardo we're gonna put a timer on the issue triage, because we've got to continue to make sure we do that. I know we're I'm bad at it. I try to do it throughout the week, but let's give it the 15 minutes that it's due.

A

um Are there any issues that folks know about? Besides the mod security timeout issue that we should review before we go into just triaging? What's there.

A

No okay go right back, go right into it, uh starting at the top of the list. I haven't even reviewed. This I haven't even looked at. This is actually we've got some new ones already.

A

um I'll drop this in here, so we can go ahead and look at it.

A

I can't see my shares my screen's still being shared right. You see the 403 issue, yeah.

B

A

Okay, so we got a third issue.

C

uh Yeah I think this one is it's right because right now the way that uh the client of uh works it will get. Your uh I mean it's, it's gonna, it's gonna, validate your certificate and then just say: okay, uh it doesn't match whatever I need so block it.

C

A solution for that would be uh to have some, let's say, um move uh the the decline validation uh to below, like the location, I'm, not sure. If this works, I need to double check, uh I think it's optional and and then it takes the location.

C

I need to double check that and uh maybe uh have some uh authentication leak nor uh field that says that all of the dash dot well known or anything that should be public, uh that you have rfcs or things that say that they are public locations by definition like uh Slash dot, well known, let's use it via, uh let's create oauth and all of those things. Maybe we can just skip all of the validations for those locations, but I need to see if this is like expected.

A

That's domain name checking.

C

uh That's the common name checking. So uh we have. We have this feature on on the client certificate of right. When you send the certificate you can check if the common name matches or rejects or like a a single string, saying like hey, I I accept certificate authentication, but just from James, even if Ricardo have a certain that on that CA as well.

A

So he's saying, when he's adding this things are breaking.

C

Yeah yeah, let me hold on James. One second, keep your keeping your uh yeah. It gets. The sir only does the validation yeah. So uh can you see uh also? There is like an SSL verified client own on on the snippet that he said: yeah yeah, just just a bit above yeah yeah, yeah, correct there yeah, uh the other one yeah with the underline.

B

C

No, the same the same value that you added, but instead of the dash underline underscore.

C

Yeah, the first one, the the yeah, the line just just below this one yeah, that's a selfie verified client. So the right way of doing that would be like having an option now. uh I think there is a flat on that uh he can set the flag for optional, but still uh feel uh that line below that line above uh it doesn't really failure, because uh there is no no match right so that validation should be inside probably the location or inside something else.

C

So you can have the optional it will shed and then you can and then you can say: okay uh if something is wrong, I want to send it to another patient somewhere else. Otherwise I can accept, but that's that's kind of tricky to do that.

A

Right, that's what the 403 is saying that it should match, and it's not.

C

I, don't think they have certain manager.

C

uh Yeah I'm not sure because the SSR verified client own. We already block him right because it's not optional, but uh yeah assign me to this. One I will try to take a look as soon as I have some time if you want, or if someone wanna drill down into a certificate requests, it's it's uh it's kind of tricky. What he's trying to do, but uh I think there is some some room for our Improvement here.

A

C

But the behavior is right: probably the template is from, but there is a need to to some extra configuration that the user needs to do or let's say we should.

C

A

Okay, avoid external uh external Hub, with rcrl I.

C

Think there is a PR for that. One.

A

Oh yeah, you do that oh Json localhost.

C

Yeah I saw the PR, but I didn't enter the ad I I'm kind of afraid of allowing local.

A

C

Because we run extra stuff on localhost right so.

C

Yeah uh I, will you don't need to assign this one to me, but that's on my my radar I will take a look and uh I think that we need a better way of doing that because accepting things to localhost may be dangerous for us right now on our current situation,.

A

Yeah I agree, but what I'm? What I don't understand is why.

A

Why would it do the extra hop if your external local, if your external Health URL, is localhost.

C

I think I think probably the user has like a second Ingress that contains something running on localhost or something like that. But I need to read the use case. I mean fine. There is already a.

A

Yeah this isn't this isn't fully hashed out of like because if your auth url is set to localhost.

A

Why would it yeah? Why would it brought out or externally.

A

That's what I don't get.

A

Do you want me to drop a from the fill out a little.

C

Bit more detail.

A

On this one, no.

C

Yeah yeah yeah I would I would I would ask some some scenarios explaining how he's testing this? How he got into this.

B

Long long generally pushes them to fill out the full like template, right, yeah and maybe just ask them to fill this out.

B

A

A

This isn't an issue I think it's because he came with both the pr and the issue. It's usually like one of the things long does ask for is like when folks have put in a PR asks them to put in an issue too explaining the use case. But I don't see any explanation on here.

C

Yeah, let's, let's just keep it simple, Let's, uh explain the use case on the authentication one. That's all, let's see how this goes. Yeah.

B

A

Ssl pass-through is enabled at the same time as proxy protocol proxy protocol doesn't work.

C

This is a duplicate. There is something else if you, if you find the issues, uh can I go to the issues.

C

I think there is a duplicate on that and the things that our current implementation of SSL pass through. It's it's bad.

A

It's like there are similar issues.

A

It's so pass through. I, don't see timeout.

C

A

B

C

So uh this is something that that I'm gonna uh take a look: uh I'm I'm, gonna refactor the whole uh SSR pass through today. The way that we do it's a proxy that runs uh inside the controller another inside engine, X Because by the timing of the implementation engine X, didn't support it uh right and uh I.

C

I think there is an issue that I have open uh to refactor, that, if you, if you, if you, if you search for that, you can just say, hey uh associate what we with the other and I, will take a look at that.

A

One opened by you, I, don't see one opened by you.

C

uh Remove the Three Edge accepted Maybe.

C

Or even try to search without the dash like just pass through.

B

C

There you go yeah.

B

B

A

They're, just removing it and having nginx do it. Natively yeah.

B

A

um I'm gonna go back and comment and just I probably should go back through and look at all the SSL pass-through ones. I, don't want to say that they're, your implementation or the native implementation will fix those issues, but we want to put them on the backlog and let them know that where you've looked at it and we're gonna refactor it yeah yeah it's down on the laundry list of things.

C

I mean the implementation, the current implementation works, but that's pretty in genetic supporting this, so we should I mean include that.

C

B

A

We do every other, Thursday is just watch me tight.

A

All right, we might have time for one more.

A

Default back in annotation is not working. That's not good! Sorry closing, all the other ones.

A

Felt back in service the handles yep, so they have a custom. One annotation has no effect. That's not good.

A

Is that the only annotation that they need.

B

B

C

B

No, no no uh I'm just saying I would need to check this because I've been playing around with default, annotations and yeah. It only needs that uh I mean default back-ends, and it only needs that annotation. But it's been working fine with from what I can see.

B

Oh, they need a, they need to set the custom errors right or something.

A

Is that what long is saying here.

B

Yeah, uh can you paste the link to this yeah custom errors, yeah.

B

Because the default back-end will only catch catch when you set custom errors. So if the uh your server responds with an error, it will root to a default back end on a specific URL foreign.

B

A

Can we link to.

B

That yeah yeah I'll I'll look into.

A

This don't want me to sign it to you. Brandon then.

B

A

What's your username again.

B

A

Yeah, this just looks like a support request. They didn't put in the the other one I've, never well I, keep saying this there's a lot of configuration options, I just never set for years. I just used the default, like I, just use a very basic Ingress when I'm using it anyway yeah cool, fun times all right. That was the 15 minutes on the issue triage.

A

We can use the rest of the time talking about some of this other fun stuff, the permanent breaking change issue. This is the thing you talked about in slack today, Ricardo.

C

uh Yep I just wanted to raise that I uh I've seen this um some repo that I use I, can't remember which one was but I thought that the idea was going to be actually great right. So uh that's a locket issue that uh you meet probably Gentile. We can. We can edit that and start adding the braking change that we figured out or before the release.

C

We just need to make sure that the approvers they know about that right uh and uh and then we can, we can keep a track on things that are going to be uh removed or uh or deprecated like on version 1.9. We are removing hap uh I want to make the change to uh disallow snippet annotations as a secure measure and tell people that this is a breaking change and uh we have renamed a white list to allow this.

C

uh It still works, but uh may stop working right so and- and this is this is actually I forgot to add. But this is the one that supports validation but validation. It's not uh enabled by default, so shouldn't be a breaking and change as well, right and um and then on version 110. We have all of those plans to to to to to to change, remove open, Jaeger, Zipkin, open tracing, zip, India gear data, dog uh and so on.

C

So I wanted I wanted to use that as a way to keep the track on things that we want to uh deprecame.

A

I I like it, um we can also put it out on the dev mailing list as well. Just so folks can track that.

A

List here too, um and also this is just for potentials right. This isn't what we're actually like what we've actually put out. It will be.

C

In the release.

A

Now the actual changes- or you want to use this as like gospel.

C

uh uh I think that we should keep on on the release notes, but also uh uh keep adding. So just so just a second.

B

A

Your headphones down.

C

No, uh my cell phone is ringing uh yeah, but that that's okay, uh so uh I I wanted actually to keep this open. So we can keep a track on things that that broke and people can just scroll down even not go into each of the change logs as well. But this is going to be like a really manual step by I. Don't want to do that uh uh automated I think it's it's a good way of just keeping the track and allowing people to see hey. Something is not working.

C

So did you take a look into, but this uh big child issue right as you've said on the version 2.0. If you take a look into the bottom, I have them Planet release with like removing all the SSL protocols, removing mod security and the whitelisters range annotation. So when we decide to do a release of that, let's say on version 1.11, we just move to the top.

A

Okay, um there was those changes um that was yeah, that was in the 2o breaking change dock that I made. Let me go ahead and add those because we had uh strict validations that the true that's definitely a breaking change, um control, plane, data plane, removing Jaeger, etc for otel and mod security, removal and or replacement, um and then Gentile has removing ajp support um that would probably not have to go into. Oh I could probably go in 190110. You've already got the pr out there right.

A

That was the reason why we moved the release uh to release branches right, yeah yeah, that's.

C

A

Not bad yeah, how do I can I edit this yep foreign.

A

And then, as we open up PRS and issues for these, should we link to them with them as well? I think.

C

I think it's a good I think it would be good for us if we do that, at least for the major breaking breaking changes. So uh yeah people can comment on the PRS right as an example, but not on this issue. This issue would be just for watching and not spamming uh people with a bunch of uh comments, random comments.

A

Now you've already got that The annotation. Those are The annotation changes. Sorry I was copying it in the wrong spot. No.

C

That's okay, we can, we can. We can take care of that later. I mean just like put add some uh some placeholder for the things that we don't know the category yet and we can just keep moving.

A

Yeah validation.

A

A

Yeah, let's see what grammarly says about your English.

A

Okay, I, like it um definitely.

A

Oh you've got the disabled, annotation validation. Oh it's defaulting to false. That's the stuff! You just finished up, okay, never mind, yeah, that's fine! um I like it. We can send out a notification for folks to watch this.

A

um I'll I can send it out to the dev mailing list, drop it in users and tweet it out.

B

A

Cool you have fun your next fun, one validation, the validation, PR's.

A

C

Yeah I think it's actually passing out of the tests. It's ready to be merge. It I, uh yeah, cool, great right and take a look into the just 5 000 lines. I think it's it's! Okay right! We can imagine how to check in so.

B

A

101 new files.

B

C

So uh uh so this thing is: is ready to be merged. I I know that there are going to be some needs for improvements on the rejects. We are relying a lot of uh on rejected to validate some annotations even having some common validators as well.

C

Right I, try to document all of the flags that can be used and keep the default right now as disabled, but allow people to enable that, instead of even doing a breaking change on 1.9 and saying hey, we have validations, but validation is disabled by default and we are going to enable that on 110 right uh but uh I mean I.

C

I know that Gentile, it's gonna, probably take a look into this uh until next week, but uh as this thing is kind of dragging for some time- and we have some uh priority to manage this. If, if you understand me, uh I would just ask people to really take a look into that see if they missed something. If they figure out, there is something uh weird and otherwise just getting that merger.

A

Yeah, did you add an end in test where it's flipped to true.

B

A

And that's what this one is.

C

C

I mean this is the one that we block. This is the one that will block based on on the web hooks right and uh there there are some uh all of the validations. They have unit tests, at least on the on The annotation unit, as they have passed it yeah, and uh there is some STAR test as well and on the starter that we were or or work based on on, the parts are being enabled or not right.

A

Yeah that's what I was thinking is that we just.

C

I've spent more time writing tests than writing the code. Actually, in this case, yeah.

A

um Because it's a it's a config Global option right.

C

uh The flag that enables the validation is the flag that allows uh that changes, the risks and the cross name. Spaces are config map.

B

C

What is disabled by default? It's it's an it's! It's false by default, so it's not disabled.

A

Yeah, what I was.

C

Thinking, sorry, it's it's true. By default, it's it's disabled. The validation is it's disabled. By default,.

A

Before we run it into one, nine or 110, or whenever we want to flip the flip it to doing the validations, should we run a whole set of the helm in in the end test with it set to true.

A

A

I'm just thinking we put it out there and it's gonna break people's clusters and we'll have another issue like we did with the path validation, annotations.

A

C

A good idea, I can uh I mean I will I will keep my. My problem would be on the test that actually we we enable or disable the flag right, but other than that uh I can um yeah. Yeah I will create just for one release, because I think we are not messing. The way that we deal with kubernetes, but just with the annotations right and uh and see how this goes and what's going to break uh yeah sounds good. I can do that.

C

A

I'm just trying to prevent us from heartache from the next time, um so the question is n2o, we'll flip it to false. So it's always validating, because that.

C

A

A major change.

C

Yeah yeah yeah, so.

A

C

A

B

A

Yeah so we'll roll it out to 1 9 and 110 with a disabled, and then 2o will flip it to True. Would we ever want to do that in a I mean there's going to be already a lot of breaking changes? Yeah.

C

I think we can. We can actually do that in one to ten, because uh we usually do breaking changes between Amino releases, not patches, and this is the idea of the the branch as well right, like I've, been released, Online, release, 110 and then on release. 2.0.

C

Probably because it's going to be a major architectural change with control, plane and data plane, then we should do on the 2.0, but the validation I would just keep on the next, not on one line when I'm going to keep disabled as they do on kubernetes with, like future flags and and the apis right. So we can keep it disabled for one eye. 110.

C

111 as an example like we say: okay, it's it's stable. By default, the test they are passing right and all of those releases so.

A

We can not just anything about it. Okay, that's fine.

A

Cool all right on to the next ones: disable snippet annotations for one nine yeah.

C

uh We have we had this uh this uh flag for some time and now and as one to nine, it's gonna have some breaking changes as well as we are removing a JP I wanted to flip this flag to a true and now. If, if you want to allow your users to use an epider notation, you need to flip that back to false and announce that, because then it's going to be like an assumed risk right. So we gave people a lot of time to take a look into that.

C

We know that some folks they are gonna, have like uh uh complaints about like hey you, don't support headers, you don't support something else, and we have that on on the backlog right, but uh I think that it's going to be good, that people know that they should now assume the risk of enabling these Snippets.

A

A

Okay, I'm still concerned about that one being said to disabling them.

B

A

What's the ramification if we disable Snippets and someone has Snippets that they're using so in an upgrade it's going to cause that would be a breaking change. Yeah I! Don't think we can change I, don't think we can flip it for one, nine, okay, okay, from a user's perspective. If they go to upgrade to one nine and this stuff, Snippets are disabled, they don't read the release, notes we're going to get a bunch of issues open up.

A

2-0 gives them that, like we need to go, read the release, notes and see what the breaking changes are.

A

Deprecations I think are fine for um minor releases, so I'm moving on to the next one I, don't besides datadog. No one else has yelled about the other ones right, open, tracing, Zipkin, Jaeger,.

C

Yeah and and data they have a plan as well right. So I wanted to state that uh we are not going to remove that on 1.9, as they asked us for some time to implement that. But we are going to remove that on 1.10.

A

Oh where'd, it go I thought it would have like put it at the top here or something.

B

A

That issue back up I penned it I thought pending. It would do something different.

C

um It's it's on on the main repo. If you click on Ingress in Chinese,.

A

Yeah, so that's this right here and we want to do that for 110.

C

Yeah I think it's a good commitment.

A

Okay yeah. Do we have an open issue for comment on that I know, data dog has put in the they've got their plan, but.

C

We have an issue for the deprecation yeah.

A

Okay, I just want to add that, like the list, I just want to put a list of like deprecations like we'll give people like hey here is here's the notification about plan deprecations and our changes in this penned permanent issue and here's the individual ones that you should be aware of that are coming and the moving everything open. Telemetry is one of them.

C

It's going to appear there.

B

A

I'll write it up, so that'll be one of the things I can ride up to just let folks know about all of this new. All of these new things.

A

A

um Yeah, let's go ahead and look at this one, let's probably be the last thing we look at um so I was talking with this person and it's very interesting this one. So again, this is a scale problem.

A

They have a large amount of ingresses and then only in the production environment when they turn the security rule. Engine on everything goes down.

A

I asked them to put in some more information about, like the scale of like the number of ingresses, the rules that they have enabled I think they put out some of the rules that they have here, but I I don't see anything off the top of my head.

A

I, don't know if there's any issues with the because we're running the latest version of The Mod security Library, which is one of three and I, don't I have not looked in to see if they have any scaling issues but again, I, don't know how large of an environment that they say that this is any other thoughts on mod security. Just deciding to give up when it's supposed to do its job.

C

What goes down is the controller, or is it in gynex.

A

The sights are down so I'm assuming nginx, isn't proxy, and once we turn the mod Security on that's why I asked I asked them what the CPU memory utilization was once they've turned it on.

B

A

Requires a restart right.

C

Yeah it needs a it needs a it needs a Reload yeah when you set it.

A

So this is one of the original audience is in place, everything's fine, with detect only, but as soon as he flips it on all the sites.

C

I think, last time what I said um and I uh I didn't check it, but uh it's uh I have this feeling that when you are on the set detection, only what happens it's that uh and it would be really helpful if they have some logs on somewhere of the engineers yeah. Here we go.

A

Yeah, so I was looking at that and um I see a lot of.

A

You know I downloaded it already. Oh.

C

I would say that probably this is uh a timeout.

A

um Stop sharing and then could I hit, stop recording.

B

A

Stop sharing I, know, I, don't know where all these buttons are um so I see a bunch of these errors from the certificate from the Lua code, but I also see the mod security. So the 400s request body exploding files is bigger than the maximum expected yeah.

C

A

A mod security error right, yeah.

C

So what happens? There is uh when you, when you set in any- and you enable the the body inspection uh and let's say you, you enable the body, inspection and you say like a PDF or 10 megabytes.

C

So my security won't won't validate that because uh it's gonna say: okay, I have this maximum of like one megabyte or something to validate on more than that. I can I can right to protect it from uh from memory problems.

C

A

Are these? Are these errors normal?

A

C

uh This is so those efforts they are probably so so it seems that someone is doing some scan right uh and and testing some random servers there right, but not passing the hostname or something like that or even there is like hostname customer one. So they may not have the certificate set on this uh on these Ingress.

C

So it's just getting that because there is no certificate configured which is which is fine like so it's it's using the default.

A

A

Mod security is doing what it's supposed to be doing.

C

A

So yeah you get an access denied, do they just needing I, don't know what these hosts are doing and I didn't ask that so so there's the stuff yeah. It looks to be just the same error over and over.

C

Yeah, so uh this one it's because they don't have a a certificate right, uh another one of on what security yeah, the other one on what security, and they would probably a help on that. It's because hold on server post configuration unexpectedly or 400. uh It's failing to reconfigure, can I see it like it's failing even to reconfigure the this is. This is actually interesting.

C

It's it's trying to reconfigure something like the back ends and it's not working access denied that called 400. uh parameter. Hey hold on this is this is actually nice. Okay. uh This is this is great James, uh okay, here's the thing: okay, okay, I I see the problem. So here is the thing we have. This slash configuration, slash, servers internally right on our internal part: okay, okay, fine and- uh and we pass a Json with the whole configuration that should be set on nginx right, okay and when they enable mod security.

C

What we are doing is actually we are inspecting our own internal request, because the Json body- it's too big, and then it's denying.

A

So it can't so once they turn it on, because they have a large number of Ingress controller objects. Security is not letting us yep, that's fine, yep, it's not, but.

A

C

We should ignore security for internal requests.

A

Yeah, because, okay, so they just need a ad and mod security rule that ignores localhost requests to this to that yeah.

C

Yeah short term, it would be, but I can take a look and do and ignore that, because it doesn't make sense actually to have much security running, because this part is just exposed internally, and we don't really don't know what what secret is. So uh it's it's. Just like probably adding to the template, saying like to the local uh mod to the local configurations. It shouldn't be passing too much security, exact rule engine should be off for.

A

C

Yeah there, let me, let me take a look here and.

A

um Script right when we're doing the default, nginx rule or default mod security rules.

C

A

C

Actually add to the template and just say skip, you know.

B

C

A

And tell them to add a mod security rule to ignore, ignore that Port that host to that URL and.

B

A

If that figures.

C

All right, yeah, I, I, found it hold on there. Is this uh uh I think this one yeah I'm gonna, send you the the link of the line here right and what we should be doing is actually on on this case.

C

We should be just ignoring.

C

A

Oh, so you can just put a block in here in the server block and say, ignore mod security and allow it through.

C

uh uh Yeah yeah I think you can. There is the configuration of what security is on uh uh it's it's inside the server definition I think, but we can set mod security that there is already one like Insurance Security will not run on customer or pages, and we should probably set this uh on the same here. This is the line we should set this to the internal configuration as well. This one I've sent that to you.

A

Oh yeah I see it right now, mad Security on and for that yeah. That's the Roblox. We can just set my security off yep. Okay, that's.

C

Pretty amazing yeah my whole recommendation would be tell them to change the template. They are using a pretty old version, so if they are willing to update to one a I think we can, we can raise a bug and share it Peak and do a release of 182 for them because they are using 164 right.

B

C

Otherwise they they would need to uh to change their template. For that.

A

Yeah they wouldn't be able to easily change that and still stay on 164.

A

There's no snippet to change, because that's all internal to our configurations that they added they tried to do something on localhost it wouldn't work. I would think I will I'll. Let them know we've identified the issue, we'll put a fix in for 182 and let them know what the actual issue was and if they want to change it on their side, they can.

C

Yep, it's just adding this line like we can. We can. We can create the pr and tell them hey. This is the change that you should be doing on your template and it's going to be really like a one line, change and that's all right.

A

Okay, well, that was fun, so I've got I've got a couple. Emails to I've got an email to write about the deprecations and we'll put in this bug, fix 182 and we'll see what folks think think about all of our breaking changes coming up or any deprecations.

A

Okay, thanks for joining today, folks.

C

See ya have a nice weekend.

C