Kubernetes SIG Network, 14 Nov 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Kubernetes SIG network meeting, 2019-11-14

Description

Kubernetes SIG network meeting 2019-11-14

A

All right, we are recording the kubernetes Signet work meeting for Thursday November, 14th 2019.

A

Why don't we started right off with our standard-issue triage robbery, yeah.

B

I'm here, but I'm still having some technical difficulties. So just the triage.

A

Sir I, do you want us to wait a moment if we come back to you and in progress? I have.

B

To connect to Google's own video chat to make it so everyone else can hear it's easy. So.

C

While he's getting that set up, we could talk about the item that I like just under this, which is yeah I, noticed that when we first started doing this, it was taking us a long time, because we have this huge backlog of issues and not the last few weeks there have only been like a handful of issues to review, so should we think about maybe trying to deal with our PR backlog or reviewing flaky tests to make sure that you know things aren't or you know something like that and have an alternating series of you know, triage events for our meetings.

D

Are we live here? I love that idea, especially the PR backlog. We the project overall accumulates about what is it a hundred PR is into our backlog, every new release, so every like consistently every release. We just go up by about a hundred years, so we should try to make me burn through those that'd. Be wonderful, wonderful thing to start triaging Dan. Are you volunteering.

C

I guess maybe I am I could lead it at the next meeting. Yeah.

B

Let me go ahead and get it into issue triage if that works, for you share my screen.

B

Alright, can everyone see this cool, alright, so support external traffic policy on Cluster IP? This is a very new one. It's been around an hour, not a whole lot here. I think.

A

This is duplicate if I recall correctly, I think it's a duplicate, I, don't remember the issue number, but you can assign this one to me and I'll try to track it down. There's another issue.

D

Like I just approved topology today, so like, isn't this exactly topology.

D

Assigned to me, I will cross-reference the typical stuff and I think and.

E

I think this also references PR I, have open on the gap.

E

In this issue, so the previous jobs are in presence, references, okay, oh.

F

Yeah yeah because.

E

That has to do with detecting and not detecting. Yes,.

D

B

All right, I was easy. Next, one downward API volume should have huge pages exposed to pawed through resource field. Ref looked at this just briefly, it's.

D

B

Siga they tried every cig, so it's.

D

Not sick, Network signal.

B

Alright, so this should be signo to only correct. Yes, I'll come back to this one and pull other SIG's off of it and remove the triage. That's every everything they could think of yeah. Okay, all right external traffic policy, local, doesn't work when the firewall is running firewall, firewall D is running, I I, don't think anyone has looked at this yet.

B

Any volunteers on this one I guess you can get it to me.

C

Yeah dan would know not that D cbw dan Winship, okay, perfect.

B

All right, node port allocation, conflates tcp and UDP.

B

Not able to connect to the port, sometimes using the same port number with different protocols. That's what this is.

B

D

C

I feel like we have an EDD test that verifies that this works.

B

Anyone want to follow up on this one I.

C

Guess you can give that to me too.

C

But I'm done for the day.

B

All right adding new handler post ready hook. This is more of a proposal specific to HIV ingress.

B

Wanted to have something like after readiness succeed, yeah, sometimes.

D

B

D

Gonna sign this into me: okay, okay,.

B

Should I just assign man hon, okay, yeah, I'll battle,.

F

B

Kubla supports IPS and Mac cni capabilities.

B

So basically, why can't couplet support I mean.

G

Cubelet doesn't really have much to do with this, because where is it going to get that information from to pass through sure we've.

D

Had we've had this conversation a little bit or something similar up about? Should there be a namespace of annotations that we automatically pass down on me to see nine plugins like a particular prefix or something I forget what happened with it, but I mean obviously it didn't get implemented.

G

Yeah, so we also we pass annotations down to the runtimes and it's sort of up to the runtime itself to figure out what to do with that. So it's like this could be hooked up to C and I, but you'd have to standardize like Tim said that prefix or something like that, like the pieces, are mostly there. But it's more of a political question, not a technical question. How far we want to go in kind of having a pass through essentially into the network. Plugins.

D

Do you want to you want to respond.

G

To this one yeah I'm not sure what you just said but tightened it up sure I'm not sure how much that's gonna help, though, because it's more about, like you know, if somebody wants to try to propose standardizing this stuff, you know I, guess they could bring it to the sig and talk about it, but I feel like we've talked about it before, and we've kind of punted it outside of cube. Well,.

D

At least at that point the issue would be out of triage and we said, like it's a legit feature request right also.

H

Dan is that this one of the recent things we did in the network plumbing group to.

G

Some degree yes, and that's one way to do that kind of thing, but this would be more. It sounds like what this is trying to do is standardize side and communities itself, which I guess would be a great.

H

Thing but there.

G

Are reasons why we have decided it not to simply.

H

G

H

I'm good epidurals is right. My point is simply that you know we did accept it in the network plumbing group, so it seems reasonable to contemplate. You know, more broadly and even in anticipation of eventual harmony with the workers and.

G

There was one group right: you know there are a lot of considerations around it, so yeah I guess we'd accept some kind of kept for that, and then we can actually discuss the specifics so yeah. Why don't you just assign it to me.

B

All right, accordion s and an H, a cluster 1.16 behaves unexpectedly.

C

F

Assign that to me or.

I

J

B

All right incorrect pod broadcast address ioctl.

K

K

Butter is a man-thing, a.

B

D

Casey we could start be thrown those at you is that so I mean.

G

The other option is to tag reg at Chopra on it, since he is actually doing a lot of flannel work. These days, I have a turn for my jet in a long time. Why.

D

Don't you that Raj.

G

It's kind of a punt, but yes, since he does it a lot, give it to him and see what happens. Cool. Tell.

K

Him he's welcome to come to our meetings. You know we miss him. Well, he's that invidia now, oh right.

K

Yeah he can still come. That's fine.

B

This issue really does not have a lot a lot of information with it, just an Apache bench and the output of that and Cooper nice 1:13, claiming an IP tables to proxy problem.

B

No one has responded here yet doesn't even say what the problem is correct.

B

Yeah during a deployment rolling update.

I

I'll conduct American take a look at it. Okay,.

D

This is one that you can push back for more info or clothes right like it. Maybe there's a real bug here, but there's not enough for us to even reproduce it yeah.

I

Yeah I'm gonna have king really more information on something. That's helped us to today's flight. Great.

A

I think this is one that I owe a response on still and I. Have it looked at got picked up a couple last time we stand able to get to him yet yep.

B

Yeah, okay yeah, so this may be getting into overlap now, with ones we've already covered, so just making sure we haven't already.

L

This one okay, I, remember: I'm, engaged okay.

B

B

This one no one's assigned to it note court service with external traffic policy. Local, doesn't work well for internal traffic.

M

E

F

B

J

B

So yeah we should close as working as intended or not working as intended.

D

D

Look forward to topology replacing external traffic policy entirely, so we are thinking about what external.

M

B

All right, Cooper, OxiClean UDP contract after node network resumed.

B

And actually I think Andrews already on this one. Well, yeah it's been closed and reopened, but I think this is still yeah.

F

He followed, oh okay,.

B

Cool and then the last one service, the pod communication, is broken and looks like Casey's already on this yeah. Okay. Well, that's it for triage this time.

C

Thank you very much, so I'm signed up for PR triage next time, but that two weeks for now is Thanksgiving. Are we meeting then, or are we just skipping that I mean.

D

The Europeans and all can meet, but I'm gonna be on vacation, so I think our next call will be a month from today or four weeks from today.

F

All right, yeah.

A

Sounds right to me and so next time we've got Dan Williams for some dual-stack discussion. Oh man skip.

G

That might as well just take it off the agenda for now, unless anybody remember.

A

There are a few other seven points from from Andrew and and Dan Winship. If you want to talk about those or yep might as well.

A

Alright and you're doing that.

F

Sure I think I wrote it so I'm trying to remember. Thank you what I said yeah. So my biggest question is, if there's a node that has one v4 internal IP in one v6 internal IP. What is the mechanism that we're using to let a user pick which one is preferred for you know generic couplet couplet actions like giving vlogs or download API things like that? Should it just be the first IP that the cloud provider sets or do we want some other way to for you to specify which one I'm.

G

Not sure first IP is specific enough. Well.

C

So so I responded to this with right. Now we always take the first ipv4 IP, either that the cloud provider returns or that we find on a network interface, and it turns out it's pretty simple. If you just change cubelet to allow you to say, node, IP, :, :, meaning the unspecified ipv6 address, then we can have it pick the default. The first cloud ipv6 address where you first ipv6 address, found on an interface and it's a pretty small patch and seemed like maybe.

G

A good way to do that. Well, and you said you got a link there dan. This says the API server does that. So if a server used.

C

To have the oddity that if you said, if you told it to bind to the ipv6 address, it would still advertise the ipv4 address and and I changed that so I.

G

What I'm saying is, it seems like having consistency between opponents is a good thing, not.

L

Deliberate but the thing that my concern is how deliberate this is I.

C

Don't think it's it's delivered, I mean it seems like all of these single stack, ipv6 stuff was originally done, and the dual stack was originally done with the assumption that, if you have any ipv4 addresses, you prefer them to the ipv6 addresses, and we can't change that now, but we can at least make it possible for people to specify that they want to prefer the ipv6 addresses I.

G

Mean part of what dancing is that? Just because you have a node with v4 addresses does not mean that you to necessarily run v4. It may be the case that you want to run v6 single stack in a setup where a nodes have both v4 and v6 connectivity, but you effectively want to ignore v4 connectivity and that currently doesn't work.

C

And that's what got me started working on this, although it does help with the case of somebody who has a dual stack cluster or wants to prefer the ipv6 addresses as well. Yep I'm, probably going to file a PR with the patch linked from there soon. So, if people can be watching out for that, we can discuss it there or if people have comments now or to think it's terrible and want to stop me before I file. It I.

L

Think what you're trying to do is right. I am not sure. If it's everything we need to do, yeah I mean did it.

C

May end up needing more patches to other places after this and I wasn't quite sure what to do about cloud providers or external cloud providers.

L

I'm gonna throw a curveball at you today.

L

The publish publish address of API servers, still single stack, I know it's somehow not exactly the topic of transfer for, but it is related. So in a lot of ways, we've never we've never sat down and said: look the cluster has an infrastructure bank owned communication, that's happening somewhere between API server, nodes, nodes, an API server right and the way everything is configured right. Now it's discretely configured.

L

You can configure this one way and then the other thing is one another way and so on, and even our dogs doesn't talk about these things explicitly like hey. The infrastructure connections needed by your cluster is gonna, be explicitly described as whatever in node object or whatever an API server configuration. We don't do that, so my vote is I think we should do what you then wants to do alright and then we need to take a take some and think about this entire infrastructure connection or whatever we're gonna call it it.

L

Just some sick total users yeah well, the thing would be configured that way, no matter how good and then you'd have control over, you will have control over IPS or whatever published, IPS or families and so on. So that's that's my goal, but that will of course, I'm thinking not something I have from top of my head right now.

F

So I'm still confused what happens if the people it doesn't set note IP in.

C

That case, it falls back to the current default behavior, which is prefer ipv4.

F

But is it prefer ipv4 today, yes, like in the note in the cubelet like when cubelet does get hosts, IP or whatever it just picks the first internal IP? So couldn't that be.

C

For the cloud provider case it might pick the first one that the cloud provider returns. I, don't remember if it may, for the bare metal case, it definitely will prefer ipv4. If there's any routable ipv4 address on the system, it will use that over ipv6. So.

L

I'm speaking on behalf of Antonio, he was posting on that he cannot speak. He was saying: go line, has a preference for before. So, if you don't bind anything, it will burn to beat the people. That's what what I think is saying at this? Also common can't attack.

L

N

Can you hear me I? Can yes now the thinnest? We have this issue with with Quran and a twist I. Don't remember this. The problem is just not with the when you bind and go binds for a bill for, and the other problem is with the DNS go. Ron yeah falls back to IP before for all the miserable addresses. So that's the problem here is is that it's just not a matter of changing changes.

N

We need to understand so all these things that are binding to pores if they are using whose names what IP addresses are using is very complicated. This problem.

N

I'm, the thinnest you can choose a video use. A cross name is going to fall back to twelve pp20 before and the problem is just up to me. Excite PPC's and host names. You are going to have a system that is working in in with two different IP families.

N

The server prefers APB six, so they the hosts, the operating system, prefer ipv6, but the go along library, professor, before.

D

Okay, so we're gonna look for a PR from Dan and keep going with that and.

L

Just just a comment on how and cloud providers at least o'clock providers I know that has supported the two finalists. They actually prefer turning ipv4 first. So that's one other things to think about, but for now yes, so.

F

So what do you mean by preferring actually for like they're, just ordering the the addresses on the node yeah, the.

L

Fabric itself was designed because all the fab, all the cloud on the line fabric was designed before first then v6, all right, so it returns the v4 first then v6, it's a side-effect of how things were the lemon tea.

F

Okay, so I yeah I, think that was kind of the answer. I was looking for like if that's what that's a pattern, we're following like that's: why I'm weak yeah? Okay, thanks! Oh.

L

The short term and I think yes, let's go with with the paradigm is proposing, and then we stop and think about this in a bigger like all the angles, because obviously we need to secure.

A

Okay, thanks thanks. Everybody next item was Satya on the side: okay, yep.

E

So the captive dealer for the camp is out I, think Korra okay to test today. I just want to bring it up and if you want to discuss the details here, but if he is hoping we can start the discussion on the PR and get to some sort of approval. So I did leave the implementation section a little bit open because it's still not marked as implementable I just wanted to see.

E

If you have consists in the direction and then get that very straight and I can follow up for the actual implementation section, because just adding three flags and then the question is since we're just adding Flags. So we have to go through an alpha which a gate or something or can we just and the flags right because often.

O

Behavior doesn't.

E

E

So I want to leave the implementation conversion independent of the directive to it and I'm hoping we can get this.

E

Of course, discussions of.

E

There's already some discussion, there were some comments. Reply to those yes.

A

So sorry, I've, taken on the carrot, I have some pending comments, so I am hoping to post this later today, or this week at least so and I'll be coming soon. Thank.

J

A

All right, I, guess I think we'll probably leave rest of the discussion of that to the cab yeah, but thanks for calling it out keeping the scientists.

P

Why can you do wanna, go next dose sure so.

Q

As everybody knows, and probably doesn't like to admit, it's code for you today for 117, so I thought I would just go through the four PRS we put forth for 117 and make a decision on if we want to go for any and get emerged or if we want to just let them lapse and I'll. Let the release team know so I'll, just pop the hood. This shouldn't take too long I.

Q

Hopefully you all can see this.

J

Q

Me know so: I had put forth these four pr's I'll go through them just quickly. One is just at support for cube proxy in iptables mode. I did a quick look through this.

Q

It looks like it needs a rebase and hasn't been touched since October 13th, where Antonio was calling out, can we take a look I, don't know this is probably the biggest of the PRS that we have do you think we want to go for getting this through I, don't is Valerie on the call, so I think I could follow up with Valerie and see if this is actually ready. Xin I think we need a rebase on the meta proxy abut. Do you think we should try and get this through?

Q

I can raise an exception or hunt it to 118.

D

We need this for we're not going data with dual-stack right nope, okay, so we need to still mater, but we're not going data I. Guess it's not critical. I.

D

Don't think this is worth heroics. Okay, somebody tie will be.

Q

I mean if there's a lot of work than HD down here so.

L

I thought we go for we go. Was this your early next release? Okay and push so put the V 118 as the beta that they did release we go wait. I was told stack.

Q

Okay, let me I will run that by Valerie, so she's not surprised about the air come here.

L

Make sure she's.

Q

In agreement and then go go from there, thank.

L

You yeah was this and then one sly stuff that we've been working on. We should yeah feel complete, so dog and the api's I, don't think will change the api's. Just me the two volume of agony that this one generates. Oh.

Q

Okay, so I will follow that one up this one you had raised cow I just want to check as this bin sunset for Rob's PR. Yes,.

O

Q

5 24 6, which is merged so that one is in yes, just a quick question for me because I'm going to write the docs does this require documentation, changes? Yes,.

L

We have one entry that says the endpoint that do this tag is mutually exclusive within the end points less no and.

Q

That's now not not.

L

Q

I will get that documented. Thank you and thanks Rob for getting that in and merged downward dual-stack down with API. This is by an ish I. Think it's good to go. Andrew had commented on eight days ago.

Q

Anish has responded to that. It has an LG TM. It's looking for approval and I think this souls Etsy hosts and down what API to grab to publish a list of IP addresses my pod IPS. So this will give you if you require either down with API for status pot, ip's you'll get a list of IP addresses and we'll also go and throw the the address in there in Etsy host in the pod.

Q

Do we want to go for getting this I think it's gone through several iterations I think Tim took the past several months ago and I can't remember this one or the the cluster side of thing.

D

Yeah, no I don't want this one, but the result that I just saw looks like what I would expect so lgt em doesn't mean smart people have looked at the code, yeah.

Q

I guess Andrew looks to have looked at at the most recently, so I, don't know if Andrews in a position to given is it we were. We were in on here. Do we it has? It just needs an approve and a milestone.

L

Anyone approval somebody with.

F

Approval right, yeah, I, just reviewed it, but I can't approve it itself.

Q

What's that am.

D

I assigned, can you scroll? Yes,.

Q

All right to me, driving I, need to learn to drive.

D

I have a tab open with all of my assigned. Prs and I will go through them and at least try to pick off anything I can pick off today, no procedurally, next time, let's not make the code freeze the day before people leave for Cape Cod.

O

But last time we did it last time we did it the week off, keep going so this is marginally better I'm, not sure I agree actually well.

D

You know we have more free.

O

Q

You want me to put a Sentinel in here, so when you show up you know, we've we've blessed this together, so you can add just a rubber stamp. It I.

D

Don't know what that means, but sure why not well.

Q

A breadcrumb to say: hey, we talked about this and Andrews happy with it. You can approve it. Oh I.

D

Can remember five things: okay,.

Q

Good good, yeah, well you're doing better than me. I can only remember one okay, so that's good! I all milestone it. So I'll put it in 117, it's like because it's not in there now milestone. Is it 117 or B 117 I can never remember that bloody thing. Usually.

D

We put the V put.

Q

Q

All right cool, thank you. This is allow multiple node, so además and cube controller manager, so we'd hard-coded this to be a slash 24 in 1/16 for the v6 prefix.

Q

This allows you to make it configurable I. Think it's in it's ready to be reviewed by Tim or Bowie. I think there was a redesign and a niche confirmed that he went and did that redesign. So.

L

That was reviewed and the outcome was because I'm back turning within each that yeah, we should have different arguments for for the cider cider mask.

D

Implementation on this, yes,.

D

All right, it's assigned to me and I, will work through to my cue today and.

L

D

The way andrew.

L

Has been doing a doing the job of keeping up the body honestly just checking everything. Sorry lucky.

D

Can you go back that one yeah yeah I'm, sorry, sorry I saw myself in the reviewer list am I in the assignee list? Yes, okay, perfect.

Q

So that was it of the four that were put up. There were some ones that were ecosystem like adding it to kind and other things, but I think they can wait. So the only thing that we're punting or have agreed to punt is the IP tables the rest are going to be put in. Is that sound okay.

D

Q

I'm gonna give up the mic. I may chase people down about docks in the next week. So if I sorry.

D

Go ahead and finish: I have a comment afterwards. That's.

Q

It I just might ping you on anything: that's merged, yeah,.

D

Guy I expect other PRS that we want to get in today, I'm. Looking at the list of peers that are assigned to me, which I know is a subset of all pr's like the Debian IP tables, one Dan.

C

So it looks good, except that it seems to be breaking the ete kind. I PB six inexplicable ways which seem to have nothing to do what the VR is doing.

C

It's possible they're, like pulling in the new Debian thing, has changed something that breaks that.

D

Environment I don't know well, I mean this iptables incompatibility thing has sort of festered for a long time, but it's pretty in our face now. Do we need to do something with this or I mean I? Guess we can justify this as a bug fix, but he's kind of a big back patch.

C

Yeah I mean well, you know it's not changing anything in kubernetes it well, I, guess it. It changes the images that we ship, yeah I, don't really know what to do. It's passing de GCE.

D

Some kind yeah all right look I mean. Did we get some of the kind folks to weigh into.

N

It I was chasing this IP pcs, that PBC's kind jobs are are flaky and I was hoping. That was a problem with a big table person, but it doesn't seem so.

D

Okay, so I don't think we can merge it as long as it's red right.

N

I, don't touch this job, I mean.

D

I, don't trust the job either, but if it's it's gonna be ready for everything. From now on that it kills the signal. No.

N

About well, we have to kind I peepee, six jobs. This is a canary. This is a job that I'm running in parallel with the full suite of end to end. The conformance test with ipv6 are are passing. The problem is with the job that is running all the end-to-end tests. That is failing. Okay,.

D

All right, I'll leave this one in my queue, I was hoping. Maybe I can kick it out, but well I'll. Look at it today to see. If there's anything I can do with it. Do you do you recommend that we merge well I, don't know if it'll, let us merge anyway right.

C

D

C

Test, that's failing is not required, so we could merge and then hope that we can fix.

L

The kind test later, how is it doing with other tests on the test? Brett, because sometimes things get broken there and we don't see it. We are.

N

We're 36 Jimmy yeah.

L

Mip v6 and other tests as well.

N

With a PVC sahaba.

L

N

L

The image will probably have some application on any tests. That I mean.

C

All the other PRS passed, except for the container d1, which has failed the last 100, runs so.

C

Although I don't know to what extent the other tests actually make use of the changed images, I think the.

L

Changes do they the Puffin changes, I.

C

Mean, like you know, the integration test probably doesn't even run the images it just you know, runs stuff directly right, 7.

D

All right, I'm, not sure what the right answer is I will see if I can discern that. But if we miss the boat on this is like how how.

C

Bad is it really? It just means that 117 will have the same warnings as 116 does about you know this will work in some environments and not work in others.

D

All right, it's not it's not the end of the world, but no.

C

It's not makes.

D

It worse, okay and there any other anybody else feel like they have PRS. That are, oh, my god. Why isn't Tim reviewing me.

M

What happens with the one? That's in conflict that you just approved technology.

D

M

It's the middle that I didn't China. No, that's a good point.

D

Yeah we can get exception like it was lgt ended. It was approved, it's just that it meant that we know there's just a conflict. I think we can justify that.

D

A

Next, all right thanks, everybody.

A

A

In that case damage it, let's know that this keep dumping.

C

I just figured there should be something on the agenda in case there was anything that we wanted to discuss before cute gun. It's I mean. Usually we do a bunch, but I don't know if it's too late to plan that at this point or if people are busy with other things, I know I have more and more non cute cotton sorts of meeting things going on yeah.

D

I would love to get together even just super informally lunches. You know, I could do Wednesday lunch or I can do like Thursday afternoon. We've got the Signet deep dive and then we've got something gross stuff right after that we could just sort of hang out after that.

C

Anybody anybody, Wednesday lunch I think there is a network lemming working group thing going on then it's my remembering the timing on that right. I think so so that might be bad I.

N

L

A I have a question: you guys talk about stuff, fun meetings and so on. There's a place. I can look and see these things happening and I'm.

C

In, like you know, with co-workers and customers and stuff like that, like like stuff separate from the I'll, get the actual agenda, yeah.

D

Yeah my calendars filled mostly with customer meetings and stuff, but I Thursday. Basically, Thursday I have thing fixed on the calendar, except for signet intro deep-dive, something about ingress following up on that proposal and then nothing so anybody's gonna be around Thursday. Maybe we could do that.

D

One time we have time for lunch. How about two o'clock.

D

No I mean I had dealings before this. Not two o'clock is late enough to be explicitly not two o'clock guests. I, don't know.

D

All right, I, don't I, don't know where we can meet up. But why don't we say.

D

Same places, the Signet deep dive, wherever that is.

D

I know the San Diego commit dissenters like a mile across so like, instead of picking a room at random. Let's say wherever this Signet deep dive meeting was whatever room is assigned for that. That's where we meet two o'clock Thursday. Yes, all right sounds good. No agenda, we're just gonna talk, I.

B

Think that transitions into my thing pretty easily, which is what is it 12 25 to 125 ingress next-gen discussion for anyone who's interested? So that's a Tuesday I had the googles community lounge I think is what it's called I don't know, but I sent an email out it's in slack and it's on the agenda so.

D

Okay, cool cool; yes, we have a community lounge put aside for this stuff.

A

Cool, that's all you got on the agenda. Any last-minute items I have.

N

A question for Rob Rob: you are the one that is doing this lysis and falling theme. So how is this gonna work? With the glut stack, you see, we don't need the meta proxy thing with the Impala in point lysis or we still need the meta proxy. We.

B

Know it's only metal products here, but now that we made a change in this 1.17 release where, instead of just an IP address type for endpoint slices, we have 5 GB for and ipv6 address types, and so it's actually it's really easy to integrate with meta proxy err and COO proxy in general yeah. So there's 1, PR I think locky just linked it in chat that implements it. Okay,.

D

Thanks reviewing that.

N

Is it gonna make yeah it's already.

Q

Only this time, Rob I, remember last release. We were a.

B

Lot better, this time.

Q

Because I think see, I was down last time as well. You.

P

Q

Like merging things blind, yes,.

R

Thanks for working with me now, I remember.

D

That I dislike.

R

Freeze to me, like three years of back honest like when you said that is like wow. That happened just two and a half months ago, yeah.

Q

Totally I feel.

B

Like that end point slices were close to being pulled out for at least a week after code freeze and eventually things stabilized I asked.

Q

You for a rivet patch yep.

B

Q

We're gonna pull it over there.

A

That's crazy, excellent thanks! A bunch see everybody in one month then whole next week.

Q

Next week, in one month, yeah see everybody in a month.

L

Or January.

Q

L

Me I'm off for a month.

P

All right spike all right, thanks, bye,.

P