Kubernetes SIG Network, 30 Nov 2017

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Kubernetes SIG Network 20171130

Description

Kubernetes SIG Network meeting from November 30th, 2017

A

All right so we're recording this is the coup Brandis sig Network meeting for November 30th 2017 we've got a few things on the agenda today, pratik had added a demo. We got a couple demos on who take one from Mike.

A

And a few other topics as well so I hadn't, spotted that we had a cup two demos until just now I'm for taking Mike how? How much time you guys think you would like for these I.

B

Prepared to make my pretty short, five or ten minutes, yes,.

C

They really didn't perfect.

A

So I think we should be able to do both. Then pratik. You want to start us off. Yes,.

C

Sure I can share missing great.

C

Are you guys able to see my screen.

D

C

Are you seeing the dog or the presentation I.

A

C

Do you guys see the presentation now yep? Okay, so the idea is behind this tool. I have been waiting to debug an application, so there is a application running in humanities cluster. So as a user, it's always a challenge to go in there and get all the Iberian rules if a pod is not able to dial out or not able to talk to any other service inside the tester. So just in the state and I have been working on this tool. It's not it's not complete yet, but I have oh I.

C

Have it at a point where I can do a POC and get some feedback like if it's really helpful or if it's later, if it's of no use so that I can think of something else so and get more feedback? How I can improve it if it is a healthy? So, as I have only two slides to cover, which will give like the flow of what I am doing so in here, I have four main components.

C

One is the controller which listens to the CRD objects, so the idea is a user can create a CRE object for the Taser and in that object they can define which pod they want to debug and the destination.

C

I will show that in a minute how how we can give that information and the destination where the part is not able to reach once the controller sees all this information, it creates a an inspector inspector right now is being run as a daemon set, but it can be optimized and inspector goes on each node on the cluster to analyze the traffic and a diaper table routes and and collect those traces from that node and inspector creates of worker, which goes which is being added by the inspector on that on the part where it is running like in the same net opening space because Kuban it is right.

C

Now it doesn't support that command, cube, CLD work, so I had to go this route to use the dock or EPA directly to attach the running container, so I attach the worker in there which just sends the traffic with the destination provided and collector is sitting on the master or somewhere in the in the in the cluster itself, which gets all the logs from the inspector and collector can send it toward different back-end to view those logs or right. Now it's just displaying on standard in so so.

C

The steps to create is I, just I'll just create I will just deploy a controller and a collector, and then I create my cre definitions in that cid definition, I have a the pod I need to debug and the destination to check and on the collector will just trace the logs where the collector is and then to clean up. We just delete the controller and collector and delete the tracer CID and everything it. Everything is cleaned up. Basically, so I'll just start with the demo. Well good.

C

C

Was able to see material.

A

Looks like we can see it for a moment and then it switches back to the slides.

C

C

So I have a twister running in gke right now. It's a four node cluster and I have already deployed a sock shop application which is using all the capabilities concepts. So it's just sock shop from VBox guys and what again I can do here is I. Have my commands here, so I will deploy them.

C

A controller and controller is just watching at this for the CID objects once those CID objects are created, it will create an inspector, so I will deploy a controller right now and a collector which will collect all the logs from the inspectors. So once I start this guy excuse me, so the collector and controller are up. Now we can go ahead and create the CR, D and I can show the definition of my cre also.

C

So that's a simple c rd definition, how it looks like it so based on this. I can then create my tracer objects and pass in all the values there, and once this guy's.

C

This guy and I have my collector up, so I can check the logs of the collector here. So collector is listening on port 80 from all the inspectors. Now, once I once I identify which part I have to debug I can go here and.

C

Create a tracer so so this is where I tell my D ID, that I want to debug orders and name a part which has name orders. So it's the selector I provide, and these are the target destinations. I want to debug like I, want to see if the pod can get outside ICMP or refuge if it can't reach a payment service within the cluster on TCP port 80.

C

So when the inspector runs up, it will start the workers and worker will start sending traffic for these on these two destinations, basically and inspector will be able to capture all the logs from there. So right now, I don't have any tracer. So, let's see ok.

C

So there is no tracer and I can create one tracer once I create a tracer controller, we'll get this information try to get where the order pod is running on which node and it will deploy the inspector on all node. But on that node, where, where the order part is running, it will also apply the worker.

C

So inspector will connect so looks like the inspector is coming up now and so inspector gets deployed on all the nodes and once the inspector comes up it brings up the worker also and worker immediately starts, and in traffic and inspector capture that and sends it to us and the collector, which collected all the logs. It will tell us where the so on this host.

C

My order part is running, that's the source IP for my orders and that's the destination we wanted to debug, and it will give me all the IP table rule it has being fast through. So every IP table rule is doing down there right now, but this can be optimized and it can also give the information about our and other things from that house.

C

We see, and so the logs now here will show the table chain which policy number it was checked through source IP destination and all the policy details what the rule was and we can debug easily from it. So.

E

Are you changing namespace into the target namespace and sending traffic from there or how are you getting that in so.

C

The worker, a worker container, is being connected into that namespace to the part target hospice. Okay,.

E

So you have to say: you're running I didn't see that that you're running then it uses to join the namespace.

C

Is using docker, ok, ok, docker run net container and the pod container ID. Ok, thank you to get to that and then.

E

What's running, it's easy: it's using docker run! Yes, okay! So it's joining that network name. Space catcher right.

C

E

Using the darker.

C

Api is basically I'm in the inspector pod, which I deployed inspector pod I'm mounting the darker sock on on it right API is to talk to talk to create the worker container. The worker container is in the same network namespace as my order for yep great queue.

E

B

I wonder if we want to support that more generally, I can think about.

E

The uses for that there's a there's already a proposal for cube cuddle debug, which does effectively the same thing, but does it at the cue cuddle level and that's largely been approved. I think at this point, they're arguing over some semantic details and then Janelle didn't we'll proceed well.

B

I was thinking of things that are not debug they're, just regular production usage, I could think uses, for you know dynamically. Attaching stuff and detaching stuff I.

E

Would love to hear the use cases for it and we could discuss it. The the target that we're focusing on with the debugging stuff was really the case of I want to deploy my pod without any tools in it for security reasons and then, when I need to debug it I want to attach the debugger as a separate container, but that was intended to be sort of a transient thing. Not a production level thing so it'd be interesting to talk about those other use cases. Sure.

B

I'd like to do that also this this business about reporting the rules that were traversed if I got that right, I'm not familiar with what has the ability to do that so.

C

How I'm doing is in the inspector is when the inspector comes up it, it runs the iptables s command and passes all the routes, so it all, the rules are there in the in the user. In the memory, and then the packet goes through inspector adds the NF log NF log policies. So all the packages are going through that those rules are being given to the inspector and inspector matches that packet with the existing rules in the memory which are painted at the time of when the instructor came up.

C

So that's how it can tell okay, this rule is matching this. This packet is basically matching this rule and it will dump all those okay.

B

So the inspector just has a copy of the logic in in iptables I said: yeah, okay did.

E

You look at using iptables trace to try to narrow that down and let the kernel do the work. Yeah.

C

The problem is for that you have to insert a module and enable that more yeah I tried that, but that that requires like root privileges for inspector, but in this case I have given only let okay but.

B

If you wanted to enable that, as a cloud provider, you could just put that you did about a kernel module right, the cloud provider could just put that on those kernels.

E

B

A minute I'm still confused kernel module right. The cloud providers say the kubernetes operator could yeah.

E

Like most modules, it wouldn't be loaded by default. It would be loaded on demand and in order to trigger the load on demand. You'd have to run it well.

F

I think what Mike's saying is that the cloud provider could just load it automatically at system startup.

C

Yeah I guess you could, but you don't want on each node. You just want that on the Nord, where the part is running to debug that form it's.

E

Not that inefficient to just load it I mean the reality. Is it's a kernel module? It's probably you know order hundreds of k-max.

C

Yeah but the the other challenge I saw with that was the traces were like so many and then you have to go and pass that syslog from there. That's true. Oh it traces everything all the time yeah. So what I tried was I added those logs in the raw in the rod table and then I was getting all the traces for all the packets going through. But in this case, I have all the filters where I can get only the logs for the targets I'm sending.

C

So when I'm sending 8.8.8.8 sir, only that destination I mean.

E

You could you could program the rules in the raw table to only use you know only destination a date, a date yeah, so yeah I can do that works entirely yeah, but you're right. You still do have to parse D message, which is terrible. Yeah.

C

So, for me, I thought this one was like faster. Okay, so I went with this approach. It seems like a good trick.

B

But it seems to like a lot of work to be to duplicate all the logic of iptables, there's actually a lot of stuff in there.

C

Yeah, that's the only heavy lifting the rest on stuff is straightforward.

E

We don't we only used, like maybe maybe like 5%, of what I potatoes can do well,.

B

You, what about what about calico and and everything else that people might be running.

C

The rules are not that many I looked at calico also, so it's so when, when the inspector comes up, it passes all the rules and it converts them all into JSON format and they are stored in the runtime, visible.

C

B

If you're reporting that that's interesting, thank you I.

G

See but it's true that you would, if someone had a rule, that you did something random, that wasn't supported with that. Just I mean.

C

We can always add that rule. It's not that complicated to add the logic button.

G

C

G

But in the like, the full implementation, where, basically, you have to reuse.

H

So say if there's a some kind of Cisco parameter that we quick and then it's namespace so like. Let's say: if it's in the network namespaces in a one and it's say one in the root name space, even though you copy all that IP tables from into their namespace. The behavior may be different right.

C

No, so this this, these packets are sorry. These IP table rules are copied from the root networking space and it's not it's not put it in the network, namespace of inspector there in memory and then the packet is being when the packet goes through the root net of the instance. It hits the NF log entries and based on that NF log entries. We get that packet and we can match that packet with these routes. Oh I see.

G

So you the trigger, is you see a packet arrived? You will have other like what is coming out of the enough or is it merely the packet yeah.

C

Just the packet and then once I get the packet I used the go packet library to parse that that I.

G

See and then you you do like a synthetic roll.

C

Trace, yes and then I match that packet with the with all those. So let's say it came in I can take an example. It came in filter, so it came in filter forward and then after filter forward and I will match all the change in that filter forward with those policies and whichever matches I'll just don't go.

G

I see so, but the thing is when you like: let's say you see the packet, but then it somehow because something happened. It went somewhere that you weren't certain, but this is synthetically kind of evaluating the rules like how do you know it actually went through all the rules that you evaluated because.

C

The rules will be the same on the route I put, even unless somebody has changed it when this was deployed.

G

C

G

Actually know for sure that your synthetic execution matches up with the IP cable, like the actual delivery of the packet yeah.

C

Because, as soon as the inspector came up, it caught all the rules from the host and if somebody has not changed it, the logic will be the same, which will be there in the IP tables and in the users. Are you.

E

Evaluating are you like evaluating yourself all of the conditionals in all of the rules so like? If there's a rule that says if source is mod, you know, if there's a particular source range, then jump to some chain. Are you evaluating that yourself or are you just saying now wait for something to tell me that I hit that chain no.

C

I'm evaluating that for inside that each chain and then when once it goes to a different chain, I get that I get the notification from an F lock. There also.

E

Okay, so you have some confirmation, even if you evaluate it yourself, yes, but you've, basically written an entire iptables process right. Yes, it sounds like fun.

B

And some chains are maintained by aggressive programs that don't let you hack them.

C

Like you can't insert the rule, oh, you can insert all.

B

You like, but this other program is gonna, take it back out again or move it or something yeah.

E

I mean cube, grotty cube proxy will will rewrite all of the rules every whatever few minutes. So, if you run afoul of that, like whatever you insert it into all of our chains, is gonna go away.

C

What I have seen is, if you append it so I'm, not appending it in the chains, I'd written back, cube proxy, but I'm doing it in like filter chain forward chain, so the default chains we have in the filter table so.

B

It's not every chain, it's only certain chains, yeah.

C

Default chains, not in the user-defined chains, so I won't insert a rule in queue. Metadata server, but I will insert the rule in for watching.

I

H

C

C

E

It's very interesting related. We have someone who's been looking at something internally here to also diagnose network failures, and basically they took this dock that we wrote a while back called debugging services and starting to codify it into a program that you run, and it would then tell you whether it's your service that doesn't work, whether you're, missing endpoints or whether your selectors don't match or whether cube proxy is not running and it tries to go through and make all those sorts of diagnoses.

E

This might be an interesting addition to that, or there might be an interesting way to merge those yeah.

C

Maybe we can talk offline about it once I have more more stuff added to it sure.

E

If you want to send me a link, I'll send it to our folks internally and they can take a look at it. Yeah.

C

I, don't have it right now, it's all past of work, so I have to clean up a lot of work. Okay, whatever you ready sure thanks.

A

Cool thanks for take.

A

So Mike I think you're up next.

B

Okay, Oh see if I can share my screen here. um Yeah just sure the desktop that'll be easiest, so um I wanted to just mention some of the stuff I've been working on. A larger project is I, think of it as using pieces of kubernetes to build something. That's not kubernetes, in my context, we're interested in building an infrastructure cloud that can serve VMs and virtual networks and I just wanted to show a piece of the work on virtual networks here.

B

So I put up a brief outline on it's linked from the agenda, so you can see so this idea, I'm going to focus on right here, is just the basics of building some virtual Ethernet and making network interfaces that attach to them using kubernetes api machinery to build a distributed control plane. We use OVS for the data plane, we're virtualizing using the excellent, so we're introducing two new kinds of API object, network and network endpoint. Actually, it's a network definition is the one we call it.

B

um There are Network definition and network endpoint, and so the central problem here is on each node, we're using the local control plane of OBS, which is limited. Doesn't anybody other know that has to be told? The issue is when you make a Linux network interface on a virtual network on one node, all the other nodes that have also have Linux network interfaces on the same virtual network. You have to do some local work in in the OVS on that node.

B

So there's this control plane distribution problem, so we did it in just obvious way using kubernetes api machinery, we introduced this type of object or a kind of object called network endpoint when there is a linux network interface made there's also a network object made in the kubernetes API servers on H, no there's a net agent. That is, has an informer on these objects and it invokes the appropriate local operations on the local OBS to to update it so that the thing I'll get stitched together and works.

B

So that's the basic idea and I can give a demo. Maybe I should stop there. Any questions at this point.

B

No okay, well I'll just go ahead and show the thing right. um Let's see here so I've got a shell.

B

Is that is that legible, a little bit bigger.

F

B

Right, let's see here, I just gobbled up a bit of scripting here, so I'm gonna go ahead and make a network definition, and you can see here what's in a network definition, it's very simple right now, all the user specifies is an IP subnet and a name and there it goes. We have a simple controller, oh right, so we're using VX Lancer. There are virtual network identifiers, OVS Tunnel IDs involved, so they there's a simple controller that assigns only these things in network an int, a a network number.

B

So here's a number that got assigned to this network. So let's go ahead now. We've also got a scene I plug-in that uses these virtual networks. Here's a little script that makes a pod. So let's have a look at that pod and the important part here is we're using an annotation to identify the network.

B

This pot is supposed to be attached to the C&I plugin reads: the pod discovers the right network makes connections to it and also updates puts in another annotation that reports the Mac that got assigned so now we've got a container on that Network, let's go in inside and have a look inside.

B

Okay, so here we go here is the network interface created by that C&I plugin. We also make a bridge a linux bridge for friendliness to key mu and that's the thing that actually got the IP address. But this is a thing created by the the Sdn. um Let's see here and now, let's we can actually look at the network. Endpoint object for that. The the name here is just the the MAC address in X.

B

So it is that guy right here.

B

So there's the network endpoint that got created for that again, it's pretty simple: it's got just some basic stuff, identifying the host and where the network interface is on that host, we can make another one and show that they can ping each other, for example. So I can do you know cou control exec, let's see, let's do unless the new guy to ping, the old guy.

B

And then IP address was in the.

J

B

It works so that's you know, coop grenades knows the IP address. Of course we reported it and there it is, and I can. Let's see oh right, so I can also go on and show a little bit about the the stuff we're doing for VMs for vm's. We actually want a dynamic set of network attachments and we're using more sophisticated syntax for that. So here's the the specification, the dynamic set of network attachments for this VM here is a command. That's going to using a different API. Add something to that.

B

And now, if we go back and look we'll see that there are two attachments and in fact there's a status, the status hasn't been updated. Yet it's going to take a minute. Let's see it gives back now yeah. So now the status shows two attachments have been made. So there you go basic idea.

B

Any questions or comments.

H

E

And it looks cool Mike. What what difficulties did you run into like what didn't work? Well, so.

B

It's a little bit annoying to have to go in the C&I plug-in and fetch the pod right I mean because the scene I plug it and buy something that already has the pod.

B

It's a little bit strange so to have a scene I plug in you know, you'd normally think it was doing a local operation, but it's reaching out and doing all this remote stuff. So it feels a little bit inverted in some sense, but it back on the basic concept of the Sdn. It seems very natural. In fact, I have people who know more about ovn and saying this is a just a much better.

B

Implementation of ovn is kind of doing the similar thing, but they're starting from a database foundation rather than a kuba API, Machinery, Foundation and they've just got a much tougher row to hoe on the it. You know, I'm told that the the goob API machinery just makes this more direct and efficient.

B

Okay, in fact, I want to argue you know: I'm I, have kind of a long history, building systems and I really like this state based management approach, I've been advocating it before I heard about kubernetes, but the coup brace API machinery is a really nice foundation on which to build that style of management right. So I'm trying to argue in general that this is actually a good thing on which to build whatever distributed system. You want to build all.

E

Those are very kind words and I will pass them on to the the machinery team, who are always suffering some crisis of confidence because they look at the code and it's oh sure,.

B

Yeah I mean there's, there's things that you know and asks that I have to. But you know it's it's going in a really good direction and it's released. You know showing a good approach, cool.

E

Thank you sure.

A

A

Any other questions for Mike before we move on.

A

So Mike, the next topic on here was multiple networks before we dive into that is that is that a quick question, or is that more of a broad discussion you want to have yeah I just wanted to you know we try to move.

B

The discussion long.

A

Okay, so I think there might be a couple quicker comments on the agenda. Let's sure get those out of the way first and then Weekender, and on that note so like we did, it started doing last week, I added a sig test review. So why don't I share my screen and we can quickly go through failures.

A

Can you guys see that.

I

A

Cool, so it a couple failures. Last week it looks like some of those been fixed, although some of them haven't so we go through the ones that are currently failing and make sure there's someone looking at that.

A

So last week we had both the upgrade and downgrade acute proxy Suites failing it looks like the upgrade isn't fixed, which is awesome yeah, but downgrade is still failing.

A

Trying to remember was it Zoey who was looking at those.

D

D

To me sometimes, and only I think it is a meaty seedy issue.

D

No one seems to be actively working on this I need in some folks. I will pull up on this. Do we do we have an idea with the kind of arrogance? Is you downgrade, I think the master eighty CD with some incompatible issue in the void, your name.

A

So this is potentially we're using some some feature in this manifest in this is available in an earlier.

A

Concerned about going into with this.

I

A

So the only other failing on here is the hoops yeah, a calico set of tests, which I think is on my plate. There was some progress made I think it's an infrastructure problem with kubernetes anywhere and yeah. I think this is with me to investigate at the moment, but shouldn't shouldn't block or concern 1.9. At all.

A

Last week we had talked about the IP vs tests. Those look to have gone from failing to flaky.

E

E

Anybody from my PBS team here- yes, hi I'm here- are you tracking the tests? Yes, any update, Porter.

K

That has the passer. If green, since is IB, said, cut the e and green I think it.

A

Looks like that might be just a couple more runs and we'll go to passing rather than flaky excellent.

E

A

We have a whole bunch of other sweets that are flaky as well. I, don't know if we want to try to cover all of those right now.

A

That is probably worth us trying to get those.

B

Passing those are all out flaking at a low frequency right, maybe there's except the last one. Some.

A

Lower than others yeah is.

B

This some bit of well, maybe some just look at this. This seems you know, infrastructure, flaking or what.

E

Yeah I mean if only there was a way to tell without looking at the logs.

A

And we also got this GK sweet. It looks like similar sort of state there. I know we had a couple failures last time. Those seem to be moving in the right direction. I.

A

Think we determined it has to be someone inside Google who investigates these yeah.

A

Cool any other concerns about tests before we move on questions or comments is.

E

Anybody from the v6 effort here.

E

Okay, I'm hey there: what is the status of test coverage for v6 Oh.

J

Gains leading that deign are you: uh are you on yeah.

L

I'm here yeah, we we're still in the design phase trying to get get some end-to-end test running in that with the infrastructure. But we have. We have a test suite picked out. We just need to create some create some containers that that run dinned, so we can run multi node in a virtualized environment. Somebody somebody.

E

Here had a really interesting idea recently that I thought might be interesting for us to try to use to test six on four. So tell me if you think it would fly or not what if we could bring up in the regular surrounding most of the test, Suites on GCE, what if we could bring up the GCE and since that set up six to four tunnels on the VM and then treated the G scoober Nettie's cluster, like a v6 cluster I did some really rudimentary testing and it seems to work I.

E

Can you know paying each other across v6 addresses? Obviously it's it's tunneled, but to the to the pod. It was a v6, odd.

J

Yeah I think we'd still need a setup either through a VM or in the container yeah.

E

J

I'm not 64, we.

E

Need to do some custom, some custom setup to make it work, but if we could do that, then it would just be a regular old test run like any other test run with it with a custom setup. Wonder if that would give us the ability to test the all the v6 work effectively in perpetuity on GCP, even though GCP doesn't support. V6 natively are.

F

You saying Tim that, like the default Nick for that cluster, at least the Nick that all the pod traffic would go over would be a 6 to 4 tunnel. Yes, ok yeah, that, on the face of it, sounds like it might be. A good approach. Yeah I mean.

E

It's really really rudimentary testing we no more than 20 minutes worth of testing, but it it seems to work I.

A

Have a vague inkling that we have done something similar to that in the past, so yeah I think that might might be a valid approach. I can ask around yeah.

L

That's that's worth considering I, not sure if we're going to have a problem with with the web hook, if github github is sending us web hooks, I'm, not sure if that only works in v4, but that that needs to be captured in would it what is a component that does the captures the web hook? But if that's in a v6 cluster I don't know if we have to translate that from a v4 packet to to VC.

E

Question I, don't know, I didn't test it with something like external traffic. I only tested it with intra cluster traffic yeah.

L

But that's a detail have to hammer.

E

Out yeah, but I think it might be worth spending a little bit of time on. Maybe once we get past cube con to sort of brainstorm on if and how we could set up the test infrastructure to have a parallel test run that would start running v6, even if it was a subset of all the tests. I just.

J

Put in the chat window, this is the COO Badman didn't project uh we shared with the team a few months back and yeah I believe who I think is on the call. I did a lot of work to add ipv6 support there and it's what we've been using for testing and GC as well as local, and it's worked out really well so I, don't.

E

Mean to take anything away from that. Certainly I'm just was thinking of if it would make more sense to be running without any of the DMD trickery, but we'd be doing substituting for a different set of trickery. So if you guys think you can get test frameworks up here enough, then that is fine. Once we get to dual stack, we'll have to maybe think about. Maybe it makes sense to only run the test Suites once and just run, everything in dual stack: yeah, yeah.

L

Yeah, it's really be a lot easier once we have dual stack.

E

Do we have a timeline for dual stack.

J

Well, thing has just been dependent on being able to support multiple pot, IDs, okay,.

H

J

Haven't been tracking that just because we've been so heads down on the one nine deliverable, but is there any updates on multi IPs for pods? Is.

E

Anybody everything anybody's activity.

J

A

E

Anyone is so yeah.

B

E

It as a 110 goal, maybe well.

B

It it does bleed into the multi network stuff, because yeah.

F

That was one of the blockers in the burnout blockers in the past, but one of the considerations but I think.

E

Nobody's at this point, multi dual stack is a real proposition and I. Don't think anybody's gonna disagree with I guess the only question is: can we enable multi IP in a way that, if we pursue multi network would still be happy with.

F

The way you could do that I think has been proposed in the past. You kind of have a I P structure that has you know some arbitrary set of tags along with each IP address and that could potentially be used in the future to say which network each IP was on. That seems plausible.

E

So it maybe we should make it an explicit sig goal for 110 to try to get multi IP support in yeah.

B

I mean I think you can factor it right. Even on one network we can have multiple IPS protect, dual stack, just look at the IP and you know what it is. You don't even need any tags. Well.

F

Yeah I mean it's it's easy enough to do: multi IP per pod without having multi network. It's just kind of a question of it would be a shame if we end up doing multi network at some point in the future, to not have the structures there to have to rev the API right exactly yeah.

B

So yeah I I, you know I, think I said this before what you said earlier, so yeah I think if we just start out with something that has the option to tell you these things that.

F

Should be good, there was also the question about which IP should be used for health check, and that was another thought for tags, because maybe you could tag a particular IP as a health check IP there.

E

You go so right, so this conversation definitely has some nuance to it: yeah, but I. The time has come for it. Now that v6 actually basically works. The next step would be to get it dual stack. I.

L

Think we'd also have to have to figure out the the dual stack for services to I mean for every endpoint. Do we do we use to both of e 4 and a v6 endpoint or or do we setup services that so we can select a list of service ciders that that should be should be used for the endpoint.

J

L

So that that's another part of the part of the problem, all.

E

Right well sounds like there's a lot of work to do then.

J

Since we were talking about v6 testing I in the chat window, put a link to my PR five six to four or five and I think it's it's failing tests, because test impre doesn't support these euro six euros. Cni, binaries and so I opened up an issue which is the last thing that I provided Izzard. How do we get confirmation or any kind of guidance to inspect you? What version of the CNI biner is a test? Impre is running when it's doing its integration and and then to us I.

E

Don't actually know the answer to that, but it needs to be whatever we're saying. We support I.

A

Think that's a kind of marquee area. At the moment, 1.9 is supposed to include oh six I support, but if I understand correctly, they we haven't pushed the kind of Debian and RPM packages for that, because we haven't released one nine, yet so I think there's a bit of a chicken-egg problem there.

A

If you're using any of the cube, ATM based infrastructure.

A

They don't they don't have it I think they only have a single release stream, which makes it hard to to maintain different versions of those packages right.

J

J

And you ought to say I thought those your six euro support I mean that's the ipv6. All the ipv6 works hangs on that. Okay.

C

J

Figured it out, you know this could be note to PR that I posted there again. It's I think it's failing because well, the test infrastructure is not supporting me, updated, CNI, binary. All.

L

J

But the issue that I opened up I, don't see I put it up, I guess it was just a day or two though, but yeah any kind of guidance we can get and it could be helpful.

E

Can you tag uh XD, I XD, why I.

J

E

J

Just doing a sign or yeah.

E

A

Done: okay, we've got a couple more things to get through on. The agenda should be quick, but I wanna make sure we get to them in the last 10 minutes. Do you had a question about declaring IPS beta? Yes,.

K

I'm not sure if we still have it at hand pita, since the OSD is and I know, we have faced all no issues.

E

So officially the window is closed and the boat has left. We would have to make a really strong argument at this point for moving it from alpha to beta after the window and explain to people why it was urgent. Do we think we can make that argument? I'm, skeptic, I.

K

A

Is the argument not just that it's a documentation thing at this point, I thought all the checkboxes were ticked off on that issue and the test seemed to be passing I.

E

Mean regardless of that, the the deadline for exceptions was last Monday. So if we're going through we're going through sort of an exceptional exception process, I'm fine with trying, if you want to try it I'm, not the release manager, so I, don't get final say.

E

But but we did sort of miss the boat.

K

Okay, I think the issue is already in Excel Excel data release, tracker.

K

Don't human the manager asked me.

K

A

Understand which boat I think we missed, so that we had a feature for this in the one nine milestone doing I thought we did yeah. That's.

E

True, then, maybe I'm wrong.

A

Okay, so let's circle back on that and double check, because.

E

Any objections to going to beta, if you're comfortable, with trying to being ready to support it and deal with any incompatibilities. The only issue is: are we gonna be able to get a PR through at this point for it? If it's already filed well, there's a feature, then probably we can.

K

Okay, whether I happen to see an interest not now so I I guess if Lou is I'm, not sure if we are to later, but I haven't, negated and.

E

Okay, I didn't go to today's burn down. There'll be a burn down tomorrow. I think. Is there an open issue for moving this? Today's there, an open PR for moving us to data no.

K

Question is there? Is a PR open back Quinta I can't I can sign it it to you later. Okay,.

E

Yeah send it to me please and I'll: take a look at it. I look at my calendar if there's a burn down, yeah there's a burn down tomorrow at 10:00, so maybe we can talk about it. There, okay, great.

A

Damn you wanted to ask that keep calm, yep.

F

Any of you guys talked about all that while I was out, but I was just wondering who all is going to cube con. It was kind of like what the current status of the like the I've session. All that stuff was I, know I had seen the proposals you had sent out a couple of weeks ago.

F

They look fine to me. Is anybody on this call doing any talks at cube con you know: are we should we meet up at some point? Networking involve people to just kind of chat or hang out or develop a rapport before the deep dive which is like the last possible day or what so just kind of all things cube con I'll.

A

Be there I don't have any talks other than the my name is against the update session, so we've got some slides for that. I think I still want to review those a little bit more and get a feel for what our presentation formats going to be. This one person gonna be presenting all of that, or should we split it up into a bunch of different sections?

A

B

Also I'll be going I tried to collect suggestions on the mailing list and nothing came in during the expected time. So in case he put forward some suggestions. Then a couple of things came in a mailing list, so you have a couple more ideas, but we decided last meeting, but we didn't record it that will just gonna self organize at the start of the deep dive. Okay.

E

I think it's fine to sort of run. It unconscious style, question I. Think the most apt question is: should we get together sometime before that and say hello, yeah.

B

F

I'm flying in a Tuesday night, so probably the most logical time would have been Tuesday I mean.

E

Tuesday's also the contributor summit, so a lot of people will be busy with that I. How about Wednesday sometime during the day sounds.

F

E

I'm sort of loathe to pick a particular time but I think we have to.

A

Capture, we am even free to thread on the mailing list or.

F

F

A

Right out there.

E

Mike I think we ran out of time talking more about maybe network, but I wanted to throw out if I, if I could an imitation I, don't know if anybody from the Amazon team is here, but now that you guys are uncloaked I think it might be cool is to get a presentation on the networking setup that you're using I know that there's a new CNI driver and it wasn't clear to me exactly what it's doing and I would love to see more about it if any ways here or if they're, not.

E

If anybody knows some folks over there who might want to pass on that, perhaps they should start attending this I.

A

Might be able to secure a couple people cool if there's nobody already here.

A

All right, I think that's probably it for today's meeting and next week's Keep Calm and then I think we're still on for meeting the week after that.

A

So I see you all some of you next week and the rest of you in two weeks.

D

Feels good right.