Istio Networking Working Group, 21 Jun 2018

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Istio Networking WG meeting - 2018-06-21

Description

Agenda
- Playback/request for feedback on istioctl proxy-* commands
- Pilot Decomposition Epic
- Critical issue triage for 1.0

A

So thanks everybody for joining the easter networking community meeting. We have actually a very light agenda today. So first item is to look at some critical issues and do a bit of trash. And the second item is some feedback requests on East, your CTL proxy commands, so I'm not sure. If Liam are you are you in the meeting I think it was you who had it out and okay cool cool and if there is anything else, please edit here, the voice would probably be able to finish a bit earlier than the normal.

A

So the first item, which is critically to try out so Louie, is not your team, I'm, actually thinking, but maybe like doing this as a second item. If Liam it would be okay to start.

A

Yeah yeah, okay, great. Thank you tell.

B

Me the proxy states is like this I'm gonna, weird branch, but I haven't updated to master that I've just realized, so I'm gonna have to use kind of a custom, build and just do the proxy company stuff, but I can leave I did a hacking into actually I don't know. I will quickly check that the proxy conflict command is working in my current builds.

B

A

You want to share anything green.

B

B

C

We're able to see okay.

B

Can you see my salmon online, yes,.

A

Although it's very the fonts are tiny, so.

B

B

And see if I can get more commands on, let's do.

B

You know: should we go, that's fine. So if I get Hans running and pointed at my dev transfer, so that's fine.

B

Basically so the proxy States at this command, which I can't actually show you, because if the we'd build I've got that I can show you and tweeted out by sanity. So I can show you exactly kind of what it looks like.

B

Basically told us to pilot and what it talks to each of the pilots and gets a list of all of the prophecies connected to them and their sync reside. Synchronization status.

A

They want to show your Twitter now sorry.

B

A

Do you want to show your Twitter and.

B

A picture of the command because I do not have a working build online, my laptop's funny very slowly, hangouts so yeah. The idea is the good news. The consi status commands kind of work out.

B

They should be stopped to go wrong load but yeah you look at it tells you it's got where the CD s is deemed LDS is synced, it tells you the percentage of endpoints, so that's endpoints compared to total endpoints are synced, and then it tells you whether or not RDS is synced, and it also allows you to kind of see which pilots each of your envoys are connected to.

B

This is going very slow.

A

Yeah this is this is great, so you basically are showing status from all the employees plus the pilots right. So.

B

What what is actually showing this is showing all the status of all the envoys, but it's getting it from pilot. So yes or we store the nonce, which is basically the time of which the last up they were sent and then envoy responds saying, with the next discovery request request acknowledging that it has received that status update.

B

So we then take that information in its newest ETL and we then synchronize that we work out whether it's synchronized or whether a nice tail or in terms of IDs here which I'm assuming you can I, will see the picture on my screen yeah.

B

So we're not sending routes at all to egress gateways and ingress gateways, because I believe they don't request to watch routes. So you can see there, it's not been sent. So this is kind of the high level I want to see everything. That's going on my mesh. You can filter by proxy ID, so you can just if you wanted to the details, one you just do SEO CTR proxy status, details v1 whatever with the dot namespace on the end, and then it will just sync to that one.

B

So you can look for specific ones, I think it also does just strings contains matching, so you could just do details that'll be one, and it would show you if you had multiple details, v1 pods, that you can show you all of them and then you could see which ones would potentially have sync.

B

So that's kind of a high-level one that you do you kind of look and maybe see that something stale and then you use the endpoint commands to do stuff like working out. What's in here, so I buy the full commands, so you do endpoint and then that's good product page.

B

Buster's, for instance, and that will show you each my own commands as I change the order. So the commands a bit more organized.

B

It's running very slow.

B

So this kind of splits up stuff, so this is basically all just summarizing information from the config dump from envoy. It will show you kind of a high level thing and then you can filter like run bots. So if I did ninety eighty, which will show me all things, will clusters with four ninety eighty which isn't in itself particularly useful, but if I filter by poor, give me three, for instance, and then ask for Jason it'll dump out the entire all done by the config for that specific cluster.

B

So you can then kind of go in and make sure the stuffs correct. It's the same for listeners as a specific thing, collection of things you can search on and filter on, which is different for each commands, they've done as Flags, because I can think of a nice way to a proper leader.

B

This is alias L as well, so I'll just do that.

B

So this news of Quadra, sport and type and then be the same. You can just stick a Oh Jason, but you can filter on whatever and go in and actually look at the confidante for a specific thing. There's obviously lots here because I just dumped everything, so you can go in and look at that specific thing. Now there is a bug you can see here where it's gone really far to the right. That's a bug with frogs above which I've raised against them.

B

So hopefully that looks fixed eventually and then you can do summary of routes as well so routes I'm. Not this is just the RDS routes, so this won't give you the routes that are in line with listeners. There's only name and the amount of virtual hosts I put in summary, and you can get the full conflict on these in the Oh Jason I've done enough influence that yeah but I'm open to what information is available in the summary commands, that's kind of what I'm asking for and then eventually the idea would be.

B

We can do some kind of like proxy div for proxy convictive or something and the idea would be it- takes there dynamic clusters from the Envoy config dump from the API, and then it compares them to what pilot gives as a risk would give you as a response. If you came as that proxy and then week and if those to work out kind of what exactly is I would think, but that's kind of a bit further down the line, I'm just kind of looking for the summary, your feedback on the summary.

B

So if I just quickly show all of them in a line so I can get so people can quickly see them. So this is kind of general fields. I can only really get or we can only really get information. That's in the conflict dump.

B

So for those who aren't familiar, there's a new admin, API config dump on command with envoy, and that's where we're getting all this information from.

B

Yeah there is I, know Andrew. You raised an issue for basically having ongoing for Unicode cluster and it dumping its full config to a basically to a binary, so a user could send it over and then basically as a config dump for their entire mesh. So we can then debug it. We are live in that as well at some point but yeah. This is just generally for this stuff and I'ma stop rambling. Unless someone else talk, this.

A

Is super useful and, like super exciting, and it's great to have this- you know like I- think we really need really need to get better at troubleshooting, and that will help us like it just in general development plus whenever this would be running production battle. I'd be awesome really, and it is already awesome like this. What we have yes and so Bay, based, like my question, related to your question about the issue I open.

A

So that is exactly to help with the troubleshooting when, whenever this is run on a big cluster- and they don't know exactly what's going on, you know what, where? Where is the issue? So it would be good to have run a simple command and gather everything from pilot and from all the employ proxies and put it in a nice binary. Attach it to the github.

A

Is you know like because the type of issues were seeing now sound like traffic loss or I cannot reach my service or you know, or things got bad after running two weeks in production right, so the other idea would be to like, at the beginning of a deployment, then run that command and get the status and then, if something goes wrong like run it again and then do the diff again, just as you mentioned, between pilot and proxy, also diff, between really the operational state at various points in time.

A

So we can narrow down the problems right yeah. So this is like yeah. It's it's super useful. Definitely it's.

B

It makes it much easier to understand kind of how the how will the cluster listener routes interaction works, because sometimes you just get a dump of everything and it's quite confusing. Actually.

A

What I would also like to see and I'm sure this is possible if I'm like, for instance, I, had to be able to trace a particular host or the traffic for a particular yeah floor host like if I give the host I'd like to see the the listener you know associated with it or the the listener throughout the class. Certain point and basically be able to understand. Where is the issue and ideally across multiple right? So so, basically do what the employ does internally, but more of from a theoretical or.

B

A

I think I think it's the so.

B

You'd say something like I'm: making a request of this host. Can you please list me how you would go, how envoy would go through the process so right, location, right.

A

Something like that, I'm sure it's possible- it's probably non-trivial, but it must be possible. Yeah.

B

There's the others that well must be cuz envoy. Does it right from the same amount of information, so it mostly yeah I can see a way that you could. You could work that okay I mean I make some notes is.

D

There any way to get the circuit breaking in here, there's a column.

B

Where is where's the circuit breaking information stores? Is it in there? Is it in um LD answer.

E

B

Is an odd yes/no circuit breaker thing is insidious. It's not seen you.

B

Probably won't have any circuit breaking service is probably not helpful.

F

B

F

The good braking is useful here, that's more like statistics from on way and not like you know, statistics between pilot non-white in terms of the SDS protocols, I mean those would belong in a proper dashboard with all of the other on my metrics that are coming in, and this is just simply listing what pilot has sent and what has been act in the protocol exchange. Yeah.

B

And the the idea is that the first kind of clusters is just a high-level summary and then you can go through on that specific cluster and just get that information. So you can get a full conflict, um so it'll, be an error. Just won't be on the summary page.

B

Like I said, I'm Otis feedback you've got any feedback. Just message me on the it's like channel discussion on things. Probably.

A

Have any plans to make it also like browser friendly, so aid add HTTP link. So if I'm on are you know, I'm on a listener pointing to a route, I could actually click on it from a browser. So.

B

Very easily make sure do we not the envoy information actually so.

B

Yes and no I, don't know I mean sure I'm a trying about this. Actually, whether or not we're doing a sto dashboard is kind of like a coupe, dashboard type thing or whatever or not. We're gonna kind of talk to people like the uncanny team- and this is something they can do- is on that type of thing. It's it's basically, whether or not we're going to do some form of lowest common denominator is vo dashboard type thing.

B

But the informations are there, you just have to have you'd, have to have some, probably API running or like service running in your cluster. That exposes this information on the API somewhere. Mm-Hmm, hey widget, just touched me: the envoy admin API this one in the proxy states. This is just calling on this. I added sync the endpoint to avoid, and you just call that and you can. You can generate all of the stuff from that information. So there's no reason it can't be done.

B

It's just whether or not we want to focus on some kind of dashboard.

A

Another question so the the sync status isn't based on the arc, not crazy from envoy. Yes,.

B

I added some code to keep track of that so on each connection, there's just a cluster a cluster sent, and then this two routes whatever and endpoints, and then it just dumps all that information. When you call the sing, see I'm thing and you just have to go through each pilot. What behind the console that does that goes against the list of pilots calls each one of them on the sings, the end point and then just combines all the input and works. Although.

G

G

The proxy config tool from release 0.8 until now, and one thing I recall I noticed- is that getting the endpoint information was not possible for a gate waste so that somehow didn't work. I mean I could get them with no from any point but from pilot but would be including ours. I.

B

Mean we don't get that is because envoy doesn't expose that information in the config dump. It exposes four things in the config dump: it does clusters list or CD s, LD s RDS and the bootstrap config. So that's kind of the only information we can get from envoy for some reason envoy doesn't dump that I'm sure I don't know well, I, don't know if there are any plans for envoy, it's dumpler. So what are you.

E

Missing the input information.

B

B

So we can get the endpoint information from Pyla yeah the envoys config dump in the admin API. Sorry any long.

E

Term, yes, ideally we would surface in Viacom big dump there you go the slash clusters handler, which is also MIT's Jason. You can actually pull out a whole bunch of it effectively. The same information like it has a cluster information and then the constituent members of the posture add. So you can actually look at the individual points and then hammer okay, I'll, take a look.

B

A

What was the reason for not adding an endpoints? Is it the song.

E

This has to do with this current complexity. At the time this was implemented. Endpoints made a bit differently than other parts of the system. They updated independently and a few things like that. But the iIRC don't know that for the full context you don't just chose map pioneering problems with this, but I think it's certainly something we could add and if someone think you're sitting doing that they can do that, but we have effectively equivalent information today.

A

B

I'll go and take a look at the clusters because we were we in the naught point a we are actually using the slash clusters, but this command. Doesn't this one just purely passes the config dump and works hard from that.

A

Right it, wouldn't it would be easy to add it. He right the spite of them yeah. My.

E

Copic dump is Mark VIII to alpha is in an experimental status; it's not going to change radically, but there's a PR and flight right now who rearranged some of the field. So you might see you might as the occasional breakage and now we'll continue, probably at least for the next week or so I'm in the moment, trying to actually fix up a bunch of pretty terrible stuff to do with actually being able to repast that Jason back into protoform, which is a very nice too.

E

Probably you have, but right now, protobuf itself is broken, um which is kind of exciting.

B

Yeah yeah, yeah I think I can raise an issue against the program repo because it the when it was going to the right of the screen here just starts adding in random amounts of white space, just wow wow that one I was.

E

Not familiar with but I'm, aha there there's certainly.

B

E

For example, the case of the fields gets, it seems to be almost ad hoc, others, the we we've been general in on whether they doesn't do the snake case conversion by camel case conversion, but it does so I fixed that around oh yeah.

H

I've noticed as well.

E

And there is also.

H

The fact that you just can't round-trip.

E

From the proto in and envoy to the Jason to another proto, it's something like you know: you're tall or somewhere internal monitoring tools and that's a problem that has you're just some collocation put a bath where, if you mess, then within a map field and any it just doesn't, handle them correctly. So, and this isn't that nice.

A

Anymore, a feedback or anything to say on this topic is.

I

There any sort of authorization or being able to access to this from misty Oct, oh I, mean I, don't even know what the capably really at the Envoy level for controlling that. But you know.

B

Ideas there I guess so what this is doing, so it works with NLCS. Is this darts? So this will only work. The coop, because cube is obviously opinionate. It's on how it does stuff, so we can assume where stuff is to what it does is under the covers. It does a cube exact into the co constantly new container and then calls a binary on SDO or called the issue of cotton binary and a command on there, and then that calls the local envoy.

B

There is basically the Kurds. If you can, when you give you a ton until as long, you would have to have exact privileges into the proxy container, basically other than that not really.

I

Listening access to the container as the with your cou service, account or user you'll, see all this and.

B

The second basis is just going to discovering love and yes reinforcing. Okay, it works through cube in its neck, and it has to work that way, because once you turn MLS on you'll, obviously just get bounced. If you try to port forward and then power logic.

A

A

Okay, that's so we move to the next topic, in fact, I added, something which I hope will be very very short. So some people asked me on on slack about the work on valid decomposition because they would like to follow on that I'm not sure like if the same people aren't here but I'd like to describe the method to follow up on this. So there is a high-level epic that I'm sharing now I hope you're able to see it. It's called mesh configure API mesh, config protocol, Politico position, epic and inside of this epic.

A

There are multiple, smaller issues which are equivalent to like lower level tasks. For instance, there is, there is one to define the mCP server, which is actually already committed in master. There is another one to extend the API, for instance, with an envelope to contain the metadata for like namespace and name for various objects. There is an equivalent one for the mesh config protocol client in pilot, for instance, that's going to be a very small small change.

A

There is one to define the common interface between the ingestion layer, which is the mCP and the pilot datastore. So I only created issues for the work we plan to do in 1-0, because there is like the scope is larger, so we need to do the migration of the failures in process adapter sort of process, but there is no not a particular rest to that before one zero. Many important part is to like to define the framework so yeah. If there are any questions around that like please. Let me know. Oh.

A

Okay, good and so gluing is not here, but he added an item on the agenda, which is called critical issue triage for one zero, so I think I know what what he had in mind. So I'm just gonna quickly show you how many issues we have it's quite scary, actually so I think it was last week, various people did the trash exercise and went through all the hundreds issues and label them with two things.

A

So one of them is milestone, so whether this is for 1 0 or the nebulous feature, and the second thing is the priority or the pipeline, which is p0 p1 p2. Okay, so let me, let me show you quickly how this looks like a bunch of them have been known, have been close, but not all of them. So right now, for instance, these are all the issues across East yon.

J

Are you presenting something you know you're not seeing because.

A

I also yeah I. Also, let me let me try to do it again. Yeah.

C

A

Presenting also I was showing that epic high level ethnic with the Sabbath sample are you able to see how yeah.

I

A

Oh yeah, no, it tells me ok. So actually what I was talking about? The previous topic was this epic and how it I, I'm, using Zen hub, so I actually am able to see all the sub issues that belong to this epic. So, for instance, they are here. I can see.

A

There are, for instance, two out of ten issues completed, ok and whoever wants to follow the work on this particular aspect can look at this epic and just see how the issues are closed or like, for instance, from the ones that are closed, go to the the actual fix. So if you, if you are interested to see the mesh config protocol server, for instance, so that's the issue, but it was fixed in this PR. So you can see the changes easily I'll try to keep this nice and tidy alright yeah.

A

So this does requires an hug to be able to get this nice view of the things right.

A

Ok, so any any other question on this now or it's good, we switch back to triage, alright, so Louise here also, so we actually started to talk about the triage and I. Wasn't sharing my screen? I have to redo a bit so yeah, so, for instance, right now we have like 700 open issue in each do, but let's not worry about all of them, but if I I go I filter by networking, for instance, I am still left with 243 right and if I go by milestones, which is like 1 0, you see we have.

A

This is I, don't know what this is, but I don't think it's used the snapshot or we do have them 1 0 milestone in the nebulous future. So let's take the 1 zero milestone, 89 open, that's already much better and out of this, some of them actually have a p0 for which of which I cannot sort. So I cannot sort here based on a p0 right reason.

K

Of that oh yeah.

A

We need to load the other board. Ok, but let's see, let's see so I can do that in a moment. I just basically want to say that out of this 89 issue, some of them are signed, but they are not necessarily assigned to somebody that can work on them, so they're sometimes assigned to the lead or to somebody who commented in it. So I think this is. This is a call, basically a call for help to the community to see you know if you can help with fixing some of these p0 issues.

A

I'll show you how the p0 looks like so, let's see, maybe I won't take these ones troubles. You think it's your CTL proxy configuring provements, since we just talked about the proxy config. So, as you can see in my screen, this has pipeline p0 here, where my mouse point. So this is something we would really like to to have in 1 0, because it helps like for a production or quality.

A

We need to be able to troubleshoot I think there is no question right, so this is currently assigned to Liam and I think he mu said to you. You do plan to to get these to a much better state. So this is great. Maybe you can remove the Community Help Wanted label. In that case right sure yeah there are. There are a bunch of other issues which, like need owners and.

A

Yeah, do you want to add something, or did you have something else in mind, so.

K

Right, Network kind of showing people how to manage and use the list. We also kind of need to go through and like maybe take some of these, and you have arguments about why they should or should not be abused. Well,.

A

K

A

K

Okay, because it's.

A

Change it's right now or let's hear about that. I'd.

K

Like to hear it so it's marked as p0 I, don't know how I got Martha's p0, but I did okay yeah, so it's 30 very important, be able to diagnose customer issues. Mm-Hmm.

C

K

Would we block shipping Emily's on this feature or not being present I'm, not sure that we would oh yeah? That's not so that already makes a p1, not p0, b p1 then targeted for the one on the milestone. Well,.

J

Not so we not necessarily this particular fix, would we block, but if we can't debug and know the state of the mesh I would argue that is p0.

J

You have to have debug tools to be able to convenient. You easily see the state of the match.

K

The thing that would make an argument there about is easily right. We've already demonstrated today that we can get the state of the mesh out right. You have to follow steps and documentation to do it.

K

Right and we asked our users to go through those steps and documentation so right, it is possible to debug the state of the mesh because we're seeing it happen, it's not as easy as we would like. Clearly, it's.

J

Probably not it's probably not documented that well I mean forgive me if it is, but you know I generally use the this form of this tool myself, but maybe it is documented well and I, just missed it, but I think it's not right.

K

So I think in the long run right we want to have the tool whether the tool does everything we wanted to do at one hour you can clear you haven't right. What we want the documentation to be is dump the tool right, one, the tool get the dump and if you don't understand what the Dom says go get help. That's, basically the state that were in yeah.

A

We also want the single command awesome short right like this. This is really useful because it's very hard to say to be has back-and-forth. You know get me this version of the command get me the other extra command right, like.

K

This is this clearly has not yet right. The only debate I'm, having is whether we would block the release on having you know: baroque instructions that were 80 percent of the time or a nice single line command than what's on representing the time so.

A

The debate here is between, let's say time, to market right, because it's we're, if we're in a crunch, to ship the release, let's say or for whatever reason we want to ship the release fast, and this has not made it in. We will not go I. Guess that's the argument right so.

J

I Louis I wasn't disagreeing with you all I was saying, like fundamentally I wasn't trying to disagree with you I'm, not saying we need this particular command or any particular command. What what I think we need is to make sure we're happy with a set of documentation that tells someone how to debug the mesh, that's all right and whether it's this command or some other command or some other implementation, I'm not saying that it has to be a specific way. I do believe, telling someone how to debug the mesh clean.

J

Clearly, okay, even if it's difficult for them to do it, is a p0o.

K

Yeah I wouldn't argue that we at least needs the documentation right, reasonable documentation.

K

It's gonna, even even that's, gonna, be hard to prove coverage right into yeah.

J

There's always say: there's an interpretation by in the eyes of the beholder right right.

K

So we can, we can make that you know writing that kind of documentation and a fixed effort and say somebody should go and spend three days making the documentation better. It should cover these things right or we can like, try and define some qualitative criteria on the documentation which is really hard right. I purely from a scoping perspective right.

B

When, when is when are we planning to release pompano because, honestly, this this entire piece of work, I, could probably doing like.

K

Difference between it's totally reasonable for people to work, empty ones targeted for the milestone. Well, that doesn't necessarily mean it's peak like Liam could go get preempted by some other work. Okay,.

A

So we also know one is spend like an inordinate amount of time on a particular issue, because we have 89 I.

K

Think the point that's worth making in general, if you look at the distribution of bugs that you have alright, a very very small percentage of those should be p0 s. It.

A

Doesn't represent any.

K

Realistic priority yeah yeah.

A

Yeah I agree with that which we still have quite a few, that.

A

K

Yeah so I suspect we're probably more likely in the range of having 10 to 20 actual key zeros yep.

F

A

Right so so here it would be like I'm, not sure we are during the span of 20 minutes. We have lengths. We can really go through this thing. Alright,.

K

So I just wanted to like to go through four or five okay.

A

And discuss about.

A

F

L

F

You scroll through the list of things right, then you just quickly get an idea of two. What are the biggest things that are like missing, I think.

K

I used to go through this exercise in triage, because we all have to do it right, like I, can go through or Sriram can go through. Around I can go through every single bug and assign priority, which you know kind of it has to happen anyway in terms of triage right within the book gets assigned to somebody right and they kind of have to agree with the rationale right. Cuz, they're gonna do the work he's.

A

Timmy suggesting should we focus on unassigned p0. Yes,.

M

A

Let's he may be assigned to nobody, it's a actually look at this, the first one about registration with non-default cluster domain. So this is actually a customer issue. That's the reason why it's a p0 and they have no workaround for that. So I, remember I mined this initially as a p1 and then they're saying that this is really like blocking for them and I am Not sure I.

K

Actually know this issue pretty well.

A

K

It is a p0, and the fact that is 2 doesn't work with any two Rhenish cluster that has anything other than the default cluster DNS structure mm-hmm.

K

It's also a really big thing to fix, ah so it's not clear that the milestone for this should be Wanna.

F

A

Number 6 3, 1, 3, I'm, sharing it I think.

F

We just fixed heard yesterday, or maybe it was something else.

H

K

My concern here with this one wasn't necessarily what it would take to do the fix, but what it would take to have test coverage for it. Do we have end-to-end test coverage.

F

Yeah, that is hard because you have to launch an entire cluster with a different DNS name, and that is you. Don't typically do that in gke or anything else, we probably have to launch it in so.

K

Basically, what that means is that what we might have a fixed, the basically the feature support is an alpha because there's no consistent testing for it. Yep.

A

But the customer would still be happy to try the face like.

K

That the word we're always happy for customers to traffic yeah.

A

K

The sort of milestone for the like is the issue resolved no right this, because we don't have tests mm-hmm.

M

K

Then the milestone would be post 100, we could say. Look the at least a an interim fixes in place for the issues not resolved mm-hmm.

A

K

That make sense good.

K

Christopher I get I, haven't looked at the PR or the work that you guys have done for this. No.

H

Yeah I think your your general coffee, yeah, the general comments make sense. Okay, thinking that further should should that be the criteria for almost any p 0, slash, I, think the majority of stuff would want to be in beta for the zoo, GA release. So does.

G

H

That we're a feature to be considered that we need to lock that on proper and ends for each of those features. So.

K

We have the test criteria right. Mark features through the different stages. Alpha beta and Gi is even a an issue template for features that has the list of requirements right. So this is a bug right. This is a customer reported issue or.

H

C

K

Be is a feature enhancement request, saying we need to support this yeah right, so they might be okay to close this one I hope they're, the ones get opened, yeah.

A

K

That one could be from another milestone all.

A

Right right, so actually I think we need to get better at that feature matrix so I had a thing, go we just like. Initially, it was rough, just the type of protocols we support you in breaking. You know just the major features, but maybe we need to get into a major.

K

Announcement request github should be informing that yeah.

A

K

Sorry should be informing the release, notes, yeah.

A

I'll, try like to gather all this, especially with us and I. We have so many like variations. For instance, I would like to make sure you know we covered. Let's.

K

Go through a few more yeah, because this is getting so I think Christopher walk. What should happen here is because this was reported as a bog right, not as an enhancement.

A

Yeah you for them it's about, because they cannot run the plaster right.

K

So it's okay to close out the bog, but we should open an enhancement request on networking to say we need to properly support arbitrary DNS namespaces right and that should be tracked, like any other future development work. So.

A

Christopher, will you please take this action because you are like Greg, whoever did the work yeah, it's Greg on the call.

H

Yeah I'll follow up with Greg after.

A

C

Ok, great that's.

A

Me you know, and yeah I actually wanted to say that for10 we don't have to have all or the majority of the features in beta right, but we need to have the core features in beta right like we may have many. Many smaller features that they can be just alpha.

C

A

Right, ok, so this is actually the result of the security bug we fixed in 0-8, where the where one particular invoice can be used as a trampoline to reach some other service by who finked the halls header and the fix was to use the inbound ports to like, basically not to capture the entire traffic. In my and to let some traffic bypass envoy and.

H

A

The requirement was, for every part, actually list their container port, so we can use them in inbound ports, and the few people have complained that this is I guess to matter. They don't know about, and we have another like another design to fix that. The initial issue, which requires either like removing the wild card listener and this thing every single service in a listener or otherwise splitting the listener into inbound and outbound. Okay, I guess: that's a p0! Yes,.

H

A

1 0 and the the Amboy part will be assigned to your true, so there is a corresponding and voice change if we're going, especially with the listener, speed and we're going to need some change in pilot, it's not going to be horrible. In fact, it's going to be very easy. It involves having to port like changing the IP tables, to direct traffic, to two different parties that the one depending on the direction of the traffic, and also to probably generate the inbound and outbound listener.

A

So it's going to be more like a flag change somewhere in dissonance. So this is not the signed.

K

Sighted, pure because he's doing the primary work.

K

That the reason why this is the p0 is because it's a major usability drag on the product right we make people do more work than we would like them to have to do, and it also didn't meet kind of the don't. The no surprises usability goal right. They installed it and they want the distant one for everything, and this deal wasn't on four ports that weren't weren't declared. But traffic was still flowing, so they didn't see it.

K

So those were the kind of two usability expectations, the product that we're getting violated and that's why we need to make it changer.

K

Okay, now that is one that would block the release. It's a major user experience time yeah.

A

I think this is probably not a p0. Does it be two so I don't have a way to sort nicely like to filter by these.

K

Are all the old and.

A

That's a p2, so the problem is we actually assign them to somebody, but that doesn't mean that yeah.

F

It's nice temporarily assigned it to like a bunch of people that they can actually ping the the posters and then ask them for updates, and if not, they would close the issuer's tail because most of them were like pointed at like 0.7 or 0.4. Release and we'd ask them to upgrade to 0.8 to see if they shoes to purses, but if they don't respond within 10 days. We just closed the.

N

Issue yeah, so we're gonna have the stale bot stuff Christopher kindly.

K

Kicked off that process and I was talking to said this morning about getting that turned on that will help right it pretty Leon customer issues right where we need more info. If they don't respond like the milk, isn't ping them yeah yeah it'll, also ping us on issues that have been assigned that people haven't done any triage on.

K

You know the usual kind of 30-day 90 day stuff.

K

You know, envoy is already doing this. I've got pinged a few times, I'm sure I'm, just gonna. Taking a few time it overlaps.

H

K

As a project management tool, so I'm looking forward to getting that turned on.

A

Okay, like the thing is it's this imbalance, I have 16 of them open assigned to me. I, don't think I can work on all of them. So what would the process be here? Should we clear you.

F

I mean so it's like I used to sit and go through, like I mean a hundred 100 of them, and just this was just like one github page, and he just like go pinging each of these authors and then again come back after a week and a half and if they haven't responded, he just closed the issues, but maybe the tailboard would probably help, but this was just a way to like you know, for a human to go and see if some of these are still relevant or not, and based on that, like you know, you could just like close.

K

So the creation process so that we know that bugs have been triaged history.

K

One of the goals, obviously is the triage- is to assign bugs you usually what you have is some kind of automated system that you know when bugs come in it just kind of ran Robins the bugs among the available people in a particular area. They'll, do the initial round of three hours they'll either pick a better assignee or they'll, remove themselves completely or they'll. Keep the bug. Okay,.

J

K

You know the first phase and then once the bug has a real assignee, then you know they're it's their responsibility to maintain the bug the stale bottle problem. If a bug is unassigned, then it needs to be triaged again more. It's marked as Help Wanted right and then it can stay on a side. Okay,.

O

K

We don't have all that automation in place right now, but yeah.

H

A

You go ahead to Chris, something like.

H

Four issues: how would you expect that's work? Just look at the owners file, maybe so.

K

What we, what I'd like to have, is when an issue gets raised right if it doesn't have an area assigned to it all right, there's a triage process that gets an area assigned to it. Each area is assigned with a group all right, so we already have github groups like network hackers right, so what the the tool would do is it would just pick somebody out of that group and say: well, you guys are collectively responsible for the area. One of you get this issue to triage it all right.

K

So that's how we would do the kind of first round basic load, balancing, okay, yeah, I'm working.

C

On that first kind.

H

Of PRS right now at the moment for Envoy, but that should be portable. That I can do that for ask you as well yeah.

K

I'm happy to maybe have a more detailed conversation or thread offline Christopher about how this we'd like to get this set up. We're going to talk about some of the steel ball, but we can talk about it for triage in assignment two and we have a tool in turn called blunderbuss, which it does. This just shoots. One of the people in the group with the issue.

K

That's a reasonable way of doing it, so I.

A

Think it would be good for everybody to actually do the exam, sighs of checking their own issues like dishes that are signed and see like what is reasonable, ready for 1-0. What Rama described like think the author? If there is a request for more info by the way there is a label called the kind info requested, I think need more info. Okay, so feel free to attach this label and also request more info, and if you think you can work on it, keep it assigned.

A

If you think there is like not a reasonable chance to work on that at least and assign yourself add helpful needed community help needed label so that we, whenever we go through the issues we can see actually what doesn't have an owner right, yeah.

K

Yeah I mean there's always a background process for triaging on assigned issues. What.

M

I would suggest everybody.

K

To do is to go through all the issues currently targeted for the 100 milestone that are p 0 or P 1. You don't have to go any further than that. If you're going to get past your P ones, I would be amazed. Yeah.

A

The only thing with the p0 I think they really need to install the Zen hub yeah.

K

You have to install the Zen hope.

A

To see the p0 like, let me see if I can like load it here. You.

K

See it in github. So if you go back to the get it uh-huh analysis right, you can filter by it, but it's showing you to do Oh.

A

Actually, yeah yeah, that's true. There is this little thing here where my mouse is mom yeah, very.

F

A

K

You filter by the assignee as yourself, then at least you're crazy, yeah.

A

Actually, we kind of have one p0 1 P 1, so there are probably more PC rows than we want like I think there are mobs for tea, I would say: I, usually inspection. If.

K

You want to filter by priority and milestone. Then you have to go into then up to do it. Yeah.

A

I, don't know what oh there it is. Okay, so then hub looks like this: it's a Kanban style, and here we can maybe.

K

Select yourself as the assignee.

A

Okay, let's see and I saw myself I want to see ways done and okay, the PC here. P ones are here and Disney issues new issues. They have not been like, assign the priority yeah, they are not. They are not triage with the priority they are with um with area, so they both belong to area. Networking right.

K

A

Understands this, this.

K

Is what I used to go through all the new issues and do a kind of first line triage, because we don't have any other tooling in place right now, yeah, but it it lets. You see how many peas arrows P ones you have and you can tie them back to the milestone. So you can kind of look at what work you're currently slated to do for wha now. Actually,.

A

I'm going on assign myself from this one right because I don't plan to work on it and I think people will not help here. If there is already somebody assigned it's.

K

Also in the nebulous future milestones yeah.

A

K

So feel free to on assign issues.

C

From your cell phones, ones, yeah as.

K

Long as they have a milestone and a priority yeah when.

O

You do it much better know.

C

This yeah, let's take Martin here.

A

Strong, okay Martin, you have quite a few, also see.

P

Those ones I haven't been able to a farm out yet so some of them need more info. Some of them I I, haven't been able to find okay, uh-huh.

A

Okay, yeah, but look at them. You have like. Oh my god, you have quite a lot of zeros. So.

K

Let's pick one of Martin's around them: let's create it for him. Okay,.

A

Do you want to choose? This is a hard one. We know.

P

That one you can skip over because those ones I am working on. Okay. How about this one? This.

A

Does not look like a p0? Does it some incorrect warnings.

K

So it really doesn't look like it blocks. Anybody using the thing in production, yeah.

A

Okay, but not somebody can really yeah so anyway, how about we Marcus is a p1 I want.

K

To pee too, maybe.

A

So I mean this my height some serious issue right, maybe.

K

We're still prioritizing it lower than knowns your sisters.

J

The way it's worded an incorrect warning, I want I would say it's p2 if it's really an incorrect warning, but maybe it's just the language yeah.

P

I think that it's, if you read through the bug, there's more to it than appears on the first few lines. Okay, do you think it still keys? Let's.

K

Go down the bottom.

A

The warning with them- oh.

K

This is the customers using it.

A

So you said it might actually hide the problem that would cause traffic laws. Yes,.

K

When you triage your bugs, feel free to change the type of the bug or actually represent yeah I think I know where this one is coming from.

A

Okay, so actually we might have a PR on that, so you might be off the hook, but maybe I should assign into our I'm glad yeah.

P

P

A

The problem is he's not in history organization and I cannot assign to him. You say like you're not here, so that there are sometimes cases when people want to work and we cannot assign they directly. The issue to them. I think.

P

You should be, though, because he's not a few peers right.

O

But they make requests to join. Ok.

K

The TOC about whether we, if people exclusive collaborators, read access so that they can be assigned issues, there's a permissioning problem in github. That makes this a bit of a pain. Yes,.

K

Either way, if is to get a Bishu if the PR has been merged.

C

N

We disclose issue right, so the issues be closed. Yeah.

O

N

Does much just right.

K

Yes, why did you not closed.

P

I think we could potentially close at this I mean I'd like that and have a look at the the face.

K

Losing bugs is always the best type of story. Okay, so I think one thing, that's generally useful. Is it's good to set aside some time to just go through all your bugs and do the triage and if you close and close them, if you like dole the priority, milestone things so that we have some idea what the aggregate backlog is? If there's right now, we just have like this heap of peasy rows and people came in when it's one ownership and you're like I. Just don't know because we won't repeat like these are us.

A

But- and there are also there is work happening that is not necessary associated with an issue so, for instance, Shriram has a list of you know things and therapy arts coming, but we don't have issues for them. So it's very hard to predict. You know like in terms of when we are.

K

A

In my role, that should be an issue whether.

K

It's a feature, an enhancement or a customer bug associated with every every piece of work. That's going on in the product.

A

Because that can be used to justify like the quality of the release and when we ship and when not to ship right. So.

K

I'm, sorry, if you are doing work, that's not associated with an issue. I go create a feature. Enhancement request, give it away levels. So we can.

C

A

That's it and if you have P zeros and they shouldn't be p0, please change them to lower priority, because at some point people will start chasing us for p0 is like the TPMS will ask us every day: what's the status on this p0? So if it shouldn't make a zero change the priority and we're actually running out of time with the try exercise, so any last minute think.

K

Does everybody feel comfortable that they can do this.

K

Or do people feel like they need more context about what's kind of needed to stay on top of this stuff.

K

F

A

F

Is a bunch of issues it says like get protocol next to beta and then you know, get protocol wire to beta and then I think, when, a few months ago, like there was like about a million issues, creators saying that incomplete tests, X I, think of them, but there is still like some very arbitrary like get G RPC to beta, and what does that even mean I guess like maybe in key RPC, there's probably one example of a long-running connection, but it's it's there's a whole bunch of them which are marked as p zeros, and these are like I think these could actually be closed or I.

F

Don't know what is the criterion for yeah.

A

So what was missing for those was a good test coverage right so that we can claim it's better. So, for instance, we discovered the problems with long-running connections right, like TCP connections, dropping because of certificates refresh. So that's so that's why those issues exist.

A

However, if you, for instance, are confident that we are, we are like in good shape, which ERP see as an example, and we have a long-running one action: we're using GRP c, jb, c type of traffic and everything right like you, have the authority to actually decide that particular protocol is in beta and just.

D

A

It's it so those who are all about testing and about our confidence in a particular feature and like all these protocols, they are I, think they are required, features for one zero right like we won't. She pissed your without teasing good shape. Yes,.

K

A

K

Those issues are really like epics right coming in. Well, they ultimately break down into is a bunch of smaller issues. I, don't think they're. Currently, tractors FX those ones I'm.

O

Not sure yeah so.

K

I think you know, as long as there are detailed bugs covering the gaps, I think it's ok to close the umbrella ones.

A

What would that help the.

K

Romans are nebulous right, oh well,.

A

Not in this case, like.

K

As an example, mm-hmm.

M

K

There was only one outstanding technical issue: around long learning connections insert rotation to Sager. Bc was fine right if you want to treat the GRDC one as a net back and say well, there's only one outstanding issue in the epoch, and it's this one or you're, not that treated as an epoch and therefore you're, just better off having a more detailed following getting rid of the old alright.

A

If it's one to one but like look in this case, high availability for pilots right so right now, what did we have here so like under H a for pilot? We have a bunch of things, a about auto scaling, I, don't think it's in yet right. It covers like lightness and readiness checks. It has like update, applying continued to function like all these I, maybe not all of them.

A

M

A

This is an epic and I. Don't want to close it because.

K

If it's not an epic.

A

Either turn it.

K

Into an epic we're done.

A

K

So sure does that answer your question because yeah, yes, yeah those those bugs really are X right there, like you know they represent the life cycle of getting a future production, call it. If someone, if you see some that haven't been treated as epics, then you know you can always think me or Andrew and we'll try and help you figure out either turn it into an epic or we can retire. It easy.

A

To turn something into an epic, you just assign the label, epic, that makes it an epic so.

K

Yeah, you use your best judgment about whether something's in epic or not.

A

Holy yeah, I, pick evil features, I think like a bigger feature that may sprout in few directions. You know we're really going out of time, so I think well, yeah, we'll probably get more feedback. Also we can get it offline, so please feel free to email, the networking, easter networking, Christy or Deb. If you have suggestions for how to improve the triage process or questions about how to assign an assigned just, you know, get a bit more predictability, I think in our process and if.

K

There's a specific issue: you want to help triage on you, crows pinging me and Arirang I. Just.

A

K

Like should I close, explain, yeah, but if you have a question so we're happy to go over the same stuff again, just ping me on slack or any other people know. There's the networking group. We can just bring topics up there to helping people. It's.

A

Very little traffic by email, I think yeah.

K

Just like it's fun, it's like is good for triage yeah,.

A

Alright, so thanks everybody and see you in two weeks, except.

K

For me, I'm going on vacation for three weeks: okay,.

A

K

I