Istio User Experience Working Group, 23 Jun 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Istio User Experience working group June 23 2020

Description

Istio working group meeting, User Experience group, held June 23 2020

A

So I had a small item. I wanted to run past everyone. I have um new columns for proxy config routes and clusters. uh Let me slowly.

A

A

A

So um several people have complained about how little you see when you do um the proxy config summary commands um for routes uh on the ingress. You see very little, you just see this, which is completely useless, and I had written before and uh some code to display more information, and I imported that code to download and made it out for it. So with this pr, the information you'll see is much more interesting.

A

It creates lines that have multiple matches and shows a new column called match. It breaks down the matches, whoever's typing, please put on mute.

A

uh It also tells you which virtual service the match came from, uh which is great. That's that's sort of how it looks on a typical english controller.

A

When done on a regular mesh pod. There is quite a bit more information in the same columns and the match is pretty boring usually, but you can see the this internal name here comes in very handy and all of the pods that sort of uh the domains that it sort of handles. So it looks pretty nice um I had for a while. I'd had a column to display where it went to, but I got rid of that because it became too wide uh anyway. I encourage people to review it.

A

There's also um just like the virtual service here for clusters. There is a destination rule column and that um should help people. People have been asking me: is my destination rule being applied or not, and you never had a you could use the describe command on an individual pod, but just maybe a different easier way to see. If it's you know fine or not,.

A

um So I wanted to um bring up the my hobby horse, the dependencies of the central studio refactor. This is the thing that I have been spending my time on.

A

um I made a small change to the picture uh earlier and I want to show everyone that picture changed first and also, I know lin's on the call and she's going to be showing some document changes and those documents don't include the sdo cuddle part it'd be good to have at least an appendix on lens document with this, and this is the latest, so we have sort of two two routes that um these commands can go.

A

So if you do a command like proxy status, sleep pod, to tell if your sleep pod is in sync, you don't want it to go.

A

Very control plane it's best if it goes to the exact control plane, the sleep pod does and we'd always envisioned it going through this sort of reflector on on the side car itself, there's a pr for that. Then, if you ask for the proxy status of all of the proxies um last time I presented this, we saw istio cuddle talking directly to each one of these, and we decided that that's not going to be feasible because their ip addresses will be private.

A

Behind this ingress, we usually no way to see them, so the story is going to be, even if we know the xds address of his dod we're going to talk to one of them and it will somehow push or pull that information from the others. So this, I think, is the picture for how it has to be done. I don't think we can get away with getting rid of any of these gray. Arrows will always steal cuddle talks to the system.

A

The the need for this grey arrow here, the reflecting one on the side car, is because of our work item to identify the correct control plane for a particular pod in the match.

A

So when we added multiple control planes, you often had to give it a revision, which was a revision in your own, is just system name space, but costing always asked me to make that not be needed, and I think the mechanism to make it not be needed is not what I had originally proposed, which was looking two ways to assist him, but it's this this way for the sleep pod to expose that and I'm gonna do the client side as soon as we have understanding of the server-side piece.

A

There's the uncharted view issue that we talked about an hour ago. um We need an owner for this, so let me bring up the uncharted view. So this is the problem. If there's yours to these multiple shards, you run proxy status. How do you get the proxy status for all of the shards and the?

A

If, if if this was http, I could imagine the implementation custom gave me uh using cluster load assignment having the pod call it siblings. That would be great combining the answers.

A

I'm I'm a little unclear how to do it in terms of grpc, I'm hoping that constant can take over this item and triage it and provide at least an implementation guide. So we can figure out um when, when we're streaming, this uncharted view of xds.

A

B

A

Oh, I can hear you.

B

Yeah so uh there's some discussion also on the on the slack on this topic. um So uh one thing we mentioned in a previous I think ufc meeting was that we can reflect those events in kubernetes events, feed and that's a very simple change. And if you have access to the api server, you can get a stream of kubernetes events and inside we can start the same information that you would get over. Xds.

C

Cost in which kubernetes server would we be connecting to in the case of centralist, dod,.

B

uh The centralist kubernetes, yes.

A

We wouldn't have access to that. Our cloud provider is provided.

B

A

B

Except for that, this would be a lot easier, so I understand uh so uh again. What access you can have is different than full access. I mean I, I it's obviously a requirement or reasonable to not have access to modify pods and do all kinds of stuff, but with our bug permissions, it's not for short term, I mean for for one eight until we have, because what you want to do, what we want to do it with xds aggregation is pretty complicated, especially with multinetwork and other things.

B

We need to take an incremental approach. We definitely want to reflect the information kubernetes events, because that's a good thing to have, and for 1.7 that may be okay for central eastern to give you our back permission to events and nothing else.

B

uh If not, we can reflect them. I don't know there are other ways to to push them to other pub subsystems. We can we we I'm very happy to add code to integrate with cncf eventing and that will allow us to them to push them to whatever knots. If you want or uh any other mechanisms that are supported by by uh cncf eventing.

C

So acosta, I'm pretty concerned about changing the entire uh payload and protocol that we're using for communicating this data with two weeks of development left in the release.

B

We are not changing need. Neither the design was that we want to use event for eventing. Just like cloud cncf. We adopt the same, make the same semantics that event. We define what inside the event and the knock again. We may. We still want to to clarify what we put in the neck or what we don't put, but uh how the event is transported was always supposed to be flexible and and and possible to implement in multiple ways. I can tell you xds with security.

B

I mean with the problems we found is not going to happen in 1.7 completely, I mean for a single cluster problem. It will happen, but with multiple network traversal, it's it's it's complicated.

A

And it may be that we drop this unsharding because I don't think it's reasonable to expect access to the kubernetes for centralist stud. You know we have to drop proxy status for sample cd. I.

B

I don't think it's a problem uh so so long term. We can get the information the way we want, but short term. We may have right now we are doing good, ctl, electric and and other horrible things accessing. Just the events is not it's a big improvement already.

C

So I'm saying that writing these values to kubernetes events will be available in the one seven time frame to us or this morning as well. That.

B

Seems like a trivial change, I mean it's just three lines of code. Mostly, I know I mean you know the callings of cuba cube api to generate an event. We have it's already extracted. There is a call site where, where it generates the xds event, at that particular point, we can insert additional. You know rights basically right to kubernetes events right to cncf right whatever you want. It's not really.

A

If, if the future is xds events- and I already have a pull request for um proxy status that uses them, uh I'm almost would rather harden that pull request um and do a hack for that. That's.

B

Also, fine, that's also.

A

Fine, you can, you can get uh get yeah sorry it may it may be that if we have a third party providing central sdod that you can't see all the pods, because you don't in the in one seven, you wait till one eight for that, I'm willing to let it slip yep. But um I wanna understand if the you know, if we're gonna continue down the road that I went in this item or if there's any possibility that we should do try to do this ourselves.

A

If I should try to put this in pilot uh or something.

B

It would be wonderful if you can put it in pilot, uh as I mentioned in the past, uh the difficulties around uh you know, multi-network and other things, not necessarily connecting you already have code to do put portfolio.

A

It's all everything is port, forwarding now, there's no more exec.

B

If you, if you do port forwarding uh from easter cattle, you can get the east geode pods and then you can port forward the unsecured port, because that's where I'm stuck really it's it's an authentication and security model for uh for east ud.

B

So because I don't want, I cannot connect to the secure port. I mean your pr. You saw in your pr, it's wonderful, it works, but it's not really something we want in production to insecure connection. But if you're going to do a pod forwarding from the port 15010, you have uh kubernetes security, providing the the encryption and everything else and it's a viable solution.

A

So, for this uncharted view, there's like a design piece and implementation. Can I assign it to you to get me the design piece just sure but document, and can you can we? Can you triage it or should I triage it here.

B

I'm not interested which design you you want here because, from my point of view uh it's I I never said that xds is the only way to get events. I always said that we want to integrate with eventing systems.

B

uh I'm happy if we use xds, I expect exactly the most scalable and and and the best, but uh so.

C

The question is, we need. We need one consistent, uh inventing system that we can integrate, whether we're talking about centralist, dod or vms or any other environment. What we heard from you back in june was that or in may rather was that that was going to be xds. If it's going to be something else, we need to know what it is uh and we need to be able to build on top of it more or less this week. So let me give you an overview.

A

Of the problem before we talk about the solution, so the commands that talk to pilot through debug are proxy status, two kinds with or without the pod, um the version command um the version of the control plane and of the sidecars.

A

This config distribution, experimental weight is experimental. Maybe we can drop it experiment.

A

Well, it can be partially working on one seven, maybe only if sto is co-located or something I mean, I want it to work in general, but what I want to do is make sure all the work I do on this is in the right direction.

A

Okay, so I've got um I've got version, is no longer exacting proxy status, they're all port forwarding it's it's we're moving in the right direction, but um finding the proxy status of a particular pod.

A

Some of these config dump things. I need to write client code that I can reuse.

B

So today you are connecting to each study and called debugging z.

A

Yes, so, yes are we for virgin we're doing that and for proxy status. We're doing that.

B

Then then we don't have a problem and if you connect to each study to to get the status, then keep doing in one second, exactly what you are doing and we you connect basically to each student in one seven continue to.

D

Collect the problem: how do we do that in central istio d, when the history runs in a different cluster.

B

Just like you are doing in one six, so I'm saying that problem is not yet resolved it's what what method will make.

D

So would you configure your instructor to talk directly to sdod for some of the commands and then for the other commands you would talk to the data playing on the other cluster.

B

Short term, it seems that's the only solution we we we can, we can implement with the time left now.

D

Okay, so so now my question is: how do you talk to stud securely from a steel condo.

B

By doing port, forwarding, like kid is doing.

D

That would require you have access to the cluster that runs still d.

D

So that's the exact way. You know the event system was discussed. Permission.

B

Yes, that's that's the reason for to have the eventing system in place, so you can you don't have to connect to anything you can you can just subscribe to a pub subs or whatever equivalent you have uh and that's why? I think it's very important to continue to to say that we support integration with arbitrary eventing systems. Not that xds is the only one and only event that we support.

B

Xds is one default or whatever, but should not be the only one.

A

So for one, so are we saying that for one seven, if your istio d is local and you are the administrator of it or at least can port forward into it and have kubernetes access, then you will see. We will do that and you'll see everything.

A

But if s2d is central, you only see a subset of the proxies because you're going to pick a random one through xd's address and then in one eight we're going to solve this. Is that what we're deciding right now.

B

My proposal was to to just use kubernetes events in 1.7. I mean it's relatively easy to add the code to generate the kubernetes event and then, if you have access to the event feed which is uh lower bugs and port forwarding, you will see all those things.

D

Yeah, can you um maybe educate us? um What are the access to that access to the event feed and also, I probably only want the event for a particular sdod, because imagine that cluster might run can instill d for 10 different deployments. I probably only want to invent one of them.

B

A

We won't aggregate it that's the whole point. We can aggregate it by enumerating the pods as we've done 116.. We don't need kubernetes events to solve this problem. If we have kubernetes access.

B

That's true you're right I mean.

A

We're done the code is already written. Everything is great if we do not have kubernetes access. I believe your claim is that we could it's reasonable to just talk to one of the stods for one second and it'll be a gap and we'll close in a one-eight.

B

I'm afraid it will not work, uh I mean that's the whole point for the status and and and things is to talk with all of them, so.

A

um Let me so let me make another point here. Let me show the picture um in this case in this picture: there's only one user pod sleep pod.

A

um If, if we did this um thing, we talked about to reflect into its existing connection, then everything would work great. If there's one pod, that would talk to the only one that mattered. If there were ones we could talk to all of them. We could. We could talk to each one until we had a representative sample.

B

uh Yes, you could talk with all of them, yes to get the to get status of all the importance. Yes,.

A

To do that, we would need this arrow to be implemented this arrow, and my question is, um I know there was a pr, maybe for that, will that be ready.

B

A

Four one: seven.

B

You know which pr to do.

A

So let me show you what I think so one of the items I have is- and this was from your request- that when you do a troubleshooting command like proxy status, it would um identify the correct control plane. Originally, I.

B

A

I would find that with labels, but of course that's not going to happen, so my thought was that we have this sidecar tunnel to xds all right. You have proposed this. I had made this work item for it.

B

A

We have not triaged it, yet it doesn't have a p priority. Do we think this could come in one seven.

B

Yeah this one, if you, if you have a tunnel to xds, uh it should work. The problem is, if you, if you want on secured tunnel, that's that's where, where the pr is stuck at this moment, so.

A

I a secure tunnel is going to be super important. Maybe, but I mean um you don't well.

D

You don't really I mean we don't have secure tunnel from sidecar to still be on the slash debug api today.

B

Exactly exactly.

D

So do we really need to secure tunnel in this case if it's just getting debug information.

B

uh It would be wonderful to have it, but I think we can. I mean it's, it's not worse than what we have today.

D

Right, I'm just thinking loud here to be able to support, slash debug on secure is probably a okay starting point, even though we do want to be secure down the road, but we're not secure. Today.

B

A

So if, if this could be a priority, it would really simplify things first, it would simplify all of my uncharting things. I think maybe I mean I would rather have uncharted, but I could live with that. um It would help me deal with the security items. So um I have this other item for security. How am I going to? How is istio cuddle going to talk to so-called stod?

A

Currently, I'm using this, this go run tool that people dislike.

B

uh They say make philosophy.

A

Yeah so, but but a make file I mean I it's. I can't tell users to run a make file to make a certificate. They pass to istio cuddle.

B

um Someone was working for vms now to the token.

E

Yeah can't we use the token instead.

A

uh John, can I assign this to you and you give me a better way to do it than what I have here, which I got.

E

Can you assign it to the security team.

B

uh Iris, I think iris did a lot of work on this area. I mean uh or.

A

How do I assign it to security team.

E

uh I think with the oh, I don't think you're assigned to a team. You have to do an individual or add the label.

E

I think it's I think. Basically, today you can use the token to get assert and then you can use the cert with xds, but I don't think you can use a token directly with xbs. Only.

B

We could add this, it's not hard, but I don't know if it's useful yeah um talk.

D

About how? Why is this needed? Because if I can have istiocado talk to the sleep pod or the ingress pod, which can establish a tunnel back to the whichever stod it is? Do I really need this.

E

D

E

Is only if we don't do the tunnel right.

B

Oh yes, but uh in in so there are two separate problems. One one was the issue of uh improving the ux for vms or how do you provision vms and- and I think what john said is the current agreement, as far as I know, to use the token- uh and there was discussion about easter cattle being able to automate the provisioning of vm, so someone working on vms was supposed to have a way to do. Istio cattle uh generate certificates.

B

There was a pr, I think, iris or someone started at some point, but it was doing the wrong way and we agreed to use a token instead.

D

Yeah, I remember that, but.

B

I don't know where it went.

D

I'm trying to connect the dots with how is this help with this cut or whatever.

B

It is if your catalysis, on a vm, it's exactly the same use case same problem with with issue on a vm everything that you do on a vm you do. It is your cutter.

A

This connection, right here um from istio cuddle to the ingress. If you talk directly, is going to need the same security that uh is needed for a vm.

B

A

Think it needs.

C

Better security, it needs escalated privileges above what just a vm would have.

B

No, no, you connect to xds you connect to regular xds. uh You will have some some some credentials, uh you should buy. I mean some speedy identity, uh but it's nothing special. I mean it's just like any vmware vm running into your system. I mean, if you, if you, if you configure the credentials to beneath your system, you have that.

C

So constant last time we spoke, we discussed that a proxy may not actually, maybe you weren't part of that conversation. I think it might have been john howard and francois having a proxy able to list all other proxies in the mesh is a is a security yeah.

B

Oh yeah, isolation, yeah, that's that's obvious, but but uh I'm not saying any proxy, I'm saying some proxies are more equal than others.

C

The auth z should be escalated for that particular connection.

B

Absolutely so isolation will make sure that if you go in the example namespace, you will not able to see anything except your name space, but if you go with a credential into your system or with with policies that gives you permission to see everything, you should see everything. Thank you that will be true for vm's and everything else. I mean it's not specific to to issue your cattle or this use case. It's something that we want to do in general.

B

If you have a sidecar in a namespace that has egress dot star you'll, see everything that that's how it works today,.

A

So carson, I want to do a a one week: sprint, where I either work on this path, port forwarding into the pod and then letting this reflection happen, or I do the path talking directly to the xds. um I want which one of these should I sprint on the left, the right path or the left path.

B

A

Both of them are going to need a little help. This one's going to need help with security. This one's going to need help with this. With this reflection.

B

Piece so the the main problem, if you talk through ingress gateway to an stod, is that that relies on the multi-networking.

B

So we have a limitation I mentioned before. I don't know if people are understand what the problem is with multi-networking really. This is a use case for newtonian network, because easter cattle doesn't have direct port reachability to all supports in in istio system.

B

So for mr cattle, you can only see the idea of of the ingress gator. You cannot go to the pod ipo of each east to the replica.

B

Your problem is to connect to all studies in order to do that, you need to be inside the network and to have the pod eyepiece.

B

So if you go to the pattern left, you go through the ingress gateway right now you cannot reach individual ports, you can you get a random one.

A

I can reach so that's why I don't like the one on the left until this event mechanism.

B

I agree, and I agree with you.

D

B

D

Think I should go well because we actually may not even run ingress gateway on the left side. So I think the right side is more resilient, regardless the networking or regardless whether you run ingress gateway or not.

B

Well, you will need some data to do. You know you cannot have easter the exposed. I mean you need something.

D

Right but it may not be an ingress, it's your english game. It could.

B

Be actually sorry, it would be a great way, not the easter egg, but some load balancers. That is there.

C

I think only dependence, the dependency we have there is that it's a uri that there's some address that we can hit uh that will get us to the sdod right, usually an ingress gateway, the way that we deploy things, but if that's a google load balancer or exactly the vm thing that also works. Yes, for this week's sprint, I can try the right path.

B

I think so forth this week. That's the.

A

Business and that is going to require um again it'll require um the so.

B

A

B

Yeah, the identity of all all uh histod servers should be available by listing by by looking at the cds response. So.

A

uh And I, when, when will I be? When will we we have a test? Pr for this? This is gonna unblock me. If I could get this.

B

uh You don't need a sidecar tunnel to east your xds. You can just connect directly to xqsds. Now.

A

um Using, uh like writing a ref uh reflector pod, what do you mean just.

B

Call it xdsc, just if you, if you use a adsc client in in in the board,.

A

So I think what you're suggesting then, is that I implement that myself.

B

You should be possible, yes, I mean you kind of already did no. That was your pr. You had okay, you.

C

B

Idsc connecting to xds, you used your dna. That was what your care was doing. No, what did that.

A

Was so I could take the code that I had written here using istio caudal, adsc dial. I could move it into um the sleep pod.

D

A

Move it into the istio agent in there.

B

Ed, that's what I'm doing. I took the code that you wrote and put it in history.

B

Well, almost almost.

A

I would love to tell my code that you have ported.

A

B

Let me see if the pr is passing the test. Now I mean thanks to john for the recent refactoring ahead, pretty much. Let's be nimble with that.

A

Okay, so I think um I think I have a plan for this week on those items.

D

And I assume that would solve all for a concern with instill cuddle commands, because the moment you can turn up to back to istiod, you can pretty much run any commands.

A

That that handles um the grey arrows, um we have two at least two other problems, so uh the proxy status that I wrote lists the proxies it does not tell whether they're acting or knacking. I cast, I believe, thinks he's waiting on me from what I heard him say at the work group leads. I thought I was waiting on him, but uh mitch told me today that, instead of solving it, um this way that we should solve it using client status, discovery service, a new envoy thing. Have you guys looked at that.

B

Yes, uh uh thanks mitch for finding it uh so waiting for me and me waiting for you. What is I was waiting for is for uh the ux working group and mitch and and you to decide what proto do you want to use. That's a perfect proto! I mean I'm, I'm very happy. If you pick this one.

C

uh Yeah, I think that would be my preference as well.

B

Then we have uh consensus. Now, that's wonderful! Oh.

D

What's the impact to pick up this one as far as the development timeline, because you guys so earlier, there's only like two three weeks left.

B

The main problem is in in my pr what we have right now in east ud is just a boilerplate to send an event with some. I think I put the node in for some some crap. That is not really useful, because what maybe it is so uh the idea is that each event will have a proto that is well defined and stable and we treat it as an api and you can rely on it.

B

So if we pick this and void apis, then we are good. I mean we'll say that each event will contain an instance of this of this proto.

A

So I think this is good and mitch is being quiet, but I think he said he might be able to implement a lot of this.

C

Costing you probably would have a easier time at it, but if you're strapped for time, I can take a swing at it.

B

I'm quite strapped right now. Yes,.

C

Yeah, so I will just uh I will publish what I've got to you early and often so that uh you can redirect me when I go down the wrong path. That's.

B

A

So I think this will be good for telling us if the sidecars are acting and knacking um for the, which is great for proxy status, for the version command. We like to list the istio version that the sidecars are running um and I think the connections api that we've been using before might do that. So the question, though, is how they sort of interact.

C

B

B

Let me think about it: does uh does. uh uh Can you scroll back to the previous uh with with uh with uh proton, let.

A

Me before I do that, let me show you yes what of my in progress p. Oh this has to do for pilot, but I had put in a more recent version. I believe the version here, but um let's scroll back, which one.

A

The um client this one.

B

Yeah yeah there's a oh yeah, uh so if we we, we could generate this pro. So basically, when sidecar connects, it will get lds and itself. You will have this event being generated. So if you, if you are listening for this particular event, you will know you can probably stop the version somewhere right.

B

I was trying to see if, if we have any any metadata included in this, uh because the version is passed from the sidecar to pilot task metadata, if we can pass this metadata in the event, you are golden, because this event will give you both the version and the knock status.

B

John, do you think is reasonable.

A

So I I just I need.

E

To take a closer look at it,.

A

I just see these configs but any place to hide. I mean version no place to hide version.

F

C

Can you do, can you explain version ed? Are we talking about the istio version, or are you talking about the xds on uh object version? It's the istio version.

A

So if I do uh oops, if I do uh istio cuddle version, it lists three things so.

B

A

I think there are multiple conversions.

B

Yes, yes, yes, uh so, um and the version is passed so so pilot has access to the data. Plane version has access to the version that mitch was mentioning. What is a current version of each of each feed? uh All of them could be made available in the product. Maybe this- maybe let's not have consensus on that proto until we confirm that it has all the support we need.

B

We could still use it and put metadata on top because we can wrap it in in you know it's in a discovery response.

C

In particular ed, it might be that this is just not the right proto to have that that there would be a different prototype.

B

C

B

That's also possible.

A

We we might already have that in the uh sdio connections. We.

B

Do because I'm blindly passing all the metadata I get from.

A

The node, so I may just continue doing that in addition and then use this um discovery status for the knack and act status.

B

That's fine as well. uh I trust you and mitch to to pick whatever proto you need for your command, because I'm not familiar enough with it. I mean whatever you put in the proto, I'm happy.

B

uh My use case is somehow to get a proto where eventually I have the ip address included and that's one of the metadata. So I only need one field in this whole thing.

A

B

And probably john has another one readiness status, because that will also I mean when he implements his ready, not ready, readiness check.

B

The plan is to use this eventing for as a way to synchronize histories across themselves, so they they react quickly to readiness change.

A

Okay, well, I think I know what to do this week to move forward so.

B

If it's easier, having a single proto to rule for both cases, I think is perfect reasonable. I mean because every connection has at least an arc somewhere and if acts contain all the information that we need, then you know we can add later if we want a lighter event, but.

C

I will put together a very, very brief design document, summarizing the use of csds for proxy status, and hopefully we can just approve it over email, yep, great.

A

uh So we have 20 minutes and two items left on the agenda: martin's and lens. How long is yours give me martin.

F

uh Well, it's I could try to do it in 10 minutes. um You know, potentially uh with with offline discussion afterwards.

D

Yeah same here.

A

Okay, 10 minutes and 10 minutes is 20 minutes perfect.

F

Okay, um so uh there's there's actually uh really two parts to this rfc uh and, and the genesis of it was uh just the desire to improve uh the the quality of of logging and uh error messages that we're getting uh currently because um a lot of a lot of support cases, both internally and externally, are I see they have their roots in in just uh you know, unclear or or overly verbose or or just missing logs.

F

So um that's where this work started and somewhere along the way way, uh there's also uh some proposal for potentially refactoring the existing logging api, rather than just having it as a cleanup exercise to refactor the existing logging api to better combine like uh some of the metric aspect of of error and warning events um and also uh make the uh the logging messages be more maintainable uh going forward because, right now, it's it's difficult to enforce. Every single.

F

You know error, warning message that that the developer writes, and so so. A second thing that came out of this is um a proposal to extend the existing log package that we have in initial packages to uh add some some optional functions to uh scope, uh to support that and ed. If you could maybe scroll down a little bit to the retrofit proposal, yeah so so this is a v2 of the um package api uh which, which originally had you know quite a lot of feedback.

F

It was a very rough proposal and there was a lot of discussion and feedback.

F

I I tried to incorporate all that feedback which I left as comments at the top into the second version of the proposal, which is what's in the document right now, um and the main thing I think that I took away from the comments was, was a not to force not to force people to do anything.

F

um So so you know, what's on the table. Right now is is optional. It's uh uh there's no required api change uh logs and scopes will continue to work as they do right now.

F

um The stuff that's in there is is intended to be optional and incremental, uh and uh it includes things like adding structured logs, so adding context and and just local values to the logs which can be passed around right. uh If I scroll down a little bit, please add.

F

And I think the the main main thing, the main proposal on the table uh where, where I think that there is still some contention, as is this idea of a dictionary, so so what this is saying is that, rather than capturing all the logging information and metric information at the call sites, uh what we like to try to do is is have that live in some file, which is at where, where basically, all these uh user facing messages are enumerated, uh and this this file comes under the purview of um you, know, folks that that are more uh have a more interest in in user-facing stuff like like the ux group and and dogs.

F

Perhaps, uh and the idea is that you know we would slowly go through the existing uh body of errors and warnings that are there in the logs and gradually retrofit the most important ones to this kind of format, where you know the the log site mostly points to uh information about the error or the condition which is contained in this in this uh one file, this error dictionary um and besides, besides the content, the user facing text, uh I also included some proposal for hooking in uh things like metrics, so so as part of the uh the dictionary definition, if you could go down a little bit, so we still have, we still have uh the same definition of a metric.

F

It's just that now. There's a list of metrics, that's part of the error, dictionary definition uh and uh when, when the actual log message occurs, like scope, dot, error, f or whatever um having this, this error, struct actually be passed in, would result in.

F

um uh You know this event actually making its way to any any metric listener that that would be consuming these events, so so the the metric listener would actually get uh the context. Labels. uh Also, the the entry from the the error dictionary uh and- and you know it could basically increment the metric or whatever. So again, this this is optional, but this is just showing a way that you can combine a metric handler and a metric definition with with an error dictionary.

F

um So maybe just just scroll down a little bit more, at least to registering a sync. So so targeting is is more just like talking about scope, but I I think that's that's like relatively uh vague at this stage, but um in terms of in terms of just how this is actually communicated to users.

F

um This is this is a again sort of an optional plug-in framework, where you could have a uh like a log handler like a load listener that uh receives notifications, callback, notifications of um local error and warning log messages and receives all the information from that call site, including any context, labels etc, and the dictionary um uh error definition for that particular event.

F

uh And it's able to do some, something like you know, use zap or something like that to log to uh load to some output. um Similarly, uh uh you know a metric handler uh can can define a callback to you know incremental metric or whatever so right. I think that that's pretty much the proposal and I'm I'm looking for an approval for it. uh The state is, you know, I've. I've passed it around, and I've talked with with a bunch of people over it uh and- and I I think, there's generally good consensus.

F

uh I I believe that costume has some pretty strong objections. Still so um you know, I. I think that uh I I don't know if you want to talk about it, boston or.

B

It's what we discussed in this meeting before about having access to sdod or not having access to studies, so I don't think anyone most people should not have access to students, but the entire discussion is relevant from all the support point of view.

A

But we need to make something that is useful, as the user experience of the cloud provider themselves, sure.

B

I I think most important is to have the users the most users of istio be happy the person who is running esqd. I mean it's not really something that uh I mean. I I think they'll be better off using the same tools that the users are using. So if all the information that would be expressed in logs is accessible to the user, I mean, of course, with the filtering isolation. So you only see stuff for your name, space and so forth.

B

Then why not use the same tools more wood behind fewer arrows? Basically, uh for me, any look yeah. Sorry any log for me is a problem. I mean it's something missing in the tooling that we have.

C

So I think uh I I think caution is helpful to think of logs the same way that we're thinking of xds events, uh if we want to take these logs and plumb them to xds events, that's just a matter of a few lines of code and plumbing. uh However, xds events don't make it so that logs are irrelevant, uh we wouldn't make it so that we never write logs. They still have their own independent value, but if we also want them on cloud pub sub or something else, we know how to get them there.

C

B

It's a mix mixed mixing, I mean we don't want all logs to be events and we don't want all the events to be logs. Yes,.

C

B

C

B

Events like arcs that definitely should not be in logs they're, all kind of logs, like debug statements or something wrong happened. I have no clue what's happening. That probably should not be an event.

B

A structure is a crash dump or whatever probably should be an event. uh We we have metrics as well, which, again, if you increment on automatically a metric that you got a packet, you got a connection, that's probably not a log and probably not an event either, and I think we need the problem is that people do not think carefully about each particular use case, as it should be because events, we need to think about the proto, the discussions that we just had about. How do we put status for metrics? It's something that you know.

B

How does it show in rafana need to be stable and so forth and for logs, usually it's whatever crap people put and which is reality today and I'm happy to see more structure in the in this, but I'm not. I don't want to be the main things that we are doing, because then we get an ms anyway. That's my opinion. Sorry.

D

Yeah, so basically the user for this rfc is actually who have higher administration permission to operate sdod. So it's not over normal users who develop and deploy microservices onto the mesh because they probably won't even have us to read a lot. I.

B

Agree but but the super user will have access to all the information that user has, and the question is: is there anything that we put on history logs? That is important enough that we will not send to the user. I mean if something is wrong with your config. If, if you have anything, it's something you would inform the user as well. No.

D

Right, I got your point. I agree with that. So my point is this: ifc is not going to help users debugging their problem into the match. You would have helped user as to you know they have to contact their support or admin to pull out logs into uh of sdod. I agree. I mean this useful error for the user. We should provide an informative way to not fire a user without bugging their support, team or admin to get the logs.

A

So, martin, I think this would be great for us I mean I would love to have better logs, but it would be good to see a demo of how a cloud provider would use this to make things better.

D

And also maybe analyze what the information we should also present to make it easier to be accessible to the user market.

C

I think part of the problem here we we have an outstanding document in the ux working group that has not yet been approved, but we're trying. We keep we're iterating on it, it's toward defining what users and what personas we support in a steal yeah. We do need that. Certainly.

D

C

Common user to be supported is going to be that workload administrator uh who does not have any backdoor access to the control plane. We'll never see us dod's logs, that's the most, that's our most common user, but that's not to say that we don't also want to support the administrator of sdod uh and actually there's a few other personas defined in that document. So that's the impetus I think is on us to get that document pushed through and approved with steve's help, and that will help these conversations move a lot more smoothly. I think, if.

B

I can add one more point. uh I I have nothing. I mean the reason I have strong feelings about this is I have a lot of ptsd with uh with uh privacy teams and because logs usually at least in vendors, like google, need to be approved for privacy, they need to be scrubbed, they need to be. You know, multi-tenant, it's a huge amount of pain to to care, for I mean privacy, information that and p0 sky is falling because you log something that you shouldn't. So I have some some. You know history with this.

B

That makes me very it's not that I'm crazy. Well, I'm crazy, but not particularly here so.

A

Now it's the same at ibm cast and I had to. I wrote a bluemix service a long time ago and I had to make sure that there were no passwords ever going through. It.

B

But it's more than passwords. That means a tenant id which can be used in I mean it's a lot of stuff that really goes into proper and then, if we can avoid the pain.

C

Yeah so martin, maybe the best thing to move forward with this document is and- and maybe it's here and I'm just not seeing it a little bit of guidance around when to use a log versus when to use something a little bit more user visible.

C

So that that it's clear what the intended audience for a given message is.

D

And honestly, I think logo would be the last thing the user want to look at in general. You know if they have a simple way like analyze or describe or look at the events.

D

People probably would prefer that first.

F

Yep yeah uh well yeah, so I I do have some some some ideas around that, but uh you know I I already have eaten uh 15 minutes of the time. So uh so maybe we could just have some follow-on discussion, uh maybe in environments or something in the uh slack channel, but yeah it's currently. This is definitely oriented towards the admins, so so that that is for sure.

F

I I haven't really thought much about uh what what to expose to users and- and you know like how to other than targeting, which was my attempt to introduce some idea of scope. uh I I haven't talked much about how to restrict things to users versus admins. It's definitely admin targeted.

E

Thanks, martin, we just make sure we still have the logs that we currently have. um Otherwise we're never going to be able to debug a complicated issue again.

F

Yeah, so so definitely, this is not removing anything that we have it's more about just taking the the ones that are commonly used and and possibly require metrics around it, and- and just you know, having a little more more clarity. Clarity around it by uh just having more review like from from other work groups, and- and just you know, rather than being ad hoc, as it is like a little bit today, uh just introducing a little more discipline into it.

B

And same for metrics, I mean whatever matrix we have as few as they are. We need to keep them stable because they're in dashboards and alerts and other things.

A

Yes, sir, yes thank you for your hard work on this. We definitely should I need to reread it. We should follow up on this either in environments or in slack lynn. We we use more than the 10 minutes. We said we would, um for your item, tell me um if we should move it to next week or if we should review it separately.

D

Yeah, that's fine! um So basically I just want to let everybody know. You know I provided uh some update to this document, so john kind of would inspire me to you know to think about how do we do this most simplified way by the fact that we already can run htod from a laptop, so I kind of for having like deployment model. One is a new deployment model and two and three are some of our existing deployment model more towards multi-cluster.

D

So I encourage everyone to take a look and I will probably talk a little bit more uh in detail in the environment work with tomorrow as well. So I don't have to you know present the whole thing and here, but do let me know if you guys have any concerns with it feel free to comment on the dog.

A

A

Lynn, um okay, everyone so we've started doing this uh every week and uh it's a good thing that we did that because we certainly had a whole hour's worth of material. Today I will see everyone next week uh and my hope is to make progress on um central stod troubleshooting commands this week, uh and I may be bothering many of you on this call with that, and also some of the control plane upgrade stuff in environments tomorrow all right. Thank you. Everyone thanks! Ed thanks.

A

F