Istio Technical Oversight Committee, 6 Nov 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Technical Oversight Committee 2020/11/06

Description

Meeting of Istio's Technical Oversight Committee on November 6th 2020
Topics:
- 1.8 Release blockers
- 1.9 Release Managers
- Feature status for Annotations (and ENV vars and CLI args)

A

I think we we need to really uh delete the release by week for the fourth fix for us I mean all right, um there's already like two fixes on the way um like I I believe they will land like already next week, so so that we can make a call like how it looks. uh It's not like far away we're following so all right, say: let's get together release by the week.

B

Sounds good to me, yeah.

C

I think that the fix can land um somewhere next week, um but but like something I I would like to say the. I think, the the fix that we have um why just for this release is not really resolve the uh root cause of this um kind of like sds uh um race conditions um from that that exist there for the from the beginning.

C

So um well, that's probably enough for this release, but in the in next release we probably need a better fix overall for the whole sds situation, because it caused the um many chances.

B

I yeah, I mean lazan, you know how it goes. We we have these impartial fixes and they fix some things. They introduce other issues and then we just kind of iterate on that. You know why not just go to a long-term fix right now. What would be, how hard would that be? Yeah, I'm not saying that we.

C

Should go to the long-term first fix for for one eight, but.

B

Yeah, but I I'm saying maybe we should I'm wondering how long a long-term fix would take.

C

um That that's well, I I haven't so so. The basically issue is the. I think the sds is not completing the um like um in the warming like init manager flow in.

D

That sense, like.

C

It could fall into uh some situation that sds is not ready, but cluster or listener to be active and then like it rejects or doesn't connect to stuff, and that that's one thing and the another thing is um for this: one that we don't have the uh for for the uds sds. We don't have the like, proper retry or timeout to form void to retry on that, so that two two things.

B

And, and are they both like? You know, what is the blast radius in those changes? Are they both kind of sweeping changes? The first one sounds like it might be: a sweeping change where you need a design dock and everything, and it would take time, I'm not sure about the second. What do you think um I I.

C

Think post would need some time um yeah it could land in one line, but I I don't think we we can do that in one hit. Yeah all.

E

D

um A second question right, so we we have a fix, I guess in the pipeline. That's fine. uh Do we have a safety net right right? We have an agent and we have an envoy. That's not doing what it's supposed to be doing.

D

Is there any way for the agent to help recover from these situations? Even if it's suicide.

B

Yeah great question.

F

I think we could yeah, we could add a wideness world that would kill itself, uh but the the issue is one of the issues is we would need to get the bug of the clusters, not warming, rather than the bug with the secrets now setting up, uh because if the secrets don't set up, then it will be marked ready and so then you'll actually have major traffic impact. The cluster not getting ready. One is a much better bug because it will never be marked ready, so traffic will never be sent there.

F

So that's less impact on the user like neither of them are good, but the cluster one is is the better one, because it's more broken, and so it doesn't even be marked as ready.

D

And how would we detect that situation externally outside the onboard process.

F

um So we have a, I think the entire readiness probe will fail because the it's not even listening on the readiness port, and so if we added a liveness port for the same thing, um it would also fail, like we'd, basically add a liveness probe. That's the same as the readiness probe. So if it doesn't become ready in like a minute or two minutes, uh then it fails.

D

Okay, I think we should consider having that at least be a full-back clone for getting the release of it.

C

F

Agreed yeah we might want in general, too, there was some reasons we didn't do it, which was a fear of false positives, but we've mostly made those fears go away. I think from other things, so it may just be a good idea in general.

F

That the wideness, that is.

G

H

G

I don't understand why the benefit of having a liveness and already this probe, that's the same test. It's kind of redundant.

F

So I think one would be um like the liveness pro has a longer timeout. So if it's okay, it's not ready for like two minutes or something, uh then we kill, but the readiness probe like immediately when it's ready, which could be hopefully like two seconds, then it will be a marked writing um it's it's for my I've seen it's not that uncommon to have them be the same check, but just different intervals and yeah. As long as.

G

Yeah, so that that's the difference there that I didn't, I didn't catch it uh earlier, if they're different intervals that make sense um as long as that liveness or that readiness probe or the liveness probe, which is using the readiness test, is not checking external dependencies because then you run into other problems.

F

Yeah, it's pretty minimal. Now we used to check like the envoy stats, which would get overloaded, sometimes by things that didn't cause the pod to actually fail. So that's why I was always hesitant to add this, but now uh we're just going through like the real traffic path. So I'm going to be concerned about adding that okay.

D

So do we do we have an issue trucking doing that john or uh I'll create one okay? Can I can I assign you this.

D

I

So what did we decide for the short term? Thank.

B

I

J

Not so dangerous, yes, no! It's huge.

K

Tactical fix and.

I

Releasing or are we making a long term fix.

D

I think lasagna was pretty clear that the long term fix right is permanent in time.

B

Yeah, it sounded like a stretch for one nine, so we can't hold back one eight for that exactly.

F

B

Yeah, it could have been one eight.

F

Yeah, what about the short-term fix? Is there anything that we should do to accelerate that or just make sure you have enough coffee or.

C

um Yeah, that's much it's probably just my coffee. Oh.

B

Lazan's on the case, I feel much better, that's good um all right! So, let's give lauzon some coffee and then uh I got diverted that the health check you know kind of detect and destroy uh the the scenario that louis was describing. Do we want to explore that in parallel.

F

Yeah, uh I think we should do that, but I would still be pretty concerned if that's our like, if we ship it and we just slap the band-aid on it's no.

B

We're talking, but we're talking about both john.

F

Yeah yeah yeah, as a fallback mechanism, sounds great, but okay.

I

Go ahead. Do we really want to do this for, in the one-day time frame, I'm worried we'll break more things, just because.

B

Yeah, if, if the health check- and you know, check and destroy thing- has got uh false positives, we're in trouble. Oh.

D

Yeah right there just to be clear, the check and destroy things should be an opt-in feature. I see okay.

B

Yeah, actually I like that so for people that are reporting the issue, we can give them a break glass yeah, that's a good approach, uh so who's working on that who wants to work on that is that you, john, or that somebody else.

F

Yeah uh you can assign to me- and I I mean I.

B

Think you'll handle it well. Thank you.

D

um Okay, so we're saying: okay to delay, pick up envoy fixes and possibly.

D

Readiness compensation, optionally, that's on right, delay, release.

D

D

Well! Kind of right to everybody.

I

Does this cover both the issues or have we just talked.

A

About one of.

I

A

We need to find an owner for the second user as well. I think it might be a different issue, so it's about aggregate cluster uh like just reported yesterday, so we won't have a root cause or an owner for it. Yet.

L

Let me indent that.

D

I

All right how's, this gonna, do the same thing. Thanks.

D

Do you want me to present the second one.

D

A

So we saw also an overcrash, and I think it's like um also not just notices in the upstream and with pr test, so uh it could be new. um Like the the reporter said, we cannot reproduce this one spring, one, seven. So it's a progression.

A

I would also mark the sensor in this blocker and we also need owner to investigate this.

D

And this occurs with onboard filter only because.

A

Yeah, this is every cluster, so away, filter.

I

Have we verified that or we are going just by that issue saying we have to create our own web filter trigger right.

B

F

B

F

Send aggregate cluster config through pilot, so we have.

C

F

In onboard filter, config, that's completely custom, that's seems like it's not the highest part. I mean, maybe it's at least, but it's not as bad as other things.

B

But we have actually told some of our users to use this to use envoy filter to configure aggregate cluster. I think we had a discussion once back when we assure him, as one of the networking work group leads um where he basically said before. We create a first class api for aggregate cluster. We should just let them try it via on voice record and then advertise it later.

D

But josh that still puts this in the development category, not even the alpha or beta category right.

B

So, okay, fair enough, not a release blocker, but who's working on fixing. This.

I

I would caveat it as a known issue in my release, though,.

B

Yeah, I agree, but we should get somebody fixing on it too. Like can this be a p0 but not a release? Blocker.

I

I don't think it's even a p0.

B

B

Knowing who I know uses this um we'll have to get this fixed I'll, take that off offline.

D

uh Yeah I mean clearly like any fix, is probably okay to go into a patch release right. I think that much is clear, yeah yeah, absolutely um and there's no reason not to fix it right. It's just a question of priority and does it block um other toc members.

D

I would be inclined my personally, I think I agree with the.

I

M

Very uncommon scenario- and I would imagine it's only with aggregate clusters- it's not a generic algorithm.

B

I think that's right right, yeah, so we're just saying um uh put in the release. Notes is a known limitation and then somebody will fix it when they can yeah and.

D

I mean right, I mean if you had an envoy crash and, like you were using dubbo right and use the only filter to configure it.

D

Agri-Good filter is a more intrinsic feature. I agree with clusters more feature of envoy than more extraneous things, but.

B

D

B

B

B

And and uh peter I'm I'm going to see if you you can die, who I know should be on this call and see if you can pick up the fix.

I

Good good good work by the release managers to surface these.

I

D

um Now, obviously, given that the other fix may take a week if we can get this fixed all well and good in the timeline right as long as it doesn't impact.

M

Other things yeah.

D

um But we're not going to get.

J

D

What should we tackle next uh there's a couple of things on friending or we can go into next topic.

N

We are still sharing the bump.

O

D

uh Are there other one to late release items.

N

uh So other than these two bugs they're testing, you still need owners. I sent out the email yesterday. There are six zeros uh but still need owners. um Yeah, six.

D

uh Do you want me to update the verbiage here.

N

uh Yes, please I I it's from the last. I forgot to do that. Sorry. Can.

D

You just update it. Yeah.

N

I'll do that thanks!

N

D

um Yeah, so please work group leads, try and find owners, and if, if we don't get owners, then I guess the toc members will follow up offline.

D

D

Do you want to go to nate's.

D

Topic or we could talk about online release managers or the telemetry api, any preferences.

N

D

The least manager stuff, maybe is uh pretty quick, so maybe we should just get through that. um So there two proposed release managers for 1.9. Do I have that right, uh jacob and uh sure.

B

Oh, I'm sorry and I didn't add also we had steven landau from google that was another nomination.

B

uh And if we only have three, then it's really easy decision. What do you guys think yeah.

M

And those three are doing good work already so yeah.

J

That's good to me.

I

Do we have some share on this call? We don't want to sign something without just asking him again.

K

I've already asked him he's more than happy to do this.

B

How do we know that you've actually asked him, and he said yes, kevin.

K

I could share the email response where he said. Yes, please. I want to do this awesome.

D

uh I'm just I think that sounds like a proof to me right.

P

Thank you very much.

D

uh I'm just gonna move that.

J

D

um Mate, do you want to start talking about uh your thing? Well, yeah, learn how to use google docs sure.

Q

uh Yeah, so this this was kind of like a journey down the rabbit hole. um I I started uh thinking I wanted to promote the locality load balancing uh label to the api since it's kind of a hidden label down within its ud. um When I, when I started kind of going down that route, I realized api uh istio api it. It uh doesn't actually current, currently document labels as an api.

Q

uh We do this with annotations, but we don't want labels um so then start looking at labels or the annotation stuff and the machinery behind generating the docs for that and realized man. We we need to give us some love. um He had taken this kind of arbitrarily assigned uh feature status and there was like this combination of incorporating the feature status in the in the name of the annotation itself, uh which doesn't seem very user-friendly going forward once it goes to a different feature status.

Q

um So I so, I think, we've gone. uh I I rolled up some prs and- and I think, we've gotten some general consensus around not wanting to embed the feature status in in the name of the annotation itself. uh But then the discussion became. What should the feature status be of all the annotations that we have today? uh So that's that's kind of like uh I think what I'd like to kind of discuss here. um So we've got a bunch of annotations in istio api already.

Q

um I I just by default they're, just all alpha, um because you know, let's, let's start with baby steps, um but I, but I think, they're, certainly not all alpha right. um So uh in some of that discussion it did seem a little arbitrary. Like you know, a lot of the comments were well.

Q

This is probably at least beta right, uh but it'd be good to to to kind of like as a group kind of decide uh how like an approach for how we should like categorize, these save for one nine, um so that we actually have like something. We all agree on and a way to kind of assign these feature. Statuses for for annotations and labels going forward.

D

Yeah thanks nate, I mean I. I actually really appreciate you taking the time to go through and put effort into corralling that topic and yes, there's clearly uh smear.

D

Yeah I was trying to be judicious.

D

D

Do we so one it seems like they were getting pretty good dialogue going on the pr where people are taking reasonable ownership of the status of the things.

Q

Do we think that that's fine, I guess uh like do we? I mean it ultimately comes down to everybody's kind of gut feel for yeah. This is kind of beta. This is kind of stable. um I don't know that everybody's necessarily using the same criteria for for doing that, um and- and I think that's a one. One question I think I think it was doug raised- was what exactly is the distinction between the status of a feature and the annotations or labels?

Q

um Do we have, uh I guess criteria, for I mean it's technically an api for the future right. You.

D

Know it should be treated like an api right, so I mean I think there are.

D

You know, there's there's some ways to make these decisions right I mean, obviously, if the annotation is basically a stand-in for a an api right, that we would use like at a different level of granularity like putting an annotation on namespace instead of using sidecar or something like that and the equivalent feature in my car is at beta or in development. Then the annotation should be at the same level unless the annotation came before the api and we communicated to users that that annotation was stable.

M

Or maybe the annotation wasn't tested as part of the feature.

D

Right, like we know if the annotations are tested. Yes, awesome.

M

So, look I I think it makes sense to be alpha by default and if there are appropriate tests it makes sense to be promoted to beta. I don't think it makes sense to be something beta without any test in place.

I

Yeah so I was gonna say why not just follow the same graduation phase that we have for other things, because these are apis right. So.

D

Right, I think the issue right, I agree, but we're we're not starting from zero. Exactly that's that we have right. We we have to figure out how to recover from our lack of progress. To this point.

M

Right, maybe it could be a lightweight process which the focus would be more. uh Does it have automatic testing that we don't necessarily need to regress to ask? Do you have an ifc? Do you have a design doc? Because you know it's already been done.

D

Yeah, our other simple set of questions that if a working group lead answers yes to, then we can accept that that feature that that annotation is beta. I think that's right for the existing ones.

D

um So, yes, is it in widespread use? Is it tested? Is it documented.

M

D

If you can answer yes to those three questions and- and you can provide some other evidence- maybe to reinforce your argument- then there's an argument for making it better.

M

D

Can you can we jot that down here, yeah yeah, sorry, definitely.

M

Maybe we can do a template for that too.

I

Hopefully it's a one-time thing.

I

New annotations just go by the normal process right. The interesting question here would be so I already know there will be annotations, which I invite spread use which are widespread documented, but don't have tests and their users relying on it. So the impetus from us should be for one nine: let's try to add those tests so that we can actually call it beta. Is that correct, louis.

D

D

I

Yeah I was just saying there are a lot of annotations, which I know which will uh suffice. The last two items in your list, but not the first one. So are we gonna, say for one nine that they're still alpha and then or we're gonna. Add.

D

Them tested, I don't think they should be marked beta.

I

Okay, good yeah.

D

Like we're not removing them, so we're not gonna break people, we're not gonna fill the grass right yeah. um I I think there was there's a question about not just alpha right, but also development right. If you don't answer yes to any of these questions, absolutely right.

M

D

Wrong, it shouldn't be an alpha.

M

Yeah downgrade into development.

D

And you might not be able to make a really good argument for that to be in out even an alpha if you're not saying yes to any of those three and if somebody has other criteria that they think that could help qualify here. Please uh suggest.

Q

So are we? Are we suggesting that the only change we would make for any of these uh annotations that exist today would be just to beta, but not stable?.

M

No, this is alpha, so they will be alpha by default and if they go through the three questions, yes to each of them, they could be promoted to beta. If they didn't have any of the questions, we were looking at downgrade them to experimental.

Q

What I'm asking is uh so, yes, they're all alpha today, this criteria would make them beta. Is there any other criteria that would make them stable.

D

Well, leaving our side our issues with crds and our ability to make api stable uh using the normal definition, we do have stable apis.

I

We have stable features, that's how we classify them in our.

D

D

I think this time around, just because we're dealing with a relatively small backlog if a working group lead thinks that a thing should be made stable, they can take it up with the toc and we can sign off how about that.

Q

Yep, this is a good thing. It should be pretty quick to work through. One last question, then, with respect to all the stuff, so we have a bunch of annotations now. Well, not a bunch, maybe like a handful uh that actually start with alpha um in the name. So I think we had talked about wanting to change that. So what sort of deprecation and replacement uh process should we follow here? For those.

D

The the renaming yeah, um so I think you I'd, asked you to track the kind of deprecated buy. So we could do the swap thing right. That's pretty normal practice.

D

You know we're gonna have to leave the existing one and it's whatever its determined state is, uh and if that determined state was beta right, then we have to follow the process um and probably frankly, because the cost of migration is low.

D

The deprecation could be for a long time right. The documentation should update, but there's probably no reason, because we now in theory have a framework where, if the decorated buy was deprecated by another annotation, with a cleaned up name like we could deal with that automatically in a framework.

D

If the deprecation was by an api feature right, so it got promoted out of an annotation. Then it should follow the the normal timeline for deprecation all right and that should be in the generated documentation. We say deprecated by this right and then we track that.

D

Does that sound appropriate to people.

I

Totally like the work.

P

I

Doing uh is amazing nathan, like we have been waiting for this for a long time. So I'm glad you.

Q

Took it on, uh let's just make.

I

It I didn't move potentially.

Q

Just to be fair, this is just because of that locality label thing.

I

We'll take the benefits uh just I want to make sure if we do what I'm seeing through that pr is there a lot that I have commented that seemed deprecated now if a annotation was alpha and we decided that nobody actually used it, it's totally fine to deprecate it right. uh I just want to make sure we clear out the documentation so that people or we give them a migration path. So that might be increasing your scope a bit, but somebody will have to do a scan of the docs and update.

D

Yeah yeah, that makes sense.

Q

We should probably just have a readme in in the annotations and labels folders. That kind of like describes how like this this process effectively. I think that probably makes sense.

M

Yeah definitely yeah. Thank you so much nathan.

P

To environment variables,.

D

uh Yes, I was going to bring that up uh good cool niche um they're.

P

Not exactly the same john howard pointed out that we we tend to think of our environment variables more as alpha level and john you can clarify if I'm misrepresented there.

M

I think we study experimental today.

F

Yeah I mean experimental are also one of those, but I can't imagine labeling an environment variable as beta and certainly not stable. That's just it seems odd.

J

There are some.

D

That are stable right. Yep, like http proxy, for instance, is a standard, and we use that in uh whitebox, sidecars.

F

Yes, you're right forgot about that. One.

D

D

Those are probably stable because they are standards so right.

F

D

F

Standard ones uh yeah, I wasn't thinking of those.

D

F

um That are, you know, used for user customization.

I

So we have removed some options for certain features that were considered. uh Maybe alpha ish, like workload ttls, for example, and as.

C

I

Know the only way to configure them now is through environment variables,.

F

Yeah, so that's that's absolutely true and I think the fix is not to label those as stable, but it's to move them to proper apis. I understand: there's a.

N

Bunch of environment.

F

Verbals that are super useful, like there's the trace sampling, one that mitch gave as an example like these are real things that we need in our api, but the fix isn't to make them the environment variable stable. The fix is to move them to a real api which, for the tracing, what we're doing with the telemetry api for the secret stuff, uh there's a bunch of those actually not just the ttl.

F

I don't know we need to do something about it, probably mesh config, but we haven't yet, but I think that would be the solution here.

R

M

Yeah another key is to kind of communicate. What are the ranges of features to your user for your for the environment of us, because today I think we just blandly mark the whole environment ross as a pre-alpha in over australia. I o, so it's really based on what you know to use them.

R

But that's that's a design. I mean the design for environment, which was always that they are not meant as an api. They are meant, as you know, short term. They will go away next. Release, don't use them except for for escape patches, but.

M

I think louis just mentioned past and actually pass will because they are standards, so they will always be there.

R

They're not true, how do you know what environment you are going to run? I mean who knows where you are going to run history and if I don't be one windows or I don't know.

D

Right, but I mean like they are effective standards costing right when we run in environments that respect those standards, that's how they behave.

R

True, but how do you pass the environment variables to something I mean if you have a multi-tenant? uh What is the sense of it.

D

Right so, in general, I I agree with the recommendation, which is like, if you ever want to have an environment variable, be beta or better.

R

You're doing it.

D

You're probably doing it wrong, and you should think long and hard about what you're doing um it may happen.

O

Right so we should agent, yes for.

R

Agent, probably.

D

Right so maybe no matter what we should probably codify them using the same mechanisms, but we should prevent them from progressing.

R

I think mesh config is a bigger problem, because mesh config is, uh you know like environment variables, but it is a bit more uh api like.

D

Right and then the the obvious thing we issue we have with right. So now we have a really good mechanism. You know based on what nate and what mitch are doing to at a very fine level of granularity capture, features, status of features and almost like it feels level. But we don't have the ability to do that on the field level in apis um right and that's really problematic in mesh config right, because mesh config is the thing for each individual field is a feature more or less.

R

I I mean, I think, the same thing that plays for environmental applies for mesh config. If you put something in mesh config, which is a global that has pretty big impact, not in a proper api, you are probably doing it wrong and we should not never progress mesh config as a better api.

R

uh There are kind of implications of how mesh config works that prevents multi-tenant prevents you know safe rollout of configs.

D

Like this yeah like it becomes a much, it becomes a very nuanced and detailed conversation. I think what I'm saying right now is we don't have the tracking ability and we don't have right and we can't even produce the dock like what nate just did for annotations. Actually lets us produce good doc right. We can go work on the docs tooling, so we can help set users expectations.

D

We need to be able to do the same thing for api yeah.

E

We just use that same annotation or whatever comment thing on fields and photos that we're putting on the annotations.

P

So we we can't use exactly the same because it creates a circular reference between our repositories, but we can duplicate it uh in the package repository and that's fine. My question is, you know: I've dropped the link to where we document environment variables per command on istio io. There are, I believe, 163 that we document. Where are we documenting that these are alpha or pre-alpha? Today? Where are we letting users know that these are not safe.

Q

I think the answer is we don't.

F

No then edited on the on the dock.

R

At the top or somewhere, yes and- and I mean command line, parameters are also, I think, are still documented uh for yesterday and other things which again, we never supported as an api, and we should make it clear that they are not. I don't know even why, where we are documenting them, since nobody can set them.

D

Up there is a distinction right between things that are right literally in development and things that are in release that you would do use in environment variables like we like. We need some graduation, even if we want to use a different set of terms which is delete, mine right.

R

Internal use, let's call it internal use. I mean there are a lot of command line, flags and environment because the api is mesh config I mean they get set from mesh config or from from values.tml, which is exposed to the user. But users does not need to know that something in mesh config gets translated into some environment variable or some command line parameter for internal use.

D

Okay, so it sounds like we're in agreement that we have to have basically the directory of environment variables right because we need that to produce the dock right.

R

D

Declare in the.

R

Indication that is not it's experimental and should not be used, except for exceptions.

D

Well, like internal things shouldn't be even on like presented okay, so they should be. They should not be generated doc, even if they are in the directory, and only the things that we would ever consider exposing to a user would go on that list.

M

That's a good point, that's a great point. We don't have a way to separate them, so it's just automatically generated. It's too.

R

Much for a user how about how about we put uh an internal right now we discuss only experimental alpha, but let's put an internal and both environment variables, even annotations and- and uh you know, definitely command line flags and other things that are not supposed to be used by user they're internal. So that's a completely different category.

I

You mean to hide it from the docs hide.

R

It from the dock or or yeah got.

I

It so louis the same. Does the I see that the command line args are right now documented, I feel like they shouldn't be even documented. I.

R

Agree I keep asking for each release. I I ask for them to be removed, but someone likes them.

I

Well, let's clarify right: is there a reason to keep them there.

P

So we we support today the operator overlays at install time which do have the ability to set from the operator or from istio install any of these environment variables or command line flags. As I understand it,.

R

This is a better api or something I mean again. It's it's it's. I don't know who decided that so things became a become api in.

P

That would be the environment's working group constant.

R

I'm pretty sure we, I would never.

P

I

To have environments.

R

To have uh common line flags.

I

Let's, let's not dwell on what was how how we ended up here, let's think about how what you want to do going forward right. So is low level command line arguments helpful for end users.

I

Are they actually useful right? That's the question. We need to ask.

R

There is no supported way to set them and uh they are not tested, they don't fit any criteria for.

P

So so the operator overlay is definitely supported, constant, it's it's.

R

Show me a test show me a test show me a test that is changing any environment variable in the operator overlay, and this is it's checking that it works.

P

Where it's under the pilot integration test, it's analysis, test.gov.

F

Not a commandment argument, but I think I mean the command that arguments are very small in number like they're, almost all about logging, basically yeah and there's no other way to configure them. So.

R

No! No for for uh history, it's true to environmental personnel! Sorry for.

R

F

Oh yeah, no one directly. Yes, that's.

I

What that's yeah yeah, I agree with costing here. So we have alternatives match here where either we masquerade it as a mesh, config api or some other high level api, where eventually they get distilled down as command line arcs, because if you're using a operator or s2ctl to manage your sq installation, if you manipulate them directly, they will get lost anyways right.

R

And really want to support helm, which is doesn't really support, editing the command line, arguments and and customize which doesn't support it, and there are kind of other environments where it's not. I mean the operator is kind of the the one off where we added a feature just for the sake of.

P

I 100 agree with costin on where we should get to in the project. We should not be asking users to modify these I'm a little bit concerned about disabling documentation, because I do think the users are in fact modifying them today.

R

I mean uh that's that's worrisome if they do, because it's probably going to break.

D

Can we try and get some signal on that? Sorry, I'm trying to keep up with the notes, because this is this- is a lot.

J

R

We I mean, uh can we find out what operator configs people are using or or.

P

So the ux working group started a conversation with the steering committee back in july about automatically collecting some data about istio installations in an opt-out fashion. uh I think the conversation just died silent, so I can try to revive that.

R

How about we deprecate the the things that allows to set and see if anyone complains, because we need to support, help, help definitely doesn't have this capability.

R

I don't want to have disparity between helm and and operator, because there is no value for users to be confused, and so simpler is better.

P

Martin is the expert on this area as far as I'm aware, but I can say for sure that cost and we at google are using the overlay extensively for our products.

O

Probably you can fix it.

I

I think we are talking about making this like an internal api.

C

I

Like there are some internal environment variables and start the process of saying it in our docs that will be deprecating it yep and going.

O

I

And here is how you migrate and use a higher level api right. I.

D

Mean if vendors want to use an internal thing, yeah that's responsible and they have to keep up with release right exactly right.

R

I know for a fact that one of the few of the recent cleanups removed a bunch of command line. I mean trust domain and- and you know some of them quite critical- they were removed. So if someone is setting trust domain as a command line flag, probably they are not getting what they expect and that's fine. I mean it was reviewed as a code change and as a cleanup and and nobody treat it as an api, because it's.

D

So do I, I guess a question here right so do we want to do the canonicalization process for args? Just like we said we wanted to do it for annotations labels and api fields.

D

um If so, we need an owner uh and then we need to work. We need to review them now. Maybe we could just do that right without doing all the other machinery things that we've started to do for things that we do want to support api long term.

R

So uh louis for for us, my my goal was to completely get rid of them and use environment variables everywhere, because uh the complexity of operator which cannot move up to to have another install mechanism, was due to uh the command line, parameter. The fact that you cannot, you know, use customize, for example, to change query environments.

D

You want to say that, like args are simply uh education of environment variables and environment variables is the canonical api for args.

R

And then are gone from history and everything else, so everything is environmentalizing. There is a you know, there is uh precedence for for using a verifiable consistently for configurations for internal configurations and we have the api.

Q

So for the discussion here, could we just like assign an owner to that topic and and to drive that discussion.

D

So mitch you picked up environment variables. Nate, you picked up annotations, uh who wants to pick up args.

R

uh If it's about getting rid of them, I can pick them up. I mean.

D

Well, the first thing: let's get the list of all of them and get them reviewed right.

R

The list is this: on this ui I mean we do have it.

P

Already, I'm happy to add the infrastructure for, for us to track status level of command line. Args it'd be pretty trivial to do um as far as defining which ones have which support levels. That sort of thing uh that that should not be me.

R

No, no, the proposal is to just you know, get rid of them uh completely: deprecate all of them and and and using environment variables, and uh one way to to configure.

D

Two points right, that's the second part. The first part is to do what mitch said right. So if mitch is willing to do the basically put together a pr which the working group leads can review, which is here's a list of all the command line. Args that looks a bit like the thing that nathan had mitch's job is not to assess what the support level is he's just providing a framework for people to do that review. That would be super helpful uh and then we can do the next thing right.

D

So there are two parts there um so mitch you don't mind doing that. The first part yeah all right so.

Q

One one last tiny topic related to all this. uh Looking at our our uh uh io page for feature status, we don't we only list the three we list alpha beta and stable. We don't list a pre-alpha, so it'd be good to get some sort of get to get that updated to include that whatever we want to call pre-alpha. Is it experimental.

R

We use the experimental is what we used in the past.

Q

That's fine, but we should just you know, uh standardize that.

D

Yeah we should- I I put a note in suggested terminology, feel free to comment there.

Q

Q

But that's not exactly feature says, though, is it right?

Q

uh It's not, but we should.

D

And they're kind of the same right like we don't doc, we shouldn't be documenting. You know even api fields, right that are in that internal or development state. So it's not clear that we need to have divergent terminology.

Q

Okay, so pre-alpha would or experimental would not actually show up on our docks anywhere. They might actually be in the repository, but they'd be marked hidden.

O

D

Yeah so currently used hide from docs is the way of doing this right, which isn't great um because we're not like that mechanism doesn't let us convey status right to people right well, then, it's there or not there, um so the last one is who wants to pick this up for apis.

I

I do think louis that for experimental apis, even if we hide them, we can have tasks and guides which say hey. This is how you use an experimental thing, and we can clearly indicate that this feature is in development right.

D

Yeah but we don't want to put it in reference material right, yeah,.

H

R

Could we could you have a setting on the site or or on a separate site or or you know, put them on wiki or some other place, so so the documentation and easter is clean, and it's only you know the things that are supported and better and stable. And then, if you want to do experimental or internal stuff, you go to the developer site or or some other.

D

Well, I mean like even that right I mean today the the analyzers don't have a canonical form of knowing right the teacher status of api use, so they have to go and codify this by other means.

R

Yeah, that's that's clear, but.

D

Like this would be hugely beneficial right, having that that mechanism be standardized and actually be usable within a framework right, because we literally even analyze right, which tells you you're using an unsupported feature.

P

Well and more than unsupported we'd like to build an analyzer that can give you an idea of your support that you're leveraging so you're saying here's all the alpha features you're using here's all the beta features you're using we, we pivot our support and our guarantees a lot on these feature statuses, but as a user understanding, what your actual exposure to them is pretty much impossible.

P

The surface is just too wide.

R

Just use only better apis and don't look at anything.

O

D

I mean that's, you know some of you would like to be in that state and they would like to know if they were in that state um right. I mean this would probably help us with some of our upgrade pain, and it would also let us make an assessment of like do. We have too much api surface. Where do we start shaving right, and how do we shave?

D

Because you can go to a customer and say, run this tool right and let us know right the stat the feature status for every feature that you're using that's really important: ux feedback when we plan deprecations.

D

D

There's a huge ux benefit to being able to work this into the tool chain, thanks, doug doug. Can you capture that, in the note, by the way, um so coming back to the earlier topic, uh we need an owner for apis and that's probably that's probably a bigger work item than the other three.

D

Because it involves going through.

D

Well, it's going to involve forcing people to go through all the protos and annotate every field and ever annotate. Every enum value.

M

And also check if they, which state state they should be in.

D

Yeah, no, that's like. I don't expect someone who creates the framework to do this. I expect them to put the thumb screws to the working group leads to get this done.

P

We also need to differentiate between the api name in kubernetes, which often is v one alpha, one or v, one beta one and the support of the individual proto fields that can be confusing for our users.

P

We are already there. Oh, I know, I'm simply saying we need to make a clear statement on it, not that I, I don't think it's something that we can fix. We just need to make sure it's intelligible to our users.

P

I

I think louis working group leads owning this, because this, I think one person owning will be very difficult.

D

I'm just talking about one person owning getting the framework working.

P

Yeah, I have someone in mind who's taken a swing at this before, but I haven't asked their permission yet so louie I can follow up if, if we're not getting an owner here,.

D

Okay, so I'll take an action item with you mitch.

D

And and maybe if there's some other toc member.

D

I

I I can help.

I

I've I've tried I've been doing it internally for us.

J

Q

Guys to circle back to my earlier question, so uh my question was: do I have approval from the toc to go back to the feature status page and to add an experimental column.

M

Jacob was asking what terminology we should use. It would be using experimental, pre-alpha and development which they're the same thing.

I

Right, I I reluctantly say this that yeah you should clarify because we release like like yeah. We are in a state where we have experimental features and they are not explained well, so your clarification will help, but I want to go to a state where we don't have those we straight away. Go to alpha. I have not seen a lot of other mature infrastructure products have a pre-alpha stage. To be honest,.

Q

It's still useful, even if we don't document them like to actually like if we, if we store like for look for annotations, for example, if we, if we create a new annotation, but it's experimental, we put it in the api repository, we mark it hidden from docs, but it still has like in the in the go code that its feature status is experimental or something like that. I think that's still a useful signal.

R

You know someone used to say more wood, fewer arrows, I mean let's focus on the beta and and stable features and and uh maybe alpha features and- and uh you know six months from now- we can worry about the internal ones. If we, if we find the need.

D

So jake I mean how to do the the kind of ui treatment for this uh jacob. Did you volunteer to help figure out how we might present this information right there? There are ways of making experimental features be kind of visually below the fold, so we don't encourage people to use them.

D

Think about that.

S

uh Yeah, um actually, no, what I was suggesting is we have things like tracing or whatever that are marked experimental, and then we have other pages that are marked pre-alpha, and so uh I think, to end users. uh It's just not clear to them like how how's experimental different from pre-alpha. What does that even mean? um So I can. I can help with that initiative um and as far as helping uh below the fold, I mean I'm also willing to to help people uh kind of coordinate that that effort as well.

D

Yeah, I think that would be really helpful because I think that's a it's an important part of this, but maybe somewhat orthogonal to the the more frameworky aspects of this work um who wants to take ownership of the terminology?

D

Do you want the toc? Should the toc just go and do that? I think.

Q

That seems like a toc thing to.

D

Me yeah, okay, uh who else would like to be involved so jacob you volunteered. uh I've heard um opinions from mitch, nate and constant on this call about this, so I'm going to consider them de facto involved. Anyone else.

I

Is this selection of the various terms, or is it broader than that.

I

We are talking about four terms here right. Is that what you're discussing or is this even broader.

D

I think I think we're talking about four essential terms: okay, and how they're described to users.

Q

Okay, that's fair and and we're maintaining the existing terms, or are we thinking about revamping the entire way we refer to these things like data?

Q

um I would rather not turn users, I'm just concerned about, like uh you had earlier, you referenced me to what you had written there with internal exposed and standard. I wasn't sure if you were suggesting replacing what we have documented today with alpha beta stable with that.

C

D

Notes here actually.

D

Which is follow apis.

O

Document better: how about that.

Q

D

Me- and there is the third thing- that there are from some things that are not supposed to be user facing there are runtime, only uh tls, state or mode or whatever. It is right, which was used by uh automatic tls is never a user-facing feature, but it is an annotation.

D

So we need a a nomenclature for that right. Some things that are runtime online.

I

Yeah they're like the status fields of crds right you're, talking about equivalency they're for annotations.

D

Whatever we, I don't think we have runtime environment variables, we try really hard not to have a runtime api, except for status, which is meta anyway. um Annotations is the exception here. I think, but are there other.

I

Runtime environment.

H

I need to drop. Can you guys uh stop the recording when you're done.

T

I

Right louis, we do have runtime environment.

J

R

One tiny comment: I don't know: if people are, I mean we had a discussion about uh uh having got environmental is normally our required start and we had a discussion about having a sexual mesh config with the proxy config having the important variables and actually making them uh dynamic. It's for your info. It's something that we'll probably do. Okay,.

D

Yeah I mean we could obviously start adding other fields to indicate dynamism right, but this is still kind of a basic status thing.

R

Few of them would have to be static, but uh rest, probably most of them. We want them eventually to become dynamic.

D

Yeah sure I mean just like we like, if we're moving environment variables into mesh config right.

D

D

We are out of time, we've covered a lot of ground, but I think we covered some pretty important topics.

R

One more thing: risk level I mean, for example, switching to strict mode or something like that is super risky. We should probably also have a kind of a dial of how risky it is to change something. No seriously, I'm not joking. This time.

D

No no, like I I I take your point. Like I mean we, we already added deprecated by hidden from docks right. We can add other dimensions, dangerous.

D

The description.

Q

Field for a lot of these things too, so I mean you: could you could express that pretty easily there.

D

Well, I think if we, if we want tooling yeah like analyzers, they need like some kind of standardized definition. uh The mitch I'm gonna, put uh something an ai for you, an ad think through how tooling and ux would benefit from this and maybe come back with some proposal. I don't know.

Q

We'll do it and let that drive the the fields that we need to add.

I

Yeah yeah, so so louie. I think what uh nathan you have a pr out. Let's make sure we just discuss the runtime field nomenclature there, so we don't have to have another document for it and maybe what jacob's pr will do when he adds it? Toc can jump on that pr to figure out how we want to convey the messaging for uh pre-alpha, alpha, beta and stable.

I

T

Yeah, that sounds good.

M

Yeah sounds good okay, so before I let everybody go, I want to say you know we are. uh We are trying to promote external control plan to alpha for 1.8, um so we already have. One group of group lead approval from environment work. Google also I'm seeking dlc approval.

M

For that. Please take a look and mark your approval in the issue.

T

While we're doing plugs, I received a grand total of zero comments on the telemetry api proposal in the last week. So um if others would like to comment, the weeks are going by.

J

T

The goal was to get that finalized design wise for one eight.

M

I

Hey lynn, I'm just curious so that promotion document uh approve the document has been. You all have been working for a while, and now you just want the final lord, or do we have to go and look at all the underneath pieces for promotion. Like is all the groundwork done and yeah.

U

I

Step pretty much yeah.

M

You know how sorrow frank and nate does review, so it's like 100 comments to get a documentation, page added.

Q

M

Well, thank you for all the help.

I

Okay, awesome: I will I'll take a look at your uh plan. Then thanks.

M

Yeah, thank you.

M

Q

But by the way, lin don't feel bad. uh I don't know if you saw what I went through with the multi-cluster docks, so.

M

M

Q

That was yours at least time for.

I

T

I

Covered a lot of good ground, but I also need to drop so.

D

Bye we're we basically done here louis uh yeah, I'm just trying to keep up with notes, guys so yeah feel free to drop off. uh There was there's a lot to.

I

You, like the scribe, rule no. I.

J

D

Okay, uh I will stop the recording.

E

Okay, thanks folks.

M