Kubernetes SIG Instrumentation, 3 Sep 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: SIG Instrumentation 20200903

Description

SIG Instrumentation Sept 3rd 2020

A

And I believe the recording has started welcome everyone to the sig instrumentation meeting today. It is currently september 3rd 2020 and I believe we have one item on the agenda. Currently, I strongly encourage people to go ahead. Take a look at the agenda. uh Add items if you have things that are missing and uh mark your name down under attendees, and we have uh an item from chelsea about the new event api, ga graduation.

B

Yes, uh so I have a question so am I able to share the screen I want to share? I.

A

Think I might have to make you a co-host in order to do that.

B

A

A

uh Try now: okay, yeah, that's.

B

Great, so let me do.

B

B

Yeah, I can share the screen right now.

B

A

B

And be able to.

A

Increase the size a little bit: oh yeah.

B

uh How about this? Okay, so, okay, I will get started so uh hi everybody, my name is chelsea, so today I'm going to give a brief overview of the new event api project, and I will also give a summary of the work that has been done so far for this project.

B

So here is the cap for this project, so basically includes the original design dock yeah, so this project was initially proposed and designed by marik his github id is gmaric a few years ago. So this is a pretty old proposal.

B

So maybe so many people might not hear about this project before um let's go downstairs, go down um so yeah, so the motivation for this project uh is that um the old events are very spammy and with very unclear uh schematics um like so, which makes events not that useful and also uh there are some well-known performance problems caused by the old events. uh For example, events can all events can potentially overload the api server, especially when a cluster is in an unhealthy state.

B

So the goal of the new events api is to improve those issues. Okay, so now, let's look at the design yeah. So first there are some api changes. uh Here is the, as you can see here is the new event structure. um So there are some new fields added and also some duplicate, um but it's not very clear from this structure. um Let's go down and here is a table that shows the comparison between the old and the new event apis yeah from so from the table.

B

You can easily tell that there are some api changes so, for example, like the involve object has been renamed to regarding, and also there is a related new field added. So this is basically represents. The second object that is related to the events and we also have a new action field, which basically means what action was taken for this event and and then we also have a message view was renamed to notes and we, the source view, was duplicated, but instead um reporting controller and reporting instance.

B

Fields um are came from the original source view and they are added to the new api and we also deprecate the first time stem lost cancer and count views, but instead we have new event, type event, time field and also there is a new series view. So this view it gives information about whether it's a singleton or sequential event and let's go up to the structure. So uh the series field, it's basically a structure and it has count uh field and also the last observed time.

B

um This state view has already been depleted and removed uh for from 1.18 so yeah. So uh this is the overall changes in the api.

B

So I think the biggest change is actually how we duplicate the events and how we limit the number of api requests sent to the api server. So that's described in the following section.

B

So so there is a very important assumption that we make for the uh for identification logic uh in the new event api. So that is uh events with the same regarding action reason, um reporting controller reporting instance related tuples. They are considered as isomorphic events uh yeah. So this assumption is very important because it's how we define the isomorphic events and it also decides how we update the new event series in a green event recorder and the dedication logic in the event recorder is described in the next session.

B

um So how does that work so? First, when a new event is emitted for the first time, you'll be written to the api server and it's considered as a singleton event. And next when the isomorphic event is emitted, we think the threshold, which means the first event, hasn't been deleted from the cache. The event recorder will start recording and updating the series view for the events and then all the subsequent isomorphic events don't result in any api calls.

B

They only update the series view for the event in the event recorder and yeah so and then in every 30 minutes. The event recorder will create and like heartbeat call, is basically an update called to api server in order to theoretically update user on number of occurrences of the events, and you also prevent the uh garbage collection in fce yeah. So basically, this every 30 minute call is is how the event recorder update the api server about isomorphic events and other than that.

B

The event recorder uh also uh go through each entry in its cache every six minutes. um So this check the goal of this check is to see if there's any attempt to emit event in the last six minutes. If not, then the event recorder will send the last call to for the smooth event to the api server, and then it will delete the entry from the cache for this event.

B

So, let's see there is an example here, so yeah you can see when the first event occurs. I look like this and it doesn't have the series field with a set and if within the threshold, uh if there is a second like the same event as multi event happens, then um it will the event reporter will start recording the series view for a new event for this event and then um and for all the subsequent uh occurrences of this.

B

The same event, you will just update the series field in the cache, and then everything means it will send to the api server to update the events yeah so and yeah. That's the overall design of the new event api and how it's being used in the event recorder, yeah, okay, so and then let's go down um yeah. Now, that's let's look at the performance impact. So uh as for the performance impact of the new event api, we haven't done any formal testing for that. Yet.

B

But here's some estimation for the performance changes, uh so we will potentially emit more events for pop creation um and events will definitely be bigger because we are adding more views to the api to the structure and the good thing is that we will emit less events for hot loops, and this table shows the use memory usage in the performance test um in class of various sizes. And so the estimation is that, because we are adding more views to the new events, we saw the increase of the event.

B

Size probably will be bounded by like 30, and then we probably will emit more events for pro creation and that impact will can be bounded by 20. So the overall estimation is that the worst case in the increase in band size will be bounded by 56, so yeah. So we believe that this can be easily afford by the membrane. So yeah, that's the estimation of the performance and also as for backward compatibility from this table. You will see three types of event: structures.

B

um The first column is the internal event and it's basically the internal representation of the event structure in api server and then the second one is the quartile event. This is the all event types and then the new new event is event.event yeah. So you can see this all these three event structures. They have been updated to be backward compatible to each other. That means we can easily convert between any two of them yeah and yeah. That's the uh that's the overall uh like design for the for this pro for this project.

B

So after marik proposed this design and he implemented the new events uh api as we won beta one api and in 1.19 I have graduated the new event from we want beta1 to v1. So now it's ga graduation and other than that. I also implement a common fallback library to the event recorder so that it allows components to easily switch between the new and the old events. Api and, and currently only cube scheduler has been migrated to use the new event api.

B

So our next plan is to migrate all remaining components, to use the new event api and other than that. There's also one very important thing, which is we we need to implement a very.

B

We need to implement a load test for the new event api in order to make sure that the new event api doesn't really have any negative impact on the api server uh yeah. So that's the most important thing that in this project, so the detailed design was including this cap. So if you're interested in you can take a look yeah, um that's that's um yeah. So.

A

Thanks for the overview and the presentation, um I guess my question for you. uh That's really great. um Do you want to maybe go over, usually there's like an implementation history section in the cap.

A

You want to maybe just go over that quickly to provide some context, and uh let us know, like you know, to thank thanks for uh presenting this. uh You know what what are you hoping to get uh from the uh the sig? Are you looking for feedback? uh Are you looking for uh like assistance with implementation?

A

uh You know like what what.

B

uh I I think the purpose of today's presentation is just to give a uh like overview of this project, because this proposal was proposed very a few years ago. So um I heard that I was told by hand that many people that pretty new to this instrumentation community might not know this project because it's pretty old, so he wants me to be for like overview, because I was previously working on this yeah. So that's the purpose for today's presentation.

B

Yeah. So do you want me to go through the implementation history here.

A

Yeah, I think just briefly that would be helpful, because uh if I'm remembering correctly, this was like a super old cap that, like never kind of got, finished up uh and now uh I think this new version of it got merged in april, uh and so that would, I think, be helpful context for folks and uh thanks so much for coming to give this presentation today.

B

Okay, so yeah, so you can see like three years ago this cap was created here. This proposal was created here and then and then yeah in the same same year, the new api uh new event api group was was added to the api group and then yeah. So uh so and after that we can see.

B

uh Also some also the schedule has been migrated. So I believe this is uh like two years ago, sometime two years ago, yeah, I don't really remember yeah like yeah last year yeah. So it's so the proposal was pretty old, but the invitation hasn't been done so much and then after the schedule has been migrated.

B

um Some, like some people, also try to migrate the other components but because other components have two they are too complicated, so they, the events, are used in many places, so it's kind of hard to migrate, so those uh I believe those hasn't been finished yet um yeah so and then in 1.19.

B

I chose to pick up this project and then I updated cap and also graduate the api, a new api to to ga and yeah, and then we plan for the next step is to migrate the remaining components to use new events.

B

So that's the overall implementation, history and the future point for this project. Great.

A

Does anybody have any questions for chelsea.

C

So I have a question: is there a recommended usage for the related object? Because I.

D

C

The regarding just translates from the old events right uh and the related like uh there's only one related field.

B

uh Are you talking about the structure like the relative view.

B

Here sorry, what's your question again.

C

uh Okay, sorry, I I missed the comment so creation.

D

C

Related objects, so, let's say a pod emits an event that it's I don't know created the pc or something like that right is that the correct.

B

Understanding yeah yeah.

C

B

So it's just relatively just the secondary reality object for this event,.

B

Yeah so yeah, if you have any questions about the fields in the structure, you can definitely go to the api to check the. I believe the common thing for each view is enough to understand those views.

B

Any other questions.

A

It doesn't sound like anything uh if, if somebody, oh.

E

I I have a I have comments. uh It looks like there are several works work in progress, but not like have a detail planned for migrating all the components using this new api. Maybe we can update this cap to include to include a series of like p uh issues or feature requests uh for saying uh updating to cover all the components to use migrate, other components to use uh the new api and we in the sig instrumentation, for example. If I interesting, I can help with some of the implementations.

B

Sure yeah, like definitely I will. I will update the cap too yeah and I haven't updated the cab after I implement the graduation so yeah. I would definitely update that and include more future plan thanks.

D

I think usually, caps go with a related feature, enhancement issue, and this issue is usually used to track. The work by which is also really important for visibility by the release team, so like like using this issue. Release team can make sure that enhancement is properly tracked throughout the the cycle of the of the component strategy. So I think those tasks could be put into the here, and here a list of components would be useful, but at the end it should be tracked separately, because I think it's very clear.

D

There was a problem with updating a list of tasks that would be done on cap, at least from some of.

A

Sorry, america, I had a hard time understanding your audio is a little bit garbled.

F

So I think I I caught what he said. So we should use the feature issue that the cap is linked to to track the work items. It may be useful to have a list of components as part of the cap, but actually tracking and the completion of all the individual items should be done in the feature issue.

A

That sounds good to me. Yes, I know that.

D

A

Like changes almost every release cycle, it feels like sometimes so uh staying on top of what the latest best practices is always super helpful.

A

Anything else for this topic.

G

I'll say thanks for sharing this, this topic actually came up and within my work team uh this week, although it came from a different standpoint, someone was asking about cloud events and that standard, and it was pointed out that that the events api is, is kind of different, a different approach. To doing that, would there be any sort of possibility of wrapping cloud events on top of what comes out of event.

B

Api, I'm sorry, um can you say that again.

G

um Would it be possible to write like a wrapper layer on top of cloud events that would allow emitting taking what comes out of event, api and emitting it out as a cloud event standard um event.

B

I'm not sure if I can. I can't answer this question.

A

I think I know what uh what's being discussed there so there there's been like, I think, there's an issue in the kubernetes.

G

Yeah, it came up like last november.

A

Yeah that continues to kind of like come up and there's like a question as to whether or not kubernetes itself should offer event compatibility with cloud events, and typically the like general sentiment has been no. We should not do this, but uh because that's like sort of an implementation, specific detail, uh but uh I mean you feel free to like raise that. Like I mean if there's like you know, if you could propose a cap to make it happen uh or something like that, uh you know.

A

Certainly, I think people would be happy to consider that.

G

Yeah, I need to go back and reread the cap and the cloud events back again and see if there's any way to mesh them.

D

Yeah, I think the overall, like conclusion, was that we are not sure that the events in kubernetes are the same type of events that are in.

C

D

You know open source remember and the second one was the best way to start that like or like. This idea would be to implement a wrapper for translation layers like outside of the community and look if it gets adoption. So it should be pretty simple like to just like write it in a couple hundred module that will read events from kubernetes and publish them as cloud events to prove that there is an interest in that feature uh like if it gets adoption. It should be pretty proof that we should change our approach so component approaches.

D

Events are for debugging and to give information within the cluster about debugging. They are not actionable by by any automation like like. This is current approach technically or this is how it was documented in original proposals for events.

A

Keeping an eye on time, we've got five minutes left. I wanted to quickly follow up uh on the discussion from last week about tests or sorry not last week, but two weeks ago, uh what's going on with the end-to-end tests- and I don't know if anybody has any other business, but if.

H

A

uh Feel free to add it to the agenda. I did actually also have one other item from sig p or they're, not sig, pm anymore, um but the uh the folks who are helping with triage, but I think that that will probably fall over to our triage session next week.

A

So um just wanted to ask folks we had like one action uh coming out of last week uh with respect to the stackdriver tests uh and we also had another uh like we went over a few different issues in terms of instrumentation and test flaking or not passing. Do we have any updates from folks on that stuff.

A

And that might be mostly directed at david and.

F

So han actually pointed out that the.

F

The test that I was looking at had one flake, but it actually looks like across the entire kubernetes test grid. The flake has reoccurred a couple times. I still doubt that it's related to instrumentation. It looks like there's actually just pods that aren't coming up as part of the test, but yeah it's it's still on my plate and I haven't gotten the chance to look into it again. Yet.

A

D

Yeah similar case um like somewhere yeah, I need to look into that. I I can. I think you mentioned that who was interested, so I can also sync with him but yeah.

D

I should be have some updates next week in next two weeks.

E

Yeah yeah I'll have something with merrick. uh I'm sorry I gotta sign like one or two bucks, but I just didn't have time for in the past week for looking at it. Yet I will make some time uh so so I can have some updates in in the next meeting.

A

Great so I'm trying to take notes and uh listed at the same time and I'm not succeeding uh any other updates on that.

A

Topic sounds like no. uh Do we have any other business? We only have three minutes. I'm gonna defer this uh feature tracking thing to the triage meeting or or the next sig meeting.

A

Okay sounds like no new topics with the time that we have left uh I'd like to. Let folks know that uh we have in like two minutes: there's a 119 release retrospective happening. That say, release is running. uh So if you are interested in attending that, I uh strongly recommend that you attend. uh I am planning on going at least for the first 30 minutes, uh and uh do we have any new uh faces on the sig meeting that would like to introduce themselves in the last couple of minutes. We have left.

H

um Hi, uh my name is keiko, um I'm from microsoft and currently I'm working on aks. So um the monitoring piece is very important and uh it's good to actually be here.

A

Oh hey caico, I was like wow, that's a familiar voice. It's wonderful to have you here.

H

Yeah it's great and by the way, I'm not doing monitoring anymore, but again this is a very important area for us. So that's why I started attending some of the sigs now.

A

Awesome welcome.

F

A

F

Always will go hi guys, uh I'm andrew um I'm from axioms and diaper labs. um I've actually been to signature training meetings a bunch of times before, but I've been absent for a number of months, so I'm back um and yeah. I don't I'm not contributing to the project, but I'm building basically um hosted kubernetes systems that all of our subsidiary companies use. So I spend a lot of time dealing with kubernetes monitoring, so I drop by when I can.

A

It's great to have you back.

A

Anyone else on the call want to say.

A

Hi um sounds like no uh and we're at the top of the hour, so I think that's a wrap thanks everybody for attending this week and see you in two weeks cheers everyone.

F