Cloud Native Computing Foundation TAG Observability, 6 Jul 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: 2021-07-06 CNCF TAG Observability Meeting

Description

2021-07-06 CNCF TAG Observability Meeting

A

A

It's a little unclear. I know.

B

That richard said he won't be able to come, but I thought that matt will show up.

C

Yeah, he might be. I know he has some stuff going on as well and bartok mentioned he's going to be about 15 minutes late.

A

A

C

I guess, as people kind of join in uh make sure you add your names to the google would be agenda and I'm sure we can give it a few minutes for people to join in. But then maybe if there are new faces that people want to introduce themselves, it might be a good place to start. While we wait to see I'm if.

A

B

So is there anyone on the call who is here the first time and wants to introduce themselves.

D

Yes, mapia hello, everyone, so my name is pierre work, for which is a diy company, headquartered in france, uh number three in the iwa market in europe in the world and number one in europe, um and I have a group of folks working in this company in charge of observability.

B

Is there anything in particular that you're interested in telemetry or you know how to consume signals or any any use case or anything.

D

uh We are quite a large company, so all the topics are interested in interesting for us and um I'm tagging along to see what you guys are doing and see how we might contribute.

A

D

A

A

Awesome anyone.

A

A

Well, let's have a look.

B

If there is anything on the agenda.

C

Yeah, obviously, if folks have something to add, please put it on the agenda below the the names uh matt did reach out to me about an hour before the meeting uh and asked uh to cover the open telemetry uh adding to kubernetes the api server.

C

So I'm happy to spend a few minutes on that, but I mean do people have other topics that they'd like to cover today.

E

I also I didn't have it to agenda yet, but uh white paper is something that I wanted to discuss today, but I really really depend on on bartek and he's.

B

Gonna be like 15.

B

yeah, then, let's maybe start off with steve and and put it on the agenda that we don't forget it. Maybe something else comes up, um but yeah you're right party should be here, for that.

B

A

Have anything else.

B

Otherwise, I think steve makes sense if you go.

C

I'm happy to kick us off. Why not uh so? First, apologies! I'm in a coffee shop so might be a little bit noisy here. But hopefully you can hear me all right, uh so basically matt reached out and uh was interested in some information.

C

If people might have heard that in kubernetes there was a recent full request that was merged, that added open symmetry support to the kubernetes api server, uh and so I thought it might be a good topic that you might be interested in to kind of learn more, and so I figured I'd cover it at a pretty high level.

C

So basically, kubernetes has a bunch of different things that generate or could generate tracing information, one of them being the api server sub being another one uh as an example, uh and there was some amount of pre-stream support in there uh vaguely. I can show that in just a second, but very recently, uh open telemetry was natively added to the api server with a go instrumentation.

C

So basically, they wrapped the http calls inbound and outbound, including context, propagation and leverage the otlp exporter, which is uk symmetry project, as well as the ability to kind of integrate to an endpoint, typically being an open symmetry collector. That's what typically consumes otlp traffic today uh so I'll share my screen, real, quick and I can show folks in case you haven't seen it so. Here's the kubernetes pull request 94942.

C

If you're interested I can put links in the in the chat afterwards, um I they do a very good job of listing out the actual enhancement requests. You can read about that. What the implementation details are, which has basically full documentation on this change.

C

I'll, probably pull up the enhancement request, because it's kind of uh interesting, so they have basically all the details of why? How considerations next steps? What have you kind of listed in here, but it also links to the kind of prior art and open symmetry? So kubernetes had this util called trace, uh which is kind of a form of tracing, but to be fair, it was kind of limited in what it could do.

C

It basically only reported the latency of operations in the kubernetes api server and it had the ability to log some information if like limits were exceeded, uh and they give you kind of usage information about that. Well, this is nice if you can add native distributed tracing into kubernetes, and you can pass that context so calls coming into the kubernetes api server as well as call something out of the api server you can actually show uh you can do some pretty powerful things, including potentially troubleshooting problems uh leveraging kubernetes like.

C

Is it a kubernetes problem or a code problem? On the other end, uh this really wasn't as trivial before uh so the idea was hey if we can add distributed tracing open. Symmetry is a cloud-native uh implementation that is kind of picking up steam right now. Maybe we can offer a generic way that this could be send data to basically the back end of your choice, so they went ahead and implemented an initial version of this just around the api server, uh the ncd and and other aspects of api are not covered.

C

Yet, though there are, I believe, other did. I actually keep it open. There were other uh tickets listing. I might have closed it. So apologies if I did yeah here, we go add distributions. The scd client is an open issue uh and that pr is already open for this. So I think they're going to look to expand beyond, but the initial implementation of the api server has been merged and is a feature candidate for the 1.22 release.

C

uh So if you actually take a look at the uh sorry to keep it open, I have too many tabs open should have done this beforehand. Apologies here we go uh feature gates for 1.22 you'll, see that the api server tracing uh is a candidate that will be in alpha for that release.

C

So this would really give people the opportunity to start taking advantage of turning it on so you kind of have to opt in yourself. You do that by basically passing a configuration parameter. I think I have an example of that.

C

Maybe I don't uh here we go. You basically pass this open symmetry config file parameter, and you can see what the uh config struct looks for that. uh So basically, you're passing in the end point that you want to send it to maybe forward information, maybe service information, uh and uh if you pass this configuration parameter, it will basically enable this alpha feature which gives you the ability to make and receive calls and emit spams from the api server itself.

C

So it's a great starting place and getting better observability natively into your kubernetes cluster. This, of course enhances or enriches the other telemetry data, including metrics and logs, that are already there today and as mentioned, this is kind of just the starting place or other hopes that could be added, uh and there are a bunch of things that are out of scope for this initial implementation. So I definitely take uh would recommend people look at this entire enhancement proposal because they kind of list other areas that this could go as well.

C

Some of this does require manual instrumentation if you want to go really deep into kubernetes, but even having api server access is a huge step forward into being able to solve some more complicated availability and performance problems with the interactions that happen in kubernetes.

C

I guess with that I'll pause questions, comments from folks.

B

Yeah this is super exciting. I'm I'm super happy when I saw that tweet first and then you know this. This definitely opens up a whole lot of possibilities. The the number one thing because I didn't really have time yet for for a deep dive. I was wondering the obvious question: does it extend to custom resources.

C

Yeah, I don't know the answer to that. I haven't drilled into the specifics and actually I haven't had the opportunity to run with this. Yet I believe bogdan has tested end to end, um so I can ping him, maybe on the cncf slack and see if he can comment, uh but that is a good question.

B

I I would actually so thank you and I would actually suggest a step further.

B

Maybe we can get a cncf webinar where maybe yourself or hotel, and don't remember the name from the the I think it was a googler who sent in the pr um doing a joined webinar on that topic and then it in action. I found these yeah. It's always very useful.

C

I agree, cnn would be really cool. That's a nice suggestion yeah. I can definitely sing some folks and see if we can get that scheduled.

A

B

You matthias david, yes,.

E

Yes, really really exciting to see that on api server, uh I was wondering if there is uh enhancement, proposals to add uh tracing to other components of kubernetes, such as cubelet or controller.

A

C

Yeah, I think the enhancement request covers some of this sorry, I don't have my tabs straight, which definitely doesn't help here here we go. Does this list.

A

Next steps somewhere.

C

They call this out. Maybe non-goals is one thing uh so here, for example, are some things that are not in scope. Currently, that could, of course, be enhanced to support this as well.

C

So I mentioned the manual instrumentation parts since all kubernetes resource types in generic manner. You can't do that today.

C

Clearly correlating with other signals would be extremely useful, uh but that's outside the scope of the initial implementation here and even the controllers themselves, the custom resource, I'm assuming it's actually not part of unit, just based on how I'm reading this um but yeah- I I don't actually know the full roadmap of this, so I didn't actually come prepared.

C

I was asked kind of last minute, but I'm happy to go drill into this and get some more information and pull in relevant people, so we can kind of figure out what are the next steps here or if people have suggestions or comments like feel free to add them to the tag observability, google doc and I can share them out with the author and other people that are kind of working in this space. uh Maybe we can have another deep dive on this. In a previous tag of durability meeting, if people are interested, thank you.

A

Thanks any other questions.

C

Cool, I think that's it for me, so hopefully there'll be other kind of announcements going forward. I know like envoy added logging support recently. uh Hopefully we'll see tracing be added here in the future, um and hopefully things like istio and m3d will have more native open cemeteries for it. So we're seeing a pretty good pickup here, but the kubernetes aspect gives you a new level of disability. Typically, you don't get that level of like platform or infrastructure type correlation. uh So this is a pretty exciting change over overall. So thanks folks.

E

All right next would be white paper, but well. What I wanted to discuss about the white paper is uh how to we are.

F

We are getting some feedback.

E

uh From folks uh on pr's like twitter, now I was I wanted to discuss how to better address the feedback, because people are opening the pr against against a tag observability repo, but but the white paper is on another pr. So there is going to be some virtual fix if we.

B

I think the technically the best way would be to use github discussion, so we can create a new discussion.

B

We should have it enabled on our tech, repo right, because then we have it in the central place, and you know because I I saw that you know questions also on slack and it's really hard. You know to receive feedback from 15 different channels and including carrier pigeons and whatnot to make sure that nothing drops. So if we do have discussions, I would strongly encourage to use github discussions to just you know, have a feedback thread or whatever there and then point everyone there.

B

A

Know if it's restrictive.

B

If someone needs to have a github account in order to participate, if that's the case, then that might be challenging, although I believe most people do have, but it would exclude people who, for whatever reasons, don't want to or can't have a github account to.

E

Verify that uh I I don't, I don't see discussions on tag observability.

A

Okay, then, we need.

B

To check why that is the case, it's likely that you just need to enable that on a repo basis, it could be that it is a cncf thing. um We need to.

A

You have access.

B

I don't think that I I can change anything there, but you can give it a try.

G

Because they need to enable it so we have this for other reports. It's not yeah yeah.

B

um Let's uh give the chairs an action item, I always love giving action items to people who are not there. They can't complain um to you know, have a look into and to get up discussion, enablement.

B

Unless anyone has a better suggestion how we could channel all the.

A

A

But going with discussion would also make inline commenting a bit hard like. I was the one who made some review comments on slack and yes,.

F

A

Problem there was that it was just a list and for with discussion we would have the same, so it would be just a list, but we couldn't really uh address specific sections of the text right.

B

That's why I'm asking if, uh if anyone has a better way to address that, I am super aware of that. You know you want to have a context, um but I guess my main question is: is there a better an alternative for you know getting the context and having in the central place, because otherwise you know it's a little bit challenging to make sure all the especially if you have um you know contradicting um feedback or whatever right someone says like. Oh, you know this.

B

This section totally should be moved somewhere else and someone else says. Oh no, you know.

A

I don't know I have not found an.

B

A

B

E

Yeah, it's hard to settle in a solution without the three chairs here.

A

B

Yeah, that's that's at least yeah. You have to have captured it anyways and.

A

B

A

E

Yes, we, they also kind of hush uh getting the white paper ready like in the last three weeks, because we had the chance to publish it on the new stack uh blog post. I think I wanted to check with bartek how is the status and this because I think the due date is like this week, but at most the next one.

B

My understanding was that we simply mention it and point to you know here is the pr or are you talking about a more polished, ready to consume version or.

E

I don't know honestly, I don't know what they what they need. So that's what I wanted to check.

B

What did we tell folks how long we ask for feedback two weeks or.

E

B

E

Tell we are open for feedback, we didn't uh oh perfect. Yesterday we didn't put any any uh date.

E

A

E

Have you opened the meeting notes yet.

E

uh We had two points regarding the the white paper. uh One is it's it's really hard to address uh all feedback, because we have different a lot of different channels like we have. I I've been receiving uh feedback on twitter slack uh github prs, if not.

E

Wondering if it could enable github distinctions on the ripple and.

F

Just gather everything that sure we can do that. um I wonder if it is worth to yeah, just just really ask people to leave this feedback on on on the pr really so the place we envisioned and if we can do that for them. So I even asked this person uh yeah. Can I copy literally their feedback into the pr?

F

I don't know if that's legal or like nice, so I'm just asking first, but you know, maybe we can do that and if you just copy yourself uh after the approval, then maybe people see the kind of advantage of having this in the pr. My worry with discussion is like yeah.

F

I mean sure we can do that for the later step, but like right now we envision this this this piece to be um a place for the discussion, because you see there might be like repeated discussion and maybe that if, if someone can see the past discussions, there are higher chances, they don't repeat, but they may be joined as you know, kind of the same flow. I don't know that would be my fault, but I can I can enable discussions anyway. uh Let's try that.

E

On second thought, actually I think what's gonna happen is we're. Gonna have uh slack twitter prs and discussions so we're just adding a new channel instead of consolidating.

F

Jesus arthur you're, so smart, so.

A

F

And so smart already.

E

A

That's an excellent.

B

E

A

B

No, no I'm just like that's an excellent point. uh So what can we do to reduce the number of channels switch off slack and shut down our dm? No, but seriously I mean we should at least communicate the preferred method right if someone doesn't want to. I cannot whatever a comment directly like did. We communicate so far a preferred way to comment.

E

Arthur, uh I think, biotech suggestions is what what I've been doing so far I'll feedback. I'm copying the feedback to the pr myself and trying to answer that.

F

Yeah and let's, let's do it consistently and also just mention that you copy it right to the to the past thread like twitter and, like say hey, uh I will be answering your question there. So just join me if you want right uh yeah, it's hard like people just prefer things but again like we need to have consistent place for you know.

A

F

Those those discussions.

F

And um anyway, I cannot enable discussions or maybe that's a good thing. I don't think.

B

Would enable I I would nevertheless independent of that I agree with with having too many channels regarding the feedback, but I would think that having discussions on our tech, repo would actually be a good thing because there might be this kind of, like you know, faq style, whatever things that might come up that uh you know, an issue is not really the right thing and we don't really own code in that sense. But discussions really have a nice way to. You know, raise issues and we can discuss things there, so.

B

Nevertheless, I would be supportive of enabling discussions for our repo, but not using it. For the white paper feedback.

F

Okay, I'm adding issue to enable discussions, because matt and rich can do it.

E

Anything else yes and again about white paper. I wanted to ask: what is this status about the the news type, what they want from us? What can we do to help them.

E

Do you know anything, are they.

F

Feedback about what sorry.

E

uh So we we move forward on the last weeks faster than usual, because we wanted to to make the white paper ready for the new stack.

F

Yeah, I think I have you seen that article. I never seen this being released, so I will catch up with the status on uh on the article itself.

F

However, what was really encouraged was a blog post for the cncf around that. So I would I would you know, treat this new stack thing as the best effort and try to come up with, like blog posts around that um announcing even public comment and like getting more feedback really uh what they think.

E

Yep sounds good to me. I just wanted to know the.

A

E

F

Yeah, I don't think it was released, so I will catch up on the newspaper as well. Yeah.

F

A

F

For catching on those things and putting this in a state that is actually commentable and and clean to work on.

A

Do we have anything else.

F

uh I wanted to kind of discuss, but just you know I think uh I know there's this ongoing discussions about cows, engineering being part of observability or not um yeah,.

A

I wonder: what's the feeling.

F

About this group, like looks like there are certain um facts that, for example, like you know, litmus.

D

F

Is like house engineering uh kind of project? uh The incubation stage is not actually you know evaluated in the observability group, but rather in, I think, up delivery or something like that. So this kind of an indic I don't know who decided that and I'm fine with this. It just kind of indicates that you know uh yeah. It might be good to clear clarify this space right.

B

Which is which is strange because in the landscape cncf clearly positioned all the the chaos tools in in our corner, so um would really be good to get clarification like I. I would be supportive to have them in our space, but I I'm kind of like surprised to to learn that this is not in in our charter.

B

F

B

Maybe we need to clarify the charter, I don't know, but uh we definitely would want to have uh amy or whoever from cncf clarifying that.

G

Maybe it could provide some background from from the app delivery perspective because we had uh litmus as a project there and it extended now even more so. The plan is to have a white paper on chaos, engineering that was driven by the litmus project team. What, then, immediately, came up that? Obviously we had the net attack network there, because there's obviously also network chaos, and then we had security chaos there, which brought tech security into the mix, and the current activity is to have a dedicated working group on activities around chaos.

G

Engineering, because the way they make it into app delivery is because they do testing and a part of the delivery chain. From I think, security. One is obviously for security chaos, networking because there's networking case and obviously for analyzing, it observability makes sense.

G

So, whoever volunteers I'll bring you into the working group charter discussion if you're interested in, I think it's spanning a couple of teams- and we had this for other projects as well, where they're assigned to multiple tags that the proposal from the tlc was then at least have a working group that works across uh tax on those topics.

G

So who wants to get involved? Just hit me up on slack and I'll get involved in that working group discussion, because I think it does make sense to have observability there.

F

Yeah thanks for the context, this is uh super helpful and okay, so looks like there is a cows. Engineering is not left behind. Someone cares about that. So that's good. um The and yeah I'm writing down on doc that anyone who wants to be participating to that. uh Let aloys know on slack uh the the kind of question here is like this is still part of the landscape, observability landscape. So the questions like do we want to do you know of any plans? Do we care to change this to to bring this topic?

F

What do you think.

G

So how it makes it on the landscape is that somebody submitted it uh onto the land with dock, and you can only add, like one area where you want it to be, and whoever submitted it submitted it for a certain criteria. That's how these things usually end up, but um we also have our meeting tomorrow. So I can also reach out to uh the litmus folks and uh point them in your direction to at least present in tech observability and have a discussion over here.

G

So they are usually very friendly and and open to to discuss their topics and incubation, more or less is done for litmus right now uh from the former criteria.

B

But your theory as in this would be something you know a singular case that doesn't release, but it's not really supported by. When I look at the landscape because under chaos, engineering they're like three six, seven things listed, not just fitness, so.

B

Or maybe I'm misunderstanding what you're saying it's.

G

How they were submitting it? So if you submit it to the landscape- and you add it for say, observability- that's how it then ends up or for uh app delivery, that's how it's ended up there and historically, this litmus was also added before seek observability even existed.

B

Right so you're saying that the first chaos engineering related project submitted it to observability and all the others followed through with that or I'm not 100 yeah.

G

So what I'm, what I'm saying is, don't interpret too much into how it's in the landscape and let's have the projects working with the tags that make most sense. But.

B

You see that's what people out there see right, I mean cncf is heavily promoting this landscape and if we internally have a different way to deal with things than what we communicate to the outside world, then that causes confusion. I'm not! You know suggesting that we start a fight between tanks, a tag of war. So to say I'm saying: let's have it consistent and have it clearly communicated? Where does this belong?

B

I think it would be a good place here, but it's up to the project if they feel these better suited somewhere else also great, but let's make it consistent and clear where it belongs, so that if someone comes from the outside and it's not as deep as you always or myself or whoever uh they have a clear. Oh that's the tag that is responsible for it makes sense.

G

Yeah, it's fine, let's just cuss with amy.

A

G

Don't have any objections there so.

F

Okay, um sorry, I didn't hear who we should discuss this with.

G

I think it's yes starting a discussion with amy and then eventually chris, I think chris will have you know this one as well.

F

Yeah, sorry, it's like a minor need, uh but sometimes you know people care more about this uh landscape kind of categorization, some people less and actually we have discussion about whitepaper. It's like. Oh, it's part of the landscape. We should definitely think about cows engineering and looks like that's not not not the case right, so it just brings some small confusion, nothing else. Okay, I can. I can take this action item and kind of start the thread if, unless anyone wants to do that,.

E

This confusion uh regarding calls engineering on the white paper just didn't, have only like when getting feedback, but when writing as well. We were we spent like several months of discussion. If cloud engineering should be included or not.

F

All right, let's, let's clarify that, thank you guys for all those contexts.

F

A

F

It from my side any other comments, topics, ad hoc topics,.

B

um Quick reminder in case you haven't seen it. I think the prom con cfp is still open for another two weeks or so so, if you have anything metrics related or more consider submitting.

F

Good point: do we have a.

F

Link, we have okay.

F

Okay added the link as well on on our document.

F

Yeah yeah, that was easy.

F

um Cool, thank you very much then looks like we are good. Sorry for being late. I was driving and there were there was traffic yeah, sorry and see ya around. Thank you have.

B

A good one bye now.

B