Cloud Native Computing Foundation TAG Observability, 7 Jun 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: 2022-06-07 CNCF TAG Observability Meeting

Description

* KubeCon EU Meeting Recap (Bartek)
* Adding profiling as a signal → OpenTelemetry

Notes for meetings are linked from our github repo (

How to join: cncf.io/calendar, https://tockify.com/cncf.public.events/detail/388/1649174400000
More info: https://github.com/cncf/tag-observability

#cncf #observability #opentelemetry #profiling #cloudnative

A

A

B

Yo, what up everybody.

C

Hey ryan how's it going.

D

C

C

B

Is everybody today.

C

Good good, how about you.

B

Doing well kind of back from the dead from between between kubecon and just like family illness as four teenagers get to the end of their school year and talk.

C

We should have met up, I don't know no, no! No. I wasn't there.

B

um I had planned on going, but but couldn't um do this in various life stuff, but um but no I mean a lot of people were off off the grid so to speak and occupied. So.

E

Yeah, I'm still I've been hopping around a bunch of time zones myself since then, because I was at kubecon and I was in california and in dallas and now I'm on the east coast and so or I guess, eastern time zone so I've. uh I have no idea what day or time it is.

B

So I'll put a link to the um the agenda and the meeting notes and slack here please do sign in there's really there's really two big things on the agenda for today, uh one hey hi henry uh one is um one is a brief recap by bartek on uh the meeting that did happen in spain for those that weren't there uh it wasn't recorded.

B

I believe we do have notes, but at least this way we'll have something for posterity about what transpired around the observates project and some other stuff um and and some takeaways uh and then uh the larger uh issue of the day. uh Adding profiling to open telemetry as a signal type uh ryan will drive that uh and we have alolita joining us at the half an hour at the half hour. So it's going to front load the other stuff.

B

um Lastly, um I can put a link in in the meeting notes, but the toc meeting, the first tuesday toc meeting, was just the last hour and we as usual, I just covered what we've been up to uh briefly.

B

So with that, um does anyone want to add things to the agenda before we start? I also need to say I almost forgot. uh This is a cncf um meeting. The code of conduct, uh of course, does apply. um Does anyone want to add anything to the agenda? That's not not. There.

B

Okay, henrik, I love your new episode by the way.

F

F

And I'm working on a summary for the week of kucon, so I was in the middle of the past production, so stay tuned.

B

Awesome yeah, we'll all shout out from the rooftops when you post so that'd, be great all right. Well uh with that, I think bartek do you want to kick things off and provide sort of your, your your sure, takeaways from kubecon and any insights uh and yeah tell us what happened. Yeah.

A

Can you hear me you hear me? Well? No, yes, yes, but we don't say thank you. Yes, yes, uh I'm on the phone, all right, so yeah thanks everyone for joining. So uh just to wake up. um I added a small link to you know the meeting notes you have below uh they're, very, very simple, but this is the high level what happened. We had like one hour actually two hours on the kubecon monday.

A

um We call those things like usually like developer summits or some kind of meetings for those who are really really interested in what we are doing and um and yeah. We have um quite a nice session. We have fully packed room and people couldn't even get in so so definitely observability is you know one of the hottest topics still and um and yeah. We started with some introduction.

A

Then we we discussed the white paper and uh you know the kind of status of it and and how people can contribute and and kind of get get uh some information about that and read that even now. uh Second thing is, uh you know, michael and henrik shown amazing demo. um It was kind of like you know, I think, only basics.

A

What we want to accomplish, but uh it's already it already started many many questions around observability solutions, uh best practices, so that was that was the end and the the really really useful um kind of thing to do and yeah kind of the we should kind of uh invest in that more um starting from my side- and I bet uh fredrik, who is here- maybe has some some thoughts as well. I think um we realized that we have a lot of community members and and lots of end users.

A

Actually, we were asking specifically about those people who are not maybe involved with any vendor and that actually wants to use observability or looking for approaches, and you know most of the rumor where were around those topics, and um many of them are starting their journey right and I think what what at least it told me is that we are still missing a lot of information, beginner information, beginner guides and those kind of index pages that can kind of narrow down.

A

You know in what direction they might want to go and and what documents and then kind of references they might uh need to uh follow, to learn more right. So I I was kind of like very um surprised, but like positively surprised that there is still a lot of work around um advocating and by evangelizing, better observability and how to not get lost right.

A

So that's my tl, dr from uh from this event, uh was really pretty amazing, good job everyone, and hopefully we have some people from that on the call and maybe on the future, calls. Of course we try to kind of uh shout out to to people uh for people to join us and and and uh join our effort.

C

When I got to the room it was so packed, I had covered fear and did not want to cram in there with all those people, so I passed on it a certain level of tolerance.

C

As I speak to you from rsa conference where people aren't even wearing masks, it's like crazy there's going to be so many cases here.

A

Yeah yeah we, fortunately we have richie, who are you know just very brutal about those who didn't have.

C

Yeah yeah, no, that was what you were saying, but the same you are.

A

C

Good yeah I've heard a few times.

B

That it was a packed room, uh any guesstimate as to actual head count. I don't know how big the room was. Yeah.

C

That room I.

B

A

C

200 maybe, but the thing is, is during the conference they had moderators that limited a seat per person and no one standing in the event before kubecon like this meetup, there were people crammed in there standing around like it was wall-to-wall people. There was no space, so I just didn't feel comfortable personally wow. I skipped it. Yes,.

G

B

Wanted to go, I.

C

B

Show up early enough, I would assume we can get a list of attendees, then um that that the cncf I mean, if they were taking, they were.

C

Not taking because this was a pre meeting, because it was just a lucy goosey, they didn't scan badges. There was no moderator. Like that's disappointing.

C

We have lists of. uh I know that there, my colleague, did a session on open telemetry. I did the jaeger maintainer session and bartek also did a session on several. I you had two right bartek, I want to say I saw one.

A

Yeah free, I have three but um yeah.

C

So you can get, uh we can get lists from the cncf of those sessions. I'm sure uh you know which.

B

C

Related to observability.

B

Yeah I'll put something in the observability channel later today, but uh long-running work item we actually now have, I can say uh successfully provisioned or had have had provisioned a google drive for the tag, so we actually have like a neutral open place where we can put stuff. uh So that might be a great place to put that, and I.

C

Just didn't know if people want like I work at a vendor. We obviously don't want people's names because, that's like you know, not good, but if we want to use them, if you want to use them personally matt for some type of outreach or something I think it's time.

C

I think if there are folks that self-identified and like their contact.

B

Info because they were interested, I would want to send out an email from the tag to saying hey thanks for joining us. Here's notes come to our meetings. If you like, just just leave it to there, I I don't mean anything more targeted or nefarious than that.

H

B

Just a thanks for thanks for joining us and and here's where you can find us.

C

Yeah, I don't know that, there's like any kind of like reach out to me thing that attendees do in specific, um but I don't know you could ask in them. Are you part of the marketing channel and or you can ask matt, you know matt right yeah, just ask matt and say like what can we do from a tag perspective to try to reach out to people at kubecon that went to observability sessions that might be interested in the tag? I don't know. That's.

B

A great idea I'll take an action to chase that down.

A

I think there are some some more feedback we can just sorry just just to time in there are other kind of things we could just think about. You know what what could we put cncf and we could do better like definitely, we need a bigger room so, for example like on our attack, observability uh meetings or not meeting, but actually the talk about introduction.

A

There were like less people than on the you know this meeting right, because probably people have had more energy. It was like a dedicated day which were much less kind of content, and I think this speaks for itself like we should invest in that, maybe a room more than even on the top right. So all.

C

Those also they also want, like you, have to pay for the other one right, the one that's before the conference.

F

And no, not those ones, those ones were included.

A

It was not co-located.

C

Oh, I thought you were talking about the co-located. No, I went to the one that was just the dave on the monday. I want to say exactly.

A

That was, that was a ticket, normal yeah yeah.

F

But yeah to come to just uh mentioned uh yeah. I think there is a in open territory with the same feedback. People are looking for getting started, guides examples uh they they all they see things coming in, but it seems that there is a large portion of the community who is not very confident on how to get started. So I think we we should put a lot of effort. I'm trying to internally get some budget for a project that I would like to run in january.

F

So I'm planning to do like a observable detour, um and uh the idea is that uh uh I will probably find some company who will be able to host those various meetups, I'm doing various mediums and various cities in the u.s. I think there could be a way of getting speakers getting examples getting and trying to bring that community together and speak and exchange and network.

F

I think that would be great for the future uh yeah because out of kubecon, which is really big uh people, sometimes I have the chance to come and fly in, but there's a big portion of the community who's don't go to kubecon at all. So I think, having smaller say, meetings regional I mean on locations, though I think that could be very useful for for the entire community for sure.

E

I had a question too about the uh I know you said you gave like a recap on the white paper. Maybe um here we could also yeah I'm curious to know sort of how that's going and if you know some of the things uh that we're trying to promote more about. You know whether it's like tutorials or that kind of stuff. Maybe that's somewhere, where we can, you know, add, add links or something or or just add some type of information, or I guess yeah.

E

I don't know I'm curious about how the white paper is going and if it's sort of uh moving closer to what we were like hoping, it would ultimately accomplish.

A

Yeah sure, so, um if you want to check if it goes along what you, what you think we want to accomplish, you can already check that it's already on the uh under our repository right and um we were just making last, you know, read proof and trying to just actually like have finally finally find final touches, and we got a little bit stuck on on our review on our kind of stump that this is what all we want to do. So it's like kind of we try to be. Maybe too perfect right in this.

A

So um I'm definitely you know. After after this meeting it, it was so clear that people really need that uh in in really not even like. You know full all that details enlisted, but at least you know the basics, um so we should. We should kind of make sure we can make it officially public and obviously kind of stable as soon as possible. Right but like the content is already there, it just last touches. So if you have any feedback, just really, you know literally make any issue like tangible right.

B

Yeah, so thank you for that. um Some of the original uh folks that were driving the white paper uh have, for various reasons, not been quite as active in the last year. um You know just due to I believe, just to time constraints and whatnot.

B

um Okay, but bartek are you on on point to kind of drive that to completion I mean, I think that would make sense, given your involvement all throughout, but I want to just be explicit that it does have an owner and it's being driven um to a point where we can actually release it uh uh or at least declare doneness on this version of it uh and yeah yeah.

A

Man, let's have a clear, exactly, let's have a clear owner. I was kind of like fuzzy. I yeah. I was already touching it. I was blocked a little bit but on other people but um sounds like yeah, I mean we, we have it so yeah. You.

B

A

B

Great, if you want to take it over and you know, obviously anything that's needed or if we need to communicate stuff or bring people in or you know if you could, if you could just provide us um or provide the tag. Rather, you know some guidance around how they can engage or what's left you know, what's the gap, etc. That'd be wonderful and I can follow up with you offline to make sure uh that any existing uh prior art uh is actually all kind of coalesced together, because there's obviously quite a lot of effort.

B

um I believe also. We have memory, serves an issue outstanding to review the chinese translation of the white paper, which has been translated, but we're waiting for someone who uh is fluent in mandarin to kind of give it to give it a thumbs up. So that's so that's something that we need that I don't.

C

Have I know a lot of people if you need someone.

B

We do so that'd be wonderful if you particularly need like someone with some domain expertise, yeah yeah, like.

C

Also builds skywalking. He's he's usually really helpful, I'll reach out to wu shane. Thank you. Super cool guy.

E

Yeah- and I guess I guess just to clarify there- I guess I was kind of seeing if, if there's any way potentially, we should alter the white paper, given the feedback from the like people who, who I guess like mentioned, what they were looking for at kubecon, I guess that's kind of um so I wasn't there. So I'm not sure if there's a way that we could um or maybe if there was even just like an issue of like hey people, said they wanted content more geared towards this.

E

That, maybe I mean I don't know I don't want to. I guess- move the uh the goal post there as we are trying to get a first version out, and maybe we do it for like a second version or a separate paper even altogether, but I was just curious. If there was um you know some way, we could put out some content that would uh sort of satisfy.

E

You know what people were saying that was missing um when they were at the meeting.

F

I I don't know if you um know the google sre website, uh that is around the sre book that was initially released and they have come up with a lot of small um exercises or workshops or practice practical examples and you can go through. I think you, the great I mean again it's a suggestion.

F

If we have the white paper who is in a very, very stable format, then we could take advantage of the observed demo environment and do some exercise where people can walk through and try to apply those concepts in a demo environment about whatever. But it will be more more because they will look at the screen. They will understand it more the actual concept and the the value of that. Again, that's that's the suggestions.

F

I don't know what you think about that, but I think it would be great as a next step to provide even more uh getting uh getting started or onboarding tutorials for for the community.

C

That's kind of interesting, maybe.

E

Could you put the link somewhere what'd, you say, google sre website yeah.

F

Let me uh find out.

C

uh It's uh sre.google, I want to say, is the website for all the books and stuff.

C

Yeah you just go to sre.google, they have all the sre books and everything else is on there. You can get uh for free thanks. It's a really useful site.

C

I already sent the chinese pr to him that so I'll, let you know oh sweet, yeah,.

B

Thank you so much.

A

But yeah I will I will on the whitepaper side. I will definitely compare that with what we have on the notes um for that meeting. We have some outstanding topics and uh yeah totally and I'll be. I would try to have like summarization, where we have gaps and where we need to so we can make together a decision. Is it something we want to finish before release or not so yeah?

A

E

Yeah like- and I guess I mean these.

E

Change so do we just do the first version bringing up how it is now and then sort of uh as like a second iteration and some of that stuff.

B

All right, uh who is that question too specific.

E

I don't know I was just saying that's in a question. I guess I was uh generally saying that we could um yeah it's more of a statement than a question. I guess.

A

Anyway, I think we finished the topic about the white pepper so far so yeah, I think some other one.

B

um Briefly, on the on on the uh the art of slos, um I can also provide a link, but a month or two, I finished the o'reilly book on rolling out slis and slos to an organization without making that organization curse your name, um it's quite a good book.

B

uh um I can probably pull it uh into the link, but um yeah highly recommend it, um particularly on the the human aspect of that right, how to kind of not yeah not how to not make everyone hate you for trying to to give tools anyway, um so alolita's not here yet, but that's okay. uh I guess next up brian, do you wanna um you get into profiling?

B

E

um Always uh yeah, so I posted some links in the uh in the slack channel there uh about sort of uh yeah there's like several links now the video.

E

So I guess, yeah to back up um on friday, we met with a there's actually like a huge turnout of people is like plus people came to just generally um this idea of getting um having profiles as an accepted uh sort of um you know, event type in hotel as um yeah as at a similar meeting to the one that this observability tag had the hotel, uh like group, had a similar meeting that was also very packed and one of the biggest uh like voted priorities.

E

There was people who were interested in um profiling becoming a I guess, yeah and profiling, as it's become more popular. A lot of companies have popped up around profiling and a lot of like cloud providers have added it as a part of their offering, and so people were saying that. um I guess I wasn't.

E

I wasn't aware, but that with previous signal types in the past, I forget if it was logs or metrics, that um a lot of people there were saying that that that it a lot of, I guess, people who were doing logs or metrics before hotel sort of like blessed a format for them um had issues then trying to like go back and support the new sort of like standard format, and so um so yeah. It kind of seemed like profiling is at a good sort of in-between point where enough people are doing profiling and enough.

E

um You know formats exist where it's early enough as well, that if there were a sort of standardized type that it would allow for the entire sort of ecosystem to move forward, not only that but move forward in a way. That's consistent with the other signal types logs, metrics and traces. um As I guess, a lot of people here are probably familiar with, as we kind of talk about this uh briefly in the white paper as well, and so um so yeah.

E

So that's sort of the I guess sort of um context there and so yeah. So we met on friday to sort of just have an introductory meeting where um it was mostly just kind of getting various people, and um you know profiling developers, uh thoughts on on this sort of effort and how it could move forward.

E

What are some of the considerations that we need to have all of that is sort of outlined in the uh I put the links, both in the slack channel and also in the meeting notes for today, um so yeah definitely take a look at those if you get a chance but um yeah, and so basically we we talked a lot about that. We didn't come up with uh the most concrete to-do's, I kind of added a section at the bottom, with just some like general.

E

um You know things that I felt would be good to make progress on before we, you know meet again as that group, but yeah. I kind of wanted to bring it up here. Seeing as this is probably the biggest group outside of um you know, outside of hotel, who would you know, have a vested interest in this and um and sort of the expertise to kind of move, this effort forward, um and so uh so yeah?

E

I kind of wanted to similarly here sort of talk with this group um about just hotel in general and sort of get you guys perspective, or you all's perspective on it and um and sort of how we might be able to move forward, ways that this tag can um can uh is distinctly positioned to help and all that kind of stuff. So um yeah, that's a lot of information, I'm curious. If anybody has any thoughts on this, I guess effort in general. This idea this concept of um getting profiling into hotel, basically.

F

I think it's super exciting. This is great, but I think the market is waiting for that. That's great news for sure.

E

Cool um yeah, I mean I'm curious here too, uh and maybe if anybody wants to talk about it, I mean obviously there's you know logs metrics and traces that um you know are already. I guess at various stages of being supported in hotel um and I'm curious what people's experiences have been with um with those are you using them? Do you find it to be?

E

I guess a you know. Yeah like I mostly want to better understand sort of how people are using the signals that are already out there, so that, as we try to kind of like you know, also get profiling in there, so that we don't necessarily make the same mistakes or have the same issues that some of these other signal types have had as they've, you know, got gained more popularity, and so I'd also be curious.

E

That's here if anybody has any thoughts on as users of logs metrics and traces and whether or not you're using hotel. If you have any thoughts on sort of like how that has gone for you um and ways that you feel like it could improve.

B

And before we go into that, thank thank you, ryan, and, and let's do that, but before we kind of drop into that, um could you or someone else who might have some direct expertise here uh kind of just highlight for the folks that might be new to this that are watching this recording later um the history around symbolic info and being able to kind of you know, there's a general.

E

B

Around hotel, I think liz uh had a pretty cool, pretty insightful comment um in in one of our slide groups about this um uh that you know this might finally be when hotel needs to add some sort of comprehensive support for symbolic information uh to assist with look at looking up symbols and other things and as I'll say from my perspective, as an aside commentary, were we to ever in a hotel, have, for example, execution tracing like any. uh You know more traditional, you know um much much lower level execution tracing.

B

We would obviously need some some variant of that um yeah.

C

Talk about you know our.

B

Tech, I know you've got some experience here as well.

B

Yeah but I didn't mean.

E

B

Right to bar tech, just whoever.

E

Yeah I mean I, I can talk a little bit about it. I'm curious uh of other people, I mean yeah. I mean one thing that I would say related to. That is just that. uh I mean I think that there's a big portion of that conversation also around ebpf as well, and I guess you know being able to find some way to.

E

I guess yeah deal with symbols. I mean it's just something that I guess is like newer, that um it is honestly kind of a tricky problem to solve in the quote-unquote best way to do it, uh there's a lot of considerations that you have of um and then yeah, depending on what language you're using they're sort of like different.

E

uh I guess, like paths that you have to go down there, um I can speak from uh from, like my experience with pyroscope like we do symbolization as well in, like I guess, like a slightly different context, but um basically just in order to uh compress the data that we're sending as we're sending it around. um We use uh trees and tries in order to um basically like compress symbols themselves, but uh yeah.

E

I don't know it's it's a pretty massive topic that I don't know uh we kind of brought it up in the context of profiling, since profiling is already like.

E

One uh profiles tend to be very large compared to the other signals and then two that profiling, sometimes the data that you're getting, is um you know, sort of like an address in memory or something that you ultimately want to be able to then transform into a function, name or file, name or something like that, and so it is definitely, I would say most relevant to profiling, but um I feel like yeah like as as matt said, it could be a topic in and of itself and used elsewhere. uh Definitely.

C

I mean uh I think logging can also have some of the same challenges just because uh you're dealing with the unstructured data that doesn't have any bounds on it. So I think profiling. Actually, you can build a better schema around it than logging for sure it's easier to put into a schema but yeah. It could support both.

B

B

um In concrete terms of how might someone engage, uh I mean? Is there a formalized working group in hotel, then that we've linked to, I know, there's a doodle, that's going around for a regular meeting, but just in terms of governance, I want to make sure that we're explicit about where folks are congregating and how others can join them. I know we've kind of got a number there's a slack group.

B

That's not on cncf slack with a whole bunch of people in it 50 or so, and then I know hotel has its own.

B

Okay. Alolita should be joining shortly, so she also could provide some context, but does anyone? Does anyone already know the answers and.

E

Yeah, um as far as that goes so yeah, it's still in the early stages. Honestly, the um yeah, like the the typical process, uh is that you, you know first kind of meet a couple. It's similar to sort of how our working group here started. I guess you kind of meet a couple times, make sure that there's enough interest and enough sustained interest that um you know that you can ultimately get someone who's going to like maintain it. Someone who's going to sort of like whenever.

B

I I meant like: is there like an.

E

Issue in hotel.

B

That we don't have linkedin, I can go looking like. Is there a formal working group? That's been spun up so.

E

No so yeah, so that's what I was doing too, so not yet sort of yeah. Of doing that, I would say for now the most uh sort of centralized place for communication around this effort would be the um in the cncf the hotel profiles channel, um which I believe is linked somewhere.

E

Oh yeah um yeah, it's linked in the uh in the tag observability channel as well uh liz posted it, um and actually I wonder how many people are in there um yeah it's got like 81 people in there, so I would say: yeah it's uh yeah we're in the process of consolidating all the different groups who are interested in this into that channel and then so yeah. I would say generally to expect communication to most reliably happen there and then also on the issue itself. That was started around two years ago or something about this.

B

Yeah, I'm just dropping some links into the meeting notes for for what we do have so far.

E

E

Yeah, um so I don't know um yeah if anybody has any thoughts, if not I mean that's fine too, um but yeah I'd be curious to hear um yeah like if people are using hotel for like logs traces metrics, whatever um yeah like if they have any ins, anything any insights to share or any experiences to share around you know integrating it around.

E

um You know, maybe the process of like switching from you know, maybe before hotel existed for those things to using hotel for those things how that process went anything along those lines would be helpful. If people don't, uh you know, have stuff to share around there. That's fine too, but I just felt like this would be a good group to ask.

F

So we're trying to collect those feedback from end user. As of now on the yeah hotel and user group.

E

Yeah I mean, I think I think that one of the things that we want you know so I feel like a lot of these kinds of efforts, can have a tendency to sort of uh you know um potentially drag on or you kind of like you know you make a lot of process, you get to progress, you get to the end and then people are like.

E

Oh, this isn't going to work for me because xyz or something on those lines which then makes it take longer, and and so all that kind of stuff- and so um I think with this like one of the things that I feel like is important- is yeah getting some of those some of that kind of feedback um and making sure we don't sort of like you know, we account for those potential pitfalls early on um yeah.

E

If that makes sense,.

F

Yes, it really depends of the language, because now most of the end user there's they have only uh focus on tracing because it was the the signal that was the most stable one and then metric has been announced during kubecon valencia.

F

So it's very hard to get feedback from from real end users, but as of now the most of the the feedback depends really depends on their language uh depending on the language they're using they have different experience and different, uh I would say uh frustrations depending on, for example, php was a the customer was that I interviewed he was very. uh He basically said he is too complicated and it seems not stable yet so, let's wait a bit until something is better.

F

um So, most of the end user that were at the end user hotel group that we had um in valencia, they were all looking for examples, more examples. More tutorials. They say that the few examples that they are currently available in the vs repo are very hello world oriented.

F

uh So it's very hard at the end, after building a hello world to get the actual value, because it's basically one component, there is nothing interaction. So it's very hard to get meaningful uh measurements out of the instrumentation done by open symmetry, so they were looking for more uh suitable examples to get a better picture and better understanding on the the actual work that they need to do to implement it on their within their environment.

E

Interesting yeah, I mean that makes sense. I mean I think things we could do. There is yeah like one make sure we have a uh input from various languages early on to sort of avoid. You know avoid that as much as we can and then two um also sounds like a good uh sort of use case for our like observates.

E

um You know project here that we might be able to um yeah to sort of add examples for different languages early on potentially or something along those lines. um You want to say something: matt, hey uh yeah,.

B

I I I was just capturing some notes here. I think, in addition to vendors, that we've started enumerating there like polar signals, psyllium, pixie, etc. um Periscope is open source. uh There's a variety of projects that I think we could identify in a landscape at large that might particularly benefit from deep and or rich integration.

B

You know, with profiling via hotel that there might be good demonstrators and or early adopters to vet some of the the ideas um so I'll start a list there as well, but you know um in particular.

E

B

Cpu bound things would go to the front of the line, um but, depending on you know the specifics of the profile and how it captures idle time, you know, might be quite a bit more broad than that. um uh Grav has their hand.

G

Up hey so I'm new to this forum, so apologies if this is a bit out of context. Thank you um but yeah. This is a super interesting topic. I would like to ask about uh some of the like two things here. um One is a minority view potentially on what would be the reasons to not do it uh ryan.

G

uh That would be one uh and to get a better understanding of that. I think will help you know, make the solution most likely better for everyone. um Second, um I wanted to ask about uh the like. Oh.

C

G

I'll, let you answer that one and then we can go from there.

E

Yeah um I mean.

B

Could you just summarize ever so briefly the question again uh for the recording have my audio crackled.

G

Oh you uh matt. Do you want me to summarize the question again.

B

Yeah, just the first one, I'm sorry, I I had some sort of connection on my side and I didn't have a question.

G

No worries yeah, I was asking about: what's the minority view on like why we shouldn't include profiling in hotel? um Is there a contrarian perspective, just yeah just to understand that very question? I don't know.

E

Yeah, I mean, I would say the biggest contrarian perspective. There is that um profiling is new and that you know some would make the some have made the argument that it's too new and that it needs to mature more before we um you know before we come up with a sort of like standardized version of it. um I think that's definitely the most contrarian view um I'm definitely biased towards um having so like. So I guess also for context around. um You know I work on pyroscope, it's an open source, continuous profiler.

E

We integrate with a bunch of different underlying profilers for various languages, and one of the things that we've found is that for each language they do things slightly differently. So um you know the way you get profiles in ruby versus the way you get profiles and go versus the way you get profiles with, like you know, go we'll have like ppro, for example, like java, has jfr um in order for us to consolidate all of those and consume them in one sort of uh unified way.

E

We kind of had to do ourselves a lot of this sort of standardization and um and so yeah, so I'm definitely biased towards uh thinking. It is definitely necessary, but I could definitely see the argument that some people think that it should be that it should be more mature, um which is why I brought up that point earlier that they were talking about at the hotel meeting. Where, like um I wish, I knew which one it was.

E

I believe it was metrics that, when hotels start added their, um you know support for metrics that, like dot net had already had its own sort of api for it, and so it became this like massive headache. Where now you had to sort of like through the net api you had to like, then it used the hotel api under the hood, and it was, I don't know it seemed like it was a very um and don't quote me on that. It may have been logs.

E

I believe it was metrics, but um then it turned into a big headache, and so I think, there's also an argument for not letting things get too mature before you come up with a standardization, because then everybody sort of digs their feet in their own unique ways of doing things and it sort of makes it harder to then go back and undo things that people have already built on top of, and I think that's just being new. It offers a good opportunity to sort of like standardize things from the beginning.

E

Assuming we do a lot, um you know the the sort of requisite amount of uh forethought in making sure that we are thinking about building it for the future versus um you know not painting ourselves into a corner.

G

Got it that was super helpful. um I just have one thing to add to that ryan, um which is, uh it might be interesting to take some of the examples from the the efforts that are outside sort of observability.

G

For example, the one that I am familiar with is around um smart home stuff, and this is completely different, but they had very similar issues um and google and amazon and other companies are trying to figure out an interface. So how do you create something that is working for a subset, but also allow for innovation on other stuff that you know individual companies and individual smartphone providers would like to to kind of provide? So if I remember correctly, then the matter is the protocol um where they had similar issues.

G

So we might want to pick up some design patterns from uh some of these interfaces.

H

Yeah, I think that's a good good suggestion, because uh um again um matter is definitely evolving and, and I I think that at the you know, layer that we are looking at right now with ryan. You know what ryan was mentioning in terms of profiling.

H

uh You know data, uh it's definitely a layer that needs to be integrated into matter. If you really were to, you know, say, look at smart devices or any kind of iot devices or client. You know components actually, inter intersecting into the larger observability stack right, because, fundamentally today, what you're seeing is that uh there's a lot of emphasis on all most of the observability projects with server-side.

H

You know integration right and and, as you see, the world and mata is a good example of this. um Where you see the client, you know world also intersect and and needs to have that same level of um observability, with the metrics and and the logs that are, that would be pushed from there.

G

Yeah, no, that certainly is a good point. I was also trying to draw attention to the fact that some of the problems that they're facing in standardizing that interface are a sound similar to what hotel is facing.

H

uh They are similar, yes, but they're, not at the same layer right.

G

I understand yeah.

B

um So I'm just gonna chime in we've got five minutes left, so I want to make sure uh uh camo or kamal, I'm not sure how yeah.

H

Yeah go ahead. Come on sorry, I know yeah.

D

Hello, everyone, uh the name is cameron, uh I'm working for polar signals and we are building a observable profiling tool uh called parka as well. I just want to comment on the sdk part of the hotel initiative. I guess, but this will be a little redundant after ryan's last comments because, like what we observed, we probably don't know what we don't know yet for most of the part around profiling and with the hotel. We had this issue.

D

I guess it was again with the metrics, I'm also one of the maintainers of client goal and prometheus client golem. So I'm like working on the client side of the metrics part as well. I don't remember anyone actually like contacted us from the hotel side and we worked on the metrics sdk as well. So all in all, what I want to say is we should definitely hold on the sdk parts and maybe focus on the otlp and how we can actually standardize the wire format. First and one part of this is definitely ebpf.

D

We are building a profiler based on ebpf and when you use the ebpf there is there isn't much need from an sdk aside from maybe correlating with the other signals. I guess that's the goal of autel as well somehow include the correlation, but one other thing you can actually have the correlation without a need of sdk as well or just a. I don't know, uh really minor touch from the sdk itself, which we actually demonstrate demonstrated this during kubecon, with wartech on the thanos maintainer track as well, so yeah the gist of it.

D

Let's, if we can do this, that was also the idea from the last meeting. I guess let's focus on the otlp part first and how we can actually optimize this format, and this includes a lot of pieces like how we are we going to include the symbolization information and how we actually encode that symbolization in for information in an efficient way. Maybe then we can start working on the sdks yeah.

H

I agree with you because I think that you know dlp is the first layer that we actually should look at to be able to handle. You know the detailed uh you know, uh nuances of the ebpf data itself right and, and then I think sdks are you know ideal for instrumentation, but don't necessarily you know need to be as complex or or extended for only ebpf or ebpf, specifically right so, uh and I think that uh folks, kamal, I think, from polar signals, have been involved on hotels. So you know uh it's just that.

H

You know you. I think folks have intersected back and forth through different parts of the project uh metrics you know like initially when we had the prometheus uh drop discussions again, uh I think frederick and others were also involved. So definitely but thanks for joining in.

B

Yeah, I I wanted to interject a little bit too. I I've written a profiler myself a couple decades ago um back when arm4 was a new thing, um but you know many of the devices that oliver made reference to. You know both mobile devices. uh Increasingly, you know arm-based uh server, hardware for edge scenarios uh and lots of other sort of socs, I would expect, would have you know arm, for example, had cp14 coprocessor 14, so you could have a monte carlo style.

B

You know profiling, sampling profiler, but instead of using a timer, interrupt you're using tlb hits or tlb cache messages or or you're or you're. You know interrupting on every on every nth eye, cache or decache hit or miss right. So you can start building profile reports, but not using a timer or using different frequencies and timers for different types of devices, which can vary quite widely. So in discussion of the otlp format, um you know I would want to make I I does anyone have any visibility or could anyone comment on you know?

B

Are these things already being talked about as being modeled uh in an open enough way that that you know otlp becomes more of an envelope uh format? You know that that allows for extensibility and for a wide variety of things, or you know I wouldn't want to make it overly prescriptive to say time-based profiling or you know something like that.

B

E

H

Friend, go ahead.

E

I was just gonna say um I guess yeah related to the the format in general that yeah that I think that is definitely the biggest uh the biggest priority right now. Is that we're trying to um sort of like narrow the the space of like what the what it can ultimately be? I mean there's a lot of um you know formats out there that the final format that we settle on eventually will be some.

E

You know it will be some combination of what existed exists, I would say maybe 80 plus you know 20 that allows, for you know what we could add in the future. So, for example, you know- and I'm not you know, I'm just using this, for example, but like let's say, like p prop, which is already you know very popular, the entire of like goling uses it.

E

So it has the ability to add, um like labels or tags, for example, which allows you know it has place in the schema for that which- and it also has a place in the schema, for um it might not be the most efficient, but for also like adding some ability for like symbolization and being able to to deal with that. And so um you know, if we were doing that, then you know there's kind of like okay. So what doesn't that have that?

E

Maybe jfr does have or that you know some other formats in finding some kind of you know common denominator of what we could do forward.

B

If, if you did decide to find a superset, you know just to brainstorm with once worked with other protocols in the past, with this type from yesterday um same ideas, but you know new languages and whatnot. um You know you could have an agreed upon superset format. If you will- or you know something that, as you mentioned- roughly hits that 80 or ideally it encompasses and is broadly enough to encompass all. But then then you can have the the folks that are implementing their end of that protocol.

B

um You know to to you know which might be p prof or might be. You know java the the java stuff or flight recorder or might be the net on. You know, diagnostics of aj. um You know they could they could define conversion functions or conversion um filters. You know that would mutate to the the standard. Otlp format see. So you get like an agreed upon mutual standard, uh but you don't have to go.

B

Have people change anything from what they're already doing and what's already deployed, which is probably a non-starter, uh and then it puts the onus on them. You know to to own their own. um um I don't know this could live in the hotel collector. I suppose you know, as a as a as a processor uh seems like the right way to do it. But again um I guess I guess the hotel, profiling group and or the recurring meetings are probably the right place to to get nitty-gritty.

B

H

I agree yeah, so I think ryan again, I think one of the results of the profiling discussions that are ongoing. You know just started uh with the discussions coming from kubecon uh is that uh on hotel at least it'll be good to kind of draft up an otep to submit you know, as we get closer to that right and and uh part of that process again has been that you know not only actually defining what are some of the common.

H

You know areas that can intersect with otlc, but also uh the semantic conventions that might need to be defined. You know specifically addressing bpf, you know types of data right, so again lots of cool work there and I'm hoping that we can actually collaborate with other projects. You know parka and others, which have already been looking at this space um to be able to actually have a well-rounded proposal, which is you know, meeting the the needs of all the projects that are working on this together.

B

Yeah, I think here you know we're a few minutes over, but just to close out at least I do think that the the tag observability like you know we exist specifically for this kind of thing, and I think I think that we can help this overall open effort by cheerleading. You know by by by broadcasting it uh by by using our own networks, um moving forward to bring more people in and make sure that you know people aren't ex inadvertently excluded because they just aren't weren't in the know already.

B

So I do think that that is a role that we have um yeah and again. Actually I put a link to our issue here that ryan created in the toc um deck as well today and talk to it briefly again to just kind of spread awareness that this is a topic of discussion, that's exciting and and and tip of the spear. If you will for where the industry is moving.

E

Yeah and- and one last thing that I'll add there too, I know over- is that um yeah. So I I added at the bottom of that uh of our meeting notes from when we met on friday, um just some like to-do's of like things that we can do actionable.

E

You know writing something down doing some actual resource and research and, like you know, coming up with results, if the, if anybody here can add to any of that um pretty much everything we talked about today is already on there um and if it's not I'll, add it, but it would be really helpful to have people from this group help contribute there as well. So um so yeah thanks everybody.

H

Yeah yeah, absolutely thanks ryan for leading this effort and pulling everyone together appreciate it.

B

Yeah um is there anything else before before we break? I don't know at what point zoom decides that we're done or.

H

Working into an hour to kick you off.

B

uh Yeah we're in five minutes too so anything else before we drop all right, uh yeah, uh henrik I'll, look for I'll look for the link once you're done with post-production.

B

um You know, I wonder if we might just as a parting thought if we might form a little cabal and when things like hey, here's the you know: here's the proposal, the kp, the hotel uh uh proposal, uh that's ready for review. You know we can all blast it out or when, when we've got a new episode of open source news or is it observable, um you know we could do the same so um yeah.

H

B

Good suggestion.

F

Clearly, I think, for the next open source news I was going to talk about to the main topic of the discussion will be profiling, so uh it could be interesting that uh either ryan or someone for polo signals is part of the discussion with michael. So that could be a very uh open discussion, bring and also be also a call to the community say if you want to contribute you're going to get. There could be way of. I think.

B

If you didn't panel format with who you suggested, uh but definitely some someone with some thought leadership um already demonstrated from the open, telemetry uh universe- I don't know if alolita or others uh come all whoever you could. You could suggest a few folks for such a panel, uh but it would be really great to bring together vendors. You know open, telemetry and then even an end user, but I suspect that's not really requisite here um yeah. We.

E

Can talk about it more offline, yeah, I'm happy to uh yeah happy to help they're happy to recommend um or if.

B

E

Doesn't fit that format.

B

um uh Henrik, you know, obviously we could. I think this one discussion that we that we host as a tag.

F

So I think the the format of the open source is very short, but it could be interesting to get more insight news about it, but I think if we do an open like we can call it the ask me anything about profiling, and then we bring four or five people, and then we ask we wait for to get some questions from the community. I think that will be also a good way of getting feedback as well. That's.

B

Great yeah and we can also coordinate with the cncf marketing uh folks as well, and just making a cncf webinar, that's featured um so I you know we we've had issues. We actually have now issues in our backlog, specifically to host webinars and interviews and panel discussions, and things like that. So we have. We have the framework and the governance in place. We have the platforms in place now um you know so we can just use them and it'll have quite a reach. So um thank you, everyone for for your thoughts.

B

uh We really filled out the hour I'll, be posting. This recording a little bit later today, once it's done with post-production, so um have a great couple of weeks and we'll see you online. Thank.

H

You everybody thank you good work.

A

Everyone thank you thank.

H

You take care bye.