Cloud Native Computing Foundation OTel Community Day NA 2022, 29 Jun 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Community Driven Breakout Session: eBPF and Auto-Instrumentation

Description

SAVE THE DATES!
ArgoCon – Register Now | September 19-21, 2022 | Mountain View, California - https://events.linuxfoundation.org/argocon/

KubeCon + CloudNativeCon North America | October 24-28 | Detroit, MI - https://events.linuxfoundation.org/kubecon-cloudnativecon-north-america/

KubeCon + CloudNativeCon Europe 2023 | April 17-21 | Amsterdam, The Netherlands - https://events.linuxfoundation.org/kubecon-cloudnativecon-north-america/

OpenTelemetry is a collection of tools, APIs, and SDKs. Use it to instrument, generate, collect, and export telemetry data (metrics, logs, and traces) to help you analyze your software’s performance and behavior. Go here to learn more: https://opentelemetry.io/

Community Driven Breakout Session: eBPF and Auto-Instrumentation

Breakout topics will be generated and selected by the community as part of the event. We'll be using collaboration tools for you to suggest, and vote on, things you're interested in discussing! These topics and small group sessions are great for discussing and sharing ideas on everything from effective ways to get started with OpenTelemetry, strategies for visualizing and understanding telemetry data, useful tips and tricks around alerting, driving cultural changes with observability, and anything else that you'd like to talk about. We'll be selecting the most popular breakout topics and running them at least twice during the day, ensuring that you can attend the sessions you want. This is the real heart of the community day, and we're excited for you to participate!

A

So welcome to this breakout session, so the intention who nobody was anyone here at the 2020 Community Day Has. Anyone done like an unconference before or devops days, okay- and you know Open Spaces in a devops days. So that's the intention here right like this- is not me standing here and talking to you. This is everyone kind of chatting moves themselves and, using you know our lovely foot boards to kind of note things and come to realizations and discuss what's important now this is also being recorded for posterity.

A

um So with that in mind, I have no clue how we're going to do this, but we have microphones that people can use. So if someone wants to you know like I'm I'm, just gonna Johnny Appleseed these around I guess and people can. Let me just put that like the Irish and someone want to be the mic minder for this side and then someone over here I want to be like some yeah that looked like volunteering to me.

A

So does someone want to volunteer to kind of take notes and help facilitate.

A

Cool come on up, you've been appointed and self-appointed hey come on, give it give it up for uh Libby. Thank you Libby. So.

A

um What I think a good way to start something like this? Is you know this is kind of two things right. This is ebpf. This is auto instrumentation, so maybe just help take some notes from people. Let's go around and kind of say, like you know what let's basically figure out some brainstorm, some topics right and then just kind of jot the topics down, and then people can kind of like rank those a little bit and people can sort of discuss those and again it's like a devops stays open space.

A

If you've been to that right, yeah so do our mic. Our mic handlers want to make sure they're on and then people start kind of talking to the mic test.

B

A

Okay, another check on the other room, so you all start to discuss about yourself.

C

So I'm interested in knowing uh when and what are the circumstances in which we could use ebpf instrumentation to replace any language, specific implementations, I guess- and you know that- might reveal that I, don't know a damn thing about evpf. How about that.

D

I was just going to say I'm behind when it comes to figuring out uh how eppf Works into my work, so I've read the basic directions but having to have Hands-On so it'd be kind of cool for somebody. Who's had hands on who knows how to tldr too long didn't read what EP BPF is for anybody who doesn't does anybody? Does anybody brave enough to say I don't know I could use a quick refresher on eppf, okay enough people here and then you've written it down? Is there anybody well before we keep going?

D

Is there anybody who can maybe two minutes on eppf sum it up for people Henrik and somebody else, okay, uh but we can move? We could spend some time like that, but maybe not now like. Let's get like the topics.

E

um So the tldr of ebpf is historically, it comes like this is literally two decades old. It comes initially from uh automating extraction of of um network data and Analysis of of network data. If you heard of the Bro project or such from two decades ago. That's that's where the root of all of this is coming from these days, the extended Berkeley packet filter and it's still a packet filter, but it is now no more generic.

E

It gives you hook in the Linux kernel, where you can um in certain you, basically a little bit like callback points where you can put more or less arbitrary code which is executed. When something happens, the thing which allows you or the thing which is enabled by this is you can have a fleet-wide aggregation of hey. How is my thing actually looking and the thing does not need to be specified, so the good thing is, you can, across your whole, Fleet extract data from whatever you're running, and it is not specific to your workload.

E

That's also a downside, of course.

E

If you instrument your code by hand you're really near to what you're actually doing- and you have all this context- and you have it in code- so it's really easy to attach proper metadata to your whatever you're emitting, because you know what you're doing and where you're doing it and why you're doing it, whereas ebpf inherently, is looking at things more from the outside yeah personally, I I do expect that ebpf will be more of use when you have large fleets, empirical hyperscalers and such of course they they gain a lot of knowledge just from looking at things from this high level, whereas if you have smaller workloads, arguably directly instrumenting is more valuable, also and then I'm going to drop um you're having a little bit of a problem similar to profiling, where, unless you have your signals and your your symbols and everything in a format which allows you to to walk back on, how that actual code looks, ebpf doesn't know what's happening and also you can't trace a specific request.

E

That's that's I mean it's possible, but it's super hard and super wasteful to do with ebpf.

C

E

It can, but on a different level, of course, it's just on kernel level, the colonel knows what is being executed, what is being done, but it doesn't know where this path is being taken.

F

The idea is that you're gonna Define the events where you're gonna run some code and the kernel will basically trigger your code, so your coaches will be very lightweight and then you can do whatever you want. You can count the number of time there's a socket that have been open, for example, but then you keep in mind that you want to exchange that information to your user space because you're at the kernel level. You cannot do much.

F

That's why there is a ebpf Maps, where you're going to take the data that you have constructed, store it into a PDF map and that ebpf map could be consulted. Through Your, observability solution that could store that information.

F

So it's very uh the way you build. Ebpf programs should be very lightweight in C program, and you know it's need exactly to know which event going to trigger what you're going to measure you can do tons of things with programming language as well. But again it's very difficult. You don't manipulate the context. You just been you trigger a code when an event happens, so you cannot add any annotations. Any context to that data that you just been observing.

D

Dumb question: what does ebpf stand for.

E

Extended Berkeley packet filter.

G

Excellent. Thank you.

D

G

C

You so this is, uh this is Linux kernel specific. This is not a there. Are there any ports that any other operating systems.

E

Excellent, oh yeah, fully yeah, so just for for the recording, uh it was just stated that Windows also supports ebpf.

E

It's for the recording course. People.

H

Unless we use the ones.

H

I

B

B

I'm curious, if anybody has practical experience where they've used it in production, gotten benefits from it. I.

G

Think you describe.

F

F

A pixie Labs is relying on ebpf Parker. Parca is also relying on evpf, so uh we have no information about how it scales, but they are running in production for sure.

H

Control plane that basically controls. Oh we're, currently experimenting at AWS I mean my I am and then some of my friends not AWS myself. So um so we are able to collect um a profile data and then send that to cloudwatch and other services through through open Telemetry, collector yeah.

H

So we're currently experimenting that.

F

H

F

The ebpf has been naturally been designed for network uh initially for the the original origin of the name. No bruh.

F

And uh the that's why it's still Jim I, don't know if you heard about psyllium psyllium is doing a great job for security or for any networking rules, because it's designed for that so I think it's I think the evpf could provide a lot of value for the future of service mesh where you will have more lightweight without any proxy sitting in between your containers. Your pots, so I think that's, but for for profiling also could be interesting. But again, it's very difficult to say so so far.

A

F

In most of the service mesh, they inject container in your pods that will do the proxy no job and in the case of EPF you don't need that anymore. So you remove one workload uh in your cluster.

A

So one thing that I think is like a open question and maybe the people that have been using Pixi or evpf in prod.

A

How are we expected to share context between the hotel context and a like an EPF context right, because these are working at different layers of the stack and open Telemetry itself kind of requires that context, layer to be propagated and everything else?

A

Is there a fundamental mismatch here that we need to figure out some way to like push things down or pull things? You know at the at lower labels, build to kind of like pull that context out. Is there a context that needs to be propagated at the kernel level or the networking stack level that pixie can that something above it can then read and Associate these Telemetry types together, because it seems to me like the full Hotel context, implementation would be pretty heavyweight to run at you know on the Kernel.

G

Yeah, it's actually not clear to me how you can you know properly contacting the kernel, because contact is fundamentally like application concept right, so kernel wouldn't know like which experience the child of you know a different span. The kernel, the pixie does uh say say he does read the you know the on your HTTP, headers and kind of figure out. You know it can. Essentially it's like you know in East.

G

You know the Easter sidecar, you know it can generate the spans, but in this case uh Your ebpf Hooks can generate as best, but still the application you know needs to handle the context.

A

uh uh Do we want to drill in more on any one of these.

D

Yeah, so, following on to that um so you're in the context of code, you, you are the program you're executing you're, thinking about the moment of execution. That's usually why we have to layer in stuff into our code right to propagate the context for eppf. Is it just it's just about Network, right uh and and no it's kernel as well right, it's hooks in the kernel. Yes,.

E

But it's not just about the Network anymore. Initially, it's coming from the networking space, yep, um of course, at what was horrendous speeds of one gigabit back then right. uh You couldn't do deeper analysis without the combination of having a hardware network cards which did on cart, Asic analysis and also hooks in the kernel and several iterations. Later, it's extended to basically pretty much everything in the kernel.

I

D

And I think one of the hard humps to get over if, if we are going, we were talking about uh replacing language with specific implementations with ebpf. That's the hardest stuff to get over is to deal with the fact that there's a million ways to write threads subreds. You know what I mean process you. You would have to be very specific in in that model, in the process and threading model to be able to propagate any kind of context and I'm sure we could I'm sure you can.

D

You can extract from layer seven the context that's coming in and follow it to a certain degree, but that it gets really tricky inside. You know, there's a there's a million ways: I could write how to how to get a program to write and how would you? How would you duplicate? Would you duplicate it?

D

Would you Fork it uh the context across those processes and threads just to get like what is actually so I think the the easy Sledgehammer right now is to expect people to write this stuff in their code right to propagate their own context, but is it impossible I think we just need to work on it? I.

F

I, the only way uh the only way I was thinking I mean maybe I'm completely wrong. Is you have to put a small agent close to your app? It will get the events uh share. It create the right stuff, because the agent knows the context. He knows which program is running. He doesn't know which function is necessary, but you have to go through the profiling level, but that's the only way to get the context because you're getting the event from there and the agent sits somewhere.

F

So he knows pretty much what it's supposed to do, but that's that's what I thought initially, but maybe even wrong.

F

So you you need to have an agent from my perspective, close to your app that will help you to get the this context.

D

And so I, so I had a conversation with a guy named Ryan Perry over a pyroscope uh which is a cool little open source thing that does profiling into continuous profiling, um and we were talking about the overhead right when you're actually tracing calls. Not only do you need the symbols as mentioned right, but you, you also incur a certain amount of overhead just to sample that stuff.

D

I, don't think, that's an intractable problem because you can decide what to sample. You don't have to sample everything sampling every every uh call under the call call stack everything in the call stack for everything is, is impossible, but I think that's the only way that we get with an agent to your point.

D

um The the profile of what's being executed has to have that context, and once you have that context up somewhere as like us, smaller segment or sub subset of the total data, then you can correlate it with. You know the layer, 7 stuff, I think.

D

Does that make sense to anybody else.

C

So I haven't looked at this myself, but there was a an experiment posted here. I, don't know if anybody's had a chance to look at this, but that uses ebb after instrument that go binaries at runtime. Has anyone had a chance to look at that? I? Don't know if they have any feedback on it, because I haven't been following it, but I was just bit me in the ear. Just no.

E

I think there's a project called or something which is trying to do this, calling Auto instrumentation on on ebpf level, which is similar to uh what what the Prometheus Java libraries allow you to do there in the jvmu auto instrument the code, because you have a place where you can do this and that's basically applied to the same. uh The same mechanism and I think this is super interesting course. At the end, most developers, or at least their managers, won't care too deeply about this big instrument that they want.

E

The benefits, not the actual work being put in so having having a baseline of Auto instrumentation would be, would be super interesting, I, don't know how this could be done specifically like I can imagine that if, for example, you have certain wrapper functions, you can actually from kernel level determine what is being called in user space and then again you have a symbols, file which tells you which is which, and so we can trace back what is happening and, in particular at larger scales. This is a super interesting course.

E

You you don't have to extract all this data from all the programs. You do it way more efficiently in the in the orchestrator. In this current, um the kernel I think that auto instrumentation is going to be one of those super important things in the medium term, because it it allows you to not spend time on instrumentation. You only spend the time where it really benefits you and the rest. You get more or less for free. It might not be perfect, but you get this Baseline.

A

uh I actually wanted to kind of not challenge that point, but I want to flip it. On said, just speaking from experience right, you know Fender, but most of our very successful users of lightstep tend to be people that are in go and the reason they're successful is because the lack of Auto instrumentation go means that you actually have to sit down, and you have to think okay I'm going to model my system with these traces and I have to actually go and I have to write that instrumentation.

A

G

Sort of thinking.

A

About observability in the industry is this is something that helps you create data to model that you know to, let you say: hey here's code that describes what I think my system should be doing and how it should look and act and function.

A

Auto instrumentation doesn't necessarily like go towards that goal, because all it's doing is showing you the shape of what actually is, and it's usually giving you a lot of stuff that you probably don't need at the end of the day. Right is there sort of a.

A

From an observability perspective, sort of a secondary or tertiary way of thinking about this, where we use Auto instrumentation.

E

A

Like very rapidly get a setup to the point like to do all the initialization to do the bootstrapping to give us kind of that basic level of context, profit, the application, Level and.

I

A

Have stuff like ebpf on demand for security for profiling, use cases stuff like that right? You know, I think when we talk about this, especially outside of people that are nerding out about Telemetry, um it doesn't necessarily translate to you know the person there into the table where they're thinking like, oh you just mean I, need to run pixie and I. Don't have to do any of this other investment in Telemetry or you know creating spans right.

A

So some of this we need to be better or more specific about like the exact role of continuous profiling in our systems. You know: have people had conversations about this with customers or you know, co-workers or.

C

You know my end users are all they do is complain that they don't have a continuous profiling and they like tracing but don't love it, and they want me to do everything for them and they don't want to write any code at all. They're like how come I don't see this stuff automatically up here. Downstream um so I have the opposite. Experience I think where it's like people want more things freely available to them like free as and they didn't have to write anything extra, um so I and in particular goal line.

C

Programmers are very finicky with me about be telling them yes go ahead and use these key value, Pairs and the attributes because they like it, they really don't enjoy the ergonomics of the interface of using attributes. So it's like yeah I'm, constantly arm wrestling people, I, guess yeah me personally,.

E

I think both of you are right, but you're talking about different things, so, yes, I, absolutely agree and as much as my experience it or most effective and efficient users and and customers are the ones that actually think about what they're doing. uh But this is not the majority and also here in this room we have a highly self-selected group of people who actually care about this stuff. There is the normal person is more on this side or on his customers or user side.

E

um They don't care, they just want to have the benefits, and yes, they won't be as good by most certainly not, but they have something so I think you're just talking about different user groups, but the thing is his intended audience is like 10x or 100x larger than the people actually care about the stuff. So, on the open, Telemetry level, it makes sense to care about this. Larger group of people.

I

Just wanted to add, with everything that's been discussed about the difficulty for new users using open telemetry.

I

This would add another barrier if it wasn't implemented in a way where they didn't need to know all of these things off the bat right.

H

um Continuous profiling, I, don't know how relevant that is with evpf, because with continuous profiling you're, like you, want to see even the private functions that you're calling within your application.

H

You want to see how long a particular piece of code is taking there any exceptions, if that's using more memory and stuff like that, I, don't know, maybe I'm just not clear on that, because with evpf you're, basically only capturing all those things that you are intercepting with the Peace of line of or I mean some lines of code when a kernel, the kernel is basically invoking your code, and that is basically when maybe, let's your let's say, you're, making a sleepy call or maybe making some grpc calls.

H

Then you want to log that, but with continuous profiling, you're like every single line, I've got the application executing is being kind of monitored. That's when your developers are interested because they want to know, tell me which part of my application is causing problems, so I can go fix it, which they don't get right. So.

F

Netflix did that many years ago, with Fame graphs, so they were with their systems using evpf they're able to get all the stack going to the kernel and they're building those flame graphs showing you. The Flames is the top of your user functions and then go down to the kernel on the bottom and they can could track and everything, and then they store every single small profiling for one single timestamp in huge database.

F

And then you can click on one single Pixel, which is a timeline of your production, and you can see the flame graph of that actual pixel, which is impressive. But but they did it 10 8 years or ten years ago.

E

So for continuous profiling, I with my Tech observability head on I, strongly believe that this is the fourth signal which which is emerging, and do you see this again and again and again after Matrix locks. Traces depending like specific orderly depends on who's the user, but you, the next logical step after traces, is usually a continuous profiling as to ebpf, specifically I, think for open Telemetry.

E

There is a difference in in how pressing the meters, if you have a go Application, you don't really need to have this in in ebpf, because you have something nice in go already. If you have a python application, it's a lot more pressing to have something outside of this course. You don't have any any other place to do this or nothing built in Java, sits in the middle. Of course, you have a different place. You can use jvm for all of those things in the generic case.

E

I think, yes, absolutely that that continuous profiling is one of the next things for for open telemetry.

D

Is that synonymous? There were two mentions of profile, one in I. Think Lolita's, uh talk towards the end of the talk is profile synonymous with the continuous profiling topic.

E

That we're talking.

D

About okay, so these are the same things: good words matter: okay,.

E

Yes, and also like with both Lolitas and my tech, uh chair, hats on sorry, I'm still, half asleep jet lagged um again. This is something which we see as one of the next things which which we as the observability community, need to engage with and actually solve in a more generic case and established standards open formats, blah blah blah blah.

D

Blah yeah yeah I imagine we're going to get into a lot of language. Specific runtime, specific uh uh translation of the every every language has their own different symbols and it's going to essentially be the same kind of situation as open Telemetry with different language, bindings and stuff for profiling.

E

That's part of the promise of ebpf. Of course you don't have this, you, you get reduced feature set and such you get. You reduce your visibility, at least depending on how specifically implemented, but you get this for free across everything which you're running doesn't matter what language run it in a Fortran doesn't matter, ebpf will still be able to extract meaningful data from it.

D

See now I really have to go home and play I.

A

Also want to point out like there's a really great.

A

um This hasn't been realized yet, but there's a huge opportunity for sort of Brownfield monitoring scenarios and ebpf, where, like you said, Fortran right, like there's immense immense program, observability programs at very, very large companies that have significant you know kind of Legacy technology, Stacks that they want to integrate. You know if you're a bank, if you're you know a big bank, then yeah you've got 20 billion, mainframes still running every single request.

A

That goes through your system at some level, so you need a way to kind of connect and view both the new stuff and the old stuff or there's a desire for it, at least if Me Maybe not in need.

E

The full mainframes you don't have ebpf, but.

A

um You got it at the part. That's actually connecting.

E

Yes, so the the thing about Brownfield is correct and also um it's about closed Source. That's one of the things with with microphone ahead, on which I see a lot. People who want to use ebpf have massive installations on things. They do not have the source code for and even.

I

If they have it.

E

They have a binary which has been 30 by certified by a state agencies, so they can't just recompile it or anything they have to work which, with what they have, they cannot read instrument this yeah.

A

So to that uh point,.

A

And this isn't really an auto instrumentation question, but maybe it is you know we spend a lot of time right now. I mean in my presentation earlier right, the most popular library or the most popular repos in terms of activity is like Java, contrib and collector contrib. So there's obviously quite a bit of energy going towards zero change, instant. You know: zero code change, instrumentation.

A

Do we think that there's a real opportunity.

A

I mean I already I'm gonna answer my question like it's never going to be anything other than that because, like you said, there's a lot of people that have binaries that have programs that they can't make Source changes to or they're on old versions of things, they're never going to update like.

A

Even if you got to the point where you said, okay, every single framework has open Telemetry built in then that doesn't matter because people are gonna, be running five or ten year old versions of spring framework and they're going to need that auto instrumentation plug-in to work. So what can we do to sort of make it easier for those Auto instrumentation.

A

Libraries to get created and maintained, like I, think there's a problem right now with sort of the the breadth of it right.

A

Is there something that we can be doing as maintainers or as people that are involved to sort of ease the maintenance burden of Auto instrumentation.

A

Discuss amongst ourselves.

D

Well, I have a mic so and I just before we move on to the uh the the maintenance problem, the Henrik and I, probably know from the performance and reliability space and then moving on to SLO space that there's always going to be people who are like just do it for me right just make it easy like magic box lists for me and I don't want to know the tension is not what you need to know in order to make that happen.

D

That's what we need to solve for how little you need to know about the thing but the, but also we have to require people to provide the context from their human brain into that right, and so the the next layer on that. What I found works really well from people who are like I just want you to tell me what my performance should be. I just want you to tell me what my SLO value should be is like.

D

Okay, give you a little bit like you're, saying fine, Auto instrumentation, just to get them to the point where then they have to ask that, could they they see the values they see a little bit of value out of it, but then they're like I wish it could do this, and then you can say what did I freaking tell you. You have to provide some context into that. To get that information out right, you have to wet their whistle.

D

You almost have to like stage the question in their mind is why is this not good enough and then and then it becomes, they start to own that problem a little bit more and that's the idealistic way of saying it, not every it's hard to get that going, but I think not trying to solve the 100, for them is actually the way forward right is is solve it just enough where they start going.

D

I need more good. You need more. You need to do this thing. You know so anyway. So then going back to what Austin uh well awesome. A second just.

F

One thing related to the example of the Fortran code: Legacy code code evpf requires a certain version of the kernel. So even if you have Auto instrumentation on evpf, maybe even this company using Legacy code won't be able to use it because it doesn't even run on the right kernel version.

I

A

I mean there's usually some sort of like connector, though right it's something a little more modern, that's actually managing the RPC. You know that's proxying, your rpcs, from whatever your fancy new stuff is into your. You know, less fancy old stuff, so it's not obviously a Flawless thing and like putting taking off my whatever hat and putting on my like Austin's personal opinion, hat I.

A

Think a lot of the driver of this is really almost sales driven in a way from like the vendor side, where we're really just trying to find something find something that is reducing time to Value, right and ebpf seems like the ultimate like. Oh you don't have to do anything to your code. You just run this and boom. You get all this data, you know, maybe that's cynical of me to suggest, like I, do think. There's actually really good applications of it, but it's hard to not view it somewhat cynically, but that's a side.

A

You know that's just an aside.

E

I absolutely agree that there is a huge hype around ebpf and not all the promises will be realized as as usual, with with any hype, I mean we are sitting in a room talking about how to use ebpf with open Telemetry and the majority didn't have a firm understanding of what ebpf is, which is not like negatively in any way.

E

But as a statement of fact, the hype was was what has driven people into this room, largely not that they already have it in use and they already derive value, and not all of this Hive will be realized. But um I I do think that when we, when we have at least this Auto instrumentation things for all the old stuff and yes giving incentives, but it's not about telling people it's about showing actual value, and then they want it. Just telling them is usually not a good way.

E

I think! That's! That's the main promise of all of this.

D

I'm just saying, if you I'm just saying the the point, there is there's only so much that you can build for people, but your real goal is to compel them to change, to act differently. Right.

E

Yes, and so it's.

D

It's like Eric prigler says right: if you ship a report to somebody and they don't do anything of anything different, you have failed, so it's kind of like if, if we provide the the very best, it's got to be just enough, where they're now able and in a mental place where they're asking the right question. You know.

E

Not all of them, I I, fully agree that the majority doesn't care about all of this. um From from their angle, this is just something which is service or infrastructure. Water comes out the wall. You don't care about how electricity comes out of all you, don't care about how and I I think there's a good argument to be made for observability to to also come out of the wall to at least some extent as a baseline part of the problem is uh with all the cloud native and horizontal scalability blah blah blah blah.

E

The fundamental workloads are the basically the same there, the same room for the last 50 or 100 years, the service delineations and which part of containment we put what type of service and what scope of service into how to do. The orchestration who does the actual orchestration? What is the scheduler? All of those things have been broken up and real arranged and rejeggled. So a lot of the things which we used to have with classic boxes and and classic servers have gone away and a lot of this.

E

It's just there and it works, and it just does the right thing for you as a decent Baseline needs to be rebuilt and that's where I see much of the value.

C

So uh one thing I want to clarify is, as far as like the experience I was describing it's more about the work required to collect the information and I I think analyzing, that information analyzing the data, the traces, the metrics.

C

um That's on the um you know, partially on the vendors who Implement solutions that help you generate insights out of it and part of that is a change in attitude right, building an SRA mindset because I'm not going to be the one that can describe what your burn rate should be and what your SLO targets are.

C

That's up to you and your product manager, but I think I want to go back to kind of like where the big burning question for me is: how can we leverage ebpf in Auto instrumentation and what are the constraints that prevent this from doing that and What problems? Will we run into if we like for me right? We um in the Ruby seg.

C

We are rewriting code and constantly dealing with changes in libraries that are getting pulled in, and we don't have that many maintainers to try to deal with that um and the the you know. From my perspective, the virtual machine isn't very amenable to you going in there and making changes to it so that you could add instrumentation at the Virtual Machine level and then we'd have different targets right. We have Jay Ruby and uh C Ruby and who else knows what other Ruby's going to come out in the future?

C

So those are all maintenance problems for our team, which is very small and I'm, trying to figure out a way to provide value or to make data collection, easier, instrumentation, data collection, easier and minimize. The the code that we have to maintain, and so um part of this is my ignorance again about the promises of ebvf and what we can do and what we can't um and the other part of it is kind of. Like uh is again just like me, trying to take workload off myself.

C

Personally speaking at my teammates, so I don't know like for anyone. Who's I asked again like is anybody had an experiment with this trying to use a vpf for auto instrumentation of a language? Does this make no sense at all to do? Should we not even try I, don't know.

A

um One thing that we are currently exploring on the community demo is popping pixie in there since they started otlp export and trying to figure out like what does it look like to have you know like? Is there a way that sort of given what we have now?

A

We can actually make correlatable data between like hotel at the app layer and then ebpf at the networking layer right, and if that you know that's going to be when we figure out their experimentation like you can go, find Carter and ask him what he thinks about it, um but I think I'm. The only one. That's actually open an issue about this so far, um but yeah. The goal honestly is I.

A

I think the only way people are going to understand it is if they have something that like looks and feels like a cloud native app. They can kind of like put their hands on and say. Oh, this is what evpf looks like in production in production right. So here's like a sample app that you can kind of compare and contrast like here's, what I get out of hotel with traces and metrics and logs, and then here's what I'm getting a vpf and it's coming out as a trace.

A

Yes, but these are tracing fundamentally different things at different layers of the stack um is that data useful from an analysis perspective huh like that's for our friends at New, Relic and lightstep and grafana, and everyone else to figure out like anyway. That said um it is half past. So if people want to switch sessions or if you want to stay here and keep talking about this, you know cool, um but I want to give people the opportunity to rotate between the breakouts. If you like,.

A

Great discussion by the way. So if you want to stay and.

E

Continue adding to the board, it probably makes sense to reiterate what the other sessions are.

A

Yeah, the other session is uh intermediate. Open, tracing and context prop. Is that it signal correlation? Yes, I believe you were the one suggested it so. Okay,.