.Net Foundation On .NET, 16 Apr 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Introducing the Diagnostics Client Library for .NET Core

Description

In this episode, we’re joined by Software Engineer Sung Yoon Whang from the .NET Runtime team. He’s here to talk to us about the diagnostics client .NET Core. Using the library, you can communicate with CoreCLR to gather diagnostics dumps from your running application.

[01:31] - What is the library for?
[03:26] - Creating a custom runtime event tracer - demo
[14:34] - Capturing traces from an .NET Core application
[17:58] - Dumping traces to a file
[20:14] - Who are using these tools?
[21:20] - Using PerfView to inspect the dumps
[23:49] - Where to learn more?

.NET Core runtime diagnostic tools Github repository
github.com/dotnet/diagnostics

Microsoft.Diagnostics.NETCore.Client API Documentation
https://github.com/dotnet/diagnostics/blob/master/documentation/diagnostics-client-library-instructions.md

Run-time configuration options for debugging and profiling
https://docs.microsoft.com/dotnet/core/run-time-config/debugging-profiling?WT.mc_id=dotnet-c9-cephilli

A

Hey we're here with sung and we're talking about the Diagnostics library, for.net core come check it out.

A

Hey welcome to another episode of the on dotnet show. Today we're going to be talking about the dotnet Diagnostics client, with song, so yeah. How about you tell us a little bit about yourself and then we'll get into the topic. Yeah.

B

My name is sung I'm, a developer on the dotnet runtime team, I work on diagnostic support for the runtime and the PCL library, so system Diagnostics tracing some of the dotnet CLI tools liked on that trace and tonic counters yeah, like that.

A

Okay, so today we're gonna be talking about one of your packages, mmm-hmm yeah, and so what what is it actually called? It's.

B

Called Microsoft Diagnostics that query client, okay,.

A

How do we go to the nougat page so that people can like okay, there yeah there? It is so it's Microsoft Diagnostics net, core client, okay right! So it's the Diagnostics client for dotnet core from Microsoft.

B

A

Okay, so it looks like it's still in preview, mm-hmm.

B

Well, no, it's not the version 1.0 yet, but it's out of previews. So.

A

It's yes, okay, those people.

B

Interesting choice: yeah.

A

Okay, so how about you tell us what this is for.

B

So this client library is for you to talk to the runtime using the Diagnostics IPC protocol. So it allows you to communicate with core CLR for various diagnostics purposes like starting at recession or taking out core dump or attaching an eye core profiler right.

A

So this is for an out of process scenario: yep.

B

A

So IPC yeah, that's like inter process communication is what that all means. You.

B

A

So we have some in process scenarios, but you wouldn't use this for those right.

B

This is specifically for out of pocket.

A

Diagnostics traces right, but it's out of proc on the same machine like if you have some off machines. Do you use this too or.

B

um Let's see really good questions, so we have plans to make it available for container scenarios. So even if you're on different machines, you should be able to like use this library in the future. For now, this isn't just on the same machine. Okay,.

A

And when you say container scenarios, do you mean like sidecars or do you mean solving different insight? Cars okay, makes total sense. Okay, we kind of framed this and I guess. The other thing is, you know. I know, I know some of the details here since we work on the same team, mm-hmm.

B

A

So you mentioned these, you know dotnet core tools that you've built like dotnet dumped and done it collect, and things like that. mhm From what I understand you had this private, the code that you know I mean it's not private and like it's open source, but it's was still a private implementation between those tools that you've built. You basically pulled out that code and made a first-class client library that both your own tool tools build, but other people can build tools on and I guess. That's really.

A

What we're gonna be talking about today is how people would build their own diagnostic tools on top of this cool client library and how it's way easier because of this mmm-hmm awesome. Okay, so do you have a demo or a tool that you've built yeah.

B

So I have a very simple, looking custom tracer here that will be going through today.

B

So one of the things that this library supports is starting a tracing session. So say you have your own eventing mechanism in your code. So writing your own event, source or say you want to consume any runtime events. Then you could do that by starting a tray session. So this will kind of show you that and have in here a vent pipe provider. So this is using vent pipe, which is a serialization mechanism for you to emit events out of your runtime.

A

Right, but the one this since this is Microsoft Windows done at runtime.

B

A

Is these are the ones that are in box the involved.

B

Events exactly so, this is the runtime events, yeah.

A

And also, why does it say Windows on there are those.

B

Yeah, that's a remnant from the dotnet framework. These this provider existed from like Don, that I.

A

See so if you were, if you were using, would you run the same code on Linux like.

B

You would say the same thing: okay,.

A

Have we thought about removing the windows name.

B

Where's that hard that might break that compatibility. Maybe we could have an alias.

A

B

We can think about that, but yeah. This is the runtime provider, okay, yeah and then you can specify sort of like the event level. So I can kind of like just write this from scratch. To like show you what writing this with look like, so you can specify in here the name of the provider, and then you can specify like what level you want it to be. So there's all these kinds of different like event level, that's accessible.

B

So, for example, if you want to make it like informational, then you can, which is that the key words specify what kind of events you want to filter for so let's say like in the runtime there's a bunch of events. Like you see the jet, the loader say you only want like the GC events, for example, then you can specify to have just the GC events right.

A

I know, and it's not a performance win having like specifying just the ones that you care about right.

B

That's how you would filter for just events yeah, but so that no additional payloads are getting created by the runtime. Okay.

A

So why do we have that cast they're like why? Why is it the case that so that's I assume that's an enum to see our events parser or.

B

No earth, that's a property. This is GC like this. Isn't you know yeah.

A

That's an enum yeah, so why does it return the wrong value for what this thing expects.

B

So the portal call expects this to be a 64-bit number. Yes, it's a lot. It's a long type. This is kind of cheating, because I already know like GC has keyword one so like this is.

A

B

It's not inside the library we're talking about today, Oh defined in trace event, so the parser for these events is what finds these keywords.

A

Okay, okay sounds like we got some API design left. You.

B

Have to do in this space yep. This will eventually have to like somehow get flushed out. If some issues with this keywords thing the VCL library there's some badly designed keywords hard-coded in there. Okay.

A

Well, let's remove let's not spend time on that I'm, probably not the only person that thinking that looks a little odd yeah.

B

Okay, yeah, but that's how you would specify like sort of the providers. You want to listen for okay and you can specify more than one provider but we'll leave it here for now. Yeah make sense and you would need to create a client.

B

So this is the client that would talk to the diagnostic server that lives inside the runtime yeah. So you can specify using process.

A

Id right and you're obviously passing that process ID to this tool. From from the outset,.

B

Yeah so that I can you know just likes there any process and sort of like attached to that yeah and then I will start an event pipe session, so that takes in a parameter which specifies which providers you want to listen for then, this false is basically a whether you want to request rundown events.

B

This is another like implementation, detail kind of thing, but this event pipe will basically give you additional metadata at in the possession and.

A

B

What would that look like that? Meditator contains various things like ready-to-run symbols, so without run down line, you can't resolve framework symbols. Actually, um so the default is set to true, because if you don't know what you're doing just set it to true, but in our case we're just going to be printing out these events happening in real time, so we won't care about the run. Rundown events, okay, full set it to false. Okay sounds good and the next parameter is circular buffer size.

B

So this is sort of the buffer that a runtime uses to store the events.

A

In this, this is a buffer in memory, yeah.

B

Yeah yeah: this is in memory yeah of the target process.

A

B

A

Because we're we're basically, this whole thing is: we are using the Diagnostics client we're gonna, be asking the process to do additional work right on our behalf. Yep yeah and.

B

The process will then allocate some amount of buffer for it to store the events in yeah.

A

And I would assume that, when it's doing ass additional work on our behalf, it's just doing this on it's not creating new threads to do this. It's just doing this on whatever.

B

Existing threads, but it's using right, so each thread will have its own buffer yeah.

A

Okay, that's good to know that that makes sense mm-hmm.

B

And once you start this session, that's going to give you back a event, type client event, pipe session object and that session object represents a single event pipe session. So you can do things with it. There are a couple of things that are exposed via this class.

B

One of them is the event stream, and so this is the stream object that the event pipe um tracing session will emit difference to okay, so here I'm using a class called the vent pipe event, source which is defined in trace event, library, which is not what I'm here to talk about, but because this library is what you use to parse the events sort of have to use it. If you want a print events in real time, so it's basically the DC or Eliezer yep.

B

This is the DC réaliser for what gets utilized from the runtime right.

A

And so you and I were actually talking about this earlier.

B

A

You only need that deserialize err for the out of proxxon Aereo's, if you're using an event pipe in proc, then you don't need all of that stuff, because you can get them pre-sterilized right in their raw form and really.

B

A

Need any of that.

B

um Yeah, you would just simply use event list in their API, so, instead of like creating this, it's much easier.

A

B

Use event listener; okay, that's what it was designed for. It.

A

Makes sense: okay, well, let's try and quickly get through the rest of this and then I actually see that the demo in.

B

A

B

Yeah, so this is also trace event stuff. It allows you to add a callback on every CL or event that happens so the source not CL art at all, I see.

A

So that's a thing that role allows to print as events comes in.

B

Opposed to waiting until then yeah, so the so like console.writeline on every.

A

Event, yeah who's that it literally says is that right there yeah so.

B

Source that process will then like start this session and yeah. We can sort of see.

A

Okay, so that's that's. Basically the flow yeah.

B

A

Meant that all that's we're actually really simple: yeah.

B

That's it: um that's all you have to do to like get a life print of all those CL events. Okay, what's.

A

What can you can you show us yeah so.

B

B

This is just Web API yeah.

A

So running so we're gonna start out.

B

Well, start time and.

B

Set another command.

B

B

A

Taking a very long time to start.

B

Yeah I wonder why oh there's some coat that I meant to comment.

B

You just caught this yeah okay,.

A

A

That's that's much better.

B

Okay, so we have that running and after that.

A

B

A

Yeah, this is our tool, so now we don't have the pit yet right, yeah, so I will.

B

Get the pig okay.

B

So this PS also has some code in here. Like that part, we have a get published processes API in the client. It's a static method that you can use to get all the process IDs that are running compatible done in process. Oh I see so it's already filtered down right, so um it'll filter, like all the processes that have the diagnostic server running. Okay,.

A

So that all makes sense now, let's, let's see this in action, so what kind of beer right we already have a web app running yeah. So let's get that going yeah and then this is this is a Web API. So gonna have to hit a particular endpoint in the browser yeah. So do you? Do you have one pre-selected from.

B

Yes, so I have a endpoint that I wrote for it that will leak okay purpose, so we can see some GC is happening. Okay,.

A

And then also, we need to start the diagnostic leader tool right, the one that you showed us, but presumably we need to find out the pit somehow yep.

B

So I have a PS command in here that will point out, although compatible runtimes, so right.

A

So there's a subset of processes on the machine. It's all the that's all the dotnet core processes that have the diagnostic server running yep.

B

So anything above 300k, including.

A

3O, including 300k yep, three, oh and above yeah, make sense so there's another reason for people to upgrade to two three one: mm-hmm.

B

A

Get this scenario.

B

Right so a lot of diagnostic support was added from 300k.

A

Okay yeah, so it's on it run and then it's uh yeah 12 5 9 6.

B

So that's 30 session. Ok,.

A

So now we're going to go. Do the thing now.

B

We just need to leak some memory for guys used to kick in.

B

So the number that I'm gonna give is how much how much bytes own allocate so guys say: I, give it like 2,000 that should have leaked some memory and I can see some GC events that happened so you'll see like GC was triggered because it went out of his space. Then it suspended the EE. Then an.

A

Ee on this case means execution and.

B

The execution is in the runtime most of the runtime had to stop and then GC started and started marking things and once it was done marking it printed some heap history stuff, then, did you see finished and it printed some heap stats. Then the EU was 38 again and finalizer is kicked in okay, so we can see just events happening.

A

Now, with each one of these events, is there more information, that's available.

B

If we wanted it mm-hmm, so this is just the currently we're just printing the names of okay, um these things, so we can see that we're just printing the event name I see, but we could theoretically print all the page. Okay,.

A

Right, cuz in and of itself I think there's very few. My developers that are going to find this yeah I find that very.

B

A

Other than just learning about the runtime behavior sure.

B

You can definitely like convert these into actual teresa's and feed them and perfu, something that I said, though, give you more information about. Each of these events right.

A

So imagine you did want to load these traces in a separate tool. Is there a simpler because you could just take all this information and save it, but is there a simpler way to just say you know, I want to I want to create this trace I want it saved to a file so that I can open it in another tool. Yep.

B

You can do that. I can actually show you how to do that. That's the next demo I'll capture is tracing first or intuition, so yeah we have summer looking code here, pretty much identical, except now we have something called a sample profiler as another event pie provider. That's added to our request- and this is a CPU sampler in the runtime, so it'll take managed snapshot of your stack and do that. Every like 10 milliseconds.

A

I think you really mean a snapshot of your manage not to have you manage. Thank.

B

You yeah okay, so that provider will get turned on. If you specify.

A

This okay, that sounds useful I.

B

A

Pretty much read the code when it's what's doing, you're.

B

A

Just yeah writing all this stuff to a file mm-hmm. Okay, so are you gonna show us that yep that demo.

B

Hey quit out of that and instead of print event, what I used here is I think trace yep trace at the same paid. Now we just have to specify the duration. Ep will give it 30 seconds. Okay,.

B

It's okay, I, have a Oh oops.

B

Index off by one okay, apparently.

A

This tool hasn't gone to production just yet. Oh of course,.

B

I just wrote this this morning: okay,.

B

Okay, so this should be like at the end point again, so let's keep make some Jessi's happen. Yeah.

A

So, while we're waiting for that, do you either I guess two questions? One is: are there any people that you're working with? Maybe that actually built you know for pay diagnostics, tools or open source diagnostic tools that are, you know, are using these api's yeah.

B

So my team currently builds for not necessarily you.

A

Know but I mean people not on your team, not on my team, yeah yeah.

B

A

Visual studio use these in any places is.

B

Planning to use this for their Linux scenarios- okay, so right now, they're on utw, but once they moved to once, they enable the Linux scenarios. They're put on profilers like using the substitute connect to Linux, provide.

A

Yeah I know there's this one company who I won't name that we're working with like it's.

B

A

A user of dotnet core that I don't sell diagnostic tools or anything.

B

A

They made a pretty heavy use of a vent pipe for actually for Linux in areas as.

B

A

Okay, so was that done.

B

Yep that should done it. There is trace in here. Okay,.

A

It looks like it's three megabytes yep.

B

So we can use perfu, which is a tool I used to view these files yeah.

A

This open source on github yep.

B

A

B

Can download your own github? You can build it yourself, yeah, whatever.

A

B

B

And there is that file. We can open that up and first we can just check if 30 events so.

A

It looks, it looks, looks like there are mm-hmm.

B

So if you wanted to look at like sort of some of the events that we saw earlier, you can just sort of like to see that kind of stuff, and this shows.

A

Like or right how much time we spent to.

B

Information yeah and like when it's 30 like timestamps, whether like other metadata about the event so like white, this GC was triggered and.

A

Like right, so I assume some event types actually are you can print pretty pictures with them like? This is all just um yeah.

B

A

Information yeah.

B

This is just pure information, but theoretically you could go to like a memory group and then this would show you like different stats about like GZ, for example. This does a special person on just the GC events. Okay, like prints this kind of like it's that's on it, okay,.

A

So yeah some of them are yeah easier to view.

B

And you can look at threat time as well, um some of missing symbols, but this is you see what took place during that 30 seconds and it'll show I, always.

A

Get right so now we're kind of more capable of doing an actual investigation right.

B

Before an investigation but yeah so right now there aren't that much happening because we're just triggering a lot of GC. So most of the time is spent in unmanaged code time. Yeah.

A

So understood that that's kind of particular to the scenario right.

B

But theoretically um it should print like which manage the code is spending most time. Most of your time makes.

A

Sense: okay! Well, this was awesome. So where would you suggest people go to get started so.

B

We have a documentation on our diagnostic sweep, oh okay, how about we shall we show people.

A

Where that is yeah.

B

So this is where my team spends most of our day and you can go to the documentation and there's a API documentation for this library, and here I have some sample code as well that you can just take and compile, and they should work if not file an issue, if not yeah following this June I'll, probably get back to you within minutes.

A

Okay awesome well. This is like a whole new scenario that we really never had before and like I said before. This is another reason to update to the latest version of.net course.

B

A

You can, you know, use tools that you find that use this client library or build your own, and you know basically be a bit more kind of like what's the word like self self directed self help for diagnostic diagnosing. Some of your performance issues mmm awesome okay. Well, this has been another episode of the Anya and Annette show, and so today we talked about the diagnostic client library, for.net core thanks. Thank you.

A