.Net Foundation On .NET, 24 Jan 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Diagnosing ASP.NET Core Performance Issues

Description

In the episode, Mike Rousos shows us how we can use tools like PerfView and PerfCollect to diagnose performance problems in ASP.NET Core applications. He discusses how we can use the tools for performance analysis on both Linux and Windows servers. Also, we get a demo on how to inspect things like large object allocations, chatty SQL calls, and thread pool health.

[T04:10] - Introducing the demo
[T06:03] - Performance Diagnostic tools
[T08:37] - Collecting performance data with PerfCollect
[T16:24] - Inspecting performance data with PerfView
[T25:30] - Common ASP.NET Core Performance Issue
[T28:49] - ASP.NET Core Performance Best Practices

Useful Links

PerfView:
https://github.com/Microsoft/perfview

PerfCollect: https://github.com/dotnet/coreclr/blob/master/Documentation/project-docs/linux-performance-tracing.md

ASP.NET Core performance best practices doc: https://docs.microsoft.com/aspnet/core/performance/performance-best-practices

A

This episode of on.net we're going to be looking at asp.net, core performance challenges, and you can learn about new techniques that you can apply to your app.

A

Hello, welcome to another episode of the on.net show.

A

Today, we're going to be talking to mike russo about.

A

Diagnosing asp.net core performance challenges, particularly cross-platform., so mike,. How about you introduce yourself and then we'll get into the topic.

B

Yeah, absolutely thanks: rich. so,, like rich said, I'm mike russo,, I'm a developer on the devdiv customer engagement, team.

B

We're a part of the engineering organization who works with customers to make sure that they're able to successfully adopt some new devdiv technologies..

B

I specialize in .net core, but also worked with azure, vs, and so on.. We make sure customers get off on the right foot that we bring back what we learned, and make the product better through fixing, bugs, blog post,, documentation,, improvements,, etc.

A

Right. basically,, whatever the customer's need?.

B

Yes,, that's right.. We figure out why this customer is having challenges, and then we make it so other customers don't have the same. Problem.

A

Right. so,: do you work with say one customer at a time or is it multiple?.

B

Typically,, just one at a time. we'll work with one customer from anywhere from a week to a month or more till we work through their particular scenario,.

B

Then we'll take a couple of weeks write up what we found, and drive improvement back into the product, and we'll be off on another one.

A

Wow. so,: is there some bad phone, like I want to work with me?? I have the need.

B

I don't know if there's a bad phone,, but we get leads from a lot of different places..

B

We get it from the field, a lot of times our director, about it from the other folks. In.

B

The engineering team or meet someone at a conference and you'd really come from any number of channels.

A

Okay. so,, let's get to the topic.

A

so,, it sounds like we're going to be the last customers. You've been talking to you been diagnosing asp.net core performance challenges, and I'm hoping you are able to get them on a good path?.

B

We were, yes. so,, tell us.

A

About the specific challenges you've been looking into.

B

Yeah, absolutely. so,, when we look at asp.net core,.

B

One of the things that comes up a lot is that customers don't know how to go about diagnosing issues, in staging and production environments,, because a lot of times that's on linux, it may be in.

B

Docker containers and a lot of the people we work with have .net experience,, but they don't have as much linux. Experience.

B

so, they're, not sure what tooling available there,.

B

And they're not sure how to get started. so,, that's one of the things I thought I could demo.

B

today, I'll show you a little bit how we do that, what tools are out there, then we can also as we're doing that, maybe talk about some of the problems that come up most frequently..

B

Every time we worked with the customer,. It's unique and performance problems are seldom just one root cause,, but there are definitely classes of things that come up more often.

B

so,. It's worth calling out as common things to look out for in your own apps.

A

Right. so,: is it the case that some of these customers just don't know enough about.

A

Linux, to be successful, or is it the case that you should give.

A

Feedback to the asp.net core and .net core teams to say,.

A

Your products aren't as diagnosable as they should be?.

A

What is the forward looking path in this space?.

B

Yeah., I think a big part of it is actually just the documentation.. It's a lot that customers understand how.

B

To use visual studio or other tools to diagnose things out windows, and we have tools for linux.

B

they available,, but up until very recently,.

B

They weren't well documented, or if they weren't documented,, it wasn't some github report. where, if someone like myself tells you, "hey go check out this doc on github, your great.", but without that, you're, not going to find it.

B

so, we've been doing a lot of work recently to actually be flipped, that documentation.

B

so, that someone new to asp.net core as they go through the docs,.

B

We'd be able to find those pointers and I'd get ramped up.

B

Make sense. okay., let's start., let's look at the staff.

B

Yeah. awesome. I'm going to show my screen here.. We can go through a little demo.

A

You're going to give us some performance, challenges, is not it?.

B

Yeah, I've got a sample app. I made effect here,, maybe I'll even start with that. I made an app that demonstrates.

B

Three of the more common problems I see in customer code.

B

Are you able to see this, all right?.

A

Yes. okay., I think we're good.

B

Perfect. now this is a little bit of a contrived demo because I tried to make it short but fit in a lot of different challenges..

B

But basically it's just a simple asp.net core app goes and gets data from sql databases.

B

specifically,, I'm using the adventure works data,.

B

Which is one of our sample datasets., this api here,, which is called test1,.

B

It goes out and it gets all of the products in.

B

The mountain bikes category with this first sql query,, which is this.

B

Then, we call some external service.

B

For each of the mountain bikes to find out whether or not it's in stock.

B

for each of the mountain bikes, that's in stock,, we then do a followup query to get a thumbnail photo for it and ultimately we're returning an md5 hash of the photo of each mountain bike. That's in stock.

B

so,. I get a little bit of a contrived sample,.

B

I don't know anyone who actually needs to return an array of md5 hashes for any stock mountain bikes,, but it's an example of something.

B

Someone might do in a sprint that chord, that showcase of some of the stuff that I think could be problematic.

A

Right. so,, what I hear you saying is the app is contrived, but the specific performance issues that you're running.

A

Into here are not contrived because they're based on actual apps, that you've seen in the field.

A

Yes., that's right., okay, good.

B

Yes., that's exactly right.

B

so,. I've got a few slides here with links that I can go through briefly,, but the thing I want to really focus on here. It's these bottom two,, because people are already familiar with the value of.

B

Having an app performance monitoring with something like app insights, or any number of other options like new, relic or dynatrace,,.

B

That's the place to start., then after that, you would go and use the tools in your ide, like visual studio, 2017.

B

Diagnostic tools, which are super useful during the just go test debug cycle for parody stuff out early.

B

But people know about these already,, where I think we can add. The most value is really letting people know about perfview. If they haven't used it before,, then more than perfview, perfcollect,, which is a script that allows us to gather.

B

Performance information in a cross-platform way so that we can go in and do some deeper diagnostics.

B

So,, if people haven't used them, these are definitely tools worth looking into.

B

also,. I did check it. My tribe sample on github.

B

So,, if you want to go out to mj rousos, the asp.net core perf patterns,, you can get the terrible code for yourself.. It's out there.

B

But the tools that I think are useful are perfview.

B

This has been around perfview years., so, people probably know about perfview,, but it's a performance analysis tool for .net for identifying cpu issues,.

B

Memory issues, and chose utw events all of that.

B

If you haven't used it before,, it's a really great tool to become familiar with. We've got some video tutorials. You can go check. Out.

B

just got to microsoft, perfview., but the challenge that some of the customers I've worked with recently eroded. Into, is that perfview. You only runs on windows.

B

so, when they have a performance problem that only reproduces in their linux staging environment,.

B

They don't know how to start with that.. That's where perfcollect comes in to fill a gap.

B

so, perfcollect is a script that you can download,.

B

Uses curl up down call aka.perfcollect.

B

This script uses native linux, profiling tools like perf and.

B

Ltteng to gather traces that perfview can consume.

B

So, on your linux, machine, you'd hop into your staging environment or even your production environment,, you download perfcollect,, you run it for a minute or so,.

B

Then you take the zip file that it generates, copy it back to your windows. Machine, then use perfview like you're used to.

B

so,. That's the thing that I can demo quickly here. Just to show you what that looks: like., great. so,, let's go ahead and do that.

A

Are some of the customers that you're working with,?

A

Are they doing just that?.

B

Yes. they are and defect good success with it.

B

so,, it's worth knowing about.

B

So, I have deployed that app out to a linux vm app in azure,.

B

And it occurs to me that this text is probably small.

A

But actually it's not that we be typing a few things.

B

The first thing I'm going to do is I'm ssh into that vm.

B

so,, I'm just going to start the app.

B

So, we'll go out to force a spec for perf anti-patterns that combined demo.

B

Now, here I will run the app.

B

Once that app has started up,, we can confirm that it's running by just.

B

Hitting the endpoint that we were looking at earlier.

B

So, I'm doing a curl to the test. 1 api.

B

Usually we get back a bunch of md5 hashes, which in themselves are not interesting but says okay,.

B

This thing is working. so,, that's good.

B

so,. What I want to do now is, I want to run perfcollect and I've already installed perfcollect on this machine,. That's just a matter of copying down the script,.

B

Running perfcollect install to actually install the tools that it uses under the covers, and then a couple of other steps that are well-documented.

B

On that github site that will explain how.

B

To set it up to get symbol, information for .net framework and .net core stuff, if you need that., but that's already installed,, so what we want to do, the one other thing you need to do for perfcollect to be as useful.

B

As it possibly can be, is you do want to have a couple environment variables, set. so,, specifically there's two of them.

B

so,, I'm going to restart my app with a couple environment variables, set. now,, I'm going to set com plus perf map enabled equal to one because that's going to.

B

Instruct the git that gits managed.

B

Method needs to also emit map file.

B

That will allow us to map back from some cpu sampling data to where this was managed. Code.

B

Then,, the second one we're going to set is: we are going to set com plus enable event log to one.

B

That's going to instruct the runtime to admit etw events about stuff like garbage collection,.

B

Threading, jetting and so on.

B

now, setting these is going to slow down perf even more so you don't want to have these set all the time..

B

But if there's a particular problem, you're running into,, you can take one instance of your app or one instance of the container,. Take it down,.

B

Set these environment variables, put it back, up, restart the app, collect some data,.

B

And that undo it all. so,. That's the process that I'm going through here.

A

Now,, you said etw event,, but we're clearly on linux. Here.

B

We are on linux. yes,. I was there not actually etw event here, but the runtime is able to emit the same events in a cross-platform way.

A

So,, they wouldn't really be etw,, but.

B

They're still admitted in a way,, the tools like lttng can pick them up and report on them.

B

Right. yes. good clarification.

B

now, in the real-world,. What I'd probably do here is, I would use visual, studio,, perf and load tests to apply a load to this.

B

I don't know if people listening to this have experience with those tools, but they're pretty nifty,. You can have a whole script of the requests that you want to make and it's pretty sophisticated product.

B

You can run stuff up in azure,, so you can have a bunch of azure vms meeting on your service to apply load., but it takes a few minutes to upload and start that up,.

B

So in the interest of time,, I'm just going to run some load tools from the command line.

B

So, this could be apache bench,, jmeter, anything..

B

In my case, I'm just going to run siege and I'm going to tell siege to.

B

Have 50 users continuously calling this api for one minute.

B

Fifty users,, which isn't even very many but it's a terrible api,.

B

So we'll immediately see problems.

B

So,, I start this and now we've got people hitting the api and in fact, before I even run perf collect,, I'm going to use perf to look at what's happening with the cpu usage on this vm.

B

Like you see, now, .net is up at the top here as you'd expect because it's processing these request..

B

But I noticed immediately that it's got very low, cpu utilization compared to what I would've expected.

B

So,, this is indicative that we're not making good use of the hardware.

B

Maybe threads are blocked or we're doing something.

B

That's not actually useful.

B

so,. That's something we'll want to look at later,, but what I'll do right? Now, is I'm going to run perfcollect.

A

Now,, for those that don't know, top is basically a command line. Version of.

A

Task manager for linux, if you come in front of windows, background.

B

That's right. yeah. good point.

B

so,, that's a little fishy,, but then perfcollect is as easy as just running the perfcollect script, collect,. Then you give the trace a name..

B

You only need to run it for usually 20 or 30 seconds to get some good data. As long as your apps in a state where it's demonstrating the perf problems and then once you're done collecting, you hit "control c.", it does the post-processing,.

B

It takes a minute or two and it gathers all the stuff up into a zip file, which then we can bring back to windows and we can analyze.

B

Meanwhile, over here we see that siege finished up,.

B

It started, so five requests per second,, which is terrible.

B

That's to be expected., while I wait for this,, I can show also. I did earlier,, let's see where is it.

A

What tool do you use to move those zips around,?

A

I would use scp.

B

That's exactly what I use, scp,, it's just simple command line tool for doing it.

A

I have to admit,, I really love that tool.

B

It's super useful.

B

yeah,. It's very handy.

B

Here it is. so,, I was going to show the output from some load tests as well.

B

so,. I mean, I'm using siege here,, but if we had 10 more minutes, I'd be showing you vs load test, because you can get information.

B

About throughput over time.

B

Actually, this is an interesting chart because you can see that our throughput with my api was very spiky,, which is another interesting symptom because it means.

B

That periodically something's happening.

B

That's causing everything to slow down or halt,, and then it's okay for a while and it slows down again.

B

so,. This is another useful tool to look into,.

B

It's just visual studio, perf and load tests and the azure devops integration. There.

B

The interesting thing about this is it's just a zip file,.

B

So I can unzip it and I mean, you wouldn't normally do it because you don't have to,, but just in the interest of demoing. What's in here,, we can't step into the contents of that zip file.

B

If we look at what's here,, you can see, we've got some perf.data files,, which are the stuff that the linux native tool perf will gather for us.

B

We also have these map files that I was talking about, that allow us to correlate information in that perf trace back to managed code.

B

This is all stuff, that's understood by native linux tools as well.

B

No,, I'm not super familiar with them,, so I'm not going do much debugging in it,. But if you are familiar with things like perf,, you can just run perfreport on.

B

This same data that we're going to use in perfview and it's able to understand it as well.

B

So,, you can come in here just for the command line and start looking at where's time being spent,.

B

So on and so forth. this console's too little,, but you can start to see interesting things that are taking time and so on.

B

so, just interesting to note that that works, that way.

B

Very nice., I'm sure, there's some people who.

A

Are super happy watching that.

B

Yeah. I'm already familiar,, I could take advantage of it and it all works with .net core. The way you would want it to,, which is great.

B

So, I'm going to go ahead and start up perfview and we can look at that. Briefly.

B

I'll try keep this quick,. I know these demos aren't supposed to be too long..

B

This one gets a little lengthy,, but what I'm going to do, because I actually already copied something over with scp earlier,.

B

It's the same as what I just dab,, but rather than waiting for a 100 megs or whatever, to copy to my machine,.

B

I've already got one here. We can look at., I'm just going to briefly go through a few things in here that are interesting.

B

so,. The first one is when you open up perfview,, there's going to be a section here for cpu stacks..

B

This is just sampling information,. It shows where time was spent in your application,.

B

It shows the call stacks. you actually get like a flame graph which in our case I'm going.

B

To quickly drill into just the process, we care about.

B

In this case, I don't think that flame graph is super useful,, but in other scenarios that can be helpful., but what it allows you to see is you get both exclusive and inclusive information.

B

About how much time was spent in each one of these stack, frames.

B

Exclusive information is how much time was spent specifically at this frame,, whereas inclusive is how much time was spent in this frame or in something it called.

B

so,. You can either do a bottom-up approach,, where you sort based on exclusive time, and you start looking at what was taking time and how did we get there,.

B

Or you do more of an inclusive analysis where you start from.

B

The top and work down to see where time is being spent.

B

Again, perfview is that complex tool,.

B

So I'm not going to get into all the details now,, but we've got a lot of videos online. For someone who wants to learn about it., I encourage people to go check those out. in this case,. I just sort by stack, frame's, inclusive, time,,.

B

At the top, you see a bunch of asp.net core mvc stuff, which is to be expected, because every request is coming in through your mvc pipelines..

B

It's going through your middleware, through mvc routing, model binding,. All of that stuff.

B

So,, that's always going to be up at the top here., but as we get down, we'll look for where our app starts to show up,, we can say, "okay, 60 percent of the time when we sampled,.

B

We were in this, get in stock.mail. Hash is one api,.

B

Which makes perfect sense because that's the controller action for the one api we were calling in our load tests,.

B

So we expect to see this on here., but we can go to the "call this" tab to see what this thing was calling and where it's spending time..

B

One of the places where we see it's spending time is.

B

In system data sql client., we can drill in here,. We can see it's spending a lot of time. Getting connections from.

B

The db connection pool and so there's something going on right, now. just seeing sql connections is not necessarily.

B

A red flag because that's commonly a slow part of an asp.net core app,, but seeing it so much at the top of the list, makes us want to go, look and see, well,. How are we using sql??

B

Is there something that could be optimized, there? so,? This is where you might hop over into azure and look,, I'm using azure sql,.

B

So I can get some telemetry there about what sql stuff we're doing.

B

We can start to notice interesting patterns, like, hey,, there's some query that we're calling 11,000 times in the past minute or two.

B

That suspicious. so,, you can come in here and see what that query. Is, select thumbnail photo from product where product id is whatever,, then we realize, yeah, we're doing something down in our app here that we really shouldn't be doing.

B

after we have the list of the mountain bikes that we care about and we've filtered it based on. What's in stock, we're making a separate tql query for each one of those products to get its thumbnail, photo.

B

so,, that's pretty inefficient.! This is that query that we saw coming up 11,000 times in azure.

B

so,. This is something,.

B

We've got the sql query inside of a for-each loop,, which I've never actually seen in a customer app,.

B

But it's a simple way of demonstrating something that I do see: a lot, which is apis and their extra core apps, that are hitting sql databases, remote services,.

B

Whatever, multiple times per request.

B

In some cases, I see someone making a request and it results in 10 or 20 different calls to back and stuff,.

B

And this is where your time is often spent [inaudible] and the core apps.

B

So you want to really minimize that,, but this can be minimized and I've got a fixed version of this api down here by having a different query.

B

Okay, we're going to do one query for all of those products,.

B

Instead of querying them one at a time. other things you can do here, is, you could cache..

B

If there's some data you're getting from a database or a remote service,, that's acceptable to be stale by say one minute,.

B

You can cache it for a minute and now that's going to save you a lot, because you're avoiding these slow, remote, calls.

B

So,, that's one type of thing. We would see here.

B

The other thing I would look at in "perf url,".

B

I'll keep this moving here,, but we've got this "events tab,", where you see those events that we talked about: earlier, etw events on windows,- I don't know why they're called linux, but they're events, and there's interesting stuff here about the gc thread pool et cetera.

B

One that you always want to look at it in an asp.net core app is the thread pool health,, because asp.net core apps are very multithreaded..

B

You expect to have a small number of threads, handling, hundreds or thousands of concurrent requests.

B

so,. If I look at this particular event,, the thread will work on "threads start event,".

B

This is going to show me every time a thread was added to the thread: pool. well,. I see that happening pretty regularly, every second on the second,, we're adding a new thread to the thread. Pool..

B

The thread pool can only add threads about once a second, because of the way how its algorithms work that that's the way it functions.

B

so,. This tells me that, over the course of the 50 seconds, or so that we were profiling, this trace,.

B

We were expanding the thread pool as fast as we could, and that's generally, a bad sign. that can bind with what we saw earlier,, where we weren't using a lot of the cpu on. My vm, tells me that we are probably starving the thread. Pool.

B

So,, these threads are busy waiting for something that they're not able to handle new incoming requests,, and we don't have threads available to take care of them..

B

The very common cause of this is not making things asynchronous,.

B

You have blocking calls that could be non-blocking.

B

so, if we come back to our app, sure enough,.

B

You notice this. This api here is not async,.

B

I'm making sql calls, I'm connecting the database, executing the reader and stuff, not in an async way,.

B

When I look to see whether products are in stock or not,, I'm calling an api called, product is in stock async, and then I'll call data result on it,.

B

That's a red flag,, anytime, you're calling data result result or data wait on a task,, that's something that could have been done in an async manner,, leaving that thread free to handle a different, request, and instead,.

B

This thread sitting here doing nothing, until we hear back from some remote service,, which could potentially be slow.

B

so,. That's what you definitely want to look for.

B

I then the last one briefly is along with those ones in path,.

B

We also have this memory group, where you can get gc starts..

B

This will show when the garbage collector ran,.

B

Why it ran, et cetera- and this is worth looking at.

B

a lot of times, people- will start engagements asking us,.

B

"Hey is it a gc problem?" and it's often not,, and so you don't start there unless you have data,, but sometimes it is,, and so it's worth looking here to find out, and in our case, we profiled for about 50 seconds,.

B

We had 114 gcs and 92 of them were gentoo gcs,.

B

Which is bad, because gentoo gcs are slow,.

B

They require pausing the application,.

B

In fact, you can see that we were paused 17 percent of the time the app was profiled,.

B

It was paused for garbage collection,, so we weren't processing any requests.

B

this lines. Up with that real spiky performance, we saw earlier, where things are okay for a moment,, then they're terrible,, then they're okay, get those terrible times when we were doing a gentoo gc,.

B

And if we scroll down,, we can see why we're doing these gcs,.

B

In this case it was all allocating large, objects,, large objects or anything bigger than 85,000 bytes,, but typically that's an array, and they're allocated on the large-object-heap, and when they're cleaned up,, they have to be cleaned up with the gentoo garbage collections.

B

so,. The think you want to look out for here is,. If you see a lot of gentoo gcs happening,.

B

You may be allocating more large objects than you really need. To.

B

In our case,, the issue was, we're getting these thumbnail photos out of sql, they're, potentially large,. So when we read them,, we are allocating 100,000 element, byte array to.

B

Read these thumbnail photos and we're allocating a new array, each time,.

B

We're then immediately throwing it away, and so the garbage collectors just go and that's trying to clean all these things. Up. again, caching comes to the rescue here, another option. If it's not something that makes sense to cache,, because it's not used frequently enough would be to use array pool of p,.

B

This is a relatively new api,. So a lot of people don't know about it,, but with array pool you can rent a large array,.

B

Use it for whatever you need it for, and then return it to the pool and it doesn't have to get cleaned up because the next time you need a large array, like the very next time. Someone makes this request,.

B

You could request an array,, get that array, back, use it again and you don't have to keep allocating and deallocating large objects.

B

so,. That can be super useful as well..

B

So that's, basically, the demo.

B

again, there's a lot more. That can be done with perfview and perfcollect,.

B

But I just wanted to kind of cover some of these., I'm just going to recap., make sure you're not blocking calls that could've been asynchronous. got a great blogpost here by vance about why that's so bad and how to avoid it.

B

lookout for "chatty" interactions., making calls to databases or other services is slow,.

B

Do it as seldom as you can, and do it in as efficient ways you can, don't get extra data? If you don't need to. again, we've got caching apis. That can help with this,.

B

And then, finally, just look out for memory issues,.

B

If you're allocating a lot,, especially if you're allocating objects that may be more than 85,000 bytes,, that's something you want to try to be really careful with it.

B

We've got array pool to help with that,. Also, if you're working with strings or larger arrays,, you could use span of t,, it's going to record tool that want to help,.

B

You can even stackalloc short-lived things, starting in the sharps,.

B

7.2 and then add its functionality to 7.3.

B

So, anyhow, that's kind of a quick demo.

B

Of some of the more common issues we see, and hopefully it's helpful to people to know sort of what to look out for and how to diagnose stuff.

A

It's very helpful. so,. Is there a good news story, though,.

A

To your particular app you [inaudible].

B

Yeah., it is actually so. worst case so.. I called this api that was terrible,.

B

I called it test one. in tests two, at the same api, but I started using array pool.

B

In test three,, I took tests two, but I now made the whole thing asynchronous, and then at test four,, I consolidated it down to just two sql queries. Instead of many,.

B

Which two is actually still more than I'd like to see,, but it's way better than like four, that we were doing before.

B

In fact, I ran a load test with that as well.

B

so,, whereas before, we had an average response time of four seconds,.

B

Which is kind of laughable- and this is very often on performance,.

B

Once I run the thing against test, four,.

B

I ran it with 500 users instead of 200,.

B

But despite having 500 users,, our average response time is down under 50 milliseconds and the chart is exactly what you want to see.

B

as more users. Arrive, throughput goes up and it never really varies from that increasing load., so, yes,, there's, there's good news and the good news is.

B

That just fixing up these few things makes a huge difference in this sample. App.

B

of course,, like I said,. This is not a real app, but the problems it's demonstrating have really showed up,.

B

I mean, even just to kind of briefly go through examples.

B

There are places where we have seen: customers flushing appinsights every time they log something., when you flush appinsights, that involves taking a lock,.

B

That's another version of the blocking problem.

B

the thing where we're getting images, from a database,, that's real,: we've had people, get images from databases and convert them to base64 strings and then throw them away.

B

Things like this do come up,, and these sorts of fixes do make a big difference.

A

Right., so what I hear you saying is, these fixes are actually not hard, meaning.

A

Yeah., you don't have to be a rocket scientist,.

A

You just basically need to read the right. Guidance, apply the right pattern or technique,.

A

And then you can really have a much better performing app,.

A

Then you had the four. exactly., because that was like 10 x, improvement on [inaudible].

A

B

Yeah. yeah., it's a big difference.

A

Okay. so any kind of closing statements that you'd like to make, before we finish up for today.

B

Yeah., I think, we've pretty much covered., I suppose in closing. I can point out that we.

B

Have actually been trying to capture a lot of this in documentation., it's a place where, in the past we haven't had docs and that's why customers have struggled,, but we've taken a lot of the same things. We've learned that I've discussed in this conversation,.

B

And we've put them up just recently on our asp.net core docs sites,.

B

You can go out and check out our performance overview..

B

We go over a lot of the same stuff along with some other ones too. That maybe were tier queue, issues that we didn't have time for here, but are still interesting to know. About,, don't use exceptions for code, flow,, minify stuff that you're delivering to clients, pool http clients,, all sorts of stuff.

B

so,. You go check this out and hopefully it's a useful resource.

A

Okay. thanks. well,: this was awesome.

A

being able to get effectively battle.

A

Tested customer-related feedback on how to deal with performance, challenges.

A

so,, thanks for being on the show mike.

B

Yeah., my pleasure., I'm happy. You guys had me on them.

B

Appreciate the opportunity.

A

For sure. well,, maybe you know in like six months or a year or something like that,, we'll do v two of this, and see-.

B

Yeah,, that would be great.

A

What other things you found. okay, thanks so much for watching another episode of on.net.