National Energy Research Scientific Computing Center (NERSC) HPCToolkit Training for NERSC and OLCF Users, Mar-Apr 2021, 6 Apr 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: 3 - Analyzing GPU-accelerated Applications

Description

Part of the Using HPCToolkit to Measure and Analyze the Performance of GPU-accelerated Applications Tutorial, Mar-Apr 2021. Slides available at https://www.nersc.gov/users/training/events/hpctoolkit-for-gpu-tutorial-mar-apr-2021/

A

So thanks helen. thanks for introduction., my name's keren zhou.. I am currently a graduate student at rice, university.

A

Today, I'm going to briefly introduce you to how we can use hpctoolkit to analyze, gpu-accelerated applications., so without further ado, let's get started.

A

since the time for today's talk is just an hour and we have to finish two talks,. I will speed it up a little bit., so today's outline is like this. at first,. I will just review the whole workflow.

A

For using hpctoolkit to profile, gpu applications.

A

Then I'll show you a few live demos for a few tutorial, examples.

A

I'll go through the profiling steps..

A

Next, I'm going to demonstrate how hpctoolkit can benefit gpu-accelerated applications.

A

By two real case, studies, for two applications, where we collaborated with local teams.

A

At nersc. in the end,, I'm going to summarize hpctoolkit and point out a few caveats when you're using hpctoolkit.

A

So as john said, hpctoolkit is a tool that supports calling context sensitive profiling for gpu-accelerated applications.

A

on the cpu side,, we unwind call stacks.

A

And for gpu apis,, we unwind at each gpu api call.

A

for gpu calling context,. We don't unwind at run time, since we don't actually have an api and we don't actually instrument gpu binaries.

A

So we reconstruct your calling context: offline, by analyzing, gpu functions' call graphs.

A

Hpctoolkit offers two kinds of views..

A

The first view is a trace view,, with which you can see a series of events.

A

That happen over time on each process,, thread, and gpu stream.

A

in hpctoolkit's profile view,, you can see a correlation of gpu performance, metrics.

A

With full program contexts that span both cpu and gpu.

A

So to use hpctoolkit,, for now we recommend you compile your application with cuda version, less than cuda 11.2.

A

Since we don't have full support for interpreting the line, information in cuda 11.2,.

A

Any version greater than cuda 11 is highly recommended.

A

On cori,, you can load the cgpu module and load hpctoolkit. on summit,. You can just load by default. The "hpctoolkit" module.

A

So here's the workflow of hpctoolkit, when you are using it to profile gpu applications.

A

at the start,. We can compile our applications with "-g" to generate line information.

A

On the target side., if you're using an nvidia gpu,, you probably also want to pass "-lineinfo" and ensure that the line info.

A

For the gpu binaries are also compiled. In.

A

Next, after the application is compiled,, we've got some optimized binaries and executables.

A

We can use hpcrun to collect call path, profiles.

A

Of events of interest., so here are some tips for using hpcrun..

A

It can be used to measure both cpu and gpu activities.

A

To enable gpu profiling,, you can use "-e gpu =". Whatever is the vendor's name.

A

For intel gpus,, currently, the support is a little bit complicated.

A

You need to point out the program environment. You are using,.

A

Like opencl or level0., but for nvidia and amd gpus, you can just pass their gpu names.

A

To enable gpu tracing,, in addition to "gpu=nvidia",, you also pass "-t" to hpcrun.

A

For pc sampling,, which is currently available on nvidia gpus only,,.

A

You can use "-e gpu, =, nvidia, pc".

A

So, as a caveat,, you need to pass "-t" for the current installations on summit and cori.

A

To sidestep one weird problem in hpcprof.

A

But we shouldn't do that, and we'll fix that in our next release.

A

If you want to see both cpu and gpu profiling, results,.

A

Then you pass both cpu and gpu events.

A

for example,. If you want to see the real time, event, you'll pass, "-e realtime".

A

Along with "-e gpu=nvidia".

A

So to use hpcrun with job launchers,.

A

Like jsrun or srun,, you can just pass the job launcher's command as a prefix to hpcrun., so at runtime,, by default, hpctoolkit will dump the measurement data to something like "hpctoolkit.

A

Dash application name dash measurements, something".

A

If you want to specify your own output, directory,.

A

You can use the "-o" option., so, in addition to gpu events,, we actually support hundreds of cpu events,, including some papi and perf events., to look at all the supported events,. You can use "hpcrun -l".

A

Next,, after getting the measurement data and dumping gpu binaries,, we will use hpcstruct to recover program structures.

A

About lines, loops, and inline functions.

A

So if you want to analyze a cpu binary, you just pass the executable to hpcstruct.

A

If you want to analyze all gpu binaries,, which are dumped at runtime in the measurement directory,.

A

You use hpcstruct, plus the measurement directory.

A

In addition,, if you want to recover gpu control flow graphs,.

A

So that you can get loop structures and device calling contexts you pass "--gpucfg yes" to hpcstruct. by default, we don't recover control flow graphs due to the limitation that nvdisasm needs to.

A

Take a long time for large gpu binaries.

A

To accelerate hpcstruct, you can use "-j" to control the number of threads.

A

There's also another knob to control the parallelism level.

A

So let's say you want to use hpcstruct and analyze.

A

A gpu binary in parallel., you can point out the lower bound size,, which is set to parse.

A

The gpu binary in parallel., if you set it to zero,, then any gpu binary greater than zero.

A

Will be parsed in parallel.

A

Okay, so now we have got the static program structure.

A

And the run time profile data.

A

next,. We want to use hpcprof to combine profiles.

A

From multiple threads and correlate metrics to static.

A

And dynamic program, structure.

A

Hpcprof is relatively simple., you can see. There are not too many options to use. Hpcprof.

A

As a single process to combine performance data.

A

You can just pass "-s" plus.

A

The application's struct.

A

And the measurement directory.

A

by default, we also have a default database name.

A

If you want to specify your own database name, you can use the "-o" option..

A

So if you want to instead use multiple processes to accelerate the hpcprof processing.

A

You can use our mpi version,, which is hpcprof-mpi.

A

In the end, after we get the database, we can use hpcviewer to visualize the calling context, sensitive gpu and cpu metrics.

A

And behaviors over time., so there are several options..

A

You can copy the database to your laptop.

A

And visualize it there or on cori you can use the no machine service.

A

To visualize it.

A

So next I'll go through a few tutorial. Examples.

A

I will first introduce them using the file. I have.

A

Then next I will switch the screen to a terminal and show you how we can get these examples and how we can derive important performance. Bottlenecks.

A

So, before diving into detailed examples,.

A

I'm going to show you a few important gpu metrics.

A

That may be of interest to you, guys..

A

So, first of all, we have the overall performance metric called gpu operations,, which denotes the total amount of gpu time.

A

Spent on gpu kernels and memory operations.

A

If you want to look into the details of the time spent on gpus,, you can look up kernel time and memory time, individually. also,. We have other metrics in addition to time. Rating metrics., for example,.

A

For gpu kernel metrics,, we have gpu blocks.

A

Which denotes the number of blocks used by each kernel.

A

Also gpu count indicates how many times this kernel has been launched in your execution.

A

For gpu memory copy metrics, we have host to device number of bytes.

A

And device to host number of bytes., there are also other metrics available in hpcviewer.

A

Next,, if you turn on pc sampling,.

A

And you get some fine-grained measurement data.

A

We also have gpu instruction. Metrics.

A

here are some important gpu instruction metrics., the first one is gpu instructions,, which is the total number of instructions collected.

A

At this frame,, which could be a line,, a loop, or a device function.

A

There's also stl_any metrics, which denotes the total number of stalled instruction samples at this frame.

A

We also have detailed stall reasons for every frame.

A

For example, stl_gmem indicates the total number of stalled instruction samples.

A

Waiting for the results from global memory.

A

Okay,, we offer four examples in our tutorial: repository.

A

I will introduce you to three of them..

A

First for the laghos example, you can just follow our instructions to profile it.

A

There are a few caveats when you want to profile it step by step.

A

Using hpcrun,, hpcstruct, and hpcprof.

A

So when you are using hpcstruct, you may want to specify "-j" to enable parallel analysis for a lot of gpu binaries.

A

in hpcviewer,. We can use the bottom up view to check out the kernel and memory copy, hotspots.

A

In top down view,, we can click the flame button to view the full calling context of the hotspot.

A

To compare hpctoolkit with nsight systems,.

A

Hpctoolkit performs both profiling and tracing.

A

While nsight systems only does tracing on the gpu site.

A

For the quicksilver example, we would like to turn on pc sampling.

A

To collect pc sampling data because for this example, there's actually just a single kernel launched several times, but this kernel is quite large.. It invokes many device functions..

A

So in hpcstruct we want to turn on gpu cfg analysis to reconstruct calling context.

A

For gpu device functions and loop nests.

A

In hpcviewer, we can use it to see detailed stall reasons at each line.

A

To compare with nsight systems, hpctoolkit does not replay gpu kernels,, so it can profile quicksilver in lightening speed.

A

Hpctoolkit also recovers loops and reconstructs approximate calling context, trees on gpus.

A

Which is something nsight, compute doesn't do.

A

For the pelec example,, we want to turn on both cpu and gpu profiling to check out the trace view.

A

in hpcviewer,. We can use.

A

The filter functionality to hide background, cpu threads.

A

We can zoom in to focus on the important gpu activities.

A

We can also use procedure-color map to highlight gpu synchronization events.

A

Okay,, so in the next I'm going to switch to the terminal window.

A

All right, it works.

A

So can you see my screen now?

A

just want to make sure it works. okay. all right.

A

So after cloning, the tutorial example,.

A

Here's what you get when you enter this directory.

A

Then you want to "cd examples/gpu" and choose an example.

A

For example,, we want to choose quicksilver.

A

Next,, the first thing you want to do is set up some environment variables.

A

You can say "source" and since we are currently on summit, then we choose "summit.sh"., so it will prompt something like this..

A

You want to set up the project, id.

A

So I'm using my own project id for now.

A

Next,, you want to say "source" again.

A

Then you want to also set up the reservation. Queue.

A

Since we are currently at day one, we will use "hpctoolkit1",, then you're, all set.

A

Then check out the make command again..

A

It prompts a few instructions for using this example.

A

So at the first you can say "make build" to build this example., and then there are a few options.. You can use "make run" to just profile and collect kernel level. Information.

A

About gpu performance., when you say "make run-pc", it actually collects pc sampling data..

A

So let's try "make run-pc".

A

Then this job is -- oh,. I didn't see that.

A

Sorry. okay., then all the modules are loaded.. You can see this.

A

Script is very handy., it actually notifies you at every step., then you can say "make run" and this job is launched into the queue..

A

We are not going to bother waiting for the job to finish.

A

I'm going to show you a few databases.

A

So this is a database. We tracked after profiling, laghos.

A

So in this example, we first switch to the bottom up. View,, then this filter button,, you can click. It.

A

And then you can see hundreds of gpu performance, metrics.

A

Of course,, we don't want to check out every metric.

A

Then you can use the filter to filter out whatever metrics you want.

A

In this case,, we probably want to focus on.

A

The gpu kernel metrics and the gpu mem copy metrics first.

A

So we choose these metrics and click okay,.

A

Then you can click the kernel column.

A

It will prompt the specific meaning for this metric.

A

So gker represents gpu kernel, execution, seconds.

A

Then,, if you click it,, it's going to sort it both in ascending order.

A

Or descending order., since now we want to sort them from the highest to the lowest,, then on the top frame. It accumulates all the kernel's running, time.

A

If you click the flame button,.

A

It's going to expand this frame and show you all the functions you executed.

B

These -- [suzanne] quick question.. What is the "I" and the "e" for these different kernels?.

A

Yeah.! Sorry for not introducing them.

A

So laksono actually was supposed to be introducing these metrics before me,, but now the order has been changed., okay, I'll, just briefly introduced these metrics., so "I" metrics represent inclusive time.

A

And "e" metrics represent exclusive time.

A

for some metrics like the running time,.

A

You want to know both the running time like at a loop and at the whole program. Node;.

A

For other metrics,, like the number of registers used at this line, or at this kernel,, you probably don't want to see the inclusive metrics.

A

Does that make sense?.

B

[John] yes., so it's basically,, is it myself or is it myself plus my children?.

A

Yeah, correct. laksono is going to show you more details about it.

A

Okay., so now that we have seen all these functions sorted.

A

By the execution time,, if you click one of them, which is the most consuming one,, you click the flame button. Again.

A

it's going to expand all the call paths so that you can see that this function is actually called.

A

Via different call paths,, and this first one actually consumes the most time.

A

Other call paths are not that important.

A

Then, let's switch back to the top down view..

A

This is going to give you more information..

A

You can click the flame button and you can see that it also shows you the loops on the cpu. Side.

A

click again.. It's going to expand this all the way.

A

Through the gpu function., so this is a gpu function that we are calling., since it's passing the function as a macro to this kernel,.

A

We are not able to show its specific code.

A

If we only turn on coarse grain, profiling.

A

But let's go up a little bit., then you can actually find the function. Body here.

A

Okay,, so in the next I'm going to switch to the quicksilver example,, which shows you the gpu, calling context on the gpu side.

A

So like what we did, first, we can see there's a gpu instruction, metrics.

A

And we can hover our mouse on this button.

A

we can see it represents the number of instructions executed..

A

Then we click the flame button..

A

It expands the whole calling context all the way.

A

Through a frame on the hotspot on the gpu side.

A

You can see that this calling context spans both cpu and gpu,, separated by a gpu kernel, place holder.

A

So this hotspot,, it's actually called in a device function, "getreactioncrosssection"..

A

Then this function is called at line. 41.

A

In the gpu device function., then we go up a little bit.. This function is called in this "macroscopiccrosssection" function at line 73.

A

And this call site is located within.

A

A loop nest of two levels.

C

Okay, any questions?

C

[helen], a quick question.

C

like the c++ host code,, the device code functions.

C

Can be unmangled, because the names looks like are just all mangled names?.

A

Yes, we do have both the demangled name and mangled name.

A

So, but for this name.

A

I don't quite know why it's not demangled.

B

[John], I think it's because a normal mangled function.

B

Would start with "__z" and it's getting confused with the device tab.

B

A

For more details,, we can chat offline.. Currently I don't have any good answer for this.

A

[John] all right. okay. thank you.

A

So, in the end, lets switch to pelec and check out its trace view..

A

So when you see this trace view.

A

You are probably wondering what this stuff is: doing.

A

Oh, okay., so at first I will say one thing about our trace: view.

A

So currently we don't collect any metadata information.

A

About the gpu streams., so every gpu stream will start with a thread id 500.

A

So this indicates the first gpu stream, which is by default, the null stream. and then the second row is the first stream;.

A

Then goes the third stream.

A

and in total we have five streams..

A

This is changing in hpctoolkit, but not for the current version..

A

So by clicking this green area on the right side,.

A

We can see its call stack so that we know that, for this short run, pelec is spending a.

A

Long time in the initialization kernel., this is, of course, not something. We are interested in.

A

So next we will zoom in to the interested sections.

A

By selecting the interesting area., then this is going to give you more information.

A

of course,. These colors may be a little bit confused, here.

A

Something you want to do is to use something called "procedure-color map".

A

So that you can focus on a specific activity.

A

For example, here,, if you want to focus on the gpu synchronization events, you can specify the pattern name and find out its color.

A

Since we want to highlight it, we will use the red color..

A

Then click okay.

A

click, okay, again., then all the synchronization events will be highlighted.

A

This is the cpu thread., so if we want to hide it, you can use the filter functionality here..

A

You can unselect this item;, then.

A

It will be hidden automatically in hpctoolkit.

A

So next we can zoom in further.

A

Into these stream activities and see what this is really doing.

A

If you move the call stack down a few more levels,, you can see that.

A

There are actually consecutive gpu synchronizations.

A

With no meaningful gpu activities in this interval,.

A

Which means we have redundant synchronizations, which we could optimize in the application.

A

All right., that's it for the tutorial examples.

C

C

[helen] question. is there a way here to see the async launches if they are from the cpu side?.

C

Is there a way to map those back to the cpu timeline somehow?.

A

Back to the cpu timeline? well,, you do see the call stack on the right, side.

A

[helen], that's true.,.

C

A

I think that's something very convenient., thanks for the question. okay, next, I'm going to switch back to our slides.

A

A

Okay., so we're collaborating with local teams at nersc.

A

Thanks to their help,, we were able to deploy these applications very smoothly on the summit. Computer..

A

The applications we have started are superlu_dist and strumpack.

A

The superlu_dist application is a gpu-accelerated, sparse, direct solver..

A

We tested its "pddrive3d" case..

A

We were using a single node on summit and using a single gpu..

A

So for this case of a real application, at first we observed that gpu activities are quite sparse. Comparing to cpu activities.

A

Usually, cpu samples are taken at a low frequency.

A

So if gpus are utilized at a high utilization rate,.

A

You can actually see these gpu activities on those streams with a broad rectangle.

A

But in this case, you actually see many cpu activities.

A

But only see very sparse, gpu activities., so it's recommended to increase the gpu workload.

A

The second observation we derived based on our profile view is that this application has spent a long time.

A

For expensive, cpu computations,, which could delay work being offloaded to gpu.

A

So, based on this profile view, we can see that for the real time metrics, there are several frames that spends long time in this code section..

A

So this small code section spends in total.

A

0.9 percent time,.

A

But the cpu-only code actually takes about.

A

One third of that code: time.!

A

If we optimize this code and eliminate these expensive cpu computations.

A

We actually achieve a speed-up for this code region.

A

About 1.78 x.

A

In the next, we also profiled, the strumpack application.

A

strumpack has solvers for both sparse and dense rank-structured, linear systems.

A

We tested its "helmholtz" case.

A

And we were using a summit computer node with four gpus.

A

At first based on the trace view, we can see that actually, this application has been using gpus very efficiently, as it's been launching frequent cublas kernels.

A

To gpu streams., these gpu activities are overlapped and, in the summary, view,.

A

So, first on the right side,, you can see every call stack to those cublas activities.

A

In the summary view, at each time, stamp.

A

We are going to stack the non-white activities.

A

On top of each other., so in the bottom line,, if a space is not white.

A

That means the gpu is active., so there's only a small fraction of the space which is white.

A

Which means gpu is active. Most of the time.

A

However,, when you zoom out in this region and see this high level picture, you can actually find some performance bottlenecks.

A

If we click the purple region here on the right side, the call stack points out that we actually spent a long time on cudamalloc.

A

And after moving the pointer to the left side,.

A

There's a similar hotspot here, where we spent a long time in cudafree.. This cudamalloc and free operations will synchronize per cpu and gpu so that delays the work being offloaded to the gpus.

A

We reported this issue to the strumpack team.

A

They switched their memory allocation to avoid excessive memory allocations and frees, achieving 1.15 x speedup on summit.

A

All right, and next,, I will summarize hpctoolkit utilities.

A

On profiling, gpu applications., so in a nutshell, hpctoolkit,.

A

Is a measurement and analysis tool that measures gpu activities and gpu instruction samples.

A

And attribute them to their corresponding, calling context.

A

It provides a trace view of how an execution evolves over time and a profile view that associates metrics with a hierarchy of individual lines,, loops and functions..

A

It collects, analyzes, and visualizes profiles.

A

Within and across nodes.

A

to use hpctoolkit, first,, you use hpcrun to dump performance measurement data and gpu binaries.

A

next,. You use hpcstruct to analyze, gpu and cpu binaries.

A

Next, we use hpcprof or hpcprof-mpi.

A

To correlate measurement data and program structure.

A

in the end, we use hpcviewer to visualize its profile view and the trace view.

A

There are a bunch of caveats about using hpctoolkit.

A

I will not go through every one,, but for your reference once you have encountered any questions, you can go back and look up these caveats and see if you can find any troubleshooting guide.

A

I would just focus on the first one.

A

so,. Many people have been asking me questions about that.

A

So, what's the overhead of hpcrun or hpctoolkit?.

A

So I would say that hpcrun measurement time might be dilated.

A

If an application has many short-lived kernels.

A

Due to the cost of call path, unwinding and kernel instrumentation, which is done in the concurrent kernel mode., you need to consider this slowdown when assessing how active the gpu is using profile or traces..

A

So for example,, without profiling,.

A

If your application runs for one second and gpu time takes about 0.5 seconds.

A

You should consider the space for improvement as 50 percent, because when we are profiling, this application, the total time, might be dilated to two seconds in the worst case.. This is not a problem;. It only affects hpctoolkit.

A

if you will turn on call path, unwinding or other expensive events in nsight systems.

A

It also has similar costs. also,, so for the current versions installed on cori and summit, we measure gpu kernels with the cupti activity that serializes kernels.. This problem is fixed in another branch..

A

If you want to actually treat these activities.

A

With concurrent kernel execution, you can ping me and I will probably point you to.

A

The hpctoolkit installation path that I have on summit.

A

Next, I would like to address.

A

That our control flow analysis for large gpu binaries are quite expensive.

A

And we do not show metadata, as you have seen in our profile, view.

A

There are a few more caveats but today,. I will not have time to go through every detail.

A

So our tutorial example is available on github.

A

To use it first, you clone this repository and then choose an example..

A

Once you are on the login node,. You first set up these two environment variables.

A

Like what I did then for each example on the login node, you first source the platform script.

A

And you say: "make build" to build this example.

A

next, you say: "make, run" or "make run-pc".

A

To plot the measurement data you would like., then you will copy this database to your laptop or visualize it on the login node.

A

All right., so that's it for my talk., I'm open to any questions.

D

[Suzanne] thank you, keren.

B

[John], so I'm looking at just running the quicksilver example and I did the "make run".

B

and after I did that, it just said: "submitting jobs".

B

And so now, what?.

A

Now you will be waiting for the job to be finished.

A

even later on,, like many people submitting jobs at the same time.. You will be waiting for a long time, but I have tested if summit's job queue is.

A

[John], I'm on cori by the way,, I'm not on summit.

A

D

[Suzanne] right., so on cori everything is done on the compute node., so "make build" and "make run",. They both run on the compute nodes and they are actually submitting two separate jobs.

D

Right. [suzanne] we'll have to wait until "make build" finishes before you submit "make run". with a reservation,. Everything is instant,, but outside the reservation you have to wait after build, has finished and then "make run"..

D

The way to do that is that you could do an alloc outside the reservation, and you can watch build and run.

D

it's just that we are limiting ourselves to not to do an alloc during the reservation window.

D

Because there are too many people.

B

[John], how do I know whether it ran or not? it just says, "submitted batch job".. Do I need "squeue" or something?.

B

[Suzanne] "squeues".

A

[Keren] yes.

D

[Suzanne] "squeues", one word.

D

[john] oh, "squeues".

B

"Squeue" is a command,, but then.

D

[Suzanne] "squeues". [helen], "squeues".

D

[suzanne] s is a no scrapper,, but it's there.

B

[John] okay.got it, got it. and that says: there's no jobs running., so that must means it finished, but it didn't. Let me know about it.

D

[Suzanne] yeah.: we can either check that or add to your batch script. Some kind of notification, --.

D

Use email or something.

B

[John] okay, so,, I did "make build",, I did, I did "make run" also,, and so it's done, like--.

D

[Suzanne] no. after "make run" is finished,.

D

You can "make view". [john] okay. and then you skip "make analyze".

B

You don't need to do that. One? [suzanne]! It's part of that on the gpu side.

D

The "make run" does "make analyze" as part of it.

D

You do need to do "make, analyze", separately.,.

B

[John] yeah. okay., I see that. yeah. okay., so I just logged on to,, so I'm going,.

B

So I did that all in a command line.. So now I'm on the other system, the,. What is it?.

B

D

[Suzanne] yeah, on your right.

B

[John], so if I go back into that, folder I should just be able to do. "Make view", is that correct?

B

D

[Suzanne], you probably need to set up -- if it's the separate terminal, source set up pe again.

D

[john] right,, right,, right., okay., [suzanne], yeah.

B

D

[Suzanne] and that should work.