National Energy Research Scientific Computing Center (NERSC) Totalview Tutorial, Dec 2020, 26 Jan 2021

Previous Meeting

Next Meeting

⏯

youtube image

►

From YouTube: NERSC Totalview Tutorial - Part2

Description

https://studio.youtube.com/video/bxFj9Otnn3o/edit

A

[Dean] we've covered in the first half the basics,.

A

And now we're going to get onto some of the more meatier topics.

A

So the next thing to look at is mixed language, debugging.

A

So c or c++ with python.

A

And I think there was some interest from people to see this.

A

So what we see amongst our customer base.

A

Is an increased usage of python.

A

Particularly to call out to applications that are written in c or c++., and that's predominantly to provide access.

A

To high performance routines and to leverage existing algorithms and libraries.

A

And there's lots of different frameworks available for writing mixed language applications such as swig, ctypes,, pybind,, cython, et cetera.

A

So we currently support swig,, ctypes, and pyblind.

A

And we know that debugging mixed language applications.

A

Is not easy because debugging, a single language.

A

Is difficult, enough., it's important to be able to understand the flow.

A

Of execution across language barriers,.

A

And to be able to examine and compare data.

A

In both languages., so what totalview provides you with is an easy python. Debugging session set up.

A

Will give you the fully integrated python and c++ call stack, and will remove what we call the glue layers.

A

So these are the additional stack frames.

A

That aren't the c and the python language stack frames.

A

So we remove the additional stack frames, giving you a very nice clean user interface.

A

You can easily compare and contrast the variables.

A

In python and c++., and it only requires modest system requirements.

A

You can also utilize reverse debugging in replay engine.

A

And memory debugging,, if you need to do so.

A

And we support python 2.7 and above.

A

And python 3.0 and above.

A

So this is a screenshot of debugging, a python.

A

And the c++ application.

A

over here on the right,, you can see the call stack,, and this is what we mean by the glue code..

A

These are all of the extra stack frames.

A

And what we do is we have the option to filter those frames.

A

And then you can see here, there's a much nicer,, cleaner user interface, much easier to understand what's going on inside the application..

A

So that's when we turn on filtering.

A

So let's have a look at that in practice.

A

I just need to go over to my virtual machine.

A

And what I'm doing is I'm running totalview, and I'm providing as an argument to totalview,.

A

Python 3.7, the debug build.

A

And the application we're going to be looking at is called test python types.py., and this is available for you within the python directory.

A

That's in the examples that is provided with totalview.

A

So you can try this out for yourself., and even though we don't have a particular tutorial,.

A

There are step-by-step instructions.

A

As to how to build this example and how to run it,.

A

In the examples directory.

A

So, let's start this up.

A

And what I'm going to do, first of all,.

A

Is I'm going to set a couple of action points.

A

And I'm going to use the application option.

A

And I'm going to set the action points on two functions.

A

One's called dot, and another one is called fact.

A

So I'm going to create those two action: points.

A

And you can see they're down here in the bottom left., so we've got those pending because totalview.

A

Hasn't yet loaded up the python 3.7 debug framework.

A

So when I press go, totalview will start.

A

And it will load up the application.

A

and you see those break points are no longer pending.

A

And in fact, we hit the first break point here,.

A

And we stop at this c example called fact,.

A

Which calculates factorial.

A

So over here, on the right hand, side, we've got the call stack..

A

This was what I mentioned about all of the additional glue frames. you can see. There's lots of stack frames, displayed here.

A

If I go up to this little filter,, I can turn on filtering.

A

And then totalview will remove all of the additional glue frames, leaving just the python and c++ module.

A

So this is a nice way to navigate for your code.

A

If I go to the python module,, I can see here that I'm inside the python module here.

A

And I'm calling the c fact function.

A

And I'm passing in a parameter a.

A

I can see down here, my local variables,.

A

I can see the value of a. and I can easily drag that over into the data view,, so just to keep that in focus..

A

If I go back up to the c++ fact, function,.

A

Again, we're taking in a parameter.

A

So the function, we've taken in the parameter end.

A

And again,, I can drag that over into the data view..

A

We can just compare and contrast and make sure the values are indeed the same, which they obviously are.. You'll notice, also that the python variable here.

A

Has a padlock next to it,, which means that it's read only.

A

So I can't change this,, even if I double click on it., I can't change that.. If I can change the c variable, though, if I wanted to,, I could change that to another value.

A

Alright,, let me just remove both of those from the data view., I'm going to uncheck the break point here.

A

Because I want to move on to the next break point., so I'm going to carry on debugging,, pressing go.

A

And now we run to the next break point which is inside the dot function. and the dot function calculates the dot product of two matrices..

A

If I have a look in the python module,, I can see here that it's taking in two numpy arrays,.

A

X and y, and it's calculating the dot products.

A

Of those matrices., if I want to have a look at the x array,, I can just drag this over here into the data view.

A

I can expand this out and I can see the value of the array: two, three,, four,, five, and six.

A

And if I want to see the same value of the array.

A

That's being passed into the c function,.

A

So here, inside the function,, we're taking in an integer,.

A

A pointer to long, another integer and another pointer to long..

A

So if I want to have a look at the array,, it's the pointer that I want to drag it over here.

A

and if I expand that, we can see the first value of the array is two,. And then I can easily cast that.

A

To a pointer to an array of size, five.

A

And if I do that,, then totalview displays the whole array for me.. So we can see the whole array, two, three, four,, five, six.

A

So this is the python debugging support.

A

That we have currently.- and I think you'll agree that it's very useful.

A

In that, you can easily transform the stack frame.

A

And you can compare and contrast the variables very nicely.

A

Let me just stop this and x out from here.

A

Do we have some questions about python?.

B

[Bill], nothing coming up on chat about python.

A

[Dean] anyone have any questions about the python debugging support?.

A

[Bill] just give it a minute or two.

A

[Dean], okay, (mumbles).

B

[Bill] so there's a question there indeed about mpi4py, work as well., so I have run totalview against mpi applications.

B

Written like the mpi4py on there. totalview, of course, will follow through the mpi launch.

B

Correctly and everything, it will acquire the python interpreters as they are launched.

B

Across different nodes and cores, and so forth.

B

And then everything that you saw with dean demo,.

B

Demonstrated on there and setting the break points into the c/c++ python modules.

B

That are loaded into python works, as you can,.

B

As we showed here., so you got to make sure that everything is kind of all set up,, so it totally follows correctly through, but then it will acquire and allow you to do the debug, just like we showed.

A

[Dean] thanks bill.

A

All right,, so the next topic that we're going to cover.

A

Is reverse debugging with replay engine.

A

Reverse debugging is a really nice feature., so without reverse debugging,.

A

If you had an intermittent failure,, you'd probably do something like setting a break point.

A

Realizing you'd run past it, reloading the application,.

A

Setting the break point a bit higher up and then hoping that you catch the problem or that it fails.

A

And you basically repeat until you narrow down the problem.

A

With replay engine,, you can set a break point,.

A

You can start recording,, you can see the failure and you can run backwards and forwards within the context.

A

Of the failing execution., so it's a really nice feature. and people that we've showed it to,.

A

They just really appreciate how much time replay engine can save them.

A

So when you're recording, replay engine,.

A

When replay engine is saving the state,, it's in record mode,.

A

And it saves the state information, sorry,.

A

The save state information is the program's execution history.

A

You can save the execution history at any time, and you can reload the recording when debugging the executable in a subsequent session.

A

So using replay engine, either from the tool bar.

A

Or the command line shifts replay engine into replay, mode.

B

So you're, either in record mode or in replay mode.

A

And there are certain commands that don't work in replay mode, things such as changing variables, value,.

A

(Coughs) excuse me, functions that alter memory and running threads asynchronously.

A

So the replay engine toolbar, then basically,, it's analogous to the normal totalview toolbar.

A

Over here on the left,, you have the record button,.

A

You have to go back button., you have the previous button, unstep, caller,.

A

You have to run back to button., you have the live button which shifts from replay mode to record mode.. So at any point during the session, you can go back to live mode..

A

You have a bookmark here, which enables you to create a replay in your bookmark at a selective location.

A

And then you have the option of saving the recording session.

A

To a file either for you to come back to a later tag,.

A

Or to share with one of your team members.

A

So it facilitates collaboration in that sense.

A

You can save the replay engine execution to a file at any time and you can load it up into totalview..

A

And, as we saw that's the same page as where you would load the core file or the replay engine, recording.

A

Replay engine bookmarks allow you to jump to a particular point in the execution of the program quickly.. So if you find a particular point in the code.

A

Where there's an issue,, you can easily bookmark that,.

A

And then come back to that,, just by double clicking on the replay engine, bookmark.

A

You can set the preferences for replay engine.

A

If you're using the new user interface,, you can set the maximum amount of memory.

A

That's available to replay engine.

A

By using the dset command.- and you can also choose the preferred behavior.

A

so by default, what happens is replay engine discards,.

A

The oldest history., so it keeps a rolling buffer,, basically overwriting, the oldest history.

A

So that the recording can continue.

A

and you can change that., there's either mode one,, which is discard the oldest history.

A

Or mode two,, which means stop the process.

A

When the buffer is full.,.

A

Let's have a look at a replay engine demo.

A

So I'm just going to start totalview as normal.

A

And I'm going to choose one of my recent sessions.

A

Which is replay engine demo.

A

They're telling me that the action point has changed that particular line., but what I'm going to do is I'm just going to set a break point inside the main function here at line 25.

A

And I'm going to turn on replay engine recording.

A

now,. You can turn replay engine recording on.

A

Either from the beginning of your debugging session, or if you've got a lot of initialization.

A

Or long running program,, you can run to a break point.

A

And then you can turn replay engine on from the break point..

A

So you don't need to record a lot of history.

A

If it's not particularly relevant. in this case,, I'm going to turn the recording on now.

A

and we can confirm that the replay engine is enabled.

A

By checking from the debug menu., you can see that checkbox there..

A

So when I press go,, we run to the first break point.

A

So we stop here at the break. Point, and now you can see the replay engine. Recording buttons are active., so these are highlighted now, because I've turned on replay engine recording.

A

And I can step into the function.

A

And now I'm inside the function. and I can navigate forward as I would do, normally.

A

And because replay engine is turned on,, I can also go backwards as well., so I can navigate backwards and I can even go out to the caller function as well.

A

If I carry on debugging, this particular application,.

A

It crashes, and we can see that we've got a segmentation violation, here.

A

We've got nothing in the call stack,, no local variables.

A

So normally,, what we would have to do is restart our application,.

A

Try and guess where the problem occurs,.

A

Set some break points, and then step through the code.

A

If that doesn't work, set the break points a bit earlier and so on.

A

and as you can see, it's an iterative process.

A

but because we've got replay, engine recording, turned on,.

A

I can simply click unstep, which takes me back.

A

Into my application. I can select a line of code here at line 50,.

A

And then I can choose the run back to option.

A

So I can execute all of those lines back to line 50.

A

Now, over here on the right-hand side,, I can see my call stacks,, so I can navigate up my call stack.

A

and I can see that local variables are displayed as well.

A

But we can start to investigate exactly what's happened.

A

In this particular application., and the last thing that happens inside this application is we're writing to an array inside the for loop,.

A

And the array v,, if I hover over that,.

A

Totalview tells me that it's array of integers and it's of size 20., so we're writing to it array. Length number of times,.

A

And that's 100.

A

immediately that looks suspicious..

A

We're writing 100 elements to an array that was only ever designed to hold 20..

A

So in order to investigate that a bit more,.

A

I can right click on array length and I can add it to data view.. So now it's in the base of view down here, you can see the value is 100..

A

I can create a watch point on that array. Line.

A

And if I have a look over here at my action points,, I've got the first break point here, which we set, and I've also got a watch point on array line..

A

So totalview is going to tell me when the value stored.

A

In the array line was changed..

A

If I carry on running backwards, now we hit the watch point, and we can see that visually.

A

Because it's highlighted in yellow. also from the command line,, I can see here,, it tells me it's hit the watch point., and this was where the value stored in array line.

A

Was last changed., so it was originally 20. and then inside this function, here, someone's changed it to be equal to five times.

A

The value of max depth,, which is defined higher up the program.

A

so immediately, we can see where the problem has occurred.

A

I can also create a replay engine bookmark..

A

So if I click on this,, I can call it something: like, error, location.,.

A

Then that's now appears in the replay engine bookmarks tab.

A

Down at the bottom., and if I was to carry on running backwards through the application,, we run back and we hit the first break point that we set originally..

A

If I want to go back to the location of the error,.

A

I can just double click on this replay engine in bookmark.

A

And it takes me right back there., so it's a really nice way of navigating around our program.

A

And then the last thing to mention is: if I want to save.

A

The replay engine recording,, I can click on the save button and I can save this to disk.

A

and then I can share this with other people or I can come back to this at a later point,, so that can save the entire replay engine recording.

A

So that's replay engine.

A

Does anybody have any questions?.

A

How are we doing on the chat?.

B

[Bill], let's see., so we had a few questions still about rdc, and a couple are both python and how it support there.

B

I did answer a question about conda's support.

B

And there just for others just to clarify.

B

so conda is a great package.

B

really does provide a lot of capabilities for python, but we've found that conda does not distribute to the some of the debug information that is required.

B

By totalview and also gdb, 'cause gdb has a level of python debugging within it as well.

B

To understand the state of your script and everything.

B

So they used to do this in a previous version.

B

in their latest versions,. They chose not to distribute that., so we haven't found a way to work around that packages like enthought the ones on systems and whatever do contain all the information..

B

You can also build python yourself., so unfortunately,, there's a bit of limitation on conda though,, okay?.

B

All right,, so do you want me to take over from here, dean?.

A

[Dean] yeah,, I think I'm going to hand control over to you., so I'm going to stop sharing and people can have a rest from my voice.

A

B

[Bill] so let's see, if I can do this correctly.

B

I can share desktop too.

B

And we'll move beyond,, let's take a look over here.

B

Okay,, so that's a lot of information about python.

B

Debugging and reverse debugging.

B

Thanks dean, for going through all of that.

B

one thing I'll just note on the reverse: debugging as it is per process.. So if you do need to, if you're debugging a parallel job.

B

And you want to do reverse debugging a particular rank on there,. You can stop and focus.

B

On that rank and enable and turn on, reverse debugging.

B

You don't have to do it from the beginning of a program.. You can do it at any point during the execution., often we'll find in,, I'm sorry,.

B

If you said this dean,, I repeated it,, but if you have a lot of setup.

B

In the beginning of your application- and you don't want to record that there is a slowdown when you do the reverse, debugging.

B

Set a break point beyond there. hit that break point and then turn on the reverse debugging..

B

And then you can begin to focus on the code that you want to record that history on, on there anyway.

B

Okay, so let's talk a bit,, I'm going to kind of meld.

B

Two sections together here with demos., I'm going to talk about totalview support of openmp applications.

B

And I'm going to do a demo of debugging, actually a hybrid application that is openmp and mpi.

B

And talk about some of the totalview latest features there.

B

and then I'm going to do a little more discussion around mpi, debugging, and I'll run. An mpi example.

B

On cori and show you again how the remote, ui and tvconnect work together.

B

To establish a debugging session in that fashion, there.

B

So,, let's start with the openmp first.

B

So it totally does for openmp, openmp as a library.

B

Or extensions that are delivered by compiler vendors.

B

It's a there is a standard out there that defines what the library and the interfaces.

B

And including the internal structures that are there.

B

For tools like debuggers to be able to read the state.

B

Of an openmp application,, the threads that are launched.

B

The variables that are specific to openmp threads, the tasks and regions and so forth.

B

And that is what the debugger uses to help you.

B

Get a more specific view of an openmp application.

B

Versus say,, just a regular pthreads application.

B

This is part of what openmp gives you as a framework on there..

B

So totalview understands that on there, there is a latest version of the spec,.

B

Actually 5.1 was just announced at supercomputing.

B

version, five of openmp totally revamped the debugging interface., and it gave a lot more capabilities for the debugger to understand the threads that are launched.

B

And threads that launch other threads.

B

threads that are organized into parallel regions., and it can then use that information to provide you.

B

A better picture of what your application is doing., and this is how you as a developer, are thinking about the application and how you're launching different tasks that are to be run,.

B

How threatened tasks may spawn other tasks and so forth.

B

so now,? One of the things just to note there is.

B

The version five and 5.1 especially, those capabilities.

B

Are just coming out in some of the compilers., so clang has a module that you can compile yourself.

B

And provides all the functionality I'm going to demonstrate on that latest version of compiled with support.

B

For openmp five.

B

intel compiler,, I believe, is coming out with it.

B

And has portions of it in place., so totalview is kind of ahead a little bit of where some of the compilers are. and as the compilers catch up on version five,.

B

We'll be qualifying totalview against their implementations.

B

It is an api.. They should have a standard way of us of being followed., so totalview should work with it pretty easily, but you always have to test for them. 'cause. Sometimes things just don't work.

B

The way you want on there.

B

and of course, totalview supports previous versions. version. Four and older implementations, openmp.

B

But, they weren't as rich on what the debugger could fetch.

B

For information out of there. so,, let me jump ahead.

B

Actually,, I'm going to jump into a demo here and we can see how some of the openmp threads and so forth, runs. so,. Let me do two different things on here.

B

So let me start totalview again,.

B

And get that onto the shared screen.

B

Okay, so that came up okay, here.

B

Okay, so again, I'm running totalview, on my mac.

B

And the system, I need to run on,, it's actually back in my data center again, I'm in massachusetts,. My data center is in colorado, and I have predefined configuration that goes to a system named microwave 1.

B

So it's going to establish a connection through..

B

I have ssh key set up to that system., so it just automatically connected through out there.

B

And we can see again that now I'm out debugging.

B

On a linux system from my mac.

B

now I'm going to leverage again,.

B

Let me close the old version: 12 here.

B

And I'm going to leverage that tvconnect mechanism.

B

That I talked about earlier., and so now I'm switched over to just a window. That goes.

B

Out to microwave 1, and I'm going to run tvconnect.

B

In front of a typical way that I run this hybrid application., I'm just using mpirun., I'm going to launch four ranks.

B

on this system, here,, I'm just going to say, oversubscribe on there 'cause. I may exceed the number of processes on our cores in the system. and I'm going to run an application.. It's a hybrid application that computes fibonacci on here.

B

So once I run this, back in totalview,.

B

Totalview is going to get that request now to say: do you want to establish this remote debugging sessions?, let's say, yes.

B

okay,, so totalview is on mpirun..

B

We don't have the source for mpiruns,, so there's no source available.. So let's start running here and totalview will launch mpirun,, which will then go through and launch the parallel job.

B

And totalview asks,: do you want to stop so I can separate points?. So, let's say, yes.

B

Okay,, so now what we have here is my source.

B

For my application, hybrid fib dot cpp at the top here.

B

And I have the opportunity before anything is run.

B

That I can go through and set break points., so I can set a break point, say on line 15.

B

And like dean had showed, here's our action points being with those break points.

B

by default, totalview, will instruct each process.

B

And only the process to stop at that location..

B

So when I run this, each of the ranks is going to come in here as their computing fibonacci and they are going to stop at this break. Point.

B

So let things run and we're going to hit go on the entire group.

B

Which lets all of the ranks run through.

B

And we can see them sitting at break point on this. One here.

B

And if I look to the left, here's our process and threads view that dean had described before.

B

and right now I have this configured..

B

So if I bring up the options panel down below,.

B

It's going to aggregate based on the share group.

B

Which is all of the executable with the same image.

B

Which happens to be all of them? Here. they're running all hybrid fib,, same program,.

B

The host name,, it all be the same here..

B

So that's not going to be that interesting., the process state and, let's turn on a thread state here.

B

Because we're dealing with openmp and then down to function., if I wanted to,, I could even look at the source line that we're aggregating on..

B

So, let's close this panel- and we can take a look here.

B

That I have four threads that are sitting.

B

At line 15 at a break point from my application.

B

There's a number of other threads that happened to be part.

B

Of openmp and even open mpi.

B

And whatever is on here that are just stopped and wherever.

B

Because the applications are stopped on this., I can refocus by double clicking on any of these ranks.

B

And we see that they are all sitting at line 15,.

B

B

I can control individual processes., so instead of issuing a command across the whole group,, I can go to an individual process and I can step that process. and now I can move that through and you can see.

B

Actually, that that went into a pragma at line 21.

B

And to that pragma,, which is openmp code,.

B

Ended up with a another set of calls into fibonacci here.

B

And that's what jumped us back into this break point again.

B

So at a high level, I can see what's going on with my ranks and my processes,.

B

But within totalview now and to bring up the view,.

B

There's a new view called openmp.

B

And if I open up that view on here,, we have four tabs.

B

Within that view, that begin to give us information.

B

About the different parallel regions, and task regions that are being hit here., so I'm going to let everything run a couple of times here.

B

As we hit these breakpoints and let some of these tasks and regions develop, here.

B

And what we can see down below is, as this fibonacci routine is recursively called into.

B

By different tasks,, mpi, tasks, and openmp tasks in here,.

B

That we see the relationships that are occurring.

B

so now in this tree we have our task regions, and, as I click on these,, it will,.

B

If I double click on it, it'll, bring me to that task. That was launched,, that's doing work.

B

and that task spawned. Another task, happened to be at those same lines where break points were hit and tasks were launched.

B

And then we can see another one at 24.. So now this relationship is what openmp version five.

B

Allows us to draw out for the developers.

B

And we couldn't do this in prior versions of openmp before.

B

So this is a big advancement for openmp developers.

B

And I've got fairly simple parallel regions, here,.

B

I only have one,, but if you have more complex parallel regions,, you would see those as well.

B

there's more information that we can gather here.

B

About the openmp configuration.

B

So if I look at the control variables,, this will give us information about versions.

B

Then,, it's a lot of internal information that the openmp library is using as part of its configuration., and some of this will come down to how it was built., and so this will give you those details there.

B

so you can understand influencing behavior of the application.

B

and then here we can show you, the actual library.

B

And api version information., so this is part of llvm openmp version. Five.

B

And tr62 is the build and the actual library that is coming from on there.

B

And then lastly, we can look at across each of these ranks here, the different threads that exist and the regions that they're in.. So it's a different way of navigating specific to openmp.

B

That we can look at here to get into any of the regions that may be going on here.

B

And I can, double-click it.. I believe I can double click on any of these to refocus.

B

Let me try here., I thought there was,, maybe I'm driving that wrong on there to get into those different regions. So.

B

Now the other thing I'll point: out,.

B

The other thing that I'll point out in the right hand area is the call stack..

B

So the call stack, now we'll also show an integration.

B

Of where you may have gone from one task into another, task.

B

And what it will do is show you a frame where they are linked together. and if I click on that link, frame,.

B

It will,, you can see here,, it should be bring me.

B

To that point here at line 24., let me go up.

B

So I can click on a frame up above it., and this is the task region that it went into as it went from one thread into the next thread.

B

And it called through., so this kind of gives you a sequence calling from the execution that went through one omp thread task region into another as it spawned through..

B

And then, if you were in a lower area,, let me see if I can get a thread..

B

I may not have any in this particular state.

B

There's ways that you can navigate up through., there's a final link frame that would allow you to click up back through to a color frame., and I don't think I have any in that particular state right. Now.

B

It's just a link we refocus.

B

yeah,, so I don't have any in that state,. But the call stack would show a final link. Frame.

B

If I click that,, it would bring me back to the parent on there as well.

B

Okay,, so let me think if there's anything I've kind of missed on the openmp..

B

This is straight on a cpu.

B

One of the other major goals of openmp 5.

B

Is to support launching tasks into a gpu.

B

And of course,, making it much easier to do work on a gpu there., there's still work, that's being done on the interface,.

B

The api side for openmp. and then of course, the implementations of the compiler vendors for launching code and tasks on a gpu..

B

But the ultimate goal eventually will be.

B

You will see threads that are running on a gpu and totalview would show them over in the left and where you are in that code.

B

Just the same as we see here, threads that are running on a cpu.

B

Okay,, so let me see if there is any questions so far,.

B

Any chat, that's come, up.

B

B

Chat messages, so nope,, nothing new on chats.

B

Any questions on openmp?.

B

I know I had one earlier, somebody's already trying to do. Totalview.

B

On an mpi openmp application with open acc as well.

B

So hopefully, this demo shows you some of the capabilities of what totalview can do in the space.

B

Okay, so let's now move on,.

B

To talk about, we're going to just basically extend what we were just looking at here,, but to do it from a parallel standpoint.

B

And then this demo here,, I actually will do the demo.

B

From launching a parallel job on cori., so let me just clean up a few things: here.

B

as I prepare for that,. Let me just exit out of my totalview.

B

B

okay, so for parallel, applications, we've already seen here a case where I've launched a hybrid application.

B

of course,, it's mpi and openmp together.

B

And, we've seen how totalview has the ability to simultaneously debug many threads and processes at a time..

B

The process and thread view that I showed in the left,.

B

That becomes a critical view for you to understand.

B

What's happening in your parallel job., you have the control over how to aggregate that view.

B

And when I bring totalview up again, I'll show you that there's three different views that you can store in there.. So maybe in one view you have a rank, oriented type of view.

B

And the information that you aggregate by a specific to ranks.

B

and then maybe in another one, you have a thread oriented view and you can change the attributes again for that, so that threads information is important, is shown there.

B

And, that yeah, it looks like woo-sun mentioned.

B

That the example there, yeah,, that example.

B

The hybrid fibonacci is one that I have,.

B

That I created on there. it's not in our samples..

B

I don't see any reason why I couldn't share that with you as well for the code.. This is actually,. It was derived from several other mpi and openmp examples that are kind of put together.

B

To create the hybrid example., so let me make a note of that to provide that.

B

Okay, all right.

B

And then we've seen here already.

B

About the support for mpi openmp, fork/exec as well.

B

If you have using those mechanisms to launch multiple processes and then different, flavors.

B

Of threading that you might be using.

B

So,, as dean showed earlier, there's several ways.

B

Of launching a parallel job within totalview.

B

There is within user interface, the parallel session dialogue.

B

And this is an easy way to get started..

B

You can provide the application that you're going to run in parallel and the arguments to the application on the left, side.

B

You could then select your parallel system that you're going to be using, the number of tasks and any additional starter.

B

arguments that go. There, is a number of different flavors of mpi parallel systems that we have pre-configured.

B

And for an environment.

B

One thing I'll note on here is the mechanism,.

B

The mechanism that's used to launch this style of parallel job is a little more forgiving.

B

Of the environment. it does some investigation around., but because of that, it doesn't scale as well..

B

This is good for smaller scale type of application,.

B

Is it's not going to leverage the efficiencies that say, mpi run or srun or others.

B

Will give as it launches up a parallel application., so I usually kind of,, it's an okay place to start.

B

But I'll steer people away from this pretty quickly.

B

and we'll look at some alternate ways through either your batch script or the command line to launch a parallel job.

B

in both of those cases, you're, going to leverage the efficiencies of your parallel system to launch a high scale. Job.

B

And then totalview kind of piggybacks on that to get control of the jobs,. Allow you to set breakpoints.

B

And then control the different ranks that are going through.

B

And so this slide here kind of shows some of those ways.

B

Of launching that alternate way from the command line here..

B

So if I go down to, you can see some of the ones.

B

Who slurm on here,, where we're doing totalview.

B

And then the dash args or dash dash args on there,.

B

Allows you to provide a program and then arguments.

B

To the program., but totalview will then run.. So here we're doing an srun-n16, and then the argument,.

B

The program to run., so what will happen as totalview will run srun and then piggyback along for the job to debug on there.

B

These are fine within an environment that you maybe aren't doing some remote debugging on there.

B

I'm going to show what the totalview remote ui again just 'cause. I think it works really well in the use cases now with remote clusters.

B

So let me move on.

B

Let me go through some of these slides, 'cause. We actually seen a lot of this already. we've seen how to control the groups, processes,.

B

Individual processes of threads through some of the demos that dean has done and what I just did in the openmp demo.

B

Call graph,, I will just mention this- is something that still only exists in the classic ui.. So if you're looking for a call graph of function, calls through that go across ranks of mpi jobs,.

B

That doesn't exist yet in the new ui.

B

There are various preferences to control what totalview does when your job launches?.

B

So there's a couple on there. one,. How should it attach?, should you attach to everything from the beginning, or do you individually attach to a process since it's running or, and also once you've attached,.

B

Do you stop all the ranks??

B

I typically have attached to everything and I have it stopped so I can set breakpoints.

B

and then I resume the job by my own.. I believe those are the defaults for a parallel job.

B

A feature that will come into the new ui is what's called subset, attach.

B

in the classic ui,, if you say, launched a large job of 1,024 ranks,. You may only want to attach to a portion of that. and with the classic ui,.

B

We show a list of all of those ranks and you can choose which ones you want to attach to.

B

and that will be coming into the new user interface.

B

Message: queues as well., so this is,, if you do, need to examine mpi message: queues,.

B

We have a view in the classic ui..

B

You can see this information through totalviews command line, interface,, which is available in the new ui.. So if you did need to investigate any of those,, you could.

B

and we'll be bringing this feature into the new ui.

B

And lastly, on the message: queue graph as well,.

B

You can see some of the ways that totalview will track the messaging that's going through in the communicators there. and this exists in the classic ui.

B

So let me just talk through a couple of techniques,.

B

Just so good to have kind of in your back pocket, as you are doing, debugging of multiple threads.

B

And deadlock, situations, and so forth.- and this is where using some of the features of totalview together, can help you solve a problem.

B

So totalview,, when it displays threads,.

B

There's several different ids that can be used.

B

Several are at the operating system level., so you can have,. What's called pthread library id.

B

And this what's displayed by default in totalview.

B

And that's usually a big long number that identifies the thread.

B

operating systems will also use what's called a lightweight process or lwp, id.

B

So both of these will be displayed in totalview., and that can allow you to equate to that.

B

If you were looking at say top or other information in your system.

B

now at the highest level, totalview does abstract and create its own,. What's called a process in thread, id.

B

It's a number,, it's followed with a dot and another number.

B

So you'll see like 1.3. and you'll, see that use predominantly.. That's how you identify a process and a thread in totalview.. So you'll see that through the user interface.

B

it's just good to kind of know the differences on there and what the debugger is showing.

B

So one of the cases that I had found when using totalview is common is trying to find a deadlock case when you're using mutexes., and when that happens,, you can have several threads.

B

That have a mutex and they'll lock and unlock that to access a shared resource.

B

and on linux,. That mutex is a structure.

B

And within that structure is a data element.

B

That holds the lightweight process id or that lightweight thread id of which one has the lock.

B

And one of the things that you can do, is, if you combine reversed debugging in the watchpoint's capability, is you can watch who owns that lock.

B

And when that changes on there,, the lock is granted from one thread to another..

B

So I've had cases where I've been able to get.

B

To a deadlock situation under reverse debugging.

B

And I'll watch that particular field of a mutex.

B

And then start running backwards, just like dean demonstrated with reverse debugging engine, until the point that it changes. and that's the point that you are granted the lock.

B

For the mutex., so you can start to understand,.

B

You need to understand a bit about how mutexes work.

B

In the data structure there,, but then you combine in,, reverse debugging capabilities and watchpoint capabilities of the debugger to then analyze. The point.

B

That that state changes and a lock is granted or released.

B

And then from there, you can start to determine how your program got to that state.

C

Okay, wrong direction.

B

And I think several other techniques on here,.

B

I think I've talked about each of these as well.

B

Yeah, okay., so let me do one more demo: here.

B

I'm going to actually do a different demo..

B

It's not going to be extensive! Demo., it's more to launch an mpi, application. and again,, we'll see how that looks under totalview.

B

Within the cori system though., so let me go back and I'm going to have to reestablish.

B

What I have out and often timed out here.

B

Yeah, so I'm going to have to log into cori again.

B

Okay, so I'm going to start totalview, again running on my mac.

B

And I'm going to establish a connection out to cori again.

B

On the login., so I have to log in, again. This is in a just a different window. Here., so give me a second, enter my information. That's needed.

B

(Mumbles) one more time.

B

Okay,, so I am,, if we're looking at my command line,, I'm connecting again now to linux and in the lower right in the status bar,.

B

It says my remote session is cori.nersc.gov., so I'm connected there to the system., and now I am going to log in manually.

B

To another place, where I'm going to submit a batch jobs.

B

so again,, I have to do this one more time.

B

Okay, so now I am there.

B

Okay,, so let me go into my demo directory.

B

And directory of prime count.

B

Okay, and so here, I have a slurm script.

B

It's going to submit a job on here., it's going to do a module load, openmpi.

B

I need to remove this statement.. I originally installed my own totalview, just to make sure things were working.. All you need to do is module load totalview before here.

B

and I need to replace that, but I'll leave it for now.

B

And I have put in front of my srun.

B

The tvconnect command on here.

B

So, let's just do a submission of this.

B

As we would normally do., I don't know,. Hopefully it doesn't take too long to get an allocation to run this.

B

And we will submit this on here.

B

and, as the system goes through the queue,, what we will see is that all of you.

B

Will then get the request to say,?

B

Do you want to accept this connection??

B

So we can say, yes.

B

and now we can see. Srun totalview is in control of that., and I'm going to begin running. My parallel job.

B

And in a moment, here, totalview asks us,.

B

Do we want to stop the job, so I can set break points?.

B

So let's say, yes.

B

And so at this point, everything is being launched by srun,.

B

Totalview is getting control of it. and then you end up hitting a magic break point.

B

Within the way that mpi applications are launched.

B

And nothing has really begun running here, but we have my application and I can go through and set breakpoints in here., so I've got a routine,, that's just counting prime numbers on here..

B

So I can set a break point up. Here,.

B

And we can see that it's set at the process, level.

B

You can control and change, if I wanted the whole group to stop, or even an individual thread to stop, I'll, keep it a process. Level.

B

and my process and thread view on the side is showing all of the different ranks.

B

And the ones that are set at a break point., it's a special break point for our totalview at this point that has gained control of everything.. So let's just run and see how it all works.

B

Okay, so we've gone through.

B

And we have stopped this rank here.

B

In my status bar,, I can see it's ranked zero, thread one,.

B

Has stopped and hit that break point at the line that I set there. and I have four of the ranks..

B

If I double click on those and refocus,, they are all sitting at the same place, just as I would expect on there.

B

I mentioned earlier different views of how I can look at the aggregated data.

B

And I can click on view number two here, and this one I can configure my own way of doing aggregation.. So if I wanted to just look at without source information,.

B

And just thread state down here,, and maybe I don't care.

B

About a share group,, then it will do an aggregation there.

B

Of which threads are running,, which ones are stopped and which ones are at break point.

B

and of course,. If there were any at an error state,, you would see those as well. and then I can flip back to my previous view as well on here.

B

To understand where things are at in my process and everything.

B

okay,, so there's,, let me think through on here.

B

What else do we want to demo? so we've demonstrated, again,, I'm running on my mac in massachusetts,, have connected out to,. Had the login node on cori.

B

And then, from there I issued a sbatch command.

B

To run my parallel job. and when that was accepted, using reverse connect,.

B

It established and tied a whole debugging session together.

B

And I'm able to do the debugging remotely now of everything here. so now I can let things run and we'll hit that break points again..

B

So I can disable that break point and let everything go through on here.

B

And then, eventually, once all the prime numbers and everything are computed,, the job will run out there.

B

Totally,, I think, we'll still kind of hold onto the srun,.

B

Actually, this time it all cleaned out correctly on here., so srun is finished,. It's not there anymore.

B

and if I wanted to, within the same totalview session,.

B

Start up a new debug session,, one of the things I have to do is tell totalview.

B

To start listening for reverse connections, again.

B

One of the reasons for this is that once a session.

B

Is loaded, we don't want you to have to keep getting interrupted to,. Do you want to accept this connection? Coming in?.

B

And totalview right now still has that lingering the old session from the srun here., even though we won't relaunch that on that,.

B

If I submitted another batch job on here,.

B

I just simply need to change this toggle to say, start listening again for a reverse connections.

B

And then I could issue another sbatch here.

B

If I wanted to. now the other thing I can do,, so let me launch,.

B

And, let's actually allocate,, I think I can do it. This way,.

B

Which is the command., so it's using last couple of days., so let me allocate a direct node that I can work on.

B

Hopefully,, it doesn't take too long to get the resources.

B

Might need to do the other reservation that woo-sun gave earlier today to see if that will work.

C

[Dean] yeah,, you can try that command.

C

I suggested here,, let me just copy it. Here.

C

Let's see if that'll work.

C

how's that right?.

C

Something like that.

C

did you try that?.

B

[Bill] yeah,, I think, is that the one I issued here to actually--.

C

[Dean] (mumbles) so interactive and maybe two hours.

C

Because the expiring at 12:30.

B

Maybe it's 90 minutes or something.

B

[Bill] yeah,, I have two hours on here..

B

I don't know if that will cause it and take it. (Indistinct).

C

[Woo-Sun], so the reservation will stop at 12:30.

C

So can you change the time to 60 minutes?

C

[bill] okay,, one second.

B

Let me regain control., so change it to 60 minutes?.

B

[Woo-Sun] yeah, one hour.

B

[Bill] so will that work?

B

[woo-sun] yeah,, I guess so., [bill] and interactive- is still right. [woo-sun] yeah. [bill] (indistinct). So let's try that.

C

[Woo-Sun] and use nintern. [bill] okay, all right [woo-sun] anyway, you got a job. [bill] yeah,. It has started to give it to me,, but you're right,. I didn't choose that, and pardon there.

B

So use nintern.

B

Okay, good (indistinct): this is revoked, unable to find it.

B

I mistyped that last part is a tv--.

C

[Woo-Sun] yeah,: that's a double dash, dash dash reservation.

B

[Bill] okay,, it's double dash. okay, thank you.

B

B

Reservation,, looking at what you typed.

B

Equals underscore res.

B

I spelled it right.

B

Let me just try your command here that you pasted in.

B

Let's try that.

B

There we go. okay,, so now we got onto a node. okay,. So now that I'm on a node here,, I can do interactive debugging. I own this, for a while..

B

I can go up to some of the labs.

B

So here's all the labs, and I've compiled the programs for the systems here.

B

And I can use the tvconnect command just the same here to debug any of these programs..

B

So dean had shown earlier the replay engine demo.

B

So let's try this.

B

and now totalview will get notified,. Do you want to accept the job?.

B

And now here's my source.

B

and I can turn on reverse debugging and I can start going.

B

Through and debugging this application just the same.

B

On here and stepping in and everything.

B

so everything's all just within now and the node on there,.

B

It's very easy.

B

It's a very easy workflow.. I've found now that I have totalview running locally.

B

And I can just use tv connect to launch my job,, whether it's a parallel job..

B

So I can go through and kill this now.

B

and if I wanted to go back,.

B

Let's close, the tv connect connection.

B

And I'm going to go back to my prime example. Here.

B

And if I, and I want to show this command here,.

B

Because this is sometimes the tricky part that can throw people.

B

So again, I'm going to run tv connect. so instead of using the batch system to start my prime account example here,, I'm going to just do it.

B

By doing tvconnect and run srun directly.

B

But you'll see what I'm doing here as I say,, which srun,, because what I want to use.

B

Is the full path to srun.

B

that way there as tvconnect launches, and the different mechanisms are going in.

B

And building up the debug connection is it gets the right, srun and the environment there.

B

And then the same,, I don't think this one is needed as much.

B

I think it will find prime count correctly, but I put a pwd in front of it just so it does that full path on that..

B

So now, as I do a tvconnect on here and do the srun,.

B

I do have to go back and tell totalview to start listening for connections again.

B

And then here comes my srun.

B

And just like before,, I can then start running.

B

Everything starts to launch,, I'm going to stop the job.

B

Before things get underway and I can set break points.

B

Just give it a moment for everything to establish, good.

B

Here's my source for my application.

B

and just like before,. I can set a break point on line 20,.

B

And run the application.

B

and now again, all my ranks are here: sitting.

B

So I've got a very convenient way on my dedicated node.

B

To do a relaunching of my parallel job, debugging.

B

To see what's going, on, fix, things,, recompile and so forth.

B

Okay, now, if I'm done with this,, I can kill the job and all the processes and go back over.

B

Tvconnect kind of holds on. it's one of the things I want to check there.. I think, because of the connection it has with totalview,.

B

I just control + c, that to say to tvconnect you're done,.

B

And totalview then cleans up because it goes away at that point. so, and then you can just start another debugging session on that.

B

So I've kind of combined a lot with showing openmp and mpi on there.

B

Any questions about doing this? does this workflow seem like something.

B

That would enable you to run your jobs,.

B

Debug your jobs within the cori environment, easier?.

B

I see your question on the chat about openmp examples. There.

B

I don't think we ship any openmp examples in that examples directory, but the tutorial does have an openmp,.

B

One or two openmp examples in there that you'll be able to compile and follow through the tutorial sessions. There.

B

Okay, I've seen various chatter around dean.

B

Is there anything else we need to address.

B

From the chat side?.

A

[Dean], I don't think so. no,. I think we're good.

B

[Bill] okay, all right,. So if there's no other questions there, at the end of the presentation, dean- and I have a couple of slides that we go through, just some usage, insights and areas that are just good to remember, sometimes can trip users up.

B

I have a dedicated slide that will reiterate.

B

Some of the areas that have gone over here in using their own remote ui and reverse connect.

B

As part of a session and just different things to keep in mind., so we will hit this again. and, like I said,, I will send a refresh slide deck to woo-sun.

B

So that he can provide that to everybody.

B

So I think next is moving on to memory debugging.

B

And dean you're going to take over the share again.

A

[Dean] all right, thanks bill.

A

So hopefully everyone can see my screen.

A

Which should say memory debugging with memoryscape.

B

[Bill] yes,, I can see it, dean.

A

[Dean] okay, so memoryscape.

A

It's lightweight, intuitive, collaborative,.

A

And provides heap memory, debugging features.

A

So it shows memory errors.

A

It can show memory, status, memory, leaks.

A

And buffer overflows., you can do mpi memory, debugging and remote memory debugging as well.

A

So when we're talking about a memory, debug,.

A

What are we actually referring? To?

A

well,? Obviously, a memory leak is a failure.

A

To free memory,, correctly.

A

We can also have dangling references., so failure to clear pointers,.

A

Failing to check for error conditions and memory, corruption.

A

So writing to memory that you haven't allocated.

A

Or overrunning the bounds of arrays.

A

The heap memory is managed by,.

A

In the c language, malloc and free and c++,, new and delete., and in fortran90, it's allocatable arrays.

A

And here's an example of using malloc.

A

To allocate some memory and then free to free that memory.

A

So the way that memoryscape works.

A

Is we have something called the heap interposition agent.

A

And that sits in between your user code and libraries.

A

And the malloc api.

A

so every time there's a memory allocation.

A

Or a memory deallocation, totalview records, this?.

A

So,, it knows about all of the memory allocations.

A

And de-allocations., and that way it can make sure that they match up correctly.

A

Some of the advantages of using totalview heap interposition agent.

A

Is you can use it with your existing builds of your applications.? You don't need to instrument your code in any way.

A

All you need is a debug build of your application.

A

And applications run,, but it really depends.

A

On your particular application,, but there's very low performance, overhead.

A

And the way that memoryscape works is it's very efficient.

A

In its memory, usage.

A

Some of the features of memoryscape,.

A

We can automatically detect allocation problems.

A

We have memory, corruption, detection., there's two different techniques. and we'll look at those in a bit more detail; guard blocks and red zones..

A

We can do leak detection.

A

and if you're, using the classic user interface,, we have dangling pointer detection..

A

So if you dive on a pointer,.

A

And it's not pointing to a valid memory, location,.

A

So to be flagged up as a dangling pointer.

A

We can also view the heap graphically.

A

and we can do a technique called memory hoarding.

A

Which is holding onto memory,, which would otherwise be released back to the memory management system.

A

And we can also do comparisons between processes.

A

And also different runs of your application as well.

A

you can compare memory usage.

A

So this is what we're talking about a memory leak.

A

In that dangling pointer., so normally when you allocate memory and you have a pointer,.

A

You have pointed to that memory, location.

A

If you delete the pointer,, but without failing to free the memory,, then you have a memory leak.

A

and if you have a pointer that isn't pointing to a valid memory location,, then that's a dangling pointer,, but just so that we're on the same terminology.

A

So memoryscape has several different debugging options.

A

When you start memoryscape,, you can choose the level of memory debugging., so you can choose low,, medium,, high, or extreme..

A

So low, we'll turn on event. Notifications.

A

And leak detection., so low has the best. Performance.

A

medium adds corrupted memory, detection,.

A

We've guard blocks. and the high turns on corrupted memory detection.

A

With red, zones. and extreme is all of the above options., and what we recommend is that, if you want to do.

A

Memory debugging of a particular application,, you start off with the low option first, and then gradually move up with different levels.

A

So you don't just immediately turn on extreme and see what happens. you start off with the lowest level first and then gradually move through the options.

A

And we also have advanced memory debugging options as well.

A

So you can control things like the amount.

A

Of what size memory allocations red zones are applied, to,.

A

And the pattern that's used as well.

A

So guard blocks are lightweight patterns that are written.

A

Around a memory allocation,, typically eight bytes.

A

And you can see here,. This is the pre-guard pattern.

A

And the post-guard pattern.

A

and there'll be written around your memory. Allocations.

A

And guard blocks will alert you to...

A

Here's example of guard blocks.

A

You can see here in this particular example,.

A

We have a corrupted guard block..

A

We can see that because the guard block in the center,.

A

The post-guard is colored orange., so that's immediately flagging up to the fact that these guard blocks have been corrupted.

A

normally guard blocks would be green, as you can see around this block here on the left.

A

We've got a green hash around the allocated memory.

A

So red zones are similar to guard blocks.

A

In that they give you array buffer, overrun detection.

A

But red zones use a whole page of memory,, whereas guard blocks are very lightweight.

A

But red zones have the advantage in that they give instant array bounds. Detection.

A

Typically,, you would apply a red zone either before or after your memory. Allocation.

A

And they will also alert you immediately as well.

A

You can also apply controls.

A

To the full management of red zones., so you can tell memoryscape, what size of memory allocation.

A

You want red zones to be applied to.

A

and that's in the advanced options., and you can also turn red zones on and off at any time during the programs execution.

A

We also have the ability to paint memory., so this is very useful if you want to examine a memory dump.

A

And you can see which you can see areas of memory.

A

That are allocated and de-allocated.

A

and if you look at these patterns,- and it does almost look like the word allocate, and de-allocate,, if you look at the pattern that gets written into the memory.

A

We also have the ability to hoard memory., so this stops the memory manager from reusing.

A

Memory blocks that have been freed., and this can be useful in detecting certain memory or a scenario.

A

So the idea is that you hold onto memory that would otherwise have been released by forcing your program to run out of memory..

A

You have the option of releasing hoarded memory when the available memory gets low, which allows your program to carry on running longer.

C

And the heap graphical view gives you an instance picture.

A

Of what is happening with your heap memory., so here we can see a view of the heap and the green memory allocations or the allocated memory.

A

The gray is the de-allocated memory.

A

The orange is hoarded, and there's no memory leak shown.

A

In this particular example., but memory leaks would be will be shown in red.

A

So here's an example of leak detection has been turned on.

A

And we can see here at a glance,, the red showing up as the leak to memory.

A

and also down here in the category,.

A

We can see the amount of leaked memory as well is 67 kilobytes of leaked memory.

A

So we can see that instantly.

A

As I mentioned in the classic user interface,, if you dive in on a pointer,, you can,.

A

Totalview will flag up, if that's a dangling pointer.

A

in other words,, it's not pointing to a valid memory allocation.

A

and it will flag that out by annotating.

B

[Bill] and that does work in the new user interface as well., so it will be there automatically, but once memory debugging is there, you'll be able to see that really easily too,, which is nice.

A

[Dean] all right, thanks, bill.

A

A

We can also do memory comparisons as well..

A

So if you are debugging an mpi application.

A

You can compare the memory usage of different ranks.

A

And so you can, for example,, see if one rank is using more memory than the other ranks,.

A

And that immediately would lead to more of investigation.

A

Also,, you can do memory. Comparisons between different runs of your application.. So at any point, during memory, debugging,.

A

You can save the memory status and then once you've made changes to your application,.

A

You can then save the memory status again and compare the two,, so you can see whether your changes have caused an increase or decrease in memory. Usage,.

A

Which is very useful., so here's an example of looking at an mpi application.

A

We can see all of the ranks, and rank zero is using slightly more memory than the other ranks.. So that may be something that you'd want to look into.

A

And we also have lots of different memory. Reports.

A

These can be saved typically as html files,.

A

And you can share these with other users as well.

A

So the reports are available, but for saving,.

A

They're, typically htmls or text-based reports.

A

And totalview and memoryscape work together.

A

So if you want to debug an application from totalview,.

A

You can just,, if you're, using the classic user, interface,.

A

You can turn on memory debugging,, which is just a check box.

A

When you start debugging.

A

and then from within totalview,, you can do it from memoryscape. and conversely, if you're debugging, an application.

A

With memoryscape,, you can also switch.

A

Into totalview as well., so the two work very nicely together.

A

and as bill mentioned,, we we're going to be introducing.

A

Memory leak detection into the new totalview user interface next year.

A

So you'll be able to start using memoryscape features with the new user interface, from the new user interface, pretty soon.

A

Some strategies around memory, debugging in parallel.

A

Run the application and see if memory events are detected, view the memory usage across the mpi job,.

A

That was, as I mentioned,, compare the different memory footprints of the processes.

A

Are there any outliers?, do you see some ranks using more memory than others??

A

Is this expected??

A

They gather all of the information in all the processes.

A

Of the mpi job and select and examine them.

A

And compare with a different mechanism.

A

Okay, so let's have a look at a memoryscape demo.

A

So,, what I'm going to do is I'm going to run memoryscape standalone.

A

So I'm going to, the command to run. Memoryscape is memscape.

A

And this is the memoryscape user interface.

A

So what I'm going to do is I'm going to add a new program,.

A

And this is a program that I've debugged before..

A

It's actually one of the examples that you have.

A

In the training lab packages.

A

So I'm going to choose the application called filter, app,.

A

Go to select, next.

A

and now I get to choose the level of debugging options.

A

So, as I mentioned, low provides leak, detection,.

A

Medium turns on guard blocks.

A

and high is red zones, and extreme is everything.

A

So we're going to turn on for this example, we're going to turn on guard block.. So medium is fine for our usage.

A

and if I just come over here to the advanced options,, if I click on advanced options,, we can have a look to see what's available. Here.

A

We can, from this button here, we can choose which of the memory. Events actually are enabled and disabled..

A

So things like a double free, event, guard corruption, error,.

A

We can turn these on or off. If we need to, default is to enable everything, but we can turn on individual events on or off, if we want.

A

Whole execution at the process accepts guard allocated memory,.

A

Here you can choose the pre-guard and the post-guard pattern,, and also the size of guard blocks.

A

And if I was to turn on red zones, and we can choose wherever red zones are applied,.

A

As an overrun or an underrun or both., and we can also restrict the range as well., so here we have the option of restricting which the lower and the upper limit of memory allocations,.

A

That red zones are applied to., so for instance, you could set a lower limit.

A

So that memory red zones aren't applied to every memory allocation, but only the larger memory allocations.

A

And we can also paint the memory. here's the option for painting the memory and hoarding the allocated memory as well.

A

So these are the other options that we have.

A

So let me go back to the basic options.: I'm going to leave the medium level selected.

A

Move on to the next section., and now I'm going to just run the program to collect the memory data.

A

So memoryscape stops and it tells us that there's an error- that's been detected..

A

It's saying the program has attempted to free an already freed block.

A

and we can see down here in the bottom corner,.

A

Here's the process.. This is a single process application..

A

If it was a multiple process, application,.

A

An mpi application, all of the processes would be listed down here on the bottom left.

A

and I could choose an individual process., so you have the option of report by process or report by event as it's a single process..

A

It doesn't really matter which of these I choose.

A

So the event is a double free, event.

A

It's telling us that the program has attempted to free.

A

An already freed block of memory., and if we have a look here,, we can see that actually at line 80,.

A

Memoryscape is telling us that there's a call.

A

To free memory here, the memory it was allocated.

A

To the variable p., the allocation location occurs at line 60 and then another de-allocation occurs here at line 69.

A

So the memory that we're allocating at line 60,.

A

Where we're freeing in two different places.

A

we're freeing it here at line 80, and we're also freeing it again at line 69..

A

So that's obviously incorrect.. We can't free memory twice that we've allocated once..

A

So what I'm now going to do is, if I go up to my toolbar,.

A

I can open up totalview.

A

as I mentioned, you can launch totalview.

A

Directly from within memoryscape.- and this is the classic user interface, as you can tell.

A

So what I need to do is in order to be able to continue with this particular evaluation,.

A

I need to just get around this second call to free.

A

But what I'm going to do is I'm just going to create an eval point. Here.

A

So here's an evaluation right point..

A

What I want to do is I want to just jump over that particular call to free.

A

Because I don't want that to happen., so I'm just going to set a go-to and I'm going to say, go to line 84,, which is the end of this closing brace., so that I don't want to have this free event called again.

A

Because otherwise, we'll get stuck here, and we won't be able to process for the evaluation.

A

Once I've done that,, then I can click on restart,.

A

And if I just minimize this window.

A

Memoryscape advises us of another problem.

A

So it's now telling us about a guard. Corruption, error.

A

It's telling us that the guard area around the block has been overwritten.. If I have a look at the block details,, I can see here that the pre-guard is fine and that's a green hash,.

A

And the post-guard, however, is colored orange., so that immediately tells me that something's wrong.

A

And if I hover over that,, I can see that it's been corrupted.

A

and if we scroll down and have a look at the actual data. Itself,.

A

We can see that the pattern should be a nine nine.

A

Actually, the first four bytes have been overwritten here.

A

So we can see that they're zeros., so memoryscape has actually picked up that something has overwritten the guard block..

A

If I closed this,, if I have a look, wants to have a look in more detail,, I can go up to the memory reports.

A

I can select what I want is a memory corruption report.

A

Because that's what's happened. and here memoryscape shows me the six corrupted blocks of memory., and it also shows me the original source.

A

So where the memory was allocated., it also shows me the proceeding blocks as well, and the following block 'cause. These may have useful information., so both the proceeding, block,, the corrupted block and the following blocks of data is shown along with the source where the memory was originally allocated..

A

So that's the second problem that we've detected., so the first was the double free event..

A

The second is the memory corruption, event.

A

So what I'm going to do now is I'm just going to export.

A

The memory data, because we can use this at the end.

A

So if I click on export memory, data, it's going to save it as a specific file type..

A

So this is the file type, and I'm just going to save that in my home directory..

A

So I'm just going to click on export.

A

Now,, if we go back to the home screen,, what I can do is I can carry on with this debugging session.

A

By clicking on start. and memoryscape run until the end of the program.

A

And if it doesn't detect any more errors,, it will say to us, okay, the program's about to exit,.

A

Now, a really good time to do any last minute memory checking.

A

Is your program's about to exit., so you should have cleared up all your memory. Allocations.

A

And once we get to that, point,.

A

We can do things like we can view the heap..

A

We can have a look at the memory data.. We can have a look for any memory leaks..

A

So here we can see event process is about to exit.

A

You can view the memory data., so let's have a look at the heap status..

A

So the report I want to view is the heap status reports.

A

And we have a number of options here, but let's have a look at the heap graphical reports.

A

So memoryscape goes away and it builds a report for us.

A

And this is what we meant when we said earlier about it being very conservative in its usage of memory.

A

So it only builds the reports when you actually ask it: to.

A

now,, it's building its reports.

A

And then it will display the heap for us.

A

And we can start to see what's happening with the the memory usages in this particular application..

A

So here we can see at a glance the memory allocations and de-allocations.

A

and I can zoom in and I can zoom out as I need to.

A

I can focus on a particular block of data and a memoryscape gives me information about that particular block,. The size,, the start address.

A

The end address,, the type,, the pre-guard and the post-guard.

A

I can also view over here on the left,.

A

I can see that the allocation is 25 megabytes,.

A

Corrupted guard blocks, the allocated and guard blocks in hoarded memory, et cetera.

A

And if I want to have a look at the other corruptive, blocks,, that's easy to do.. I can just select that from this dropdown box, here.

A

And then I can move through the heat, memory, and memoryscape shows me the other corrupted blocks.

A

If I want to reset at any time,, I can just click on reset zoom as well.

A

So what I'm going to do now is I'm going to turn on leak. Detection and memoryscape is going to go away and check for any memory. That's still allocated.

A

That doesn't have a valid pointer to it., so basically we're about to exit from the application..

A

We should have cleaned up the memory.. We can see that in fact, we've got 25 megabytes of leaked memory.

A

You can see here,, that's quite a lot, from the heap graphical view.

A

So if we want to have a look at that in more detail, if we go up to the memory reports- and we want to have a look at leak, detection.

A

One of the reports is called a source report..

A

So if I click on that,, I focus on the largest block of leak memory,, which is in my class b, and I can drill down into that.

A

And I can see that line 49,.

A

We leaking 19 megabytes of memory., so memoryscape has taken us right to the actual scene.

A

Of the crime here, where most of the memory is being leaked.

A

That's a very nice feature.

A

So if I go back to the home screen, another thing to show you is the memory comparison.

A

So if you remember, after the second error.

A

I exported the memory status, memory, debugging file.

A

So what I want to do now is, I can add, that memory debugging file back in..

A

So if I'd click on browse, here's the memory debugging file, I saved to disk., I can open that up and we can analyze that,.

A

So we can do a memory comparison., so here's the memory that I've saved to disk.

A

and here's the current running process.

A

If we do,, we click on compare..

A

Now we can compare the allocation, so we can see the difference in bytes between memory allocations..

A

We can see the difference in de-allocations, and we can see the difference in leaked memory as well.

A

But what's also quite nice is we can have a look at the memory usage and there's a chart report.

A

And if I go over here to the options, we can see, we've got a bar report,, a stacked report,, a line or a pie chart..

A

If I have a look at the stacked report, here, we can see on the left,.

A

This is the current running process..

A

So we can see the total virtual memory..

A

We can see the heat and we can see the text memory.

A

and then over here on the right. We can see the file that we saved to disk earlier on.

A

So immediately, we've got a very nice visual comparison between the two., so what you could do is you could use this as a way of comparing mpi processes, or you could use this as a way of comparing different runs of your application.

A

So if you save your application and then you make changes to it, and then you want to see, well, how have my changes affect the memory usage?.

A

And then you can load up the previously saved memory, debugging file.

A

So,, that's what I wanted to show you from memoryscape.

A

I think the other interesting report.

A

Is we have a high-level process reports and we also have a detailed program in library report.

A

And this is just sharing the memory usage of all the different libraries within the application., so you can focus on how much.

A

Each individual library is using.

A

the memoryscape, as you can see, it's very powerful tool,.

A

And it has lots of capabilities.

A

Let me come out of this.

A

Okay,, so did we get any questions about memoryscape on the chat?.

B

[Bill], let's see here.

B

How is memory corruption happening in this case,.

B

Was one of the questions that came through.

B

So in this case here you just overwriting the bounds of an array or how do we corrupt the memory in this program?.

A

[Dean] so we're allocating, we're doing an allocation.

A

And then we're overwriting the bounds of that array.

A

Just overwriting the end of that array.

A

And then memoryscape is picking that up.

A

Okay,, so where are we in the presentation?.

A

Okay, so--.

B

[Billy] transformations,, I think we're skipping through, that wasn't one of the topics for today, and I think cuda is next..

B

So I can take over on that part.

B

[Dean] I'll stop sharing., [billy] and yeah,. Let me pick up the sharing.

B

So how are we doing time check here?, so we got about 40 minutes or so left.

B

I want to leave some time for questions at the end.

B

and if anybody has any questions about the labs or anything.

B

Okay,, all right., so you should be able to see my screen, okay?, so I'm going to go through some slides on cuda, and actually I can do a live demo on one of my cuda machines here at the lab that we have to just so. We can see how a totalview looks debugging on the cuda application.

B

So,, like I mentioned earlier in the...,.

B

That's not the right one,, that's transformations.,.

B

Cuda, here we go. all right, so, like I mentioned earlier, totalview supported a debugging of nvidia gpu accelerators for quite some time.

B

We support up through the latest set of physical hardware for the gpus., we're validating on the latest a100 amperes.

B

Now,, I have one in-house just doing the final setup,.

B

One of our systems. and we keep up to date, aggressively.

B

On the latest releases of cuda from nvidia as well.

B

There's a number of capabilities that are supported through totalview. if you're using dynamic, parallelism, support for that.

B

If you're, using gpus across your mpi cluster.

B

And especially in a multi gpu per node type, configuration,.

B

Totalview is got support there for debugging on those different gpus.

B

One of the advances that you will see coming next year.

B

Is that same level of aggregation that you see at the process and thread level is going to be applied to debugging all across gpus,, especially in a multi gpu case.

B

of course,, when you're dealing with thousands and thousands of threads across lanes and warps and so forth on a gpu,, it can be tough to understand what is happening with your code as it's executing., so to help with that, we're going to be applying different ways of doing aggregation,, enabling you to do different queries.

B

Of the state of the code on the gpus.

B

So watch for that coming next year.

B

And it just further advances to support. Cuda development.

B

As you see,, when we look at some of the screenshots.

B

And I bring up a totalview debugging nvidia gpu.

B

As you can look at things in a physical fashion or in a logical fashion., we have ways to control the navigation across the different aspects of looking at the gpu in that way, and the state of the code running on them..

B

And inside of a gpu, there's memory, checking that goes on.

B

And if any events that are raised out of that through cuda memcheck totalview can capture those events.

B

And show you where memory faults are detected by the gpu.

B

Now, the totalview model on that one,.

B

When we look at code, that's running on a kernel.

B

And just the way that code starts up and is executed.

B

So of course, we'll start from a host processor,, the cpu on the system, and it'll kind of bootstrap.

B

The application that may be code that runs over there.

B

and then it'll launch code out on the kernels.

B

Now totalview does not know what that code.

B

Is going to look like until it is executed,.

B

And loaded and executed onto the kernel.

B

And that can create a challenge as far as setting breakpoints. it's similar to,. If you think back when we talked about the python demo, when python loads, a module,.

B

It does a purely dynamically and totalview won't know enough about it to really know where to set breakpoints until the code is physically loaded in a gpu, code. Running on a gpu kernel is the same.

B

So we'll see how you can set your break points in a fashion.

B

In that code, before it's even launched on the gpu., so it makes a nice clean, workflow.

B

And a there's a whole hierarchy of how the memory is organized through the gpu.

B

And you'll see annotations on totalview.

B

As you look at variables.

B

so you'll see, if a variable is a,, it will have an annotation.

B

If it's global or local or parameter types of data.,.

B

And so that's a different, just a different way.

B

That totalview will highlight the data that you're looking at that's running out on the gpu.

B

Okay, so let's see.

B

As code is running out there,, they are run,.

B

And a lot of this is specific to the nvidia gpu architecture,.

B

Warps are advanced synchronously across.

B

And they share a pc on there., so their totalview is a fairly high level view.

B

Of what you can do. and one of the implications of that is, when you run to a particular line, you're, going to move a lot of code, that's going across those warps to that line.

B

There isn't really individual control., as you might think, of individual threads running in a program,, and you can step that thread and the other threads stay where they're at and they can diverge.

B

it's a bit different behavior when running on a gpu..

B

So you have to keep that in mind as you're debugging your code across a kernel.

B

We have some notes in here.. I won't go through all these details..

B

You probably know a lot of this and just specific for your environment or how to debug it,.

B

Set up your application for debugging and so forth., I'm going to actually demonstrate this tx cuda matmul application. In a moment, here.

B

And we just have some notes there on what you might need to do for the different style of gpu to take advantages.

B

Of the advanced features of the gpu.

B

And then, finally, here it's really when the application is built and ready to go, it's just running totalview against the application.

B

there's nothing else you need to do. totalview will dynamically detect that cuda is on the system..

B

Your application is using cuda to launch code out onto it.

B

and then will reconfigure itself as it moves, as it goes through and starts up with double, enabling debugging. so,. These screenshots are kind of blurry., so I'm going to move ahead, and actually,.

B

I think I'm going to move into the live demo on here.

B

I can show off some of these other features,.

B

The program elements.

B

Device window, we're going to skip over because that is going away. and then the memory checker part, we have,.

B

So actually let me actually switch over to the vnc window that I have.

B

So, here I'm logged into a system.

B

That is in my data center in colorado.

B

And it has,, I think, a pascal,. This may have a pascal.

B

We have pascal, volta and tesla gpus.

B

This actually might be a volta on this system here., but I'm going to run totalview, 2020.3 and load that to tx cuda matmul., so literally from the command line,. We'll do that, and load the application.

B

Okay,, so a couple of things I'll point out in the source here. here is mean,. This is where gpu code would start running.

B

And going through and executing,, but quickly above at the targets top of this program,.

B

We will get into segments of code that you can see here.

B

This one's marked as global, and this is going to get run on the kernel., and even though we can see all the lines.

B

And what will be run on here, totalview in the left., so totalview we'll highlight where it knows it can set a break point.

B

and because this code is run out in the kernel and it hasn't been loaded on a kernel yet,.

B

We can't actually know where, what are valid: break points,.

B

Like line 94 will be a valid break point..

B

I can set a break point there., but what totalview will do is put a hollow breakpoint marker there to indicate, to say, okay,. I understand you want to set a break point here..

B

I don't know the actual location of it yet,, but I will try to resolve it once this gets loaded out onto a kernel., so it kind of will remember that.

B

and what you will see is further down at the bottom of this code unit for the kernel is it will plant a secondary break point., and this just happens to be what we call sliding down to find an actual location of where we could plant that break point..

B

So what will happen when your code gets loaded onto a kernel? Is it will resolve at line 94 on here.

B

That it can't plant a break point here and it will then slide back up this other location.

B

So it seems a little complicated,, but that's the world.

B

That totalview knows about at this point.. It only knows what it's given for information at the start of loading, this program.

B

and it will discover more as this code is launched,.

B

And out onto the kernel out here.

B

so what's nice is that I haven't had to run anything.

B

At this point. we can set the break points and figure out where, in my code, that I want to debug on here.

B

so we'll go through and I think there's another part of it. Here.

B

Yeah,, this is running up on the device as well., so I can plant another break point up in this code that will get hit.. So now we can start to run.

B

Okay, so we've hit this break point at line 94.

B

And what you can see now is it's a solid filled in rectangle.

B

And that indicates that this code was loaded and totalview found where it should properly plant.

B

That break point out in the kernel code on here.

B

I am stopped within the kernel. and one of the indicators that helps me know that.

B

Is when I look at my thread, id, totalview will give cuda threads a negative thread id index on here, so we're at process. One thread id minus one.

B

And so now I know that I am in gpu code.

B

I can look at the variables,, the local variables.

B

That are over here, and we begin to see those annotations.

B

That are on those variables., so in my data view,, if I want to explore further any of these,.

B

I can drag them over.

B

And then I can be doing expand, look at the values,.

B

I can change the values on here.

B

and we see, at generic on here, the at local,.

B

Using those indicators of where this information.

B

Resides in gpu memory on there.

B

I can step through., so at this point here, I can step this code.

B

And we're going to hit our other break point, 'cause as it starts to go through, we're executing more code on the gpu in the kernel, here.

B

And I could look at the values of those variables and I can continue stepping, and so forth on here.

B

And I think eventually we will return back through out.

B

And then we should go back up to the other routine, that launched from that kernel point into here on here.

B

So I've got full control of the stepping, look at the variables,, the values..

B

If I want to change any of my blocks or threads,.

B

If I'm looking at the gpu in a logical fashion,, we do put these two controls in the tool bar up here.

B

And I can move back through,, I don't know how many blocks or threads are configured in this application..

B

I think it's a fairly simple application., so there may not be really,.

B

There will be multiple threads there,, I think, are running,, but in this case here, it would depend on how you configured your code to run onto the device on here., and if you want to think of things in a physical fashion, here,, you can change your sms,, your warps,, your lanes,.

B

And so forth, to look at your code as it's executing.

B

Through those aspects of the device.

B

Okay, and so if we go and run here, again,.

B

I think we'll just hit the break point here at the code.

B

again,. I can change my attributes and how things.

B

Are broken down and aggregated in my display.

B

And if I want to go back to host code at any point.

B

Now, let's see which thread was actually on that host.

B

Like window three., I'm not sure which thread was on the host, but I could get back into the host code as well at some points., we'll jump back over to the gpu here..

B

So I we'll also note that unfortunately,.

B

Reversed debugging doesn't work on a gpu.

B

So it will not record what happens on the gpu.

B

But if you do enable reverse debugging on the cpu side,.

B

It will record everything that happens there., and so that could be helpful if you needed to debug,.

B

Reverse debug how your data may have been set up before it launched over to a gpu.

B

Okay,, so we flip back over to the slides..

B

I think that is it, yeah,, that's just animated graphic.

B

That goes through.

B

okay,. So any questions on gpu debugging before we move on?.

A

[Dean] there's a question on the chat, bill.: it's about a large scale, complex application, debugging.

A

I'm not sure if it's gpu specific,, but it's about long running applications.

A

I was thinking about maybe using tvscript.

A

To kind of catch, some of the errors.

B

[Bill] right, yeah., so yeah,. If you want to go ahead and talk about that for a second, dean,, I can actually flip to,. We can just take a moment here., so we're doing okay on time., so let me, and we won't go through a full thing in here.

B

But if you just want to talk about what tvscript is, and how it could be used in this case.

A

[Dean] sure,, so with a totalview,, we have the option of running it in batch or unattended mode,, which basically all of the tv, all of the totalview commands.

A

You can run on tvscript., so you could create a script to run your application.

A

And then you could say, within the script.

A

If any particular error occurs, then do the following.

A

And the following could be: you want to write out.

A

The call stack, the values of particular variables,.

A

You could print out,, you could save the memory debugging file, which we saw earlier.. You could save that to disk., there's various different things that you could do from tvscript..

A

So tvscripts has got, it's very powerful..

A

It's based on the tcl programming language.

A

So basically, you can create a script.

A

That runs your application.

A

and then you can just let it run and leave it.

A

and what will happen is you'll get two different log files.

A

You'll get a summary log file which tells you what, what has happened.

A

for example,. It's hit this break point or it's encountered this error,, and it's done this action.

A

And you'll also get the log file,, which gives you all of the information that you've requested. so yeah,. I would suggest maybe tvscript.

A

Could be one approach to using, to tackling those kinds of issues.

B

[Bill] right, okay., hopefully that helps on there.. So there is decent documentation about tvscript.

B

And how you can use it like a dean describes on there.

B

And that's a great case on there, where you could have a large job, run it on a tvscript. and if it were to crash on there, it's going to save off the core file..

B

If you're running under reverse debugging,, you can instruct it to save a recording file.

B

and any other information there to help you in a post-mortem way, try to reconstruct what may have happened with your application.

A

[Dean] one of the labs covers the tvscript as well., so you can have a look at the lab,.

A

But if you have any questions about how to create a script,.

A

Have look in the reference guide because we use the guide, then the reference guide.

A

and then the reference guide has got the information about tvscript, but also,. We can help with that as well.

A

[Bill] and there's a question about gpu direct,.

B

Does totalview support or work with gpu direct?, so I don't, gpu direct is a way for the gpu's.

B

To very easily access data that is being exited.

B

Through either network, adapters, storage drives or other locations., so you don't have to move as much data onto gpu.

B

And so forth, there. now,. I need to try some explicit programs too.

B

Before I can say conclusively, if it will work with it or not., I believe with the gpu direct, they're, just additional directives like we saw,.

B

For accessing data that way.- and it just goes out through to access the data directly.

B

On either through memory or storage drives on there.

B

So I'll have to get back to you on a more definitive answer.

B

If will it work with the gpu direct or not.

B

Okay,, so let me jump ahead., and so I can, that actually covers.

B

Most of the agenda we're going to cover today.

B

The last couple sections was to talk a bit about.

B

With the totalview roadmap,.

B

Kind of what we're looking at for, for next year.

B

And then dean and I wanted to go through a few of the slides of the common areas that developers may have.

B

Either some questions about with totalview.

B

Or sometimes things that can trip up a developer.

B

As they're going through, and they're new to different parts of totalview.

B

so, here,, this is what we've come from through 2020.

B

We've had,, we totalview does four releases a year.

B

And you can see the progression through.

B

Of different features that we've incorporated, including this latest release in june., that includes that remote ui capability, that we saw today,.

B

The openmp five support.

B

I will note for anybody who are students.

B

At your organization, or you know, of students that could use the power of totalview,, we have a very friendly student license.

B

That they can apply for and get a license for free for that.

B

and we've enhanced its capabilities this year.

B

To allow them to use the full power of totalview and everything., so it's great to have that offering for students as well.

B

So, looking into next year, you'll see common themes that kind of run through.

B

We are,, like I said, aggressively, keeping up with nvidia, they're, moving, really fast.

B

and so we'll be continuing to add, support.

B

For their latest gpus and cuda advances and other technologies, there.

B

so you'll see that in our progression of four releases for next year that are planned.

B

For users that use macos,.

B

I know that the community generally keeps up very quickly to the latest version., so we plan to add big sur support in our february release.

B

That'll be the first release of the year..

B

Initially, things were looking okay, but some of the bundling that they've done.

B

Because they are preparing to have a dual bundling support.

B

Of x86 and arm as they move their architecture over to arm.

B

So that does impact the debugger, of course,.

B

In what it has to do to load debug information., so we have to incorporate some of that into the debugger..

B

We saw today with the openmp and navigating between threads and so forth.

B

Sometimes,, it can be confusing on where I came from and you may want to get back to a particular starting point., so we'll be incorporating a feature called like a dive stack. and as you, either go through threads that have called each other or you go through functions or anything like that, much the same as you. Click through links on a browser, you'll be able to return to previous ones. More easily., so that'll be a feature. That's coming.

B

We've talked about the memory debugging., so I'm very much looking forward to incorporating a memory debugging into the new user interface.

B

we'll begin with leak detection and looking at the overall state of the heap.

B

And then, through subsequent releases of the year, we'll be adding into the ability to export memory. Data,.

B

Compare memory, data,, look at buffer overruns and all the features that dean talked about today as well.

B

I want to incorporate array, 2d and 3d,.

B

Eventually, array, visualization, and other types of data visualization and the product, a number of users that it's very advantageous to look at their data. That way as they're, debugging, it.

B

and building on that, as well as improving array, debugging support and allowing you to slice and stride.

B

A portion of your data.

B

So you can focus on what you would like to see and filter that data as well.

B

And then going later into the year,, we don't have assembly view yet into the new user. Interface.

B

And, as you start to get to more advanced code, often you do need to get down to the instruction level and look at what's happening as data is moving across registers and through different instructions., so that'll be coming. Across.

B

And then some advancements we want to do to continue to invest in reverse debugging and allow you to more easily navigate across that timeline.

B

And see events that happen across there as well.

B

And we mentioned some of the other areas, such as incorporating subset attach,.

B

Better data aggregation..

B

I have some improvements that I want to make on that tvscript technology that we just talked about there.

B

it's very powerful to be able to drive the debugger in a script like way.

B

Not only for running in a cluster, but I have a lot of customers who are doing dev, ops and working in a continuous integration environment.

B

And running under the debugger, say within jenkins, can be advantageous to capture the results and a failed test and everything, and show that to developers when they come back and revisit a failure.

B

So still a lot going on there.

B

One of the things that really helps us is any feedback is,.

B

As you use, the product features that you wish were there,.

B

Features you have problems with.

B

anything there,. We want to make sure the product.

B

Is doing the best it can for you and helping you get through issues in your code and solve whatever programmatic mission that you're working on.

B

Okay, so,: let's talk through a few of these, dean,.

B

That we've compiled on here.

B

So,, we talked a bit about,.

B

I think there was a question that came up. if you couldn't find the totalview source.. Just a couple of things to keep in mind,.

B

Make sure you compile with -g.

B

g is what instructs the compiler to build in that debug information and tells totalview where to go, find source files.

B

And if you have done that, or maybe you've moved your application from where the source is, and to adjust totalview.

B

You can have it,, you can specify new search paths.

B

For the source. and in either ui,, you can just go to the preferences dialog.

B

And then look for the search path. and you can add in new paths of where to look for source.

B

Okay, for python debugging,.

B

I talked about the issue with conda earlier, so that sorts of people should be aware of that.

B

There is a level of debug information that is available.

B

Either with enthought or as part of system packages., and that gives the debugger the insight to what's happening into the interpreter, and allows us to do the filtering and show you.

B

That integrated call stack between python and c and c++..

B

I will also note here,, so anybody working on advanced latest versions have sent to us or red hat enterprise. Linux, seven,, eight,,.

B

That python has changed some of those internal data structures.

B

and we have to have totalview accommodate those.

B

So if you're on one of those operating systems using that latest version of python, you'll see totalview.

B

Doesn't quite do the right thing there. and if you check our release notes, there are some work arounds on that, so that you can get totalview.

B

To do the proper python debugging.

B

we'll be adding,, it's basically a porch that we need to do to support the new data structures that they have changed to in the latest version of python. On that version.

B

We talked a lot today about using the rdc and the remote ui.

B

And I see a question: that's coming up from michael there.

B

I'll answer that in a second on there.- and I do have some information for that.

B

X11 forwarding performance can be bad.

B

I don't encourage people to do an ssh-y.

B

And try to forward a total display.

B

x11 was just not built for rich user interfaces like totalview., and you will be frustrated with the performance, especially with the new ui..

B

I'm much more advocate use of the remote display, client.

B

Or the remote ui feature, and not deal with the problems with x11 forwading.

B

Michael asked a question,: is there major differences in running totalview under the intel versus gnu compilers?.

B

For the most part, no.

B

and normal debugging,, they both produce good debug, information. and totalview can show you variables,, including variables.

B

In inline functions that can be often be very tricky.

B

And we'll give you very good support of both compilers.

B

Where we do see differences start to come.

B

About in those two compilers, is, when you add in a level of optimization with debug information.. So if you compile with -g, and then you say, -01 or -02 or -03,.

B

And sometimes you need to do this on complex codes.

B

Because you just need them to run as fast as that, compile will make them run..

B

We have found that the gnu compiler actually does a better job with its debug information. When you begin to optimize code.

B

And, as you go in and out of, inline functions.

B

Which a lot of them will get in line by the compiler,.

B

Totalview will be given better information to show you, those variable values to trace through the code.

B

You do need to just be aware that, as compilers do optimize, their code,.

B

It just doesn't always run in a sequence., you think about it, running as it goes. Through., that's how it optimizes it and makes it faster..

B

It's still correct,, but it's just running faster..

B

So as you're stepping and going through your code,.

B

It may not follow a logical flow.

B

you'll see that pc jump around as it's going between other inline functions.

B

So, just keep that in mind there. it's not a problem with the debugger,, it's just following the instructions of what the compiler gives it..

B

The other thing you can keep in mind is compilers will be very aggressive in what they store in memory. Cache memory.

B

And so forth and registers., and there are times that you'll look.

B

And say that variable should be in scope,. Why can't I see it. and it's because the compiler will optimize the variable out until it's needed at some point later in a routine..

B

So often you may enter another block and then the variable will come in and you'll be able to see the value. and again,. This is very tricky areas for any debugger to understand.

B

What's going on with the compilers there. so,, it's a good question there.

B

and as a programmer on the outside,. Sometimes you just don't see that insight of what these compilers are doing to make your program run as fast as possible.

B

Dean, we're through all the different totalview action points in ways to use those.

B

And use of watch points.

B

So one thing that is really neat is watch. Points can be used to watch memory, and when it changes,.

B

It will halt the program on there., keep in mind if you watch a local variable,, a local variable is on the stack.

B

and when you exit a function on there,, that stack is going to change which will trigger the watch. Point.

B

totalview will give you a warning to say you're watching memory,, that's on a stack,, so you're aware of that.. It doesn't prevent you from doing it, but it's just something to keep in mind.. It's just, again,! It's some of those internals that creep into what you have to think about as a developer.

B

And,, it's not always pleasant,, but when you get down.

B

To the lower levels of what the compilers do, and what the debuggers have to deal with,, sometimes you just need to have some of that understanding, so you can still effectively debug your application.

B

Let's see variables and scope and focus.

B

Oh yes,, so by default, when totalview hits a break point.

B

For a thread,, it will change the focus to that thread.

B

This actually, this default changed a while ago, but it is something that you can control in totalview.

B

And if you find that totalview is annoying because it keeps flipping to the thread that hits a break point, and you're focused on some other code,.

B

You can tell totalview to not do that automatic switch.

B

And so that's just depending on how you prefer.

B

Or do the characteristics of your application,? You may change that preference.

B

We talked about mpi in this first one here of the different ways to launch.. If you want maximum performance, launch like you, normally would do in a command line or through a batch queue.

B

and then hook totalview into that process of the tvconnect,.

B

Or have totalview launch there.

B

on the reverse debugging side,? We talked.

B

Through some of the,, we hit on a few of the ways there.

B

That you can control the characteristics., so reverse debugging is recording the execution of your program.. It uses a circular buffer to store that history.

B

And when it fills,, it will age out to the old recorded history and replace it with new history.

B

You can control the size of that buffer if you find it's recording too much for your application.

B

So that can help you control the amount of resources it uses..

B

The other thing you can do,, like I mentioned earlier, is don't turn on reverse debugging, until you're close to the point that you want to actually begin recording.

B

and so I'll typically set a break point at some point later.

B

During my execution. and at that point, I'll turn on reverse debugging and then let it run.. So everything beforehand is running full speed.

B

and then I'll take the penalty of doing the reverse debugging after that.

B

Dean talked through a lot of these points on memory, debugging, here.

B

So one of the things, don't turn on all the option progressively work up through and try to find the issues there because the more options you turn on,, the more impact it's going to have on the performance, your application,, the resources that it takes on there.

B

We've talked about the new ui in the classic ui, how to switch between them. and hopefully, we've done a pretty good job of identifying where the gaps still are in functionality..

B

So there's no surprises there.

B

A reverse connect feature.. We use that a lot today. that tvconnect is really powerful..

B

One of the things that I've noted and I've got an enhancement request for my team to improve.

B

On this is,, if you'll remember,, I used the pwd command.

B

Which gives you the full path? Before,.

B

Where you currently are to run an executable.- and I do that so that I fully define the whole path.

B

To my program that is going to be passed to tvconnect.

B

And the reason for that is,, sometimes under some circumstances, tvconnect, can't find it.

B

As it goes through, this launch sequence., and it gives out an obscure message that says something about incompatible execution file format for the target.

B

And it's not quite right., it can happen, but most times it means. I can't find your program. so, the easiest way to solve this.

B

Is to provide the full path to your program., so in the second bullet here, I show changing to use the full path.

B

To my program that I'm to use with tvconnect and that should alleviate that problem.

B

And then of course, there's a lot of other helpful hints.

B

If you go up to a totalview.io,, our health system is there and we have a whole set of video tutorials that are organized into classes and tracks, and much of what dean and I showed today,.

B

You can just watch short, four or five minute snippets of that functionality there to help refresh how to use a particular feature, and so forth.

B

And then the nersc environment is,, this was actually.

B

My first real opportunity, to work with cori.

B

My sport team uses a lot of engineers that use on it., so I'm grateful to woo-sun for giving me access, and get to all of you working on there..

B

There were a few tips that I kind of picked up when going through that, hopefully, will help you as well.

B

So for the remote ui,.

B

I configured the front end using this information.

B

I just named it, cori.nersc.gov., you can name it whatever you want. and then the remote host is just your username.

B

And this is no different than ssh again,. It's what you would do to ssh into the system.

B

If you don't set up your ssh keys or anything,, you will be prompted at the terminal to enter your password., and your otp, like I did, and dean, had to do. Today.

B

But if you set your keys, up, it'll just automatically launch through much like I did when I connected to my system in colorado.

B

And then make sure to give that full path to where totalview is installed., because when we connect into that cori system,.

B

We're running parts of totalview., so if we can't find it there,, then we won't be able to launch.. It will give you information to help you figure out. What's wrong, so--.

C

[Woo-Sun] hey, bill., just to give you the version number to default,.

C

Instead of 2020.3.11.

B

[Bill] you could, yeah. if default goes to that latest version.

B

All the time, absolutely, so.

C

[Woo-Sun], that's what I'm doing. so,! Whenever I got a new default version,.

C

I changed that, create a soft link.

B

C

[Woo-Sun], I mean, it is a totalview.default,, not default.,.

B

[Bill] okay, all right., so totalview.default.

C

[Woo-Sun] yeah, same thing with the memory, memscape.

B

[Bill] okay, all right., so that's good., so you can just use totalview.default then.

B

and that's good, because then you don't need to change it.

B

As new versions of totalview are put out, there.

B

[woo-sun], yeah,, exactly., [bill], okay, and so, and enhanced that we will be adding.

B

To this, process, is to allow you to run a script.

B

Before you do the setup and so forth out there.

B

One of the things that I found I did in my bash profile extension was: I did a module load totalview.

B

so that set up the totalview environment there.

B

so I'm hoping for february,.

B

It should be a fairly simple enhancement that will allow you to run a script that could do say: module load, totalview, module, load, openmpi, whatever it is that you need to commonly set up.

B

For this configuration.

B

keep in mind, you can have as many configurations as you want.. So if you connect and you have a target application,, that's using intel and openmpi and other technologies,.

B

Well, you can have it run, a script that does all that.

B

and then you can set up a second one, that maybe sets up your environment differently for another application., but it's still connecting to cori. But it's just setting it up for that. Application.

B

And that will help you there.

C

[Woo-Sun] yeah, so about that,, I created the remote init script.

C

Which loads totalview.?

C

So you know the demo display client setting.

C

If you click the advanced option, there,.

C

You should have put that the fullpath for that script.

B

[Bill] right, for the rdc.- and we want to have something similar here for the remote ui too.

C

[Woo-Sun] but you don't that kind of thing for the--.

B

[Bill], not yet, no,, not yet., so you kind of have to set it up through your environment a little bit which is not as ideal, but the next release will allow you to run that script.

B

Okay, and then you can see how I insert the tvconnect.

B

Into my slurm script here.

B

So that, once my remote ui is set up.

B

And I submit to the queue if I want to debug this way,.

B

Then, once my job is run,, it will establish a connection back to my totalview.

B

That's all set up and going.

B

and then lastly, if you're, using a pre-allocated node,.

B

I ran tvconnect. and again,. You can see where I'm using that full path.

B

It looks kind of weird but which srun will give you that full path to srun.

B

And then I do the same here on my current working directory.

B

I have it insert that in front of prime count,.

B

This should actually have a slash there, like that.

B

Okay, and those are the patterns that I did as well, today.

B

So hopefully this helps a little bit., I'm not an expert in your system yet., so there may be other things. woo-sun,. What I would recommend, is maybe you and I take some time to refresh.

B

Some of your internal documentation,.

B

Wiki pages or whatever system you use., and we can bring some of this information there and it could be a good reference point for your users.

B

[woo-sun] yeah, that'd be good, thanks., [bill] okay,, all right.. So I will work with you on that.

B

And that covers about what we wanted to go through today.

B

Are questions that I can, dean and I, can answer, or,.

B

And I'm happy,, like I said,, I'm happy to,, I'm going to let dean go if there's nothing else for him. 'cause it's getting late in his day., but I'm happy to stay around.

B

If people were working on the labs and had questions there as well.