National Energy Research Scientific Computing Center (NERSC) Totalview Tutorial, Dec 2020, 9 Dec 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: NERSC Totalview Tutorial - Part1

Description

Part of the Totalview Tutorial at NERSC, December 9, 2020. Please see https://www.nersc.gov/users/training/events/totalview-tutorial-december-9-2020 for more information and presentation slides.

A

And introductions on my side,, so my name is bill. Burns am the product manager and also the team technical lead for totalview..

A

I'm very happy to spend some time today with everyone here and show what totalview can do.

A

When you answer questions and everything to back me up.

A

And assist me through today and do we're sharing.

A

The presentation is my sales engineer: dean steward., I'm based just out of my boston, office,.

A

Dean is based out of the uk and then my account manager.

A

Bruce ryan is also on with us today, so any of us.

A

Will have to answer whatever questions that come up today.

A

Okay,, so let me take a quick look. At,, I kind of worked out.

A

In the agenda with woo-sun before we started- and these are the topics that we wanted to hit.- it's a lot for actually a four hour period,. This is training.

A

We customize it based on the time available.

A

And the customer's needs on here,, so for this four hour block we're taking something that we could depending.

A

On the amount of interaction and help that we do.

A

With the labs and so forth can actually spend two days.

A

we've done that with some of the other deal with labs, we've condensed down what we can here to provide as much value in this four hour block..

A

So we have a lot to cover on here..

A

What we'll find is on the beginning, portions of this presentation, today, we'll cover a lot of totalview.

A

Features and we will do some demonstrations.

A

And then you'll see the use of those features in after the break in a lot of the actual, hands-on piece.

A

That we're demonstrating here.

A

and I'm going to talk about the labs portion here in a second.

A

As well just so that we can understand what those labs.

A

Are and how to use them. okay,. This is interactive. So if you have questions.

A

Or anything like that, during the don't be afraid to either through chat or just bring your question up.

A

Okay,, so let's first let me just skip through that,.

A

So just a quick thing,, like woo-sun, had said to perforce acquired rogue wave,. I think two years.

A

In february, and so perforce has been expanding.

A

Its capabilities through acquisition, much of what rogue wave had for products complimented what perforce mission was and their products.

A

So it was a great fit when they acquired us in there.

A

So we span 80 countries now about 9,000 customers, worldwide.

A

We cover half of the fortune 500 and our customers out there,. So what's nice is that we have a pretty wide.

A

Portfolio now of products in four different areas.

A

So we're going to talk a lot about totalview today, but just so you're aware, with these other products out there.

A

For agile management, code collaboration.

A

And that's where totalview falls under there and the other product that people probably most well know perforce, for is their tradition called the perforce vcs.

A

Version control system- that's now called helix core now.

A

And that falls within the same tier where totalview is? We have a number of tools for own application, management and components within there.

A

And then we have a big push in automated testing with helix, qac or perfecto.

A

Which was another acquisition done about the same time as rogue wave and our clockwork tool.

A

Which provides static code analysis., so there's a wide breadth of other capabilities and products here that we can talk about is any needs that you may have.

A

So let me start off., so woo-sun mentioned some labs that he has made available.

A

Out there,, so there's two files.

A

one is a tar file of nine different pdfs.

A

And they go through each of those pdfs is a specific lab.

A

With an agenda of what to teach you.

A

On how to use a totalview in those areas., so as you open each of those up,, they will have instructions.

A

On which application you're going to use how to set up your environment? And then there is a series of steps to teach you like in lab one,, the bugger basics, startup, basic control,.

A

Of processes and navigation of processes and then going.

A

In from there to examining data, viewing data, editing, data,.

A

So doing data debugging and then so forth across there.

A

Labs five and six will go into the memory debugging and, as woo-sun mentioned, the memory scape tool right now.

A

Requires use of the classic version of totalview.

A

It's a separate user interface, but it's not integrated into the new ui.

A

and just to give you a preview of what's coming. We are actively working on and I anticipate the first.

A

Release of memory debugging, the new user interface in our february release of totalview, we'll have the ability to do leak, detection of your applications and examine the heap blocks.

A

And heap usage of your memory there. and throughout our releases in 2021,, we'll be building.

A

On that initial functionality and eventually deprecating.

A

And replacing the memory scape user interface as everything comes into the new ui..

A

So it's a really nice enhancement that I look forward. To.

A

So a few things I wanted to note in the notes portion on the right hand, side.

A

So I labs five and six do use memory scape on there., so it's not the new ui, like we just mentioned..

A

We provide a series of sample programs in to makefile to build them.

A

The makefile, you want a tailor for it.

A

By default was configured for using the gnu compilers.

A

So if you want to use your default of intel compilers, you can tweak the makefile.

A

one thing I'll note when you do, that is that we have what are called saved. Breakpoint files.

A

In as part of our tutorials.

A

and totalview will load these and automatically plant.

A

A series of breakpoints in the application.- and this allows you to hit the proper points as we go through the steps of the labs. And then you can do the exercise at that portion of the lab.

A

Those are specific to the compiler., so I ran through and compiled everything yesterday using the default intel compiler and I had to recreate those breakpoint files.. So my recommendation is to configure your environment.

A

Do the module load of the gnu compilers and use the makefile and compile them that way? It'll just make it a little bit easier for you to prepare the programs for using them in the labs.

A

The other thing to note is a couple of the programs: do use open mpi., so you'll want to do a module load of open mpi, so your environment set up there before you run, make and then mpicc and other applications that are.

A

Required to build those are ready to go there.

A

And then lastly,, so we will talk about the ability.

A

Of python debugging in as part of the demonstrations today.

A

And there isn't a lab right now. That is specific.

A

To python debugging,, but underneath totalviews.

A

Examples area,. So if you go to, when you do a module load, totalview, and you,, and you then find the directory.

A

Of where totalview is installed, you'll see underneath.

A

There it's actually under the platform., so typically it'll, be under,, say linux, x86-64.

A

You'll see an examples directory. and underneath that is python, examples.

A

And here we have some read me files and how to set up your python environment, how to build the examples that are there and.

A

And then how to debug them.? So what dean and I need to do is formulate this into a lab.

A

So that it'll be another iteration and sequence of the lab on there.. So just as a reference of that being available.

A

Okay,, so in the end of today,, if anybody is interested.

A

In working through any of the labs or questions about building the programs,, I will be,. I will stay on after the presentation and having happy to help on that as well.

A

Okay,, so let me just do a couple of quick slides for people that might be new at a totalview, here.

A

and then we'll start jumping in to a few topics.

A

Little more meet on them., so what is totalview used? For??

A

We really look at it as two ways..

A

We talk about totalview, being a great debugger for finding bugs in your applications, illegal use of data.

A

Or data, that's not being computed correctly.

A

Finding crashes in your application, anything like that, using the leak detection or other capabilities.

A

Debugging, maybe your cuda or anything in there and just making sure that the program is trying to find.

A

The errors that may exist in there., but as you watch what dean and I demonstrate,.

A

And what the labs will show on there when you run through those,? Is it's also just a great tool for doing what we call dynamic analysis.?

A

You can understand, get a good understanding of how your code is running and that may help you to learn.

A

How to formulate, how you may need to refactor your code.

A

you're, building onto your code., it might be working perfectly fine, but now you need to add new capability on there,.

A

Staring at static code. Sometimes doesn't give you that picture that you need to understand. What do I need to do to get this code.

A

To then refactor it, to add new capabilities., so you'll see that there's a lot of pieces to totalview and it'll give you that understanding of how it runs or to fix problems there.

A

The other part to just think about,, as we go through on here.

A

Is you're going to see a lot of capabilities of totalview, whether it's setting a breakpoint looking at data.

A

Or anything else, there, they're, all small little tools.

A

And capabilities of totalview,, but when combined together can solve some pretty tough problems.

A

and that, I think, is the trick of any debugger, totalview.

A

Included on there is understanding its capabilities and how to put them together to solve a tough problem.

A

And that's, one of the goals that dean and I have today.

A

Is to have you understand, what is there so that you can.

A

Apply them as you hit different problems in your code.

A

We will run through the different features and I'm not going to spend a lot of time on this slide.

A

But we are a comprehensive c and c++ and fortran debugger.

A

We've been built from the beginning to handle multiple processes and multiple threads at a time.

A

And you'll see those capabilities and how they're baked into the debugger as we go through different demonstrations. Today.

A

we've been built from the beginning for hpc,. We understand.

A

Mpi, open mp,, we have been working side-by-side with nvidia.

A

Since cuda really became something that was being used.

A

Within the hvc community., so it's more than 10 years plus that we've been developing.

A

Our capabilities for debugging cuda.

A

We will discuss other capabilities, such as reverse debugging,.

A

The ability to bug python,, finding leaks and other features in totalview as we go through.

A

Different demonstrations today.

A

and we work on linux, macos and a variety of unix platforms.

A

So pretty much everything you see today, you can do on the different platforms.

A

So it's quick, just quick summary of the features there so,.

A

And as woo-sun kind of talked about from the beginning.

A

We have a new user interface in totalview., it was put as the default,. I think it was mid. 2019.

A

That we made that switch., the classic user interface, is still available.

A

As we go through some of the features today, dean- and I will highlight where there are some differences still in what might exist in the classic ui versus what, in the new user interface.

A

So you'll find that there are some new features in the new ui that don't exist in the classic ui.

A

and so we're,. We work hard to close that gap and continue.

A

To do that, feedback from users like yourself.

A

Help us understand what features are really important to you and need to be in the new user interface. So I can priority that for my team.

A

in general,, some of the gaps are around visualization of data,. We don't have as, in the old ui.

A

You could visualize a 2d or 3d array.

A

we're going to be embarking on a project for that for 2021 to add visualization support, memory. Debugging is in progress right now..

A

The first release of that is, I anticipate february, to do leak, detection, there.

A

and when I say high scale, mpi support, we've debugged.

A

Several thousand processes without a problem in the new user interface, but we haven't debugged up to 10 or 20,000 processes there..

A

So if you really going high scale,, the old classic ui has been tuned to really hit.

A

Some of those high scales on there. and we'll be doing some of the further tuning on there.

A

So those are just some of the areas there, we'll highlight others as we go through .

A

One of the things I like about the new uis, it's themed., so you can have a light theme or if you prefer, a dark theme,. You can choose that as well.

A

And that's just under the preferences., if you go to the display portion of the preferences,, you can choose whether a,, whether you want the classic.

A

Or the new user interface and if you want the light or dark theme. so before we get into talking about features,, one of the things that dean and I wanted to do was to highlight well,. How do we get in,, especially.

A

To your environment and begin using totalview so,.

A

Woo-Sun mentioned a couple of different initial ways to do this. Using the remote display client for totalview.

A

Or there is x11 forwarding,, but we do not recommend.

A

That, unless you have a really close connection on there, x11 is really not built for long distance. Forwarding.

A

Of displays and we'll find,, especially with the new user interface since it's a rich user interface, that it just may not perform as needed on there.

A

So that's where the remote display client can come in and assist with that scenario.

A

And then I'm going to talk about a new feature that was just released in total new 2020.3, just a week or so ago.

A

And demonstrate how to use that feature to establish a remote bugging session on cori.

A

So I think what we're going to do here is I'm going to hand off to dean and he's going to run through a set of slides here.

A

That detail the remote display, client and, while he's doing.

A

That I'm going to just make sure everything's all set up to then talk about the remote user, interface.

A

So dean,, if I can stop sharing on here,, we go to stop share,.

A

Now you should be able to take over and.

B

[Dean], alright bill, hi everyone.

B

So hopefully you can see my screen.

A

Yes, I can do.

C

B

Okay, so as bill was saying,.

B

One of the ways that we have of setting up.

B

A remote connection is the remote display client.

B

And I see,, we already have a question in the chat window about that., so I'll go through how we set up a remote display, client session., and it consists of three different components..

B

You have the client, which runs on your local machine, which you've seen how you can access that it's provided.

B

As part of the installation.

B

And the server which runs on any system that's supported by a totalview and then the viewer, which is the window that actually appears on the client system..

B

A remote display client is available for 32 bit. Linux.

B

64 bit linux, as well as windows and for mac.

B

So remote display client is free to install.

B

You can install it on as many clients as you need..

B

You don't need a license to run the client, because the license is on the serer side.

B

So that's where the flexlm license manager, sets.

B

And it requires ssh and x windows on the server.

B

So information that the user provides,.

B

You need to provide a username and optionally.

B

You can use a public key file or other ssh information.

B

You need to provide the directory to where totalview is located on the remote machine and optionally. The path.

B

And name of the executable be debugged., and if you need to take several hops in order to get.

B

To your server,, if you have like a login system.

B

Then you can optionally provide additional hops as well.

B

And if you have a batch submissive systems.

B

Such as pbs pro or load leveler, you can configure this in the remote display clients.

C

So a couple of questions.

B

Can we,, can you run the rdc from arm64?.

A

Yeah so I, there are question back of. Are you looking.

A

To debug on the arm or so you'll get up run the arm bug app on, on cori.

A

So from armed cori,, so we don't have a client for that supports arm 64. Yet I can jot that down.

A

As a request to do that.

A

the clients we have right now and just windows, linux,.

A

Linux x86 do the specifics and mac.

B

[Dean] okay, thanks bill.. So this is what the remote display client window looks. Like.

B

You can see here, it's a single screen.. We have a remote host at the top,, a username.

B

So you have to enter your password,, it doesn't cache any passwords.

B

and then you have a path to the totalview executable.

B

And so you can configure either totalview or memory scape.

B

And then under advanced options, you can configure additional options as well.

B

And you can store the sessions you can see here over here on the left,. We've got a session which you can store.

B

You can save your session profiles, so you can easily come back and you can use these again.

B

So this is just about session profile management., and you can share these with other users as well.

C

So let's have a look: how this works: then.

B

So I've got, a linux virtual machine running here.

B

So what I'm going to do is I'm just going to run the remote display client,, which is in my path.

B

And I've already got configured a session for cori.

B

So here, I've got the remote host name..

B

I've got my username for cori up here.

B

And then down here in the path,, I've got the path to where totalview is located on cori.

B

So that's pretty much where you need just,, just those three pieces of information.

B

And then I can click on launch debug session and it pops.

B

Up a window over here., and it's asking me for my password and my one-time password.

B

So I've got a really long password, which is going to take me about five minutes to type in., so just bear with me.

B

And then I need my one-time password as well.

C

So if you set up mfa with ssh proxy.

C

You will not be prompt to enter password plus otp.

C

So mfa works fine.

B

[Dean], as you can see,, it starts totalview.

B

and if I also click on my last session.

B

It brings up the totalview window.

B

And you can see that what running here is the classic totalview user interface.

B

Or the old user interface.. So if you want to run the new totalview user interface.

B

Or if you want to run a script like a slurm script,.

B

You can close the totalview windows down.

B

But I've still got my connection..

B

I still have my connection here. and then I could do say I could do module load,.

B

Totalview and then I could do a module, load,, open, mpi.

B

And then I could start totalview and just give it the minus new ui flag.

B

Which will then bring up the new totalview user interface.

B

And then from here, I can choose a number of things.

B

I can debug a program,, can debug the parallel program.

B

I can attach to a running process and I can load a core file.

B

Or a replay engine recording file.

B

so for example,, if I click debug a parallel program.

B

With browse to a program,, I've got an example: here.

B

Hello mpi.: my system can choose the parallel. System, choose open, mpi.

B

Choose the number of tasks, say four and then start the session.

B

and bill is going to go through mpi debugging with us later.

B

So he'll show you how to set up an mpi debugging session.

B

On cori, using your particular environment.

B

But this is just an example of how you can use remote display client.

B

and then I can set a breakpoint and then I can start debugging.

B

And we can see that I'm running four processes.

B

And we hit the breakpoint here at line 13.

B

B

This is how you set up and how you can run remote display client.

B

Any, is there any questions?.

A

So dean,, you also created a video on this too, right?.

A

So if users wanted to see what you just demonstrated.

A

Again, they can go to totalview.io.

A

And then we have under resources, video courses and it's just a short five minute, video that runs through everything there too.

B

[ dean] absolutely, yeah. so, just to go over it again, what the information you need.

B

Is the remote host,, the username and the path.

B

To totalview on the remote system.

B

But yeah, we have lots of resources available..

B

So if you go to the totalview.io website, you'll be able to access the resources and watch the videos.

B

I'm just going to end that debugging session.

B

So we got some questions.

A

Okay. yeah,, I don't see anything new out there.

A

All right, so dean, do you want me to continue on with the other couple of ways?.

B

[Dean] yeah. so, I'll hand back control, to you.

C

(Dean clears throat) [dean] bill. You got a question on that memory. Leaks detection.

A

Okay., let me check that there,. How does totalview know where to (indistinct).

A

How to detect memory leaks.

A

so in dean's case there,, when you have the remote client connected.

A

And you have that virtual desktop in place.

A

You either the easiest ways to run memory, scape.

A

Which is just the memscape command as part of the totalview module., when you load that and later in the presentation today,.

A

Dean will do a thorough demonstration.

A

On how to find leaks and everything but,, what dean is showing you there and what I'm going to show you next suggest: how do we get totalview set up, so that you could then do.

A

What we're going to show you, through today's session.

A

So you could see in with the rdc.

A

How to set up that remote desktop and you could see on there, you know, deans, is coming from many thousands of miles away connected to your system,.

A

The performance was good., he was able to debug and everything there., and it was once that session set up on there and you have a successful connection.

A

It's easy to just double click on your session.

A

And re-establish that remote desktop in with the rdc.

A

So we'll talk more a bit about memory, leaks.

A

In the second segment, today.

A

I see, question: how does totalview know where the hello world code is?.

A

So when an application is compiled,.

A

Within that, when you build an application with what's called debug information,, typically, that's done by passing the -g command switch to a compiler.

A

It embeds within their hints of where to find the source code.. Now, as long as you haven't moved where your program is,.

A

From where your source code and compilation has been done,.

A

Totalview, we'll just find it automatically.

A

This is a number of search path: algorithms, to locate.

A

Where the sources and pair it up correctly, based on what the compiler has said.

A

now,, if you have moved your program and totally you can't locate the source, there are ways that you can point totalview at the source location,, and it will then pair it up again correctly of what the application is saying versus where the source resides say. Maybe some other space in your desk.

A

And final question,, if you want to use the rdc, then you can still opt to use nx, instead,, yes.

A

(Dean and bill mumbles).

A

Okay,, so let me now go into an alternate way,.

A

That is a new way of doing this., so I want to share this tab. Again.

A

And so I'm just going to go through past dean slide here that he had.

A

Okay,, so for an alternate way of connecting to cori.

A

Is to use a new feature that we introduced in totalview 2020.3 and that's called the totalview remote ui., and so this establishes the convenience of kind of.

A

Of what you saw with the rdc of connecting establishing in the hops to connect to a remote host.

A

But it allows you to run that totalview user interface.

A

Locally on your system.

A

And so I'm not dealing with having to set up a remote, desktop application and dealing with all those bits.

A

And everything going across..

A

What this allows you to do is harness the full power of say your laptop to run totalview and communicate to the debugger.

A

Which is running on a remote system on there.

A

The communications are crossed to secure on there., we can save the sessions and it supports another technology.

A

That I'm going to discuss and quickly demonstrate here called reverse connections., so we move on here..

A

So this is kind of a picture of what remote user interface.

A

Is going to look like. so on the left over here, I've got a remote system and today I'm going to demonstrate running totalview from my macbook pro, and I'm going to establish a connection.

A

To cori and a front end node there.

A

And later on, today,, I will submit a job to get run.

A

Into the batch nodes and using reverse connect.

A

Be able to debug all of that from my laptop.

A

So this is a new capability that, through that I'll make it,.

A

I think a lot easier,. I found myself it's very nice workflow.

A

And easy to establish and do debugging remotely on a system.

A

So is pieces of this. Here is when you connect it all together.

A

the remote ui that allows me to run totalview on my laptop and connect to cori., but when I submit, something I want to debug either.

A

Into the nodes, or even if I've manually run an application.

A

I can use a coupling technology called totalview, reverse connect that allows that to start.

A

And communicate with the totalview that I set up..

A

So how does this look on here?.

A

Before I get into some architectural diagrams on this,.

A

To issue the and start the reverse connect process.

A

All you really need to do is prefix what you're going to run.

A

With the new command called tvconnect., so you can see it as part of a slurm script here that I have my normal s run of the application.

A

That's going to be issued as it's submitted.

A

To the batch system,, but I put tvconnect in front of it.

A

And what that will do is create a handshake between when the job is run,. The waiting totalview.

A

That will accept the debugging session as it gets launched by the submission queue and then from there to the handshake to get that whole debugging session. The interaction debugging session up and running.

A

So this I'll walk through a couple of steps here, so you can kind of see what's happening behind the scenes., so again, we have our front end node here.

A

And totalview is running on here.

A

now, in the remote ui case,. There would be one other box.

A

To the left and that's my laptop, but I could be sitting on their front end node as well,.

A

Maybe I have a x connection, they're already set up with vnc and you can issue the same type of workflow there..

A

So when you run that tvconnect command,.

A

It's going to write a request to a common area.

A

That totalview is monitoring on there.

A

And then totalview will read that request and ask the user.

A

To say, do you want to accept this incoming request?.

A

Everything here is secure., it's all in your home directory.. Nobody else can write to it, but you have control to accept it or not..

A

And then you return a response and if it's a yes,.

A

Then tvconnect will read that response.

A

And then issue, basically an exec to start the totalview.

A

Remote debug servers,, which will then run srun, which then launches the job as in a normal way of starting a parallel job. and once that is up and going that remote totalview server will then communicate back to what we call the totalview client or the user interface.. If it's connected to it in this function. Here to then allow you to start that interactive session.

A

So there's a lot that kind of happens underneath the covers here,. But let me run through.

A

And show you kind of how it works in live here.

A

So we increase the size of these so that they're easier to read.

A

I'm sure that I have, okay.

A

so just to get a live session on cori here.

A

Okay,, so I have two windows open on here.

A

This, the one on the left is on my mac and I'm going to run totalview there.

A

and then the one on the right, I've already logged into cori, and I'm going to use that tvconnect command and we're going to start running. One of the samples that are in the labs called combined on there.

A

so I'll talk about this in a moment. But let's first start totalview.

A

So again, this is starting up on my mac and I'm just going to get the window, should come. Up, pops up in my other display.

A

I'll try and get over on here.

A

One second on here,, I'm going to.

A

One second: I just going to get it.

A

One second, I'm just trying to get the display..

A

I don't know if zoom is messing with the display a little bit and,.

A

Right, lets try one more time,. Let me switch back., I'm just alternating between the themes on here.

A

To see if I can get the display with zoom to work, right.

A

Here we come, okay.

A

okay,, so totalview is up and running.

A

So you'll see with this latest release is the center area.

A

Called launch remote debugger.

A

similar to the rdc, is you can define a set of configurations.

A

On where to launch totalview to debug against,,.

A

To use for debugging., so I have several here that are configured and I have configured for cory on here.

A

So there's three pieces of information you give, and this should feel like the rdc cause it's doing similar in that it has to establish that rogue remote connection.

A

so I've, given it a name, just cori.nersc.gov.

A

And here's my user information., so what it's going to do is ssh from my system, out, into cori on there.

A

and, of course, I'll have to authenticate.. If you needed to go through a gateway system onto other systems,, you can use a comma separated list of hosts that you want to go through on here.

A

And then lastly,, just like with the rdc,, you have to say where is totalview? so where does it run totalview on that remote system?.

A

So now that I have that all set up on there.

A

I can then simply,. I want you to note something on here.

A

Right now, if I look at the command line of what totalview says, it's debugging on my mac, mac, osx darwin.,.

A

So now, when I switched to cori on here,.

A

And I have to now authenticate, I have not, windows over here,.

A

Hold on there is different window to launch everything.

A

So, where I launched totalview from,, it's going to ask for my password and my otp.

A

So one second and again, woo-sun, I need to do the same.

A

As what you suggested for dean and getting those keys set up, correctly.

A

Alright, so I'll put my password in and then my otp.

A

Okay,, so once that's in there what's going to happen, is now totalview, what we call the client portion.

A

Or the totalview debug engine is now running on cori.

A

And you'll see here now I'm connected to a linux environment., I'm no longer debugging on my mac., I'm doing things remotely..

A

I could launch in and start debugging your program like dean, said with the bugger program. and we'll talk more about this as we go through the usage of totalview., but on the flip side, here,. What I'm going to do is now I'm logged into quarry over here., I'm going to use tvconnect.

A

and right now, I'm going to do out is just a simple local application here.. So let's do tvconnect and you'll notice I'm giving the full path. Here, use back to pwd.

A

And combine cause sometimes, if, depending on how environments.

A

Change on here, totalview may not be able to find.

A

Your executable so as always suggest putting the full path.

A

To the program you want to debug on there., so that's why I'm doing that.

A

totalview is getting that request,. Do you want to accept the connection? yes,? I want to accept that connection..

A

So now the program is part of my debug session here.

A

I can see the source and I can start to step into my program.

A

And start doing debugging with totalview.

A

again,, I'm sitting in boston, my totalview user interface.

A

Is sitting on my mac here with me and I'm communicating and running now out in cori in california.

A

And we'll have very good response,, very good ability to look at data, to bug the application, set breakpoints.

A

In everything there.

A

You can do multiple of these tvconnects if you were to end one session and start another one.

A

If you have a job, that's running there, once its accepted you can and run,.

A

You can then begin that in totalview as well.

A

Okay,, so this is an alternate way and I think I know myself.

A

I find it a very convenient way to just get totalview running, log in to cori, I'm working in that environment and then, when I'm ready to either submit a job or just start debugging an application..

A

I use tvconnect and it establishes this connection on here.

A

And I just start debugging within my session here on totalview.

A

Any questions on that?.

A

Okay, alright., let me go back to the slides and see where.

A

So dean and I will be using different techniques.

A

As we demo today,, I will do a demo similar to this.

A

But we'll actually submit an mpi job into cori's queue later on and show how to do mpi debugging in this scenario as well.

A

So, dean,, I think we're onto you to kind of go through startup.. Let me stop sharing.

B

[Dean] alright, thanks bill.

A

And I'll take a look at some of the chat here and see. If it's anything I can answer.

B

[Dean] okay,, so let me share my screen.

B

So hopefully you can all see, start up.

A

Yes, I can do.

B

[Dean] right,, so when you start totalview.

B

For the first time with the new user interface.

B

What you see is a screen similar to this one.

B

And you have a number of options.

B

So, starting from the left,, we can debug a program.

B

So that's straightforward.

B

You, we can debug a parallel program.

B

Again,, that's very useful.

B

It helps you in the sense that you fill in the required information..

B

So you choose the mpi implementation that you want to use,.

B

Number of processes and the number of nodes.

B

We can attach to a running process.

B

And we can load a core file or a replay engine loading file.

B

So this is actually a screenshot from the 2020.2 release.

B

And you'll notice that this reverse connection is there.

B

But remote connect, which builds just shown, isn't on that figure. That's obviously something that's brand new.

B

And we also have a list of what's new., so it tells you what's in the current release.

B

and then over here on the right. You have your recent sessions., so any program that you've recently debugged you can give it.

B

A session name and you're able to just click on that and then you can load up the session very easily and you can start debugging one of your last sessions.

B

So when you debugging,, if you choose debugging a new program.

B

This is the screen you see,, as I said, you can give it a session name..

B

So the program that bill showed us just now was called combined.

B

and you have a path to, to the application,.

B

Optionally, any additional arguments, and you can turn.

B

On reverse debugging and you can turn on python debugging.

B

if it's a mix, c or c++ and python application.

B

We could also choose environment variables.

B

so for example,, if we were debugging an openmp program.

B

We can set the number of mps threads in here.

B

And we can also do redirection of standard input.

B

Then you just click on the live session and each can start debugging your application.

B

If it's a parallel application,, then you choose.

B

Your parallel system from the dropdown list, any of the mpi implementations that we support.

B

Optionally and you specify the number of tasks.

B

Any additional startup arguments, the path.

B

To the application itself and any arguments for the program.

B

And again, you can turn on reverse debugging and python debugging as well.

B

If you want to attach to a running program,, then this is any of the processes that are running on your application., so you may have a long running application. That's running.

B

On your system- and you can simply attach to that.- and you can either do it by choosing from the list of processes from the tree list, here.

B

Or if you know the process id, you can specify that in the list and you can attach to that.

B

And this could be very useful when you're trying to attach.

B

To mpi programs as well.

B

You can also open a core file., so if your application produces a core file,, if it crushes.

B

You get a segmentation fault, and, if your, your systems set.

B

Up to create core files,, then you can load up a core file.

B

And so you specify the location to the core file.

B

And the name of the core file, and also the application name here..

B

If you have a replay engine recording, and we'll look at replay engine a bit later on after the break.

B

If you have a saved replay engine recording file, you can also specify that here and again,, you need to specify the replay engine recording name and the application that that relates, to.

B

And as I mentioned,, we have,, you have a list.

B

Of the recent sessions., so these are,- these are recent programs that you've debugged., you can,, you can click on any of these and that will take you into them as well.

B

and the little pen icon here is just a quick way.

B

You can click on that and then you can start to edit the information that's in the session..

B

So if you need to change anything about that session, you can do so just by clicking on the little pen, icon.

B

Okay,, so that's that startup, was that were there some questions that we need to look at?.

B

I saw something in the chat.

A

So dean, I'm answering a few of the questions going through,. I don't think there's anything specific on start-up. Yet a few remaining ones on rdc that I'm answering now.

B

[Dean] sure, alright.

B

so,. Are there any questions specifically.

B

Related to totalview startup?.

B

Okay, so we'll continue then,. The next topic is user interface, navigation and process. Control.

B

So totalview default views,.

B

The new user interface is built with qt.

B

And we've got a very nice user interface over here.

B

On the left,, we have the processes and threads view,.

B

Next, to that we have the lookup file or function view.

B

And then lastly, we have the documents tab.

B

In the center of the screen, we have the source view.

B

And over on the right, we have the call stack.

B

And underneath that the local variables view.

B

And then, in the center at the bottom, we have the data view,.

B

We have the command line, tab,, we have the logger and we have the input output, tab.

B

And at the far left we have the action points, tab.

B

And the replay engine bookmarks.

B

And if you find you can always drag and drop the windows around, you can resize them..

B

You can basically arrange the view, as you would like to see: it, user interface., and if you want to change that, you can easily do that.

B

From the view menu,, you can reset the view, quite easily.

B

So the processes in threads view shows you the current state.

B

Of your application- and it shows you.

B

All of the running processes and threads in your application.- and you can group these by various different attributes.

B

And that's what we see here in this window here., so you can turn on or off the different attributes that you want to group by.. So if you're debugging a multi-process application.

B

An mpi application, you're, probably going to want to turn on.

B

Process state and process id.

B

if it's a multi-threaded application in that case, you're, probably going to want to look at thread state and thread id.. So you can configure exactly what attributes.

B

You want to group by. and this little arrow here, this little half circle, arrow,, that's the reset.!

B

So if you want to reset it to its default state, you can easily do so.

B

So the source view gives you a view of your source code.

B

As you'd expect,, currently, we only show the source.

B

Whereas the classic user interface you're able to see the source and the assembly code side by side., so if you need to look at the assembly code, then you'll need to use the classic user interface.

B

For the time being. over here on the left, you can see the line numbers.

B

And the line numbers that are in bold are line numbers that you can separate points on.

B

Is there an executable line of code., so the call stack view and the local variables panel.? The call stack obviously shows you the stack frame., so you can move up and down the functions. and the local variables obviously shows you the variables that are in scope.. Then we have the action points.

B

And the replay engine bookmarks tab., so the action points are all of the action points that you've defined.

B

And the replay engine bookmarks are the bookmarks that.

B

You can set when you're doing a replay engine recording.

B

So replay engine bookmarks are a very nice way of being able to easily navigate around your code, just by double clicking on replay engine bookmark.

B

You can go immediately to a particular line of code.

B

And we'll see that later on in an example.

B

And then we have the data view,, the command line view and the logger..

B

So the data view is a really nice window where you can.

B

You can drag and drop the various local variables so that you can keep them in scope, and you can view them.

B

The command line,, you can see exactly what's happening.

B

In the debugging session- and you can also type.

B

Commands directly into the command line as well.

B

And the logger just gives you some additional information.

B

So lookup file or function.

B

Allows you to search for expressions.

B

So in this particular example we're inside this file, here, read expr.c.

B

And if we type,, if we start typing, so example, ex.

B

And totalview will bring up all of the matching items.

B

And you can simply double click on any one of these and it will take you directly to that particular.

B

Even to that file, or to that point in, in the source, code.

B

And then display settings, under settings, you can choose.

B

The user interface style,, so you can choose.

B

The new user interface or the classic user interface.

B

and then, when you restart totalview, that will default to whichever one you chose.

B

And you can also do that from the command line as well as we saw earlier on.. So if you want to start with a new user interface.

B

You give it the dash new ui command, flag.

B

Or if you want to start with a classic user in spaces that might dash classic ui.

B

I'm looking at the totalview toolbar then,, these are the totalview toolbar (indistinct).

B

So the green arrow is the go command, which sets the thread.

B

To running until it reaches a stopping point, which is typically a breakpoint, but it could stop for other reasons.

B

next button across is the whole button which stops the thread at its current execution. Points.

B

The red square button is the kill button.

B

Which stops program execution.

B

The restart button restarts the program from the beginning.

B

So the existing breakpoints and other settings remain in effect..

B

So that's the same as clicking kill, followed by go.

B

Next moves you to the next line of execution.

B

Step actually, steps into a function call.

B

And out steps out of a function call and run to execute all.

B

The lines of code up to a particular line.

B

So, with the run to option,, you can highlight a line of code and then you can select, run to.

B

and it's a very nice way of executing all of the statements up to that particular line..

B

Okay, so looking at the stepping commands in more detail.

B

Then here's an example, we're at line 30.

B

And we can tell that by the program counter here.

B

Which is highlighted in yellow and we're at this line here, setjmp context.

B

And if we were to select a step,, it would move.

B

On to the next line of code,, which would be line 31.

B

So if we select step again,, we then go into the read expression: function.

B

Next would step over it or execute it.

B

And we also have the option of resuming execution from an arbitrary point.

B

So from the thread menu you can choose a set pc,.

B

Which is set program counter.

B

in this particular example. The program counter is pointing at line 1110.

B

And you can select a line of code, for example 1115, and then you can choose a set program. Counter.

B

And what totalview does it will then resume execution.

B

From this line,, but without executing all the previous lines,.

B

Between 1110 and 1115., so basically, your totalview is allowing you to jump control.

B

To this particular line., so that's how it differs from the run to because if you choose the run, to execute all of the lines of code up to that,. Up to this particular line.

B

So it's a way of,. You can just have totalview resume.

B

Execution from an arbitrary point within your program.

B

Okay,, so I've just got a brief demo here, just to show.

B

You some of the things that we've been looking at.

B

So I'm just going to start totalview, start user interface.

B

And I'm going to choose a recent session.

B

And this is called the qt thread: example., I'm just going to click on this and it's going to load your application for me.

B

and this particular application is a qt based gui application.

B

So when I press go to start debugging,.

B

It's going to bring up the application and we can see here that it's a very simple gui,. It's allows us to.

B

Visually see four threads being incremented.

B

And if I go ahead and click start,, we can see that the thread counter each of the threads is going up.

B

And it's just a nice visual way.. So if I right click on that and say always on top, then we can keep that in scope..

B

If I go up here and pause execution by pressing halt.

B

We can have a look at the attributes and these are the attributes that I was mentioning..

B

So we can see here that we're focusing by share group.

B

By thread state and by thread, id,, which looks.

B

About right for this particular multi-threaded, application.

B

If it was an mpi application, we'd probably want to turn on process state and process id.

B

And you've got various different options.. You can also choose to source line as well.

B

So you can turn on source line.

B

And we've got these radio buttons here which have different views.. So it's a nice way of choosing the focus.

B

And you can also focus by tree view or by a list view as well..

B

If you have lots of,, if you're running with lots of processes in threads, probably the tree,.

B

The tree view is the more appropriate way of focusing.

B

So if I want to change focus at any time, you can see here at the top,, I'm focusing now on process. One thread one,, which is a 1.1.

B

I want to change the focus at any time.. I can just double click on a thread. and now I'm focusing on thread 1.8.

B

If I want to have a look at the source code,, I can go over to my call stack and I can see here,.

B

I've got some c++ functions. Here, here is the do work function,. I can easily focus on that.

B

And now I'm inside this, do work a function over here.

B

Also, I can, from the dropdown list.

B

At the moment is set to group control,, which is all of the processes and threads, I can from this dropdown list. I can choose an individual process or an individual thread.

B

And then all of the toolbar buttons will just activate.

B

On an either a process or an individual thread,, whichever is in focus.. So if I choose an individual thread now, I'm focusing on thread 1.8, and we can just step that individual thread.

B

So if I start to run that individual thread.

B

I also need to run the main gui thread.

B

Which is either thread 1.1 or 1.2.

B

It's usually the one that I don't choose., so it's probably 1.2.

B

and then, if we run that,, then you can see over here.

B

In my qt application, that we're incrementing.

B

Just one of the threads.- and this is a thread 1.8 being incremented.

B

We can see that thread. Counter is just going up., so using the totalview controls, we've got very fine.

B

Grain control over individual threads and processes.

B

And this is a really powerful feature in totalview.

B

If I go back to the group control., I can then pause the application, change it back.

B

To group control,, which is all of the threads.

B

and then I can resume execution and now you can see.

B

That all of the threads are being executed in this particular application., so totalview lets you control individual threads.

B

And processes- and this is great for things.

B

Like race conditions and deadlocks.

B

Let me just stop this particular application.

B

And exit out from here., so this was just a summary,, a recap of the things we've looked at in the user, interface.

B

So any questions about the user interface controls so far?.

B

B

How we doing with,.

B

Okay, not related to a user interface.

B

Other questions?.

A

Yeah I know,- I don't have any other specific to what you've been going over yet dean,, just chatting on a few other things around rdc, still.

B

[Dean] yeah, okay., so let's carry on and move to the next topic.

B

Which is action points.

B

So we have four different types of points.

B

That we can use within totalview., we have the standard breakpoints,, which we've already seen.

B

We have evalpoints,, we have watchpoints.

B

And we have barrierpoints and we'll cover each of these in a bit more detail.

B

So setting breakpoints is very straightforward.

B

As I mentioned earlier, any line, that's a bold. You can set.

B

A breakpoint on- and you can do this by single clicking the line number, or you can right-click the line number.

B

And use the context menu separate point.

B

or you can go to the action points, menu option.

B

And use the separate point option there.

B

so there's three different ways of setting breakpoints within totalview..

B

They all have the same effect and you can view your breakpoints in the action points. Tab.

B

So it will show you all of the active and non-active.

B

Breakpoints that have been set for your particular application.

B

So this is just to reiterate why I mentioned you can set it.

B

From the action points menu at location.

B

and here's an example, put a setting a breakpoint on line 119 in the file readexpr.c.

B

So that's,, if you want to set it on a particular line, number in a file,, that's how you do it using the hashtag.

B

You can also set it on a function,, so if you know the name.

B

Of a function in this case, draw a circle.

B

You can type the name of the function in there and it was set and totalview will set the breakpoint at the first executable line in the function.

B

And pending breakpoints,, this is when you set a breakpoint on the library.

B

That hasn't yet been loaded into memory and totalview.

B

Will set a pending, breakpoint.

B

and you'll, see that it's upending breakpoint, because in the action points tab it will say (pending).

B

And then, when the library is loaded into memory,.

B

That breakpoint will become active, so it will no longer be pending., and we will see this later on when we look.

B

At the python example.

B

You can modify breakpoints., so if you right click on a breakpoint.

B

You can enable disable or delete the breakpoint.

B

And you can also adjust the breakpoints width.

B

So you can change it from a group which is all the running.

B

Threads in all the processes. In the group.-.

B

You can change it to the process, which is just the running.

B

Threads in the process that contains the thread that hit the breakpoint. or you can an individual thread, and this is the first thread.

B

That executes to this breakpoint., so when you set your breakpoints,, you've got again.

B

Very fine grain control over where it's group processing.

B

Of thread that you will stop when a thread hits the breakpoint.

B

We also have evaluation points and evaluation points.

B

Are a really nice feature that allows.

B

You to basically stop the process and its relatives.

B

And you can use this to test out potential fixes or patches for your program.

B

Without having to recompile your application.

B

So you can test out as a small patch inside an evalpoint.

B

And if that patch fixes the problem.

B

Then you can roll it out into your production, code.

B

You can also include a goto, so that will transfer control.

B

So basically, what you would typically do is you would write a small patch and then you would have a goto.

B

In order to jump over the lines of code in the application that you didn't want to run., you don't want to execute the code. Twice.

B

And you can also execute a totalview function and you can set.

B

The values of your programs variables.- and you can also use it just to just to do a printf,.

B

So I know lots of people like printf debugging..

B

If you want to put a printf inside your evalpoint,.

B

Just to print out the value of a variable to you to the command line, then you can do so.

B

So here's some examples of evalpoints.

B

And you'll find more examples in our documentation.

B

But as I mentioned, here's an example of using a printf to print out the value of a result.

B

Here's an example of using a goto.

B

so basically transferring control to a particular line.

B

Of code, and you can do that to jump over a section of code.

B

Typically,, you would do that after you use the patch.

B

To try and fix a problem in your code.

B

And you can also use it to stop a loop after a certain number of iterations., so in this place, we're checking. If I mod 100.

B

Is equal to zero and in that case it's going to print out.

B

The value of I and then it's going to stop.

B

so $stop is one of the reserved totalview.

B

Key words that you can use inside an evalpoint.

B

Evalpoints are very useful and very powerful.

B

The third type of action points we have watchpoints.

B

And watchpoints are set on a specific memory, location.

B

So what happens is execution stops when the value stored.

B

In that memory, location, changes.

B

And the difference between breakpoints and watchpoints.

B

Is that a breakpoint stops before an instruction executes.

B

Whereas a watchpoint stops after an instruction executes.

B

So if you want to check when the value of a variable changes.

B

You can set up a watchpoint on it and then totalview.

B

Will stop execution when that value changes.

B

So here's some examples of some watchpoint expression.

B

And a totalview has got two variables that are used.

B

Exclusively with watchpoints., so $oldval is the value of the memory location.

B

Before the change and $newval is the value.

B

Of the memory location after the change is made.

B

So here's an example here, we're checking if a particular.

B

Variable is not equal to either 42 or 44.

B

And then we're setting the new value and the old value.

B

And then here's a second example is checking.

B

If the old value is greater than zero.

B

And the new value is less than zero and if so,, then it's going to stop execution.

B

So that's again, using the totalview reserved keyword $stop inside the watchpoint.

B

And the last type of action points that we have are barrier breakpoints.

B

and these are used for synchronization.

B

They're used to synchronize a group of threads or processes defined in the action point.

B

And basically, what happens, is you create your barrier.

B

Breakpoint and then all of the threads or processes.

B

Will be held until all the threads and processes arrive.

B

At that particular barrierpoint., so if you were debugging for example,, an mpi application.

B

With 16 mpi ranks and you have to set a barrierpoint.

B

Then totalview, you would hold all of the mpi ranks.

B

Until all 16 ranks have arrived at that particular barrierpoint.

B

and then when they have all arrived at that barrierpoint,.

B

That barrierpoint would set to be satisfied and totalview would allow the execution to continue.

B

So, the next time you press any of the go or next or step statements, it would advance all of the mpi rank.

B

By default,, totalview doesn't save your breakpoints.

B

So if you want to save them,, you can either click.

B

On the save breakpoints option and save them to a file.

B

Or you can turn on the save action points option.

B

So you have two different ways of doing that., so the default is the action points aren't automatically saved,. If you want to save them,, then either save them. When you finished the debugging.

B

Or you finished, setting your action points or turn on the option to save the breakpoints automatically.

B

Okay,, do we have any questions about action, points?.

A

Nothing in the chat, dean,, so far.

A

B

Right, so will carry on them with the next topic.

B

Which is examining and editing data.

B

So the call stack view consists of two different panels.

B

We have the call stack panel and underneath we have the local variables panel.

B

And the local variables panel displays.

B

All of the variables for the thread of interest.

B

So the data view panel could be used.

B

In order to drag and drop the local variables.

B

And you can do that either by right, clicking on the variable in the local variables tab.

B

So if you right, click on it, use, the right mouse click.

B

You'll have the option to add to data view,.

B

And alternatively, from the local variables window,, you can just simply drag the variable across.

B

Into the data view, and it will then be available.

B

For you to view inside the data view.

B

If you have an array or a structure- and you want to view.

B

All of the elements of that array or that structure.

B

In the data view,, you have an arrow here.

B

If you just click on that arrow, so that it becomes a downward facing arrow and totalview will expand.

B

Out all of the elements of the array- and you can see the entire array.

B

If you want to dive on a single individual element,.

B

Within the data view,, you can do that.

B

and a total totalview will show that element for you within the data view panel..

B

So here, we're looking at item seven inside an array.

B

And if you right, click on that and select dive,.

B

Then totalview will display that as a separate line item within the data view, panel.

B

You can also type expressions directly into the data view.

B

And a totalview will display the value of that expression.

B

And you can increment variables as well,, so you can perform addition, subtraction, et cetera,.

B

Within that particular data view and totalview.

B

Would display the value for you as well., so you've got quite a lot of flexibility.

B

Within the data view for displaying variables.

B

And for incrementing variables, as well.

B

If you dive on a variable, isn't dereferenced automatically.

B

So if you have a pointer,, for example, this is a pointer.

B

To a string,, you can make your editable.

B

And you can dereferenced the pointer and then totalview displays the variables value.

B

You can also cast variable types to a different type.

B

And typically, you might want to do this.

B

If you have a pointer to an array- and you want to display the entire array., so you just need to tell totalview that what the size.

B

Of the array is just by double clicking inside the variable type, and then you can specify the number.

B

Of elements in the array and then totalview will display the entire array for you.

B

If you're viewing data in fortran and you're looking.

B

At a module,, then totalview displays in the call stack.

B

It displays the qualified sub-routine name.

B

And the local variables as well appear.

B

In the local variables tab qualified with the module name.

B

(Dean clears throat) for each common block within a sub-routine.

B

Or a function, totalview creates an entry.

B

In that functions, common block, list.

B

and the names of the common block members.

B

Have a function scope,, not global scope.,.

B

So if you select them in the call stack view.

B

The common block and the variables appear in the local variables panel.

B

And if you have any user defined types in the local variables panel,, you can add these.

B

To the data view for more detail as well., so again, dragging and dropping over into the data view,.

B

Or right clicking and selecting add to data view.

B

Okay, any questions so far?.

B

Okay. so let's carry on to advanced c++ and data debugging.

B

So totalview understands many of the stl containers.

B

Things like arrays, lists,, tuples,, maps,,.

B

Sets, vectors and others..

B

So when you're, looking at an array or a list or a set,.

B

Totalview won't display the raw stl data.

B

As we see down here on the bottom, what you'll actually see.

B

Is in this particular case, we're looking at a map,.

B

That totalview we'll show you the key and the value there.

B

So, in other words, it's giving you the information that you're most interested in seeing.

B

Totalview understands the stl containers and it automatically does the transformation for you. So you don't need to do anything.

B

In order to see the values.

B

We also have advanced support for c++ 11,, 14 and 17.

B

Features including things like lambdas,, smart, pointers,, auto types,, et cetera.,.

B

So if you're working with the latest c++ features.

B

Then totalview will display those correctly for you.

B

If you're working with an array.

B

And you want to be able to slice the array or stride.

B

Use the striding or filtering, in the classic user interface.

B

We have this array window which is, you can get this.

B

By right, clicking on an array and selecting dive and total will pop up this variable window.

B

And then you can type an array, slice or a filter, expression.

B

And totalview will display that, for you, correctly.

B

So that's in the classic user interface.

B

And also, if want to look at an array in the 2d table, format,.

B

Again in the classic user interface, we have this array, viewer and you're able to select.

B

Any two dimensions: the I and the j column here.

B

So if you have a three or a four dimensional array for example,, you can specify any two dimensions.

B

Totalview will allow you to choose the start index.

B

And the end index and optionally a stride pattern as well.

B

If you only want to see every say second or third element.

B

In the array and the array, viewer is scrollable as well.

B

So you can scroll vertically and horizontally.

B

Again, if you want to use this array, viewer you'll need to use the classic user interface at the moment.

B

We also have array statistics,, so if you're looking at an array,, you can bring up.

B

Statistics about the array, things like the mean,, the max,.

B

The mediums, standard deviation and a check sum as well.

B

So quite a lot of information about the array itself.

B

And as bill mentioned at the beginning of the presentation.

B

We have the ability to be able to visualize array data.

B

And this visualizer is available through the classic user interface and you can do it through the window..

B

So if you basically dive on an array,, you have the array.

B

The variable array window,, you can then select visualize.

B

And totalview will visualize the array for you.

B

And you can also do it programmatically as well., so you could create a breakpoint,, an eval breakpoint.

B

And you could use the $visualize and then the name of the array and totalview would,.

B

It would hit the breakpoint and then it would pop up the visualize window as well..

B

So it's a nice way of being able to view.

B

Your data graphically,, which in a lot of examples,.

B

Is more meaningful than viewing an array of data.

B

You can particularly look for outliers in your array, data.

B

Are we getting?, we do.

B

We also have the option dive in all.

B

So dive in all allows you to easily see each.

B

Member of a data structure from an array of structures.

B

So in this particular array,.

B

We have an array of structures.

B

And each structure contains an element.

B

I can say, if you right, click and say dive in all.

B

Totalview will create a completely new array for you, but only focuses on the a element.

B

From within the original structure.