National Energy Research Scientific Computing Center (NERSC) Data Seminars Series, 9 Aug 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: 2022-08-09 - Gene Cooperman - Transparent Checkpointing: a mature technology enabling MANA for MPI

Description

NERSC Data Seminars Series: https://github.com/NERSC/data-seminars

Title:
Transparent Checkpointing: a mature technology enabling MANA for MPI and beyond

Speaker:
Gene Cooperman, Khoury College of Computer Sciences, Northeastern University

Abstract:
Although transparent checking grew up in the 1990s and 2000s as a technology for HPC, it has now grown as a tool that is useful for many newer domains. Today, this is no longer your grandfather's checkpointing software! In this talk, I will review some of the newer checkpointing technologies invented only in the last decade, and how they gate new capabilities that can be adapted in a variety of domains.
This talk includes a tour of the 15-year old DMTCP project, with special emphasis on the latest achievement: MANA for MPI -- a robust package for transparent checkpointing of MPI. But as a prerequisite, one must have an understanding of two advances that brought DMTCP to its present state: (i) a general framework for extensible checkpointing plugins; and (ii) split processes (isolate the software application to be checkpointed from the underlying hardware).
In the remainder of the talk, these two principles are first showcased in MANA. This is then followed by a selection of other domains where transparent checkpointing shows interesting potential. This includes: deep learning (especially for general frameworks), edge computing, lambda functions (serverless computing), spot instances, containers for parallel and distributed computing (Apptainer and Singularity), process migration (migrate the process to the data in joint work with JPL), deep debugging for parallel and distributed computations, a model for checkpointing in Hadoop, and more.

Bio:
Professor Cooperman currently works in high-performance computing. He received his B.S. from the University of Michigan in 1974, and his Ph.D. from Brown University in 1978. He came to Northeastern University in 1986, and has been a full professor there since 1992. His visiting research positions include a 5-year IDEX Chair of Attractivity at the University of Toulouse/CNRS in France, and sabbaticals at Concordia University, at CERN, and in Inria/France. In 2014, he and his student, Xin Dong, used a novel idea to semi-automatically add multi-threading support to the million-line Geant4 code coordinated out of CERN. He is one of the more than 100 co-authors on the foundational Geant4 paper, whose current citation count is 34,000. Prof. Cooperman currently leads the DMTCP project (Distributed Multi-Threaded CheckPointing) for transparent checkpointing. The project began in 2004, and has benefited from a series of PhD theses. Over 150 refereed publications cite DMTCP as having contributed to their research project.

Host of Seminar:
Zhengji Zhao, User Engagement Group
National Energy Research Scientific Computing Center (NERSC)
Lawrence Berkeley National Laboratory

A

Yeah: okay: let's get started welcome to the nurse data seminar. Today, I'm very happy to have a professor, jean cooperman from northeastern university at nursk to present a talk on transparent checkpointing.

A

Professor cooperman currently works in high performance computing. He received his bs from the university of mexican in 1974 and his phd from brown university in 1978..

A

He came to northeastern university in 1986 and has been a full professor there since 1992..

A

His visiting research positions include a five-year idex chair of attractivity at the university of toulouse, cnrs in france and sabbaticals at concordia university, at cern and in india, france in 2014.

A

He and his student xin dong used a noble idea to semi-automatically, add multi-threading support to the million lines info code coordinated out of cern. He is one of the more than 100 co-authors on the foundational synt4 paper, whose current citation count is thirty. Four thousand professor cooperman currently leads the dmtcp project for transparent checkpointing.

A

The project began in 2004 and has benefited from a series of ph diseases.

A

Over 150 referred publications cited dmtcp as having contributed to their research project.

A

Let's welcome professor coppermann.

B

So, thank you very much for the uh privilege of speaking here at nursk. uh It's a pleasure after we've been uh collaborating for two or three years so.

B

So, in any case, uh I guess I started in transparent checkpointing a little more than 15 years ago now and for me, I've always been fascinated by it. The idea that you can stop a computation take that file put on the usb key, take it home and just start and then continue it and it just continues. It always seemed like magic to me.

B

So originally I thought this is going to be a one-time project. You do it for uh who knows your pc and then you're done, but at each point I uh was fascinated by the as new hardware appeared. Well, that's interesting! I wonder if you could checkpoint that.

B

So, uh let's start.

B

What's happening here there? Okay, so this gives you uh an idea of where the talk is going. First I'll talk about the mtcp itself in mana. There are a bit of the history and then two key ideas which have made it possible now to in fact, I would say, make it practical to transparently checkpoint mpi, and this has been a long-term dream to transparently checkpoint mpi over a large cluster, with good accuracy, good efficiency and so on.

B

So let's take a tour to see how we got here after that, uh we'll talk about uh mana itself uh and at this point parts three and four in the grade area uh I'll probably have to emphasize one or the other. I don't think there's time for both, but one is maybe a tutorial of some of the internals of mana how it works, why it works, what we're doing differently from people who have tried this in the past and then part part 4.

B

It would be a short overview of some other applications of transparent checkpointing. Besides checkpointing mpi and as you'll see, there's a kind of theme going on here.

B

So we'll go through this part fairly quickly, but, as I said, it has a history here are some notable papers from our website where we list papers that have actually used the mtcp in their own work, so intel corporation.

B

In fact, they've been funding us for a number of years proceedings of the national academy of science, examples for super computing, even back then, and so on, uh and it goes on and on so what is checkpointing uh good for?

B

Well, it's obviously hpc, uh but even there in hbc, in the old days, people were saying: well, once you have a thousand nodes, you'll probably have a crash every two hours, you'll need a checkpointing just for fault tolerance, and it turns out that's not really the killer app for super computing, as many of you already know, the killer. App really is that you can chain together multiple allocations and make them seamlessly appear as one allocation at the same time.

B

To do this to do this, with no burden on the end user, the end user doesn't have to rewrite their application, but here are some others, so obviously we have full tolerance uh yeah. I guess I can point in the room and for those on zoom you'll just see the slides anyway. Oh I know I should use the cursor and that's how fault tolerance uh oops wrong way process migration debugging.

B

So you just take periodic checkpoints. When you hit a crash, go back to the previous checkpoint and then go forward from there slowly to see why you hit the bug, you could save graphics. Imagine a video game where maybe there's no save point, but you can create one save and restoring workspace.

B

There are various kinds of interacting interactive sessions. Matlab would be a perfect example. Matlab has a command to save workspace and very often it does work, but not absolutely in every situation. Matlab has plug-ins a lot of complex internals.

B

So if you want to be really sure about saving your workspace, hopefully the transparent checkpointing that we provide gives you a better uh approach in those cases uh faster, warm startup.

B

So maybe it takes 15 minutes to start upload all the data initialize and get going so after 15 minutes just take a checkpoint and then you'll restart next time and you can restart in less than a minute instead of taking 15 minutes the ultimate bug report. So the developer keeps asking you well: why is it crashing?

B

So now we have the final answer to that developer. Just take a checkpoint periodically and when it crashes send the checkpoint to the developer and say now you finish running it and you'll see the crash that I've been seeing okay. So there are two big ideas that have brought us to where we are in terms of checkpointing.

B

So here's the first of the two big ideas: uh it's um virtualization. We often call this process virtualization. So everybody knows about the other kinds. There are virtual machines.

B

There are containers that provide a lighter weight virtualization and everybody knows that there are virtual machines. There are containers and then there's the bear machine with process virtualization.

B

We can also virtualize on the bear machine the bear operating system.

B

So the first example I'm going to give the example for process ids, but in fact it works uh on any kind of namespace, perhaps file names you're on a super computer you're running in the file, which is called slash, slash, slot, 17, slash my file and then on restart you're on slash batch, slash slot three slash live file, oh no, but uh how will my software know that the name of the file changed?

B

So that's another example where you want to virtualize names, ids and so on.

B

In fact, if you have a friend who is building a checkpointing system- and you want to know uh what is the main approach that your friend is using, you can find out by asking a friend one simple question: ask them how they checkpoint and restore process ids.

B

So you of course you can change the kernel itself. Another way is you could use uh tricks with proc maps? There are certain issues there. Maybe somebody else is already using the process id that you wanted to reclaim.

B

So maybe you have to kick everybody off or maybe you'll do everything inside a container and inside your container. You control the process ids, but our idea is: let's make it very general, so we can coexist with all other processes on the bare machine so that we don't have to use containers and virtual machines if you want to you can, but you don't have to so, let's go through this. Hopefully the diagram is mostly self-explanatory, but I should say a little, um so we have a user process, the the joke that we tell each other.

B

Is that never let the user know what their real process id is. So this user process calls get pid the kernel says sure your process id is 2652, but we have a library of wrapper functions that wrap every system call in lib c which might use a process id.

B

So we see the 2652 we quickly copy into our private table 2652, and we remember that this is virtual process id 4000 and then we tell the user your process id is four thousand and now, if everything crashes and they have to restart the user process, remembers ah my process id is four thousand.

B

Let me do something with it and, of course, when we restart the kernel is going to say that you have a different process id, but as we restart, we just update the real process ids according to what the kernel tells us now, the user says I'd like to kill process id 4001.

B

We look it up in our updated table and then we tell the colonel, he doesn't really mean 4001 or she doesn't really mean that what he or she really meant was- and we look it up in the table and then the kernel kills that for us, yeah.

C

So so does that mean that I you have to basically change the ps command, the queue command so always ld preload launch the ps command q command.

B

We do some of that uh not completely, but we do it enough that we even virtualize the proc file system. Oh so, if you look up slash, proc, slash eid, 2652, uh actually on unreasonable, we will virtualize that as slash prox, slash 4000, and then we will uh give that to the user code.

B

So even that works and ultimately that's what's at the heart of the ps command. So typically, the ps command will continue to work. For that reason,.

A

Since we want to answer another question online.

B

A

So what is the overhead of the virtual piv method.

B

So the question is: what's the overhead of the virtual pid method, and usually it's really tiny, because the way we do this is we put a wrapper function around each hole to ellipse c, but that wrapper function is incredibly lightweight, because all we're doing is either updating this table or accessing the table to do the translation and that's much much faster than most lipstick calls.

B

So, in our experience, even the jitter by jitter, I mean the time difference between. If you run it once and run it the second time that jitter will usually be more than our overhead good. So now that we know what we mean by the process, virtualization, let's go on to the second big idea.

B

So the second big idea is something that we call split processes and- and this is uh came about more recently about three years ago- it's in the thesis of one of our phd students uh former now- and he graduated and also in a conference paper down here, hpdc.

B

So what's the idea? Well, normally we just load one program into one process and everybody's gotten used to the idea. There is a process table with a process id and the name of the program in that process, but we play this trick.

B

It's mean it's not you, oh okay, so everybody else can still see this yeah, oh good thanks. um So we played this trick where we managed to load two programs into one process and how do we do this.

B

B

If we load one program into the process, the kernel knows how to handle that, and then we imitate what the kernel would do to load a second uh process in.

B

uh It turns out that this is a lot easier than you would think, because the kernel really only understands the old model that we all learned as undergraduates text segment, data segment and stack. The colonel doesn't really know about dynamic libraries at all, dynamic libraries are handled in user space, and so, when you, when the kernel loads, a program, in fact, it's not loading the program a.l that you specified.

B

Instead, it's it's loading. The dynamic loader program and the dynamic loader program has an argument. A dot out and the dynamic loader program will look up where the dynamic libraries are and that's great, because now we can make just a tiny change in the text, stack and data model of loading and that's it and we automatically inherit the same dynamic libraries.

B

So now, we've got two programs in a single process, maybe even a single thread for two programs, and the two programs means that you now have two copies of live c. Two copies of whatever libraries were loaded to library, search spaces.

B

It works everything, so we like to call them the upper half program and the lower half program and the upper half program for us is going to be the mpi application and the lower half program will include the actual mpi library, along with network library, lib c and so on, and for us this solved a crucial problem. That was that we think was the reason why it took so long for people to finally uh learn to checkpoint mpi in practice they wanted to. For many years.

B

There was an excellent effort, really well-known blcr coming from berkeley labs here and that was used uh for a while, but ultimately, as the linux operating system grew that blcr edward failed to keep up with it. So we wanted something where we wouldn't have to constantly change it every year to track linux. That would just automatically track as linux changes.

B

So how does this work? Well, we have the mpi application. The mpi application might call mpi init. Well, we have our own wrapper function here, also for the mpi functions, so they call mpi init. We look up the address in the actual mpi library down here and we call this mpint.

B

So this is a lower half program that we wrote. That's our program, that's not part of what the user wrote. The user controls only the upper half so now, if they ever want to do anything with the network or with mpi, they go through the lower half and that's great for us, because now, when the user says it's time to checkpoint, we know for every memory segment either. The memory segment is part of the upper half or the memory segment is part of the lower half one or the other.

B

So when we checkpoint, we save the memory segments in the upper half only that's it. We do not save any of the memory segments of the lower half and then when it's time to restart well, we know how to restart the lower half. That was our program so typically on restart. We will call our lower half program first.

B

If we originally had 100 mpi processes or ranks we'll call it with the parameters to set up 100 ranks, the mpi library will call the network. Now we have 100 processes spread across the network and now each process we'll call well mpi, emit privately and asked the mpi library.

B

What is my rank in mpi rank, of course, is the id and mpi and so to ask what is my rank and the mp? The real mpi library says here: is your rank 17 and then it goes looking at all of the checkpoint image files and it finds one for rank three four five six and then eventually finds the one who likes 17..

B

It restores the checkpoint image for rank 17 in the upper half, and now we can continue executing again. So this is really interesting. It means we can pull tricks like if we don't care if two mpi processes were on the same node or on different nodes, because it doesn't matter if they're on the same node or the different nodes, the mpi model, the mpi library, doesn't distinguish and that's all we care about and then so, we'll simply call npi send down to the mpi library. And it's up to that mpi library to decide.

B

Is this on the same node or a different node? Maybe it was on the same node before checkpoint. Maybe now on restart is on different notes. Has nothing to do with us. We just handle the upper half. The lower half uh is now, hopefully future proof, and so we won't get get caught by having to constantly upgrade every time. Linux changes at least that's the plan.

B

So here we are doing this again uh this here we see it in a bit more detail, so the lower half recall is our program.

B

Our program will uh basically publish the addresses of every mpi function in the actual mpi library, and then it will register or it'll place that in a well-known location, the upper half. Now whenever it wants to call mpi from the application, it goes to a stub function, our pretend npi library, our pretend mpi library, looks up the address and then calls the actual mpi function down here and it works. Even if there's just one thread, it works. That one thread is sharing the upper half and the lower half good.

B

So now, you've seen the two big ideas. Now, let's talk about mana itself, so for mana, here's some of the history of how we got here. As I said more than 15 years ago, I just fell in love with the idea of checkpointing. We did what I thought was really cool in the original dmtcp um we were able to transparently checkpoint well we're done well what about high performance computing? That was starting to become much bigger and standardizing on mpi, but in those days mpi was usually using tcpip.

B

So that's not really a problem for us. We can virtualize all the ip addresses just the way we virtualize process ids, so we're good. We showed how to do that. We're done uh no in hpc, then they brought in infiniband uh so and that's rdma totally different technology.

B

This simple virtualization of ids was not enough. We had to create a shadow structure for the infiniband structure and there's a longer story. A student got a phd thesis out of it and some very nice work and then okay, we're done well actually there. Now there are newer networks, there are infinite band extensions.

B

There is until on the path cray aries now crease and now also create slingshot, and it looks like it's going to continue like this there'll be more and more networks. So now what do we do? So?

B

This is the point where I could see that I wasn't going to be able to bring in enough phd students one for each new network that wasn't going to work, so we needed a new idea and the new idea came out of the thesis of rohan garg, and here the idea was split processes which I just told you about, and now the great thing is we don't care what the network is, because we stop at the mpi interface and mpi wants to use a network go ahead.

B

It has nothing to do with us so and then uh another student, in fact one who will speak today, twinkle jane, took this idea one step further and showed how to make this work for cuda in a work called crack, and that also, I find very interesting work for certain reasons.

B

It's more difficult than mpi, for example, with mpi mpi mostly assumes that the mpi library is is not going to be that the app the user end user application will allocate memory and the npi library will not allocate library in a way, that's obvious to the application, but in cuda they change that anybody can allocate memory. The cuda library can allocate memory the end user application can allocate memory.

B

They even have something like a kind of a unified memory address space, similar to the idea of virtual memory in classical operating systems and therefore a page on the gpu and a page in the host operating system. They should play nicely you should be able to page in and page out, between the gpu and the operating system.

B

So that's another work which uh I think uh twinkle jane will talk about and obviously the reason that's interesting is a lot of the super computers today are using hybrid systems, gpu plus cpu, for example, uh perlmutter here at nurse okay- and this has been a robust elaboration, three ways with obviously nursk. Otherwise we have no place to do our work.

B

uh Memberge who's uh is represented here. You've heard other talks and uh our group, the mtcp, coming from northeastern university, largely although now we've students have graduated and other people have become interested outside of northeastern okay, and what we see is that the killer app is not fault, tolerance, the killer, app is to chain allocations together, and there are different reasons why you might want to chain allocations together.

B

Maybe it's because the assist admin will give you a 48-hour allocation, maximum and they're forced to do that, because they have to worry about bringing down parts of the system for maintenance. All sorts of things happen.

B

So they have flexibility to keep the whole administrative system going often they'll limit you to 48 hours, but maybe your job takes one week or another example is after they fill up most of the supercomputer with a large job. They have some notes left over, and so they can offer two hour allocations, but you have to give it back quickly because they might need it soon and they might even offer a discount.

B

So can you chain together allocations of two hours each, and hopefully this reminds you of things like spot instances in the cloud where people try to do exactly the same thing.

B

So this, I think currently is the killer application behind checkpointing chaining together multiple applications in hpc in the cloud and in other cases, and then just a small comment. As we approach extreme scale computing, we I'm not trying to claim that once we have uh a million cores uh that mana alone is going to checkpoint everything in one huge global checkpoint.

B

All I claim is that, as we continue to scale up, this has to be one piece of the puzzle as we assemble together an ecosphere there are other strategies, but eventually the strategy of last resort is save. Everything and transparent. Checkpointing allows you to save everything, at least on a reasonable scale, including the scale that we have so far here at nurse okay.

B

So I'm going to skip this.

A

C

One question.

A

From brandon, so he said, are there limitations on what the mpi features can be used? I think he's talking about the mana and then he said, for example, npi mean analytic, shared rmh.

B

Excellent uh point: uh can everybody assume also hear the question or should I repeat,.

A

Go ahead: okay,.

B

So the question is: what are the features of mpi that can be supported by mata and then the example was given for uh mpp mpi win family uh which uses uh rma remote memory access.

B

So, um ultimately, the game we've been playing in mana is one of coverage, and so uh we do have a student now tyrone malvia. I believe he's going to be talking toward the end of today on exactly what he's doing to support the mpi with family.

B

But I'll say right now that he's concerned mostly right now, with shared memory on the same node, not with remote memory uh in principle, both could be attacked.

B

um But this is an example where we're extending we're extending our coverage uh according to what the applications need, and at least so far it seems that most applications are now asking for mpi win the big exception here is vasp 6, where they want mpi win for shared memory, and uh so I'll tell you what the solution was there. uh Hopefully I'm not giving away too much of the punch line from telru's talk, but uh mpi win essentially is supporting inside mpi and abstraction of what people in operating systems already know about.

B

In operating systems, they they know about system, five shared memory on the same node and they know about using rma between nodes and the developers of mpi put together a single interface as an abstraction for both of those, and they try to pretend that it's just one abstraction. But if you look closely, it's really two abstractions melded together.

B

So right now we're attacking the one, the abstraction for system, five shared memory and there the simplest solution was not to. When we see an npa win call, we do not pass it to the lower half.

B

We already know it's for system five shared memory, so we wrote our own tiny system, five shared memory, thin layer, and now it goes to our thin layer for system five shared memory which we'll directly call lip c and not even try to call the mpi library, the lower half so yeah. Ultimately, sometimes we have to be creative as we extend the coverage, but this seems to be the solution. The right solution for now, um okay, good, so we've seen split processes.

B

Split processes are important because they solve the m by n problem. If you have m different implementations of mpi m for mpi and n different uh implementations for the network and for network, then what are you going to do if you want to truly support mpi in the future?

B

The answer is you can't possibly handle all those m by n individual cases. So that's why we claim that we really need this idea of split processes. That's the only way going forward in the early days of npi sure everyone is going to use m pitch and everyone is going to use tcp whatever. That is here. I guess ethernet, but obviously that's not where the future is headed.

B

Okay, so split processes crack three-way collaboration. We've talked a lot about this uh and it's a great adventure with a lot of collaborators. So I'm very fortunate to be able to speak here on behalf of so many people who uh have very interesting ideas and, as I've said, the killer app is really cheating allocations and when we change allocations. As I said, there are two places where you want to do it. Maybe you need longer than a 48 hour allocation, or maybe you want to change together.

B

The two-hour allocations, so I believe currently at nurse they offer a special queue with a 75 discount if you're willing to use the two-hour allocation, so that makes it very attractive.

B

I'm going to skip this and yeah. So let's go now to a tour of the internals good. I have enough time so I think I can do some of both the third and the fourth part so for the third part, the tour of the internals.

B

What is the key design decision we had to make in implementing this approach up until now? I've really just told you about the approach approach makes a nice academic paper and we can do toy examples. But if we're going to implement this, where are the limitations there?

B

Well, the limitation is that when we do a checkpoint, we have to make sure that there is no mpi process which still is in the middle of a function called to mpi, because if the mpi process is in the middle of a function called to mpi recall, this is only a single thread.

B

Therefore, it's a single stack and in that stack we must now have some high-level call frames from the application and user code, and then we have some low-level call frames that would be inside the low-level mpi library that can never work because, among other things, we want to be able to restart, maybe even under a different mpi implementation.

B

For us again, we think we've solved the mbi m by n problem, but if we're going to do that, so how we're going to do this so that we guarantee that no mpi call is in the lower half at the time we checkpoint.

B

So obviously it means we cannot checkpoint immediately, we'll have to wait until we reach some kind of stable or consistent state, but hopefully that weight is not very long so that we can still maintain the ideal of transparently checkpointing when you ask for it without any extensive weight.

B

So here are the ways that this issue gets elaborated as you try to implement mana.

B

First of all, there are communicators, and we've talked about virtual ids when we restart.

B

How do we know which communicator is which so here we're going to use the idea of process virtualization that I described earlier, just as we can have a virtual pid and a real pid and our own internal table to go back and forth? We can do the same thing with communicators and then when it's time to restart well, if we did this right, we've already logged whenever we created a communicator and if we free the communicator, then hopefully we remove that from the from the list of what we log.

B

So now we have a log only of the communicators that we are still using right now. So that's list is not too long and then on restart we replay and as we replay the log that log should list the virtual id for each communicator.

B

So when we restart mpi gives us a new actual communicator and we have a table virtual id of the communicator actual id good, there are various strategies for point-to-point calls.

B

So I think, as all of you know, mpi is dominated by two major families of calls. Point-To-Point calls and collective communication calls.

B

So from point to point. The idea here is, we have, uh we might have blocking mpi calls if it's blocking that's going to be a problem.

B

So right now what we can do is inside our wrapper function, for each blocking send or blocking receive.

B

We convert that into a non-blocking send and receive and with a non-blocking send and receive ultimately means you make a non-blocking send and you get a request structure back or you make a non-blocking receive and you get a request structure back and so now we'll play the same game of log and replay.

B

But now we will log and replay those request structures so on restart. We have all the request structures that we need.

B

So that's what's uh described here at this time. This does not appear to produce excessive runtime overhead. If we discover later that it does produce excessive runtime overhead. There are other strategies where we could continue to allow for say a blocking send.

B

Well what happens if you have a blocking send, and this time to checkpoint seems like that defies what I just said. I just said: we've got to make sure that nobody is ever in the lower half, but if we have a blocking set, then clearly we're in the middle of ascend in the lower half well, the answer is at the time of checkpoint, some other mpi process can offer to receive from everybody, who's blocked and it will receive it will copy the buffer into an mpi internal buffer.

B

Now, whoever was blocked is no longer blocked and then on restart. We go back to a log and replace strategy, remember which locking sends had to be relieved of their message and we replay those uh blocking sends one more time and restart.

B

So, let's, let's take a tour of some of the other uh types of strategies that one can use. So hopefully, I've convinced you that there are strategies for point-to-point that this can work out.

B

How about collective calls, so in the original thesis of rohan garg he described two-face commit more recently, we've had to generalize that to something that we're calling hybrid two-face commit. This is coming out of the work of a phd student.

B

And uh he'll say something more about that later today, so but first, the hybrids who two-face commit so here I have it an outline. Maybe the best thing is luckily. I have slides after this which expand on each point. So in here we have details of log replay. I think I described it well enough details of point-to-point.

B

I think I described this well enough. Also.

B

And then we come to collective calls. The two-phase commit so for collective calls to base commit. We have uh we go through a wrapper function in mata as before, and in that wrapper function. Instead of immediately calling the collective call, we call mpi barrier first, so we call mpi barrier and then we call the actual collective call and what's the advantage of this, the advantage of this is that when it's time to checkpoint, we now know that we are in one of two different cases.

B

One possible case is that many of the mpa processes are in that trivial mpi barrier that we asserted it's ours, the one that we know about, but not everybody. Some processes have not yet arrived.

B

So if some processes have not yet arrived at that internal mpi barrier, then we know for sure that nobody has left the mpi barrier. If nobody has left the mpi barrier, nobody is in the actual collective call so case one. Nobody is in the actual collective call. uh This is great, so we simply checkpoint and on restart.

B

Well, we talked about login replay or this internal mpi barrier. We don't try to log in and we don't try to read and in essence, we just restart the wrapper function and we replay the npi barrier one more time, so we just throw away that internal mpi barrier.

B

So that's great. What's the other case well, the other case is that everybody has finished the mpi barrier and some processes have now entered into the actual collective call. But what so? What we're worried about we're worried about? Maybe there is some mpi process, that's still doing some work from ages ago, and it will take another hour before it even reaches this collective call.

B

Well, if it's going to take another hour before it reaches the collective call, then we must have been in case one and in case one. We know that nobody is actually in the collector call yet they're only in the internal barrier that we'll throw away.

B

What's the other case, the other case is everybody has reached this collective call. Everybody has reached our wrapper function and because everybody has reached our record function, everybody is present and ready to execute the collective call and an mpi collective calls are quite efficient.

B

Usually much less than a second, so we'll wait an extra second until everybody finishes the collective call and again we're done so that's the idea of two-face commit for a hybrid two-faced commit I'm. uh I won't go into the details here. That's uh work, that's still being developed by uh yeah, but he will talk about that. More I'll. Just give you this one teaser.

B

um Why do we even care what's wrong with two-face commit what's wrong with two-face commit? Is it turns out that that reduces a lot of runtime overhead? So we had a question earlier about virtualization of process.

C

B

And there I said it's less than the jitter. We don't have to worry this time. We do have to worry um and we're estimating that this two-page commit can add in the case of mass, maybe percent or twenty five percent runtime overhead.

B

If your mps dominated by points of point calls, you won't see this, but if it's dominated by collective calls the wave aspects, then you will see large overhead.

B

So the hybrid two phase commit is an effort to reduce or eliminate this runtime overhead and the short answer. Is we every time we enter? One of these wrapper functions for a collective call. We increase the sequence number, so we say for this communicator we've seen one collective call. Two collective calls three collective calls.

B

This is now collecting call number 17 for this communicator and then because we have that information when it's time to checkpoint, we can ask all of the mpi processes to tell a central coordinator.

B

What are the communicators you know about and which is the last sequence number that you've already handled for this, which is the last collective call that you already handled and now we can draw our own state diagram. Well, everybody had finished, call number 16. call number 17. These processes have finished but have arrived, but we're still waiting on two other processes.

B

Okay, so these two other processes, oh they're, working on the other communicator they're at sequence, number 103, okay and now we can have a dependency diagram and in this dependency diagram we know what is supposed to happen before what so. With this dependency diagram, we can do the explicit happens before, and now we see we can, we can globally see who we are waiting for and now we can take the appropriate strategy, and so ultimately, during normal run time overhead, we just maintain the sequence numbers, that's very light overhead, almost nothing.

B

And then, when the request comes into checkpoint, we will begin the two-phase commit. We will basically transition over to two-face commit and when we transition over to two-face commit that gives us the safety properties. We talked about two cases uh case, one or case two, and we know what to do in either case.

B

So now the story is sequence numbers normally during runtime, when the checkpoint request comes in transition to two-phase commit when you're into phase commit, you have two cases it turns out. We can handle either of those two cases- spying, okay, so good. I think I still have a little more time, and maybe this will help us to get closer to the original schedule, so potential future applications of transparent checkpointing.

B

So maybe I should stop here at this point: we're done discussing mana and mpi if there are any questions about mata or mpi. Let's talk about that here before I move on to checkpointing in general,.

B

Okay, so there are questions in the chat yeah. Somebody could read those things. Oh good good,.

B

Okay, so uh so here are some other ideas, so these are, in my opinion, on the forefronts of what checkpointing will be able to handle in the future and are starting to handle now. um So, as I said, yes, uh traditional supercomputing based on mpi, but what else?

B

Well obviously, hardware accelerators twinkle jane will talk about cuda, then opengl, there's some very nice work on checkpointing opengl by the people at memberg, in collaboration with us, but they were leading that effort and you'll find that in the last year's workshop on super check, then people talk about tpus because they're worried about machine learning and deep learning still further out in the future. But people are starting to talk more and more about fpgas, so with fpgas.

B

What would we do well, in some sense, we'll try to learn the lesson of cuda. The lesson of cuda was instead of trying to transparently checkpoint in the middle of a cuda task. We allow all tests to complete and then we checkpoint and in cuda in almost almost always, the tasks are short. They finish usually in less than a second and some code. Applications will even go through 5 000 tasks per second.

B

So hopefully, if fpga follows that same path, then we can support fpga using some of the same principles.

B

uh There are newer infrastructures, we've talked about spot instances. Remember chaining is the killer app so spot instances, that's where you need chaining. Also, if you want to overcome the limitations of spot instances, another one that's becoming very hot and very much of interest in at least among academics, but also now in industry, uh serverless or uh serverless computing, or functions as a service f-a-a-s, and so maybe they're again. Maybe the idea is: can we just chain applications together?

B

That's the killer, app, so functions will be even shorter than a spot instance. But that's okay. If you can chain together different function, calls you can play that same game up until now, serverless or function as a service has assumed that those functions have no state that they're stateless, but if you can do transparent checkpointing, maybe there are ways to generalize it and make them stateful, which can then open up newer applications that in the past could not be used that in the past, were not compatible with functions as a service.

B

Another one is fog, computing or edge computing, they're different but related. The idea is on the edge of the cloud. Sometimes you want to move an application from one location, geographical location to another.

B

The user is in a car or truck and they're moving. So let's move the application close to them. Why? Well? We always want to have low latency.

B

So when we move it, this is a kind of process migration, but we can also think of it as chaining. Together, multiple allocations, it's the same technology uh for large data um I'll, say I better start moving faster or else I won't keep my promise of reducing the time deficit uh real time or on demand computing. This is very well known here at nurse, so maybe I won't stop to say anything.

B

uh There are newer applications which I've been very interested in. For me, debugging has always been interesting.

B

Remember I said that if a if a program hits a bug, how can you convince the developer that this is an important bug answer you could take periodic checkpoints and then send them the last checkpoint? That's your bug report, but how about now, if it's multi-threaded or mpi multiple processes?

B

Now what are you going to do because now you have a race condition, and now you need to reproduce the race condition you need to describe first. This thread did this: that thread did that for the first thread did this again and because of that particular schedule, that's why we had a problem.

B

So for that you want to use model. Checking model. Checking is very successful if your program runs say under a minute, but if your program runs longer, you have an explosion of states and mobile checking just doesn't work, but now, if we can take periodic checkpoints, maybe every minute now when it, when you hit the bug back up to the last checkpoint and lie to your model. Checker tell this that this is a new program. A new program called restart my old program and restart my old program, we'll just load in memory.

B

Loading in memory is deterministic the model checks. You know that very easily and then after that, um after that, continue to mulch check and find out why this race condition happened. What was the bad schedule?

B

uh Algorithmic debugging, no time to say much on that uh stragglers, no time to say much: there are newer container environments that will be interesting to support apptaner or singularity.

B

This has the promise of easily supporting parallel computations across nodes in a container environment. So containers are useful. I have nothing against containers where they work. Let's use it, we can still put dmtcp inside the container and now, if we have a parallel environment well, we know this. Some hpc environments are difficult to build, build it once put it inside the container, uh and now you have the benefits of containers, namely that one person does the build, and then it runs everywhere uh alpine linux.

B

I won't even stop there. um Another one. I've been tracking with a colleague in france is a sim grid model. Checker, that's to look at parallel, mpi, algorithmic, debugging, there's an interesting idea there, but no time some really nice work with members by all means. Please talk to my members, colleagues about this.

B

uh There's a really nice video on youtube, where there's a about a five or ten minute section in which one of the hollywood developers for cgi talks about how really really valuable this is and how can change things, but the idea is there's a way to use uh checkpointing of opengl graphics um and restart okay. Let me quickly tell the story, because it's to me it's really interesting. There's this uh autodesk has this framework maya with maya the hollywood.

B

Cgi developers can develop um graphics, but uh if you're doing it for a movie, that's going to cost 100 million dollars. uh There are great designers who want to try the leading edge graphics. These are used going to be a third-party plug-in for maya. A third-party plug-in for maya might very well have bugs for corner cases, so you can try it, but maybe it'll crash.

B

If it crashes, it might take you three hours to set up your complex environment again, wouldn't be great, if you just say: checkpoint now, try it and if it crashes, no big deal just restart and then five minutes is working again. So this is what my members is offering there navigational programming. um This is twinkle, jane and, and I've now gotten very interested in this.

B

It's a longer story, but uh with navigational programming, it's another promise of taking a complex parallel application and showing how to easily transform it from the original mathematical equations into standard software. That can be understood by anyone right now. When you convert to mpi, you have to turn things inside out. You have to access remote data and so on, um and then what else?

B

Well, I think, I'm out of time so not very much else.

B

Okay, so, as I said, killer apps have become very important: the killer, app of chaining allocations.

B

We've talked about cloud spot instances functions as a service.

B

B

Ultimately, here are the four related themes in this conclusion: we've seen split processes, we've seen changing together allocations.

B

We actually we skipped over, but we tried to talk about migrating processes to the data instead of migrating that into the process and then finally, there was the idea of deep debugging using model checking, and I claim that all four of these actually are closely related, and so I hope that the next decade of checkpointing will be even more exciting than the last decade. Thank you.