Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2022, 11 Nov 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Computational Fluid Dynamics (CFD) Analysis With Kubernetes, Kubeflow, And OpenFOAM - Erik Jacobs

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe in Amsterdam, The Netherlands from April 17-21, 2023. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Computational Fluid Dynamics (CFD) Analysis With Kubernetes, Kubeflow, And OpenFOAM - Erik Jacobs, Red Hat

Speakers: Erik Jacobs
Frequently, organizations build dedicated clusters for high-performance computing (HPC) workloads. These clusters may sit idle when there are no HPC jobs to run, which is a waste of expensive resources. Kubernetes clusters can run these workloads alongside all of the other applications typically deployed, which helps improve overall operations and drives higher utilization of resources. In this presentation, you will learn how the Kubeflow project and its MPI operator can be used to run computational fluid dynamics (CFD) jobs with OpenFOAM on top of a Kubernetes cluster.

A

uh Good morning, everybody thanks for showing up today. This is a computational fluid dynamics. Kubernetes Cube flow open phone um before we get started. I just wanted to thank two folks, uh I, don't think either of them are in the room today, Eduardo arango and although coquincador they were super helpful. When I was going through this sort of process of figuring all this stuff out, um Eduardo's, actually not at Red Hat anymore, he moved over to Nvidia, so either way still helpful or thankful for his help.

A

um This is not a super serious presentation. It's Friday we've been here all week. It's you know all tired. It's gonna be fun. I, promise, I, hope. uh If you want Sirius, there's a blog post that I wrote that details all the Gory GitHub repos and all this other stuff. This presentation's on this get website. You don't have to worry. We can go back to this whatever.

A

um So, if you don't get a picture of this, it's fine um and like I said this is going to be fun. So bad, jokes, terrible memes, all that kind of stuff, so I'm getting at least one round of applause, so we're good. Okay, here we go, this presentation is built like building blocks right, we're just going to try to put things together, get deeper deeper deeper, as we go to explain how I actually was able to achieve this, because in reality it was mostly just kind of an experiment.

A

um Before we get started just a couple questions if your kubernetes user raise your hand, real, quick everybody. Okay, that's not totally a surprise who's doing HPC, not necessarily on kubernetes, just anything that would qualifies HPC only a few anybody doing MPI stuff about the same number and then lastly, open foam existing one, okay cool.

A

uh How did we get here so me, I've, been at red hat for 14 and a half years, not a software engineer, I'm, not a kubernetes cubeflow, open, foam developer, I'm, not an aerodynamicist, but I do have a race car and it does have a wing on it.

A

So uh I ended up getting this email from an account team out of aipac and they were like. Oh, this automaker is doing open foam in a container. They want to do it in VMS and I was like. Can we do this? On kubernetes and I, remembered I have these friends who do aerodynamic stuff and they offered me some of their um uh models to play with and I was like cool, okay, let's, let's, let's do an experiment right.

A

Let me see if I can run their fluid dynamics, Wing job in the kubernetes environment, so review, open, foam and MPI. What is MPI? It does parallel or distributed computing. So we have lots of things that we do then lots of little things in parallel. Open foam is for computational fluid dynamics. Areas of fluid water is a fluid. It can analyze how it flows around objects.

A

So we have this big job. We want to break it into little jobs and that's where MPI come well, but not really right, MPI doesn't do the breaking up it just does the processing open foam is a tool. Well, it's actually lots of tools to help. Do fluid Dynamic stuff water isn't fluid. So we take our big fluid dynamics thing and we break it up into little pieces with open foam, and then we use MPI to do all the things. uh Those things are actually really like individual servers.

A

For most folks, they're used to HPC and clusters and Etc, and it does this over SSH right so MPI causes it finds all these systems with SSH goes into them. Does the things we need some kind of shared storage, because all the little pieces need to find the work that they have to do so to speak right, so we got to figure out uh figure that out as well.

A

Okay, it's not really that simple, but for the sake of Simplicity we're going to draw this picture: open, foam, NPI, lots of stuff rwx cool clusters and kubernetes. So how many of you have something that looks like this right? You got a rack of servers, other stuff, okay, cool, so 2014. This cool kubernetes thing happened, and then people started trying to do this right. We want to run all the workloads in kubernetes, so this is a workload. Can we run it in kubernetes?

A

Let's see what we can do, starting to draw parallels right, so open foam, MPI, fluid dynamics, job we've got kubernetes, kubernetes runs pods. If you remember the previous picture, we actually have a bunch of little things that we're doing fluid dynamics. That's a lot of pods.

A

We have SSH, we have rwx storage, we have kubernetes, we have RWS storage, we'll figure out the rest, um I'm not really sure, but we'll get there right. So, let's go deeper kubeflow. How many of you know what kubeflow is have ever heard of it? Oh wow, that's folks, Okay cool, so kubeflow project dedicated to making deployments of machine learn. Well we're not doing machine learning, we're doing fluid dynamics right, but it has. This operator called the MPI operator. Now it says it's for all: reduced distributed training.

A

It's just MPI like we're doing an MPI thing: let's do an MPI thing for one thing, so let's go deeper uh operators, how many are familiar: kubernetes operators, most everybody in the room, okay, operators- are this it's too complicated, so operators are just a pod right. There is a controller pod that looks at instances of a custom resource custom, resource definition, extends the kubernetes API and then it does stuff in this case we're talking about the MPI operator, the stuff that it does is MPI stuff.

A

So we've got MPI operator. We have this MPI job, I'm running a wing cfd job, it's going to happen in a lot of pods, okay, I I, don't know so lots of PODS kubernetes rwx, all the things I used, openshift that happens to be the red hat kubernetes. We have a storage product. uh Well, it's moving to IBM, but we have a storage product, open data Foundation that gives me rwx, and then we had to do stuff in the Middle with kubeflow.

A

So let's go deeper again: Wing cfd MPI operator when you use the MPI operator and you instantiate an MPI job object. It fires up a launcher pod that launcher pod then runs MPI into all the job. Pods that do the work. They talk to storage, if you have storage defined and all this happens on top of your kubernetes environment.

A

Okay still does this over SSH, though it's like that's gross and I, had to do some nasty tweaks that are in the blog post to make it work on openshift, but whatever we've got it working, it's fine! Okay! Here we go. So what is the open foam process? So uh the first step is this meshing process? You take a 3D object, you break it up into like a meshy thing: okay, cool! Then you decompose this into lots of small jobs right. That was the thing we talked about early on.

A

uh Then you do the problem solving and figure out what the fluid job is, and then you put it all back together, relatively trivial, so I'm really good at this, and fortunately the uh open phone project provides a bunch of examples for how to do stuff and I also have these guys to give me things to do. I actually helped Moorland build their open foam.

A

Ubuntu cluster, like they're in Atlanta I'm in Atlanta I, went over there and had lunch one day it was fun um Okay, so Dan break is the example job that's provided by the open foam project. It's a very simple. Like water flowing over a thing, let's try to run that job and see if we can make it work. So what do we actually have to put in here right? This is where the the rubber finally meets the road uh trigger warning, there's actually code.

A

It's really. This is not code, it's barely code. So this is the the dam break example. So you you do some stuff, you break it up. You run the parallel job, that's the meat, and then you put it back together.

A

So how do we get this? uh How do I get that script into this launcher? Pod right? Well, fortunately, I actually know a thing, at least maybe about kubernetes. We have this thing in kubernetes called a config map. It makes it easy to inject files into the containers. Well, that's just a script. Why can't I just inject that as a file into the container and then make the MPI job run the whole script as the entry point to the work?

A

So that's what I did I put in a config map that I defined against the dam break job that went into the launcher, pod that launched the wing cfd pods that talked to the storage on the container platform, okay, cool and then yes, it did work. This is what it looks like. It was doing things the first one is the dam break job. The second one is motorbike, which is another example within this is the openshift user interface that just shows like hey, they're, using CPUs, there's pods stuff.

A

That's the log output from open foam for the one person who's using open phone they've. Seen that a lot they spend a lot of time. Looking at that text, this is the Moreland job. um This is not really a picture that I generated, it's one. They sent me, but that's what I did I ran this job and it worked well sort of um remember this picture. Yeah I, don't have one of these I got this, but I wish. That was me. That's not my laptop. It's not my background.

A

It's that'd be cool place to work every day, I guess or just really humid. In mouth and mosquitoes anyway, so I didn't have one of these we did have one of these I really I had that right. So, uh fortunately, I didn't spend a lot, but I did have this so again, I just kind of was messing around uh I used m5a Forex large, which are not compute optimized instances.

A

uh I was like all right. Well, let me do two CPUs for pods I'll just do a boatload of PODS, because more is always better right. um 512 CPUs it took five hours and Moreland was like man, that's really slow, and so the reality is. You have to know what you're doing right tuning tuning tuning tuning is important. um I'm, not really sure you know, I I could have used. The compute optimize instance. Type I was probably uh I, probably should use bigger pods that are using like the whole server.

A

Instead of like lots of tiny ones. um Maybe I was over slicing the job. You know when you, when you have the surface and you break it up into lots of little pieces. You end up with overlap and then you're, basically just reprocessing the same things over and over that get edit it out at the end, when you put it all back together or just know what you're doing right actually be an aerodynamicist and know what any of these things mean.

A

um Where can we go from here right? So PMI X is a new thing around openmpi that gets rid of SSH there's like an actual server Damon kind of deal. It can run on kubernetes, so it'd be great if we didn't have to legitimately, like run an SSH Daemon in all the pods and then like SSH via MPI from the launcher into this is just like gross.

A

um Of course, more tuning um Nvidia is working on doing GPU enablement for this particular project for open foam so that you can use accelerated uh processing via gpus um But, ultimately like what was the point of this whole exercise right, it really was an experiment. Can I even do this, and so the reality is that I wasn't trying to test. If AWS was a good place to do this, because the reality is AWS really isn't a good place to do HPC your data is probably not there.

A

So moving all your data into there, that's going to be ludicrously expensive um and it's just expensive to do your instance. Types might not be available if you need a boatload of them like there's all kinds of problems with this. So most of you, if you're doing HPC, you probably have clusters on site that big picture of all this, like you can do this right, it actually did work. We were able to run open, foam and- and more generally, you know whatever MPI type job you have. We were able to do it.

A

um One of the important things to think about, though, is that for those of you that are raised your hand that you are doing HPC, you know kubernetes doesn't think like most HPC schedulers, it is a scheduler I would like you to run my job. It will do. It has no concept of priority preemption cueing, like that kind of stuff, like traditional slurmy, whatever your HBC tool of choices, it's not doing that today, but there's a whole Army of extensions and schedulers and other things that are coming about that are doing those things.

A

Red hat is investigating how we can do some of them. We've actually got the product manager for that in the room right now, um But, ultimately, the the purpose of the job. The purpose of the experiment was like. Can we do this? Yes, we could okay, great, um that's it. That was really fast wow. That was like barely 12 minutes.

A

um So yeah happy to answer questions I realized again. This was supposed to just be kind of a fun happy one um but yeah. That's all I got for you.

A

No questions, oh okay, we got a few hold on. We got a microphone.

B

uh Great talk, um so, where are you going to go from here now, like is? Is this project done? Was this just kind.

A

Of like yeah, so um it's good thanks for asking that um I I'm talking to the Nvidia folks about it, they're they're interested in figuring out how we can do that. I've done, Nvidia GPU things on AWS before on openshift and kubernetes We Know It Works. um So it's just a matter of how do we do that with open foam? The other thing is um uh where's the script so like these steps, it I was thinking about it yesterday. Actually it doesn't really make sense to do all these steps in the launcher pod.

A

Maybe it would make sense to like do the pre-processing and decomposition in a tecton pipeline and then take that and then attach that as a stored object into the launcher pod and then just run the MPI thing, and then, when that job finishes, do another text on Pipeline to run reconstruct like? Does that make sense? I don't know?

A

Maybe um it would probably work but like how much effort would it be to actually automate it would I need to write a tecton task to to launch the MPI job and then like another, so I don't know it's it's an experiment that I'd be interested in doing, um but that that's kind of the like where where would it go from here and then um trying to figure out some performancey stuff? uh uh Where were we yeah here so rejiggering?

A

Some of this make it look more like moreland's environment um and see if I can, you know, get close to the performance or whatever we we do have some clusters and and Labs at red hat. That I could run some of these things in um and then just figure out. You know like. Is there more storage, optimization than I need to do or those kinds of things so that that's kind of where? Where does it go? You know again, it was really. It was a thought-ish experiment. Like can we run these types of jobs?

A

um Grav like I mentioned, is the PM. That's handling a lot of the HPC stuff, um just doing more HPC things um like this, not necessarily more open phone, but just other HPC things and see how do they work and just lots of blogs and other things. It's a long answer.

C

A

Sure well go ahead. Yeah.

C

I I noticed under the MPI run command. You didn't have the allow run as root flag just curious. Like uh did you have any issues with setting it up to like run as a user instead of yeah yeah? So where was.

A

It privilege so many images lots of memes nope that wasn't it. It was I, don't remember anyway, we'll we'll see um so. The the most of the challenges I had were because of SSH and because of the open foam container, and so the open foam container expects to run as a very specific user and trying to run SSH as non-root is like a monstrous pain in the butt um and so for the sake of just like get it done. I I just kind of blew the permissions wide open and tried to do it.

A

However, I could um Red Hat has done a lot of work around randomizing user, IDs and other security enhancements in in openshift to um to reduce the blast radius of like misbehave workloads, so to speak and be more secure, and so we have some weird tricks that we've done in our Builder images to like do things to Etsy password as the container comes up so that things think they're running is the right user anyway, I just I didn't want to spend the time to go through it.

A

So tldr and your question is um yeah I just blew the permissions wide open to make it run, but you you could probably spend a little bit more time to make it run not like so so badly.

A

You wanted to see one of the this one.

D

Thank you, you're.

A

Welcome no no question about this slide. Just want to see it.

D

A

B

E

So, are you or red hat doing more on kubernetes and HPC, or was this just kind of a one-off experiment.

A

For me, it was a one-off experiment because of that email that came in, like you know, they wanted to do it in VMS and I said well. I, probably don't need to do it in VMS. You could probably just do it directly with with kubernetes.

A

um You know, red hat is not going to suddenly become an open phone company right like that. Wasn't the purpose of this experiment at all.

A

It was really mostly like what what does it look like to use the MPI operator redhead is invested in the kubeflow project, Community for other reasons around AIML, um so there happened to be people that I could talk to easily to to fuss around with this thing um but yeah in terms of like any kind of strategic Road, Mappy thing like it's, it's it's not this specifically, it's more generally like we are exploring how to do HPC in a kubernetes world and what we could maybe productize.

A

um You know or just what we can support, who we can partner with uh you know and so on and so forth. um Of course, we're doing a lot of great work with IBM in this Arena, but there's a lot of other companies and projects that are out there that sort of play in this space as well. Okay,.

E

And if I was going to do a follow-up.

A

E

Okay, um when you were running an AWS, were you using their MPI drivers? No.

A

So open thumb and open, MPI and kubeflow that don't require any kind of special drivery bits. um It was really just I. Did a vanilla, openshift installation on um on AWS, you probably could have also used eks on AWS I used the um open data Foundation, because I needed rwx, you probably could have used EFS.

A

You know you just need some Central uh rwx storage that everybody can read and write to, and the only reason I needed rwx was because the launcher did the initial writing and decomposition of the job and then wrote that out to the shared storage and then all the workers needed to access it and then write their stuff back so like if I had done it in a pipeline, anything I might have actually been able to get away with some other way to do it.

A

But I again, I just didn't explore that deeply, um but yeah there was. There was nothing special about the way I was using AWS or what I was doing on AWS. If I had had a large cluster of Hardware like at my disposal internally easily I mean we have a lab, like my group I. Just didn't feel like figuring out how to use it. uh This was easier. We have a system. I can order a cluster. It's like yeah wow this guy anyway. Yes, sir.

B

Question for context, so what is moreland's workflow right now, then, if they're not using something like this, what are.

A

They doing he has a rack of Old Dell, 1rus I, think he's got like six or seven of them or whatever, um and it was mostly like I wasn't trying to get them to go kubernetes or anything like that. It was more like well, I know somebody that uses this tool, so I can at least poke them, for you know if I run into run into a jam, um but yeah they're they're existing workload today is just 100 bare metal.

A

um No, no Cube, no containers! It's just like Ubuntu installed directly on the on the servers and then um you know open foam installed in the OS.

A

Well, no, he I mean it's it's using MPI, so so he's got a head server. uh I mean it looks a lot like that in a way like there's a head server where he runs um the the pre-processing stuff and then that's done on a NFS share. The head server is also the NFS server and then they do MPI run, which then goes out to the other, more powerful servers, um and then they are pulling from the NFS share and doing the actual processing. And then the reconstruct happens at the end.

A

And then it spits out the cool image.

A

The microphone disappear.

F

Hi great talk, I I was curious with regard to and I'm not super familiar with open form. uh Do you have to? How do you handle the scheduling of all the yeah? So all the all the parts on on the on right hand, side do all of these need to be scheduled and do at like one go? Do we essentially what I'm trying to say is: is this a gang scheduling problem that we are trying to solve here.

A

Yeah so I'm gonna partially answer your question because I don't fully know so so take what I'm saying with a grain of salt, the MPI operator in the actual job definition and I, don't have the full job definition you you write a pod spec effectively for the launcher and a pod spec effectively for the workers, and you tell it how many workers you want when you run the job or sorry when you instantiate the job instance, uh the CR.

A

The operator creates the launcher pod and creates all of the worker pods, and it pretty much expects that they all can run at that time and I think if it can't run all of the workers, it just sits there and waits. So it doesn't like run one of them ahead of time and wait for it to finish and then oh now, I can fit another one and run that on the back. So I guess it is a gang operation like it wants it all to happen at once.

A

I, don't know if that's a limitation on the MPI operator like if, if there's a way to tell it like it's okay, to run something ahead of time, I just never looked into it any deeper.

F

And when you run this on a cloud provider AWS in this case, uh did you have to like pre-scale the compute to be able to handle all those parts and then scale it down?.

A

So I I, what I wanted to do was I actually wanted to do this as a cluster Auto scaling job and so the way openshift is built- and you can do this with with most of the kubernetes um there's a way to make openshift Auto scale itself based on workload demand, and so what I was hoping to do was have like nowhere near enough actual nodes and then run this job.

A

And then, when the cluster Auto scaler detected that there were pods waiting, it would start jacking up the number of nodes to actually run enough stuff. But I just I didn't I didn't get there, so you wouldn't you, wouldn't necessarily have to have that ahead of time.

A

The only caveat I guess is I, don't know what the timeout is if any on the MPI job operator, and so if it took like a really long time to get enough nodes to then schedule the pods I don't know if the job would have already failed because it was like oh I, I didn't get to do it never mind. I gave up like I'm, not sure what the timeout kind of deal is.

F

Yeah, that makes sense. So my next question is on uh I think you mentioned that multiple parts were running on like one bare metal instance. One VM is that accurate. Did you see resource contention in multiple parts were running on the same VM.

A

Yeah um no I didn't see resource contention because the Pod, when you write the Pod spec, you specify like a a request and a limit to do the best. Sorry guaranteed quality of service tier so for those of you, who've ever played with the quality services in kubernetes like you've, got requests and you've got limits. If they're the same, it says like make sure I absolutely have that much to run or don't run me um and then underneath the covers.

A

You know the Linux kernel and c groups and the scheduler, and all that other stuff make sure that there's no contention if you wrote the spec for these badly and didn't, have a a request and a limit to put you in the quality. uh The guaranteed Quality, Service tier and you ended up in either best effort or like uh whatever the other one is like.

A

You could totally end up with resource contention where it just would throw two pods on the thing and they're both trying to use all the CPU and they both one of them, starves the other one out so like you definitely want to put limits and requests to make sure you're in the guaranteed qos tier, um otherwise you'll end up with contention yeah, but that's like normal. That's not specific to the HPC stuff. That's like generic kubernetes, stuff, yep.

D

Just wanted to uh confirm so when it came to the cube flow, it was uh providing your MPI operator. So then, all the communication from the launcher pod to the compute pods, uh was just being solely managed through whatever Cube flow, provided you correct.

A

uh At that point, it's all uh where's the SSH picture.

A

So it's it's all still SSH between the um the launcher and the workers. So the operator ensures that the launcher pod is created and the worker pods are created once all of the pods are up and running the jaw or the script will run inside the launcher and that MPI run command.

A

Mpi itself will then look at the host file to find well, there it'll look at the MPI host file to find all the targets, but those targets are all just pod names and so then, literally like MPI fires off a bunch of SSH connections into all the workers and then executes whatever.

A

Yeah so that MPI run MP4 interfoam, so everything after the four is what gets run um effectively on each of the workers? Okay, um so so the all the kubeflow all the MPI operator, part of kubeflow is doing is watching for those um that's not bright enough. It just watches for these and then creates the pods and then executes the command. It's not managing any of the like communication-y stuff that all happens effectively over the kubernetes service layer. Well over the kubernetes software-defined network. I.

A

Don't think it's actually even using Services, because it's talking directly to each pod by hostname, I.

D

Think that's where I was curious, where the obfuscation was happening or the the um yeah of the traffic it sounds like you know, Kube flow is providing you the MPI. So if you have questions regarding MPI, you got to look at kubeflow's source code and then regarding uh uh like in the the communication outside of that SSH or or otherwise, um you're. Looking more back to kubernetes, it's all.

A

uh Yeah sort of I think again, most of the communication is coming from what what is happening in the MPI run, commands from a like Network or storage communication perspective. The the SSH connections from the launcher to the workers is just over the local sdn, whatever Network you've configured for your kubernetes, it doesn't have to be a software defined Network, some people who do it on bare metal. They don't use an sdn and then, of course, you've got the storage communication, which in this case is is network.

A

Communication between the workers and the the storage that contains the little job bits and pieces from a tuning perspective like open from, is pretty good about the storagey bit, because, basically, all the workers like they, they load it up into memory once and then they do their thing and then they spit it out. They're not constantly talking to the storage, but if you have a different type of MPI workload, that is constantly read writing to storage. Then, of course, that storage network is going to be important to ensure that it's got.

A

You know high performance characteristics or that the discs have high performance or both and.

D

Then I haven't had a chance to look at your blog posts which I look forward to, but um when it came to open foam I take it, you probably just pulled a like a Docker file from somewhere or you pulled the.

A

Container yeah I think I had to rebuild their image to make like a mine. Oh I had to rebuild their image to change the SSH configuration to to make some of that stuff work because the the the open they provide a container for you to do stuff in the open phone project, but the container that they provide is like super hard lock to one specific uid and like behaves weirdly.

A

If you don't use that, and so I had to like mess with the bash RC to make it load the right, pathy stuff to find all the tools and everything it's a pretty minimal change, but basically, yes, I'm just using their their container image and and all anything I had to do from a container image perspective. It's all linked in the blog post and it's in a repo I think there's a couple repos that have the different bits and pieces. Okay,.

D

A

Foreign I have not I know we have customers that do do some of those more advanced network storage things um I, don't even know that I have access to Infinity. Somebody at Red, Hat, somewhere in a lab, hasn't been a van that I could try, um but the reality is that, like that's kind of invisible to the environment, it would be more of a like the beauty of kubernetes and, to a certain extent, storage and and the kernel, and all these other things is like that stuff.

A

Shouldn't kind of matter like maybe I, would have to define a PVC that exists as a as a um as an endpoint on that infiniband storage. Right I, don't know, is infiniband like iscsi-ish or is it more of a fiber channely or none of the above all of the above anyway, tldr.

E

A

Got it? Did you hear any of that sort of he's? So so you may have to attach infiniband devices directly into the pods, which is, which is something that that you can do. um There's ways to attach devices into pods. um It would probably make the the wording of the um the job and, like the Pod definitions for the launcher and the other things it would make that kind of start to get hairy, because you got to like be adding storage definitions to everything.

A

I had to do that anyway, to make these connect to the rwx. But if you also had to add devices to the pods like it would, it would just get trickier.

A

Yeah and I don't know if open Foams commit like tools, speak infiniband.

A

Yeah all kubernetes is doing is is effectively if you think about the the path of how pods get launched, like kubernetes, tells a cubelet on a node to run a pod and then under the covers cryo and run C, and all these other things are being instructed on how to bring up that container and in that process. That includes, like you, need to attach this stuff from the host into this container.

A

At this point, so in the case of infiniband device, you know it would like, if you think, about a podman or a Docker run command. It would be like you know, attached slash, Dev, slash whatever into the Container at slash Dev, slash whatever, um so there's, no there's, no knowledge required of kubernetes of infant abandon the last you're talking about like some CSI infiniband driver thing, but that would be baked into the CSI driver for infant events anyway. So back to the original question, I did not try and find a band.

A

um I might be able to find some somewhere, uh but I. Don't know that that open phone would know what to do with it. Even even if I did it does okay. Well, then there you go.

B

G

It cool so you alluded to a couple race conditions and the MPI bootstrapping process. uh Can you speak a little bit about those and whether you address those.

G

No, so you you said it was 512 ranks. Is that correct.

A

um What did I do.

A

uh So I did 256 pods with two CPUs per pod, so that ended up being a total of 512, CPUs and I. Think it ended up being I, don't know how many that ends up being per node I'll.

G

Make up a number it was like, but presumably it was one rank per CPU one. What one rank per CPU one rank MPI rank.

A

Oh yeah, yeah, yeah, yeah I mean the MPI run, command was pretty vanilla um and the host file that gets generated is also super vanilla, um yeah. So there's no there's no fancy MPI stuff going on.

G

The reason I ask is that we ran into bootstrapping process problems with about 80 ranks or so so. We'd see race conditions where, whereby the workers would report ready, um would sorry the workers, the MPI launcher would start before all the workers were ready, sometimes, which we caught a crash in the bootstrap and then the other thing that would happen sometimes is the workers would start and would report ready before SSH was ready, which then would cause the MPI bootstrap to fail again.

A

Okay, yeah yeah I, so I definitely did see this problem, um especially when you're dealing with large numbers of PODS, the launcher pod will fail like a bunch of times, because nothing else is ready. Yet so MPI comes up and it tries to do its thing and it's like I couldn't find half of these machines.

A

I don't know what to do and it crashed loops and then kind of backs off so I don't know that they could probably do things to make it smarter one number two I'm, not sure if you can Define liveness and Readiness probes in the template, spec for the for the job I mean it should support the full pod spec. But the challenge is like you'd have to write a really nasty liveness, Readiness probe for the worker to like SSH into itself. You.

G

Can you could do a postdoc hook where you just like check to make sure the ports Port 22 is open, for example,.

A

Yeah I mean you could do you could do things, it's just I. Never I never got to that point. Unfortunately, the um the pre-processing- and this is why doing it as a tecton pipeline could be beneficial because all the pre-processing steps would get done and then the only thing that's failing is the MPI run. If you look at this like block mesh, that feels decompose, they're happy to run multiple times while it fails, because ultimately it does the first three things then runs MPI run and that's when it fails.

A

So if you have other steps that come before the actual MPI that are not I'll use, the word ident, which is probably the wrong word to use here, but we'll go with it for now like. If you have things that would blow up if they got run twice obvi. This is going to be bad. Bad, News, Bears, right, okay, we're getting that we're getting the hook thanks very much. If you have other questions, I'm happy to answer them outside uh again, the presentation is uploaded to the portal.

A

um You can find me pretty much on lots of places on the internet. I appreciate it have a great trip home travel home, safe.