Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2019 (San Diego), 22 Nov 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Serving HTC Users in Kubernetes by Leveraging HTCondor - Igor Sfiligoi

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

Serving HTC Users in Kubernetes by Leveraging HTCondor - Igor Sfiligoi, University of California San Diego

High Throughput Computing (HTC), sometimes also called batch computing, has long been and still is the major workhorse for most R&D organizations. Typical workloads include parameter sweeps, Monte Carlo simulations and partitionable dataset processing. Kubernetes by itself is not very well suited for such workloads, which are submitted by hundreds of concurrent users and rely on the execution of thousands, or even millions of small tasks. This presentation will provide an overview of how HTCondor, a prominent HTC system, can be used to effectively and efficiently manage such workloads. The author has been running such a system on a Kubernetes cluster operated out of the University of California San Diego, and will share his experience and issues he encountered during that time.

https://sched.co/UadF

A

Okay, so welcome everyone. Try to give you a sense of what it takes to support a htc community and why I think that, Kerberos by itself, it's not a perfect solution and how HD Condor can help make the user experience better. So just a couple of words about me, my name is Polly I'm. Actually, local here I work. A couple of miles: north UC, San, Diego I, have been a longtime HCC user and I'd explain what HTC is in a second just trust me, and mostly working with open science grid.

A

I have I'm relatively new to kubernetes just about the year as part of the PRP again mostly focused here at the UT San Diego. But we have a communities class that the spans the world on one side we have Australia and Korea. On the other side we have Amsterdam all of the UI is in between and it's it's managed as a single kubernetes cluster. So we have a little bit of experience with with kubernetes. So let me start with defined what HTC is and HTC is high throughput computing.

A

A lot of people call it also bad computing, although not all bad computing is HTC and the way I like to think about this is the infrastructure for ingeniously parallel computing. So what do I mean with ingeniously parallel computing induced parallelism is when you get a big problem and you manage to slice and dice it in such a way that you can schedule it independently over many many many independent pieces.

A

Really, the important thing is independent, and then you try to make it as simple. The requirement should be as simple as possible so that you can use as many different kinds of resources as you can get your hands on and I know that a lot of people call it call it.

A

Imbalance, link ly parallel computing, but it should not be called that way because to actually get a in the compose that way, it's really hard hard work where, if you think you just come prepackaged in such a way, you have to think it really carefully in order to have really very few dependencies between your object now I understand that not every problem can be cast that way. I have been working with an application that was trying to do a global FFT.

A

You cannot do that this way, but there are a lot of applications that actually fit very nicely. Oh Monte, Carlo simulation parameter, sweeps event processes, feature extraction, you have millions of independent images, they independent treat them as such. Don't try to put dependencies artificially where they are not, but there are many, many more and the real clever people come with ways of making it very very loosely, coupled all the time now how you may say yeah, but this is fringe.

A

No, it's not I work for open science grid and we have we managed using the htc paradigm about 100,000 cores DC. Here you see the last six months. You see yeah. It goes a little bit up and down, but we are about 100,000 course. So it's not small and there are all kind of Sciences they rely on it. Yes, physicists dominate I, think that quite smart, but there are all a lot of other scientists which have typically lower demands, but they are all there. You have economies, biologists, we have nutritional sciences agronomy.

A

We have some zoologist that do it. There are a lot of places where you can actually do and sorry I don't have any industry in here. We are just not funded to support industry. It's just science but I'm sure that there is plenty of use cases when in industry you can do the same now, having defined what I mean with HTC is kubernetes a good fit. Well, yes, the all the pieces are there. You, okay, deploy one HTC job per pod. You can manage multiple parts on a single node. You can have multiple nodes.

A

You have a flexible orchestration and VPN overlays shared storage. All the things, however, when you get a user say. Ok here have kubernetes use it. They said yes, but it doesn't have what I'm used to. How do I do index parameter passing? What will who will manage my inputs and output and I? Send it mean I, said well: I have 10 million pods, and only in 10,000 nodes.

A

Where is the fair share? The scheduler in kubernetes is mostly priority. Fifo and I. Don't know if you can scale toward millions or tens of media support, which is something that it's the norm in in the HTC community. We literally have tens of millions of jobs waiting in the queues all the time, plus something that is not really a fault of kubernetes is just user inertia.

A

The api's of kubernetes tend to be quite different from the ones that batch users are used and also, if I want to use any other resource. How do I get it in my kubernetes cluster? We just heard about the clustered API, but that's just a very small subset of what you can do so.

A

How about we just take HD Condor and use it to help the HTC users to use kubernetes? Now why HD Condor? Well, it's one of the major batch file systems. It's HTC focus it's not the case that is called HD Condor, although they were forced to rename themselves from corner for other reasons, it's very flexible and it's often used in very heterogeneous environments. It has 90 supports compute for containers, which it's something that everybody that wants to use kubernetes one stone and it's highly scalable.

A

It happens to be what we use in open science grid, so I'm very familiar with it and I'd like it a lot and really it is highly scalable. We just did this big run when we created the biggest-ever cloud agency, pool 350 petaflop across all the clouds and guess what it was all managed by HT condor. So it can scale that big, no big deal. It was about 270,000 CPUs in their course now.

A

Why is it that it works that well well, because it's architecture fundamentally meant to be scalable and flexible than anything else, and it tries to do what h HD is see. Users need on one side, you have the Q, which is persistent and users, usually login into their SSH or whatever and sub it locally, and once they do, they can go away and it will keep all the stay there.

A

On the other side, you have the execute machines and you will install essentially a demon that manages it, and then everybody just talks to a sector collector. So all the state is completely dynamic and just whatever the collector knows exists exists. So it's easy to add the remove nodes at any time and Condor will just deal with it after you know what what is where you put a policy engine in place and that's called the negotiator for in case of Condor, and it has all the goodies- you want it's scalable it. It has fair share.

A

So as users use more, their priority goes down, still use less. It goes up, it suppose, Bost per user per group it does all the matchmaking cliking kubernetes. You would expect. If I need three CPUs, it will find me slot with three CPUs and not one without new GPU, and make sure that you start there. But it only does policy when the Skadi it requests a match, it gives it a match, and from that point on is the skip d and the executor source they talk directly.

A

The negotiator is not involved at all anymore, so all the file transfer all the handshaking and keeper lives. It's all managed between those two and the claims issued out for long periods of times. So we have five second jobs, which some users, like the skiddy can manage the send those thousands of jobs to the executing back without putting undue pressure on the policy engine now that period can be tuned depending how real-time you want to be.

A

But the default is about twenty minutes, which is a good compromise between being able to keep the fair share reasonable and not putting excessive pressure on the policy engine. Now, how do we use HD Condor with kubernetes?

A

The most obvious thing is you have already your scheduler and central manager that users are using for other reasons, and you just deploy a pod that contains condors 30 in kubernetes voila, your kubernetes resources are now accessible to the users through the same mechanisms. They have been using the previous on-prem resources, easy easy. We have been doing this, for example, for our friends LHC CMS here at UCSD, and they barely notice any difference.

A

The other extreme. You put everything into a kubernetes, you put the skid div on one side, a center collector and then a bunch of worker nodes all nicely inside the kubernetes cluster.

A

You don't need to provide a way for the users to read your skin D, but from apart from that, they don't notice any different compared to the previous use case, and you manage everything through kubernetes no big deal. We actually have deployed this as a way to get OSG users from outside to use our PRP kubernetes cluster because they don't know anything about kubernetes. So it was an easy way to of if you want legacy users to start using kubernetes classic.

A

That is used by for ten thousand other reasons, and you don't know to retrain the users how to actually use the kubernetes cluster. They don't see it while still using all the benefits. And the third use case is yes, you start with kubernetes cluster with at least asked it'd be an a collector, but then you can join resources that are not managed by kubernetes at all.

A

You can have some machines sitting in your closet that you don't want to manage by a kubernetes, for whatever reason you can still get those resources to be used from a controller, this kid deed that is actually inside kubernetes, so mix and match. It will just work now, a couple words about HEC users and containers. The HTC job is typically an argument, an application for some arguments, plus some data.

A

The container part, is just a convenient way to package software dependencies. Most HTC users couldn't care less about the fact that there is a container there. All they want is their application to work and container. It's just an easy way to make sure that the software is not a mess and usually the content. They don't build the container themselves, but they use something that the community have provided now.

A

Hd Condor itself allows a container to be attached to the job, and typically people use singularity for that and Condor would invoke similarity after binding the application and the date of course, inside the container, because the users need it and voila the job starts, but from the user point of view is just apply the application and running some arguments. The container is just an afterthought now I said contour is a singularity, that's simply because our users lost singularity, for whatever reason docker would be an option, but it's currently not supported.

A

If there is a big demand, I'm sure the condor team would spend like two days to make docker work the same way they use singularity. Just there is no demand right now. As far as I understand so now, I said we launched the condor demon inside the pod, so I'm already running in docker. How does this work with singularity? Actually thanks torrents and developments in the Linux kernel, it works pretty. Well, if you have a fairly recent Linux kernel, you can launch a star D inside the completely unprivileged kubernetes pod that contains singularity.

A

We actually import singularity from remote detail, and then you invoke singularity inside docker and the user. Just is the singularity image it's effectively a nested contain containerization and it works great. Assuming you have the latest kernel and that's what we do in PRP, because we have diligent users that, like very recent kernels, no matter what the underlying operating system is. However, that may not.

A

This may not always be the case. You may either have hoes that use old, kernels and buy all that means. St. OS 7 is not good enough, not with the stock kernel. You can upgrade the kernel, but if you use the stock kernel, it's 3 3 10 so way to ancient.

A

So you cannot do a nested continuation or your user might not need like singularity, and there are reasons why you may not want to use singularity. One very good reason, in my opinion is singularity, insists that the root partition must be read-only, so you, you cannot run a gem install inside your container. It will just fail, but a lot of people are happy with that, but not everybody.

A

So, in those case you both of those cases using docker inside similarity inside docker- it's not an option, so you really want to inject Condor into the user image. So the the container itself is just run wind, but in one level of continuation, the darker one from kubernetes.

A

We have done that too, especially for CMS users, but asking users to manually, install condor and configure condor. It's a big no-no. Users should be completely ignorant that condor is there so the best way to do it, you inject it at runtime and that's what we do. Essentially, you have a sink simple init container that injects condor with proper configuration into the user provided image and everything just works. I actually created there, a simple template: how to do it?

A

It's not completely ready to be deployed to just copy and paste, but it should give you 95% of what you need, and you just need to tweak it a little bit for your specific details. If there is more infor interest in how to do it more generally, I will be happy to talk to anybody who would like help. It was just not a priority for me so far and finally, I would like to stress that most atc jobs, tolerate preemption and kubernetes allows preemption, which is great, in particular in case of HT condor.

A

If a job is preempted, condor will just reschedule somewhere else. So, yes, you, you may have a little bit of waste, but from the user point of view, it's a non-event. The job will just start somewhere else. So if you put HT condor to manage your preemptable, pods users will be happy.

A

They will never complain that why things are not working because they are used to use some preemption and you can deploy hundreds or thousands or millions of pods at the lowest possible priority and keep your kubernetes pool busy at all time without affecting at all the productivity and the user experience of regular users, because the moment a normal priority in pod is installed.

A

The htc pot gets kicked off and it's a matter of delaying that other pot by maybe a second or two depending how it unit and that's what we do in PRP and it has been. It has increased the utilization of our pot drastically. It was very, very useful for one of our use of kubernetes, which is completely interactive. We have this huge virtual reality cave where, with that has what it uses: 17 GPUs, but it's interactive so the moment a user walks in and move the mouse it wants it. Whatever has been used there.

A

It must be all used for death, so you cannot schedule normal jobs there. You need to want preemptable ones and we schedule HCC preemptable jobs managed by condor because they just were great and the 23 hours a day when they are not used for interactive views. They run scientific world, which was a great great benefit for the community. So that's pretty much. All I have I.

A

Think kubernetes is a great foundation for HTC jobs, but it's a little bit hard to use for users that are useful to certain HTC features, which I think are very useful and they you just want done, but HD Condor can add that added glue, pretty effortlessly and it makes the ACC in uses very happy, while still maintaining most of the features that you want from kubernetes and we shall be using in osg and PRP this for over a year now and very happy with it. So that's it and just a little bit of acknowledgements. So.

A

If you have any questions I'm more than happy to answer, yes,.

A

Yes, HTC job can handle preemption very nicely, preemption means you're killed and he said.

A

It just got killed so most HTC jobs will just be killed and restarted from scratch from the beginning somewhere else. Now you you can put in checkpointing. So every ten minutes you could check point, but checkpoint is expensive. So the best way to deal with preemption is run for moderate amount of time.

A

So we recommend HTC jobs, try to be shorter than an hour, because preemption is a way of life and if you are extracted from from the beginning losing an hour, one per Mille of the time is a non-issue try to run for 24 hours, and you may never finish so as long as your HTC jobs are short enough p.m. you. Don't need checkpointing, because it's more expensive that to checkpoint then actually be restarted. A percent of the time.

A

Which checkpoint is definitely an option? If you want condor, will bring checkpoints back to the scaredy, so you can be start from the last checkpoint, which parent.

B

A

Yeah right now, this is the setup that we use for oil G. However, we have also another setup that is like this, where the schedule is outside kubernetes and the kubernetes provides all only the execute notes. So we have both both the tops are currently used just for different communities.

A

The pros of just having the worker notice I already had a head node and the scaredy that was deployed so I don't want the user to be required to log in somewhere else. So this was the most obvious one for this one. We didn't have anything so all G just wanted to use our kubernetes resources and it had essentially a portal didn't need the batch system behind so I deployed the batch system behind in kubernetes itself. So I put everything in it.

A

It really depends if you have already escaped and collect or deployed in that case, if you're happy with the current setup keep it, but it makes actually a lot of sense to put everything you cobranet is because kubernetes is actually a great solution for managing services and the collector and the skiddy are just two services. So kubernetes is a great tool for that tool.

B

A

Right now we don't have user that ssh into this Skadi. When is it played like that? Osg has a portal application that users can submit to and their portal that submits to the Skadi, but it will be trivial to install the SSH on that and expose it to the users.

B

A

Yes, this so the question was: is the scary part just waiting there doing nothing and waiting for users to submit yes, but 95% of the time we have way? There are hundreds or thousands of jobs cute in the skiddy. The whole HTC paradigms assumes that you have way more work than you have resources available. So if you have only occasional jobs, so you have more resources than jobs. Then some additional work is needed so that you don't have pods they'll, execute pods they're sitting doing nothing, but that's a different story. You can definitely manage this.

A

It's we have. We have been doing something similar, not for kubernetes but for the old style grid computing. It's called gliding WMS, which has have been involved for the last ten years. It's definitely possible. We just have imported to kubernetes because we pretty much have all all the time some work. There are some short periods when there is no work, but it's more an exception than the rule.

A

No right now, I think I have each pot currently requests like 8 CPUs and one GPU in a single request and then a corner partition them and give them out to various jobs, but that's completely configurable. It could go put one slot per pod or who not report whatever you think is more appropriate for your kubernetes scheduling.

A

Infrastructure. Condor is perfectly happy to give out the large be managing a large number of resources.

B

A

Yeah one problem which HPC it's, that the the the jobs are not singularly scheduler scheduled, so you have a lot of dependencies between jobs, so it's all or nothing. So you need to schedule about 1,000 jobs at the time or none at all, because it's if one of the thousand fails all thousand fail. So that's the big difference. Ht Condor was never optimized for HPC.

A

It really works in terms of one drop at a time, so it has a little some rudimentary support, but it's not really well used. People use most is learn from these days for this and you can do the same with learn. But you have to be aware that things like preemption, essentially a no-no for HPC because of this all-or-nothing scheduling needs, but everything else would just work. Yes,.

B

A

Well, Dakin is supported, but not in the same way. So doctor was supported before singularity, but though docker containers in condor are used in completely different when the singularity containers. So if you launch a docker container in condor, it will bring out it will checkpoint. The docker container at the end then bring it back which nobody wants. So very few people use it the docker container. Today everybody goes with the singularity one because it just has a different it was. It was designed in different times for different use cases in mind.

A

So, even though they are both containers, the semantics, it's completely different yeah it, the singularity support, came out of the whole grid, moving toward two containers and say well: first, they didn't want to use docker because it doesn't really play well with that with other HPC systems and told the docker universe. Didn't really make sense for the that use case, but they can just kept kept it because they thought people would want to submit containers and get back. The check check some of the container, because docker allows you to modify the root partition.

A

So they was it. Well. People want to store their data in the root partition, so we will bring everything back and, as far as I know, almost nobody uses that, at least in my community everybody likes the singularity approach where the root partition incident only and you want to explicitly bind in and out the data that you want so yeah. It's it supported, but not in the way you would want to use them inside for HTC. Really it's similar to their virtual machine universe, which has pretty much the same idea.

A

A

No-No-No, if you have a docker container- and you don't have the need to modify the root partition, you can launch it into singularity and everything will just work. The the only problem is when you want to use it for like interactive use where you want to launch an instance and after an hour it is running. It says oh I forgot to install this. Is the this application yeah we just um install it, um install it because it's easy. You cannot do that in singularity. It's not really something you want to do for thousands of jobs.

A

So it's a non-issue. That is why people really love lot of singularity for HTC. You want the container to be immutable. It's actually a good thing, so singularity is perfect for that. But not all users are happy with it, especially when they want to do interactive stuff. That's why I'm saying they currently not not support it, and sometimes people want to use the docker semantics, but it's not something most users want for HTC. It's really singularity works great, and this is why everybody has moved to singularity in open science grid because it just worked.

A

It's great for the purpose. It is needed.

A

Any other questions.

A

Nope so I guess we are done. Thank you for coming.