Cloud Native Computing Foundation Kubernetes Batch + HPC Day EU 2022, 19 May 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: How to Handle Fair Scheduling in a Private Academic K8s infras... Lukas Hejtmanek & Dalibor Klusacek

Description

How to Handle Fair Scheduling in a Private Academic K8s infrastructure - Lukas Hejtmanek, Masaryk University & Dalibor Klusacek, CESNET

While the usefulness of container-oriented computing is widely recognized, its adoption in academic environments is not so straightforward. Existing orchestrators like Kubernetes are not primarily designed to support fair execution of (bursty) workloads belonging to various researchers and/or competing projects. While public providers are using efficient pay-per-use model, academic use-cases often expect traditional fair-sharing mechanism which is widely available in current HPC installations. This talk will discuss the challenges related to the application of containerized computing within K8s-operated infrastructure used by various users and research groups in the CERIT-SC infrastructure. Specifically, we will discuss how CERIT-SC guarantees that eligible pods will be executed in a reasonable time frame, making sure that running pods of other users will eventually free their allocations to guarantee fair use of available resources.

A

Hello, let us present how to handle fair scheduling in a private academic, kubernetes infrastructure.

A

First of all, let us introduce ourselves. I am lukas hitmanik. I am architect at maharik university in czech republic and my colleague julie, berkley kaczyk is researcher at cesnet, which is national research and educational network, also in czech republic.

A

So the czech research infrastructure is currently built on two approaches. One is a hpc and the second is emerging kubernetes infrastructure.

A

Hpc infrastructure consists of 30 000 of cpu cores, 15 petabytes of storage capacity, and this infrastructure is used by 3000 active users. Those users are running 20, 000 jobs. Each day, kubernetes infrastructure consists of 2 thousand and 500 cpu cores. It seems to be smaller. It has half petabytes of dedicated storage capacity. It's currently used by 130 users and those users are running about 1 000 in books. Every day this talk is focused on our kubernetes infrastructure that consists of a large single multi-tenant cluster.

A

It means that the cluster is shared by many users. They do not have admin. Privileges are and are only given namespaces to run around their ports.

A

Basically, we have two kinds of jobs, interactive and hpc jobs, so the reference interactive job needs to run as soon as possible. Also they usually do not have limited runtime and they typically burst. I will speak about this a bit more later.

A

Contrary hpc jobs can wait in queue. They have a strict maximum run time limit and usually we have more jobs than resources, so they have to wait in queues.

A

Of course we want to to use our infrastructure efficiently, but it presents some challenges. The first one is bursting nature of interactive jobs. They typically run for a while and resting remaining time, but what to do with allocated resources?

A

Those jobs are mostly spaceful, so they cannot be easily restarted and in the graph below you can see typical runtime of interactive job.

A

Another challenge is how to prevent resource wasting. We collected a statistic for several months and we see that most port had a significantly overestimated request allocation. The red line represents idle state, but, as you can see, most allocations are below those namespaces are actually do not represent. They do not represent uh a single port, but usually several parts. So the situation is even a bit worse.

A

We also see other problems, such as it is impossible to modify both priority dynamically or adjust to general or tight pot allocation, because if we change allocation the potential to be restarted, this is no problem for stateless micro services, but it is usually bigger deal for long-running scientific computation.

A

You can easily imagine that if the such computation is stateful and runs for one month, it's not a good idea to restart it several times a week or something like that. Luckily, there is community work in progress. One comprise change resources without restarting pot. This is very promising work in progress and also checkpoints.

A

One is directly bernatech will support both checkpoint and restore. We can deal with some of the problems as well.

A

There are also problems bound with scheduling as a common hpc, but uh scheduler uh behavior, in the way that when the file system is when the system is full and new user arrives, you can always they tell the user. What is their priority or really wrongly estimate when the running jobs of other users will terminate or even provide them? A non-destructive reservation scheduler also make regulation for big jobs.

A

It means that if user request job that, if your whole node like 64 cpus or something like that and then hpc scheduler will do reservation on that node and on the sum of catch nodes and prevent a smaller job to arrive or occupy this nodes, and this is so automatic in kubernetes. It is impossible to estimate pot wait time when we are out of resources.

A

There are no guarantees and the pot either starts immediately or possibly can never start, because there are no free resources and we, I still put do not have any limit on runtime time. Those resources can be occupied forever. Also we all- or we can manually, adjust the priority of new ports to evict some already running port. But this is problem, as I have said, with scientific computation.

A

Also, resource reclaiming is not solved because there is no put to life cycle management and also there is you know such thing as fair share in kubernetes, meaning that kubernetes could guarantee a user that the port will eventually run and also kubernetes scheduler. Just cannot do any reservation for big bots only only at cost of eviction again, but this is still not good for scientific computation so and also there is no automation.

A

So this is all from our lightning to a short lightning talk and if you have any ideas that could help us also, if you know to solve the problems, we would be happy if you reach us at the contact below. So thank you for your attention.