Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2021, 29 Oct 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Implementation Challenges: From HPC to Containers in the Academy - Lukáš Hejtmánek & V Spišaková

Description

Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Implementation Challenges: From HPC to Containers in the Academy - Lukáš Hejtmánek & Viktória Spišaková, Masaryk University

Container computing does not have a lengthy history in the academic environment. Traditionally, HPC holds the first place because of high resource availability, direct multi-tenancy support, and researchers having legacy experience. This session will introduce efforts made at the CERIT-SC/Institute of Computer Science of Masaryk University to implement Kubernetes infrastructure and move scientific computing to containers. We will present challenges of multi-tenancy assurance, deploying applications under users, resource sharing, and building trust towards containerization among the research community. We will discuss several created solutions, present a couple of European open-source projects and demonstrate how containers help in the academic environment. Furthermore, we present other issues we fight and propose ideas on new features.

A

Hi today we are going to talk about implementation challenges from hpc to containers in the academy and who we are.

B

Hello, I am luka hitmanek. I am from makarik university and existence from the czech republic. I am rit architect of our infrastructure, which consists of both storage and compute nodes.

A

And I am victoria, I'm also from asari university, where I work as an I.t specialist, but essentially I take care about our kubernetes infrastructure.

A

So our czech national research and education network uh is called a infrastruct and it operates uh hpc environment we have approximately 20 000 cpu cars, 200, gpus and 60 petabytes of storage and the computational resources are accessible mostly through batch system. Pbs pro and storage resources are accessible through carburized nfs version. 4.

A

storage can be also accessed by s3 or cf rbd, but just the minority of users choose this way and we have about 1000 active users.

A

So in hpc we have two types of resources: computer resources and storage resources, users interact with compute resources via creating shell scripts and running them in a batch system, bbs pro they have to have ssh experience, because these batch systems do not provide any graphical user interface.

A

We try to change this by implementing open on demand. That was our attempt to provide some graphical user interface and, secondly, storage storages are directly available on worker nodes.

A

However, storage resources are spread among various czech cities and this creates confusion for users because they have to take care about compute and data affinity. On the other side, these storages can be mounted to users. Computers, hpc brings some troubles. First of all, there is no straightforward way to monitor running computations, so users can't easily check the state of their jobs.

A

Secondly, older scripts are not compatible with today's technology. I bet most of you know python 2 versus python 3 issues. Furthermore, every user must possess at least average unix skills because they have to interact with batch systems that are run via command line.

A

Another problem is caused by time limited access to carburized storage. After certain time, users have to renew their access token.

A

Last but not least, setting up nfs client is a hard task for every user when he or she wants to access their data from their own computer.

A

So far I have been talking only about hpc, but as we are at kubecon, I will move to containers now.

A

I think that majority of you have already heard about ngc containers or biocontainers that are meant to be used in hpc, but how do we use them in hpc in a shared infrastructure? Docker is mostly prohibited due to security issues, so we can use singularity, but this tool has problems of its own and also we have another option: that's podman, but if we think about it, why don't we use native container infrastructure as an infrastructure manager? You can choose between building shared container infrastructure or letting users run their own container infrastructure.

A

Building shared infrastructure has advantages mainly for users. They don't have to deploy and maintain the whole infrastructure, which is in fact much more complicated than just being able to work with ssh and write some shell scripts.

A

Instead, they can focus on research and their work. However, some infrastructure managers just provide users with right tools to run their own infrastructure, such as openstack magnum.

A

Here in czech republic, we decided to go with the former approach and build distributed container infrastructure. We operate several kubernetes clusters that are built on ranger kubernetes engine second version, together with ranger dashboard users can work with these clusters in multiple ways.

A

They can either interact with native kubernetes api or they can work with pre-deployed containerized applications or frameworks, or they can work with rancher dashboard to explain a bit more. In case users choose to interact with kubernetes directly they get their own or shared project. It depends on use case, together with the namespace.

A

They can utilize various persistent storages, such as nfs, cf or s3, and they can also utilize gpus and infiniband.

A

In case users do not want to interact with kubernetes directly, they can take advantage of applications which we have predeployed, such as containerized jupiter hub and binder instance of bioinformatics workflow tool galaxy.

A

We also offer kubeflow and 3d accelerated, desktop that can be used by users to run gpu intensive applications such as ansys or matlab remotely.

A

Furthermore, we also use various frameworks, such as european gh4gh tests and west standard next flow or snake make having all of this. What are the benefits for users? First of all, users, don't have to be skilled in shell, scripting and other nasty, unique specularities.

A

They don't have to interact with kerberos and configure nfs. They completely don't have to be aware of enron topology.

A

They don't have to know about software modules and their dependencies because we provide them in pre-prepared containers and overall, this is a direct way to run hpc containers easily.

A

Every coin has two sides and from the beginning I have been talking just about one side, the brighter and better side. However, lukasz will now talk about the second side, which is much more darker.

B

So, let's look at the dark side of containers in hpc when we think about containers in hpc. There are some challenges that I briefly introduce here. The first challenge is kubernetes and hpc integration. Why do we need it? Imagine you are running some kind of infrastructure and you just can't say: user is okay. We are finished here and from tomorrow. Everything is different, it just can't happen and also, if you don't have unlimited budget, you cannot just build new parallel infrastructure, so you need to make some kind of transition in place here.

B

The second users are familiar with queues and they also expect some fairness of the infrastructure. Excuse our natural thing in hpc, the third one scheduling.

B

If you have significantly more users than compute or storage resources, you need to schedule them somehow and last but not least, we need to get users trust, so they are willing to use the new container infrastructure.

B

Okay, we are talking about kubernetes and hpc integration. What does it mean? It means how to unintegrate existing hpc infrastructure with kubernetes hpc infrastructure typically consists is consistent from three parts: its um authentication and authorization infrastructure.

B

It contains compute nodes and it contains storage, and if we want to integrate it, we have to deal with all these three parts.

B

Well, authentication and authorization infrastructure can be shared between kubernetes and hpc. This is because, once you have some user database, you can actually use it, for both credentials can be used for both part. So there is no problem here.

B

Also work nodes are easily shared between kubernetes and pbs pro, because you can easily drain the node from containers or drain for pbx projects. As long as you notice free, you can assign new type of function on it, so you can create between kubernetes and pbx pro and, moreover, you can even run both types of workload. At the same time, if you utilize pbs pro kubernetes connection connector, but we do not use it.

B

On the other hand, storage storage is the real challenge and we will look on it in more detail.

B

A

B

I have told hpc storage integration is a real big challenge.

B

This is because hpc infrastructures usually are based on nfx or afx file systems and those file systems are meant for large data that the user can store and then stage it to work nodes and can do the computation on them, and you would need some the same for kubernetes, so how to access hpc storage from kubernetes.

B

The short story is you can't? This is because you have to deal somehow with user authentication. The user authentication is usually done in two variants you can use either access tokens or the whole authentication is based only on user ids problem with access tokens is that those tokens do not understand. Namespaces and, as you may know, whole container world is based on namespaces.

B

Also, there is minor problem. The access tokens are usually time limited and you need to solve somehow how to renew the token. The.

A

B

Option uid is, there is a problem that most containers run as user 1000 and you can't distinguish between users only on user ids, because all the user users have user id 1000. So you need to remap somehow user ids between local user 1000 and some user id. That is on a remote site on in hpc.

B

So for storage we tried or somehow utilize three kinds of file systems. One is the old and known nfs second is sshfs, and the last one is a common internet file system in which you can known from windows world like samba protocol in the case of nfx. Well, there is no uid remapping possible. On the other hand, it's fast- and there are many csi drivers for nfx, but user id remapping that is missing is a real problem.

B

You can use an effect for persistent volume claims for local storage that is dedicated to kubernetes, but you can't do integration with h with hpc that use variable user ids for that for every user there is different user id for in in the case of sshfs, uh it can remap a user id, but on the other hand, it's low and also there is a problem that csi driver must not restart, because if you restart csi driver all modes get broken and the user lost access to his or her data.

B

In the case of common internet file system, it can actually do user id remapping. It has acceptable performance, at least in the latex versions, but unfortunately it cannot widely support it in hpc work. So if you came, if you come to hpc administrator- and you tell him or her that you want to utilize common internet file system, the answer is no. You can't because we don't offer this file system.

B

As I have told queuing and furnace is another challenge: queuing is currently not present in vanilla kubernetes, but you can install some addons, such as armada that present some systems of jobs and queues, and this is really similar to hpc system.

B

But do we need queuing system at all? Queuing system in hpc is usually used for distinguishing between different kind of worker nodes, such as smp nodes or just hd nodes, but for this functionality you can fully utilize the kubernetes system of labels, and we believe that the labeling system can actually replace queuing system that is used in hpc.

B

However, we also need fairness. We need that some say: greedy user does not eat it, whole infrastructure, so every user can eventually do computation.

B

So we need some kind of fair use policy and we need some mechanism that can force this fair use policy and in and in this area we are not aware on solution yet, but we think that resource contacts and priorities can at least help in this area, and maybe this is all we need at all.

B

Another challenge is scheduling: pbs pro contains complex scheduler that is able to do complex, math computation of where the particular job will run and when, on the other hand, kubernetes container is a simple scheduler that does not contain such a thing as a fairness and complex computation of order of pots and also pbs pro scheduler can handle situations such as a big job eventually will run on any node, so the node is drained and the job is running eventually.

B

This is something that kubernetes scheduler doesn't do, and we have problem that and we also solve such problems, that user submit job that is greedy on resources, for instance, it needs a 60 cpu cores and kubernetes does not drain any node to satisfy this requirement.

B

It can be called by port eviction if you configure the infrastructure that put can be evicted from from a node, but this approach is not good for hpc case, because imagine that pot is running some computation okay for one month and you definitely do not want to evict in the evict it almost at the end of the month, because it means that the whole computation will run from the beginning.

B

Also, almost all scheduling, algorithms assume that jobs has infinite running time, but many kubernetes resources such as the deployment or stateful set, does not comprise time limit, so they potentially run in unlimited time, and this is not a bug. This is something that this kind of recorded design for it. They should run on in unlimited time, but for schedule for scheduling this kind of uh pot is real problem because they can't deal with such such situation.

B

So this is still open question how to do scheduling in kubernetes. That would somehow be similar to ppx pro, but our goal is not to reinvent just new pbx pro system that is fit to kubernetes infrastructure.

B

Last but not least, challenge is how to gain user trust. Users currently use hpc infrastructure, they have problem with it they, but they have at least something it eventually work. But if you introduce them a whole new infrastructure, they are naturally afraid of changes. Will it work? Is it stable and mainly?

B

Is it some hype, or will it be stable infrastructure for next years? You have to convince them that, yes, it will work, yes, it's stable and, yes, it will survive next year and it will survive for so many next years. We also need to build better portals to make it easier for the users.

B

Our future plans contains continuing transition from pbx pro to kubernetes, and we also plan to build some experimental setup.

B

We imagine that work and also will be equipped with large ssds, and we want to build fast, shared storage from these ssds from the volcanoes, and the challenge is how to provide reasonable data. Redundancy mold object solution consists of just replicating data, so you have at least two or maybe three copies of data, but in such a case you effectively effectively reduce the capacity to the half or even one third, and we want to find a way how to do it more effectively.

B

For instance, utilizing some rate six equivalent or rich element codes with better redundancy, and we need all this because we are pretty sure that some of the workaholics will be down for some amount of time for say reports or they can have hardware failure, and it's not acceptable in such case that whole storage is not, it can cannot be used.

B

So to conclude, they provide unified container infrastructure in czech infrastructure.

B

It's multi-tenancy, so many users can connect it's suitable for both web services and heavy hpc loads, and we are already running hpc logs as you, there was mentioned that we utilized the next flow framework, then so from this framework, we already run tens of thousand job just to prove that our infrastructure is working.

B

So this it's all from us and thank you for your attention.