Cloud Native Computing Foundation Kubernetes AI Day North America 2021, 30 Oct 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: A Better and More Efficient ML Experience for CERN Users - Ricardo Rocha & Dejan Golubovic, CERN

Description

Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

A Better and More Efficient ML Experience for CERN Users - Ricardo Rocha & Dejan Golubovic, CERN

Experiments at CERN such as the Large Hadron Collider (LHC) generate petabytes of new data every year, to be stored and analyzed by thousands of physicists around the world. In just a couple years, an upgrade to the LHC will trigger a 10x increase in the amount of data posing a challenge to the existing infrastructure. This session covers how machine learning has been gaining momentum in the high energy physics (HEP) community and particularly at CERN, as a viable option to handle the data growth with a similar amount of resources. The focus is on one particular service based on Kubeflow, and how we extend the existing functionality to offer our users a familiar and seamless integration with site services. How centralizing resources has improved our overall resource usage, how we extended existing functionality to manage end user tokens and credentials allowing access to on-premises storage, and how we explore tools like Harbor, Trivy, OPA and Falco to ensure a reproducible and secure flow from interactive analysis, to model training and finally serving.

A

All right so hello back and thank you for um for giving us the opportunity to present our experiences.

A

So today we'll talk about our experiences with centralized machine learning service at cern and using coop flow, and uh how will we've been changing it to to make it better for our users? um My name is ricardo rocha, I'm a computing engineer in the cern cloud team. I do a lot of containerized work, networking and some machine learning as well. I also I mean a member of the cncft technical oversight committee and I colleague the research and user group also of the cncf.

B

Hello, I'm glad to be here. My name is dan, I'm software engineer at cern cloud team. I work on containers, machine learning and cube flow and I will start with some introduction about our project and then ricardo will discuss things in more depth.

B

So we were concerned. Cern is a research organization for particle physics in geneva, switzerland and it's operating the largest particle physics laboratory in the world.

B

The mission of cern is to find the origins of the universe to answer the fundamental questions, such as what is the universe made of and to understand how particles behave at the smallest scales so to understand how particles behave at the smallest scales, we need to use very high energies, so to do that, we build particle accelerators and lhc is the biggest particle accelerator in the world.

B

So it's a 27 kilometers ring of superconducting magnets that runs 100 meter underground in geneva and in that magnet particles are accelerated near the speed of light and then they're collided at four different points for collision points. So after particles collided these high energies, physicists gather, results using detectors or experiments, and then they use this data to get valuable uh insights into the into the science of the smallest scale particles.

B

So we can see how magnets look like live, uh and this is one of the detectors we can see people here and how how this structure looks like- and this is essentially electronics to extract data from the collisions.

B

So how do we use machine learning at etser, so the data acquisition system works that works in a specific way. So uh there are 40 million collisions that happen uh every second, basically at the lhc, uh but computing infrastructure can only sustain 1000 events per second, so we need to somehow go from 40 million to 1 1000 and to do that we use trigger mechanisms.

B

So essentially, those are algorithms that select interesting events to to save and to further process so to which the question is what is interesting and how these interesting events are selected and to do that, we can either use deterministic algorithms or we can use machine learning algorithms, and for that we would use supervised machine learning, algorithms that can either run in l1 trigger on or at high level trigger to select the interesting events and this this works quite well.

B

If we know what we are looking for, but the question is what, if we are looking for for some new physics, so so far, machine learning has been used extensively, at cern, for example, to find to prove the existence of higgs boson uh boosted. Decision trends were used quite quite a lot, but there are other physics theories that were not confirmed by lhc data, which were expected to be confirmed, for example, supersymmetry or extra dimensions, so that hasn't been found in lhc data.

B

Yet so the question is what, if uh our signal hypothesis for trigger algorithms was wrong. So what what? If there was some kind of bias in supervised learning? So this calls for some unsupervised learning some algorithms that could actually learn during online processing that could train on experiment data not only on simulations.

B

So, besides these uh high level overviews of machine learning there are, there are multiple groups at cern that work on machine learning and they all have their own local infrastructure. So our motivation is to provide the centralized infrastructure where users can actually use gpus and fpgas and tpus, and they can that they can have a user-friendly platform to run their workloads.

B

So this is our motivation for kubeflow to develop a centralized platform that can be used by by different groups, and we have been working on that for for a bit.

B

uh The idea is to provide the full machine learning life cycle with kubeflow, to get data from detectors, to perform data, preparation to run some fast iteration jobs such as notebooks, and to validate machine learning models and then once we're happy with our models to do distributed, training and model validation and actually train our models and then to store the models and use them for serving for inference and production and with kubeflow. We can do all of that, and this is why we are using it.

B

So we started with a single user cubeflow 1.0. At this point, we were exploring uh available features, making sure that pipelines work that cutive jobs work and that we can run our machine learning workloads. uh So this was the initial stage. Then we moved to 1.1 instance with the multi-user and we integrated that with other services. We integrated that with the single sign-on with we are managing that with argo cd and we have on-boarded the users at that point.

B

So we're we're at that point still working closely with users to gather their feedback to discuss things to provide support more closely, and we were discussing the previous kubecon in more detail. The are bursting to public cloud, and currently we are working with one tree instance where our focus is on security.

B

We want to provide credential management and namespace management and vulnerability scans for docker images and then also some runtime checks. The idea is to provide the general availability of the service to be able to open it to thousands of people who who work at sir- and this is our our idea and ricardo will take over.

B

A

You uh I'll actually build on this one, so uh this this diagram really shows the evolution of the service. At cern, uh we got to the point where we could scale to the to the size and the number the amount of resources that our users needed, uh but then, before opening it in production, there were a couple of things that we we had to focus on, uh and this is the list that we see here under one three, and this is the requirements we had before, making it generally available on premises.

A

So the things I'll be covering here in addition to the resource availability I'll cover the management of credentials for the users, some uh like decoration of namespaces, with user metadata, uh the scans and and checks for for the images and then using things like opa for policy, informant enforcement and runtime checks of the workload. So these are really requirements that we have that are not only for a machine learning service, but are quite quite important if you're having a like multi-tenant multi-user deployment.

A

But the first thing I will cover is uh regarding resource usage, so um dan introduced that one of the motivations we had was to kind of improve the efficiency of the the resources we have at cern. Instead of having multiple groups having each each of the groups, several gpus, we wanted to have like a central uh pool of resources uh that would make it uh more efficient overall.

A

So we we have an example here where we have uh different groups uh at cern, say cms and atlas, which are experiments at cern, so atlas susie's super symmetry.

A

We had also groups in it doing anomaly, detection or in beam calibration doing reinforced learning uh while they calibrate the lhc beams. All of this is kind of inefficient, because each of the groups has to maintain their own gpus and also the resource usage is restricted to these individual groups. So moving to something like this, where we basically have a single entry point for everyone to come and and and then benefit from these gpus is, is a big improvement for us. Also, we can integrate things like fpgas and other accelerators as required.

A

It does pose some challenges because you suddenly start sharing the resources between users. uh The other thing we wanted to to do is which dan also mentioned, and we presented last coupon. Is we want to scale out like the amount of resources we have on premises are actually not enough for what we need in terms of accelerators, both in terms of gpus.

A

We don't have enough, but also accessing specialized accelerators, like tpus, which we'll probably not never have, and they are restricted to to the cloud providers or ipu's in the case of azure gpus for gcp.

A

So we want to abstract all of this so that our users don't have to understand the infrastructure they just have to run their workloads, the other part. um So here it it's where, where the integration starts, we we rely on coop flow as as we mentioned. But then, if we have this peak pool of resources, we might have say nvidia, p100s and mv100s, which are quite good at double precision, very good for distributed training when you're doing things like deep learning and we expose this via pci passthrough.

A

We also have nvidia t4s, which we use for notebooks training and and inference again using pci passthrough, but then, if you're, just using a notebook for validating your model with a small small amount of data, you probably don't need a full gpu. So we started looking at this idea of doing virtual gpus and the previous talk was was talking about sharing resources for model serving.

A

This is similar a similar concept and then also we started looking at adding nvidia a100s. They will arrive soon and here we can actually do physical partitioning instead of just time sharing as with the t4s and vgpus.

A

So we wanted to expose this to our users uh when they spawned um a notebook at cern, so they won't see. Just like. uh I want a gpu nvidia, they will actually have like a drop down box. That says, I want an uh like a full gpu or I want a virtual gpu, and here I would highlight what we are aiming for and we'll come back to. This is when you see their gpus available to use. No, we want to express to the user. Actually.

A

What is the availability of resources so that they don't just try senselessly to to get a resource? That is not there um and again. We we want to integrate the public cloud resources uh into the same same setup, so we are like halfway there. I would say the other part which is uh yeah I mentioned.

A

I mentioned the a100, so if you've played with the nvidia t4s, you know like you, can virtualize these gpus or v100s uh with this time sharing, which is kind of like a fake partitioning you're, not actually partitioning the resources, uh but with a100s there's this multi-instance gpus support, which is really exciting, because it gives us a lot more flexibility on how to partition resources to the end users without having to to like do any compromise in terms of the expected quality of service and then really building.

A

On the previous talk, I would, I would complement uh the things we are doing so I mentioned nvidia virtual gpus with uh time times for time sharing uh there were. I will just put a note here if you are using this. One thing we learned is that uh this was not really suitable for all our users, because the ones that need gpu profiling or want to use a tensorboard or something similar- they actually require uh the profiling to be enabled on the gpus.

A

This was a limitation with the 12 drivers, the version 12 drivers of nvidia. This is actually fixed in version. 13 we've tested this, so we actually are about to deploy this to our users. This will make the use of our t4s much better, but again, the the next step is what we are getting early next year, which is this new nvidia m100, where we can do up to seven times physical partitioning- and this is something that is uh supported directly on kubernetes.

A

The nvidia drivers are able to do to manage mig uh mig cards, so this this is really great, because the vgpus were are not something that kubernetes uh handles uh natively um yeah and then the other part is the multi-model serving. This is also a requirement we have to make.

A

The best out of the gpu is to be able to split to to reuse the same same gpu for serving multiple models, and this is uh something like this is a very basic example where you basically create a single inference service, and then you have one or more actual models that are linked to that inference service. So this is something that we also try to do, and it's uh it's uh like a follow-up to the to the really good previous talk.

A

So the second part I would mention is this neces this requirement. We have to integrate with our on-premises resources, uh so we rely on mutating mission web hooks for for all of this, uh we are actually using the opel gatekeeper in if you've deployed coop flow. You know that you can do a lot with customize in terms of changing the the yaml that is being used for the deployment, but this is not enough.

A

um One example is, for example, the notebook template to customize the notebooks in the jupyter web app, it's quite limited on what you can do this. If you want to like change the template dynamically based on some uh runtime information, you can do this with customize, so we do this with the mutating web hooks.

A

This is used already in different parts of coop flow. I'll, give some examples and then I'll go a bit deeper.

A

One thing if you want to, for example, to start a pod, speed, notebooks or pipeline jobs with the proper uid gid of the local users. For multiple reasons we we do this change at runtime.

A

Another thing is that we need to manage the credentials for the user so that they can access like storage systems or any kind of other internal system. We inject those credentials also at run at deployment time of the pots creation time um yeah. The volume amounts for for the internal storage systems as well.

A

So this is an example for the credential management. uh I would say mostly, we have two types of credentials that we have to handle. The first one is kerberos. The second one is oauth2.

A

um Most of our services require one or the other or both so the in both cases they are short-lived, which means like you might have a credential. You spawn your your training, but it takes hours, they will expire. So what we've done? We've written a small tool that actually manages the credential renewal for the users uh transparently, so when, when they they basically get a notebook, they will upload say a covers credential to their namespace, and then the tool will know how to handle this and keep the credentials up to date.

A

So if they then submit a pipeline uh training, job uh or any kind of other workload on the cluster, the the amounts of these secrets will actually make the credentials available for for the workloads to use- and this is true both for uh oauth2 and and corpus, and you can see here like a diagram. The first step is for a user to like push the credentials into a secret and our job will renew them and then the actual workloads have this mutating web hook.

A

That is actually uh mounting these secrets and making them available so that they can access things like storage or the internal spark clusters. uh The batch cluster, which is based on hd, condor the registry to upload their models. Things like this.

A

The second part, is what I mentioned, that we sometimes have to annotate workloads with the metadata of of the user. So an example is, is again, notebooks have to have a uid gid that matches the actual user. So we do this by actually when, when a new user is on boarded they get the private namespace, but we have a component that will basically fetch all the metadata from say ldap.

A

In the internal ldap and we'll put all of this as annotations or labels in in the namespaces of the users- and the same is true for groups, this means that we have all the metadata. We need about the users to then do mutations that at runtime, when creating the pods to to deploy them with the appropriate security context, all the proper user group that is required. This is a like a requirement from our security team, and then the last one I would mention is the internal registry.

A

So all we mandate that all the workloads running on the cluster have to come from our internal registry, which means that they've they've been through the vulnerability scans with 3v. For this we also sign all the images.

A

Then we have the different policy enforcements at creation time, which, for example, prevent external images from being run, but also can do checks like is the security context, correct or acceptable or do? Does the workload have all the metadata we need like for accounting, for example?

A

So we do. We do have some some sort of caching and replication from dr dockerhub into our internal registry. That, then, triggers all these scans and then we we go from there. The second part is that at run time like this, this will help us at deployment time. But if you have long running jobs, we also need need to do some checks at runtime, so we are using falco for this, and we do basically two main things.

A

One is to see that whatever vulnerability checks were done are still valid, that there's no new vulnerability for some sort of long-running notebook, for example, and the second one is that the workloads are doing what they are expected to do so, there's no shells being spawned inside the container there's no packages that are not supposed to be installed being being installed. Doing some weird system calls or network connections. So we do.

A

These checks live and uh one thing that is quite uh uncommon, uh at least, but I think everywhere is that when we described the stack to our security team, they were actually very happy, and this having a security team being happy about new services is not common. So this was.

B

A

B

I think so so far we have been working with couple of groups at cern who have been using our kubeflow instance uh and in the next couple of slides I'll describe some individual user feedbacks.

B

So the first one the users really like, is the integration between notebooks to pipelines. They really enjoy using kale, so if they can go from a notebook to a pipeline seamlessly without writing any docker images or anything with kubernetes that that's really a big advantage for users. So this is something we have a very, very positive feedback. So far, then users really enjoyed the ability to get resources on demand.

B

So, for example, if they have some models that need hundreds of gpus to properly train, they can't have that locally and on the public cloud we can provide access to this number of of gpus. So users really like that and they ask for such uh abilities and then there we have some advanced users. For example, we have a group at atlas, one of the experiments that they have some. They have a repo with machine learning project and then on that project they have continuous integration and yeah.

B

Whenever they commit they want to run some continuous integration. uh They would like to add a step that would trigger a pipeline into our cube flow instance within their uh ci, so they need the api access to to cube flow.

B

So this is something that we also need sometimes, and then we need some better ui for when uh resources are not available, for example, for notebook creations for gpus, uh we implemented this additional tool that tells us if we have uh available gpus in our system and as ricardo was mentioning to select uh the profile for for a gpu. So if users could actually automatically see if they have a gpu it it's much better for them so to to sum up, kubeflow has been very well received at cern.

B

um We are about open to all users and all groups after we validate with our security uh requirements, that ricardo was discussing so yeah. We want to provide credential and name space management as well and for the improvements from the actual cube flow. It would be great if we could have a complete isolation, for example for uh pipelines.

B

It's already done in the back end. It would be really great for us to have that in in front end as well also pipeline artifacts to be isolated and also we would need some better debugging of katip jobs, because users can't see logs on the ui and yeah, maybe a better feedback. Speaking of resource availability, but in general uh cube flow is like whenever we have a presentation in certain people are very excited and our users so far really enjoy using it. So, thank you very much and guess.

B

If you have any questions, let us know.