Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2020 - Virtual, 4 Sep 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Reimagining the Worldwide LHC Computing Grid on Kubernetes - Lukas Heinrich, CERN & Alessandra Forti

Description

Don’t miss out! Join us at our upcoming events: EnvoyCon Virtual on October 15 and KubeCon + CloudNativeCon North America 2020 Virtual from November 17-20. Learn more at https://kubecon.io. The conferences feature presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Reimagining the Worldwide LHC Computing Grid on Kubernetes - Lukas Heinrich, CERN & Alessandra Forti, University of Manchester

At CERN, the home of the web, the Large Hadron Collider at CERN hosts some of the largest physics experiments ever constructed. The distributed infrastructure built over the last 20 years -- the World-wide LHC computing grid (WLCG) would today be described a federated, multi-cloud and multi-cluster deployment, mostly made up of a diverse set of research institute batch systems deployed on bare-metal or VMs. The rise of containers is an opportunity to reimagine the WLCG deployed on Kubernetes. In this talk we will cover how we leverage CNCF tools in this transition. Using e.g. containerd snapshotters for worldwide image distribution, Helm GitOps for reproducible configuration of multi-cluster setups and our deployment of services and batch workloads in that system. In a demo, we will deploy a member of the federation which will then receive workloads to analyze subatomic collision data.

https://sched.co/Zetd

A

Okay, hi there welcome uh my name. Is lucas, heinrich I'm a particle physicist uh working at cern and I'm going to give this presentation together with uh alessandra uh who's, also connected. Who is a senior system administrator at the university of manchester and we're both working uh for a particle physics, experiment called atlas and we're going to talk about how we use kubernetes, to streamline our infrastructure and so just to introduce our work a bit. So uh the main place of our work is cern.

A

It's the european lab for parking physics, research and it's located in geneva, switzerland, and so here we do many experiments relating to particle nuclear physics. But the main experiment uh that you might have heard about is the large hadron collider, it's a a very big particle accelerator that is a 27 kilometer in circumference and basically it's a big underground tunnel, 100 meters underground, where we accelerate particles and and bring them to collision at four interaction points.

A

So we have two particle beams, one going clockwise in this big circle and the other one going counterclockwise and at the four points that are labeled, alice, cms, lhcb and atlas. We bring those particles uh to collision, and um here we can see uh the uh what it looks like in the tunnel underground. So this uh huge magnets that accelerate the particles to almost the speed of light, and so they traverse this 30 kilometer tunnel 10 000 times a second and at these interaction points.

A

uh Basically, what we have done is we have constructed giant cameras that uh take uh basically three-dimensional pictures of these collisions, and these uh pictures help us understand nature uh better. So here we basically collide 40 million. uh We see 40 million collisions per second, and so we generate a huge amount of data, and so just to give you a scale of what these uh quote. Unquote, cameras look like, so this is a real life picture of one of these detectors.

A

So this is the atlas detector that both alessandra and I work at, and you can see a human uh at the center of the image just for scale. So these are some of the biggest um scientific experiments that have ever been constructed and we use that to understand uh the basic building blocks of matter. So some of the things that we are trying to uh understand with this large head-on collider is what the basic structure of nature is.

A

So we want to understand how the elementary particles uh work, so you might have heard about electrons or quarks, and but there are a bunch of other elementary particles, and maybe one of the things we would really be excited about is to find new elementary particles and through that uh uncover the nature of dark matter or maybe dark energy, because it turns out, though, with the current theory of physics, we only understand five percent of the energy content of the universe and actually 95 of uh what's out there in the universe is unknown to us, and so one of the other big questions that we're also studying is what exactly the difference between matter and anti-matter is because we see a lot of matter around us and while we know that anti-matter exists, it's not very abundant in the universe, and this is a big mystery that we're trying to answer with this uh big experiment, and so one of the things you might have heard about the large hadron colliders.

A

That is, uh we used it to discover uh the higgs boson. So this has been a central piece of our physics theory and it explains what the nature of masses and particles have mass and so in 2012 we analyzed our data.

A

We could find these new elementary particles, and then uh professors, uh higgs and anglair uh subsequently uh were awarded a nobel prize through uh because of this discovery that we made here at cern and another uh reason why you might have heard about cern is because, uh as we're all uh connected online uh through the web, uh this um is in part because of the the world wide web was developed at cern, so tim berners-lee, when he was working here in the late 80s er in the early 90s, uh he uh developed the weapon, it's uh basically the foundation of what we all use today and so uh we need to.

A

You know: we need to record a lot of data to um make these types of discoveries, and so it's not it's not surprising that big data and a lot of it infrastructure has always been very closely linked to the physics that we're trying to do so.

A

Not only do we have this magnificent harbor, these huge detectors, but we also have a quite expansive it infrastructure, and so this is uh one of the control rooms that we have at cern and so just to give you an idea of how much data we're taking so the big atlas detector that you've seen earlier in the slide, has something around 100 million readout sensors, so that all stream data back to our disk, and so we see 40 million collisions per second, and that gives us a data rate of almost uh petabytes uh per second.

A

So obviously, we cannot store all of this data um to this, but we do an uh real time processing of the data and then, in the end, we uh store to data to this uh something like gigabytes per second, and so this accumulates order of 10 petabytes per second. That all needs to be analyzed in order to give us a potentially a discovery like the higgs boson, and so in order to process all of this data.

A

We are not only relying on the central uh laboratory at cern, but we are a worldwide collaboration of physics, institutions, the national labs and universities that all pool resources in order to process this data. And so this is what we constructed during the 90s and early 2000s. And we call this the worldwide lhc computing grid. And so alessandra is one of the leading experts in this area and she'll basically introduce you how we use that and where we see kubernetes uh playing into this.

B

Yeah, okay, so I'm going to talk about the wlcg computing grid, which was created around 2000, so it is 20 years older, it's federation of those of in the individual academic institutions and they provide the resources to global scientific collaborations.

B

The lhc experiments are the primary users, but there are other collaborations using the grid participating to the grid. There are 150 sites that actually pledge the resources.

B

They are from big international labs to smaller universities, to even department clusters and, at the last account they offered roughly 1.4 million cpus for around 10 000 users.

B

Of course this is a distributed environment, so the challenge is to provide the users with a uniform experience across a heterogeneous computing landscape and the greater middleware was developed to act as an overlay to provide a common api to all these different resources.

B

Of course, the.

B

Was also flattened a bit in the past by the experiments and the sites agreeing on what is required for a site to participate to the grid, for example, uh the main computing resources that the worker, not behind the batch system, had all to be had or to install a reddit family operative system.

B

They needed a internal cpus and at least two gigabyte of ram processor, also a site in general needed a number of middleware services to be deployed to be recognized as a grid site.

B

The most important one, of course, for this talk is the batch system, but there are a number, a number of other services uh like a storage with a grid protocols for transfers, information system, authorization, monitoring, squid software space, etc.

B

This is a quite heavy duty and rigid deployment model and, of course, can be applied only to pledge the resources, but the resources are diversifying and becoming more opportunistic and we don't have any more control over them and what they deployed.

B

um So the ewcg experiments have already started to adapt to this new way of deploying by deploying, for example, containers to ease the requirements on the operating system. For example,.

B

But this is not the only thing that needs to be done so for how? How can kubernetes can help the deployment in this case well for sites it offers scalable and centralized service employment?

B

Many of the services mentioned before can be now centrally deployed by someone from the experiments or someone decided, maybe in the in the region- the uk, for example, as an an organization that can do this kind of deployment.

B

This reduces the needed to train admins and grid services, which are very, very peculiar to our environment and can allow to train people instead on something that is more required also around the world also k8 allows horizontal scaling, which allows the services to be scaled and up and down depending on the requirements, since a site might be diversifying, for example, they might want to deploy resources behind the jupiter hub rather than a batch system.

B

At a certain point, these can be very helpful and also it allows arbitrary, arbitrary size deployment you can deploy kubernetes on a laptop or in a large have a high availability cluster, and these, as we will see in the demo, uh is a quite an interesting feature as a batch system.

B

It is good at doing a resource management. Of course it has a major missing feature for us, which is fair shares. So right now we are deploying it uh mostly at dedicated sites. Dedicated means that it is different from pledged dedicated means that they run only one experiment.

B

So, in particular, since lucas and I work in atlas- say how does this work out in atlas uh in atlas? It allows a to streamline or to provide a common basis on ethereum genius resources, in particular on top of the standard, the batch system sites. We have started to use hpcs and cloud, and our system.

B

Has a brokering system that calls to this service uh called the harvester harvester? Does the submission of the workflows and has an api for each system when we arrived to create this simplifies things, because now we can submit it to a laptop to a university or, for example, to google and lately also to amazon.

B

B

You can see here, we started to work on it last year and there is a state decrease of resources and of sites that are starting to deploy, and you can see that there is a there are all the resources uh represented.

B

There is cern, there is a university of victoria, uh there is a university of chicago, but also google and amazon, and in the future we expected to this to increase the resource. Taiwan, incidentally, which was the first one to deploy.

B

How how did we achieve this?

B

We have, as I say, that is a system the the user submits to the broker. The broker creates a central queue depending on what the workflows requirements are. He talks to the harvester and the harvester brokers to the appropriate resources.

B

Harvester doesn't submit the workload the workflow directly submits, something called the pilot. The pilot is, um and as a set of scripts that do intelligent things on the worker nodes stages in the data runs the payload stages out. The output monitors the the job monitors the resources.

B

If, for example, the jobs exceeds the memory requirements, it kills it also it here, it isn't really shown, but in reality the job is run in a singularity container and what we have done is uh forever kubernetes is simply to do all these incubators.

B

Both are stereo has been reinstalled as a kubernetes resource and it submits it to a cooperative cluster and the pilot runs inside the pod. So in the case of kubernetes, we are also running nesting containers, which is another interesting feature, uh mainly his container, the singularity inside the container d.

B

Eventually, in the future, the declarative paradigm of kubernetes may take over it's at least what we open. But it is a bit too early and uh we will simply transform the pilot features into kubernetes.

B

Files configuration files, job configuration files so and with this I'm passing back to the to lucas for introducing the demo.

A

Okay, yeah thanks alessandra, so uh in this case we'll do a little bit of a demo. So if you remember so that worldwide llc computing grid, you can basically think of it as a very large federation of clusters and so traditionally well there were standard batch clusters as you would use them in in academic institutions.

A

We of course now have kubernetes that we could use to join this worldwide federation, and so what we will try to do in this demo is to set up infrastructure on a brand new machine to uh create a kubernetes cluster and join this worldwide uh federation.

A

Then we'll also submit a workload into this federation that will then end up on our uh just deployed infrastructure and do some physics work for us, and so the demo that we prepared is basically um simulating high energy physics collisions, and so this is a very important aspect of our physics work. So we need to kind of understand what our data looks like and so for that we kind of simulate this uh using some uh simulation code, and so with this I'll.

A

Just pass on to a demo video that we recorded and after the video we'll come back for the q, a.

A

Okay, so in this demo we will try to deploy a miniature version.

B

A

Infrastructure.

A

One cluster that I'll name the job cluster, uh that's a cluster that actually does the physics work. uh Once uh you know, work is requested and then another cluster called the harvester and its role is to monitor the queues of the global lhc user community uh to see whether or not new workloads are coming in and they will assign worker jobs dynamically to adjust to the workload and those worker nodes can then fetch new jobs from the queue as they are coming in and so at.

A

First, we will deploy this job cluster, so we will do this using mini cube and to prepare it properly. We need to deploy also a certain specific file system called cvmfs and we deploy this using a helm chart, and this allows us to mount specific, persistent volumes into the parts that we will be starting, and so uh once uh those uh this file systems are installed, we will also provide the cluster with a secret that allows it to communicate to the global lhc job queues and okay.

A

So this is done and now we will deploy the second cluster, which is this monitoring cluster called the harvester, and so we will use this time kind to deploy this cluster, and so we will wait a second to get this cluster up and running, and once this cluster is up and running, we will then use helm again to deploy this uh harvester service, which monitors the queue the lhc cues for uh work and okay. So.

A

Here we're using a helm to install the service, and so once the pods are up and running we're ready to submit some workloads and we'll just wait. A second to get the pods up and.

A

A

Okay, there we go so at the top, then now we use our standard uh user facing job submission uh command line tools to submit a workload into this global queue and our cluster that we just connected to this global federation will pick up this work and perform this work that we requested for us, and so we get a new job id and we will actually go to the lrc uh global monitoring page to uh see this new workload that we just submitted, and so we can see that the workload has been registered.

A

And so since now we have a bit more workload on the queue that we submitted to. The monitoring service submitted a bunch of parts in order to satisfy this demand, and so we'll see that one of these containers is now um creating, and so once it's creating, we can peek into the container just to see a little bit of what is going on inside of this workload and okay.

A

So now it's up and running and we'll exact into it and we'll peek into a few uh log files as they're coming in, and so uh here we see that it's now connecting to the global queue and it's asking for new work, and so uh we see that uh fetching some new work and uh once it's gotten its new work, our workload will start, and so I will just uh pause here for a second uh just to show that uh this workload is requesting its work to be performed in a docker image, and so we will need to run basically nested containers.

A

So uh here we see that it's requesting a container image uh called atlas, slash at s simulation and uh now. um Actually the workload is running in a singularity container inside of the pod that is running in kubernetes, and so in order to uh start this workload, we of course need to first pull the image and singularity will do this for us and pull the image for uh from docker hub for us, and so we'll just monitor the progress of this software acquisition.

A

And uh so here we see it's um pulling and those scientific images are uh usually quite big. So uh this particular image is, for example, uh something like 15 gigabytes, so so we'll speed up the video a bit just to uh see that this is pulling correctly and in the meantime, in the monitoring hours, job status has changed uh from register, to running uh just to say uh so to signal that uh actually one part has taken on this workload and is trying to complete it, and so.

A

Okay, the docker image is now uh completely pulled, and now we can see that singularity is now starting the actual workload.

A

And so this is where the physics start, and so we can see uh various log files where now we're basically running the simulation job and the workload in question here is basically a simulation of how particles traverse through this particle detector that we've built, and so it's basically using a virtual version of this detector that we have coded up in order to simulate what our data that we actually record underground is going to look like, and so we can see a various uh physics thing going on, and so now the workload is also finished and we can go back to uh the pilot which manages this workload internally.

A

In this part and see that the resulting data that this workload produces is going to be staged out again to our global storage, infrastructure and okay. So now this is all done and we can go back to our uh monitoring to see how our job is uh being reported there. And so uh one of the nice things is uh is that this is the actual monitoring page that we use for all our work and we can look at what host actually executed the job.

A

And so we are checking the hostname here of our pod, and we can see that the it reported the right uh host or report that the cluster that we just deployed uh actually took care of this job and completed this workload. And so, if you refresh and now the job should also show up as finished, and so we see this, and we can then look at the log files. And here we see basically the same log files that we were just tracing in the pod itself.

A

And so, if obviously not everybody has access to the clusters. But everybody gets basically these trace logs uh at the end of the workload okay. So this uh concludes the demo and it's a a very quick version of the infrastructure that we're trying to deploy on the wlcg.

A

Okay, so uh this was the demo, and so this basically showed us how we can use cloud native tools like kubernetes and helm, charts to very quickly deploy. uh You know, resources into this global federation that actually run in production, so we actually connected these kubernetes clusters that we just created to the global federation and ran a job through all the all of the production system.

A

I think uh we're quite excited about this, because it shows us what opportunities we can get from cloud native tools to streamline our processes and manages worldwide distributed uh quite heterogeneous resources, uh that is, uh that are managed by many different institutions, and so uh before going to q a we just wanted to acknowledge uh a few uh people that actually did the hard work and put together all of these building blocks that we used in this demo and now we'll go to the q a.

A

Okay, hello, can you hear us on the q a so we've uh received one question which is from daniel uh who's asking if anybody can run the helm chart that we used in the demo, so all of the helm, chart and code that we used in the demo is open source, but obviously so in the demo. We deployed this to production in order to connect to this worldwide federation. uh We needed to um have secrets in order to connect to this.

A

So it's not like everybody, can uh kind of connect to this um without having any proper authentication, but the code that we're developing to construct this federation is open source.

B

Okay, so I'll ask for the other one. The reach is asking how many sites they participate in this. At the moment, we have a free universities, plus we have started to use google and amazon.

B

Three universities, plus cern, of course cern- is a main player here.

A

Yeah, maybe to just add to this so um so in total, the worldwide lcd computing grid is many more sites, but um so we have a handful that are currently deployed in kubernetes and we're hoping to expand as we gain more experience with using kubernetes in this academic context.

A

Hopefully, we'll have a larger fraction of these sites run on kubernetes.

B

A

There's another question: so marcus is asking: do you have specialized nodes, for example, gpu nodes for gpu-related workloads? um So the answer is uh yes, so this is something that we've, um especially with the rise of machine learning that we're using in our science, are very interested in, and so there are various institutions that have gpu resources, uh among others, uh the manchester site that alessandra is working on, and so here uh you know, the container paradigm is really fitting.

A

Well, because the software that is used for these machine learning workloads is highly dynamic, and so we are relying, for example, on base images for tensorflow and pytorch, or these popular machine learning tools, and so we are also experimenting with integrating those with kubernetes and, uh for example, the advanced scheduling that kubernetes uh kind of allows. Us will be helpful here, but so far we've integrated gpus with the standard batch systems and connected them to the federation. But we use containers, but it's not yet. On.

B

Kubernetes uh there is a question number five that kevian is asking. If we have solutions for my memory, topology, eternal jn80, uh I knew my.

B

Screen creating some.

B

Peculiar solutions, because you have to in the resources.

A

Yeah, the next question is from uh michael. um So how are the kubernetes clusters interconnected, or is it more a loose coupling and can those clusters limit the resources available to the g? And so the answer is yeah. It's a quite uh loose coupling. So the nice thing about the workloads that we're mostly interested in is that uh they're, embarrassingly parallel.

A

So every collision uh that we're analyzing is independent of each other, and so we can uh parallel uh embarrassingly parallel uh distributors across the world, and so we don't require a very close coupling between the clusters. And so this um kind of coordinating service, which we call harvester, is the one that uh has all the uh q config files and can coordinate, and uh you know add clusters and remove clusters again. And so it's a quite loose coupling. We are, um and also the cernit team is experimenting, a lot with um federated kubernetes solutions.

A

So we've been looking to add more rarely or we're, also developing our own federation solutions and we've tried cube, fed, v1 and v2, and but a lot of that stuff is not really required for the workload that we need for our science and so this type of loose coupling has been quite useful and it's very easy to limit or completely remove a cluster again from this federation by the management infrastructure that we have.

B

So there is another question and there is asking if why we run docker images within a pod instead of running directly the workload.

B

The reason is that the system that we have in place at the moment is quite complicated, uh so we run what I call the pilot, which is this cytoscripts, and the pilot calls a singularity container rather than a docker container, and this is working across all 150 sites and the easiest way for the moment was to simply uh adapt what we have and run it in the pod.

B

Like a it, was a a a virtual machine or a bare metal worker node in the future. We hope to split these and to even split what we call a job and the workload in a smaller pods and use kubernetes more natively.

A

Yeah one technical requirement is that we need to also stage in data and stage out data, and so you see this in other operators like tekton task, that uh you know you need to be able to run the sequence of pods and which goes a little bit beyond of what initialization containers and kubernetes provide you. But before wrapping up, I did want to uh answer a person who is asking whether or not the difference in cpus between different clusters uh makes a difference.

A

So in the demo we were able to submit a workload and schedule it uh to a specific target cluster. So if this makes a difference in our workload, um our brokering will take care of it, but mostly we don't care and if it takes a little bit longer, just because it landed on a cpu that is not as powerful.

A

We usually just deal with in a weight and but take the advantage of this global federation. Instead of focusing on the high performing cpus, which might be uh overcrowded um okay, so I think we're out of time for the q a um both alessandra, and I will be on the slack for the next couple of minutes and are happy to answer any additional questions and okay. Thank you very much.