Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2022, 1 Jun 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Jet Energy Corrections with GNN Regression using Kubeflow @ CERN - Daniel Holmberg & Dejan Golubovic

Description

Don’t miss out! Join us at our upcoming hybrid event: KubeCon + CloudNativeCon North America 2022 from October 24-28 in Detroit (and online!). Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Jet Energy Corrections with GNN Regression using Kubeflow at CERN - Daniel Holmberg & Dejan Golubovic, CERN

The Large Hadron Collider is the world’s largest particle accelerator measuring 27 km in circumference. It accelerates beams of particles in opposite directions almost to the speed of light before making them collide. The particles emerging from the collisions are then measured in large detectors such as the Compact Muon Solenoid. An especially important object of study are so-called jets composed of multiple particles shooting out in the same direction from the collision point. Data-driven methods are used to correct the energy values for these jets, and what we’ll present here is the utilization of Kubeflow to enable state-of-the-art graph neural network based corrections. Kubeflow’s pipeline component allows us to define our machine learning workflow in a well-structured and reproducible manner, and its built-in training operators are used to scale up the training with ease. This work is expected to pave the way for future adoption of Kubeflow among the physics community at CERN.

A

Hello, everyone thank you for being here to listen to our presentation. My name is dan. I'm here with my colleague daniel. We both work at cern, and today we are going to share a use case of machine learning application in high energy physics, and you know to show how we can scale such workloads with cube floor.

A

A bit of introduction about cern cern is a particle physics, research organization located in switzerland. Its mission to is to expand the knowledge of the universe and how subatomic particles interact with each other, what their properties are. uh It's operating the largest particle physics lab in the world and it is an international collaboration of over 17 000, part-time and full-time employees from over 110 nationalities.

A

Cern is studying the subatomic particles and it is doing that through with the help of large hadron collider. A large cardron collider is the largest particle accelerator in the world and one of the most massive machines ever built. It is a 27 kilometer ring of superconducting magnets that is located 100 meters underground at the swiss french border.

A

uh It works by accelerating the beams of protons from opposite directions until they meet the until they approach the speed of light, and then these beams collide at four collision points called uh detectors or experiments, and this is where all the data is uh gathered and analyzed uh here we can see our globe.

A

The the exhibition building at cern here is an illustration of the lhc magnets, and here we can see the cms experiment and we can see the magnitude of the experiment now a little bit about machine learning and cube flow at sir um there. uh The lhc produces a lot of data. There are 40 million collisions happening every second in the lhc and even after filtering that translates to around 90 petabytes of data produced by all experiments per year. So there is a potential for machine learning in different stages.

A

We can apply machine learning in data acquisition or later in the offline analysis, and we can utilize machine learnings to machine learning algorithms to find the patterns of interactions between particles. uh In this slide, I will cover a couple of examples of machine learning. uh Applications at cern and later daniel will cover one specific use case in more detail in data acquisition. We have uh at the cms experiment. We have uh so-called trigger mechanisms. These mechanisms are extremely important because they select, uh which of which collisions, are interesting to store and process further.

A

It is impossible to store all 40 million collisions per second. The computing infrastructure just doesn't support that, so we have to make a selection which collisions are interesting to store and further process. So the trigger mechanisms do that and the different machine learning algorithms are deployed in the trigger mechanism such as boosted decision trees.

A

But in the recent years there is an ongoing research uh effort to apply deep learning methods as well to the trigger mechanisms, uh and this there are different challenges: there: first to train the high performance models and, second, to run these models for inference in in the fpgas.

A

So that's one example: uh another one is in particle tracking and reconstruction, so with the lhc we're observing short-lived particles that otherwise cannot be seen in nature, and they are analyzed by monitoring energy depositions in the calorimeters.

A

So, with a machine learning, algorithms, we are doing particle tracking and we are reconstructing the the events of a single collision, uh then also there's a application of machine learning in simulations as a faster alternative to monte carlo simulations and in addition to physics, use cases, we have a lot of data coming from the facilities themselves, such as the lhc, the experiments and even our cloud computing infrastructure.

A

So we use machine learning, algorithms, there to detect anomalies and to proactively monitor our systems to keep them in peak performance.

A

Now there are different challenges of machine learning developments at cern, one of them being that there are multiple groups working on different projects and mostly they work with local infrastructure. That means couple of local gpus, very often one or two, and that requires researchers to set up everything from scratch. That means they have to install gpus, to install nvidia drivers, cuda python environments and then to share that across all users in the group and then, if they want to scale their workloads to more gpus, they have to move to another platform.

A

So that's a lot of overhead for researchers and that that was the main motivation to deploy a centralized machine learning platform using tube flow. uh With this platform we're trying to reduce this maintenance work, we are providing access to gpus on demand and also scaling capabilities both on our local instance and to the public cloud.

A

So our instance is in production since 2021. It is an on premise: cluster, using openstack.

A

We have an integration with many different cern services such as hardware registry, a single sign-on, then manila csi or network file system called eos, and we do our githubs with argo cds so that we can easily deploy our instances to different uh public cloud providers.

A

In our previous talks, we were discussing in more technical detail, our admin workflows and how we manage our instance and uh the the technical details of it and, if you're interested in that, you can have a look at our previous talks, and today we will cover a specific use case from a user perspective and to demonstrate how cube flow is used in high energy physics and daniel will take it from here.

A

B

You deja so I'll present a use case for you for high energy physics, where we run our stuff on kubeflow. What I've studied is called jet energy corrections. We get so-called jets coming out of the detector, so initially protons collide at the large hadron collider.

B

They release a color charge patterns such as quarks and gluons, and they cannot exist freely in nature due to a phenomenon called color confinement, so instead they dehydrate and create color, neutral particles which we can detect, and these are then clustered into a a single object called a jet.

B

The energy of this jet is pretty important for further physics analysis. However, the detector is far from perfect, so we need to calibrate this energy and what I've done is use. Machine learning for this calibration previously in production is non-machine learning methods. So we have a baseline that we can compare to and see if machine learning unit can help with correcting energy.

B

So this requires a bit of a special kind of machine learning. um These jets are or they create or consist of unordered sets of numbers. Basically, so you can't really apply a basic feed forward neural network. Instead, we can borrow from computer vision because you can create a so-called particle cloud which is analogous to a point cloudy computer vision.

B

You use the coordinates of the detector for the particles and that's why you can create a graph of these particles and use, for example, graph neural network.

B

So here is an illustration of what I mean: you have a bunch of particles. This is a set of particles which, uh with uh some uh unknown length, it can be how like anywhere from like 0 to 64 particles in a jet, basically- and you want to apply some machine learning method to these particles or to these feature, vectors and map that towards a target, in this case, it's an energy value for a whole jet. So it's a single single value target and furthermore, this also has to obey order invariants.

B

So the order of the particles that the machine learning model takes as input it should produce the same map to the same target anyway.

B

What I've used is to to models from that are currently used in in computer vision as well.

B

One of them is deep sets and it has been adopted in a high energy physics community as particle flow network. It takes as input these particles, you see them at the top over here, the charge, neutral particles and then the secondary vertices. These are all constituents of the jets um and what you do is you apply a feed forward, neural network to every member of the set I.e every particle vector, and then you aggregate them using some sort of order.

B

Invariant pooling such as summing or max or max pooling, or mean pooling, or something like this. Then you can feed that output, which is now like, represent the representative of the whole jet. You can map that towards the target energy.

B

If you want to go even more or use a even more complex network, you then create a graph. So here you see the k n block over here. This is this, creates like a you represent the particles in the detector coordinates, and then you take the k, nearest neighbors of every particle connect them with an edge, and then you have a graph, and you can then use that to get some sort of locality. Information of the particles which this provides more information and and will potentially prove improve the results over the purely set-based method.

B

So here's where kubeflow comes in now we have our two models. Now we want to run them on. On our internal resources at cern- and so we in this case we created a pipeline uh where we begin by running an uh an automl experiment with the tune, hyper parameters for the neural network.

B

um We've then export this or the best model of them all. We export this story on s3 and serve it using k-serve.

B

The training set was 10 gigabytes. We started on s3 as well, and the target here is to minimize the some loss, and then we tune the hyper parameters using random search. It's a pretty one of the more basic hyper-permitted tuning algorithms, but also one of the best, apparently just picking random features and doing that over again or random input parameters and doing that will eventually find you the optimal model so yeah.

B

Regarding scalability as well using the pi touchdown operator, we can use multiple nodes and also within our or in the pythage code. We also can use multiple cpus to read data in parallel and train on multiple gpus in parallel, and you can also run many categories in parallel, so this is really really scalable. In that sense, in many different ways, we monitor the training uh while it happens using tensorboard, and this is an example of that.

B

So for inference, we export the best python model to onex, which is like a framework for multiple different, deep learning frameworks where you can export so to one universal format, and then we can serve this using nvidia, triton and and then, when served, we can use the python, client or whatever client we want to to request inference predictions and statistics, and we also did some analysis of the different like how fast it is to to request inferences based on the batch size that you request at once, and this way you you can speed up the influence a bit.

B

So, let's see if a live demo goes well I'll, try my best. um So what I'll show you is the is our kubeflow interface. Here's the the pipeline that we built that I showed earlier we can inspect is by the way the font size good enough. Yeah.

B

B

All right, um so here's the the initial component, the hyperpermanent tuning experiment. We use the cathed component to do this, and uh so you see it takes a few input parameters. We have some output, so among the inputs are, for example, the data training data and test data etc stored in s3.

B

We can specify the number of cpus gpus how many workers we want for the training. We also did some experimentation with the visualization, so here's you can, as you can see, you can use markdown or tabular data which you get in the ui of cube flow.

B

So this is our all pretty neat. These are the optimal parameter that the hyper parameter tuning found.

B

And furthermore, then we export this. So this is a separate component and it also has some some parameters.

B

You know, and then we finally serve this model. uh The third model is then in an onyx format, stored on s3, and then we go on to the to the experiment itself. So this is a user interface for for experiments, and this is what it looks like.

B

When the when it has finished so we can see all the trials and highlight them as we wish, and then we can sort of inspect and compare which uh parameters are good and which or how how different parameters affect the the loss that we want to minimize.

B

So this is all.

A

B

Convenient way of of getting an overview of what what works and what doesn't work.

B

All right, and and while this trains and after after swallowing, you can connect the tensorboard to the s3 bucket that the model logs to so here we can see the tensorboard.

B

You can have different types of metrics that you log um so, for example, let's see you know: oh this discovers it but anyways. um You can then follow along the lines and see how they measure against each other.

B

And then we finally get the model that we served. So we can see here that we, the predictor, is the triton triton one. We had to change the runtime a bit to fit our needs because yeah, the built-in one wasn't new enough. So that's a that's one of the issues that we came across um and then let's, for example, check the particle net one and we get here an overview of the model.

B

That's served, and now we have an inter internal url which I'll copy here, and we can then request inference or predictions from this model that they observed so I'll just go ahead and paste it and wish that I have internet connection and I get some predictions so first I go ahead, load some jets and with some with some features- and I in in- and I started writing client um to this uh specific url that I copied and then I I try to request some predictions. So here we see the input jet.

B

I plot it in detector coordinates and above above the jet. You see what the what particle net predicts for energy or transformative value and versus the true one. So it seems to be working, I think yeah, and you can also note that, for example, if the there's a big bubble, the predicted pt is also bigger. So the the this. This is a plot of the like the constituents, the particles that make up the jet and uh if the, if the marker is a bit bigger, then it has a higher transfer momentum.

B

So uh yeah that relates to the predicted value, and that was it for the demo.

B

B

B

In conclusion, yeah, so this is the physics result that we get. We just requested some specific predictions for one specific jet at the time here, but these are the final results that we got. We get a 10 improvement in the resolution I mean, in other words, how accurate the the energy predictions are, and we compare that to the baseline shown here in blue like a mark.

B

So this is the baseline, the blue markers and then we get a 10 improvement, so we take the ratio towards the baseline and it's a 10 improvement which is very neat uh and then on the on the other plot. Here we see a a comparison between jet flavors, so these are up down strange charm, bottom and quarks and then a glue one as well, and ideally the energy response should be similar for like, even though the jet flavor is a bit different. The response should not be so.

B

We also hear machine learning improves by a factor of three the the flavor dependence of the energy corrections against the baseline. Both models work very well particle net, which was the one that used graph. Machine learning might be a little bit better compared to the one that used purely set-based corrections.

B

A

Okay, thank you danielle for this nice demo. So here I would like to mention couple of challenges that we had uh while implementing this demo. um So the first one was um finding a correct version of triton server image to make inference servers the default one with kubeflow 1.4 didn't work, so we used the latest one and that yeah. Then then everything worked as expected.

A

uh Now we had to do some customizations with the tensorboard, so we had to customize tensorboard controller to automatically pick up the the secret for s3 and to mount to uh tensorboard deployment so that we don't have to do cube ctl edit.

A

So we did. We did it downstream um in a way that it fit this use case, but maybe this could be uh generalized in some way uh now uh some advice, maybe for new cube flow users.

A

uh If you're using katib, then the best option for metrics collector would be the file matrix collector because, um if you're using std out matrix collector- and you have a lot of logs- that are not always under your your control that may not work, because the metrics collector may pick up something that that is not actually a matrix uh and then yeah. Maybe catbui could offer some better visualizations for uh multiple metrics, but in principle those are all minor, minor things. We didn't have any major stoppers to make this demo work, which was quite nice.

A

So to conclude, if you go back to cern's mission, to expand our knowledge about the universe and do fundamental research, we see that machine learning has its place there and that it has its application and that it can help us- and we have seen on this example of gen jet energy regression, that we get the 10 improvement, which is great, and there are many other uh applications like this. uh So that's why we need.

A

uh We need to work on uh machine learning infrastructure to make sure that it can support all these use cases uh with kubeflow. We can facilitate the scalability of such workloads so to go beyond jupyter notebooks, to run pipelines, cutif jobs, uh to serve models, uh so it really allows a lot of flexibility there.

A

The mutual integration of components is, uh is great, so we in a sense that from pipelines we can submit cutting jobs, uh distributed training uh serving.

A

Model serving sorry so it it all works pretty nicely and we offer customizable and reproducible environments for our scientific users etc. So kubeflow has been very well accepted, and here I would like to thank qflo community for all the great work um and yeah. We hope um this was useful for for everyone. So thank you very much for your attention. If you have any questions, let us know.

B

B

So I wanted to ask.

A

No, no, we used the latest version of triton, so the official one. Yes, we didn't do any customizations there yeah.

A

To repeat the question yeah, the question was: which image did we use for a serving? Did we customize it or we use one of the official ones? So we use one of the official ones. The the latest triton server image.

B

Yeah, that seems to be it. Thank you.