National Energy Research Scientific Computing Center (NERSC) NUG 2014: A Celebration of 40 Years of Science and Technology, 9 Mar 2015

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: The LUX Experiment and NERSC

Description

The Large Underground Xenon (LUX) experiment at NERSC, Carlos Hernandez Faham, Berkeley Lab

A

Our next three talks are about some of the exciting things are happening with data and computation on data and simulation in support of data, so I'm very happy to welcome our first speaker, Carlos mayhem, Carlos joined the Berkeley Lab is in Owen Chamberlain postdoctoral fellow last September he's working on the large underground, xenon, dark matter, experiment or Lux, quite heard of which recently became the world's most sensitive detector of weakly interacting massive particles or wimps.

A

After an 85 day run Carlos is working on using nurse resources to improve Lux performance related to computation, storage and analysis, in particular, he's interested in using a massively parallel database for interactive Lux data analysis and visualization, so Carlos. Thank you. Thank.

B

You return alright, so I, so I work for Lux Lux, the large underground scene on detector. It's a relative newcomer here at risk, so I'm gonna tell you about what we're going to do, not necessarily what we have done and there's some exciting prospects for us here at risk.

B

So I just want to tell you about something that happened in 1933. It's it's! Not this, but I just want to put it in context 1933. This is the mallet. There's a que there that I'm a smile look machine.

B

It can solve ten simultaneous differential equations, which was it's very nice and, of course, now everyone 2014 I, don't want to make the Mallik machine self-conscious in front of an.

C

B

So, let's just say that you can also do that.

B

All right, so so now that you have that framework 1933, this is this is 1933. This is fixed Vicki, an astronomer. He sparked a revolution in in thinking. It was a very long revolution took a long time for people to really take his ideas seriously. To summarize that here's, a quote by steven lara.

A

B

Researchers, talk about neutron, stars, dark matter and gravitational lenses. They'll start with the same way. Vicki noticed this problem in the 1930s. Like then, nobody that was the story of his life and.

C

B

Didn't help that that was just his favorite phrase, so he was a remarkable man. He coined a term supernova, he discovered a quite a number of them and he also dabbled in in Jet Propulsion I. Guess you did that back. Then he had our 50 patents, huge I believe the first one to propose using supernova standard candles in many other things- and most importantly, this is the point of this talk- is that he found a problem between a discrepancy between the visible mass that you can see in a galaxy and it's kinetic mass.

B

What the equations of motion tell you about how much matter is in there he called his dark matter and since nineteen thirty three, this has been present in pretty much everywhere. You look in the universe, so that was it then, and here's a now. We have a wealth of evidence, I'm just going to go through them. I wish I could explain each one right now, because it's beautiful physics, spanning many orders of magnitude to novel prices here, see George moved here. That's one of them self promoter, others. Some excitement in bao recently as well.

B

There's a wealth of information and they all point to the same picture is that we live in this kind of universe. It's not shaped like a pie, but that if those are the ingredients, this is us, unfortunately, every every. So often we get a new picture of the universe and we just get smaller and smaller in this picture. So this is us. This is regular matter all the particles in the standard model of physics that we know of are here, and this is matter that we we don't know what it is. We know it's there.

B

We call this dark matter and this is dark energy. This is something that we don't even know how to begin thinking about it. There may be an exaggeration, but this really is an open field right now, so.

C

Talking about dark matter, this.

B

Part of the high- we really don't know what it is, so we want to try to get more puzzle pieces in there, so you can try to see it in the sky. What's called indirect detection. If you can easily and easy lates or does some sort of interaction that it gives you buy products, you can possibly see those we telescopes there's a direct detection which is which is lux.

B

We have a big something some sort of detector here on earth and as the Dark Matter passes through us, we can potentially detect it, and then there is a production at the LHC. You could potentially make this new particle and see whether it fits the cosmological parameters needed for for.

C

B

Of these puzzle pieces, so here's a picture, sixty-nine percent dark energy. Let's not talk about that dark.

A

Matter: twenty-six.

B

Percent and 22 significant if a'kin figures, that's four point. Eight percent- and this is some sort of concordance model where many different I'm not gonna, explain this too much. But there is a concordance between different measurements that that tell you that this is what we live in and in Dark Matter, total matter, sorry and an energy dark.

C

B

Alright, so how do you detect dark matter? I wish I could tell you what this is. This is how we're attempting to detected this is this is our galaxy artist rendition, and this is all we can see luminous matter and you can detect the presence of a dark matter. Halo, spherical, roughly spherical around the Milky Way and the solar system is, is moving here and the earth is moving within the solar system. So the motion of the earth relative to the dark matter is it's not the same?

B

Okay, so you should feel this wind of dark matter. If we could feel it, there will be millions of these particles coming through you just like the neutrinos. You just don't feel them there so they're, so weakly interacting, so we're assuming that dark matter particles are able to interact with our detector. Just like neutrinos from lisa is going to tell you all about new treatments in the next talking.

B

How how we actually have this technology that the tech neutrinos and we put a detector and of all of these flux, going through it once in a while one of it is going to interact with a weak force. We don't know whether interacts with the weak force, there's a lot of theories that support new particles that interact with a weak force to kind of match. This need that we have in dark matter for new particles. So, let's just say it interact with a weak force. Let's look for new particles that are scattering now.

B

Lux is not the first to do this. In fact, there is a very long history of people trying to do this. This is the brief history of Dark Matter, directly affection here, plotting the year starting in the late 80s, and here this is the probability of interaction. Okay,.

C

B

Called a cross-section and we're assuming some sort of mass, because if you have heavier particles which changes the cross-section these terrible terrible unit is our we've inherited since then of 10 to the minus 40 41 42 centimeters squared for cross section, but just bear with it units. So we started up here, homesick and Orville funny now we're back at home. Stick now for Lux, and you see this technology going through the germanium detectors.

B

Then people started thinking about cry of detectors, so cooling down this germanium gave you much more sensitivity, and here is another trend, so you can see that the rate of improvement per year is actually getting better and better I, don't know, there's a Moore's law for these with there should be doing this and then there's the novel liquids. These are the right-hand side of the periodic table, the noble gases that we liquefy and they turn out to be excellent. Particle detectors for this kind of work.

B

So here's the noble revolution there's another King candy in the rate as a function of time and here's where we are right now we have the best limit, so we're the best at seeing nothing, unfortunately so dubious dubious title, but if there is something, on the other hand, will be the first to see it.

B

Okay and just to show you what the scale of the locks result is. I'm gonna pick on the little guy here, sorry oroville, um here's lux one minute after one meaning of data taken we have that limit sensitivity. Here's five minutes, here's one hour, 18 hours. This is CDMS to most of some of you may infer that it was a very successful program. Still is running at university. Here, Berkeley burn are seveal a he did a lot of his work here.

B

People at LVL and other universities use is yes, then here's locks for days this is seen on hundred and here's us. Sorry here is an abuse. Here's us in 85 days and we're gonna ramping up we're beginning this year, a new run, hopefully April, but I don't want to give any dates. A one-year run. They'll put us here and we'll see what we can find, so you may have heard a dog matter. Direct action is also known as underground science.

B

There's there's a reason for that: it's not because what we call the lure of the underground it's a New, Yorker image it it's not really because we want to be underground is because we need to be underground.

B

There is the shower of cosmic rays going through you every second, you extend your hand right now and if you could feel these particles going through you, you feel about one per second in your hand- and these are extremely energetic particles- I mean the particles that the LHC product produces pale in comparison with these energies that we're talking about, and when you go on the ground, you use the earth as a shield.

B

So you have this massive mile-long shield of rock and when you go on the ground and you extend your hand again, you're going to get one per month, you're going to wait for a month or two or three months until something hits your hand, so that makes the these low energy rare event search as possible. If we use our detector here on the surface which actually we did it for calibrations, it is the shower of high-energy particles all over the place.

B

We cannot see anything about that background, we're very lucky to have a fantastic facility here in the US. It's in state of South Dakota right here and here's an aerial view from a Cessna here is the local color, the headframe, the shaft. That takes you a mile down too well much much further down than if the frame can can see. You can show you. This is really one mile of cable that just unroll. I wish I had a video to show you what is quite quite remarkable.

B

They just take you with one cable whole way down and then up again and here the luxe luxe use this facility. This was a duplicate of the underground layout that we use for commissioning our detector. There's.

C

B

The show Deadwood and you you know exactly what I'm talking about gonna saying cuz. This is Deadwood South Dakota that that's right there you should see that show is fantastic by the way and.

C

B

Here's a Davis cavern, it was literally a hole in the ground in 2011, late 2011. This is called eager, grad students. This is to be written, my mind, one of them, and this is a year later, exactly a year later, these go bc, grad students. This is a laboratory space. All finished up polished. The detector is inside this water tank under our feet, all instrumentation in and ready to take data.

B

Here's the lower level, so our feet were here in the previous photo and and here's a water tank that itself it's a shield for the detector. We want to keep neutrons out because neutrons look like dark matter particles in order detective and here's.

C

A nice video of the.

B

Harvard-Smithsonian Center for Astrophysics annenberg mediately for us shows you the scale of the detector. Unfortunately, we didn't crow mater our cry of step, but it is roughly that those dimensions and that's.

C

B

That that we took right before we fill it with water, so I'm Assad I'm a sucker for animation. Sir excuse my cheese here. Alright, as you can see, we have photomultiplier tubes in there as well.

B

These are photo detectors, so whenever a cosmic ray particle goes through, it will give off light of share Inc, apply the same principle detection as ice cube and you can detect those and veto the system. So you don't take data during that time, because you can produce neutrons that mimic dark matter, particles, okay, so here's how the detector works- and you know I'll tell you more about our data and computation. That's yes, minus 100! All that the water is room. Temperature and xenon is minus 100 Celsius.

B

That's a silly question: I think it's a few degrees above regular that the rock temperature is much hotter, but they have a huge ventilation system that keeps it just a little about room temperature I forget what the rock temperature is, but you can you can notice. It is different, warm as you keep going down down. It keeps water warmer.

B

So here's how the sec detector works. You have the you know you can almost Hoggett is about this Vegas. It's a big vat of liquid xenon, so a meter in both directions and you liquefy the scene on here's, a liquid level. You put photomultiplier tubes that there are photo detectors, they're, going to take a single particle of light and they're very sensitive devices, so you put 61 at the top 61 at the bottom and then you put an electric field with grids.

B

Here's a great here's, another great there's, an electric field that I'm showing here so any particle that comes in will interact with the scene on and the scene was going to do two things it's going to give you light it's going to give you charge you going to take the light immediately speed of light, and here it is.

B

We call that the s1 signal and then the electron signal it's going to propagate in the electric field at a very or relatively fast speed, and then it's going to jump from the liquid into the gas and in the gas is going to glow. Like one of those neon signs, it's called electro, luminescence, okay, and that's going to give you a much bigger signal, so a pair of an s2 which is the electron signal and then, as one is one event and the delta T between them, the time equals Z. So more time is deeper.

B

Less time is higher, ok and then, with the heat pattern, the the neon light. If I may, the electroluminescent light is very close to the topper rate. So it's going to give you a hotspot that gives you X Y, so we have X Y Z for every particle in the detector is very powerful because most of our backgrounds are going to stop at a surface. The skin of the scene on that and dark matter is not going to care, is just going to go in and interact if at all in our detector.

B

So we can pick a virtual volume inside what we're looking for a dark matter and most of our backgrounds are going to be indeed in the outside. So here's the locks collaboration. This was just before we release her first results. Last year, vicki was around. You probably say that to us, so here's just highlighting some people.

B

This one is hard to see. I know if Gil is around, so this is Bob Jacobs and Kimmy. Let's go Gil Gil crease in the shadows. We came under myself. These are better pictures of us and also.

C

We have me a cookie.

B

Monster she's about to graduate all right and we had our first results in October thirtieth 2013. We had quite a bit of press coverage which was excellent for a detector future.

B

We made the top of the Google News science, which is aggregated, which means there's a lot of references to it, which is good Forbes and the economist I'm not putting the new york times. If you read it, you know why, and- and this was a good time for the experiment- and this is a result- a lot of work put into drawing a line in this parameter space. But this is quite powerful. Here's the mass of the dark matter in GV, here's, ten hundred thousand- we don't know what it is.

B

You can see that we have no idea what we're doing, because you spend so many orders of magnitude, but here's a probability of interaction again the cross-section and you can see this was the state-of-the-art in 2012 2011 2010 so on, and we just surpassed it by a good factor. Factor of three factor of two down here: low mass is there were some contour plots strong that you can see. These are signal people saying we found our matter. Some of them don't say it very loudly, but that is that is the implicit assumption.

B

When somebody draws a parameter space blog saying there is something here we rule those out quite robustly. We've been saying that for a while that those don't seem like signal in our new run, we're probably going to go even even deeper, but we're very happy with this result unhappy that we didn't find out matter, but at least we can say something about this. All of these claims that have been dubious at best. I think, okay, so let's data this is this is a part of Tyson asked, so we.

C

Have we have a lot of Lux.

B

Data- this is what the data looks like again. This is the s-1 and the s2 separated in channel number going from 1 through 122. We have 122 photomultiplier tubes channels and the data is very sparse, but there's also a lot of data I'm, not doing justice here to the amount of data that we've been collecting.

B

We collect every pulse, we don't trigger. We just collect every pulse that our detector gives out, and you had that gives you a lot of information. Then offline. We chunk it into events like this one, where there's an it's one and under stew, and then we do analysis with that. So we have this, keep everything find anything model we we want to have high efficiency with. One of me is dark matter.

B

If it turns out that our theories have this great idea and we could only go back and take that data again with a different trigger. So so we have an offline trigger. We don't throw any day that way, none whatsoever. So in 85 days we acquire 50 terabytes of waveforms. This is compressed data when you blow it out there. So there's a big factor there and and reduce quantities which is derived values based on this. What forms of 25 petabytes this you know compared to some of the bigger experiments.

B

This may not seem like a lot, but we do keep everything we want to find anything. So it's a rare event, search we're looking for wind dark matter, but we're also looking for a lot of different things. There's action dark matter. There is inelastic, dark matter. Double beta decay in the bigger detectors, coherent, Audrina scattering in our next phase and many many detector calibrations.

B

So we have data processing chain that we developed it takes this, this time, series data and spatial data and gives you you know the energy reconstruction for each event, the number of electrons and photons position height with symmetry, a lot of digital filtering, that we do the quality of the poles and we call those reduce quantities or our cues. So this is what we do data analysis with, but at the other day you always want to go back to this and make sure that the event made sense. You just want to look at the event.

B

Now I just want to say that we also do monte carlo simulation output. Excuse me that also gives you this sort of data. We can model the data from the detector directly. It gives you waveforms and we run those through the data processing chain as well, and just to give you an idea how rare the event searches. This is all the triggers again. We don't throw any of the non triggers away, but after the data is being collected.

B

This is these are the things that passed some triggering scheme for the electron signal, and we collected in 85 days, 83 84 million, when you start doing some cuts for a particular data analysis such as the wimp dark matter. You pick some energy ranges so that really cuts down. You know 20,000, and then you pick the fiducial volume, the inner region that I spoke to you about in the detector and that really cuts down 260 events and out of 160.

B

There is one that is in our signal region compatible with a with a wimp recoil, but is really a background, but fluctuated down we're not claiming that thats dark matter. So 84 million won that that's the scale that we're trying to the world with and.

C

Every time you.

B

Want to do a different energy regime for a different event, apology for different type of dark matter that you're looking for these changes and gives you may be another one.

B

So here are the resources that we're starting to use we're starting to use carver for our data processing. We got an award for data-intensive pilot program, those those great for us. We can get started with this award. We're going to do all the data processing in simulations here in carver and pursue also independently PD SF node acquisition.

B

For the same reason, then, on storage, we're putting all of our data, we have a full mirror, hear it at lvl, a nurse all the process, waveforms reduce quantities on disk and hbss tape, all of the raw waveforms and then we're starting to use Global's Globus online for transfers.

B

Okay, and just to tell you about two exciting projects that we're pushing ahead here at risk. We want to evaluate potentially use idvd for data analysis. We think this is really going to take the locks data analysis to a whole new level if it works the way we wanted to and and then an online event viewer.

B

So so here is the file system. Analysis model that we're using right. Now that pretty much sums it up for you that that sums it up alright file system based analysis is very difficult for, for data managing purposes to find anything, it's just very difficult. It's slow because you have to go. You know for every file and the analysis. Results are not easily replicated the code, listen song POSIX computer, and he did the analysis many years ago. Can you redo? It well have to go through all the data etc.

B

So we we don't believe that this is a model, that's going to that's going to be best suited for our needs. I think this is sometimes I feel like this. If you read this this XKCD, you know this meadow rectangle full of little lights. Yep. It's been most of my life pressing buttons to make the pattern of light. Change, however, want okay sounds good, but the way the pattern of light is all wrong. Oh god repressing more buttons, so it just doesn't feel like the file system.

B

Bonnell is the way to go so we're trying to decide to be it's a massive database. It has native, are a support, so this is what we what we use for all of these reduced quantities. It's a parallel data access share, nothing! So here's our data, here's chunks of data that you're going to assign two different notes and when you want to add something it just gives it back to you in parallel, and you can do regret operations, you can do many.

B

Many operations are very very useful, so we want to evaluate that right now, we're dumping all of the data into sci baby in a start-up account and see how how well that works for us and also, if you want to look at an event. Let's say you have all these are cues and you have them plotted. You have all these little dots in this on the screen and you want to say, I want to see that event right there. That little thought show me the waveform. Well, you have to look at the event.

B

Number then go to the server download the file that corresponds to that event. Number put on your computer load it and look at it. Ok show me that.

C

B

C

And that other event.

B

So on so you'll be great if we had a browser, this has been done before, of course, but this will be good news for us.

C

B

Go to a browser you go to one of these nurse portals. Here's the event list show them to.

D

B

I, don't have to get the data I, don't know where it is I, don't have to know where it is I just know the event numbers and I want to plot them or I mean a data analysis session right here in inside EV. Just show them to me just plotter right there, while I'm doing analysis, so these two pathways were for event. Viewing are going to be very powerful and reduce the amount of time we spend trying to get the visualization, ok and just want to tell you about the next stage.

B

I have a handful of minutes left so luxe and Zeppelin Zeppelin is the UK a UK collaboration and Russian and Portugal a couple of other institutions in Europe.

B

They have a Dark Matter detector, just like ours, we're merging the collaborations to make what's called the LZ detector, and this is lux 370 kilograms of scene on and just to give you a sense of scale. This is LZ the big brother of Lux and Zeppelin. It's going to be seven tons. This is metric tons level in with liquid xenon. You know three times four times as many channels and we can have a lot more data surface.

C

B

More background and we really want Lux to set the stage for LZ right.

C

B

In the planning stages, where LZ is the right time to start looking ahead into what can be done with, you know, parallel massive arrays in a database visualization, etc. So we want to pave the road for.

C

B

And here's the summary uh so luxe is running. We want to keep everything find anything we collect every pose and then do many different analysis to see to see different topologies we're starting a new Dark Matter search very soon. We expect over 200 terabytes of raw data simulation and with these quantities we're starting to use norsk. All of these resources are fantastic: Carver, pts, FG, pfsh, PSS and Global's fur for data and computation, and we're evaluating some of these massive parallel databases for data analysis. Thank you.

A

A

Ok well, this is this.

D

Is fantastic and.

A

I'll point out that I'm thrilled to find another Fritz Zwicky fan, because ever since I've had it my own machine, it was been called wiki already hugs me and say well, who the heck is that.

D

So you have those big neutrino detectors. You can't use that you can't combine the two experiments. What's the difference between the two, the neutrino.

B

High energy and Dark Matter very, very low energy, so we have a very low background detector at low energies for these kind of searches yeah, it's a different energy scale. Eventually we're going to be thwarted by neutrinos when we get very, very sensitive, we're going to start seeing the low-energy neutrino interactions. Well, that's! Maybe, ten years down the road.

C

Sorry time I like to you, know Trina's, when your ego.

B

C

You talking about like Gio neutrinos.

B

Coherent scattering coherent scattering from solar neutrinos on mospheira neutrinos yeah, so.

C

I was wondering you said: Lux was going to take data for another 300 days, so.

D

C

You give me an idea of sort of the time scales and then, after those three hundred days, does it keep going or is it well.

B

Be something I didn't mention. That's a very good question is that this water thing is the lux water tank, so that that that gives you an answer roughly, that whenever Lux goes out in Z goes in not in other words, Lux won't go out and feel LZ is ready. Yeah.

C

A

You thank you. Thank you.