National Energy Research Scientific Computing Center (NERSC) / NERSC 2021 Early Career HPC Achievement Awards Seminars

Raythena utilizes the Ray software (a high-performance distributed execution framework) to distribute the highly intensive ATLAS Geant4 simulation workflow across a few hundred HPC nodes. Geant4 simulation is the most computationally expensive step of the ATLAS Monte Carlo simulation chain and represents about 50% of the ATLAS computing budget. Conventionally, it is run on ‘grid’ sites and each simulation campaign takes a few months to simulate the desired quantity of proton-proton collision events. Raythena is a solution for running ATLAS Geant4 simulation efficiencly on HPCs and it could significantly reduce the duration of simulation campaigns in the future. The goal of Raythena is to process as many events as possible with a given CPU-hour allocation on an HPC as fast as possible. An effective mode of operation at NERSC’s Cori was found to be running 100-200 Cori KNL node jobs in the flex queue. Raythena is a central application that orchestrates the workload management across all nodes using the Ray API. On Cori KNL nodes, 132 Geant4 processes were spawned on each compute node, amounting to more than 25,000 Geant4 processes running in parallel. Raythena handles communication with the ATLAS central PanDA database where it retrieves the input events and feeds them to the Geant4 processes. The Raythena framework was found to scale very well up to 100 to 200 nodes on Cori KNL with virtually no delay between the consecutive processed events.

When an electron emitting surface is in contact with a collisional plasma, a unique regime of plasma sheath may form, an inverse sheath. The inverse sheath regime most notably differs from the classical Debye sheath by having a floating potential above that of the plasma potential. This leads to a restructuring of the plasma flows. The ubiquity of emitting surfaces in laboratory plasmas means there are a number of applications of the inverse sheath such as extending hot cathode lifetimes, cooling of the local plasma, and modifications to emissive probe theory. Due to the collisional nature of the sheath and the trapped population of ions, kinetic simulations are required to answer outstanding questions which remain about the sheath’s formation, and transport properties. To address these questions, we have developed 1D-1V and 2D-2V kinetic continuum codes which include collisions and allows us to explore the inverse sheath in relevant configurations. In this talk I will introduce the fundamentals of the inverse sheath, the codes we have developed to solve these problems, and the applications of the inverse sheath.

Many astrophysical observations support the existence of a dark matter component in our universe. However, after a few decades of active research, the nature of dark matter remains elusive. In this context, the LZ experiment aims to detect dark matter using a multi-tonne detector filled with liquid xenon and located at the SURF underground laboratory. We expect a handful of signal events over a significant background during the five years of operation in such a detector. Observing deviation from the background model would sign a dark matter detection. Hence, the LZ collaboration is engaged in a great effort of developing a robust signal and background model using Monte-Carlo simulation. Simulating such a detector requires the usage of any computing available. Therefore, we developed a framework to take advantage of all the HTC (PDFS) and HPC (EDISON and CORI) resources available at NERSC for the second and third mock data challenges. Using a centralized job submission system, containerization technologies, and software distribution service (CVMFS) made it possible.

The Antarctic Ice Sheet is losing mass at an accelerating pace, mass which is entering the ocean and increasing sea levels across the globe. About half of this mass loss is due to ice flowing out over the ocean, where contact with the warm water melts it directly. The other half of this mass is lost as broken ice - icebergs that calve into the ocean, which float away and then melt. Though both of these processes have been identified as possible sources of dynamic instabilities in ice sheet evolution - the Marine Ice Sheet and Marine Ice Cliff Instabilities - most large scale numerical models have yet to reliably reproduce changes of observed calving front positions, resulting in significant uncertainties in projections of sea level into the next century. Furthermore, when ice enters the oceans, the earth’s viscoelastic mantle responds to the redistribution of mass, affecting the drivers of ice flow and the progression of these instabilities. I will discuss the challenges associated with incorporating solid-earth feedback and representations of mechanical failure of ice into numerical models of continental ice sheets, some solutions to them, and the effect these processes have on projections of sea levels.

The Vera Rubin Observatory LSST is going to provide the astrophysics community with an unprecedented amount of survey data with which to contain the evolution of the universe through time. In order to leverage this dataset, we will ultimately require extensive simulations in order to validate scientific pipelines ahead of the survey ever seeing light. The LSST Dark Energy Science Collaboration (DESC) Second Data Challenge (DC2) represents the largest simulated sky survey of its complexity. Generating such a simulation required managing a complicated and rapidly changing workflow across multiple compute resources. We demonstrate how we utilize containerization and the Parsl parallel scripting library in order to create a portable and scalable workflow to meet the challenges of this computational task. With this workflow we were able to generate a simulated survey volume covering 300 square degrees and five years of image depth, utilizing 100M hours of compute and up to 2000 Cori KNL nodes at a time. We discuss possible improvements that could be made to the workflow for future survey simulation, both from the standpoint of utilizing the increasingly common workflow nodes at high performance computing (HPC) centers and that of how the underlying image simulation code may be altered to benefit more from computing at these scales.

In this seminar series, the recipients of NERSC Early Career Awards for using HPC at NERSC, describe their research and how HPC was an important aspect of it.