National Energy Research Scientific Computing Center (NERSC) Data Seminars Series, 16 Apr 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: 2021-04-16 - Zachary Ulissi, Larry Zitnick - The Open Catalyst 2020 Dataset & Community Challenges

Description

NERSC Data Seminars Series: https://github.com/NERSC/data-seminars

Title: The Open Catalyst 2020 (OC20) Dataset and Community Challenges

Speaker(s):
Zachary Ulissi, Assistant Professor of Chemical Engineering at Carnegie Mellon University
Larry Zitnick, Research Scientist at Facebook AI Research

Abstract:
The Open Catalyst Project aims to develop new ML methods and models to accelerate the catalyst simulation process for renewable energy technologies and improve our ability to predict activity/selectivity across catalyst composition. To achieve that in the short term we need participation from the ML community in solving key challenges in catalysis. One path to interaction is the development of grand challenge datasets that are representative of common challenges in catalysis, large enough to excite the ML community, and large enough to take advantage of and encourage advances in deep learning models. Similar datasets have had a large impact in small molecule drug discovery, organic photovoltaics, and inorganic crystal structure prediction. We present the first open dataset from this effort on thermochemical intermediates across stable multi-metallic and p-block doped surfaces. This dataset includes full-accuracy DFT calculations across 53 elements and their binary/ternary materials, various low-index facets. Adsorbates span 56 common reaction intermediates with relevance to carbon, oxygen, and nitrogen thermal and electrochemical reactions. Off-equilibrium structures are also generated and included to aid in machine learning force field design and fitting. Collectively, this dataset represents the largest systematic dataset that bridges organic and inorganic chemistry and will enable a new generation of catalyst structure/property relationships. Fixed train/test splits that represent common chemical challenges and an open challenge website will be discussed to encourage competition and buy-in from the ML community.

Bio:
Zachary Ulissi is an Assistant Professor of Chemical Engineering at Carnegie Mellon University. He works on the development and application of high-throughput computational methods in catalysis, machine learning models to predict their properties, and active learning methods to guide these systems. Applications include energy materials, CO2 utilization, fuel cell development, and additive manufacturing. He has been recognized nationally for his work including the 3M Non-Tenured Faculty Award and the AIChE 35-under-35 award among others.

Larry Zitnick is a research scientist at Facebook AI Research in Menlo Park. His current areas of interest include scientific applications of AI, language and vision, and object recognition. He serves on the board of the Common Visual Data Foundation whose mission is to aid the computer vision community in creating datasets and competitions. Previously, he spent 12 great years at Microsoft Research, and obtained a PhD in Robotics from CMU's Robotics Institute.

Host of Seminar:
Brandon Wood
Data & Analytics Group
National Energy Research Scientific Computing Center (NERSC)
Lawrence Berkeley National Laboratory

A

um It's so it's my pleasure today to introduce uh zach, ulysse and larry zitnick. um It's been a lot of for me to work with them over the past year or so.

A

Zach is an assistant professor of chemical engineering at carnegie mellon. He works on the development, application of high-throughput computational methods and catalysis machine learning models to predict their properties and active learning methods to guide these systems.

A

The applications of his work include co2 utilization, fuel cells, additive manufacturing he's been recognized nationally for his work, including the 3m non-tenured faculty award and the american institute of uh chemical engineers 35 under 35 award, among others. Larry zietnick is a research scientist at facebook, ai research in menlo park.

A

His research areas of interest include scientific applications of ai language and vision and object recognition recognition. He serves on the board of common visual data foundation, whose missions to aid computer vision community into creating data sets and competitions. Previously he was at microsoft and obtained a phd in robotics.

A

So with that, I will let them introduce the open catalyst.

B

Project super thanks, brandon for organizing um great to see everyone again. I wish we could be doing this in person. I really really enjoyed my visits to lbl in the past, but um hopefully some point in the next year we'll be able to do that again.

B

I know that we've or I've had a chance to talk about some similar ideas in the past. There was a nurse seminar last spring that I think a lot of you were at. um I also talked a little bit about presentations at one of the at the summer school in the fall, and so I think my plan is basically to keep it pretty brief on.

B

Why we're doing this and just sort of introduce things and spend most of the time talking about the data set itself and let larry talk about the current modeling efforts and some of the cool things that we're working on with brandon and others.

B

Cool you want to move forward, hey you want to introduce the team.

C

Yeah, I mean, I guess, the uh it's a large team, both from cmu and fair um from parasite, there's a bunch of uh engineers and scientists working on the project across the organization uh and obviously zach and his team, plus brendan from nurse and uh a summer intern uh waiha that we had from stanford coming and joining us last summer.

B

Yep and it's been super fun, because a lot of the things we're going to be talking about have um not all those people are full time in the project, but um there's a lot of things going on in the doe and elsewhere on how to apply these sorts of ideas to catalysis and materials discovery. And so I think, there's going to be a lot of impact off of this.

B

So the general high level background for why we're interested in this problem and why we're spending time on this is to try and make it easier to find materials for electrocatalytic chemical transformations. And so this is a a very high level overview of how you can take different energy sources and use it to drive different transformations.

B

We can take more traditional power sources and take the co2 from that and turn it into biomass, or we could take that co2 and try and turn it into a more interesting building block through some sort of hydrogenation reaction.

B

We could take renewable energy and directly power a car through some sort of um battery storage, or we could take that renewable energy and we could take nitrogen or co2 from waste processes or from the air and water and turn it into building blocks. We can take those building blocks and run them through thermal catalysis pathways. We could also take the outputs directly things like hydrogen and use them in fuel cells.

B

All of these are really interesting from a chemical engineering standpoint, because most of these are new processes. There's issues of scale.

B

Things have to be cheap and durable and active in order to get them actually in use in practice, and so there's a lot of challenges here.

B

Moving forward.

B

Cool, so at the heart of that diagram was taking electricity and building blocks like co2 and water and nitrogen, and turning into something more useful and in a lot of cases, the right catalyst in order to do those actively and selectively and stably are not really known, and so the standard workflow for how to understand how we should modify a surface or design. A new surface is something like on this scheme on the left. This is from a review by costum reuters group who's.

B

Now at the fritz harbor institute, we start with some sort of understanding of what descriptors are important for a specific reaction. We use those to screen and down select for any particular surface that we find interesting. We might come up with an active site model to say which facet or which surface or which site is most interesting and worth spending time. On whichever surface we come, we decide on. We can start doing elementary processes and start thinking about individual reaction steps.

B

If we still find it interesting, we can combine all of those steps into some sort of microkinetic model, perhaps with spatial diffusion and off of that microkinetic model, we then try and get understanding about which steps are limiting and that might inform the descriptors and the process can repeat, and so what we're going to be talking about today is just one step of this puzzle in how to predict energies of small molecules on surfaces which features in the upper right.

B

If I'm interested in a kinetic pathway, this plays into what is the reaction, energy and that's also correlated with the transition state energies. It's also used in things like linear scaling relations in the bottom right, where we can also estimate those barriers directly off of the reaction energy.

B

If we can have models to tell us what are the energies of various intermediates, we can make this process a lot faster, and this is really at the at the center of most catalyst screening and discovery. Efforts that you see or read about.

B

So this is just one example. I think I've shown this video in some of the presentations before this is a relatively simple example: there's no defects, it's a relatively ordered surface. This is an example of an o-h moving sites from a green atom to a purple atom.

B

We see that there's three different types of atoms: large purple, large, green and small green as well of the oh in the center- and in this case this is a local relaxation. The oh is swinging around from the um to from the green to the purple, so there's a little bit of a site movement, but not a ton, and even for the simple case, it's something like probably one to five thousand service units at nurse.

B

If we were doing this on corey, and so this from my perspective, looks very, very, very simple, but it's still a relatively costly calculation.

B

If you go to a c2 molecule, life becomes a lot more difficult, and so again this is a relatively simple surface. We see the entire surface, moving a little bit at the very beginning of this movie, but now because this thing has a little bit of flexibility, the oh group is swinging around on both of them and it's still staying in pretty much the same site, but we can see there's a little bit of a wiggle and now this is a much more complex problem and so currently right now there are no machine learning models.

B

That can tell me what is the energy of a molecule like this sitting on an arbitrary bimetallic surface, and this is something we want to be able to address.

B

So um actually trying to solve these problems is, is a long hard process, and what we're talking about is just one step in this process before we can talk about applying some sort of fancy machine learning model to try and solve these things. It's really helpful to have a data set that is large and well distributed and sparse and has other nice properties things that larry and I had a lot of discussions about, and he really drilled home how important it was before we really went down this path.

B

So this has been super super super fun, um but really everything we're talking about right now is just the data set and the first models. There are still a ways to go from actually connecting this to direct predictions.

B

To give a little bit of context, data sets and catalysis have been quite small compared with inorganic or inorganic materials or small molecule efforts, and so I just wanted to highlight a couple of efforts by myself and others.

B

The catalysis hub, which was organized by the suncat center at stanford, is one of the larger dataset efforts around right. Now they have about 80 000, unique calculations, maybe up to 100 now of very small molecules and biometallics, mostly from one large screening study, but they also have a lot of other transition states.

B

My group in the past also made some data sets for co and hydrogen on bimetallic surfaces, which were particularly tuned towards finding active co2 utilization materials.

B

Both of these are going to be quite a bit easier to fit than the data set. We're going to talk about in a minute, because they're much more dense and less diverse and they're more focused on a specific are smaller adsorbates in a much smaller composition space.

B

Other data sets that I've seen are various efforts that were deposited in things like iot, mbd or nomad, which is sort of large collect all the electronic structure. Calculation approaches out of spain and germany respectively, but really in catalysis. There aren't too many that are really really large scale, and so that was something that we were hoping to address with this study.

B

Okay, so, like I said the first step in this process is, we need a large data set before we start training models, and we want to sort of ask: how far can we push things just by making the data set larger and the models more interesting? So what I'm going to be talking about? Is this open catalyst, 2020 data set?

B

If you have questions about after this, I'm going to try and go a little bit later in the details, because I know that brandon understands all of this, and if you have questions you should go and chat with brandon afterwards and or ask us questions at the end.

B

um This oc 20 data set uh is quite broad and sparse, and what we did was we tried to choose descriptors and small molecules that we thought might be interesting and broadly um broadly, applicable to catalysis, not focused on any one, specific chemistry or pathway, but really indicative of the effort day-to-day of our community. So we have nitrogen-containing molecules, c1, c2, single atom, descriptors.

B

We started with stable materials from the materials project awesome that they had already screened for things that were stable ahead of time. That made our job way easier. um It includes things like metals, intermetallics, dope materials, ionic compounds, 2d materials.

B

The pictures in the center are randomly pulled from the data set, so it gives you some idea of what what you can see. Some of those you can see are already 2d materials.

B

You can also see that there's different adsorbates different configurations, there's quite a bit of complexity, in addition to the actual relaxations in the upper right for a subset of the calculations that we did. We also did some really simple, off-pass sampling and some very, very short time scale md on the surface, just to get some idea of other degrees of freedom.

B

Some very simple electronic structure analysis- and our hope again is that this helps ideas like how to design materials for fuel cells, agriculture, co2 utilization and so on.

B

So the way that we did this was specifically not trying to densely sample all of the possible calculations on a surface, which is what we would do if we were actually trying to understand any particular surface for a catalyst study.

B

Instead, what we did, because the space is so large and there was no way that we could report this brute forces is we tried to design a a random pipeline where we would first choose a random number of comp, a random number of components.

B

We would then select a stable structure from the materials project with that number of components. So we try to distribute single versus binary versus ternary surfaces. We would then enumerate low index surfaces with pymagen and randomly choose one. We would enumerate adsorption sites with cat kit, which is sort of similar to what is in pi megen um to randomly find an absorption site for a particular durbate.

B

We would run a very simple dft relaxation do some very quick checks for convergence or calculation failures reference the energies like we normally would to the bare slab energy and the gas phase energies and then do some quick follow-up calculations on electronic structure and again this is extremely sparse on any particular surface. In a derby combination, we have maybe one calculation.

B

So this is not really at the point where we can say we have the best configuration for any given combo. This is really just saying of the entire space. Can we get a random sub-sample to train models on, like I said, in addition to actually running the calculations, we did some electronic structure calculations to try and figure out why things were the way they were. We haven't used these for any models yet, but I think they're going to be interesting down the road.

B

These are things like beta charge, analysis to assign charges to the metal atoms or the small molecules and high cop analysis, with this lobster code to try and figure out bonding versus anti-bonding features in the electronic structure. We also did some very short timescale: md and rattle geometries.

B

I think the correct way from wrong larry, I think maybe the rattled ones are maybe up now or they will be soon. The other ones are still working on data sets cool larry says yes, so I think rattled is up. um If you're interested in the beta chart or the icop analysis, I don't think that's been posted yet, but just shoot us an email and you can be part of getting the format right and making it easy to easy to use. Our hope is that someone builds a better model off of these additional features.

B

In addition to just releasing the data set, we also spent a lot of time talking about what are the tasks that we want people to solve, and this is where I want all the people on the call to sort of jump in and start thinking about how to how to solve these tasks and do a better job. There are three things that I wish people could do from a catalyst point of view. The first is, we run dft and it's a little bit like a black box.

B

I wish that we had a method to just directly give us the energy and forces from the structure. So this is a lot like machine learning, potentials and neural network potentials. That's the most general challenge. If we can solve that, we can also solve the other ones.

B

So s2ef is pretty straightforward but hard, because the data set is large and this is a very complex space. In addition, two things that a little bit closer to where we do day to day that I think we can build machine learning models for without solving s2ef, maybe is is2rs initial state to relax state. Given an initial state, can I guess the relaxed state of the local optimization starting from that initial one? We do this all the time we run um hundreds of calculations every day we just do local relaxations.

B

I bet, if you did an analysis of all the vast jobs at nurse. Probably 80 percent fall in this local relaxation category.

B

I really wish that we could just guess that without having to do the full steps along the way and the third one is initial state to relaxed energy, this is the one that we might use in a microkinetic model. I want the energy of a particular durability on a surface.

B

I don't really need the structure for that one, and so it's possible to just guess that energy directly, that's where most of the effort in the catalysis community has been, but it also has some subtleties if things are moving sites.

B

There are reasons why you might also want the structural information as well, and all of these three tasks are interrelated in that the top line up here on the right hand, side, that's the standard approach. We get the relaxed state by doing relaxations on the dft potential energy surface, with a local relaxation method like the fts, we could replace dft for a black box s3f model. That's the second row. We could directly guess the the relaxed state. We could also go all the way and just predict the relaxed energy. All of these are possible.

B

We encourage creativity in solving these methods. We don't care how you solve each one, but if you can solve it, that would be awesome.

B

And then I think, one of the final things that I'm going to talk about before we sort of flip it over to larry is we also spent a lot of time talking about what are the right train test, splits that are indicative of what we're doing in catalysis, and so the ones that we came up with are the simplest is in domain.

B

We're asking you to solve those three tasks for a material and adsorbate that you've seen before. So from a machine learning perspective, this is going to be the easier one, but in addition to that in catalysis, often what happens is we do things for one chemistry, but then we switch chemistries and now we need to do it for different adsorbate and we want these things to generalize, so there's also an outer domain adsorbate. This is an exurbate that was never streamed before in the training set outer domain catalyst. This is a binary combination.

B

That's never been seen before or outer domain, both which is the hardest one. That is really something that doesn't look a lot like what was in the training set. All of these are also on the leaderboard. So if you have results for one of the others, you can submit.

B

Obviously the in domain one is going to be the easiest, but we really want things to generalize, and so we we spend a lot of time talking about what the split should be like. Most of these challenges, the validation set is up. The test set won't be released when you submit your your predictions on the test set, um those then get judged, and then they get posted to the leaderboard.

B

Okay, and um the very last thing is data format. um This isn't something that you have to like email us and ask for the data and we'll email back at some point and days in the future and get it to you all of it's up on github.

B

If you go to the website, there's a link to this data set markdown page, you can also just go there directly. We spent a little bit of time talking about how to package this.

B

It's not exactly the same format that we would have done if it was just a catalysis project, because these these sort of standard, asu trajectory files are a little bit large and have a lot of a lot of overhead that was sort of unnecessary and the data set is large enough that we didn't really want people to waste time downloading that extra information.

B

So everything is in this extensible xyz format, which is super super super, simple um human, readable and compressed.

B

We also spent a while trying to make things easier to ingest in the machine learning pipelines, and so we settled on this lightning map memory, database format, lmdb, which is basically just packaging, a bunch of information together into a single file, so that you don't have a million files in a directory from experience. I know that scratch file system does not really like that situation, so we're trying to avoid it. um Obviously, this is not all of the data that we collected things like out cars charge cars.

B

Everything else that came out of asp were saved. If you have a really good reason for why you want that or something else that we missed or something that would be helpful, send us an email and we'll try and figure out a way to get it to you. It's just not up on this on github repo.

B

So I think with that, larry is going to talk about the current modeling efforts.

C

Thanks zach, um one thing I want to touch on really quick is some of you might be wondering you know. Why is facebook, and why is fair? You know interested in this problem um and I think it's important to you point out two reasons for that. One is uh just the problem itself. You know it's one, that's meaningful! It's one! That's important! uh You know climate change and renewable energy and making things sustainable is something that you know a lot of people fair care about, and you know being able to work on something related to.

C

That is something that a lot of people on the team are excited by and that's why there are so many people in fair working on it, but if it was just that it wouldn't be quite as exciting to the folks affair, what really gets them excited as well is just the ml problem itself. um I I, after like pounding on this myself for the last eight months or so, and working with others on this.

C

um I'm just humbled, and you know I just it's a thoroughly enjoying problem to work on, because it's it's so much different than the other type of machine learning problems that we work on in the field like in computer vision- and you know language and that sort of thing um it just opens up a whole new. You know set of of interesting things to look into so um you know, I think, I'm hoping that the machine, the rest of the machine learning community sees this as well and gets excited about the project all right.

C

So let's talk about um approximating dft with ml and really the high level goal that we have here is you know in for the data set, the our average relaxation took about eight hours and um we're hoping to reduce that basically to one second, uh which is possible if the ml models are accurate, so right now they are fast uh as you'll see in a second, not quite as accurate all right. So let's talk about uh the models that we're using.

C

So the um all the approaches that we've been looking into are using something called a graph neural network where every node in the graph corresponds to an atom, uh and then all the edges correspond to the neighbors and generally we define neighbors as any atoms within you know, six angstroms of each other and then the inputs to the system are, you know, just the 3d positions of the atoms and their atomic numbers and the output is just the energy of the structure and the per atom forces.

C

So you know, especially from an ml standpoint. This seems like a super, simple problem. It's like easy inputs, easy outputs. You know like plug and chug. It should just work, um but what I think is underappreciated- and this takes a little while to appreciate- I think, from an ml perspective- is just how complex this problem is because it's not just a bunch of atoms. Moving around, like um you know, a bunch of ping-pong balls, you know, there's all those electrons flying around.

C

They have these really complex interactions so and then the the subtle positional changes in the atoms can make a huge difference in the forces, and this is something that I've come. You know to respect in working on this problem as well. So all of these things uh make this problem much more difficult than you might think. uh Initially at least from for the ml folks. uh You guys probably have a lot more context on this and you know a lot, a lot more humility on this um than the ml folks.

C

So let's talk about the models themselves uh again inputs. You know: 3dm positions, atomic number, output, energy and 3d atomic forces um and then there's two like from a high level. There's two types of models that we we've been looking into. One is what we call an centric model.

C

So what this model does is takes the the atom information as input and then the output is the energy of the structure and if you want to know the 3d forces, what you do is you basically do a backprop step and you compute the negative gradients of the energy with respect to the atom positions all right.

C

So basically do a forward passive network and then a backwards passive network to find the atomic forces, and then the other approach is a force-centric approach where you just take the atom information, and then you just do a single forward pass and you just predict the forces directly.

C

So let's look at a couple approaches in each of these um a little bit more detail. So an energy centric approach. You have the 3d positions of the atoms. You feed that into your your model, your neural network. Whatever it is, you compute the energy and then you basically do a back propagation step, the same thing as when you're learning the weights and then that allows you to estimate the 3d forces of the atoms.

C

Now, what's really nice about these energy-centric approaches. Is that they're energy conserving? So, as you guys, probably know via three points you know in in in space, is if it's energy conserving, then, if you go, you know in a circle the the forces sum to zero. This is really important. You know for applications like md, not quite as important for relaxations, but you know this is just a in general nice property for these networks to have now. Let's look at one of the simpler models that that uses this approach, and this is shinnette.

C

uh This model is, you know three to four years old now, but we found it is it's a good baseline and a good place to kind of start thinking about this problem? So what it did is you have your energy e that you want to compute. You basically sum over all the atoms in this structure, and then you have a neural network here: f, which then takes a hidden state of the atom, runs it through the neural network.

C

You sum that together- and that gives you your energy and then these you know hidden states are refined over k iterations.

C

So let's talk about how we update these hidden states, so our hidden state here is, is equal to the previous hidden state, plus the sum of all the messages coming into that node. So we have a bunch of source atoms here and we have our target atom here.

C

If we look at the message update, it looks like a formulation like this: I'm not going to go into the details of exactly everything here, but what's important to know is that this neural network, this series of neural networks is a function of the source, node's, hidden state and also the edge information and the edge information. We should think of that, as is just the distance between the two atoms right now.

C

What's interesting, is this distance between two atoms that you feed into the network goes into a function b here, which is a function of the distance between the atoms and you might think? Okay, how do we take the distance between two atoms and make that usable by the neural network? You could just feed in that information directly, but you know neural networks generally, don't do as well with just a single. You know a single value.

C

You want to put it into a form that neural networks are a little bit easier and ingesting, and one of the easiest things that people did initially was just to discrete discrete ties space. So you go from zero to six angstroms and you just discretize it. You know like a hundred times, uh then you have a one one hot encoding.

C

The problem with that is that creates a discrete filter and then, like I mentioned before, if we want to compute the atom forces, you have to do that back prop step and whenever you have a discrete filter, it's not continuous and therefore you can't do that back problem.

C

So what the biggest you know advancement that chanet made was they said: okay, let's not use a discrete filter, let's use a continuous filter and they basically used a radial basis function or you can think of it as a series of gaussians um evenly distributed from zero to six angstroms or I forget what exact cutoff they use, but you know something close to six angstroms and then you get this continuous filter that you can back drop through.

C

Now, if you look at the actual filters, they look like this, and what's interesting here to note, is there's a lot of interesting patterns but you'll notice that the filters are the same radially they're, you know invariant with angular information. So all chinet is capturing is basically the distance between the atoms. Now the thought is that you can capture a little bit of angular information, because you could have three atoms and, as you're passing messages in between you can kind of look at the relative distances between you know and e3 and that'll.

C

Tell you a little bit of information about the angular information. That's there, but you know it's kind of hard for the neural network to kind of pick up on this, because you have to do several different iterations in the message passing from the network to be able to figure these sort of things out.

C

So there was a follow-up work which called dime net, which wanted to make this angular information more explicit and the way they did that is they basically looked at three atoms. So you have you know your your um your target, node here j.

C

You can have your actually the target node here is, I you have your source, node j, and then you have another node, let's say k1 here and you can look at the angular information between um k and I with respect to j, and this allows you to basically looking at triplets of atoms to basically encode this angular information, which is really important for modeling the interaction, the interactions of these atoms, and then you can see some of the the functions that he learned.

C

They encoded the angular information using spherical harmonics, which is a great way of encoding. You know angular information on a sphere and then the learned filters look something like this, and now you can see that it not only has a distance information but also encodes, some of that angular information as well.

C

The problem with this approach, though, is that you have to look at every triplet of atoms. um You know so that's that's going to be computationally expensive and especially if you look at the overall network architecture here, you don't need to understand this all. But this part here in the middle in blue is what you have to compute for every triplet and that's a lot of computation and this results in this approach, while being accurate, being computationally a bit expensive.

C

So now, let's switch gears to a force-centric approach so, and this model does is very simple: you take the 3d atom positions and the atomic number information of the atoms. You feed that through your model and it just and it directly computes the 3d forces- and this is you know how you know- somebody I would say like if I was to purchase problem the first way.

C

I would would try to tackle it and what we did is we- and this is the approach that we um attempted uh last summer with an intern, and we called this method forcenet and what we did is we took the net message passing scheme and we basically just updated it to make it a bit more complex, so it took into account not only the hidden state of the source node, but also the hidden state of the target node, and we fed this information in at multiple places within the overall network, so that allowed the the neural network to learn a little bit more complex interactions and- and formulations of you know how to update these messages now.

C

The problem is: is that this est, basically the edge information? Is we not only fed in the distance information? You know how far away two atoms were, but we also fed in the you know: how far were they apart in the x? You know dimension y and then z, um and this gave it more information because it actually encoded in in essence, some angular information as well.

C

Now this is good because it gave the network more information, but it was bad because now the network is no longer rotationally equivalent because, as you took the entire structure and you rotated it, then the forces that it would compute wouldn't be consistent. You know they wouldn't rotate with it. So that's it.

C

That's a downer, because that's a property that you would want in these networks, but what we found is that in practice, if you use lots of data- and then you use data augmentation to basically rotate the systems and then train with rotated systems as well, is that the network essentially learned this in variance it wasn't perfect, but it did a pretty reasonable job all right. So let's look at some results so for training all of these models we trained them on 64 gpus.

C

We use um amp to speed up training and then, if you look at the training times uh for chinette dime net enforcement, you can see that you know the schnet and force net are relatively fast and you have dime that is taking you know quite a while to compute. Obviously we didn't wait. uh 1600 days you had to divide this number by 64. You know to actually get the the amount of time they took us walk along time.

C

This is a nice summary of the some of the results that we've got and here we're showing is the force mae on the y-axis and then the training time it took in gpu days, and you can see generally. What you want to notice here is that you know four stand and force net large, get roughly the same accuracy as a dime net here, but they're taking.

C

You know an order to magnitude less of training time and then, if we look at inference time versus average force, amee we're basically noticing the same trend where force at large the larger version of horsenet and dime that large are roughly the same and then force net and dimenet plus plus, are roughly the same but again at reference time.

C

They're a lot faster, and this is due to the triplet thing I mentioned before, but also the fact that the four-centric models are in general a little bit faster, because you'd have to do that backward pass as well.

C

Now, if we look at the different validation, splits that zach mentioned before, uh what you'll notice is that the in domain values are lower, you know, they're, like the accuracies, are better, just as you would expect, um and then, as you go out of domain, you know with the absorbate and the catalyst and then with both you see the results get worse. So this is kind of you know. This is what you'd expect what's nice here. Is that when you go out of domain, you know absorbing catalyst?

C

Is the results aren't totally garbage like the results are a little bit worse but they're not a lot worse, so it means that the networks are generalizing at least some now.

C

One good question here is like: can these models actually replace the eft and one of the metrics that we so the force me and the energy me is good to kind of track progress? um However, we wanted a metric, which also said: is this good enough to replace this practically so we created another metric called energy enforced within threshold, which is essentially are the energies and forces predicted within a certain threshold of the ground.

C

Truth um and that threshold is set to be quite tight, so that way, if it does, is if it is below that threshold, it is good enough. You know we thought it'd be good enough for practical use and, as you can see here, um none of these methods are doing that. Well, uh you know nothing's even hitting one percent. You know here, you know I don't know. Almost all of them are close to zero and some of our latest results.

C

You know getting close to point one or point six percent or point seven percent, so we still have a long ways to go uh before these models can be uh um practically useful in this way.

D

Can I just sorry, could I just ask a question on that because it seemed quite important. I just wanted to understand this. This metric is what what is the number that you want to achieve. I think.

C

For energy we wanted it to be in within 0.2 um and then the forces it was 0.03 or was it 0.05 exactly remember.

B

I think it was maybe 0.02 for energy and maybe 0.03 for forces and those are motivated by if you're doing a relaxation you normally terminate at about um 0.05 or 0.03 ev for angstrom, and so you can't even get the basin correct. If you don't have that sort of baseline accuracy.

D

Okay, so that's like, if it's within this threshold, it's good enough, and so then you want to get 100 on this.

C

Yes, okay, right and the forces are measured for every single atom. So even if one force, you know if the force for a single atom in the entire structure is above that threshold, it's count. It's counted as no good, okay cool. So it.

B

C

Really challenging metric.

B

I think one other comment on that. um uh Inevitably, one of the questions at the end of this talk is going to be um what are what about dft errors or other inaccuracies or other other residual issues that we know could be cropping up when you do these calculations um and like there's no way of getting rid of all that numerical noise. So there's a little bit of wiggle room here, but I think this is a good time to bring it up, because um I think those are not an issue 100 of the time.

B

I could see it being issue, one percent of the time or five percent of the time or something but we're at like way less than one percent, and so um I could believe that there is a ceiling to what edfwt could be due to some other random things outside of our control.

B

That is maybe 80 90 95 99. I don't know what that number is, but we are really far from that. So I would be a little bit surprised if anyone ever got a model that got 100, but that's the goal um I I feel like 90 should be should be very.

C

Feasible all right um so then, even though the accuracies aren't that good, it's still an interesting question to say: okay, what? If we take these ml models and actually use them for relaxations, so we basically, um we basically use these models as a replacement for dft.

C

Where you know we uh take down the positions, we keep the forces, we update data positions, compute the forces again until until you hit convergence.

C

So, ideally, what we'd have is something that looks like this. This is a cherry picked example where we have dft shenette gns, a method I didn't describe and then force net large, and you can see that as the as you iterate, basically the force at large and dft kind of converge to the same energy.

C

But if you, but if you replace the x-axis with time- and this is iterations- you plus replace the x-axis time. What you'll notice is that um all of the ml models are just do this much more quickly than the dft approach. So I think here we're talking about 10000x faster, and this would be ideal because, if you're hitting about the same accuracy, but just a lot faster, looking at some qualitative results, you can see here some initial starting states. We then have the relaxed states, that's computed by dft.

C

You can see what chanette would produce and you can see what dimenplus plus produces. There are some good examples here. You know the these two columns here, where dime that does a fairly good job of predicting what dft did.

C

But then you have some other cases like this one here in the blue, the second one over um which is quite different, uh and you can see there's differences here too in the far right one and then you can see in general chanette just does you know much more poorly and again, I think that's because it's not modeling, um you know a lot more of the information. Just is looking at the distance information.

C

If we look at quantitative metrics, we have two different metrics one is adwt, which is the average distance within a threshold.

C

This is looking at the the distances between the where the atom should be, as predicted by dft and where the atoms resulting are predicted by the ml models, and then we have a threshold on those distances and then we can average across those thresholds and this metric here the top one is what we wanted to use for um the research community to see if we're making progress- and the main thing you should look here- is the absolute value doesn't matter as much as more of the relative values that matter here and you can see- and this just reinforces that you know the dynet models are performing a lot better than the schnett models, um and then um you know and then the larger the diameter models, the faster they perform, and then we have another metric which is again geared towards.

C

Is this particularly useful? So this is the forces below threshold. So what we do is we take the ml computed relaxed state. We run a single point, dft on that, and then, if it was a true relaxed state, then the forces should be really close to zero. So what we're saying is: are all the forces on the atoms below a certain threshold as computed by dft for the ml computed relaxations all right, and here again we can look at the numbers and, not surprisingly, you know we're not doing that good.

C

You know we might have the best models were almost getting up to one percent. You know accuracy here um so again. The ammo models here are not producing what I think we would. You know term as a proper relaxed state, even though the ml models think they did hit a local minimum. Can I ask why you use a threshold rather than say the mean squared error?

C

um We were thinking that it would be. You could use a mean squared error, um but then that can be you don't know. If the error is due to you know one you know outlier or if it's just muslim or close to zero, we the forces below threshold, we thought would give you a better indication of given the number of ml relaxations that you do. What percentage of them do you think would be successful and that's how you would interpret this metric.

C

E

Thanks so excited, excuse me, I have another question about about this, so so that's the forces from a single point. Calculation right, but are you, are you still? Are you still closer to to uh the relaxed geometry? So like? Let's say how close are these right? So you know, did you if you can get 90 of the way there and then um then you're in a quadratic regime where newtons will find you the answer in like three steps or something then um I mean, wouldn't you still be happy.

C

Yeah, I know it's a great question and it's one that we want to look into and we haven't looked into thoroughly yet, which is um you know you could start from an initial state and do a dft relaxation and in general those take. You know 200 time steps.

C

However, if you started from the ml relaxation, the hope would be it would take a lot fewer. You know, dft calls to get the same, relaxed state um and we don't know what that value is yet yeah. We just haven't done those, but we do think I think in general that the ml relaxation should get you a lot closer than the initial structure, so not like just flying off into craziness at least what um what's prevented.

B

You from trying that out is it just like you had other things on your plate, or is there some kind of inherent difficulty to doing it.

C

Yeah other things on our plate, um the there's going to be more and basically what what we're working on one of the the things we're working on right now is basically the entire pipeline to go from a surface and absorb it, and then compute the absorption energy by trying out all the potential binding sites and there's so many different ways of injecting the ml models into that pipeline.

C

To make it faster, because, at the end of the day, what you want to do is reduce the total number of dft calls that you're doing to as small of a number as possible, and that's something that you know the engineering team is currently working on is there's.

C

You know three or four different knobs that you can play with to try to reduce number dft calls, and what you guys are mentioning is is one of those, but we don't know that we don't know the actual numbers on that yet because it just happened at the time.

B

I think it's also worth pointing out that if you take that strategy, you can only do as well as the minimum number of dft calls. So if you bring you down from 100 to 4, I think was was what um I I forget, who asked brendan, I think um was was saying um you get a factor of 25, which is awesome and I'll.

B

Take 25 right over over nothing, but that's also not um 10 000, like eight hours to one second, so um yeah there's a difference in scale there of how far you can push the idea.

C

Yeah, if you just do a single dft, call that swamps any ml that you're doing so. Ideally, what you'd want to do is just do only ml on everything. Then you find you know the thousand or ten thousand most promising. You know candidates, and then you do dft on those.

C

um That's our hope, but I don't think we're gonna we're not gonna hit there for a while, still because of the accuracies again, I'm quite there. Yet.

C

All right and then data set size not too much to say here. Other than bigger data sets help. We are seeing diminishing returns and we're. Looking into that.

C

One of the questions is, um you know, as you see, diminishing returns with data size, sometimes that's due to um even more data, not helping, but is it because the models can't um uh you know, use that much data, because the models aren't big enough and that sort of thing or maybe the models, aren't expressive enough to take advantage of that additional data and we're also looking at data augmentation techniques to kind of help with training as well.

C

So I think, there's a lot more to be done in you know, in exploring in this space all right so going forward. I think zach. You are.

B

Yeah sure yeah just a couple of slides just to talk about some things that we're working on and how this connects with some of the things going on at lbl and nurse. um The first one to mention is that, as the models get larger, as larry said, things tend to get better, which is interesting, and I think this data is already out of date. um Large here is eight million, but I think on one of larry slides.

B

He showed a model that had 35 million parameters, and so one thing that has been interesting is that even scaling up to 35 million, it seems like every time the models get larger.

B

The validation errors go down, so we aren't seeing overfitting from model size which, from my perspective, is very interesting in most of the data sets we worked with in the past, there was a very strong overfitting, because the data sets were a lot smaller. So the obvious question is: how far does this go and um is there a limit to how far how big that regime is and or um if we just take a single model and scale it up to something like perlmutter?

B

How far can we push that number down and that'll be really interesting, both from a um a scientific perspective and from a machine learning perspective like? Is there a way out of this just to model, or is there some fundamental limitations that we have to work around, and so these are things we're working on with brandon and steve and mustafa before him?

B

In addition to the scaling question, um brandon's done a lot of awesome work on doing distributed, hpo and getting that ready to go on promoter so that we get the right hyper parameters as we scale, and so I think um I think things will be really exciting for the for the next few months. As that machine comes on.

B

Another example I just wanted to point to- and I see duo and ryan on the call is a project that we proposed a long time ago. I think, maybe two years ago now um and at the time we didn't have oc 20 or anything like this available.

B

We said that we were interested in screening for nitrate reduction, catalysts for water quality applications as part of the nawi hub, which is led out of lbl and anabol jain and waitang are the other um two pis on that and helping out um what we've been doing is just seeing is any of the data that we've collected for oc20, already sort of correlating with what happened in previous benchmark results.

B

So um there were some great literature studies by brian goldsmith in raleigh spring neural sing at university of michigan, and so um this map on the right hand, side is a microkinetic model that they came up with.

B

This is data that um richard and joe collected um tie lines are basically between numbers that we get from oc, 20 just randomly, and the numbers that brian and nirala reported in their paper, and so it's interesting that for these pure metals, we're already seeing a pretty good correlation and that's not even with the actual lowest energy binding site, just sort of randomly choosing calculations.

B

There's already some information embedded in what's going on here, so the obvious step is to use the models that larry was talking about and the additional data to screen for new materials, we'll see how far we can take that- um and the last thing I just wanted to point out- is that um the website is open.

B

There's a leaderboard on it. You don't need to email us to get on the leaderboard. There's an automated submission script that um devi parikh, who is joined between georgia, tech and facebook, and her students are really helped to get up.

B

If you have results- and I really really really hope that some people on this call come up with something creative that we haven't, thought of and and published in this area, um you can submit and we'll show up on the leaderboard, and if you go forward one more larry. I just wanted to highlight that we're already seeing people citing the data set and trying new things, and so the first two that I'm aware of are aj medford at georgia.

B

Tech has a paper up on archive where they're, using this sort of pseudo-electron density, representation for machine learning models and um they're getting closer to being able to train on the entire data set. So you can read that in archive if you're interested and shurong g at texas, a m is more from the machine learning graph side.

B

He also published a put a paper up on archive on spherical message, passing networks, and so I think this is a great indication that people are already starting to think outside of the box and try new things and again. I really hope that some of you.

B

Find the data set useful, can train interesting models and then can add stuff to the leaderboard.

B

Very very last thing: if you have questions, feel free to email, myself or larry. There's also a discussion board up on open catalyst project that that's up now, where you can ask on public questions. You can also post issues directly on the github repo, if it's more more detailed, okay and I'm just wrapping everything up um again, it's just worth highlighting that this by no means solves the entire problem of the entire pipeline of predicting what an electric catalyst is going to do.

B

There's a lot of super super interesting questions about putting these models into development software engineering about scaling, um comparing with the high throughput experiments, actually making things finding industrial materials. So this is just the beginning, but I think there's a lot of really really interesting projects coming down the pipeline.

B

Okay, that's it thanks. Everyone.

A

Thanks zach and larry.