GeoSci.xyz SimPEG, 1 May 2018

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: simPEG Meeting May 1

Description

Dom Fournier shares his work on plugging SimPEG into GIFtools

A

And then then, there's like two weeks of spring and then it's not pergolas. If.

B

Things go so quickly. Yes,.

C

A

And you got those really long days: I mean everything's, not okay, I gotta grow up, I got to reproduce.

D

In a few different times of the year, it's spring August and then I've never been there in the dead of winter. So I'm not sure how cool it really gets. Yeah, but.

A

That's very very nice because it's really clear, sunny dry.

B

Everything gets colder, it stays seems like you get these.

A

Big high pressure regions, but it's just beautiful theorem you just have to like wear sunglasses all the time. Oh well, I, that's good for an extra 20 degrees when it's sunny and dry compared to that Cougars and easier I think to keep warm to put on extra moon. You just don't have that penetrating humidity yeah.

D

Which is actually nice it'll be hard without making you want to die.

E

It started by saying that Sonny and I have started about two months ago already started talking about.

E

Polishing I guess at one of the new one coats and yeah you can't, because we're kind of a very interested in is that we have a six to eight months window where we have the industry sponsors.

E

Most of us are involved into digital project. We have to finish.

E

At the same time, we're all that programmers are all day lockers for this library, so we kind of have this left nice nice window, where we have the chance to be able to ease the attention of these. It's people right. There was like a handful of users, but there are users that have needs constantly needs any happen to research budgets, correctly, initiatives being really well they're released version, fonts are increasing, so those have been X here is gonna, be a pretty good year for everybody, so I'm ecstatic with it were.

E

We've been really good. That started like pushing the code rules and pushing invariance and all fronts, but we haven't spent much time on the user on an end user part of it right so right now we're publishing everything. There's our scripts notebooks, which is good for other programmers, but like for industry, there were no they're likely not going to dive in it's kind of a and catch me to love. You only.

C

Have time to live out the end user part.

E

But at some time it's me them because movies using the code and whatever I didn't sin been in the long run. If nobody's you, so we won't have as much explore regrets that's possible.

E

So the idea here is that including little sounds you can just connect movie in their face that we have our users to be able to leverage this. This effort of lucky make me inversion easier, just connected with the open source, the open source- and that sounds me. Yes, then, on sniffing sighs, no need to worry about visualizing and handling file version, and on to flip side when were like sensing.

E

So this is another. The first step was like: how do we connect.

E

And send jobs on hours time.

C

In line develop new codes.

E

Or like experimental new things and just give the other access to.

E

Did you a scientist, you don't want to do so.

E

Here's the first I guess.

C

E

Will show this and what I would like to be able to talk about with you guys is how we ease that connection out of my environment, because, right now it's a lot of scripting, but we've thought about it before right. You will do it in JSON files or whatnot. So.

F

I think it would be.

E

A good time to read like that.

E

Ok, so you guys are all familiar with this neither's.

E

Tools, the basic edge it feels dizzy, it's a fairly simple other face its building in Java and a scrub down being used be able to view operations import. They create initiatives, written versions of Obama and currently this is like a hundred percent bill towards running UBC goes all the fortune, a schools, but because we built synthetic like it's us in our language. That UBC is you, students there's a conversion methodology. Anything the connection is fairly straightforward. They will do instead of sending a job to the Fortran.

E

Both you just sent to so in a juice tools, world. Let's see we have a one. He had versions will focus on the one, because sucky and I recently.

E

Yeah, yes, that in jiff tools we have an inversion object which come things which knows about about Meadows. It knows about theta or get stuck with my bucket and all that stuff and inversion parameters which were used also Sunday once this object is created. Usually good way to go is that we just wanted a job of the UVC codes right, so it just goes and calls a poor trap. Runs it to the bag.

E

Anomalous returns to eat the result, things for insight here, all of them all out them so far, is just how to add the option that we're going to run this impact same baked. Oh that's so!

E

Click on this and what's happening in the background, is that it calls fight on from the outside and currently just built an inversion script right. So this is just creating breaking this sort of any like we're, usually doing the books. Written is certainly the problem organization and just exchange all the parameters that we seven just little just send them to this MPEG and run out.

F

Just the wrong here, 166.

E

Okay, it's the wrong diversion here: we're saving your thinning directives. Sorry this one yeah, all right, I'm, saving, all the outputs- and currently this is just rip them in the math file format. The dot map, which is the structure so again sent back anything inside our models.

E

Fine, despite meets all the things we need to come up to the Rd acres, but he did they do and then on a map they math aggressive I, have a small script and math out of it once this is run, just slows this guy back in and then reformatted for justice people to digest again. Did they once the inversion is dead, I'm, al d'amato agro them, and then we have the same functionality that that chip- those hats where we can visualize the model.

E

Right visualize, what's in that good and visualize are pretty good.

E

F

E

And then we can also, if I repeat a curves same as foreshortenings, so that we connected all the function after that the users already used to see inches goes to just do it through this waiting.

C

With the minor comment, yeah, it's probably worth looking into instead of Savannah's, a dot Matt's, giving its hdf5 it's a bit more stable and like it'll, be easier to introspective something go wrong, because it is very language. This.

E

E

C

Could I mean because then it's very easy to load it back into Python and figure out what's going on and if MATLAB ever changes how things are safely null hi and the people we sort of found that between Python to you in Python 3, there are few changes, and so they made it backwards compatible. But it was a little bit of a headache, whereas hdf5 is yeah so good. That would be easy way to go and then to if somebody wants to look at it with different software, that's an easy thing to standardize.

C

B

A big effort to kind of transfer.

F

Things over to HDS right, yeah,.

E

So it's just it's just stories as static as a stack of objects means random means of age, 5, arguing today the I don't search 3d art the query from the outside. No it's a container, but it's not really easily very wearable. That events did you guess, just one hasn't you slower didn't eat burger, saying like what, if someone wants to interact with a pony on the outside, okay legit will not read and write from a safe spot, but it's not structured and easy. We didn't just we're keeping all the items of a nice.

A

So that's to to accomplish that. That would be kind of like a wholesale rewrite of a lot of things. Then didn't.

E

Typical stuff, so I'm gonna have to review the HR. It's not gonna happen. Any guess since is moving on it's going to be a GOI.

E

C

E

I was get this one.

E

C

And in this case another thing we need, we don't need a fancy age, five database for this, like all we're kind of, do a store dictionary, so either JSON or h5. Just like capturing the dictionary, that's all that we need to do. Okay,.

G

Nothing so is currently.

E

Like we've done, we've done a better job, that's some stuff! You do not have in Python. It's not. We don't need.

C

Right asking files we just pass.

E

With us by news right, so we're writing out the math laughs.

E

Is there a better way.

H

E

Do with is there a better way to send stuff across it's pretty fast more than that not far away so.

C

I guess I, don't understand necessarily the communication piece and how much back and forth is needed.

E

We're currently it's saving, it's just grabbing all the objects, all that all the data it needs, yeah. Those are just all vectors and yeah creating and then creating that this is just storing everything.

E

C

Can you store it as JSON.

C

Because that, in terms of like deserializing, because the way we're moving with simpang is, we should actually be able to like create I mean the simulation object or an inversion object. Serial is that which then saves all of the parameters that you need to recreate the thing yeah.

B

C

In this case, what you're effectively doing and like this is sort of looking forward, but it's setting up an easy way for your user to just generate that like set of parameters, basically like the set of JSON. Well, I mean it's a dictionary. So your your set of dictionary parameters, and so then all we need to do on this in peg end is. It should be as simple as just deserializing.

D

C

To create all of the objects, then that you need yes.

C

Yeah, okay, um so this is where, like JSON or hdf5 I think is a much better way than dealing with.

E

C

Because I think, like in terms of simpe, it's immediately very reasonable for us to implement an hdf5 through Sabre and loader, and that benefits like both shift tools. As well as somebody who just.

B

C

To use only simple Python right because for me I mean, if I generate a like an inversion. Basically, that's ready to go on my laptop, but I, don't necessarily want to run it there. I want to ship it over to the cluster hmm like doing that. Eh ef5. It's a very easy and stable way to do that, whereas writing, civic dot, mad and sort of creating all that functionality doesn't make sense, not work. Full-Time yeah.

E

And then once them within the education area,.

E

C

And so how that will happen, and this is something that we can sketch out and I think the simulation refractors can help clarify that. But basically, what we're doing is we write things with properties is recreating like in a census of the amount of how to interact with sin. Think so like set your frequencies and all these kinds of things. What are the key words and then how are things nested? And so basically, once we have that structure right, you don't need to know about the inner workings of sin peg.

C

We just need to know how to get stuff in and out of that structure, and so that structure will then define the data structure of like in hdf5 or JSON of us.

I

So how far are we play bit so I could sort of getting there simulation part Oh, baby, yes yeah, but the inversion like oh, oh I, like I, read like it can clear yeah. How are you serialize infer because, like it's a multiple object and then then you got multiple pieces.

I

C

So that's something that I thought.

I

C

In that case, what you like all that we need is that all of the pieces serialize.

C

So let's take the simulation, for example right now the simulation you need a mesh or any of the PDE simulations right, and so the meshing is actually a distinct object became, but because it serializes we can.

B

C

Serialize that one be serialized the simulation right, there's all the instructions, all the way down, that like dependency path, are serializable, so that than what we need to implement on the inversion side as well is then looking at all of our directives that I think the optimizing or the objective function. Stuff is really close, if not they're, basically already yeah, but then once we have like directives, the optimization serializing and the objective function, stuff serializing, but then basically means you can serialize the inversion object right.

I

Is it possible, but give their multiple properties, for instance, and like we're not doing all of that, for instance, but but but my guess at the moment, like.

B

I

B

I

Everything so what.

C

We need to be careful with, and this is something I'm trying to do, and this is why I like I sent that over well when I'm in looking at the simulation stuff I'm trying to tease apart what needs to be a property and what doesn't, and so there's some cases where properties isn't used and it needs to be injected. There's other cases where properties is being overused, and so what I mean by that is that if there's something that can be generated based on user input data, then it should not be a property.

C

I can show you an example. If that's helpful, yeah.

I

H

I

H

I

I got the point yeah.

G

C

G

Me just find this first.

H

Yeah, that would be nice as a really beginning. You said, like you, said,.

H

J

Because whenever I see invisible eyes.

H

May need to develop process so that each by the beginning, with rather and that's.

C

Something that we should be sure to set within simple, so that it is easy to set a random seed and serialize that as well.

I

C

In specific examples, but we don't have it like serializing.

D

C

Example, so that that would change like because if you ran the whole inversion and didn't save your random seed, unless you were recreating from a script, so it can elaborate on the random seed. So, right now, if like in most of our examples in say fake, we set a random seed pieces.

A

Just for knowing something.

C

You know so the beta estimator, for example, you just a random seed to create a random vector which we hit against both the J BEC, as well as that.

A

C

The back yeah, but.

A

The rad, the random seeds, are always or generating some kind of around the vector so that you make sure that you got reproducible all.

C

Right right well, so that when you set a random seed, what that that means is like we have picked a specific random vector right well beyond the present, but right now, unless that is in your script, we don't actually save the seed any messy.

E

Okay, so we don't know your.

G

Results, yes, yeah.

I

It's kind of kind of how I do with your different.

C

Machine is a little.

I

Bit better yeah version because, like your Maddox, a little bit the right set so I don't know everyday. That's it's not very different.

C

I

Of yes, okay, but it's not exactly the same, so it's not quite reproducible, so you can set like everything really seeing.

I

C

Kinda in my skin for a sec well,.

A

It's actually very important if you're, especially when you're testing with you know, say a few days, as did you know, frequency-domain p.m. right and then, if you start off with your adding errors, you're having a different round receipt, I mean you actually can get quite different errors and what results can really be quite different, so we're not paying for the errors.

A

No, whatever right, but yeah.

C

You might well do it for.

A

The errors, okay, so if you're, if you're, locking in like you're trying to do a Jupiter notebook or something or anything, it's reproducible, then you want to add random noise. But the random noise is the same in every instance. Right yeah, yeah,.

C

A

Easy yeah: that's a.

C

Really good point and that's something that we can easily implement in synthetic data object is a random seed so that you always like once that objects created seasons, yeah, and that could be something that the user could input as well.

C

So this this is just one example that is- and this is like not a knock on anybody's coding like it makes it makes perfect sense. But this is an example when we're thinking about serializing of redundant information that we don't need serialized. So in this case, when we create a survey, the survey this actually the source list will be a property, and so I know I can't remember if that's done yet or not, but it will be uh so.

C

The source list is a property and it contains all of its locations and then each source has its receiver list. So in that case, all of the information about the electrode locations is actually contained in the source list and the receiver list right, and so in this case, if we were to serialize this object, that information is redundant right, the a locations, B locations, m and n. All of that is completely redundant because it's already contained in these sources receiver list.

C

So then, in that case, what we want to do and I'll show you sort of what it's looking like when I.

C

But then, in this case, like oh, that we would need to do is instead of these being properties. They are then just at properties.

C

Yeah and then they are computed and they can be stored, like lazy loading like what we always do, but it's just sort of recognizing that they don't need to be serialized.

I

But let's say it's: let's say that right so I know, I think the lazy loading we could yeah yeah.

C

Because so what I was implementing and I forget which branch this was on, but basically then you do a like a location and then, if well, if it's not there.

B

C

You get the and so.

B

Yeah, so that's like.

C

One simple way to do that.

I

Okay, but there's other cases where, for instance, like a transmitter and types presence, then for using, let's say, I'm using vertical magnetic dipole, though I have a lot of other options, then that's a property, its fees.

C

Well, that is based on the class right, so when we serialize something that's.

B

C

Far, that information is communicated okay, because it is a distinct class. Maybe I can show you.

G

And you just use like that, is instance command to figure out what kind of classes you.

C

Don't actually even need to do that, it's basically set up under the hood, so it is like the the class name is serialized, and as long as we have the instructions of where the class is, it can be immediately recreated. So.

G

You don't necessarily you shouldn't, have to go through and do this searching so they'll be something like that: a property thing that just those properties that you were specifying under the app property it'll just go through and it'll automatically reset that property accurate. You well.

C

It'll set it so like it'll, be very similar to the mesh, for example, when you're creating an edge curl right. So the very basic information that we need to create a tensor mesh is the tensor dimension right, and so, when you save the mesh, that's the only information. That's going to be saved like that and you're working right. So those those are the pieces of information. We need to recreate it.

C

So then, when you recreate it, those are the only things that it knows about, but then because we have the set of instructions attached to the class about how to create the edge curl. If it's not there, when you call it edge, curl it'll get cream. So it's very much just like. So we, when you initially set up a class, it's going to behave exactly.

I

What's gonna happen like I, let's say I, that's a problem.

K

Convinced everything.

I

Plugged inserted it nothing like there if I.

F

I

That, what's what's gonna happen like a kind of like a go all the way down and sterilize it yep and then put it into one jason and.

C

So if we have problems with space or like, if things are very large, we can specify how its serializes, so we may end up wanting to do something a little more clever and like compressed data files and things like that, but that can all be taken care of in the same way. So it was kind of nice is like we can. Basically, as long as we can specify the schema and like how what keywords there are that you need to set as a user, then how we compress and store it actually is irrelevant.

C

K

I

First, gender into Jason, not.

C

Necessarily so like we can provide a set of serializing instructions that do that say like hey. This thing should be.

C

This thing should be saved as hdf5 like this thing should be saved as JSON. We can provide.

E

All those instructions.

C

E

Chip is not for us right, yeah. It's a nice solution, yeah well,.

C

Actually, like we can readily serialize to hdf5 or to JSON, like we can just say so, like save as JSON or save at station. Five isn't.

E

Helicase that you would not do like it across degree where you have a JSON file trigger that knows, which you store everything neither I nor I I.

C

Mean we can go in and do both if there's a reason to out of the box, we have the ability to serialize to JSON and to hdf5 like serialize the thing. If there becomes reason to sort of do a hybrid, we can do that.

C

E

If everything is like, save it as a JSON.

E

C

To like, with respect to data files and things like that, we don't necessarily have to resave like us big data file, we can be smart about that in the sense that, like as long as we know, for example, it is in like our iam data and this gos I, like X Y Zed file, and we know we should be using this parser like we see, realize those instructions to say. Look at this file use this function.

C

In that case, your idea, so that's sort of like down the road right now we probably will have to copy it because that's just easier but yeah, but those types of things, because, especially when, if you start getting to very large data sets, we certainly don't want to be generating copies. So that's yeah. Those are all things that are doable.

C

Okay, that's right so because I can show like the.

C

Actually, that may not be anymore.

C

So here, for example, when I serialize.

C

The my simulation I have my mesh generator and then you can see that that's another dictionary, so that's actually a whole nother object that has been serialized underneath.

B

C

Unfortunately, we don't see it wrapped, but like this is all that it looks like right. It's basically just a dictionary.

G

That's why you.

E

Don't want to have to write to that yeah.

E

C

And then so then, in this case, like some of the things that I've been playing around with and don't necessarily have, it figured out yet well. I've got a first pass but like of how to point like this, can either be an array or it can be a pointer to a file and so that it's working like substances so.

I

Look like can I just store that as HD player, yeah.

C

I

Okay, so if that's a two path- okay, I'll choose Jason lectures, HTF I'd, better get structured yep. Did you.

E

E

Yes, how this is called and everything shimmered we're still just need them to rival. Yu-Gi-Oh Mathis, like yeah.

C

Yeah well I'm, right now, I mean like you can work exactly what the same dictionary that you've structured.

G

If those are the set of parameters that you need is first faster yeah.

I

Yeah yeah, we have a function for a basic font like a saving.

G

C

Should be something built into MATLAB.

G

That lets you save yeah.

G

I

Sure, maybe like that in that file could be already I, guess not sure, but.

E

D

I

Sure yeah we just.

D

I

D

Check, yeah not sure pretty straightforward to I, would put this stuff and then run it on a cluster somewhere.

L

Else, yeah yeah, so now I'll see you just yeah. You hear all these impacts. That does it for you yeah and we don't have to write anything for okay, cool yeah.

C

Again, so that's one of the things they did with this cusing repo to you is like wrote, a driver that just takes the serialized objects, is inputs and then runs the simulation. So like money.

D

Right: that's why you basically had a drug or something similar to that foreach yeah.

C

D

C

Actually, all that I needed to run the simulation, so this is getting closer to how we actually want to work with this. All of the stuff below was for it, okay, 2d, but this this is all that I needed. So ideally, that should be like how we should be able to serialize and run. This is a forward simulation of what we should be.

G

Able to do the same thing.

E

C

What this like, there's a number of things- I, really don't like about this right now, because it is it's a weird extra layer of simple but like I, was sort of using it to think through what the simulations you did look like. So it's sort of like these weird classes, strapped on to simpe, to create the way that I wanted to interact with it, and so ideally, this actually should just be how we promise like a a time domain simulation. So this should be core same peg.

C

One of the things I would like to look into too, and this was something I would appreciate. Your guys's opinions on is when you're sending. So it's like two clusters setting up a job. Are you writing, like you're writing a Python script and you're calling Python I.

G

Bet the stomach weight.

C

Good yeah, that's one okay, because um what I think would be nice to do is to be able to call simpe like from the command line, um and so what I mean by that? This is a package. I was reviewing sauce, which is kind.

E

E

Spaced from this yeah.

C

C

So what he's done is created like a structured ya, mol file that then you can just run on the command line. So in this case, what we would be doing is creating the structure JSON file so so like this is how you they're calling their Python code. It's just like you know. The package name simulate this, and so it would be really kind of cool and this interesting to see if we have like simple simulate and then your simulation file or simple game per conversion file. Okay,.

C

Yeah at each to have that command-line option like would that ease things from okay.

K

Okay, that's what's happening is that it is flowing everything yeah, no.

C

I mean it would already have to be installed like you still have to have the right environment. It's just you're, not directly calling Python on.

E

This script in my key advantages, though, because, though we do way that the system interacts with the code, that's dick I didn't do its high level spells of running in the background, rather than.

E

C

The other thing that I think could be nice, too, is that if we do change like simple things in the API, this like we can sort of do all those change itself consistently. Whereas right now, if we change something in the API, you then need to update your jiff tools, the mechanism with which it writes the script that make sense so like if we change um I, don't know something to pay paid so NC.

C

If we change your menses number of cells, you then potentially have to go in and change that when you're writing out when you write your script and how things are assigned, but as long as we remain consistent in how the file is written out, you would only then need to change it in one place, which is writing out the file rather than in both writing out the file and generating the script.

C

Does that make sense.

J

F

Like can you show me, what's that yellow hot distillation yeah.

C

C

So it is like just their parameters. Basically, oh I see, um so it is a effectively a dictionary. Yamo is meant to be like a little more human writable for us. I. Think JSON is better because there's more information that we need to inject than this, like you, don't want to be going in and trying to manually enter time, steps and stuff like that.

G

C

Yeah, so something like that is worth thinking about and just like as you're writing like see, if that is something that makes sense, cuz I, don't necessarily know how to do it offhand, but I don't think it's birthing to do as long as we get the objects right, like that's the biggest thing.

I

E

Just you know every time I.

I

Talk on slide, okay,.

E

And then the other thing is that the drivers.

I

Like I'm good to have that mentioned one being a cannot for simulation, it might be. It's.

C

Not German.

I

Good to know yeah.

C

I

Things to change yeah, oh because Matt.

C

I

Okay, this is in class and that's how I want to serialize it yeah kind of showing that okay I don't know how that's gonna happen. So yeah.

C

I, don't think it's too far out. I need to turn to time beforehand, just to make sure that we can use our date well, whatever yeah, we can try and do that in coming weeks.

E

Okay, you want to sit them on them to get a list of drivers, so you want each each problem to have I guess.

C

Like the simulation should get rid of drivers, if that makes sense, know.

E

What the the part that will take on the JSON and then recreate itself recreated summation, there's a later on no.

C

So the simulation object should recreate itself like, so you still need to like set up the correct namespace. So I guess right now, I would say, continue. Writing scripts. That interact with cynic. That's a good way to go over now.

C

The driver files like when we create I guess like the idea, should be that it should be no harder than if you have a simulation created and we know which model we want to input serialize all of those things. It should be no more than two lines of like DC realize this object and then.

C

So that those steps should be on this impact site, but for now it makes sense that you're doing it on the Digital side, todrick I think so, like we're. Trying to I guess, bring these things closer together, but right now, because there's still a development that needs to happen on impact like as long as that sort of interfaces is fairly clean with how you're writing out the script. Yes, then there won't be any problem like transition.

E

C

Yeah cause like I think we want to be able to do like this right. I mean that should be all that you need to do on the Digital side to run a synthetic inversion. Okay,.

E

That's right, but where, where it's this living, you know so, let's see your name user and I. Give you a JSON file. Yes,.

G

E

G

Yeah, oh that's, fair yeah that should be on the Geo cool site, yeah yeah,.

C

So because, like the cippec sim page should be able to generate all of these things and intake them, yes, but actually, like the dot, run button that should be on the jewelz side. Yes,.

G

Because you want to wait, yeah.

C

Yeah, so jiff tools generates all of these pieces that you need. Yeah simp knows how to ingest those and then, when you say doc, run like chief tools triggers this in thing.