National Energy Research Scientific Computing Center (NERSC) Jupyter Community Workshop June 11-13, 2019, 13 Jun 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: 16. VCDAT 2.0: Data Exploration in the Jupyter Ecosystem

Description

June 13, 2019 Jupyter Community Workshop talk by Carlos Downie and Charles Doutriaux, Lawrence Livermore National Laboratory

A

So we did I'm travel to UI, developer, Lawrence, Livermore, National, Laboratory and I'm going to climate program and I'm here.

B

With character.

A

To the mandible and.

A

So today, history and.

C

A

A context about back and then with so.

C

A

My stone and then tell us with you a table and then we go back in so the history. No, we started with the program with the.

A

Decoration of Mandela quoted one I, can't know.

A

This environment, community, recording and together and we by organizing meeting we've, got meet the data comparing together and that's on the science they're.

A

Evolving with one English another great herbs with iteration near yet, and which Trinity's with me.

A

Originally, together to hold everything in to be so yummy green, exterior, siding and, as you see here, the science.

A

To help visualize to science, and so that's CF, which uses a program on the graviton, is written, the NASA community.

A

We try to believe it also just, and so let me see that it's the extensions and the components are CD image, which is older than I.

A

I understand you mean early rise, emptied over race, and you know then again in class software understand where it is and that's into the overall which in pcs, which is a butyl addition, because.

A

Quickly, so whatever nine and incorporated, we can we party, and this never usually send.

A

In suspense, and also that really extremely pleased to think so, but so wish it, it's totally two-way based.

A

Way to a graphical front-end for Syria for various I love people and.

A

We using service to community to do that for you, and so, as I mentioned before horribly Western, and then we decided okay too much different. Today, wait browser web a that occasionally easier and when we started I remember seeing oh wait: I never keeper and he.

C

A

So we had to go away this in, and so we have a CFO, no, it's okay, but then by Sam goes when I Drive. It sure enough. It is just scratch it and it's engineer it and you have been the freeway because every time there's a five, but you can old nice thing about it. What you want, but you also get back to.

D

A

Line so whatever our vacation we give to the user when it.

A

B

All right good morning, thanks for this opportunity, excited to show you guys what we've been able to get it through in Jupiter lab it's in a total learning experience for me. I started this project just last year and I've actually been pleasantly surprised to how how much potential there really is on this platform. So let me just switch quickly here.

B

Okay, so I'm doing the risky- a live demo here, we'll see, so this is kind of your just normal super lab environment. So you start off with your files here and everything normal, but what you'll see here is an icon. This is our extension Icahn. So if you click on this you'll see some side buttons and if you click load variables, it'll just take you back to the file browser, but it's basically indicating okay load variables. What file do you want to actually open?

B

So we have a cnc file here if I click on this double-click, we get that air that we mentioned earlier this week, and then you pick your kernel so now, they'll just tell you: ok, we're loading, the core modules and voila. So now it read the file, it said: oh there's three variables in there you can actually choose which variables you want to load from the file. You could also subset variables, so we have like you know you could change what axes you want to actually include.

B

We also, if you wanted, you could also rename your variables something else.

B

We'll just leave it at that for now and you load, so what it does is it'll inject the actual code that you need to perform that operation, so in a way it's kind of a middle ground between pure GUI and command-line.

B

So now you can actually by doing this, you did the GUI route, but now you know how to do a command line as well, so it's kind of a very powerful way to teach users initially just how to communicate with the backend API and we have the ability to edit. So you know, if you decided, oh actually, I wanted to change the subset. After the fact you just click update it'll quickly put in the next parameters.

B

You could also choose some graphic methods here, so we have a bunch of graphic types: I'll just do and I so fill, and then we have under that. We have some I still feel graphic methods, so I picked that it injects. It creates a new variable there now I could also choose a template. I'll leave a default for now. If you click plot, it'll quickly do the plot and it will create a sidecar plot right here. So now you have a plot and it was literally just click.

B

Click click no command, no programming needed, but you also have the code right in front of you. If you ever want to change anything or you're just trying to pick it up yourself.

B

The other thing is, you can export it. So I just made a plot. Actually, let's, let's make it a little nicer will do overlay, so I'll switch to ISO line so I'm just going to overlay mode, which means I'm gonna plot. On top of this I'm gonna put some lines. So if I click plot again now we have two plots one over the other and I can export that and I'll just call it test export.

B

So we have it exported and if we go back here, we see it's right here. So you have your PNG file. There's. If you saw the the dialogue, you saw that there's more than PNG, you can do PDF and all these other types as well so turn over lay off plot again. So now I just override and again you see it's it's putting in the code that you would have to do to perform that operation. So it's pretty helpful and if I decide. Oh okay, now I want to start fresh on another notebook.

B

Well, that's fine! It recognizes so I could switch between notebooks. You know remember what I had selected and what I was doing. I can load another file, here's a larger file, so it's gonna load. Those. Now this file has a whole lot of variables. This is the file you can't just directly load, but that's the nice thing about seedot. Is it lets you kind of subset pick the data you want to see. You don't have to know, load the whole file and we have our search here.

B

So we can, you know, specify what variable we're looking for just like that click it and obviously you can pick multiple variables load them all in, so it just loaded all these variables. You can check that pick, which one you actually want to plot. You can delete a variable if you said oh I fir. I didn't actually want to load that one delete. Oh yes, there you go so it just deleted the variable. It's I mean this is simple Python code, but it helps for especially a new user. Just to see.

B

Oh, that's, all I have to do okay, edit again, so you can see all the different axes for this variable I can rename it something else update.

B

So now we actually we load a new variable and we delete the old one with that name. But again it's just kind of click, best of both worlds.

B

That one didn't have much exciting data okay anyway, so you can clear the plot. We obviously there's a lot of room for new features to be added on there and there is still so much potential for this. This is kind of our first actual release and stable, getting it to work with. You know, multiple notebooks and recognizing what variables one thing. Oh yeah, let me show you another thing. So, let's say: okay I want to make my own variable, I'll, just say: CLT and I.

B

Want it to equal CLT times, I run the code, so now it actually recognized Oh a new variable showed up, so it actually keeps trying to that. So now, I could actually edit this new variable, I, created and you'll see the parameters match the other because it's basically a clone, but you can still subset it and so forth.

B

It'll automatically put in the correct code for that now and you could save your variables as well, so if I click save put in tests or actually I'll just do my bar and see save so now we have our variables saved here can open that it'll open like normal, but we'll see that this variable was there there's also some just little hints here saying: oh by the way you already have a variable CLT loaded.

B

So if you load this one you're going to override that one, so we have you know little tips that are just helpful. If you just forgot something, but there you go just did it. So, let's see oh yeah. We also have obviously there's some reference material here. So we have some links here. If you want to look up the actual reference material, we have online it's right under the help, menu and.

B

Let's see you can also not only open da and see files, you could also load XML files, for example, any any file that is recognized by see that as being oh, this can hold this type of data. So if you just load that- and you see it just opens the correct files and it loads it. Based on what you wanted.

B

There's more things to but try and remember what else we can do here: I think the provenance sure. So, let's, let's see I'll make a plot. Oh.

B

Right, it can't pop two different variables here: plot that one okay, so we'll export that and again we'll export it as just so now we could also capture the provenance in the PNG itself. Actually, so, when you export it's saving the test file- and it's right here so now- this is the same type of PNG file on the surface, but in the backend it actually stores important information about the provenance of this variable, or this plot I mean so. You can actually then run one of our scripts that we have in the backend.

B

Eventually we'll put this in the front end as well, but you could actually in your console, you could put into the script to actually yeah, oh yeah. Let me.

B

Actually, I didn't have that part of the demo here. Oh that's what I was doing with a console like that.

D

B

I didn't actually practice that part of the demo. It's just the last-minute addition yeah. If you want you could take it over and do that part. Yes,.

A

It's good as a cool factor: okay, I guess it's here. Just do a new terminal.

A

Making sure we have a good Titan, yeah and I think it's good generate see that netbook and the my sister didn't remember all the options you know, and my next I suggest to you put it mhm yeah think so.

D

Here's action you can.

A

Actually recreate the whole condo environment and everything it's all starting to the metadata, but I'm not going to do all of that I'm just going to say mmm and and.

C

You can give it name.

A

For your pond environment that you want to create, but I think I just say, run or just yeah.

D

Actually, it is just create a notebook.

A

And there's a sweet taste to the time you wanna good, so I just going to open the pink fine and now I've got the test to wipe your notebook and if I click on it I can choose. You know which granted I want to run it with and that's automatically generated. So if I had send my king to anybody and I, never computers, you just get the script and that's how I made bad.

A

Yeah, so in the king we have all the information we have actually the whole Yammer for the corner.

A

Fine, the better one limitation is, if I, do it on a Mac and assigned to you know, Linux is just not being able to generate the Conda, but you could still generate the you can still donate a notebook, though so you know you have to manually, create your own environment, but you will know exactly I mean you can tell you which version of every package was you, so it's just a Yammer, the Yammer for Mac, so we will bar from Conrad.

A

Yeah basically dictionary it as a header somewhere.

B

In slides now, I think that that's the end of our demo live demo. There.

B

Go back to this okay, so so that's where we are now with our 1.0 release. There are definitely as I mentioned before, for improvement things. We want to do up ahead, so one thing is getting some tutorials just to acquaint people with the actual UI just so they know. Oh, this is this: will open a box and now I can use the sliders and so forth. So we have I've already got some prototyping done on that we're thinking of using some Joyride tutorials.

B

Those point-and-click kind of things point out two features and you click on them. So that's something we want to do and then we'll put a little link under the help. Menu.

B

Also just allow you to actually get a dialog for the files, so you can actually select the NC files and not have to see all these other files that aren't related kind of thing, letting you save to different places like a local file system, also incorporating some color maps, so our previous 1.0 version had GUI, for you know, making your own custom color maps and modifying your graphics method with that.

B

So we'd like to you know, bring that functionality back as well in our 2.0, so we're definitely looking at a lot of new ways and its really promising to see that the community itself is just creating so many new tools. You know to actually make a lot of these things, not only possible, but actually not that you know you don't have to reinvent the wheel kind of thing. A lot of things are actually already being done.

B

Add some average standard deviation and weighted average options when you're loading your variable right. Now we let you subset it and choose things on the axis, but there's more that we actually can do behind the code. So we just need to update the UI to. Let me do that as well, and then obviously get user feedback find out what it is that you know. Other helpful features of people are looking for, especially when they're working with our tools, data analysis tools.

B

Well, what more would they like to see any shortcuts or just helpful features that they can themselves using a lot and then for the community overall, so this is. This is kind of like one thing we're working towards, but the earth system grid Federation itself has a lot of front-end tools that separate teams and groups are developing they're, all very cool and diverse, but they're, not they're, not all kind of integrated they're not unified in a way.

B

So if you switch to a different tool, you know it's a whole different UI and all that we feel like the Jupiter ecosystem, and the super lab platform gives you this opportunity to actually kind of put a lot of things together, unify this group of applications in a way that would just be you know you can have like a climate data analysis, suite of tools and so VC. That would be one part for the visualization, but we're thinking you know you can have like a search download app.

B

That would you know let you download data from Globus and or receive a C wte there. We have several different applications and we feel like they could all inter communicate and also connect on this platform through a front-end way.

B

That's one of the great things about Jupiter lab is just it has that extension mindset where you know you can create these powerful extensions that they they allow you to do so much it's not. You can basically become almost like part of the architecture. End I, think that's just a very forward-thinking decision on the architecture and I think there's definitely room for us to take advantage of that. Those possibilities also like right now.

B

Our app mainly works with Python, but there's other languages that the backend you code, you could actually write so it'd be nice to allow other types of you know kernels and a lot of room for like earlier this week. We saw a lot of presentations on improvements that are being worked on and extensions for, like file streaming, for instance, to get rid of what we were doing there. That gives you that little error message that can be corrected, there's a lot of room for all of that stuff.

B

So if you want to try yourself out, there's a link, we have a github page and with that, are there any questions.

B

We could actually some of the earlier code that I had would actually rewrite it, but we figured for now. We would just go back to appending in certain cases. I know the capabilities there and I've used it. So I think you know in the future. If we find a need for. Let's say we just want if they click this button, it'll just rewrite that cell instead of create a new cell. That is actually a possibility. So there's definitely we could definitely do that. For now we just decided okay.

B

Well, they want to open a file, for instance, we'll just open it and then close it I used to an earlier version. It would actually create a header cell where it would open the files there and leave and rewrite it there, but we decided to kind of move away there because well what does they actually wanted to be closed because they're gonna modify it later or there's things like that? So we consider that to.

B

Yeah yeah so early on, so this was kinda. We kind of brushed over this, but when we were trying to get a demo out, we kind of just needed it to work, so we mashed it all up.

B

But since the release actually I spent- probably the last month or so not only cleaning up the code and adding features and making sure it's stable, but I also worked on modular Eisen it more so, for instance, all of our code injection commands are put in a separate file for Python commands and then I have a separate utility library that just has some really simple functions.

B

That will, you know tell the duper kernel Oh, send a request for this code, get back the results, so we have functions that could easily just be implemented and we have a code injector library which it used to be. You know the different components. You click the button and it'll. Do the injection I moved all of those commands into a separate module so that now, if you ever want to add a new injection command or remove it's all in one place? So it's definitely and it's pretty easy to add.

A

C

B

So if you would see me create a new variable like a equals five that one shot, it's only going to show the variables that actually are CDMS variables have the you know have right and we yeah that way. Your drop-down doesn't get like flooded with non important variables. Yeah.