Jupyter Community, 25 Jan 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Jupyter Community Call - January 25, 2022

Description

Recording from the Jupyter Community Call in January 2022.

The notes from this call can be found here: https://docs.jupyter.org/en/latest/community/community-call-notes/2022-january.html

Read more about these calls on Discourse: https://discourse.jupyter.org/t/jupyter-community-calls/668

A

Here we go yes, oh and I need to change my view. That's right! Okay, cool cool! Here we go hello, everyone and welcome to our january 2022, jupiter community call. This is the first one of the year, I'm cheering for all of you, because y'all are being polite and being on mute, but I'm seeing some cheers in the background, so excitement for that.

A

A few things before we get started. My usual things. In fact this is a jupiter community gathering, so we are all held to the jupiter code of conduct, while you're here that can be found at absolutely any time at jupiter.org conduct and additionally, I am recording this call. It will be posted publicly whatever that means for you, there's your heads up and from there on out. I think we're good to get started. If anyone hasn't been here before or just needs a refresher.

A

What we're going to do is follow the agenda that I've linked in the chat, so we're just going to go through short reports, then longer shares and with probably plenty of time for discussion, but I am so excited. We have lots of things on the agenda today. So yes, and that's what we're going to do. The first thing we have is a shout out from zach.

A

Would you like to take it away? Zach yeah.

B

Hey everybody zack sailor here, my shout out is to isabella because we're starting a new year and she just ran the community- calls for a whole year. This last 2021- uh and these are awesome. I know I ran these for six months and they are horrifying because you never know people are gonna show up, and so congratulations of a great year. Thank you for thank you for running these. They you do such an amazing job.

A

Thank you, that's really sweet, and that means a lot coming from you. I appreciate it and yeah tony thanks for linking the blog post. uh Tony mostly wrote this, but it was a really good discussion of kind of some of the things that we did to make it consistent, because I think that's a struggle with any meeting. Not just community calls, so he did a really good job summarizing that and then I added bullet points so yeah.

A

That is the only thing currently on the short reports, but did that inspire anyone else to celebrate something? A new year is a great time to celebrate.

C

Oh, what let's see how red we can get isabella? I want to give a shout out to isabella for organizing the accessibility workshops the past two weeks. um This weekend we probably got like nearly a dozen uh contributors in on documentation. We had a great talk uh the week prior, so there'll be some assets coming out from that. So another shout out to isabella happy new year.

A

Thanks yeah, that was another set of events. um I'm glad that the first two are done. We do have two more coming up in march, just as a little follow-up for that, but yeah it was. It was awesome. Those pr's will be in the works at the end of this week, because we're giving contributors time to polish up if they want outside of the rushing sprint environment.

A

I yeah actually, okay, no first any more celebrations, shout outs, quick things: we had a release on this project and I'm super proud of it. Those kind of things hi.

D

Everyone um I want to shout out to zach for being an early adopter of the new governance process and setting up the uh jupiter server team. We're all really excited to be part of that. So thanks, zach you're doing a great job leading us into the future. That's it.

A

C

I think we got to give a shout out to darion for going around and making sure that everybody's aware of the new governance process changes. He has to be in so many meetings voluntarily during the week that god bless his soul for doing that, um and we hope to follow suit, with what zach's done very soon.

A

Yeah I agree: darian has done a great, at least in meetings. I've been in done a great job, pulling apart the document into steps to take. I really do appreciate it.

C

Do we say congrats to the binder team for not letting binder fall over too.

A

If you want to, I mean, I think, that's something.

C

That happened this month, uh congrats to binder for keeping that alive. I don't know what I'd do without that, but uh good free offer cert not all of those credits. Oh no.

E

Yeah, that was a big shout out to chris holtgraph for um organizing at least a set of short-term credits and another one min who, like works on the technical side, to reduce the monthly cost, so that short term credits are actually viable.

A

Yeah, that's, I know so many people that use that so very grateful that that kept running. Thank you all for your various tech and money. Gymnastics is what I'm hearing more celebrations. We still have time.

A

B

I'll call out a tweet from uh chris holtgraph. He showed a pretty sweet demo of uh or jiff. If I don't know how to pronounce it of, uh I think a missed parser in jupiter lab notebooks, which I thought was really cool. I hope chris is.

F

Here, to show us that hi.

G

Oh, don't you worry.

B

G

C

G

Writing preview.

C

You've been converting code into python from javascript, and now you finally have to write javascript congratulations on going full circle. Chris.

H

I know it's a a snake eating his tail kind of situation. Isn't it scary.

A

Wonderful, I will be quiet for a few more seconds. I love all these celebrations. It's a great way. I don't know about y'all, it's a great way to start my here, yeah.

C

One last one: let's celebrate everybody being here and then we move on. Yes, thank you all for being here.

A

Right, that's true. Zach made the point that it's hard to keep these running, but they really wouldn't be runnable without all of you as absolutely cliche and cheesy as it sounds, it's very very true. um If I just sat here alone, it wouldn't really be a community call, so I appreciate it and great. Oh those were so wonderful. I'm definitely going to revisit this recording start later.

A

Thank you, everyone for the for the joy and with that we will take it away with all the agenda items we have a couple today, which always makes me happy. I think the first one is I apologize, but I so tell me how to pronounce your name properly. Jitendra.

I

Yeah, that's that's right. Okay,.

A

Are you ready to share.

I

Yeah, so uh how much time I got.

A

We have for sure two things on the agenda. I'm checking to see there's one person who said they might be on the call, but I'm not seeing them here. So I I think you have at least 15 minutes.

C

Let's try and keep it at. Let's.

A

Shoot for presentation at 10.

C

And then questions oh with a little extra and we'll see if we can make some time.

I

Yeah good all right I'll share my screen.

I

All right uh so uh here I'm. I want to talk about uh okay, so by the way I'm jitendra pandey. um So we have this small company called infant store. It's a startup in the ml ops space, uh I'm a co-founder and uh cto, and our ceo is also on the college again. uh So so we recently built this ml flow kernel. uh This is basically a an extension of the ipython kernel uh and the the motivation is that to have an integration with ml flow, which is becoming very popular in the mlaps space.

I

To make sure that when you, whatever activity you are doing in jupiter, gets recorded in the ml flow in a in in a very seamless way, so ml flow has integrations with lot of libraries like uh tensorflow or xc boost or or in fact most of them, and uh and when you you can enable auto logging, it will lock certain parameters and and matrix and all and whatever you want to record.

I

But there are certain additional things like okay in the cell, whatever output cell produced or what was the exact code that was executed in the cell, uh those things are not captured in an auto log. So for that we need, uh we need a kernel that can actually intercept and and send it to ml flow for uh for recording.

I

uh So that is what I want to present today and I will show you a short demo, uh so so ml flow kernel can record uh all the cell outputs which includes like, if you have any graphs or images that you are using to visualize your uh uh your data, all that is captured and and recorded into ml as an ml flow artifact, I'm hoping that you guys are somewhat familiar with ml flow, but I will show you an interface so that you have uh a mental image image of what I'm talking about so uh and and also the organization of of your jupiter notebook execution.

I

Like typically, you start a kernel and then you have uh you executed cell by cell. uh So so all that is captured. uh You know in a very well organized fashion, like uh the duration of a kernel uh session uh is recorded in a parent run and all the cell executions within that are recorded as child runs uh and, as I mentioned, that it integrates with the auto logging.

I

So so all your like, like, if you are say, recording the model in the nml flow or you are recording matrix so that all is captured in the same place, along with your cell outputs.

I

So if time permits, I will show you a demo of the deployment, otherwise I'll come back to this slide and then show you the steps, um and in short, this is my contact. So if any time you can shoot me a message all right, so, let's head back to the demo part all right. So uh I recorded this so that I'm in control of the time and also because training it looks, uh takes a little longer so to make sure it fits within the time frame. So this is a typical notebook.

I

I got this code from from kaggle. uh Essentially, it's a lan facial landmark detection.

I

Basically, you have a face, and and this code uh can trains a model to detect the location of the nose or eyes or a lips on the face, it uses a convolutional network and, and it has a hand, annotated training data which I will visualize shortly.

I

So, uh as I mentioned yeah, this is uh implemented in keras, a very simple model, and I don't claim that it's the best model out there, but uh but it does a decent job.

I

uh So, like there are a few cells, we load the data, we do some analysis and uh some certain augmentations and and at the end of it yes split into training and test and perform the training here so and in here like during the training phase, we also enable auto lock to showcase that uh the demo flow uh behind the scene will capture everything along with the parameters that the code wants to launch.

I

But you don't have to worry about starting a run or stopping or anything all that is taken care of and all your code is also preserved. So we first pick the kernel, so ml flow kernel is an extension of ipython kernel, so actual execution is all happening in ipython.

I

uh We are just a layer that captures and by the way this code is all in open source. Github link is also available, so this is our inter face for our ml flow. This is ml flows, free service that infant store offers. So you can sign up anytime.

I

I will give you a link for that also, and this ml flow kernel in this uh notebook is already configured to talk to that ml flow service. So we have started a run. First, one will take a few seconds. It has to connect to the ml flow service and and start the runs, and now we are loading the data and and it will also produce some output and then we do some further analysis and and a little bit of augmentation of the data, so so yeah. So this uh this execution.

I

Now here we are visualizing the data. We can see that what data looks like in the data. We have some hand annotations of the uh of the location of the landmarks like eyes or nose tips. So now, let's see what we have achieved in the ml flow, so here ml flow shows that we got run started.

I

There are four cells executed, so we have four individual runs already captured here and if you click on the last one, you see that the cell output is captured. The code is captured, so so this now this record is forever. You, if you go and modify rerun it that will be a new run, so you can always go back to what was an and even your visualization history is available so that you can go back in time and see that okay, what really happened in in in that cell.

I

Yeah so now we go back, go to the training, so here we will first do the splitting of testing and training data and uh and then we launch the training.

I

So training will take a few seconds um actually in real. It takes a little longer, but I have expedited it and using only one epoch, so the purpose was to show the ml flow integration part, not really the accuracy of the model.

I

So, okay, so training done now. It is logging the model to ml flow. It is uh also going to the same ml flow service and um and that plumbing is taken care of and in the end we do a little bit of I mean prediction on certain um images so um and the prediction output uh is, is also displayed as an image, and- and this is what data scientists typically do. They look visualize the data, and now we have.

I

uh We can see that okay, so we got logged out, uh but we can log in back.

I

This is uh ml flow service, it is secure, so password authenticated and all everything encrypted.

I

So in here the last, but one run had the model training. So in this one we can see that we have captured a lot of information, so some some of the information has come from auto logging. Some of the information has come from logging, the model itself, and so here is the code, so we keep tweaking the models, but at every step, this is how we preserve the code and any standard output if the user is interested is also captured or and and all the details of the model are captured, including the dependencies.

I

So I mean, if there are a lot of parameters to be tuned and and capturing those parameters is, uh is always helpful and everything is going into the same run. So the record of everything is in one place.

I

uh The matrix are also captured depending on how the uh framework has logged the matrix, and we can also look at the maybe model summary that, okay, how am I yes.

I

Okay, I've talked about the matrix already so so yeah, so this is basically ml flow, integrated uh uh with a jupiter kernel and uh and okay. So this is the the visualization of the model output. This is very key because uh user data scientist wants to go back in time. Okay, how my model did and look at it visually if you had plotted some graphs here, although that will be visible, those images will be available here to always review how what your model did in past.

I

So in here yeah, so jupiter notebook execution is done yeah. One last thing that I wanted to show was uh for those who are not that familiar with ml flow, that you can manage your model here also because now, if you register this model, it will be registered in the ml flow service.

I

We can store this model as a as a version of an existing model or create a new model and go to that version and manage the life cycle of the model like you can designate it as a in production or staging, or it is an archived model. So, typically, we create a new version and old versions are archived.

I

We have all the permission controls here. Also, like you, can authorize people to access this model.

I

And uh yeah: this is how we transition the state, and this is linked to the run, so you can always click on the run and go back to and see that what was the code inside what was the and all the outputs that I showed previously all right. So this uh brings me to the end of uh this uh presentation uh for the deployment I have an another short clip. I can show that, how do we really deploy it and the login and authorization part as well, but any questions here? I would love to answer.

A

I actually have one if no one else does okay, so maybe I'm missing something, but I wanted to ask you. It sounds like a lot of this is for recording right all the parameters that got you the result that it did, but then, if you like, if you wanted to reproduce that, is that something you'd have to do manually, recreating all these parameters, or is that kind of what you were talking about with the register model that that creates some way for you to automatically set that up again.

I

Okay, so the integration of ml flow kernel is with the notebook. So every time you execute the notebook cells, it is captured. So if you are reproducing suppose you want to go back in time and reproduce something, and at that time it was a different code, then there is will be a step that you go to that run. Take that code and paste it into the jupiter notebook, but that's a good idea. Maybe that could be a feature for the future that we somehow uh provide an easy way to click and and read and pick.

J

A notebook, maybe stitch a notebook from all the different child runs very interesting. Actually.

I

And uh actually I mean as a company, we have a compute feature in our product where we capture these notebook code as what we call as transformations and there you have the ability to actually go back to a transformation and click a button, and it will execute, but uh probably some other data will showcase. It will be a longer presentation so, but yeah throw questions at me feel free, interrupt me anytime. I just want to show how to deploy it uh because visually it will be a lot uh more interesting.

I

So this is our um and I want to show particularly like how we deploy it with our free infant store service. So this is how you sign up for the service.

I

um It's a one small page: you have, you need to provide an artifact location where all the output will be stored and you can specify an im role for authentication and authorization, but I have already created this account here and in which we will do the logging uh so and all this ui is a standard ml flow, but our we have a more scalable enterprise backend, so you can create a new experiment or use an existing experiment when you are configuring, your ml flow kernel. So let me create a new experiment so that.

C

Does your kernel, let folks do all of this programmatically.

I

uh So this is uh at the uh so there are two pieces to it. One is the m, ms, the kernel itself. Another is the ml flow service. So um so what I'm showing right now is ml flow service which you'll have to set up, and once you install the kernel yeah then rest of it is will be automated okay.

I

So so we have it's a pi pi package. We have installed ml flow kernel right now that version number 1 10, but uh the version numbers are graphically changing. uh So second step is to actually add this to the kernel: spec uh kernel, spec, okay, so right now we have python kernel, but, uh let's add ml flow kernel in the spec and this another step will create the add that to this pic and- and that is it- uh we have to now put some configuration, so configuration location is in the dart jupiter directory itself.

I

It's a json file and we need only a couple of a few parameters there. uh So we need a ml flow tracking url, basically the and it could be infinite stores, semiflow service or any other ml flow service. If you have, you can use anything there and debug enabled is useful and- and you specify your experiment name, so we just now created this phase research. So let's put that there so that all the runs are created in the in that rain. And now we start the jupiter lab.

J

So the really cool thing about this, and the first thing that occurred to me- was often data scientists go back to the same cell, tweak something and read on it and tweak something and read on it. Every one of that is captured, so you can really go back and look at it and say: okay, I you know fiddling with the fiddling with that here's. What I really found useful um that I find very valuable.

J

I mean the notebook itself is a pretty awesome structure and it saves the final state of what you end up with, but this uh detainer as invention, basically records every activity you take, and I find that really cool.

I

So uh so we selected ml flow kernel, because now it was in the kernel spec. So so, when you were doing it for the first time with ml flow service, you will need to log in so it will prompt you for the this small piece of code here, which you just need to log in we can. We will use the same account password that I showed on the ui that you will create when you are signing up for the service.

I

If you have your own ml flow service, which does not need this step, then this won't be prompted, and so once the login is done, you don't have to do it again. It will be automatically refreshed. We can get rid of that login code and we restart the kernel now at this point it is ready to use we again and the rest of it is what I showed in my previous demo.

I

So now we start the run and we go back to cml flow and we should see a run in the in this new experiment that we created.

I

And you see that it was a just an import code, but the code is captured and everything is expected.

I

So that brings me to the end of my presentation. Thank you so much uh but feel free uh shoot me a question anytime or.

J

There's a question in the chat: if all the felt sources python would it be possible to implement these features as a shell extension instead of kernels.

I

Okay, so uh okay, that's yeah! That should be possible because it's a standard kernel, so all the a kernel apis are uh supported. uh I have not played with that, but uh um but the ml flow kernel is created in a implemented in a is just an extension of ipython whatever it does not understand it, delegates to to ipython kernel and just hooks up with the ml flow service in the middle. So that should be technically possible.

I

All right, yeah, one last thing to mention that we do have sage maker integration, also, if some users are using sage maker, the steps are available on github and also on our website or just drop me a note, and we will help you set up.

I

We have created an image, also image with ml flow kernel, enabled which you can actually directly use uh to to get your size maker, uh studio up and running uh with ml flow kernel, um but if you are using just stock jupyter from open source, those steps which are also highlighted here, will completely suffice.

A

Wonderful, any more questions.

A

Okay, then well, thank you for such a in-depth share. I always like to see what people are working on and saving. All these different versions seems like it could be really helpful right. Recording is the first step for knowing what you're doing so. Thank you so much.

I

A

And then, next on our agenda, if I can read the agenda, is we have demo, I'm assuming? This is from chris based on what happened during the announcements earlier, the demo of hey yeah.

H

Okay, yes, why not yeah I've just been there messing around with them. um Let me share my screen.

C

Did the naas folks not show up.

A

I was gonna check it then, but I don't see the person who said that they would be here. So I was just going to check again, but if that's not true, let me know at any point. Sorry thank you, chris.

A

That's not the right screen.

H

Or is it I'm confused yeah, that's the one.

H

No yeah, you can see uh the uh vs code. Yes, that's the name scene, okay, cool yeah. I thought I'd give a quick demo. I've just been uh throwing together a bunch of stuff over the last few months uh in the acceptable works worlds um so trying to pull it all together here so um yeah.

H

These are the the kind of six things in my time to fill the rest of the space time uh is: writing notebooks with this text based markdown format, um then how you can use that with vs code, how you can also use that in due to lab, and also some extra things of how you can execute these notebooks and feed them into uh creative documentation with mrnb and jupiter book.

H

So mist is essentially is this format that we put together over executable books projects designed to extend the common mark, which is the common markdown specification with more of advanced features for writing: richer documentation, scientific articles under like admonitions, figures, tables, etc. It's what we use within jupiter book to generate this, these documentation, the books and websites.

H

So we want rich documentation, uh but also we'd like to be able to uh hook into jupiter, as the name suggests and execute some code. um So what is missed? One of the main things it adds is these role syntax and the directive syntax.

H

So what it looks like it looks like when you write markdown, you have these kind of code, inline code blocks and before it now, you have a name essentially of how it's going to be interpreted and the same for um admonitions, and you see they're all nicely syntax highlighted here in vs code. You can hover over them. You can also complete.

H

Based on ones that are available- and this is all within this markdown extension- and also if you click on the preview you'll get, some of them actually are done, because it's a older version, what it turns these admonitions in to uh their kind of final format, their uh their rendered format. So you see here we have a note. We have a note nested. In a note, we have a figure here with the caption, uh this list, format for um tables etc.

H

Now also, as well as being an extended markdown format, this document is also a notebook um in a slightly different format. So if you look at the top here, you might have noticed it has all this uh metadata this front letter- and this tells dupa text which you might have heard, of how to convert this into a notebook um and so you'll see, for instance, these plus this plus syntax tells g by text that this is starting a new markdown cell, and then we have code cells and we have raw cells.

H

So if we now go over to tubes lab with the dupy text extension installed, we can open up this markdown as a notebook and we get a lovely notebook and also with the jupiter lab mist extension.

H

We also have all of our nice rendering of all the markdown cells. So you know you can change this. They look like that and yeah. When you execute it, you get all of your rendering of all your html and so obviously very much like how the the end product of your book will look like, and here, for instance, yeah we've got the cell tags that it's captured and we have our. uh We have our code sales to run.

H

So uh that's yeah, that's the first step really. um So this is all built on uh mark.net, which is a markdown parser, that's very extensible, and within that we've written a number of plug-in extensions and, and they all run together. So you get this consistent um approach over vs codes and over jupiter lab. Also we're collaborating with some other guys on curve note, which is uh another kind of scientific writing platform that also uses miss markdown.

H

um So yeah. That's uh that's all nice, so it's nice now that you can properly see what you're actually writing with this miss markdown. It doesn't just look like raw text. um You can properly have all of these nice uh rolls and directives.

H

Then, um oh quickly, the other thing. So with these with these uh markdown uh text-based formats, um kind of one of the problems is obviously that they don't store the code cell outputs. um Now this is nice.

H

These you know the text-based formats are nice for for saving and give having on github and seeing all of you or having uh nice gifts and things and having a nice uh editor experience, but they're not yet great for that. For.

H

For saving the notebooks, uh so this is where jupyter cache comes in, and the idea of jupyter cache, which is integrated into tube to book and missed mv, is that you can have a project. So you have all of your notebooks within the project and it knows how to read them and what it does. Is it essentially, it caches the execution.

H

So after you've executed your notebooks, it stores them within this cache. It stores all the outputs and one of the nice things about it as well, is that it knows whether a code has changed and it needs to be re-executed or it's just markdown which doesn't need to be executed. So if I type in something here, you'll see that when I list it, it's still saying it's still: nice, green and saying: okay, that's fine! If I change this, then it's now saying ah I've no longer got that notebook cached, because it's got a different.

H

It will have a different execution. So if I execute the project now it'll look down what notebooks need to be executed um and execute only them once and catch them again. So this is a way that we can use to have these. Have this these markdown format, these text-based format, executions, or you can even run this with notebooks. If you have lots of notebooks and you want to know, I want to keep them up today.

H

I want to make sure they're all executed, but I don't have to open them all and execute them separately and things, and so this is how we do this for jupiter cache, and the last thing I wanted to just touch on is the kind of how do you? How do you take these notebooks?

H

So we have these kind of preview formats uh within jupiter, lab and vs code, and we also want to eventually create a final kind of representation, either in html or latex or any other format, so with uh mrnb, what I've been working on is kind of having a simpler being able to have a simpler standalone cli for converting them to other formats, let's say like html or lartek.

H

um This is somewhat so. It's kind of similar to uh mb convert which you may well have heard of, um but it's uh different to nba, convert it hooks into docutils, which is the underlying package behind sphinx and things like this, so you get these really rich features for creating documentation. So, for instance, if I run if I do this, it's firstly going to execute the notebook and it's going to set it up, and then it's converting it to this.

H

This docutel's asked format where it knows what to do with it or you know we can run this straight with html and get out our html output.

H

um So again, I haven't got to link it into jupiter cash at the moment, but you can also do that. So we get out out so we get it out that way um cool. I think. That's the main things I wanted to say, there's a quick rundown of what we've been playing around with any questions.

A

I not a question but just a comment. Maybe it was just me when you opened the something with a file extension.md and it showed the notebook icon. That was a real ride. For me. That made me laugh really yeah.

H

I think yeah it just uh this is all yeah, you'll, say good work by um jupytex and being able to do this, and you can have these linked. You can have yeah, you can have synced, notebooks and markdown formats and and all this good stuff, um but yeah certainly so tell you to go and check out check out that project.

C

So I'm curious chris how if um angus's uh jupiter markup work uh informed any of this stuff yeah for you and if there's something.

H

Yeah yeah, yes, definitely give a shout out to you.

C

I said I would give a.

H

Shout out to him um yeah, so this all builds on um agus. I did tell you, I was doing this for him to pop him, but uh I don't think he's here. um This builds on angus's uh mark down there, um jupiter lab markup.

C

uh Tool, so is a dependency of your project or are you go? Are you? Is it in jupiter lab markup.

H

uh No, so what jupiter lab markup is um does very much actually what vs codes um markdown implementation does, which allows which starts up a this markdown it for um markdown passer, creates a kind of instance of that and then allows you to add plugins to it yeah. So it's how you, how you add tables and roles and these directives and everything like this, um so we plug into that. So essentially, we take uh angus's, jubilant markup and we say here's some extra plugins for you.

H

Here's uh which say how to pass them and also you know how to convert them out into html and with the css and everything.

C

Very cool man yeah the jupiter lab markup extension's, like one of the best extent like I feel like everybody, should just have it, um and also github on their roadmap, uh has um like mermaid diagrams and some of the features yeah that are in jupyter lab markup, but between jupiter lab markup and what you've demoed here like it's. We it's starting to really see like a really rocking document, editing platform, encoding.

H

Yeah, exactly that's the that's the hope, yeah that you you get all of this lovely. You know the kernels and everything you get jupiter, but also this richer, markdown, environment and yeah. There's a lot of talk over in the this, this jupiter lab markup about how we um can extend that and improve that with, like syntax, highlighting and also kind of the lsp features.

H

As I say, within the vs code, I've started to add these kind of lsb features where you get like the auto completion and the hover over to to be able to write these, um that's not currently within the jupiter lab extension. By certainly like to look into that. There's the jupiter lab lsp, so we're hoping to kind of work with them with those guys to do that as well.

I

So, uh chris, a question on uh the cash. uh This is for preserving the output uh of the of yeah and the state of the notebook right, which includes the outputs or anything that your code produces. Is that yes,.

H

Yes, so the idea is that it essentially hashes your notebook, um looks at just uh kind of ignores any of the markdown and just looks at the code and- and you know the the the uh the metadata for the colonel and things like that and says, has any of that changed since I last executed.

G

H

If it doesn't, then that's fine, it says: okay, here's uh and links that so hash is your notebook in terms of the code cells and the code metadata and then links that to something that it's already executed um and if not, then it has the mechanisms in there via mb client, which is another good project that people are working on, which is essentially taking what was in mv, convert and kind of making its own standalone projects and how you execute notebooks outside of jupiter lab or jupiter notebook just by the cli and things but yeah.

H

That's the idea is that it just has a big. It has a cache of everything. Let's get excuse it.

I

Yeah, actually, I noticed the parallels that, because I was talking about recording everything in ml flow kernel with versioned outputs like every step, so jupiter cache is something that records locally and, uh and it has some additional features, um but uh but if it is integrated with cml flow in the back end, then actually we could have a version recording. We can go back to previous versions, but I mean there will be some integration work needed.

H

I'd certainly be interested to hear from that I say: yeah. Our main focus on call of duty cash was obviously how can you yeah? How can you create books and things where you don't want to every time you change a line of uh a line of your documentation. It has, to you know, re-execute every uh everything every time, so it's that kind of that kind of thing that we're after, but it can certainly be extended to things like this. Like version.

A

I believe zach raised his hand.

B

Yeah um this is super sweet chris thanks for for doing this demo. um This is lift up to your expectation.

J

B

For sure, for sure um I wanted to this is more of a comment. The um a couple summers ago. Isabella knows this all too well. Some of the interns at cal poly worked on a front-end extension to basically provide like a. What you see is what you get a wysiwyg editor in markdown cells in jupiter lab um one of the things that one of the challenges of that was like when you wanted to do something more complex. Like add a note, or something like that, or something.

G

B

The standard markdown, like you were limited by what markdown could do. I see this as being like a way way more powerful for something like that. Where you can have. um You know you have a markup set of of syntax, that you can build a uh an editor that allows you to just automatically populate that down into the markdown, but show it as a wysiwyg um and yeah.

H

Certainly, for.

B

H

Oh yeah- and I say I was just uh going on to this- I'd- certainly give a shout out to the uh the curve note folks that we're working with um quite a bit um they're working on this. uh They say the curve note, which has these more kind of, I guess- richer. Almost wysiwyg features where I guess you are more.

H

We with with the jupiter lab, missed extension, obviously you're writing within standard markdown. Then, once you execute kind of render, then you click. You know your shift uh tab or whatever it shift enter to x, render the cell um the curve know folks, are working on this more kind of integrated way of doing it as well, um where you are doing that. The full kind of what you see is what you get approach, um but again it's all. The idea is that it all kind of flows from the same uh base code.

H

The bottom end we reuse as much as possible.

C

I love this. The most inclusive future is that everybody, just programs in markdown, no python versus java wars, anymore. Everybody, just programs and markdown to write a program is the story. It'll be great.

H

That's the dream.

H

Cool well I'll, uh stop sharing.

A

Yeah, that was those were both really cool. That was interesting too, seeing a little bit of synchronicity at the end there, with some related tools we yeah. Hopefully we can continue that yeah no right, but that makes me so happy right that we can it's. Arguably one of the points of community calls so just nice to see it happen, but yeah.

H

I see uh sorry, I see nick uh sorry, bold, I think, is it.

J

Yeah he's, obviously I was saying.

G

H

Jupiter um lsp and things it's the big proponent of that I guess.

A

Yeah, I was also just skimming the chat to make sure we didn't miss any questions but yeah. We definitely have a few minutes left for any more thoughts on either share or final, exciting things and I'll be quiet for a few moments for y'all to stew. On that.

K

Yeah I mean we don't. There's there's been a lot of discussion over time about a bunch of these features, and um you know it gets to the end of it of who decides what is the markdown and if we're just some other folks, you know it's not a reference implementation in perl, but it's these reference implementations in these two environments, it's inclusive, if you're in the game right- uh and I just I want to make sure that we're not creating.

K

You know some new thing. That is now this other language specification and you got to be running these six versions of you know jupiter text and all this other stuff- and I don't know it's if there is not a way to describe what these extensions are, that you're having then you're gonna run into all these same problems that we have with magics inside of python and magics inside of derivative kernels and stuff like that, um so I hope at some point we do have a jupiter lab markup.

K

You know that is or a jupiter-wide markup that is described in such a way that we can use it in a bunch of places confidently and if they change migrate them. You know everybody poo poo's on the notebook format, but gosh darn it. If that thing doesn't take care of itself right, it's one of the few self-healing formats out there. I just don't want to lose that, for you know, whatever gains we're getting also on the uh on the ml flow one, have you guys looked into kernel uh the enterprise kernel gateway?

K

It seems like if you wrapped it there, where you can intercept all of the uh all of the kernel messages. Then you could instantly have this, for you know every kernel and it wouldn't just be python. You could have the scallop kernels and you could have the spark kernels and you could have the julia kernels.

I

Oh yeah, okay, that's a good point yeah I will. uh I will look into it. I did some experimentations with.

I

We want to be able to basically somehow want uh work with different kernels, particularly not for the reason of different languages, but for like different kind of hardware. If you have different kernels now, one running in with gpus one running on some cpus and have some ability to uh multi multiplex between them uh that kind of uh so directions we want to. We are thinking of, but that makes a lot of sense. uh We can have an ml flow kernel for all the languages. That will be interesting.

A

That sounds really cool actually.

H

I can't say that.

H

Yeah, no, I just uh no I'm as we speak, I'm working on a kind of mist we're trying to work in like a missed specification and things, and these kind of things to make things more uh regimented. I guess more standardized.

C

Quarto did a good job of that right um where they took the common mark spec, and then they wrote their own specification over top of it or you go into like that level of detail. Chris.

H

um So I've just I've started: writing this a format, uh unist uh and then there's m dust on top of it. It's essentially just a json format for common mark extensible, json format um uh that remark uses is a jupiter package. um But that's what I'm writing, because I think that's that yeah. It basically has what we need.

H

It has it's json for jason ball, so it's yeah language, agnostic and it captures all the information like position and things like of all the syntax stuff that we need for like lsp stuff like where is something in the document and things. um So that's the idea to kind of build on that that has the common mark and then they also.

G

I mean they have they're already like a common mark.

H

And then a github flavored, markdown extension over that and then yeah. The idea is a missed one.

C

Does our markdown do something like this, because it seems you guys are mimicking our markdown syntax in some way right.

H

um I mean obviously the yeah- I guess I mean they certainly read across um it was I mean. Obviously it was more coming from the level of how of how can we get all the features of restructured text really, rather than looking at? I think markdown specifically, but they're very similar, obviously concerns of how do we get these kind of extensibility.

G

H

Been marked down, I guess of the directives and the roles and things so um certainly been looking at it, but yeah there's there's a lot of read across in all of these, and hopefully we can converge on on something.

C

Nice congrats on making boring work, look cool.

F

Spawn a few satellite mate.

A

Wonderful, we are approaching time. I don't want to cut anyone off, though so any any last things, if not I'll be linking.

A

Okay, then, I am going to link you all to our little feedback form in case you have any thoughts. This is mostly I do read these to make sure nothing goes wrong with the recordings, audio quality and just a few of any ideas for us, and it's a really uh not pretty google link but yeah. If you have any comments on this call, you want to give you can give it there. We will also have our next community call in february.

A

If I can find the agenda, I believe it's the 22nd same zoom time same zoom channel, and you can already sign up if you're so excited so inspired by today we do have the agenda for next month's. There totally open, just making sure people know you can share anything, jupiter related that you want it's just a big show until where we get excited. So thank you so much for all your time. This was a really fun call a great way to kick off the year and still kind of early week. So thank you so much.

A

I

Thank you thanks.

C

Isabella thanks. Thank you all.