Jupyter Community, 29 Jun 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Jupyter Community Call - June 29, 2021

Description

Recording from the Jupyter Community Call in June 2021.

The notes from this call can be found here:

https://jupyter.readthedocs.io/en/latest/community/community-call-notes/2021-june.html

Read more about these calls on Discourse:
https://discourse.jupyter.org/t/jupyter-community-calls/668

A

Here the the voiceover tells us that the recording was successful, so hello, everyone and welcome to the june 2021 jupiter community call woohoo. That means we're halfway through this year, so congrats on making it and having the time to be here, I love to see everyone's faces and what we're going to be up to so hooray for context for anyone, who's, new or stumbled into this by accident. Somehow this is a place where we try and connect across communities understand what cool things people are doing.

A

It's kind of like a big show and tell where people are like I'm excited about this. I made this thing or somebody else made this, and it really helped me out. So it's here to help us stay connected across projects in jupiter and just get excited for the work we're doing.

A

Also you already heard it community calls are recorded, though they are posted on youtube if you ever need to catch up or you're just curious or if you love a share and want to go back and figure out what was the name of that cool project someone shared five months ago.

A

uh There also remember this is a jupiter community event, like all other jupiter community events, we are held to that code of conduct that you can find anytime at jupiter.org conduct. That includes me in case. That's needed for any reason, and what we're going to do is follow the agenda, which I can link again if anyone needs, and that means we're going to be starting with just the quick shares which these are like announcements that people might have that just take a moment like this was released or this.

A

So I think we have one on there from frederick actually so ready to go.

B

Yes, sorry, I was adding a last minute bullet so because people have pinned me about it, I'm just telling that totally v5 for python has been released with a major change for jupiter app integration. In the sense, that's no federated extension, so that's mean you just have to pip, install or condense tag the package, and you will have the extension inside jupyter app without anything else to do another nice thing is that there is only one package in the front end now and the last main change that I did, and so I'm.

C

B

Of pro, the guy is to be able to load the the library uh in a lazy way. So that's mean that if you are not using portly, but it's installed, you won't download the javascript library inside jupiter lab up to the point. You really need it, and the other point is that oh thanks steven with the help of steven.

C

I've learned how to.

B

Make release of jupiter web so we just released jupiter fv, 3.1 beta0, and if people want to try it go ahead, there is nice features coming in like a real-time collaboration.

B

The debugger is coming also with ipad kernel v6, and so there is quite new stuff and there is also enhancement for the interface stuff, like that, so don't hesitate to to try it and report any back before we do the final release and hopefully in next month, or something like that. So that was all for me. Thanks isabella.

A

Thank you. That's awesome. I love hearing about releases and great jupiter stuff which, by the way when that does go out, the feedback is very much wanted. To reiterate what frederick said. So, please that would be awesome to know what y'all are thinking. I'm the next one on the short reports.

A

Actually I linked to a jupiter disco, yeah discourse issue where uh I just for full disclosure, have been asked to give a five minute discussion of what's happening with all the jupiter projects at scipy, and uh I want to make sure that I know what's happening because, even though I try and stay across community, it's not like.

A

I see everything so if people have things that they want in the pool for potentially being brought up, that would be great if you could add that there just so, I'm aware, because I definitely don't know everything that's happening here. So that's my little thing, but I think we have some more reports after me. Now it's nick or jeremy. Next.

D

Well, nobody spoke up, so I guess that's me. um So we uh we soft, released uh jupiter light the cli package. uh I broke the packaging because total rook amateur here sorry, um so it doesn't have a readme on pi pi. But what are you gonna do? um Anyhow with that now you don't have to hand build the federated extensions. You don't have to hand build contents.

D

It will do all that junk for you, you just you know, jupiter light build in a directory and now you've got a static site that you can host and since the last time we showed it, we got uh real uh ipi widget support like up to plotly and and uh and vq plot and leaflet. uh Pythag py3.js is hard, um so we didn't quite get there yet uh because of nested buffered binary things, um but uh making a lot of strides there. The rtc stuff is looking kind of insane.

D

um So you know more good stuff is coming workspaces. Gonna hit that uh going to figure out how to grab stuff straight out of wheels and until condo forge comes along and gives us. You know a wasm target. We're gonna have a lot more uh work to do to patch existing python stuff so that it runs in the browser, but um this is really good stuff guys uh this is this is gonna.

D

I mean this is how we get jupiter to the next billion people um that have a chromebook or whatever, because uh you know in 2021 it's still really hard to install python, and uh this is really gonna is really gonna move things forward. um Another clever thing it has is, uh it can do. Reproducible builds um at least if you're, not on windows.

D

I don't know why I haven't looked into what magicjunk it's putting in there, but uh if you run jupyter lab archive and you give it a source date epoch, you will get exactly the same side out, um given the same input which is kind of nuts, because you know, reproducible builds are hard, um so anyhow, no demo, because it's the same thing as last time.

D

It's just you can build your own now uh and we're gonna be working on some sphynx integration and stuff so that it's even easier to get that stuff built and published on a good web host yeah. That's all.

A

Fantastic, more awesome, release news and more super cool things to play with when we have time. I look forward to probably, though future demos, as things continue to get more wild over there. So thank you. uh What about? Oh, we have kevin next. Would you like to speak.

E

Sure, thank you. um I just want to. Let folks know that the kernel, provisioning pull request is ready for review um kernel provisioning, essentially abstracts out the the p open layer that the kernel manager interacts with the kernel process with and allows a pluggable interface to um bring your own uh provisioning provisioner.

E

If you will so that you could launch kernels into different environments within the existing jupiter framework, um so it's essentially just a process uh layer, a process, interaction layer. So, if folks would like to review it, I'd appreciate the comments and we'll we'll move forward from there. Thank you.

A

That's wonderful and thank you for doing that. I know you spend a lot of time in projects with with less community members around them, so appreciate you keeping things running. Thank you and we have more. I love seeing all these announcements. You all are doing so much congrats on that we have luik.

F

Yes, uh sorry, I wasn't there at the latest jupiter lab meetings. I was kind of busy lately, but things are still still running up. I mean we are still working on things and notably, um we have a new release for jupiter lab h5 web, which is landing this week. This is not released yet, but we hope to to make it this week.

F

So for those who don't know, I already presented this at a previous jupiter lab community call, but jupiter lab 85 web is an extension to explore hdfi files which are filed where, basically, there is a whole file system in there and for those who know hdfire files. Sometimes you can have issues with external link that links to other files and compression of the datasets that are that are contained in such files, and the latest release should address these issues and bringing the support for these features, and also we are also bringing so.

F

The extension is based on web viewer, which is called h5 web, and we are adding a lots. Lots of features of visualization into this web viewer and the next release of the extension should bring. Also, these new visualization features inside the extension.

F

So there is a lot of stuff that is going in this release and I hope that it will interest people that are looking at hdfi file in jupyter, lab or elsewhere.

A

That's fantastic. It's been awesome to hear all the updates from that project, so maybe one day we'll see another demo with all the cool new tricks. After so.

F

Yeah, maybe thanks.

A

Possibly no pressure there please, but yeah we now have nick or mike. We have options again choose your own adventure uh short report. Go ahead.

D

And I don't uh kind of tell who's here, but anyhow uh so mike and I have been uh working, the uh language, server protocol, integration for jupiter lab and jupiter server for some time. Now uh I don't even remember- and uh it's you know it's pretty good, uh but we're ready to graduate it out of uh uh mike's personal repo and and start really moving forward. um It's gonna have impacts on a lot of places. I have a feeling, um because you know it's a whole.

D

You know four times as many more messages than uh than jupiter kernel messaging. um So if anybody's got feedback on that, that would be super welcome as to what you might do with it or uh how you might be able to use it in your project or your product. um You know it's it's one of those things where it's it's kind of a.

D

I don't know it's. It's a community driven spec, but kind of not really uh there's that organization that I see a lot of attendees here today. From that you know they're kind of kind of holding the close, so we're trying to get to something where you know we can more broadly embrace it across a bunch of projects.

D

um We've got a bunch of you know a lot of good stuff to see there if you haven't seen it before, um and it's just there's a lot of tricky bits on on making all that stuff work correctly for everybody's use cases. um That is all.

A

Woohoo, I hope you get some good feedback, since that's gonna have a lot of impacts and finally, we have matt.

A

A

If you're speaking that, I can't see, I can't hear you at all: oh there you are.

G

Okay, one second here I was using the wrong mic. Okay, can you hear me now yep yep, all right, so I had two things. The jump one reminded me there's a gem to to look at as well um so ambi converts 6.1.0 um had been sitting there and a bunch of um community members had submitted different pr's and then there was kind of a poke for hey.

G

It's been eight months, which is basically from when right before I had my daughter um since anyone get released foreign, so uh I tried to put together as best I could some testing and make sure everything was smooth and didn't rc for a week and a half or two weeks um to see if everything got there and got some test fixes. If I haven't had anyone report like major issues, I think all the stuff went through pretty cleanly, but if you do see something there, I do raise an issue.

G

um I'd say the biggest thing that repo needs by the way is it's people to help make example docs and upgrade the example docs like mb convert examples, for example, is uh 5.x, not 6.x patterns and no one has updated. Those and some of the documentation could be better about how to do. Custom um templates, and I just haven't had a lot of time to put into that.

G

That's the thing I was looking at what like needs the most attention in that repo, so if anyone does have a little time to help some of the documentation there that's familiar, I would love to have some help on that and then there's um uh jeff that's been sitting for a while, probably ready to go at this point. But not everyone has kind of taken a look.

G

So if you want to um for making that link, uh if you want to take a look at um the proposal to transmit cell metadata on uh execute request, that's been up for a couple months. We should probably look at you like wrapping that up or getting final feedback on it. So for the folks here give a look at that, so we can move that forward uh and feel a little more confident to be great.

G

I can talk more to what that one does. If people want.

A

To be clear to people, those are your seconds of silence for for asking.

G

Questions alrighty. Well, that's my update.

A

Yeah, thank you so much. I I'm loving hearing about all these releases today, so glad to have another one, and always we need doc's help. So a good call to call for that as well. uh This I'm going to give a few more seconds if anybody else says. Oh that reminded me of this short report and then move on. But here we.

A

A

Okay, then, thank you so much for all those great reports and it's big agenda item time now. So with no further ado, if you're ready to go eduardo, it would be so exciting to see what you have for us today.

H

Yes, thank you. I'm gonna share my screen.

H

Okay, can you see my screen.

B

Absolutely cool.

H

uh Well, thanks everyone, I'm gonna talk to you about the project. I've been working on, um it's called bloomberg and the idea is to help data scientists build production, ready pipelines from jupiter. So this is the github repo and I'm going to show a demo, which I think it's more exciting. So um this is the idea we have. um We are we want to build a data science pipeline, so usually the way it works uh with some games is that they built everything in a single notebook and then it's really hard to manage right.

H

If you have a notebook that has 1000 sales, you break something it's really hard to to fix it. So the idea with bloomberg is that it allows you to break down that gigantic jupiter notebook into small parts uh that then you can test separately and that some, like more than one person, can collaborate in different parts. So what I have here is a jaml file, which is the way you declare a pipeline. So we can see that this pipeline has four tasks.

H

So first important thing is that these are scripts, that open as notebooks, so I'm going to show one of the scripts in this pipeline, which is the get a script that just gets some data, so I can open it as a notebook. um Why? Why is? uh Is this? The case? Well, sometimes is well, it's really hard to do code reviews with jupiter notebooks. So the idea here is that you simply use scripts.

H

You work on them and then you can do code reviews and do like git, div and all those things. But when you execute this script, it's going to generate a notebook as an output. So you kind of separate your source code from your output, and this helps quick go to reviews. Now, that's the first important thing.

H

The second important thing is: how do you concatenate different scripts or notebooks into a pipeline? I'm going to create a diagram of this pipeline to show the relationships between these four scripts. So now I have this raylan.

H

So I can I'm gonna move this here and I have four scripts, so I am getting some data then I am. This is a simple machine learning pipeline really basic using the ibs data set. So I get some data and then I generate some features. Some feature a feature b: join everything and train a model right. So how do we establish these relationships?

H

That's really simple! So let me open one script. That has a dependency, so I'm going to open filter a so. If I go and open figure a.

H

I'll see that I have this upstream variable and I can say for example here I'm saying I want to use the output from get as an input, so I simply have to say get I put the name of the task and then, when I reload the the file, so actually I'm gonna, I'm gonna do this. So if I set this to none and then reload the file.

H

So this is how you would start right. You have a script and then you say well. I want to use an output from a previous task as input, so I say I want to use get so I save, and if I reload again I'll, see this new code, which is saying well, you want to use, get as input now here you have the output from that file and then I can simply execute my thing without having to oh.

H

I didn't I didn't run the previous task, so I don't have the actual output I'll just do that, so this thing works. So let's say I want to get some data. I open the first task uh generate some data frame.

H

I'm missing one folder here so I'll just create one output.

H

Okay, all right now I saved the output from my first task and now I can continue on a separate notebook using that output, and then I can run my thing here.

H

So this is kind of like the interactive experience. How can I build something from small notebooks and then I can run the whole thing from the command line and say just run. Every notebook um like this is essentially the experience you separate your logic in a script you break down in small scripts, your orchestrate execution, and then you can do other nice things like parallelizing, independent tasks, for example. Here I can run these two tasks in parallel or you can even with some little extra configuration.

H

You can run this thing in the cloud, so we currently support airflow aws batch and kubernetes using our workflows. So the idea is that you don't have to move back and forth between writing a notebook. Then copying copy pasting that code to an airflow bag.

H

You can simply write it here and then you can export your work and run it in production. That's it.

A

Fantastic, thank you for a demo and I I don't know I love seeing when things kind of break and seeing how people fix them. So I find that informative, but questions or comments from.

A

Others and if no one does then I do, I am.

F

I mean I have a question, um so you specify the dependency of your pipeline in the script right. uh I was kind of surprised because, usually in the yaml file in cip pipeline and things like that, you specify the dependency in the yammer right.

H

Yeah, so that's the default option. That's a great point. That's the default option. uh You can specify the dependencies in the jaml file itself, uh but when, when talking to users, we found out that this is kind of simpler. uh I guess it has to do with the thought process right. You open your open script and then you think what am I going to use as input here, but you can change that option. Yeah.

F

Okay, interesting thanks.

A

Can I ask what inspired like this particular approach? I know people have done workflow things in a lot of different ways. So I'm curious what what made you and the rest of your team from looking at the repo.

H

um I think so kind of this project has been going on for uh more than one year, so actually the first version didn't support notebooks at all. uh It kind of evolved from being like a sql python functions approach and then at some point we thought well for some tasks in the pipeline. We usually need a jupiter notebook because it just makes exploration so much easier and that's one thing: data exploration. The second thing is when we train models. We want some report right.

H

We want some evaluation, metrics or some tables, and then we thought well. Actually, the jupiter format is great because we can have a single file that has everything that we need. So if we run something in the cloud, we just get this jupyter notebook back and we can take a look at the model evaluation uh report, so that can that's that was kind of the inspiration and then people started using it as a jupiter notebook-centric tool.

H

So we started working on that and- and I think it's been- uh we have received great feedback from a couple companies that are using it and think that this is really helping them organize their pipelines, because data scientists no longer have to copy paste between notebooks. They can just write something in a notebook and then they can move it to production without having to rebuild or refactor the whole thing.

A

Well, that was a great explanation. Thank you. So much luciano did you did you want to say your comment.

I

I was just gonna ask eduardo if he he was aware of lyra, uh so we we have a similar approach, but we uh basically doing most of the the workflow uh visually uh uh and- and I think in terms of back ends is similar. We support airflow.

I

uh We support argo and tacton via uh kfp, so um just take a look might be some synergies there that we can work on on integrations so.

H

Yeah, absolutely I know about leira. I watch the jupiter con presentation, uh so I think it's a great project yeah. Definitely there's a lot of similarities, so maybe we can chat offline to see how we can collaborate. Sure.

H

A

A

Okay, I thought that was someone. I thought that was someone turning on their mic to speak, but I guess not. Can I actually ask one more thing, though: you've been talking a little bit about user feedback? Is that just through um I don't know, I'm curious how you're getting that. That's always interesting to me in open source projects, because some times people get it easily and sometimes they don't.

H

It's been from different channels, I would say some people just open issues on the github repo, but I think the most engaged users just sent me emails with lots of questions. So then I think that's great, because you kind of start building this uh relationship with users that maybe are afraid to ask like they think dumb questions on a public public space.

H

uh So that's been great and I sometimes talk to them like consumer or anything like I keep in touch with them to understand, what's going on how they are using it- and I think I mean it's an exciting point for this project, because the next release that we make is going to be entirely driven by users feedback. So we are no longer thinking well, I think this is going to solve certain problem.

H

We are actually asking users and then fixing things and that's how we are by kind of building. The next features.

A

That's so that's so nice to hear music to my ears. Well, congrats on that. That's amazing! Thank you.

A

I don't think there are any.

F

A

F

A

Leave a moment again.

F

I just have one last question that was supported, but what you were saying do you have an idea of the demography? Let's say of your users: what are they doing with this tool, mostly machine learning or things like that.

H

Yeah, so I the so the set of users that I know more about is a company in the us, like health care, company, healthcare technology company. They are doing so right now. There are only a few people at that company using it, but they want to expand it to like data scientists who mostly use jupiter for for work. I think they are mostly using machine learning pipelines.

H

um They are probably going to expand to maybe some like analytics, but mostly machine learning, and I think it also has to do with the kind of community that I've been um like speaking to which, which has been mostly machine learning, but definitely, and also most of the examples, are machine learning pipelines, but there's definitely room for some other use cases.

H

A

Yep thanks everyone for a great discussion and thank you for eduardo for taking the time to demo and show up it's. I love seeing new demos. I love seeing new people, so welcome welcome to our community more formally. I don't know if this is formal.

H

Yeah, thank you thank you and thanks for all the questions.

A

Are we ready to move on to the talk that, according to the notes, at least, is about to be given by a bird though I have a feeling, it's nick or maybe tony, you know.

A

I don't know now: whoever has the next item on the list, since you did not.

D

Mean can't we can't.

A

D

Tony, we cannot hear you plug and replug use your clapper thank.

G

D

A

Clarification for your amazon or something.

D

So, with some tap dancing music, while he was working with stuff pidgey- is a uh evolution of um some literate programming stuff that we built uh primarily for for doing presentations and demos um jump in any time. Tony nope, you can't jump in he's still silent. I can't hear him, uh but he's he's cursing it's kind of funny watching it actually.

A

D

Here's smoking, um and so it's you know it's been, it's been guided by failure driven development. Basically, can you take the idea that markup has that you can't write, broken markdown um and apply that to writing code and documents that are interactive there? We go. I have a microphone, oh my god,.

C

I'm so sorry, thanks for thanks for keeping that balloon in the air nick, it was good. I'm you know anybody who advocates for pidgey is uh destined to be uh made fun of, um but I need everybody to open up their minds. Forget all the you hate about notebooks and just let's think, into the future a little bit please so, like nick said um we actually have been so pidgey is a project that deathbeds has been working on for an exceptionally long time. uh It's never been quite right, but it's always been quite fun.

C

um So one of the more recent uh talks that we gave uh this one was super fun. Wasn't it nick the ten pounds of talk um where uh we basically spent a lot of time comparing uh the jupiter community to uh playing around uh as if computing were skateparks? So we start talking about these computational skatepark ideas and when we're giving these presentations, you know it really becomes like a multimedia collage at some point and what we're really doing is we're combining code and narrative together um to tell stories right.

C

So uh if you look at this, this is an example of an old pidgey version that we've used. But it's marked down there's indented code and then there's more markdown. The indented code gets executed, uh the rest of the markdown gets shown um so from one cell. We can make a lovely little composition of exploring communities of practice and skate parking and open source and computational architecture. So it's really fun for composition. uh We actually have some places where uh we put some graphics in here.

C

um We've got some uh images and tons of tons of fun stuff, but um this is how we've used it in practice, um but I want to tell you a little bit about the current state of things.

C

um So before I get going, I want to talk that nick mentioned the idea of the failure of code right. So if I were to in the notebook- and I typed print 10- um and I didn't have my parenthesis here- I should get an error right. So what we notice is that the code cell um is actually being rendered as a piece of markdown.

C

If we tab it in and we run it now, we get errors, uh that's not the right error, but we still get errors. um So what I do in practice when I'm working in the notebooks, even if it's a normal notebook, this isn't for pidgey notebooks.

C

I always indent my code, because if I have like hella blocks of code and at the end of the day, I want to actually print 20. um What if I were to render this untapped?

C

It looks like that and it's not code if I tab it in it, looks like code and when I render it as a markdown cell, it looks like code, so in a lot of ways the notebook has taught me to really program and markdown um one of the features of pidgey.

C

If you put a url in here that you can iframe, it just shows up right. So this becomes really nice for taking notes.

C

um One of the first demos I'm going to show you is uh a riff off of um brett victor's early work in tangle.js, where you know the idea is, is that you know you can have this object and you can change the narrative in line and update things now. I haven't gotten quite quite to this point of integration and mind you as I'm, going through uh the fancy parts of this demo. This is not there's no javascript in this. It's a pure ipython kernel um and it's written purely in python. So this is this.

C

Is this? Is a python project all right? So let's recreate this uh demo here and uh what we're saying is that hey uh now I can go and drag this. My pidgey cell actually contains ginger templates. It doesn't compute this stuff for me right. So when this widget's changing it's actually going and updating the template, I make a little widget and I say it's a cookie.

C

If you look, there's actually an emoji, um so that's another feature of support in pidgey is uh using emojis for things. uh What happens? Is the emoji name gets munged into a python alias, so we could have a baby um baby, chick baseball.

C

Penguin and then, uh if we were to put all of these things together, so I added some emoji completion. That's a feature of pidgey here. So if we put all these things together- and we say this equals 10, then we now have a baby chick baseball, penguin variable. um For me, this really helps with naming. I don't like naming. I think it's a waste of time, uh but nevertheless, let's move a little bit forward.

C

So we have these ginger templates in here and what I thought would always be cool is if the ginger templates could hook into the ipython display formatter and give us html stuff right, so we're going to uh make a little. So basically, this is a pidgey cell and we can think of it as a caption, where I'm going to make a data frame in pandas.

C

It's just some random data frame, um the figure I'm going to plot the figure it's going to go right in the ginger template as an image, I'm going to tell you a little bit about the shape, I'm going to show you some data frames from it and I'm going to run it.

C

So if I go and move this new output here, um so we have a new output. Oh, that might be an old output view. Let me uh just double check here all right: let's get new output view all right radical.

C

This is my favorite part all right. So uh with ginger templates you can find out what variables they need. So I know what variables I need to re-render this, and I know that if dataframe changes, then I'm going to keep re-rendering it.

C

So what's really cool is without javascript and just using python and the ipython shell and the ipython kernel we actually, we actually have like non-linear interactivity, like the code will update as you change stuff, um and this not only goes for the display, but if we think about doc testing, it's a really great literary programming interface for writing tests right where it's just like. Let's make some narrative and let's mix some code together, so I've actually put uh testing as a first class citizen in um pidgey. So now it says, oh f is not defined.

C

Of course it's not defined. This is a dot test that says: oh f should return a value of one. So, let's go and say, f returns a value of ten, and we run this so now our dot test goes and updates. It says that it got 10, it expected one. We can go change this and now we get a check mark and we're satisfied.

C

So pidgey has a lot of affordances and ideas into it, but a lot of the main goal is to strengthen the narrative. This is not about code, we'll turn it into code later on.

C

uh My favorite thing in this recently has been adding the emoji support. um We can import pandas as panda face, using or as a real emoji. um We can uh make all of these different names of things, and you know it winds up, adding a splash of color right and we can write these like cryptic little poetic uh code statements that actually execute some code.

C

Now this isn't just for python. We won't work in all the languages. That's what's super cool about markdown right like at some point. We should be able to take advantage of all the kernels and have pidgey doing smart stuff with all the kernels, but until then we do have a list flavor, so you can actually write a liter literate lisp and learn how to play around with lists. This is using hi, uh which is a python uh flavored lisp. um So this is another feature. uh The testing features go a little bit further.

C

In fact, so we we allow for doc testing in here. The check marks indicate a successful test. We can run cells and they will continue running even if they get a test failure which is super important, because some notebooks might be about a failure. We can use emojis in our test. We can test functions, um we can test unit, test classes and so on and so forth. um But this is a lot. Pidgey does a lot of things. It's super fun.

C

I hope to make some good getting started docs at some point, but one of my favorite outcomes of this is that all of your markdown becomes a program. It becomes a test. So uh I really like this project. Do it uh for task task execution? um So if you look, it says with pidgey loader. uh This is context manager from task import star. I actually have a file called tasksmd.

C

All of the indented stuff is code, so this is a literate markdown program. This just runs python. It works as a python script and I find it really helpful when I'm working at this heavy heavy interface of language and code, that these abilities to weave narrative and code together become very, very important for me. Communicating my narrative not only to myself but to my colleagues, uh so I'm gonna stop there. That was uh the pidgey demo um and uh I can share any links or whatever.

C

But um thank you for letting me share that with you all.

A

Thank you so much for sharing. That was like high speed. So much happened. I have so many thoughts and questions, but I have so.

C

Many things happening: yeah right now has a lot of stuff.

A

But anyone else have a overwhelming thoughts or maybe not overwhelmed. Maybe y'all were great you're all so.

A

Cool, I feel like I saw a lot of faces that kind of like had like the pinched brows and, like the like the hand on the chin, so I feel like there has to be questions here. Besides me, there are no. I talked to tony kind of regularly. I don't want to push you all. I'm just saying I.

C

I yeah, if any, you know I'll start out. If anybody is like why on earth, would you do this?

C

I can try and answer that, but for the most part like, I think, that's the biggest question, um but you know I want to be able to talk and code at the same time like a guitarist in a band plays the guitar and things at the same time right.

C

So I need these deep, deep combinations of code and narrative to be able to do that, and this is kind of an evolution of the notebook because we have markdown cells and we have code cells, but now we can bring them tighter together until cooler, yes, lsp nightmare, absolutely an lsp nightmare.

D

And so I mean to that, to that point I I you know created my own question. um One of the things that we were talking about on jep72 um many kernels introduce syntax that does not work with their host language and uh pidgey's is just an extremely extreme case of that it doesn't put it in notebook metadata, but otherwise it's it's insane um and formalizing.

D

If a kernel could report the nature of its broken syntax that it provides- and you know round trip transformations uh at the token level at the line level back and forth, we could start doing some of this stuff in a more automated fashion like right now we support a bunch of um ipython magics by hand uh in typescript, which is not where we want them to live.

D

um But you know that at the at the end of this there might be something to that of polyglot documents that their runtimes themselves actually tell you what makes them broken and how to unbreak them. In the case of static analysis, in the case of testing in the case of packaging right, because we we, I don't think we have, we actually shipped any package notebooks. I think we took it out, I think we took them out. I think we and why?

D

I don't know anything, but you know with with import.

C

D

Yeah, it's about the only well, but I thought we took it out like we pre-processed anyhow, we want to ship notebooks right like all these complaints about how, on whatever they are, I mean at the end of it.

D

If you crack open the docks for your file and oh it's a friggin notebook that you can run and it's got examples and it's got narrative, I mean I want that more than you know, sphinx output, so um these are all these are all kind of these future things and and where we really want to get to with pidgey is, is pros, mirror.

D

um You know when we start defining some of those ginger templates and things like that, where you drop into something that looks like word, you know that's again for the next billion people right like a blank, a blank document, that's blinking at you that is not going to start, throwing pink salmony text at you as soon as you type the wrong character is, is really empowering um and we've been, you know, tracking the nature of learning, computational thought or whatever for a couple of years.

D

Now and it's I don't know if any languages are really moving it forward. You know it's a ui problem. It's not really a code problem, um so we're uniquely positioned to do something about that.

C

Yeah, I think to the one of the points to highlight there is like we want this to be a word processor. We want word processors with compute behind them.

C

I think scientists need that right, like we can't tell stories without data and code. Now we have too much of.

C

C

Yeah, all of the above, oh geez, go ahead.

G

Are you what's this jupiter cell type that you're saving, are you saving raw cell or is it saving as code or markdown, when it knows what, if it's mixed,.

C

So it's really leveraging the interactive shell right, like I've, got a bunch of cleanup transformers and apps parsers and compiler sort of things that happen in there. But at the end of the day it goes through the transformation process. The code gets run.

C

If the first line of the cell isn't blank, then it just makes a markdown display, um and I have these markdown displays that are sort of like widget lists where they use the traitlets and the ipython display handle to update themselves so they're just normal to markdown displays um and so are the tests. The tests are also marked on displays.

C

So, basically, after the code, the code runs, the execution comes out, so you might- and this flow is important, because if you get a figure, then you can almost consider your input cell as a caption right. If you did it the other way, then it breaks the caption uh sort of convention. So you get your display and then you get the markdown display out of it and uh yeah we're not doing anything really weird.

C

It's just code set it's if it's a code cell, it runs um and that's one of the conventions that pidgey brings too, which is like instead of having code cells and markdown cells, we have on and off cells now right um where a markdown cell doesn't execute, and a code cell just executes.

G

Yeah, I'm just thinking from the format save like. If you take a pidgey file and load it and load it in classic or lab, but without pidgey what it would look like.

C

Oh yeah it would, it would render perfectly fine uh the code, wouldn't the code wouldn't compute, though um in some cases you might, you know just get the ipython text wrapper um like that's, not trusted, uh but yeah. You can use this without a pidgey kernel. You can actually use a bunch of the magics and stuff like that. So you know it works in interact. It works in classic. It works in lab um and in fact uh you can act.

C

It actually turns the markdown editor in jupiter lab into a really awesome ide, because the markdown editor already has line selections and you can shift enter on those. So you can basically be writing side by side and it's a super fabulous like literate programming. Experience.

D

Yeah the one thing missing: there is the same thing from the rtc thing: where we don't our markdown renderer has no idea where its source is. It has there's no way to jump back to it again on the lsp thing, but uh getting the sync scrolling um between a pidgey source or between an rtc document, and what you're doing, I think would be really massive. I looked into it for uh the jupiter lab markup thing that angus started um it's hard. It's it's really hard.

D

uh Your parser really uh has to know a lot about, what's in your code, to get that that kind of resolution, but it would be massively helpful.

C

Yeah, while all of this stuff looks sort of wild, I'm actually not breaking rules, I tried to stick as much to the uh standards as as I could and if it didn't, if there wasn't a standards approach, I didn't choose uh that feature in the in.

A

Pidgey constraints do make more interesting designs.

A

Also, I don't know if we really went over how you you were using the like interactive widget on a web page that you had embedded in the notebook. There were a lot of things there, but before.

C

A

It below but um yeah we are, we are getting a little closer to time, but we obvious, I think we still have time for discussion, but any any other thoughts with that people have been very friendly.

C

I want to know where people on uh uh in this room sit on emojis variables, I'd love to have that discussion, because I think naming is only a problem. If you say it is.

D

Well, I mean you know: we've had that, since you know apl and fortran and stuff like right, like there's. No reason we can't have mathematical symbols, you know, that's that's how you get the that's, how you get the the pocket protector crowd involved, but you know the emojis really resonate with a lot of people from a linguistic point of view. They are for whatever reason you know kind of internationally uh accepted, and I I think it's a, I think, they're I think they're firing.

D

I mean, if that's the message you want to be telling that day you know um and uh yeah I'd love a jupiter, emoji font that'd be great.

G

Yeah, I definitely think emoji should be allowed to be supported. Even if standard conventions say don't use this, I don't think they should be blocked from being used as a in in coding, though, I personally believe that they shouldn't be encoding.

G

I don't think anything should stop users from using them if they want to.

C

It's tough is accepting people we want to be like. You should use this, but it's like I mean it still makes me cringe right, I'm never going to put it in my real code, but in my notebooks and the things that I present to people, I I can't look cute physically, but can at least make my notebooks look cute.

G

Yeah, it could definitely flavor up uh presentations at conferences and stuff that would be very entertaining, especially in the not in-person conferences that we will probably still continue to have for at least six to nine more months.

A

I feel, like you kind of just said, like you, can't write code or can't demo without being cute, and I I think that's a fun sentiment. So thank you.

A

Awesome, I I it's sounding kind of like we're winding down a little, but maybe.

C

A

Wrong last bursts of inspiration you've been visited by a muse from on high.

A

This is your moment.

C

Oh, I don't know if.

A

C

The draw io uh update that who is it carlos is working on. Oh no. Somebody posted that on twitter today, but that's pretty exciting they've got rtc draw draw io demo, move it along. That's a very exciting piece of technology.

J

um I wanted to re-state what um one of the things that frederick said, which is there, is a um release candidate of ipad kernel version 6 and it supports debugging, and if this is something you have any interest in, please pip install kernel to get that release candidate version and play with it in jupiter lab 3, which will automatically enable the debugger once it detects.

J

You have a debuggable kernel and uh this is shiny and new and hopefully fun so try it out.

C

We can add more stuff to pidgey now more.

A

Things, oh thanks for finding that uh steve. I was looking for that. I.

C

Was looking for it too, thanks for finding it steve.

A

Great well, it's sounding like we're kind of oh gosh. Where are my notes? We're kinda we've got all our info here. We got so much info today. Thank you, everyone not only for showing up, but for sharing. So many different things happening. I love to see what's happening here. That's why I love to do these calls.

A

So I hope that's why you love to be here, though, if you do have thoughts on that, we actually have a feedback form um which you can give as soon as I get my life together and link this, and that is now in the chat, but I also want to say if this was your first time extra welcome to you and if you haven't shared before, if you have, I hope, you've seen, we have a real range of shares and like presentation styles here, it doesn't have to be.

A

Sometimes people have an idea of polished, I think- and it doesn't have to be that's totally up to your comfort level and how you feel okay sharing. But if you do, I do already have an agenda for next month. If you want to sign up for that, as.

E

A

It's there whenever you are ready and our next community call will be on july 27th. I should double check that. Yes, I was correct this time, so that'd be great. If anyone also ever wants to run these calls, I'm going to continue extending that I love to run them. You don't have to do it alone, but it's probably better if more than one person knows what's happening. So that's that's my little.

C

Spiel at the end- and you know it does need to be.

A

Updated you're right. Thank you. It's been.

C

A weird week- and you have a bunch of uh resources for getting started with these events too. That, like, if folks, do want to start these there's they're not getting started from scratch anymore.

C

A

Yes, no, I I was actually thinking the next time. I may share those because they've kind of been work in progress, but I I think they're working, because these calls keep happening. So I may share those more publicly so yeah, thanks for noticing that needed updating, y'all, I'm gonna do that right now and yes, that's good to know that it may overlap with things. So, thanks for letting me know claudia- and I will stop recording now, actually, as I'm continuing to mess things up there, we go. Thank.