GitHub GitHub Universe 2022 - AI, 21 Nov 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: GitHub and VS Code tricks for data scientists - Universe 2022

Description

Did you know that there are over 9.5 million Jupyter notebooks and data science repos on GitHub? Come and learn how to use GitHub and VS Code to supercharge your productivity authoring notebooks, building machine learning models, and collaborating on data science projects. Edit and run notebooks on GitHub from the comfort of your browser with vscode.dev. Also, learn how to take advantage of free compute and GPUs on Codespaces to train models.

As always, feel free to leave us a comment below and don't forget to subscribe: http://bit.ly/subgithub

Thanks!

Connect with us.
Facebook: http://fb.com/github
Twitter: http://twitter.com/github
LinkedIn: http://linkedin.com/company/github

About GitHub
GitHub is the best place to share code with friends, co-workers, classmates, and complete strangers. Millions of people use GitHub to build amazing things together. For more info, go to http://github.com

A

A

Ers and welcome to looking at vs code and GitHub using tricks for data scientists, I'm Harold, kirschner, I work on vs code. I do look at a lot of data I run surveys. We do a b testing I run. We work on the website, so there's a ton of data around me, but I'm, not a data scientist I'm, not sure why I'm up on stage here to talk about this, but let's learn data science together and I. Think you saw another keynote already, what's probably coming so we're going to start, of course, with codespaces I.

A

Think everybody these days has a job of looking at data scientists. So even if you're, if it's not your title, even if you work in the team and you have a data scientist being able to bring the data to a discussion- empowers everybody in team. That's why I'm so excited about things like code spaces templates where I can just open the jupyter notebook. So it's here to set up and I started getting a data science I had to learn. Notebooks I started. Putting all these things on my machine and codespace has shortcuts.

A

All of that I don't have to manage my environment, have to keep it maintained, install new dependencies as they're coming out, keep it secure all these other things and I like vs code I, know my way around vs code. I don't want to keep using that to learn notebooks, even though it's at Super notebooks notebooks now as a concept morphed into so many more apps, including these code. So let's start looking at that.

A

If you want to try this out at home, just use this template on jupyter Notebook I already have it warmed up, so we can Dive Right In, so the great part about vs code and browser. It will bring you all the keys code capabilities with just everything you need, including notebook support, so we're looking at notebooks in vs code.

A

But this is my vs code, so you probably notice that one important difference sidebar on the right where it belongs, which is important, because if you actually close and open it, it will work much better with everything else and wouldn't move your code as much around. It's also my theme. This theme is actually Dracula, so feel free to steal it.

A

It has my extensions that I, curated and groomed over the years, so I can actually understand uh property markdown with Emojis, which is also an important fact, and it has my keyboard shortcuts I'm notified you I've, been using a few editors over the years, I started with Sublime and Adam mvs code and you kind of accumulate shortcuts in your muscle memory that are hard to get rid of. So my stick is command shift d for duplicating lines.

A

I just go in here: I can duplicate my line away, and it has been always, like my left hand, is just I can't get rid of it. I keep hitting it and nothing happens. Actually, the debugger opens I think so. I had to shift it around now.

A

The debugger doesn't open anymore and I can properly duplicate lines, and all that comes with me through setting sync, so I'm logged into GitHub um interviews code with GitHub, and as soon as you switch on settingsync on your desktop machine, it will bring everything everything you have in vs code into the cloud. As soon as you open the code space, it will come down and will just look like your vs code, which is great also, of course, this is a vs code in a code space.

A

So it comes with a terminal and that's a big difference to other notebook environments that you open. That gives you a nice notebook environment. It might be preloaded with a lot of things like this one pips3 list, let's see what's here for python, and we get a lot of these packages already. So where did they come from? They come from a Dev container, which you might have seen in Keynotes as well really so this is your config as code way to define your development environment on all the dependencies.

A

So settingsync brings everything that you already pre-configured for your vs code and then Dev container defines what environment you're working in in this case. This is the pre-built GitHub package, which comes with a lot of python and packages already and then in this case we just do a pip install to get all the rest that we need. We also get these two extensions. So if you start working on notebooks locally get the Jupiter and the python extension, they will naturally be recommended to you once you start, which then brings you this beautiful notebook support.

A

So notebooks are a first class citizen envious code and have been for a while. If you look on the marketplace for extensions, you will find several other notebook formats. For example, the vs code team heavily uses GitHub issue notebooks to manage all their backlogs at work. So there's many ways we found notebooks help people to understand not just data but learn new languages and a lot of other things in this case we're looking at data science.

A

So this is The jupyter Notebook version we're looking at and if you know notebooks, you will feel comfortable and at ease. You have your markdown sales with vs code markdown editing, which is of course amazing. We have code sales with a run button on the left. This is kind of the most natural thing to do in a notebook, editing code and keeping running it on Hover and focus. We see these other things up to execute code on top and Below splitting cells all and, of course, a lot more options there.

A

So if there's a familiar notebook UI from everything, we know then of course comes the outline which is important for especially longer ones. Let's look at one of the other notebooks that's in here, which is the image classifier. So now we see the outline really shining make it bigger. The outline shines it highlights every single header in the file. So even have you had a longer notebook if your analysis grows and grows.

A

If you write a book in Jupiter notebooks, which people do you can still navigate that, and on top of that, let me add a little screencast mode. If you haven't seen it before so command shift, o is go to symbol. So if you already have these memory shortcuts from navigating your own code, they all just again work in notebooks. If you haven't adopted them, I keep looking for them, look it up in the command palette and definitely add it to your muscle memory.

A

So, looking at the notebook now we have the the hover buttons that I showed before, but there's also a lot more customizations. So vs code is well known for its massive amount of UI settings, so you can hide and show UI really based on your your actual knowledge of how to reach those.

A

So if you can go into zen mode and have no UI, that's probably the one extreme, so the notebook layout as a really quick way to to abstract what you're seeing within the notebook, for example, in this case, I already hit change this setting to show the toolbar on Hover versus just on click. So now, when I, when I open, this I will see this only now and this will show on Hover still this is toolbar. So that's that's one way. You can customize it and really make it feel.

A

If you already know notebooks, you can get to all the check boxes to make it look and feel like the notebooks. You already know so, as we now run, this and I ran I think past a bit through in the past, so I click run all and I see in the outline already which cells are executing, so I can follow along, which is important.

A

You want to know, what's actually being printed so I can go in and see the the output, but I can also see how long things took, which is important, because we don't want to rerun cells to take a minute to run and then oops I did it again. So this is nothing to keep an eye on, for, let's take a simple notebook to actually look at some of the actual editing. So this is great from a notebook perspective.

A

Everything is here, but then, from a vs code perspective, you will bring in all the amazing language capabilities that vs code already has. So if we go into markdown, we can actually go and edit uh in this case, let's try to understand markdown, so markdown is, is important for notebooks to actually describe what you're doing in this case, I added a read this link to the readme. So as we're editing, adding more code finding insights, one actually document them back in markdown, so markdown is a is a first class language in vs code.

A

So it understands even the autocomplete within some of these files, so I can go in here, hit autocomplete. It will understand I'm looking for the readme, so linking between files becomes really easy. As you tell your story, it doesn't just understand the file. It understands the content as well, so I can also autocomplete this header within the file and then because we have code navigation, I can also click it and open it later on.

A

So it makes code list much more interactive as you write a markdown now looking at code, this was marked down the content. Writing so it's easy to split cells and execute new ones, and if you start actually going in and writing code, you can get the amazing order, completion from vs code because it has a python interpreter and language server built in called Pilots. So all this, these panders, documentations and API calls become really transparent to you and how you use them and autocomm yourself.

A

Once you have code in front of you, you want to make sure to hover things, to understand them and there's these inlay hints to give you an idea of what types of variables you're looking at.

A

So that's where vs code really shines and not just writing code, but also when you start looking at a notebook to actually understand how the code works.

A

So now we got the basics of Mark of notebooks, we're all actual data science experts. You can add that to your LinkedIn skills. Now, let's see what else we we can come up with. So we we know data science. Now, let's collaborate, some some more within the actual notebooks, the.

A

So, to get your templates go to slash code spaces, which you have probably seen a few times today already and we're going to move on into how we collaborate together.

A

So it's important in data science, especially as a team, to have review to properly have your analysis, Version Control, to have the data you're looking at Version Control. So GitHub is a great place to do that. Your developers are on there, so they can start actually understanding where the metrics come from, how to act on them. They can create their own metrics and bring them in so I. Just bootstrapped.

A

This simple data science collaboration repository I try to document what are the best practices, how we're going to work together here how to collaborate and then I started opening this first pull request. So pull request as you do an analysis putting it already in a pull request to share early to think about what you're doing it helps me, especially to get feedback, sometimes I'm, not fully sure on data science. Like is this. The right approach. Is this actually MIP hacking myself here and trying to find insights?

A

So it's a great way to to get to work together on data science and Empower everybody to contribute and everybody can jump in and learn in this case, I did pull in the stack Overflow server from 2021 and thought about like four vs code. There should be some amazing insights here and I didn't get too far. I have a draft open and already changed a few files. So if you've seen notebooks on GitHub before that's how they look like so what's in private preview, right now is actually the proper notebook diffing, that's much richer.

A

So that's a feature preview in private beta that you can check out and it's going to make notebooks on GitHub, so much more powerful, but we're also going to get the same experience in vs code. So if we go back here and now open this, so usually, if you just want to check out an analysis, you would get it on your local machine. You try to Pivot in the right environment. It's a lot of a lot of work.

A

I, don't want to have this on my own machine, but I do like vs code on desktop and the way I can do that. I already have this code space here you go and open in and open it in motion, Studio code on your desktop and of course we did that just to be less less Wi-Fi, dependent and or this demo is by the way running over my phone. So this is still a good. It's a very uh Safeway, even on a low connectivity to work in these notebooks.

A

So this is now vs code on my desktop and it gives me the same experience like because I have settings sync. It looks like the other vs code. So usually, um let's look at this notebook. What I said what I created I have a stack Overflow survey: that's in the CSV over here then I had some questions already. uh The csb by the way published um under the open database license and it's already filtered down to self-described data scientists, then I import, pandas and Seabourn, which I love, which are have great apis.

A

I, don't necessarily know the apis, but I know they have great apis. I uh apply some options. These are useful to make it look. Nice, uh I'd, love to see CSV and then I look at the data, so this is already hard to tell because this for something 49 questions in the survey, so not too too insightful to look at um and then I'm trying to get this first Insight from it. So what if I compare coding experience with compensation? That must be the first shot at the least say like?

A

Oh, if you spend a lot of time in data science- and you know data science- you probably earn more money, but this is not the inset I'm looking for so this is this is what happens if you don't clean up data, if you just like try to guess from the fields like what should be happening, so, let's fix it up. The great part here is what I'm going to demo gonna run this for now see if it's still executes the same thing up here running running and the plot yep still looks broken.

A

So, let's go in. We have, of course, the DF head which tries to describe the data and now we're going to show off data Wrangler, so data Wrangler is not available, yet it's going to be available end of the month. This is the sneak sync preview of this tool and the first public demo, I guess so. I'm really happy and thank you to the team for letting me show that it's it's great. It's gonna be amazing, so this is the tool to have data exploration and cleanup within your notebook.

A

According to the Anaconda survey, over 45 of data science is just prepping data. It's about loading and cleaning it up and trying to make sense of it, and it's just a lot of time that goes into looking at histograms. Looking at columns, removing columns, cleaning things up, and this is the tool that brings it into vs code, so less context, switching less opening things in axle cleaning them up exporting them. You don't know what people did in Excel. You can't repeat it, so this is really what, where this tool finds a sweet spot.

A

So it's data. Wrangler brings this whole new UI into vs code. On the right side. We have operations. We can do on each column. The summary so we're going to look at now, so we have 4084 rows. So that's the respondents and with the 49 columns. So that's how many questions they probably answered. We see some some answers that uh weren't provided as often because it's a survey it has branches. If you only, if you pick a us, then you can pick a state.

A

So that's a good good sense like what actually the values we could want to use. That makes sense. So if you look at then this main row, this is where all the columns will be broken out to and already give you some summary.

A

So we get histograms if it's a number and we get these percentages if it's only string value, so we see only three percent of students, so you probably don't want to get too many insights out of this for students and we can look at countries well, so us is only at 21, that's good, India, 2012 and Germany, so something a few countries I'm using China is probably the same size developer wise as the us. So we definitely don't want to get insights into like a global economy of developers.

A

So it's all these things to keep in mind as you look at data so now, if we get into it, uh let's start actually cleaning it up. So we know that we wanted to look at comp total and that's kind of the first one so years code Pro. That was the other variable uh that looks good, that's a number with some missing values, so we might want to take care of that.

A

We know come total! Oh! This is this is not a good histogram right. This is some somebody book system. How do you these numbers get so high? They get so high because they're bound to a currency, so that's complex and then there's a com frequency, so some are yearly and some are monthly um that that was my mistake to use in in a graph. But is there a better version of that I know it's a preparation.

A

Somebody audited that in the past, so there's a converted comp yearly column that we have, which is in dollars with commas. So that's not a number is used and we, if you plot that it probably makes like as a messy graph as we have now so there's something we have to clean up. So, let's start going into the operations and doing exactly that. We can start by select count. We have a few operations available, so feel free to explore once it's out but select columns winds it down to just what we want.

A

So, let's reduce this table down to just years. Code Pro and current computer and I saw one of the questions I put in like we want to have some at least fun color. Not just more experience means more money. So what what else could be in here? um That could be interesting, so I found. This is a great question. What do people do when they get stuck and I?

A

As you see I like to go for walks, so this will be my go for Walk activity that I want to actually go segment, people on which is like one of the simplest segmentation. Usually you have way more elaborate ones that, let's see if people go for work, so we have the MS stuck stuck stuck new stack this one.

A

So this should give us you'll see a preview everything that's being removed. Let's apply that. So this is where we end up. If you're, only three columns much easier to read, um let's next one remove the missing values under find and replace, we have dropped missing mirrors. That's a very typical data. Science, cleanup problem of just I. Have data I need to wind it down to the data. I want to actually look at in some cases. You want to replace it differently. So there's definitely several ways. You can do this in this case.

A

I just want to drop them, so apply, you saw which columns will be moved so we're just slowly working our way towards cleaning this up and what is actually the outcome like will? We just say the CSV and say magic happened and we clean it up. No, it's a the great way about data Wrangler. Each time we now did something it actually generated the code. So in this case it generated a drop not assigned.

A

Then in the four set of columns it actually selected The Columns we wanted so every one of these operations. We do ends up being code. So when I later on switch to the new stick, overflow survey, which hopefully has a similar format, I, can still use the same code and apply the same. Cleanup and I get hopefully the same data out. So it's a repeatable process and somebody can actually review it versus the Excel version of I removed stuff, so we're in the operations. Now. No, we have we're close enough.

A

We have a comp yearly, one that still needs cleanup, so I guess if I would know python I would remove that dollar sign and remove that comma and I would have a number but I, uh I I, don't totally know python, but not that that, while um so, let's use Flash flow, flash floods where data Wrangler really shines, Flash Fill is a custom AI that can extract patterns from an input that you provide and we'll apply that pattern to a whole column of other data.

A

In this case, let's start by naming the column, this would be column yearly, just something shorter and actually selected converted from yearly.

A

So now it asks us give me an example of how this one I'm just going to show you this one should look like so five one, five, five, two, it's a number simple: let's do, let's do it so now, flash field will will look for the pattern, apply it to the first input and then apply it to everything else and Magic happens. We now have an actual number. This is typical hard problems in data science now become just matching patterns Within within the dweller. So let's apply this, so we have a cleaned up column.

A

We can actually drop this one and let's look at the new stuck version. So this is a other data science problem, that's very common! You have a survey, it's multiple choice. People click other things, but you're kind of interested in only one of them.

A

So the way to to could just uh string match uh do some other things, but I'm gonna use the formula for multi-label text binarizer. This will split this input, the column and for each value it finds for each string. It will create a new column which will make it really easy just to apply. Machine learning apply several heuristics across the set in this case we're only interested in one, but it gives us a really nice zero and one just classification of the column.

A

uh I guess, of course gives us way too many for what I'm interested in so again we go in so the column and select only years code Pro. This should be go for walk and we have com yearly.

A

So this looks great and I'm going to keep forgetting. Is making this an actual number because see numbers. Look. Nice numbers get a histogram. This is just unique values, so it's still a string change column type. Now it's a clearer number. We can just convert it with change column type and convert it to U in 164 yeah. That should be good yeah. So we have the preview. We can always know what what's the output and check up front. So now we have well that's no! No, we still don't have something good.

A

This is still broken, but we can actually go in and just hit filter and now, with the help of the histogram, we can understand, like let's try everything less than 500 000. might actually give us five hundred thousand give us a better histogram, so we can start cleaning up. Maybe we can still go down to 30 300 000.

A

and get closer to what we're looking for yeah. This is a much better histogram in proper data science. You want to think about how you remove people and who you're removing this is. This is demo so calling out the caveats.

A

So we apply this. We now have three nicely cleaned up columns and we're going to have questions afterwards. So we've been hit me up afterwards, so this is nice. Now every company says we checked in before now we have nine steps that get generated, that we can actually export and the way it works, export code back to notebook and exit. This will now bring back some space, bring back this column of the cell into notebook and move it up under where we had our head.

A

So what does this do? There's a clean data function that generated and all the code we saw before nicely lined up and then creates this new data frame for dfclean.

A

We can now go in and use this new data set in our plot. Let's see how it actually looks like we can execute this, this line and yeah. This is what we just saw in NATO Wrangler. So it's perfect, and we know this is now comp yearly. So we can use that column down here and we can see how this data plots.

A

So this is the graph I wanted, um but let's see what co-pilot can bring to a conversation. Of course this is on the AI track, so copilot has to be mentioned and in notebooks it works amazing compared to other code, where you run it, it will know the code, it might know some of the variable types depending on which language you use, but in notebooks it has also the state of the whole notebook and the background. So if you look at variables, these are all the things they're already loaded up within.

A

uh It will be running, including all the columns, so vs code knows it. But let's start now with just same plot but wider.

A

So I can iterate if I have a graph that looks good. I can start iterating on that graph with copilot and just continuously add and advance and learn from it now I know aspect is the way is an argument that I can use in the future to make my graphs more readable when there's more data.

A

So what if we take the same graph and then add color for go for work, option, I, don't know that works. So apparently it's hue, so now I know U I can use to color my graphs if I have to in the future. So let's see how it looks like, though,.

A

Yep exactly what I wanted. It's like! It's really fun with copilot, just to go in start from one graph start from one analysis and maybe give it some more complex things.

A

So, if you're not doing code and then do another, one is box plots we want to because we did want to answer this question of how do we is? Is walking actually more advantageous for some so box plot to compare.

A

um That's not, we want compensation.

A

Across not years of coding, but across go for a walk versus no yeah. This is compare box plot from Pure conversation, no box plots from school, so that's traditional work and then yeah. This is box plot. This is good and it X is using this column and Y is using comp yearly.

A

Let's see it looks like yep, so it's clear. We found proof that people who like to go for walks um have higher comp. So don't don't quote me on that. I first have to review this with my colleagues, but let's do another one uh same plot same plot, but for years of coding experience the amazing part is once co-pilot really guesses, which columns you have. It becomes really magical, so same plot now puts in years use Pro.

A

So it starts being those conversational demo that we had in the keynote on the first day, where you're more talking to your notebook I ended up when I did the demo I just ended up typing my comments and executing them which trying to help our EVS code will read my mind and put in the right code. So this is where this eventually brings. You you will just try to explain the best as you can, which visual you want to put up, and then it will come.

A

So there's a lot of great C1 graphs. It has a very rich API, there's a lot of great Seaborn, a lot of pandas apis and co-pilot really helps you over time to just get to know them. There's there's lots of uis that will come up, for example, pair grids, another one peer grid to compare conversation.

A

It's multi-line perfect, so peregrid is there's just so many. This is not what I wanted. So, if you don't get what I wanted I highly recommend the hovering and then open GitHub copilot, you actually see all the recommendations that it came up with. So this is this looks much simpler pair blood on clean with the U. So we're going to accept that one and rerun it so it is, is a trial out and you can you can play around with it anytime. So, as you have, as you have one function, it works.

A

You can try the next one. So that's one way to do data science and to keep leveling up. You have your notebooks in vs code, the place where you also code, it's easier than to pick it up and write it in Python and all the other places you have and it's it's the best on GitHub as well. So in this case now we already have it open. So I could go just in here commit it back. Did it?

A

I can now go into the vs code, built-in PR experience where this already shows up so the same div. You previously saw on GitHub also works on vs code, so you see all your changes. You can start commenting on them and now I can leave a review for okay got some insights.

A

I can submit I can see the pull request again within vs code, including my comments and now I can say ready for review, so somebody else can come in and actually look at my notebook to see if it makes sense, so all that with India's code without leaving it ever and I can later on, maybe take this into a python refactor. It do actual debugging.

A

So if you want to try it out code spaces for data science use code to explore and analyze notebooks is built in thanks to Jupiter and many other notebooks, and then data Wrangler will be out end of this month. If you already have vs code and follow us on Twitter, you will hear about it, but definitely try out copilot in notebooks, it's magical so, and it makes it really an expert within data science. Thanks so much.