GitHub GitHub Universe 2022 - AI, 21 Nov 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Building an AI artist with Codespaces and Replicate - Universe 2022

Description

Presented by: Zeke Sikelianos

Deep learning can do extraordinary things: it can drive cars, write code, and make art. But it's still really hard to use, and many models require specialized hardware to run. Thanks to the new GPU support in GitHub Codespaces, working on these models just got a lot easier. Replicate is making it easier to build and run machine learning models, so more people can use them to solve real-world problems. In this talk, Zeke will show how to use Codespaces to fine-tune a generative text-to-image model and deploy it to the cloud as an API. Check out Zeke's resources here: https://github.com/zeke/universe-22

As always, feel free to leave us a comment below and don't forget to subscribe: http://bit.ly/subgithub

Thanks!

Connect with us.
Facebook: http://fb.com/github
Twitter: http://twitter.com/github
LinkedIn: http://linkedin.com/company/github

About GitHub
GitHub is the best place to share code with friends, co-workers, classmates, and complete strangers. Millions of people use GitHub to build amazing things together. For more info, go to http://github.com

A

A

Hey welcome to GitHub Universe today we're going to be talking about building an AI artist with GitHub code spaces and replicate. So all the slides for this talk are up on GitHub, of course, at Zeke universe 22..

A

So, if you want to follow along, you can hit that repo, my name's Zeke I'm, a machine learning engineer at replicate, which is a small startup in the Bay area, but I used to work for GitHub for a number of years before that I worked on electron and atom and documentation and internationalization and all kinds of really fun things. So it's really cool to be back here at Universe. Let's jump into what we're going to talk about today. Stable diffusion is a generative text to image model.

A

It's an open source model that was released a few weeks ago and is sweeping the internet right now. Let's also talk about replicate, which is a cloud API for running models, we'll cover GitHub code spaces, which recently announced support for gpus, then we'll jump into fine-tuning our own model using a technique called textual inversion and, lastly, we'll deploy that model to replicate as a cloud API. So, first, let's talk about stable diffusion. This is kind of the quintessential image.

A

That's become popularized by open ai's Dali, and that is an astronaut riding on a horse, and that is a completely non-existent image and sort of a proof of the idea that a machine learning model can generate images that have never been before seen in real life. So this is stable, diffusion running on replicate.com. You can see that there's a web interface here where anyone can come to the website, put in a text prompt and generate an image, and this is a an image of a 19th century otter gentleman.

A

So one of the most exciting things about stable diffusion is that it's open source, unlike other sophisticated image generation models like open, ai's, Dolly and mid-journey stable diffusion is completely open. Its source code is open and its weights are open, and what that means is that there's been a lot of room for really exciting innovation in the open source space around around this model.

A

Stable diffusion generates very high quality images, unlike any that we've ever seen before from existing open source models, it can also run on Modest Hardware, that is to say it can run on your M1 laptop, for example, or you can even run on some machines that only have a CPU. Most importantly, stable diffusion is really fun to use. So, let's look at some of the innovations that users in the open source space have created, with stable, diffusion, first off, we'll look at tiling images, so this is another model on replicate.

A

It's a fork of stable diffusion that is able to generate tiling images. This is an example showing oranges and pomegranates on a Blue Table tree bark, Mossy, runic, bricks, ocean waves, red hot magma, a field of grass with little yellow flowers and a line cut print of cats and dogs. I could do this all day, but I'll stop and switch to the next.

A

Let's talk about another innovation in the stable diffusion space in painting in painting is a process of taking an existing image, erasing certain parts of it and allowing the machine learning model to replace the empty parts.

A

So here in this example, you can see that we are erasing the cherries from a bowl of cherries and changing the prompt to a painting of oranges and then again erasing parts of the image and changing the prompt again to bananas and regenerating a new image in painter is an open source app and, of course, the code is available on GitHub all right. Another really exciting innovation in the disabled diffusion space is animation, so someone has had the brilliant idea of taking the generative capabilities of the stable diffusion model and generating animations with it.

A

So this model can take a starting text prompt and an ending text prompt and interpolate between to create an animation.

A

The possibilities are really only limited by your imagination,.

A

So we talked a little bit about stable diffusion. It's a really amazing model, and people are doing really interesting things with it. Now, let's talk a little bit about replicate, which is a cloud API for running models. Replicate is a website and a set of Open Source tools for running machine learning models in the cloud. You can run models that other people have published, or you can publish and run your own models and they can be public or they can be private.

A

So we saw this image before this is the replicate website and anyone can come to this website and enter a prompt and generate an image, and that's really a great way to familiarize yourself with a model to just jump right into the browser. Look at the inputs and see the outputs, but what's really interesting about replicate is that it has an API.

A

So once you've created a model or you're using another model, you can take the capabilities of that model and integrate them into your own app, your own mobile, app, your own website, your own Gallery installation, replicate, has lots of models, not just stable diffusion, but lots of other models that can generate images from text or models that can take images and generate prompts from them in the opposite direction. Models that can take low resolution images and turn them into high resolution images models that can remove rain or fog from images or restore old photographs.

A

So we've talked a little bit about what replicate is: let's talk about GitHub code spaces and what's new about them, so code spaces is essentially Visual Studio code in the browser. It's a fully functional development environment that is running in a virtual machine in the cloud today. We're going to be forking a repo on GitHub and running it into code space and using the GPU capabilities of that code, space to train a new model. So this is the repo we'll be working with.

A

You can see that the readme of the repo has all of the instructions you'll need to get up and running, but let's take a quick look at how to run through it. So whenever you're, on a repository page on GitHub, you'll see this green code button when you click that you can actually open the contents of that repository in a code space without installing any software.

A

When you come to this screen, you'll notice, there's a new Option here to select a GPU as your machine type. When you create the code space for the first time, it takes a little bit of time to prepare and boot. Once your code, space is ready. You've got a fully functional coding environment right in your browser. You've got your code. Editor you've got your files. You've got a terminal everything you need to start working on a new project.

A

Let's talk about Cog Cog is an open source tool created by replicate for packaging and deploying machine learning models if you've ever worked with Cuda or Pi torch or any of these kinds of tools and try to deploy them to the cloud. You'll know how difficult that can be, if you haven't then you're in luck, because you get to skip all of that. There are two files that you need to know about when you're working with Cog cog.yaml is a file that describes the python dependencies of your application, as well as the system level.

A

Dependencies predict.pi is a python file that describes the interface for your machine learning model, describes the inputs and the outputs we'll run this command to install Cog on our new code space and then we'll run COG so code spaces has some settings. Let's talk about those first off, we have dot files. Dot files are a really cool thing, so, if you ever are working in a bash cell, shell or a z, shell you'll know that you can use dot files to configure your shell, so you can add aliases and all kinds of interesting functions.

A

So if you have a DOT files repository on your GitHub account, you can configure code spaces to use it automatically. Also, you can decide whether you want code spaces to open directly in your browser or in your Visual Studio code desktop application. Lastly, you can configure the timeout for your code, spaces environment, so if you're going to be training, machine learning models- and you want them to run for a long time- you may want to set this all right. So we've talked a little bit about GitHub code spaces.

A

Let's actually jump into fine-tuning a machine learning model. Let's talk about textual inversion. Textual inversion is a technique for training, a machine learning model on a specific set of images. So here we have an image of a sculpture and from that image we can generate new kinds of images in different styles. This is Mona Lisa, octa-cat, I'm, sure you're, all familiar with she's, the Ascot of GitHub. What if we could generate new versions of the octocat using a text prompt well we'll need to start with some training data.

A

So luckily we have the octodex to turn to the octodex. Is a website created by githubbers that contains various octocaps created over the years by designers and artists at GitHub for training our model, we'll select a few images that are representative of the style of an octocat. Next, we'll upload those training images to our code space and one of the really cool things about code spaces is that you can just drag and drop the files directly from your computer into your code, spaces window and it'll upload.

A

Then we'll edit, our training script a little bit specifically to make it work for octocats, then we'll run bash train, and this is a training process that takes about an hour or so once it's done, we can run Cog predict and we can see our first output from our newly trained model Okay. So we've talked about stable diffusion, we've talked about replicate and code spaces and we've dabbled in fine tuning. Now, let's take our fine tune model and deploy it to the cloud.

A

First we'll go to replicate.com create, and here we can create a model, we'll call it octocats. On the next page, we'll see the Cog commands that we need to run to actually publish this model to replicate, we'll go back to our code, space, Cog, login and then we'll Cog push our model to the replicate registry.

A

Now we have our own model on replicate just like the original stable diffusion model, but this one is trained to work specifically with octocat style imagery, so we can actually provided a prompt that includes the word octocat and C novel output. Perhaps even more interestingly, we have a new API for our new published model, so we can take our octocat image generation into other domains.

A

Here's a sampling of some of the images that I was able to generate with my fine-tuned, stable diffusion model, so we've covered a lot today in a short period of time, but now it's your turn to go forth and create something with these new skills. Again, the slides for this talk are up on GitHub at Zeke. Universe 22. I can't wait to see what you create with these new models.