GitHub Open Source Friday, 1 Mar 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Next level repository management with Octoherd

Description

The management of multiple repositories in an organization can become time-consuming.

Join @bdougieYO and @gr2m in learning to leverage the octoherd/cli to manage multiple repository updates all at once.

0:00 - Start
1:32 - What is Octoherd?
9:19 - Octoherd Demo

As always, feel free to leave us a comment below and don't forget to subscribe: http://bit.ly/subgithub

Thanks!

Connect with us.
Facebook: http://fb.com/github
Twitter: http://twitter.com/github
LinkedIn: http://linkedin.com/company/github

About GitHub
GitHub is the best place to share code with friends, co-workers, classmates, and complete strangers. Millions of people use GitHub to build amazing things together. For more info, go to http://github.com

A

uh Hope you enjoyed the last conversation if you were sticking around uh to hang out with us. This is open source friday, uh a time when we sit and talk to open source maintainers about their projects, um their pain points, their wins, and uh today we've got a returning guest you're, actually, our first returning guest gregor. So congratulations um do you want to give a quick intro of who you are, and why are you here.

B

Sure yeah, thank you so much for having me again, I'm gregor I am the maintainer of github's arcticat for javascript and typescript, and I do maintain several established open source projects in the node in javascript, community and yeah, and I'm here today to introduce you to octoherd, which is like a project that I just started out and I thought might be interesting for your viewers.

A

Yeah- and uh I think this is probably the the earl the I guess the most, how would I say the as far as age goes: it's probably the youngest project that we've had on the on the stream. uh But we've had a lot of conversations about this and um I just wanted to point out real quick folks. If you did not see the original conversation that we had around automating open source, main uh maintenance with bots and octokit, uh there is a youtube video, so definitely check out the youtube, the github youtube channel.

A

uh You can check that out, but today we're gonna talk about next level, repository management, so, like we're moving to the next level, uh and if, if you don't mind, I guess that we could just go ahead and jump right in to octaherd and what octohurt is so. You have a brief explanation, but uh do you want to talk about like what? Where, where it came from and what it's sort of doing today.

B

Sure so it came from me creating lots and lots of like atomic little scripts to do tasks, because I'm an engineer- and I don't like to repeat myself so soon- I see myself like doing a thing three times, I'm like nah, nah, nah, I'm going to script this and because I happen to know the kitab platform apis very, very well, because I built all this tooling in a position where I can create the scripts very quickly and my I probably a hundred folders of these scripts and I just create them once and then throw them away and never use them again.

B

But over time I just seen like recurring patterns, and I thought it's about time- to put it into some kind of form, which makes it easier to share with others.

B

And octoherd is basically it's more a pattern than any kind of framework. It gives you in the possibility to create a single function, which gives you an octocat instance and repository contacts and options that the user passed and then you can run whatever you want on each repository that you loaded with dr heard um cli yeah.

A

B

It you know it's not like. You need to know what you're doing it's. It's. It's not like super end user friendly. But if you are a coder and you know the github apis a little bit, then this is a great tool for you at this point and.

A

I wanted to provide some context real, quick too, as well for folks who don't know who you are. um You maintained this project um started this sdk called octokit rest.js, it's the node library to manage interactions with github's, api and web hooks and and then and then some but my context with this is I actually have- and I wonder if I have a terminal, that's ready to go.

A

uh I have actually have a repo where I have scripts for interactions with the github api and um I don't think the terminal is going to come up, but actually I'm not sure why that's not coming up but we'll we'll move on from there, but I have a basically a long script of like if I need to manipulate a bunch of repos, for I have a second account called not be dougie and I wanted to archive all those repos, because it's it's where I put a bunch of junk code as I'm testing stuff, I fork a lot of stuff and manipulate actions and in pro-bot, apps and stuff like that.

A

So I wanted to archive all my all my projects on that one organization account. So I have a octokit script that I just run whenever I want to archive a bunch of projects. um I have another use case too, as well, where honestly, I'm thinking this, I know we oh there's my terminal, uh I'm not sure why that took forever, but um I've got this other thing called github craftwork, which is a way for me to teach automation uh on github through github.

A

uh So we got this little repo, but then we have all these other repos uh that everybody who starts the workshop with me or does it on their own, creates a repo as you go through it. So I wanted to write a script to transfer repos to the owner, so that way I can just get rid of them out of the organization.

A

um Those are scripts we don't have to solve today, but these are scripts that I have sitting in a folder uh in a basically a javascript file to automate the process of this, because at this point like it, just gets to the point where this is too many people in the organization and too many repos that I need to do a script where I just look for the word ci and not my name, and hopefully people named it the proper way which we have one person who didn't and then it makes it even more challenging to to run a script like that.

A

But um since we're uh we started a little bit late, uh I'm curious. If you wanted to jump in and give us an example of things that we could. We could automate through octoherd.

B

Sure yeah, I just wanted to say regarding octokid: it's not only for node, it's also for javascript and it works with dino now as well. Oh really, I'm very interested in dino, um especially for cli. This is really great. Did you.

A

uh Okay, did you work on the the dino um or dino denno option.

B

So yeah um it's the same source code right and you know it's just web platform and octokid works in the browsers, so it should mostly work out of the box.

B

There are some differences and also do not support typescript out of the box, so there was some things I had to do some problems that came up with you know they didn't come up with the other platforms, but these are resolved now, so you can run dino scripts with typescript support um right now and it's great because it's basically like, like a cli with the browser security sandbox.

B

So you can just run a script that is just hosted on a random website and it won't be able to harm your operating system because it just can't randomly read and write files on your system and stuff.

B

So it's very interesting and it's actually perfect use case for of the herd, and I really want to make octopus compatible with you know. So at this point you know you were talking about your scripts and basically you could just you know, take one after the show and just try to convert it. Yeah.

A

B

Like I have like a few, you know scripts that I created as examples here and I just want to start with one that is a little bit more complicated because it has been asked for and it's a way to synchronize branch protections. So you know you use the octaher cli and you pass in user authentication token, and then you pass the path to the script and then you can pass on what repositories do you want it to run? So you can just pass a star and it will run on all repositories.

B

You have access to or octahertz star runs it on all octahedra repositories and after that are user options that are specific to the script. In this case I want to load the current branch protection settings of octoherd cli and then just apply it to all of the hurt repositories right. So this is our use case. Like branch protections are kind of a pita, so um we want to load it from one repository which is cli.

B

Let me see what they look like octoherd cli settings branches here on the main branch, so we require pull request, reviews and we require status checks and that's it. So it's quite simple, but we don't have the same on, for example, the scripts folder. I think.

A

uh I'm also understanding the name now on the reason.

B

A

um It's clever because, as you um like, did you have maintainers in mind when you build this maintainers who have multiple projects like yourself as you're, trying to manage the herd absolutely like.

B

Maintainers, this is for you, if you are, you know if you do tons of open source maintenance or maintenance for your work and your funds of repositories, and this will save you a lot of time all right. um So here I just deleted the branch protection and now let me just run, run the script as it's shown here.

B

Let's see if this will work, so I have to pass in my actual github token here.

A

Okay, I was going to ask if you're going to actually paste in a token, but it seems like you, you came prepared.

B

Exactly I think the rest is good. Oh no! Actually, because I do not have access here, I'm in a different folder, okay,.

A

B

Should work so this would this needs to be like a path of an actual script and I think that scripts and then sync and script. So this is the file that I'll be running, and I will show you the file in a moment, but let's just see what's happening, um and then I can also up to her. I can just enable more debug logs and now play you know, let's pray to the demogods as well.

B

This will actually work.

A

B

Yeah so with mpx, it installs it on the fly and just runs it. So if you have known installed, you can basically just just run it and then again like with dino. There would be no install like you would need to install the dino library, but you don't need to install any binaries or anything, and the script could be directly run from a website all right. So it's running here we have some locks, so it's loading all the repositories.

B

um It found the branch protections here and there's some blocks that I should remove anyway um and now, like one thing that it does, for example, is by default. It asks you to confirm any destructive request, so whenever you do a request, which is a delete or a put or post it asks you to do it, you know just because you don't want to break stuff, and this is not on git. You cannot really just simply undo it. So here it asks me like do.

B

I want to update the branch protections for after cli and no I do not because this is already my template repository, and this is up to her script. So here I can say yes, and so it sent a request. You know it has like the debug information here with the properties that it did and then it continues, but that's already it so I can just you know, say no to the others and then go back to my browser and in theory I should now reload, and here is my branch protection so ta-da.

B

um This is the same protections that I had on the default branch on the octa. Cli.

A

So far so clear, yeah yeah, that sounds great. I'm actually intrigued to see what the the script looks like too as well. um Is there, like I understand under the installing the cli locally, uh but the script itself? Do you need to pass in parameters for it to interact with the cli.

B

So you pass the path to the script right.

A

Now, okay, um but inside.

B

The script file it doesn't.

A

Need to know anything other than it seems like octo kit.

B

Yeah, it gets basically, it gets an octokit instance um passed through here, which is a custom optical instance from octoherd. So it has some some features built in like pagination and throttling and request retries and the confirm um question the confirm prompt that you have seen, and we probably add some more stuff to that.

B

So and then you know it's checking. If I pass this template um cli argument and if it didn't it would just pro so in the script now I have octokid and I can do whatever I want right. So octokid has request method, and here I can you know just whatever like get the get the current repositories, for example, um and then what I get from the script as well is the repository object.

B

So I can directly access that and say, like repository owner, and you see you get like the whole intellisense, because octokid has very good typescript support for all the endpoints yeah. So you don't need to guess so you can just you know. I know it's repository and then its name and then like. I could store the data like this so and now I would get all the information about the repository which I already have, so I don't need to do it, but just to showcase.

B

Basically, all 600 plus endpoints are here for you and you can do whatever you want and you can also um use graphql requests as well with the graphql method, and you just pass in your query here.

B

So in this case, I'm first checking if I already loaded the template bridge protection settings.

A

B

If I didn't, then I read out the parameter, and here I actually get the repository, the template repository and I get the default branch and for the default branch. I'm loading the protection and then I have to kind of translate the response data into input parameters, because the format is not the same.

B

So what I get from github looks like this: it has like url properties and they're like enabled objects, and what I want to for the update request is just the false and the true, for example, and I cannot pass all the url stuff, so here's a method I created to clean that up.

B

So once I have that I have my template branch protection settings and then I go here through the current repository. So I load the branch protection settings for the current repositories default branch.

B

And then you know I log it, and if there are no branch protection settings, then I then we are fine and I create the branch protection settings and otherwise I lock the current branch protection settings out. So I can compare them and that's it.

B

So you know it looks more than it is, and this is a pretty complex script yeah, but uh yeah there's a lot of stuff happening that I don't need to worry about, basically, which is what the whole octavert cli does for. You.

A

Yeah and um yeah- I love it and then so. My next question is uh so you're a maintainer of multiple repos in an organization you have, let's just say, 25 repos and you want to now you want to enable auto. So you want to get auto merge, so you need branch protections.

A

um So can I run this on a number of repos at the same time, or do I need to update the script to do this as well.

B

No, you can just do whatever like we can just do it on octokid right now, right, like the script that I had um like. I have this installed, so I don't need to use npx.

B

So it's just a parameter like instead of octaher.

A

B

Just just run it on octokit and and use like, I don't know, um templates and then octokids rest.js as something as a template.

A

Oh, it should just work and that's it right. Okay, so now now I understand I missed this part where you actually pass in the parameter of the organization, but your template is not. I was assuming the template was something you were pulling from the file structure somewhere else, um but your template is actually you're, mirroring the template of octocat rest onto the rest of the octagon.

B

Exactly genius.

A

B

Got it like, how do you, how do you define the branch protection settings right? I mean you could do it whatever you want, and you know, as you said, this project is very early right now and once kind of the foundation is solid. I want to make it much easier to create the scripts and to fork these scripts and to share the scripts.

B

So, instead of you know blowing up the script of a thousand options, I want people to come in and just fork. My script yeah.

A

B

Then just say you know, I want to read out the branch protection settings from environment variables or some kind of file or whatever, and that should be easy to do. And then we have, you know let a thousand branch protection scripts bloom and you just pick whatever you want and if it's not there pick what is closest to what you need use it as a template and edit it so yeah. I better stop there because I don't want to update any ruby protection stuff.

A

um uh So I guess my next question is then, uh because I know we're short on time. Where can folks find scripts or suggest scripts um to for octokid or maybe like? I guess, let's just leave it at that, where? Where can people find existing scripts but also request scripts.

B

Yeah, so um right now the best place is probably the octaher scripts repository and they can just check out the scripts that I have created here and you know they can create issues and ask for more, and I think that you and I we had some discussions ourselves yep um I've also just yesterday found out that stowe uh created the same repository and they have you know just did the same thing and they just created their own repositories right here.

B

Scripts right here, so that's easy enough, um but I think what I want to do next is just publish scripts to npm one script per package and then just make it very easy to create a script and to maintain these scripts and to test these scripts and to easily update these scripts for compatibility with new versions and so on.

A

Yeah, so that's.

B

Probably the next thing that I would work on so yeah the best way to start is you have an idea, you want to do, create an issue and then just try to build it yourself, and I will help you along the way and I'm pretty sure you come up with use cases that I didn't consider yet so we can, you know iterate on the foundation and then the next step is, you know, to make this script more easy to distribute and to share yeah.

A

B

Once we have that or in parallel, you know, as I already mentioned, I want this to be compatible with dino, which will make it much easier to run these scripts, for example, in github actions too, because.

A

B

Are no dependencies right, they will just run and you don't need any build step or anything.

A

Oh, that's and.

B

Actually, pretty powerful.

A

Real quick uh with github.

B

A

What not a lot of people know, but github's grip is a it's. A optical powered um get up action and um you could actually run shell commands uh within your github actions as well. So imagine if you had a cron job of like keep everything consistent across the the board for all your your repositories, uh I saw stows um labels, something that I try to take consistent with across my repositories, which I'm not really good at is keeping labels consistent.

A

uh Keeping the other thing is like the contributing md like I have the same rules in all my projects, exactly, I don't keep them up to date everywhere. So, if I update my sort of, I don't know my pinnacle project of like in your case, maybe octaget rest- uh you perhaps you want to you- want to update the rest of the octaget libraries with the same contributing md, the same code of conduct, the same, etc, etc.

B

A

B

Yeah, but you know then oftentimes it's like in 90. You want that, but in some cases you don't want to override the contributing mds that you updated specifically for this right.

B

So it's something that you can kind of automate to an extent, but then you want to keep some control, and that is something I want to take into account with the cli.

B

So it would be nice to kind of have diff used, for example like hey. If you run this command like this, is the current contributing md, and this is the new one. Do you want to have this update, but instead of going manually through every repository, you will have it right there and you go step by step and you don't lose contacts um so yeah actions are powerful and this is definitely a good target for that. If you want to automate them also, what I didn't show you this, you can write scripts to gather data.

B

It's actually an idea from you yeah that you had right. So I created this um this script to find this is not mine.

B

To find releases.

B

And then you know what it does is. The whole logging is um at the end of my at the end of the script. It will look like a path of um where it created a file with the with the log data, and this is just newline delimited json, and then you can use this json and do all kinds of analytics with it. So you could use like jku or what I did is.

B

I can quickly show I uploaded it to big query: okay, um basically, the output of octahertz right and then you can load a bigquery as a source for this data studio and can very quickly create this kind of reports, um and that's something that I think I'm also very interested in, because for one you have these scripts, that you want to update stuff and keep it in in sync and the other one.

B

Is you kind of want to run this complicated or like get data like once per week, or something which you could also run an action and then just write the raw data log somewhere and then have some reporting on it? So you can do that with octoherd as well. That is within the scope of the project and then, ideally I want to be able to share these reports too. You know it could be, it could.

A

Be very interesting, yeah, it's valuable information and I think it's the the challenge is the the limitation of like github's rate limiting like we do, have a rate limit uh so, like you can't just scrape all your stuff all the time you know every hour um yeah, but to be able to have scripts that are ready to go and models that folks maintainers can share across organizations.

A

um I like this and I like the fact that these um we're starting to see. Well, we have this script, which is octoherd script, but also stowe, who also has an octahedr uh script as well, um so seeing the the collaboration um is nice. So I'm curious: what's your thoughts on centralizing all the scripts? uh Will it be like a database uh in the future for folks to look forward to.

B

Yeah, so I think why I want to go is to make it more atomic. You know before I call you talk about monorepos, basically, script.

B

Repo is a monorepo right now, but I wanted to be um separate repositories or even just gists, but I think repositories would be great and then I would probably utilize a github action which you install on all these repositories and that way I know all the scripts out there and I can also help you run your tests and you know, run run your tests against a new version of the cli and so on automatically basically like a dependent board specific for octoherd scripts.

B

So that's my idea. Right now and with that information I could also create a website where people can just search and and just find the script whatever they want and then run it directly.

B

So, ideally, they would have the dino cli and you would get like one url that you paste and then it will include both the cli logic and the script, and you can just run an out of the box um yeah and for me, the end goal of this is not the end goal, but kind of the next goal would be to just have a nice web ui, where you run this stuff yeah, because octokid works in browsers. So why not?

B

And then you can provide a nice ui like an oauth login and you know better, like ways to define settings or like automatically provide the reports that could be published together with the with the scripts themselves. You could basically publish like a report script which reduces the data.

B

It is something usable um and then you know you can kind of just have an online editor, ideally with all the intellisense that you've seen in vs code load, one of my scripts or a script that exists and just edit it and run it right after and rerun it and rerun it until it does what you want yeah once we have that, then you know we'll be at a point where you can really just create a script like this in five minutes and run it, but also leave it in a place where others might find it, and even if you never use it again, they might need it or they might use it as a starting point for their own script.

B

And this is what I want to get with octoherd. So it's a great time to get uh involved.

A

Yeah yeah folks, if you're looking to contribute to a project, uh provide insight, some value or just get value out of it. um The way to get value is by asking questions and uh seeing if this thing will work for your projects, um so gregor um uh we're gonna have to round up the conversation now um folks, if you want, if you're interested, uh definitely check out the the meetup page. uh This is where we're gonna be posting our future conversations um with other maintainers.

A

I believe next, uh in the next couple weeks, we're gonna have a github action. Maintainer uh talk about ai powered github actions that do visual testing for your websites. That's in that textbook! Next week and the week after we have the maintainer of octoprint, which is a 3d printing library, sorry, 3d, printing, open source project, so gina is going to be coming on to talk about that as well, so um like and subscribe hit the the bells and all that other stuff and uh we'll we'll see in the next.

B

A

Gregor thanks for coming.

B

A

Thanks for having me.