Internet Engineering Task Force IETF Tutorials, 24 Mar 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Tutorial: GitHub Tools

Description

This tutorial at 12:45-13:45 UTC on 24 March 2019 provides information about using Github for work in the IETF.

A

I'd like to welcome you to the github tutorial, I'm Carolyn, Donahue I'm, with the ietf education team, I'd like to seems a bit echoey. Doesn't it? Oh? You can't hear me now. You can hear me okay, folks, in the back, can you hear me a bit better? Oh so I just need to be louder. Okay, so we can cut this part out of the video anyway I'm Carolyn Donahue, with the ietf education team, I'd like to welcome you to the tutorial I'd. Also like there's going to be a survey link in there.

A

I'd really appreciate any survey results that you have any any thoughts you have on the tutorial itself, additional tutorials that could possibly be offered, and the other thing is. The edge' team is actually looking for additional help and additional resources. So if you are willing to a help, provide any tutorials or be help Shepherd other tutorials or do other work associated with the ietf like helping organize the ietf education materials on the website or on the wiki, there's just a long list of opportunities, and we would love to have you with that.

A

I'd like to introduce Mike, Bishop and he'll, be talking to you about the github.

B

C

Right is that better all.

B

Right just have to eat the mic, so my name is Mike Bishop. Hopefully those closed captions are not too distracting and might be helpful for someone and this transcribes anything like my name. We can laugh at it we're talking about the tools that are available for using markdown and github to keep track of your internet drafts, so we'll be looking at basic concepts.

B

You know we'll be looking at basic concepts around markdown, get the template and how you use github and CI systems to tie all those together, I'm not going to go into as much detail on the market, because Dan already did a session on that in the previous timeslot.

B

But if I can just get a show of hands how many people are already using markdown to write their drafts, fair number and who's familiar with git, whether an ID context or not, most people I will skim that section very quickly then, and do use github already for things similar number. Okay, so we'll focus mostly on how to get things set up and not as much on the tools themselves. But I will still take a quick look around how the different working groups are using things.

B

So things we're not focusing on the XML RFC v3, not diving into that very much. There's an excellent talk about that previous time slot. But most of this is going to output, v2 for the time being and you're also shielded from the exact version of the output. I. Don't want to get into the details of whether working groups should use github. There's a working group later this week on the best practice as if they're going to- and you know the pronunciation of the boss.

B

That did not intend to form a working group and did let's leave it alone. So I think the best tagline for using markdown. It is certainly less cryptic than XML. It's intended to be something that somebody can read through and it's easily consumable in the base text format. But then we have tools that can take that text format and spit out nice. Easy work with HTML, that's convenient for reading and the text format that we all know and perhaps love the frontmatter helps generate all the boilerplate.

B

And this is the same thing that you would find at the beginning of an XML document, produce XML RFC, basically grab somebody else's document and adapt accordingly. I would not try and know what all the different things are that you have to put up here to have a beginning document.

B

There are a couple different ways that you can do references this was alluded to in the previous session. You can be very explicit and write it out yourself, and this is Gamal. As one comment noted, you have to get the spacing exactly right if you're going to do it that way, but each document has a normative and an informative section at the beginning, or you can explicitly list out the documents and just let it pull from the database of all the RFC's all the internet drafts. Yes,.

C

B

C

B

Really don't have a good way to adjust the lighting, but the slides are posted online and.

C

B

So the comment was that it is hard to read some of the text. That's up there, probably because the lights are shining on the screen.

B

So you can take the you can put the number of the RFC and just let it pull in and populate that from the online database or what you can do is just do the inline references and not even mention anything up front, but when you need to make a reference to a document, then you just put the two curly braces with normative or informative exclamation mark for a normative question mark for an informative reference. The other nice thing about that is that you can give a different name to that reference.

B

So if I want to call RFC 7540 HTTP to instead of always referencing by number I, think that makes for a nicer output and all of these things are supported by the XML, but you can use Graham down and markdown formats to lead into it so just skimming through here with in the middle on the back sections, you used the hashes to do level well top levels. I can love all third level headings. If you go past third level, you can get really deep, but anything below third level doesn't show in the table of contents.

B

You can do cross references either two sections or figures by putting the anchor tag, so you just indicate the anchor that was provided with the section and if there isn't one that's put when the section was declared, then it just takes the title and it converts that so you can see the the third level heading I didn't give it an anchor tag, and it just takes the text of the title, drawings and tables again.

B

I would suggest, grab an example document and the ones from quick are a good instance of that, but all of those are supported by crammed down in the markdown format and will turn into nice tables both in the HTML and in the text output.

B

As far as get looks like most people are familiar with it, so, let's just skip through there. We have commits that are snapshots at different points, branches and tags.

B

So the idea is, we have these tools that let you start with markdown and use cramdown or in mark or some other options to turn into XML, and then you can take XML tar FC that can give you the HTML output, the text output. You can use PostScript to turn text into PDF and those are all great as standalone tools that you can use.

B

But when you want to tie them together, there's a tool called the ID template that you can get from github. That.

B

Just wraps them all up and uses a git repo and fundamentally the ID template is the hideously complex set of make files that tie all of this together and let you take a document and have it spit out all the right things.

B

I, don't know how well I'm going to be able to eat the mic over here.

C

B

There was a question on how well we could see earlier. Are you able to see see that as well.

B

All right it are you able to see, is the mic on or do I just need to be even closer? Okay,.

B

So what we have in this repo is just a markdown file that has an existing internet draft in it. So the tools that are there build on make so that, if I run make it's going to go and download all the necessary references and spit out now we have the HTML on the text version.

B

So if I go pull that up.

B

So it generates all the boilerplate here that you would want.

B

So if I wanted to go and make a change to this document, let's say I wanted to add some text.

B

And when I add the boilerplate from RFC 69 19 to define the terms must, but we know you won't, should consider all those other terms that we aren't officially supposed to use but use all the time. And then I will.

B

So I'll try and commit that, and it's good to tell me that I actually made a reference to 69 1999 and it will refuse to. Let me commit it because I'm making a reference to an RFC that doesn't exist. So one nice feature here is that it will actually sanity check the document when you try and commit it.

B

So I'll fix that and go back and now, if I commit it, it did the repo it's there and so that that lets you get different outputs and you have the check before you go in other things that you can do just from the local repo. If I tell it make diff, then if I had RFC DIF installed sorry, then it would build the output of the comparison.

B

From the last time the draft was submitted to the current state of the repo and, if you're, trying to run everything locally make submit, it says it doesn't need to.

B

Okay, well, apparently, we didn't make enough sacrifices, the demo gods. What makes admit does is it builds the XML and the text versions that you would need to upload to the data tracker when you are ready to submit the document.

B

So looking at github people are probably familiar with what what github offers it's essentially a hosted, git repository and then a set of workflow tools. So the way the quick working group has been using that.

B

The renown we have a readme file that lists all the documents, and this is actually auto-generated by the ID template that for each document you get a link to the editors copy, which is a live reflection of what is currently in the repo we use the issue tracker. We use pull requests to.

B

We use pull requests to manage text that people would like to bring into the documents who are not the editors. So the way we have it set up, the editors are the ones who have write access and then anybody else can clone the repo and propose changes.

B

So if I wanted to go in and take a change, I'm actually not going to go into how to use github, because most people said they were familiar with. That.

B

But the process when you start hooking github in along with circle CI, that's when you get really nice automation around all of these things. So you have your local copy of the ID template and you're making edits and each time you can build it it'll do a sanity check and produce the local HTML and text that you're interested in when you're ready to push that to github fine other people can see the markdown to you just that you've made, but github enables change traders, so external tools can observe when something changes in the repository.

B

And so if you set up a trigger with circle, CI or Travis, which are both supported, CI mechanisms, then you can have them run their own container with a copy of the ID template and what they will build. They'll take the documents, push it back to the gh-pages branch and that's how you get that editors copy there.

B

Anybody with a web browser can go and see those documents. So what that actually builds.

B

So the gh-pages gets published.

B

On github il and the name of the repo now quick, actually hasn't set up to redirect to a particular domain name, but that the generic form is your github username dot, github on dot, io /, repo name and the IV template will build for the master branch at the top and then for every branch. It will show you all the built documents of the current state of that branch.

B

The other thing that can happen on CI is that can respond to tags and the repo. So if you tag a particular document as being on draft version, zero zeros you're a 1. Whatever your next version is, then it will actually take the document once it's built and it can upload it to the data track and it can upload it to the data tracker and so your actual workflow, when you're ready to submit your document, is simply build.

B

It tag it and push the tags, and then you get the email that asks whether you in fact were the one who submitted that document and if you want to approve it for publication. So this can help take out a lot of your interaction with the data tracker and just get the publication directly from the command line. Where you're already working on the document.

B

So when you're trying to set up a repo from scratch, a lot of this is really convenient once you get it running, but the difficult piece is getting bootstrapped trying to take an empty repo and get it set up. The the ID template setup script is quite good about telling you, when you haven't done something correctly, but when I was getting started with it, I would often just do a loop of run set up. It'll tell me: oh, you haven't done this yet and then I need to go fix. It fix it.

B

There are some setup scripts that rich saul's has written that are intended to help help you get the repo setup in the first place, so it will create an account on github for a working group, and it will let a working group or an individual with an existing account, set up a new repo, but it still comes out as an empty repo. You can upload a upload, a template to it.

B

So what I'm going to do now is just walk through taking taking an empty github repo, and you can either use the script to create it.

B

You can use the script to create it or you can do it directly on gum, so I'm gonna, say I want a new repository and sure fantastic journey and just take the name that it offers.

B

So right now we have an empty repository and then I'm going to come to the command line and do the basic setup. So the first thing you do is clone it down.

B

And before anything else happens, you need to put a very basic version of the draft, so Rich's script would populate this with a with a template that has no content for purposes of this. In this demo, I'm going to copy the change that we did from the DNS service draft.

C

B

So the setup is first, you have to have already committed and pushed so get push just interact with the remote repository.

B

C

Just a question is the fantastic journey to.

B

Working group- in this case it's just the repository for the document, so one of the things that we'll be discussing in the get working group for from best practices is whether you should have a single repo for all. The documents like we've, mostly done with quick or whether it's better to have individual repos per document. I have mostly done one large repo, but I know. The tls working group has a separate repository for document and either one works functionally. It's a quick one.

B

The template can handle having multiple drafts present, so it will build each draft that it finds.

B

So I pushed that and then, if we're going to set up the template.

B

So the ID template itself is on github, it's under Martin Thompson's account and the URLs and the slides. So you clone it down into the referee called lib and it contains the setup script. So.

B

You run make and point it to that setup, location and then what the template is going to do is first check that your draft actually is present and billed and has already been pushed up to github. So it wants to make sure you have a good link and then it will start populating a readme file that contains the name of your draft and tells people what working group its affiliated with tells them. The discussion should happen on the reading on the mailing list. It populates a contributing.

B

Contributing document that points them to the note well and just helps to solve the issue of how do people who do drive by contributions on github know that they need to comply with the note well and the IPR policies of that. And it helps populate the things that you need in the repo.

B

So, at the end of that setup, if we look at the commits, we see that the setup script added a new commit that creates all these files and added a gh-pages branch, which is where the built documents created by the change by the CI. The continuous integration are going to live. But if we go back to our github repo all this lives.

B

If we look at the gh-pages branch, what we'll actually find is, even though the index.html file is there it's empty, so that relies on getting things set up in the change at non continuous integration system.

B

So once you've created the repository, you need to go to circle CI and tell it that it should actually be building from that repo.

B

So you tell it to add project and you find your new repo and it'll try to walk you through creating the dot circle. Ci config amyl, that's actually already added by the template. So when you run the setup script, that's created for you, you can just say start building.

B

So at that point.

C

B

I know what I missed.

B

It created the it created the commit on my local repo, but I never pushed that to get up.

B

So if I come back here now that I've pushed it will fine, then you'll see that it's populated the readme file, with the name of draft the links to them where the editors copier is going to be. The contributing guidelines will show up here for anyone, who's interested.

B

Now we should be able to see, yes, it does run, and so what this is gonna do is every time you push to the repo, it will build the documents and if the documents don't build correctly than circle will send you an e-mail. So that is a convenient way to keep keep track, particularly if you have lots of people collaborating on a working group repo that if somebody breaks the build you find out about it.

B

So let's say before we get to that. There is one thing: that's still missing, though. If we go to the gh pages, it doesn't actually push back because circle needs permission to do that. So when you set that up, you need to go into github settings.

B

And they hide it under developer settings to get the access to events and an access token is something that is not your password that can be used to talk to github on your behalf, so you'll create one with whatever permissions you want. The only permission that it requires.

B

The only permission that's required for the CI system is public repos, so long as it's not actually hidden. Can you take that change to okay and you'll come to come over to circle.

B

So that in the settings for each project that goes in as an environment variable now, since I already have it set up, I will go to a different repo and copy it over. The name of the name of the variable is gh token, and you paste end the token that you got from github the other. The other place where those tokens are useful is you'll need them at the command line to interact with github.

B

If you have to factor off setup with your main account, you can create a token using two-factor off and then use that token as the password for a tool that can't interact with the to learn to vector.

B

So now that the token is there.

B

Anytime, I have a build from here on out and I'm. Just gonna restart the same, build with the version of the draft.

B

It will not only build the documents.

B

B

So now it's going to push it back.

B

So if you look at the commit history, first thing it's going to show: you is with each commit, so you know when CI tries to build it, even if it doesn't have the toca and it will report back to github whether that version in that commit was able to build successfully. So you can easily see this one it wasn't able to build.

B

Then we pushed it after setting up the ID template and you get the green checkmark and every time you introduce a commit, it's going to add a new commit to the gh-pages branch with that version of the built documents committed by the ID bot- and this is what gives you.

B

We go over here and now it has built the index page with the HTML version and the text version.

B

When you're ready to submit.

B

You do an annotated tag, I, don't even remember what brew from the draft this is actually on, but I'm, not gonna, let the submission go through anyway.

B

One thing to note is that, in order for circle to do this and do the auto submission to the data tracker, it has to be an annotated tag, which means you have to give it a message to go with the tag, but the actual content of that message doesn't matter at all, so you have to pass something. It will fail if you don't, or rather it will bring up the editor and ask you to type it.

B

So, if I create the tag and now I push the tags up to get.

B

If we go and take another look in circle,.

B

You'll see that that tag is also something that it will build on and the last step when it finishes the build, is that it's going to use curl to upload it to the data tracker, which will generate an email to all the listed authors and I've already told Ben that you should not respond to any emails from the data tracker about this draft today and I'm going to cancel the submission when I get the email.

B

That assumes that I guessed right on the fact version that in the data tracker, doesn't reject it for me, so the last step is part of the build process. You can see the upload to data tracker and it's looking for a 200 okay coming back and if it doesn't get a success on the submission, then what it will do is email me that the build failed on that tag to say it couldn't couldn't submit to the data tracker. You may need to do that manually.

B

So, to recap: how we do that setup? First, you need all the local tools you have to have get and make for the template to work. Xml to RFC depends on Python from down RFC two-six two-nine requires ruby and those can be installed with their appropriate handlers. Actually apt is able to take all of these directly. So if you just go and ask for those four, you can get more current versions of XML RFC off of the PI pi installer and cramdown can come off of rubygems on the cloud side.

B

You need to create your repo either with the script or manually, and you need access tokens to give to circle and then on circle. You need to set it up to follow the repo and you need to have the access token in the environment variables if you want it to actually populate back to the repo on your path, so full disclosure, while I, have made some contributions to the IV template I'm, not the one who wrote it. That would be Martin Thompson.

B

So if this looks like a tool that will make authoring internet graphs easier for you, I would say, buy him a drink. I have found it to be a lot simpler than trying to mess with the XML, which I did at the beginning of my time in the IETF. But at this point, I am marked down for basically everything mostly for reviewing the slides.

B

Later I've got some basic like what commands you would use and get for different tasks, and that is the end of what I've written we're a lot we're way under time, because I skimmed, through the things that look like most people, were you're with. But if there's anything that you want me to go back and go into more detail on I'm happy to do that happy to stick around the Minister questions.

B

And the deafening silence in the room is either okay.

C

Sebastien go ahead, hi.

D

Mark expert village I was wondering if there's a protectorate, mint I got it through github versus other forms that just it lab, or somebody goes out there.

B

Github is what we have primarily used. I believe I believe that it would work with any remote git repository so long as it can integrate with circle, but I don't actually know how reliant circle is on using github and not the other versions and.

D

Then, maybe is sort of interest result I also suck at this report for Travis. Yes,.

B

It is still present and supported, but I'm not sure whether the newer features like auto submitting to the data tracker are supported and Travis or not.

D

Okay, thank you.

B

We know we have experienced that circle is more reliable for building things and also it passes more information into the environment of the build that the template is able to pull on. If you open up some of the make file scripts they're part of the template, you can see comments like if this is Travis. This is not populated. We have to attempt to guess it.

D

E

Rich, so Travis might also be on its way in decline, because most of the staff has been fired. I.

B

Imagine that doesn't help.

D

Actually, on that, I could continue yeah, please the the reason I asked about github and I mentioned it lab as well as because a good lab it could be self hosted and there wouldn't be on you know, startups going out of business or things like that. I mean granted Microsoft owning that how this pretty reassuring, I guess in the long run, but for some of the other tools that are related to it. It might make sense for by Jeff tools to also be open-source and be able to be self hosted by by by authors, yeah.

B

So a lot of that discussion is likely to happen in the get working group later this week. Their Charter is to come up with best practices for get git and github use at the IETF we're chartered to focus specifically on github, but the question of whether any of the tools need to be self hosted and managed by the IETF. Even if there are third-party equivalents is definitely going to come up.

B

Any other questions.

C

B

I will stick around for a while if anybody wants to ask anything not at the microphone. Otherwise, thank you for coming.

B