DFFML Development Skills, 16 Apr 2021

Previous Meeting

⏯

youtube image

►

From YouTube: Testable documentation with Sphinx consoletest extension

Description

Intro to consoletest, a Sphinx extension that can be used similarly to the doctest extension to test code block directives Sphinx and ReStructuredText uses. https://github.com/intel/dffml/blob/master/dffml/util/testing/consoletest/README.md

A

Build documentation, websites with sphynx um and sphynx mainly works with markdown or restructured text files, um and so these restructured text files they they have all these little. You know keywords and stuff like this, creates a heading, and this creates this little code block that that highlights the syntax is bash, uh which is the shell right and so you'll see here. This is bash. Shell, syntax um and, and we've got nice little highlighting and- and it allows us to write these these tutorials and things all this documentation.

A

That looks good, but we can write it in these simple files that can be tracked and get um which is nice because then um you know you're not putting things in like a wiki, which is a format that you can't do reviews on right. So if, if you track everything in git, then you can do reviews and editing through your regular pull request, workflow, um okay.

A

So now what we had is we've got a bunch of tutorials on um dffml, the same sort of thing: um let's see, what's one that's okay, so this is actually the one that is the essentially the the the uh the demo of the allow list tool. um Basically, get repo maintenance, tracker, abstracted into maintained or no maintained, um and so what we've got is is you know this is the app that we're building right and we take people through the tutorial and we say: hey write out these files.

A

You know you're going to need to create these files. This is your applet you're, starting with um and now go through and and and then we're going to modify the app right so we're we have our existing application right and now we're going. uh We set up our database yada yada and now we're gonna start modifying it so that it's going to you know, have machine learning involved.

A

So we basically have all these little console snippets for them to run. You know in these files for them to write out- and this is sort of you know a standard thing that you get with lots of tutorials right, hey, write, these files um run, these console commands yada yada, so a problem we ran into is obviously this gets out of sync with the code rather quickly, because if you change stuff in the code so say you change.

A

um Where could we go here so say we change one of these command line parameters or the way that the command line parameters work. Now unless we go and and retest this tutorial manually and copy paste all these code blocks into the console, then we don't know that this tutorial has broken, and so, when you have lots of people working on a project, you you end up in this situation where you, you are really relying on your automated testing, to tell you when things are to tell you what needs to change when people make changes.

A

So if I make changes in part a of the code base, I may know that that docs associated with you know part b, actually need to change, because there's this trickle-down flow um and- and so that's- I'm only going to know that if I have some kind of automated test that breaks when I change uh when I make a change right, so we needed a way and the other thing is that you, you really need some way to to have the um because there's one thing to write test cases and then there's another thing to write test cases that are the exact same stuff.

A

That's happening that you're actually displaying to users in the documentation right, so we started out with a bunch of python files that basically just said. Okay, these are code snippets. We include the code snippets, we run them using the python file, but now we have this sort of. uh We have this intermediary land where okay. Well, if we don't update the test file, then you know if we change the documentation, but we don't update the test file now, we're not you know the user, we're not testing the same thing.

A

The user is seeing so what we needed was we needed something that uh allows us to test exactly what the user is seeing uh and so sphynx, which is this utility that that lots of people use like you know to to generate their documentation. It has something that's specific to python called doctest and now doctest. What that does.

A

Is it basically lets you go through and if you're familiar with python, you you've seen these little arrow arrow arrow things, which is a python prompt, and so you can write your code just like you would write it. You know, within these code blocks here like clear linux. People do with bash, but it only supports python. So you can write your little python examples and then it'll run them uh using this. It has this whole extension infrastructure and builder infrastructure.

A

So, just like you would build your html docs, you would build your dock tests and so that essentially, instead of actually building anything, it just runs all these tests. um So, for example, you know here this is your your little piece of code that gets run and then, when you run the the doctest builder, which is a command line command it. It goes through and tests all your docs, and it tells you if anything failed.

A

So this works great. If all your examples are python, but if you have some intermixed, you know python code and maybe you know javascript files to write out and then console commands to run. You end up in this space where, okay, now I can't I can't test my docs, um so what we came up with uh well, what I came up with was uh so blame. The insanity on me um is this thing, which is a console test, is what I've been calling it?

A

It's think, since it's a string since sphinx extension that basically is the same thing as doc test, but for these code block console and code bash well code block console um directives, and this is what these little every time you see, one of these things with the dot dot and then colon colon, that's called a directive in our restructured texts, speak. um So it's this little python module eventually we'll we'll split it out and refactor it into its own thing.

A

I just I haven't had time to do that um and so what it lets you do is you can? You can essentially write your restructured text and then you can add this little test option and so restructured text has options in this or anything so directive is dot dot and then blank colon and an option is between colons um under the directive, and so here you'll find like the content of the thing that you're about to see, and then here you'll find options associated with that.

A

So, for example, you can toggle line numbers using options and then you know the displayed output. You would end up with line numbers here, um so they're they're things that toggle the way that they get the these blocks get interpreted or displayed, um and so we can do is we can? We can take our existing stuff, which is like you know.

A

We use these existing directives when we're writing our sphinx documentation, like code block and literal include, and then we just add this little test option to it, and the test option means that that this console test plugin will not. Basically it ignores everything that you haven't explicitly marked with test um and so once you've marked it with test it'll come through and it'll say all right, hey, okay, you want to you want to do something with this readme and what it's going to do is it's going to create a new blank working directory for every?

A

So you start with this fresh empty directory. Every time you run uh one of these console tests on a document, so each document gets its own blank working directory to work with, um so it starts empty and then you'll start saying. Okay, like okay, include this file, and this is sort of the flow that a user might go through, and this is why we mimic that flow so that we try to stay exactly as close to what a user would see as possible.

A

um So then you say, and actually I'll just run you guys through a demo here. So, um okay, so basically this is, we've got a command line usage for it as well. Just in case you want to run a off, but it's also a sphynx builder. So, okay, so let's see did my okay great I've got my whole thing queued up here.

A

So basically, as you run this stuff you're going to notice that um it's it's you're in a temporary directory, so we so we write this directive for console code block and we say that we want to test it and we can, you know, run commands in there, like you know, for example, pwd is uh directory and we can see over here and as I save this file, it's going to start running stuff over here.

A

So as we can see over here that that that you know, there's our working directory as this temporary directory and if we save the file again bam new temporary directory. So and this you know we can include so this is this file is readme rst.

A

So if we include the file into directory- um and then we run- you know what what is in here, we'll see that okay, now readme rst, is in this temporary directory.

A

um So you can also do things part of part of what this makes this interesting and I'm excited to at some point, hopefully have a little more time to extend this. Is you can do things like when you're testing you often run into cases where you need more temporary stuff, albeit directories or ports or whatever, um and so a lot a lot of times?

A

What this ends up being is a port right and you're trying to run all these services and you need to set up this environment, it's sort of like an integration test, almost right and you're like okay, I need you know a docker server running on. You know a docker running my sql on on 3306, but actually you know, I might already have my sql server running, because I might be running several tests into parallel um and and I'd really like that to have a random port assigned.

A

Now we don't have that functionality yet, but that's sort of an example of what we want to go to. What we do have is the built-in http server. um So python has this built-in http server and you can basically say it also. It does a small cgi, um so it can run scripts, and so uh we can say we can say to console test that you know this thing that I'm about to run is a daemon.

A

So it's going to run in the background and uh eventually the idea is that we'd be able to extend this based on commands, so there's some kind of infrastructure. Behind this to say there is infrastructure behind this say: hey what command are you running? So if you type cd right change directory you end up with uh you know, you change your working directory right and that's something: that's local to the shells context, and so essentially, what we have is almost this emulated shell, environment and it's very minimal.

A

It basically says: what's the commands, you know it basically goes through. You know the the arguments each thing with a little shell. um What's it called a prompt right? The little dollar sign gets interpreted as as one command to run right, so we go through, and if we see cd, then we're going to run a specific command for cd. If we see this python http server we're going to run a specific, you know command that that actually, is it's not really a shell command that we end up running.

A

We run actually some python code that allows us to go and bind to replace that port with a random port and then tell you know, and then we're telling our console test infrastructure here that we're actually running this random port that we should map what what to the user sees as port 9000.

A

We should be treating that that, as you know, whatever random port got mapped to port 9000., um so yeah we come in here and we can start our server and it basically does nothing because it dies at the end of the thing, but we can check that it actually got allocated a random port over here.

A

um So, every time we rerun, you know it's going to end up with a random port uh and so that that allows us right like like, I said you know we, we need random ports for things when we're running multiple tests, so you don't get collisions um and you can go through and you can do things like you know. The idea is that we're going to extend this to say.

A

Okay now, if you have a specific type of command that you know allocates a a random port in specific way or some kind of temporary resource, you know you want to take something that would be a permanent resource and make it a temporary resource and then map that into the rest of your tests, we're that's. The idea is eventually we'll, hopefully build out the infrastructure for that right. So right now it's just this http server and you can go in and you can basically say, okay, what the user would see is this right?

A

The user is going to try to access that website that we just started and curl down this readme file, which is this readme, which we copied into the temporary directory. um But you know we started it on a random port, so we want to go grab whatever that random port was and replace it.

A

And then you know we'll end up with with um you know the last few lines of this file, which is what we did here and since so we were support pipes to other processes, but I don't think there's support for pipes to actual um to to files at this point, um and so then the other thing is that if you're familiar with sort of like auto formatting and code style and and you care about those types of things um that dock test stuff, you can't there's this auto formatter called black.

A

um That has been gaining popularity in the python community, and that has been that's really nice. Although you can't put auto formatted, you can't black, doesn't know how to auto format code that exists within these blocks within your restructured text files, so part of the nice thing about it, and now I'm going to immediately break that paradigm here, but uh part of the nice thing about it is that you can you can take these?

A

You can literally include any files that you might be wanting to auto format, and that way you know you're you're, ending up with the same file that you can run your auto formatter on, but you can also you can also test that code. um So then you- and if you had something unfortunately now I'm about to demonstrate python- which I would like to have a format usually um to maintain code style. But but you know say it's something else right and I want to do something with the contents of these blocks.

A

That is not running, console commands. You can write them out to files right, and so we can say, write this the contents of this out to this test file um and then all of a sudden. We end up with um this test file and oh yeah, now we're we're curling the end of the file, and so that's what we're seeing um and uh then once again this is just another example of replacing the port. um So now this is gonna run that test file because we actually started a little cgi server there, um but yeah.

A

So that's the gist of it, um and so this allows you to sort of keep your. The point is: if you're writing documentation, often times you know you, you end up with the situation of okay. How do I test this documentation? How do I validate that?

A

It works without me, going through before every release and pasting every single command and writing every single file before I release to make sure that everything worked and sort of the solution as well, let's make something that that that acts like a user and reads your documentation, like a user, so that you know you end up with a consistent experience um and a testable environment um so yeah. So this is the console test, plugin and- uh and you can go, install it if you please um it's a part of dffml.

A

So if you install dfml, you get console testing if you're saying to yourself. Well, I don't want dfml. Well, then I guess that's too bad for now and we'll split it out for you later, but yeah.