DFFML Rolling Alice: Engineering Logs: Volume 0: Architecting Alice, 27 May 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Architecting Alice: Volume 0: Context: Part 51: ADRification of Alice Discussion Thread (Roll?)

Description

https://github.com/intel/dffml/discussions/1369#discussioncomment-2823067

A

B

Huzzah we're off all right. Okay, um we need to back this up because uh it's uh it's choking on me. So um I'm going to update this thing and uh well, let's just say github- is not github's. Ui is no longer corresponding with the size of this discussion thread and it needed to get backed up so we're going to investigate the graphql api for discussions which was referenced to us by a very helpful mislaw.

B

Thank you very much.

B

Okay, so uh what are we seeing here? So it's a graphql query um and it looks like we're looking at a particular owner of particular repo.

B

We want to get the repository object given that owner and repo we want to get the first 30 discussions ordered by okay nodes.

C

B

Repository that discussions repository discussions.

B

B

Created at updated, okay, so these are these discussion. Orders field enough field field is this field.

B

You know, discussion, number field, discussion or field; okay, this enum with these values; okay, there's one of the values all right uh discussion categories category here.

A

B

I'm not sure, okay discussions.

B

Page info discussion, edge discussion.

B

Implements comment: little lockable, robust update, will answer.

B

Answer chosen at author body type html type string category discussion, category that replies to the discussion um identify.

A

B

I'm hoping we'll be able to do like some kind of streaming. Graphql query for the data flow execution, so.

B

Database id in check the collector okay, so comments comments now, after before discussion, comment connection.

A

B

No, it's coming first, ten. How do we iterate over this thing.

B

B

I think we can only get the first hundred.

B

Whoa, okay, the tennis court versus the replies connection is requesting up to three million hospital nodes which exceeds the maximum limit of fifty thousand.

B

Okay discussions, so how else can we do this so discussions.

B

How do we discussions.

A

A

Which is paging for in discussion.

B

Okay, so we can use pinned discussions in this case, because alice's are only pinned as oops analysis are wrong.

B

This is our only pin discussion.

B

B

What for everyone.

B

All right so she's the only pin discussion so we're going to just pull her down like that. Oh come on.

B

Comments doesn't exist on pinned discussion, my ass okay, let's see, pin discussion connection. Where is that.

B

Discussions discussion.

B

Discussion category connection.

B

B

All right, this is under the blanket okay discussion category connection.

B

Discussion category.

B

Discussion added category edge.

A

B

So what is it pin? Discussions gives you discussion, category connection which is nodes, okay and then within nodes. We have discussion category.

B

Come on, oh oh here it is: okay, implements node repository, node fields, name, okay! So no it's time. Okay! So there's no title.

B

Description, it's a category discussion, category connection. How did we get here, though,.

B

Discussion order, discussion, pin discussions, pin discussion connection.

B

And discussion connection.

B

Right so so this gives you pin discussion connection which has nodes, which is pin discussion, which is.

B

Pin discussion, which is.

B

Pattern pinned by discussion: okay, so we want this gush in.

B

Then we go to the discussion. This has the title and the body text okay, so this thing okay, so I think we just needed to wrap in that. It's always wrapping in something.

B

Okay, let's check it out.

B

Excessive paige nation.

D

Boo boo, boo, okay,.

B

Do we lose anything when we do that? How many are there on this.

B

236 comments with 201 replies.

B

How many comments are there on these.

B

uh I don't understand how you.

B

Okay, so we say so comments. First, 100 all right.

B

After before cursor, so will it give us a cursor? Maybe we can okay yeah, so maybe we can say so uh mv, okay, so we say mv my query.graphql initial discussion, query.

B

Graphql and then we copy that into uh page and the shin discussion query and in that one we say first 100 after and then we say.

B

Okay, how the hell is this going to work.

B

B

B

Okay, so this is the replies.

B

Okay, so we have one so.

B

So on this comments, one so, basically, if we are return to cursor, if we return to cursor from our response, so we will do.

B

What is this, what did we call it.

B

Initial discussion query.

B

B

So now we'll just pipe that through python m json tool and.

B

And we'll pipe it to a formatted version, so this is output.json.formatted.json.

B

Okay, let's take a look. So where are we.

A

Wait, let's see.

B

Nodes nodes replies nodes. So let's go so replies comments, so I'm going into comments. Okay, so it looks like there was not another okay, so reply says no yeah. So the only items in both comments and replies are just nodes, which means that none of the oh wait a minute. We have to look through all the replies.

B

Okay, let's see. So what is that level at? So? Basically,.

B

Okay, so we need to grep, so we need to look through here for anything that says.

D

B

Replies: okay,.

B

Okay, so it looks like everything's good on the top. So what did we do here? Well, we, okay, so we went through and we said all right. Let's look for reply, so we look for everywhere. Replies was mentioned. It always says nodes, there's nothing other than just nodes below that key. So now what we need to do is we need to.

B

We need to check all the rest of the nodes, and so we can do that because they're all at the same indentation level, we can basically just issue a check for anything coming in at that indentation level. So what's the end level for this would be here, so anything coming in here is going to be what we want to check.

B

Right so this and then close paren.

B

B

Is minimized false, so, let's take a look, so let's see is minimized false, so if we're at the bottom? Okay. So if we're at the bottom of this, then is minimize false yeah, as always so the last node. So we have successfully confirmed that we don't see any more than 100 replies in any of these um and we know because we're looking at pin discussions and there's only one ah or let's see so discussion yeah within pin discussion, it's nodes and then within the nodes.

B

Object is just a list, so there's no other there's no other um field within pinned discussions, so that means we're not paginating there either. So we don't need either of those other queries, then, okay, so we can take this now and we can uh dump this to json.

B

So I think I have a little or dump this to mark down. So.

B

The question is, how do we want to do it um so.

B

All right, so I realize this is.

B

Okay, so let's take this.

B

Let's add all this to the thread.

B

C

B

And we'll call this this file.

B

And we'll show the command we ran.

B

All right there we go.

B

B

Okay, so here we are, we have our dump command, so we got everything out. So uh I guess we'll do you know.

B

These are all markdown docs.

D

B

C

B

Thinking thinking thinking.

B

I think alice should contribute these docs.

B

D

C

No okay, okay! Fine! We won't do that now.

B

Okay, so let's just.

B

Let's just take this and look at it in python for a moment.

B

B

All right did that work did okay,.

B

B

All right there we go all right, so this is our input. Data and oh, this will lead us into another fun one, and this is our output data.

B

B

Maybe we'll put her on the wiki for now.

B

Now because the wiki.

B

B

The wiki is not all on page, though it's nice to have it as all one page um with the little sections is really nice. I think that's what we'll end up with um okay, so.

C

B

So if we output to mark down okay, so we'll output to mark down.

B

Well I'll put to set a markdown files.

A

B

B

So, basically anything that's. We have a few of these that are labeled where they have everything in the first line.

B

I'm just going to get out.

A

A

Dump discussion.

C

All right all right, let's go.

B

B

The usual suspects ospathlab temp file,.

B

B

Create a path object, so we create a temporary directory. We we create a path objects to reference, that temporary directory or stripping nuts all right.

B

Too much at night.

C

B

B

B

Pin repositories.

A

B

Pin discussions.

B

Nodes nodes, okay, so nodes is an iterable, so we can iterate over nodes before discussion.

B

B

There for a discussion node in.

A

B

Discussion node.

B

B

Field modifications off there, we go.

B

Body text where's the markdown.

B

Are you you are kidding me? Where is the markdown.

B

Where's, the markdown.

B

Why can't I see the markdown use your content, edits.

B

Give me a heart attack here. Why can't you you give me the markdown? Why can't you give me the markdown.

B

B

Or your list of edits to this content user can delete user can update.

B

What is this discussion.

B

B

B

B

Body: okay, please tell me it's in body.

B

Body text, body text.

B

Thank god mark down.

B

All right, so we got body from the discussion node, so where's. The comment, node calm.

B

It comment commit comment, commit nope all right, well, discussion, comments, discussion, comment, create connection, knows, discussion, comment, body, okay,.

B

Okay, comments. Comment.

B

Replies, comment comments.

B

No, it's body notes, okay, wait notes, replies body, replies, body, replies, replies, discussion, comment, connection, discussion, comment, okay, glad. I finally figured out how graphql works uh discussion comment, connection.

B

uh Discussion comment, connection comment.

B

A

B

My buddy buddy all right, whatever, let's see what happens.

B

B

Fragment user fragment comment. Oh oh, oh! Well! We don't care about this.

B

B

B

No, I understand now I understand how this works, so this is some kind of it's almost like that. Yamaha unpacking thing.

B

Comment comment.

B

C

B

Comment comment, comment, notes, comment, comments, comments, note, comments, comments, replied. ah It looks right to me. What happened?

B

Why? Don't you like fragment user, was used.

B

ah Yeah, we don't care about that.

B

Oh, thank god. Okay, mark down!

B

Oh so much better!.

B

Okay and let's go ahead and replace all these unicode characters.

B

And what is that? What it looks like oops? It looks like a quote to me: a single quote.

B

um Yeah single quote, so, let's look for some other backslash use. What other unicode do we have in here.

B

Quote unquote: oh great.

A

B

All right, so we do uh input equals json.loads, um pathlib.path, output.json.formatted.json.readtext.

B

uh So we'll import, json and move the path import to the top.

B

And we'll get rid of this blob okay, so there we go. There's the discussion body and then we're gonna go into comments, nodes.

B

Comment: node common node comment body.

B

So comments, nodes, discussion.

A

Excuse me: okay,.

B

Discussion comments, print comment, body comment, body.

C

Print comment: okay,.

B

Body, oh hell, yeah. Okay, we got it. Thank god. Okay,.

B

B

I'll loop through oh.

B

B

Okay, so comments node, o and then replies so comments. Note body replies.

B

B

And this isn't comment: node, okay, reply, node.

B

And this is string, apply node.

B

B

B

Oh in replies, nodes print, apply, node, okay,.

B

All right, great, all right, fantastic, so um now we're just gonna dump this to a bunch of adr. So.

B

B

B

How do we structure this? Oh wow, there's a lot of text here.

B

What is this word count at?

B

B

B

2500 words.

B

Oh wait a minute. Oh.

B

Is this thing really forty two thousand words holy.

B

Okay, wait: okay, it has a whole chapter of alice in wonderland in it. Okay, thank god. Okay,.

D

B

All right, let's take out.

B

B

And then, okay, so 2000 words, so 40 000 words wow.

B

Well, a lot of those link pasting.

B

Damn: okay! Well, no time, no time to uh no time. Okay! Wow! That's a lot.

B

Okay, so after removing the first chapter of.

B

Okay and we'll say before.

B

B

Interesting forty thousand hmm so.

B

A

B

B

No we're getting this uploaded. Okay,.

B

Okay, so it is good now.

A

B

B

A

B

All right so now I'm checking for headers.

B

All right so we're looking at all of the titles of all of the okay. So what can we do here? So it would be nice to have the date stamps on all of these.

B

With all their edit history.

C

So what about the edits? Can I see the edits.

B

Okay, security arrest all the way down: okay volume.

B

Zero introduction volume one volume zero and then the title is volume zero as well.

C

B

All right we're gonna, do like a literal translation where we translate okay,.

B

Miscellaneous things all right chapters and then miscellaneous.

A

B

We'll group them by.

B

B

So if we are a chapter.

B

In that chapter,.

B

I'll create a directory okay, so.

B

So for the top level, one we'll do tempter path, dot.

B

We won't do too we'll say: uh output, markdown, uh output, directory, okay,.

C

B

B

Graphql query, output.

B

Okay, so it takes the graphql query: output, which is this input and writes everything to this temporary directory output directory dot, join path.

B

Index dot empty.

B

I would say this is more of like what's the first line.

B

Okay, so the first line.

B

B

Oh yes, okay! So.

B

B

And it's really.

B

B

B

Dot, okay, so piss rolling on us.

B

Okay, so what do we do.

B

B

That's our title.

B

Okay, so this is like what index rolling alice dot empty? Okay, so.

B

Rolling alice.md and then.

B

B

B

All right, so we got the title.

B

Okay, so death title.

A

B

Title link line.

B

All right, it said no paste.

A

Okay, so we got that title. We got that title so title file, name.

B

uh Total phone now.

A

Substring that found.

A

B

Chemo manifest okay.

B

All right, so anything that starts with the hash.

B

Just couldn't turn into that so we're trying to talk about that.

B

B

With nothing and space dot replace.

B

Space with underscore.

B

All right and then just pay replace to replace hyphen.

B

What's underscore.

B

Okay looks good.

B

Turtle's file name, okay, so tell the file name.

C

C

I have zero introduction.

B

C

B

B

Okay, so I'll put directory.join path.

B

Okay, oh and then I have to fix up all the links. Okay,.

C

ah Fix up all the links.

C

All right now, it's tight.

C

A

B

All right so on the rest of these basically.

C

Where do they go.

C

B

Oh each reply: this reply gets written out to them file.

B

These triple gets written out to its own file.

B

Okay, come on come on, come on, okay, come on okay, each reply gets written out to its own file um and we're going to apply all the get diffs and create commits with the correct dates.

B

B

Okay, um I wonder.

B

We're just going to put it all in github format. Right now, we can figure out what to do with it later.

B

We're going to put it all in git format, right, okay,.

B

All right so outer loop, I inner loop j reply, jay.

B

C

B

Pad f string python.

B

Pad and f string python.

C

Is it it's like.

B

B

Let's do con zero five.

B

B

Yeah we can do clone zero. Four, all right- we're not gonna get anywhere near this, so it doesn't really matter we're just dumping right now.

B

B

Okay, doesn't like that.

B

All right can we do this rolling ass.

B

And then we just don't give it a file name.

B

Fun name parts, so here's the following parts.

B

A

B

If yeah, if that's a real header, then do that, if not so then say.

B

Okay, right out the body right out the reply.

B

Okay, so this should be in a similar fashion,.

B

Right, okay, so so what's happening here, so basically we're we're dumping um so output.

B

The uh a file for the comment, I'll put a file for the reply uh create the following, which will be joined by underscores.

B

ah Create the filename for the top level.

B

Okay, I said system.

B

B

Tempter all right here we go.

B

Yeah there we go.

B

All right, so we got a slight tweak. uh We'll put the the plus markdown should have been outside. Of that.

B

That's out of that right there there we go that's more like it.

B

Okay, we're still getting a few we're getting some markdown dot, markdown how's that happening there.

D

From other parts.

B

All right so increment I by one.

B

Enumerate start.

B

B

239 files with start equals zero 239 files with start equals. One.

B

B

Okay, so now adding it up here on the top level document.

B

B

Is everything in order everything looks to be in order holy smokes? It's a lot of mark down.

B

Everything's an order fantastic.

B

A

A

B

That looks good okay, so, let's basically just say.

B

All right so wow, that's a lot of files. Okay,.

B

So these are all of the files.

B

Okay, this all looks good.

B

Okay, that all looks good okay. So what what about the edits, though,.

B

Sometimes I hear myself: okay.

B

I want the edits to.

C

Use your content edit, I can keep my eyes open all right, the list of edits the content.

B

Or at least commit this good comment box art as uh initial.

B

Commit without edit history.

C

I find that ridiculous.

B

B

So backed up with out edits here.

B

All right and here's that commit all right so here is our markdown.

C

D

B

B

Well, I think we're going to call it for a night, um the realization that we're at 40 000 words on this thing, it's kind of like it's time to go to sleep so good night.