DFFML Rolling Alice: Engineering Logs: Volume 1: Coach Alice, 10 Jun 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Coach Alice: Volume 1: Chapter 1: Part 4: Fighting with Overlay Infrastructure for CVE Bin Tool Scan

Description

https://github.com/intel/dffml/discussions/1369#discussioncomment-2663771

A

All right, let's take another stab at this, are you with me here with me, fantastic all right.

A

So I think we're grabbing the wrong thing. Merge dataflow,.

A

Oh, so we do have a to do for this. I think I was ranting about that the other day, okay yesterday or something so saying that we need to do that. So it's still not done, then I take it all right, okay, um so this overlaid so merge. So we were supposed to accept this merged so potato flow. We are applying overlays too.

A

A

Merge implementations.

A

Right, so this is why we diff this before we did it so now it's now, it's screwed up.

A

Okay, so let's take a look at this again, so we end up with the right flow, but we don't end up with the right implementations right so.

A

So that looks like it worked.

A

A

Range implementations refactor to function.

A

B

B

A

A

So, okay, we merged our implementations.

A

Okay, so we need this dude, so we're applying this over this one. But okay, um I think I have to bail on this whole thing. Right now, I know come on.

A

Repo2 directory, oh yeah, so that's okay! Yeah, because that one has an implementation. Everything else instantiated fine, because it's installed so um okay, so.

A

So we're not supposed to grab this overlays merge, so we're supposed to do. Data flow, we're applying overlays too, by running overlay data flow and passing as input. Okay, so data flow being overlaid.

A

Let's look at the execution so.

A

Okay, so basically all we ended up doing here was going data flow to data flow aztec and then to from dict, okay, so to dict from dirt.

A

Stage output merge data flow. We have too many things happening here. I think we wrote too many things at the same time. Okay, so because look at all these output operations so.

A

A

And what is this overlap, therefore, to apply it over there?

A

Okay, so just do flipping overlays, okay, so.

A

Main package overlay.

A

Are we applying an overlay here? Are we doing.

A

A

Operation implementation, non-instantial directory.

A

A

Overlays installed so does overlays install get called so if overlay's installed just getting run, then we must have. What path did we come through here.

A

High level data flow.

A

All right, so we came through here.

A

This is where we enabled the debug logging, so we turn it off there. Then we see nothing.

A

A

There we go okay.

A

Just if I click, let's read the notes, this effectively creates a new system context, a direct answer, answer of the one that got passed in and the overlay. Therefore they are both listed as input parents. What does that says.

A

That doesn't really make any sense.

A

Therefore, they are both listed in the input parents. Let me set this out, it's called uh from whatever is all right, so I think we should delete that part of it and we should keep this all right and we should say so some context, all right. So.

A

A

Upstream equals results.

A

A

And links equals.

B

Data flow overlay.

B

A

A

All right, so that is what that system context looked looks like none type is not callable, so it doesn't like this.

A

Oh sh, wait what oh, it's just running the flow okay.

A

So basically, we did not apply the overlay in this instance. Okay, I think we need to bail on this thing. For now, I think what we're going to do is we're just going to use the.

A

Don't know yeah, I need to stop this.

A

So, let's install the release version and see what happens.

B

A

Okay, oh god, okay, oh okay,.

A

A

A

So we're gonna do.

A

Extend cv bin tool and pass in.

A

B

A

A

A

Resultant system context equals system contacts.

B

B

A

Back to the fact that this is not instantiable.

B

A

A

All right so high level data flow run. We um use overlay as system context deployment, I'm going to push okay time check, 37, okay, so base that's wrong up there. So we're not going to commit that uh it's wrong as it is. But okay, active system contacts.

A

A

A

All right, so do you have system contacts?

A

uh What was this so active.

A

Active system contacts I take uh parent and only upstream config, as config.

A

So we have that's uncommitted, it's this mostly cli stuff. Okay, we don't want to mess with that right now, so, okay time check, 39.

A

Directory now found.

A

A

Okay, so we need to use this merge implementation. So the question is how all right so do we need something? That's like the opposite of extend. No that doesn't work because everything's, a stream. um We have to collect it all.

A

So we can do.

A

That more basically.

A

A

So if we were to collect all of the two.

A

Okay, do what to apply as overlay.

A

Main better, therefore, being overlaid.

A

Careful being overlaid as sticked.

A

Merge merge data flow so so that merged data flow.

A

Therefore, after overlays merged.

A

So when you run the main package overlay, so you run the main package overlay.

A

So the upstream is the main package overlay wait a minute yeah.

A

Upstream is the main package overlay, the overlay.

A

Oh, and this is why it's confusing so main package overlay needs to get passed in to this as data flow being overlaid.

A

And then, therefore, we're applying overlays to okay. So this is the format package overlay.

A

Data flow equals.

A

Okay, so if overlay is not done.

A

Overlays installed.

B

A

This isn't working, let's just skip it.

A

B

Okay level, good flow.

B

A

In progress, failure to apply fails to apply overlay so skipped for now.

A

Okay, so different df memory, two two one, eight, so two two eight.

A

A

Okay, so what is this an output.

B

A

Okay, so this thing only got one value: oh that's right and we have to move the input into the.

A

Repo to directory that's outputs.

A

Yeah, this is not good.

A

Okay, so I now put items okay.

A

Let me go to directory.

B

A

We do repo to directory and then we do okay reboot to directory, and then we want the should. I operations right so.

A

Where should I.

A

I don't do it for now, sam okay, so let's go back to.

A

A

So this is what commit we're on.

A

A

A

Overlays to add functionality.

A

Such as should I, where appropriate,.

A

A

Let's do so we'll note that we test against this development version of dfml.

A

A

uh Scan scan of python repo scan a git repo.

A

So currently python, cves or python certainly runs. Should I.

A

Operations, or should I scanning um and then.

A

And then we'll pass the directory right so dot. Okay, so.

A

Files, as usual.

A

We'll add overlays to build, so in future we can add overlays to run the build, then scan.

A

A

So, for example, if we see that a build is available via github actions, we could fork the repo we could initiate a fork run. The action consume. The build then do the scan.

A

Right because it's all arbitrary with the overlays once we get them working so.

A

A

A

A

That did flip a scanner for multi language support.

B

B

A

Here's our pull request.

A

A

All right so now I'm going to turn this into a checklist here, so support python, the inclusion of should I flows.

A

And I can't spell language apparently so I need to fix that up here as well.

A

B

A

Okay, so existing scanner uh functionality, yeah data flows.

A

Okay, let's go ahead and put that over here.

A

Is this the right one yeah.

A

We might run into the.

A

Really don't want to have to deal with the graphql page nation. I don't think this is 100 comments. Now right, yeah, that's not 100 comments, so we're close all right.

A

Okay and here's our nice little restricted text file that we're working on okay and then we'll have to take away these test blocks. Okay, I'm gonna! I'm trying I'm trying to get github to support this, um we'll see how it goes so that to support displaying it without you know, being a problem.

A

A

Let us combine with the rpm file stuff as well, so just for conclusion. So rpm file, okay, so operations been sick.

A

um The data flows add and as an extra within extras required.

A

I didn't do as an extra via extras, okay, so what we can do is within cv, bend tool. We can say all right. So on our entry points here, we can say see eventual scan data flow.

A

Multi cv been tool: multi cv, bin multi tool.

A

Cv bin multi-tool, no cv, bin tool, scanner, data flow, and so then here we can list diff mail. Diff now feature git.

A

B

B

Okay, why did I name it then.

A

All right install with extras.

A

hmm Saw cve bin tool um extras require.

A

I think oh nurses require.

A

A

Okay, all right, I gotta go.