DFFML Rolling Alice: Engineering Logs: Volume 1: Coach Alice, 8 Jun 2022

Previous Meeting

⏯

youtube image

►

From YouTube: Coach Alice: Volume 1: Chapter 1: Part 1: CVE Bin Tool Integrate DFFML Memory Orchestrator

Description

https://github.com/intel/dffml/discussions/1369#discussioncomment-2663771

A

Hello all right, the screen is flipped, fantastic, okay tricks with obs, um I'm assuming you all can hear me so we're we're um here. We go it's it's ju! um You know in in hindsight. I don't know what I was thinking uh trying to get two months. Work of work done in one month that didn't make any sense um makes perfect sense. Now I feel a lot better, actually starting this video and realizing we are on track. So um this is, we were gonna. We should just call this june tutorials.

A

I don't know why that we're gonna do planning and execution in the same month. That doesn't make any sense um so anyways so on to volume one chapter, one down the dependency rabbit hole again all right. So what are we gonna do here? Well, we're gonna, build some threat models um and uh we're gonna do some basic threat models to start um yeah? Basically, uh let's see does this link give us anything that will be nice for the video?

A

No okay? uh Well, that's! Okay! So uh we'll we'll see some we'll we'll see some results by this weekend so uh effectively the the what we're gonna do is we're going to take um cbe bentool, which has great facilities for scanning a great plug-in system, great great, very, very, very mature, um we're going to introduce uh some of the stuff from. Should I eventually down the way um to we're basically going to take cv bentool make it multi multi, you know more than just binaries, and how we're going to do that.

A

Well, we're going to leverage the df model orchestration, so we're going to write a new, a new scanner. That's going to be a multi-language scanner effectively and we're going to um then uh you know you use it so so the first thing we need to do is go. Take cv, bend, tool, um where's it at here. It is so.

A

So we're gonna take cv, bento we're going to add an alternate scanner to it, um using uh dffml we're going to make sure that scanner has a pair has parity with the old scanner right and then we're going to extend the scanner to new languages uh by uh effectively doing a overlay of the should I flows right and so then uh we should have uh wait. Basically, we should have ended up extending cv, bend tool in the process.

A

What we're going to do is we're going to add a new output format and that output format is going to be threat model. Okay, let's get at it all right. So, uh let's jump into the scanner code, so cv bend tool is structured. Basically, there is the the main scanner and then there is well it's been a while, since I've been in here so yeah, so the version scanner- and we had some great great work that was done in here, where everything got made async. So that's going to be fun, um so we love that.

A

Obviously, um okay, so here we go and let me just make sure I'm pulling the latest version of this, because I know this code base is influx. There's a lot of great work that happens here.

A

Okay, so where are we- um and I should probably check with terry too, make sure nobody else is doing this.

A

Oh, I was scanning log for oh that's right. Last time I was in here I was back in december. We were scanning, so there's some examples that we have on scanning for log4j using data flows, um and so I was going to do this support. Then. Obviously, it's six months later so well here we are uh we'll. Do it now? um Okay,.

A

All right, let's so, let's just play around here and see what's going on, so we do a recursive scan now. This is just this is just ripe ripe for data flows. Okay, so let's check it out. um How are we going to start this? So what branch are we on we're on log for j? Have I done anything here?

A

uh No, okay, so it looks like we're just based off the 3.0 branch. So, let's check out main, let's pull uh and we'll see we'll see what's up, so uh that was our our current version of scanner and now, let's check out where we at okay so april 14th. um Let's just do a quick search through the.

A

A

See if anybody is working on this right now.

A

Okay, improve language mega specific package, private structure for.

A

Okay, so we check, if there's any, um does for multiple parsers. That's probably terry readiness. She writes good. um Oh no! Let's see this is actually dragon four months ago, all right, let's wear multiple parsers current structure stamp it to package parsers.

A

Okay, general system, any new park, parsers look.

A

Okay march, 2nd.

A

Okay, so, let's just say.

A

Let's just comment.

A

Okay, let's say uh we're looking at doing this with alice for this so and we can link to this video and we can link to this log entry here. Let's make a log entry right, so uh plan. Okay, you know this is this thread has gotten out of hand. So, let's just say we are planning to implement multi language.

A

Support and see then tool the introduction.

A

Orchestration of let's see, data flows to handle. Similarly,.

A

Or, should I and then we'll link to, should I right just so everybody knows what's going on, we want to make sure that we communicate appropriately and maybe, let's actually link to.

A

Okay, or should I, and we should also link to the other, should I tutorial so that we can? We can make sure that people understand right. We gotta assume nobody knows anything coming in here like about what we're doing you know, they know everything about what they're doing, but they know nothing about what we're doing right and they shouldn't have to. We should tell them um it's not not their job to know everything that that we're doing so, okay, so we're planning to implement multi-language, support and cv event tool.

A

The introduction of data flows to anal similar to handle uh scanning uh will.

A

Will then uh extend via overlays?

A

Similarly, to should I uh we plan to contribute these changes back upstream, okay, so we will, and since this has a link, remember we're putting the links in the recordings and let's go and edit that comment. References.

A

A

So let's go ahead and say we.

A

Started work on this here and, let's just do jhpr list see if there's any open pr's against this.

A

A

A

Okay, um data flow scanner.

A

All right so, let's go through and do we have a docs all right. We do so.

A

Let's just go write a little bit about what we're up to so data flow based scanner.

A

Because this would read the docs, it is actually so, let's go ahead and make that an rst file so go ahead and make this an rst file. And then we can test it using console test right.

A

A

And then we can copy paste.

A

And we can say cv bin tool, actually we'll do python m cv bin tool. So, let's check out what do we got here in our um looks like we should go update? This so looks like okay great. We have output engines, we have checkers and we have okay. We have our entry points so.

A

uh You know so one thing that is a common problem that people have is that their path is not set up.

A

A

So let's play with this and see what happens so. I started work on this here and then we'll put a link to the pr.

A

Okay and we add the doc okay and now, let's create the file.

A

Okay, so scanner.

A

Maybe we'll say scanners and we have to, of course make our init py file in there, because or else python will not pick it up and we'll put our dataflow scanner in there.

A

Where's this file.

A

And we'll say they don't see: dock dataflow scanner for more information.

A

It's just in case somebody finds this file and doesn't know what the hell this is or where it's coming from right. So important and important all right. So third party input, so dfml is now a third party import right because we're within cv eventual, and so if we're following our uh you know, reverse christmas tree dependency includes um where we basically stack.

A

We basically stack at least character count dependencies at the top in the first party library, in the standard library to third-party libraries and then to uh you, know second party, uh something that's plugins or something, and then you know first party being whatever package you're in so here now we're in tv ben tool right, and so um let's see scanner data flow okay. So what should we do? What should we do? Why don't we do python? Then we have um cv bin tool right here: okay, so uh path, dot. Okay, so um we'll say.

A

uh You know cd path or, let's see.

A

Actually, let's just go ahead and give that url to.

A

Tv bin tool, uh we'll just down we'll just do the git clone.

A

Okay, where are we at time check? Okay, we got 10 more minutes here. So what should we do? What should we do all right? um Let's see, let's, okay, so let's take.

A

We've just done this like 10 times different ways: okay,.

A

So that's so good just make sure that this is up to date.

A

Manifest branch computer see, if not working computers- oh come on this one: better work: okay, so import! Okay! Now we're going to read bfmo uh entities alice. So, let's just let's pull in alice here all right. We can overlay alice.

A

Okay, so what alice doing so.

A

Okay, so there should, I install that's gonna, be relevant for python packages. um Should I use it's gonna be relevant for multiple things and then should I contribute. That's really focused on get repo right, so, okay, so s bomb and we'll replace spawn with the cv bin tool stuff.

A

Okay, so I think what we'll do first is we're going to integrate within the cv event tool code base and then once we have a clean integration within the cve band tool code base, then we'll explore okay, let's pull it back to alice's level and see how does alice interact with cbe been tool now that we know that cv event tool has been flushed out to a you know, a multi-language support because really doesn't make sense to maintain. Should I uh shoulda is really just a very specific.

A

You know it's kind of see: we've been to one, should I or too similar, let's throw it on cv, bend tool um and let's see how it goes right, it's a great place for it. um Okay. So what should we do here? What should we do here? We should pull in the inner source.

A

Let's see what we've got here, and so what do we got here? Okay, some shitake, config? Okay. So this does clones given url all right.

A

Clones given url, okay and.

A

All right so we're going to create a main function, we're going to run it with async io right. um I think we can safely remove this talky stuff for now, not not not an issue for us. um I don't think we care about counting lines of code at the moment check. Okay,.

A

And, let's see so, let's just go ahead and put in the url, so we have the url. It's a nuget branch given okay grab all the get feature operations: okay, let's create some operations for here as well. Actually, let's just do the operations within this file.

A

A

Let's do from where are we we're in data flow yeah? So let's just do. Let's just do this: let's just put the operations within this file right and then that way, um yeah it'll be it'll, be within the same file. So, okay.

A

A

So, let's define a system context.

A

A

um Tv bin tool.

A

We're just going to see what happens here right, we're just going to play to play with this all right. So.

A

Async for results in cv, bendfield data flow right. So let's this system context right so cb band tool. Data flow is a system context.

A

What does that system context? Do look at that? Okay, definitions.

A

Your nail name run record, set.

A

A

A

So we want to pass this in, so we want to run it like this.

A

So, let's see what happens here, nothing.

A

So that's what we already renamed it scanners.

A

Okay, so we remove operations.

A

Okay, great so, let's go and remember: okay, so let's reference our modification of the git repo here right, so we want to do. This is an example of an overlay happening where we basically take this gear repository checked out and we run the lines of code per language, but now we're going to do the reverse right. So we want to download.

A

Pipeline package context directory.

A

All right so pi pi package contents is ending up in dfl mapping, create that creates a repo object right. So we actually want to do the reverse. We want to say in the event that we have a repo object and we detect in the event that we have a repo object.

A

We download it and then we scan it to see if there's any binaries in it right and so then we'll just go ahead and put a binary in a repo and call it a day um and then we'll have implemented data flow scanning sort of um okay.

A

So at least that's what we're gonna have to call it a day of that today. So okay. So how do we do this?.

A

That's value, so what is our objective? So we have a gate, repo.

A

Okay, so this is just a giant okay, yeah and that's right. Then we figured out that we didn't need to do any of this, because the output operations, what it, what a concept? Okay, so random, cooper language with this conditions.

A

Okay, so this says that we're just needing sure cocky. So let's do.

A

Okay, this is um get repository.

A

A

Okay, so we can take these out here. This is just that group by spec.

A

Okay, let's forget quarters, we don't care about quarters right now uh we don't care about new branch given right now, um so we have our collector data flow grabs, any operations in here. Okay, so this should. If we write this, we don't even need to do this. So if we just say.

A

All right, so if we just say that.

A

This thing, whatever it is, um I have to go all right. I have to go all right. We're just gonna make an operation.