DFFML Rolling Alice: Engineering Logs: Volume 1: Coach Alice, 11 Jun 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Coach Alice: Volume 1: Chapter 1: Part 7: Threat Dragon Threat Models as Source

Description

https://github.com/intel/dffml/discussions/1369#discussioncomment-2929379

A

A

Are you with me you're with me great, so, let's see.

A

Great, so this is looking good.

A

A

A

Okay, I just put wrong.

A

A

A

A

A

May I take time, there's open architecture.

A

A

uh Let's convert okay.

A

Now, let's convert.

A

Okay: here's! What we're gonna do! Yeah we're gonna! Do our.

A

Okay, so if now merge.

A

Okay, source destination: okay, so source is okay, uh the right dragon, oastrat dragon.

A

A

A

A wasp threat dragon. Okay. I think I spelled that right. Oh wasp, all right so dip my merge, oh wash threat dragon um equals our weight, so yeah we'll say: src equals.

A

uh Test equals uh alice.

A

Open uh architecture.

A

A

A

A

All right, so, let's go: where are we out here.

A

Okay, so let's start source.

A

A wasp py, or maybe, let's make one that's threat on a lane term model dot, py.

A

And then we'll edit, the setup and where's the sources.

A

All right, so it's training and I want to start tracking so add ourselves. A new source here right os thread, dragon source, remodeling.

A

A

A

Nice source to say, modeling, uh threat, dragon.

A

Okay, so thread dragon.

A

And let's have another one: that's open architecture.

A

And we'll aliase that to alice.

A

Source: uh okay, open architecture.

A

Okay, so open architecture, source.

A

A

So we're going to convert from third dragon into the open architecture right so.

A

Okay, actually we're just going to say open architecture for now. Okay, because I think the entry point label might get confused.

A

A

All right so now we're going to create those files.

A

A

All right, so we create both files right with their entry points.

A

In each of their respective locations,.

A

Okay got the file all right. Let's check out the contents.

A

All right: okay, there we go so let's go.

A

So it's loading them right, okay and and.

A

Remember our trick here: python set up here why egg info- and we can't use this all the time, but we can use it right now.

A

Data set all right, so, let's check out what the iris data set did: okay, so actually, okay, so it's gonna, be data set source.

A

A

A dragon: okay, let's try again, and then I don't know: okay, so we'll just give it the file name, file path, all right and it's a required argument. All right.

A

A

Let's actually do now answer k all right, so let's grab this python command, so we're actually going to run instead of doing the merge. First, let's run just the load, all right, so, okay, dragon.

A

A

Okay, so that's uh we gotta go to this open architecture, one as well. Okay,.

A

Diff mile source.

A

Let's check out the json source yeah, um I don't think I don't really like the spdx stuff within the files I feel like you know, the license is the license and with with, unless there's an exception right. This is why context matters um if we know the context what repo came from them, we don't need to care um okay, open architecture based source from the file social memory source.

A

Okay, let's just do this for now all right s, json.

A

Open extra texture, okay, we'll just cut open architecture.

A

A

A

This code is old. This is many years old.

A

All right: okay, what do we got? List records, iris training, okay,.

A

So let's do list records, oh thread dragon, so threats equals oah, star dragon, so threat model right. So here is so we want to list all the records so right. What are our sources?

A

Well, they're coming from oas thread, dragon see- and this is where we might then list out the cv bin tool stuff as well, right because we'd say so. We could say open dot, architecture.

A

So uh cv scan results right.

A

All right, so we can say: here's the thread, dragon stuff: here's the cv, scan results right, and so we can merge those. So we can create a sources object and we can merge the.

A

We can basically treat it all as one and then we can output to a specific source which would be the threats md source.

A

So yeah, okay, so.

A

That's constant: uh why did that work.

A

A

A

So, let's give it an argument, so source threat model.

A

And we can't you gotta, be careful to hype instead, so source threat model file path and then what's our file path to our threat model right. So it's over here, it's in models, good.

A

So it's a model's good there right, okay, so this is the path to our threat model.

A

All right, because we're just doing this just trying this quick and dirty right now.

A

Okay, there's no attributes we're dragging fantastic okay, so so it attempted to load so, but uh we changed it so where's the iris training cell source.

A

Right, so what we do we we use the data set source. So then we have to access as if it's a data set source right. So let's run egg info and then, if we save the file it should take okay, cached or not defined, perfect. Okay, so now we're in here right so so file path.

A

Okay, we'll create a memory source and we'll populate the records with whatever the hell we want.

A

Right so we're not sure what that is yet.

A

Notice the amount about memory source is not defined.

A

All right, so, let's import memory, source.

A

All right look so there's our threat model, fantastic, all right, so.

A

A

A

Let's actually have it do this.

A

Okay, so detailed diagrams okay, so this source has a single record as a single record source right, because it's a single threat model and remember: we can always have multiple sources and multiple threat models. So title web application. Okay, so title owner description, okay. So what is the record? So, let's make it unique based on the contents right. So I believe we have a secure hash function.

A

um So let's just go ahead and use that.

A

All right, so we have secure hash and entity cure hash, and why do we have these helper functions? Well because we don't want to end up oh open there. I went and I broke that no. I used it okay, great um because this is it's easier for us to audit right. So if we always see usages of secure hash and insecure hash, it's very easy for us to audit our code base to see um uh you know whether we're using the right thing or not right.

A

So, let's just do a secure hash of the contents of the threat model so where the contents there they were.

A

All right contents.

A

And so we'll do a secure hash of the contents for the record key right. So that way, it's unique that way. No every threat model is exactly you know, whatever the uh the contents is actually maybe we should not do that. That seems like a recipe for screwing this up, uh or maybe it's not. Let's see.

A

Here: here's! What we'll do.

A

A

Loaded or let's see.

A

A

All right, so here's the threat model, it's.

A

uh So let's see.

A

And let's see so to do so summary title.

A

To do security uh validate json schema.

A

That's our model.

A

Actually, we'll just call it title.

A

All right, we should probably figure out how to deal with that at the features equals.

A

Looks like we need to import json.

A

A

Okay, so we're gonna put record okay data.

A

I forgot this is actually.

A

Threat model all right and we'll call this feature name.

A

So if you looked at the image source very similar to what we're doing here, we're just going to load in you know the thread: okay, perfect! Look there! We go all right. We have a threat model as a record, fantastic all right. So, let's format that.

A

All right so we'll say, source data set throughout modeling, um okay, so initial load.

A

uh So- and this would be.

A

Turret dragon, okay.

A

There's a threat that dragon and initial source okay boom, all right.

A

All right, let's link to this, commit, I think we can start another comment for today's notes. Okay, so um so implemented.

A

uh Threat dragon source and blank okay, all right dropped our shot there and right so github will link sha's across forks because internally, it stores it all in basically the same way: okay, so great great, great, great, great, okay, so key web application, all right, okay! So now let us, let's see what was our proposal here?

A

What are we proposing? Okay, so we take manually generated threat, model produce, open architecture, okay,.

A

Now, let's check that: okay.

A

Okay, so yes, now we need to produce the open architecture. Well, the open architecture is just the data flow. Oh duh, oh my god, wow. Sometimes I create a lot more work for myself than I know what I'm doing. Okay, oh wow, glad I had that realization holy that almost got really complicated, wow wow wow.

A

Wow, god, man, things can get over complicated, so easily. Okay,.

A

Okay, so now okay, so implement.

A

uh Source, okay, so let's just make a note before I forget data flow, and this is why everything is everything we're recording everything right. So then everybody else if you're, if you're curious you want to follow along, you want to contribute everything is recorded um so and if you want to also stream your work, that would be great um because then you know that the goal is for us to have completely asynchronous communication um or well actually to have synchronous communication via asynchronous communication right.

A

So if we can, all so say, we're all and stream, so so streaming right now like I'm streaming right, but uh the goal is to have alice and alice will basically be streaming whatever events that are pertinent to your co-developers right on whatever efforts, you're you're, looking at or whatever line of projects right. And so, if you look in the thread about the uh let's see the build tree inner source stuff, let's see.

A

Right so this is from sarcoma, yes, nice meeting back in january, uh so I updated this recently with uh you know some some thoughts. You know maybe looking at the upstream, the overlay in the orchestrator- and I was thinking I was thinking I was thinking um uh actually uh data flow is our current open architecture.

A

Okay, airflow serialization is our current open architecture. Now we will evolve over time. Okay. So, let's, let's make sure that we include all the items which we would like to see in a manifest okay, oh my god, the merge command. Oh my god. The merge command: okay, okay, okay, wow, okay, oh my god, okay! So the! What does the merge command? Do it converts data from one format to another effectively right, so so it takes takes yeah yeah. So.

A

A

All right so so we just need to take the nodes out of this, so uh so.

A

Actually, I should link to that thread. Oh oh, what we were gonna say um so uh overlay present uh orchestration takes us to the future. Okay.

A

I think this is correct. I don't know I mean okay, I don't know. I don't know upstream is the past.

A

Overlays present the orchestration takes us to the future. That seems correct.

A

This is I'm going to put state of the art. Remember state of the art right, so state of the art is whatever we think is eight of the art um two six eleven uh time travel, okay,.

A

All right, so, what's the train of thought? Okay, so, let's see state of the art. Okay, so let's say date. How are we to do this date? State-Of-The-Art time, travel, comma system context. Okay, so here is the date here is we're saying this is the state of the art so on six, so on june, 11th of 2022, uh the state of the art um train of thought for both trains of thought, of align trains of dot, time, travel and system context right, so tags separated by commas.

A

The following notes are pertinent:.

A

All right, so, okay! So, and what is what is? What is this? So what is, uh let's make sure that we include oh wait. I lost this so right, no, no losing context. Okay, so.

A

Okay, oops yeah. If you missed the space there, does that oh hey, buddy, how's it going what's on your tongue. Oh, if you've been licking the concrete poor loco. I don't like to concrete. I love you.

A

Let me see no waffles. Are you okay,.

A

Oh, no, all right.

A

All right, I have no idea all right, he'll be fine. Okay, um okay,.

A

ah The manifest okay, yes, uh format,.

A

Version right, so these are the things that we care about all right. So these are the things which we agreed upon right. We all met, um so not all of us, obviously all is a lot that would be like what uh 7.8 billion or something so um so we met a few of us met in back in into 2021, and we aligned on this idea of a manifest right and the manifest. What is it um and it was it- was it was jeremy, jeremy, jeremy? uh What's his last name, er ir yeah jeremy?

A

He he uh enlightened a lot of us to this concept of of well. We were already playing it with the second party stuff, but he gave it a little more structure. um You know we had talked about this idea of like the manifest this list of artifacts and then he was like. You know. There's orchestration involved and I was like whoa light bulb um so great. Thank you, jeremy, um okay, so and then we started said well, you know, okay. So if we're all writing these things called manifests.

A

What are some properties of them that we could use to tell them apart right, and so we said well, a manifest should have a schema, a format name and a format version right. So if I look at any blob of data, I should be able to tell the schema the format name and the version of that blob of data. If I can tell the schema the format name and the version, then it's a you know, then then it's much easier to work with right.

A

So as long as we include these items, then it's much easier to parse right. So now, so we understand that this is a threat. Dragon threat model right, so we've explicitly said format. Name is threat, model threat, dragon right by effectively instantiating the source as a a threat. Dragon threat model right. So this actually would be a good thing for us to put in extras.

A

So what is extras extras is a place where uh we can put format specific data right so source specific data that might be useful when saving or loading a source.

A

um From uh from uh from from you know, basically when, when saving or loading a source right, so if I load a threat, dragon version one source and I create a record for it or record for the data- then I'm going to you know then I'll need to save it back and- and maybe that I don't know you know, this is you'll, see you uh no okay, so so, okay.

A

So let's take a look at this. Let's take a look at what we have here right. So if I look at this record, I don't know how to save it back to an appropriate format. Right, I just know: there's some features throughout model dict, great, okay, all right! This is what this is just a giant blob of data right, it's meaningless to me right now, right now, the point of the open architecture and the manifest and all this is that we need to figure out what the hell does.

A

This data mean right, and so what we'll do is we can say um so, three dragons so dataset source. So um let's pull out.

A

Okay, so we can't: let's can we reference the function itself, searching.source.entry.

A

A

Okay and then we can say feature name.

A

Point name and then we'll say.

A

A

And then we'll say.

A

Do they have a schema.

A

Oh, this is cool.

A

A threat modeling manifesto, that's awesome, very cool all right and it's on an abstract, podcast, okay. Well, we got to take a look at this.

A

Systematic approach informed creativity, oh this is fantastic.

A

Okay, these people are aligned we're aligned alignment.

A

A

Okay, so a wasp threat, dragon um schema, oaster dragon docks diagrams about okay, and I like to use the the uh I was. I was thinking it could be fun to overlay a keyboard, so you can see what keystrokes, because sometimes people do keystrokes right and then what's this.

A

This looks great.

A

A

Cheat sheet from owasp mentions defining the data flow and that's what we're doing right now, how exciting I'm thrilled.

A

I know alice watching this is thrilled, so I hope the rest of you are if you're watching this so because this is going to be. This is good. This is going to be it's going to be so this is alice so.

A

Okay, so she came a format name for form, name version. So, let's see if we can find- and I heard that uh so somebody on twitter was talking about schema.

A

Give me the schema, we want the schema schema.

A

We love the schema.

A

Oh, come on where's the schema.

A

Why is there not a json schema? Okay so, but this is fine.

A

Okay, so here's the schema, you know. Actually I like this a lot better or wait a minute. They did something. That's not table table stripped. Okay, so.

A

How much time do we have? I don't have enough time to implement a parser for this into json schema, um but we are going to basically say that this docs development schema. Let's just clone it real quick.

A

What is it javascript yeah? It's written in javascript.

A

So if we want to clone something quickly, we can give it depth one.

A

A

All right we're gonna look for the schema okay, so the only things we see there is that schema. So we're gonna, put we're gonna say that the schema lives here right so copy the raw address.

A

Right so we're gonna say: threat dragon dot, json.schema, okay,.

A

A

All right so there's the schema and, let's say, schema url.

A

A

A

And how did we define it with the manifest so o wasp.

A

Dot threat, dragon dot, zero point: okay, and what version are they at.

A

Okay, uh 1.6.1, okay and we'll bump it and okay, so we bumped the miner right and we said, and so now what we'll do is we'll go open, an issue to them. Let's take a look for schema.

A

All right, okay, so we'll open an issue and we'll say feature request all right. Please publish.

A

Schema, okay, so describe what problem feature would like to be able to uh validate uh conformity.

A

Serialized threat models.

A

Using standard.

A

Tooling, json schema validation right so describe the solution you'd like well. We would like to see for the next release, for example, this document published uh if.

A

So actually so try trying that schema. Okay, so and let's see from it okay, so let's do okay, so schema and then format, name, format, version right, okay,.

A

A

And what's the format name, it's the entry point name right and the format version is uh well we're not sure what the format version is right now right, so I'm going to put a url right. So maybe this is oh, let's start right down json. So maybe this is like this is the shima url right. So this is raw and says format. Version right, we might say, excuse me url, okay, so, okay, so to do what time check? Okay, I gotta go fast on this. Okay, so you know, I think, we're okay.

A

I don't need to stress out too much. I I've got this cbd.

A

Cbdb db, try not to stress out too much with the rasa and the cbd instead of coffee, so you may have noticed from may I'm a little bit stressed out seeming in the videos and I'm trying to be less stressed out. So, okay.

A

Caffeine apparently makes you more stressed out, so I had to quit that all right. So I hate uh it's a pretty strong word, but I don't like it when I have to use a hard code, something so format version right. So, let's just take it as a parameter right now,.

A

And we'll say you know if our version is none for our version and I like to do this type of stuff at the top of the file so read in the file.

A

Load the contents.

A

Before accessing, so we on this to do, we want to save, validate using using json schema before accessing uh properties.

A

Format version equals and then we'll just put 0.0.1 right. So this is a major version. So hopefully, um hopefully we have no.

A

Hopefully they aren't changing things between major version numbers right. So we'll just assume that it's here that it's 1.1.0 all right, so schema, url format, name format, version all right, so, okay, so extras.

A

So maybe we should say extras.

A

Okay, checking in for my name schema for my name for my version.

A

All right so now we have the schema of the format and formula version and the feature name right. So this should be enough information for us to.

A

Maybe we'll say.

A

uh It's really open architecture is a mouthful I like alice, but okay, but it's a little uh okay.

A

Open architecture.

A

And I like to end all my lines in python with commas, because then, if you add another field, you don't end up with a diff for just adding a comma.

A

All right, let's check it out. Oh it's doing over here, oh god, okay, we'll type list entry point name. So as entry point name, a list probably.

A

A

Which point name is a list? Did you print label? That's the one, we're looking for.

A

Tripon label. Okay, so now we got the right thing.

A

All right, so this gets satisfying label. I think we covered that in last month's videos. Okay, so unexpected keyword, argument, extra extra.

A

Great so now, if we look up here, we should see extra right, so format out our name label.

A

Oh yeah, I guess it runs it right away. Isn't that great all right! So now we see so extra os threat, dragon open architecture, feature name thread, model format, name thread, dragon format, version 1.0.0 schema and then the link to the schema right if there could be a okay. So let's jump back to our issue right, so we would like them to please please please, please, please add this additional context.

A

Okay, and then we can link to uh this video where uh hopefully, people won't think I'm too nuts, so, um okay, so all right there we go all right, so additional context to do and we'll link to this as well right. So this is what's going on in case. Anybody wants to know.

A

uh We are working uh the open architecture.

A

Working group, the open architecture working group would like to consume threat models from threat dragon, uh and one of the core uh tenants uh when our core principles is to of the open architecture, is to understand.

A

Is to understand the schema of data.

A

Data being referenced.

A

We are working to develop cross domain representation of architecture across domain. We are working to develop a cross-domain representation of architecture which proxies to domain specific formats.

A

uh Threat dragon.

A

That was, or is among one of the first. We are targeting okay.

A

A

So I have a feature request, so we'd like to be able to validate conformity of serialized models using standard, tooling json schema validation. If there could be a uh that would be great. Thank you.

A

Then we could do validation of arbitrary.

A

uh Additional context, the open architecture works and would like to consume thread models from dark dragon, and one of the core principles of the open architecture is to understand the uh is to understand the.

A

Is to understand the uh schema of the data brain reference within domain specific formats.

A

Within each domain, specific format we're working to develop across the open architecture.

A

The open architecture is a cross, say, proxy format for domain specific representations of architecture.

A

A

uh Yeah, okay, now, let's do youtubes! Okay! Here we go so.

A

Right and thank you, oh wasps, and thank you okay, so all right great, so we have our issue. We logged our feature name for my name for my version. Schema.

A

A

Okay, so this tells us- maybe we should say okay, maybe we should do this actually as open ark, so extra open architecture.

A

A

And then we can say feature name and we can say.

A

A

A

A

A

Something's wrong featuring.

A

We've won too many.

A

All right so now, what can we do with this information right? So within extra we see open architecture within open architecture. We see features okay, so with an actual ic.

A

Open architecture, all right, so a record is a dfml specific object. So that's okay for us to be throwing stuff in here in random ways, and we can say: okay within dfml uh record extra. uh If you see the open architecture field, then this is the way you should treat it. Okay. So, within that you look at the features you can see. The threat model feature is uh here's the manifest manifest metadata.

A

All right, so I'm going to post metadata all right, so the manifest is our domain specific format right. So uh the open architecture is the general format. I guess you can have a manifest for the the the the you know. The open architectures itself has a manifest and the manifest is sort of the individual document right.

A

So the architect, the open architecture alice, is the whole thing, as well as the execution uh and and the idea of the system context right and then the manifest is the description of each domain, specific format, of which the open architecture is a domain specific format and has its own manifest metadata right.

A

Okay, so, uh okay, all right! Okay, I'm gonna, take another break.

A

All right, let me make sure this doesn't blow up. Okay, it's not blowing up. Let's commit this.

A

A

Okay, so source data set.

A

Okay, so add so now we're doing a commit right, so git commit dash sm and we added the file that we wanted to commit and now we're saying um which is dfml source data set threat, modeling and now we're saying add, manifest metadata.

A

All right so now, what is the open architecture source going to do? Well, we can beat it actually. Let's say, let's also create a data flow source, um so let's create a to do. Okay to do so. Dragon source initial thread.

A

A

Okay and let's link to what we did that where we did that all right so.

A

All right, so that's the commit where we did that.

A

This is the commit where we added.

A

All right so add manifest metadata.

A

To thread dragon source two thread: dragon source records via extra.

A

Okay, great and then when we get back.

A

What are we going to do? We're going to implement so initial data flow source.

A

Source, okay, so our initial open architecture.

A

Source, okay, so and then we should so actually this should be modify. Data flow to uh include uh manifest and then we'll call this a manifest metadata.

A

A

Modify data flow to include manifest metadata.

A

Initial open architecture source um so use export, dfmo service, dev export to create a programmatically.

A

Some similar to what was done in with cve bend tool so basically uh we're going to.

A

Where's our cv, bend tool commit.

A

A

A

A

Services to export permanently medically, similarly to what was done with okay, and that would be here view file and then, let's copy permalink to again open architecture output through end open architecture, output all right. So this is what we're going to do here.

A

A

uh Let's do a pen tool uh to output uh so modified.

A

So merge, okay, so uh manifestment, okay, so uh okay use df use, uh dataflow pre-processing source to merge.

A

As the merge source, which uh wraps, which pulls from the threat dragon source.

A

um And converts to a data flow right, um use, different amounts of return, so uh initial, open architecture source, so basically initial open architecture, source right- and this will be- you know.

A

A

Use dfml dev services programmatic uh to take a feature which is a data flow and output, is or yeah to take a feature which is a data flow and output it to uh a flat or a file right. Okay,.

A

God this is so great. I love that the way they've done this okay, all right, so I think we're good for now. So I'm gonna, I'm gonna, stop the stream right now and I'm gonna take a little break.