DFFML Rolling Alice: Engineering Logs: Volume 0: Architecting Alice, 23 May 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Architecting Alice: Volume 0: Context: Part 42: Onward Through Overlay Land

Description

https://github.com/intel/dffml/discussions/1369#discussioncomment-2796678

A

Okay, we're good all right. Where were we.

A

Okay, these are a few operations we're going to merge stuff together, we want to run.

A

We want to run with this.

A

B

So we're gonna run.

B

We're gonna run.

A

Data flow, pre-process source context, we're going to run a source operation.

A

We're going to run a source operation where the source is the data flow pre-process source, we'll run it with all for single. Do we have any operations for sources? Let's check.

A

Operation model; okay: we don't have any operations for sources, um let's go through and just make one real quick give my operation. So, oh no, there there. It is different operation source, okay, convert records to list okay. So this takes.

A

Features and predict features and iterates over the okay convert list to records okay, so we need another operation in here. That's just um you know so.

A

The input is nothing.

A

And the output is.

A

Okay, so it takes a source, it instantiates the source um and uh dump so.

A

So uh records it's called source records right, okay, and so this will say, expand equals records or result right. So we dump, we say: here's a source dump all the records, expand every record so that it enters the input network as a separate item right now now we run those, let's see so we.

A

Yeah, so we run this source records within our df, our df where's dfml, df source, df, okay, so data flow source, okay, actually within data flow source right.

A

Okay within data flow source.

A

A

Because we're doing all for single, so we do data flow source with base source under that being.

A

The memory orchestrator um and then we merge yeah, so we execute the merge within.

A

The data flow source or, let's see so no the source.

A

Why does this have a source source to wrap data flow source.

A

What is this? This looks just like preprocess.

A

A

All right, sctx, sctx, okay, so it's not used anywhere. Why the? Why is it here.

A

A

A

Looking for command line, usages looks like data cleanup classification uses it.

A

Source plug-in converter.

A

Tf source, cfd, okay source csv search file, name text equals df data, merge data, cleanup operations; okay,.

A

Let's run through this tutorial.

A

All right so we're going to run the console test for it.

A

A

So copying the file path and replacing the file path.

A

I'm going to stash the changes, real, quick because it's not happy with them.

A

All right, so it's stash and let's roll back by three.

A

All right we're running the console test, so we're just going to double check because I'm not sure what the ci says so we're just going to run it here, we're going to see if this tutorial works.

A

It could be that our usage is out of date, but I'm not sure what the source does right now. I can't see the source context doing anything in this code.

A

So this, I think we might have missed a rename right because this source here.

A

This, I think this is probably me fixing it up.

A

No right so here's source context, so this isn't even used anywhere. It's just entered right. So what do we even use source for we don't end up using the source length.

A

All right length: that's yeah, like that's, not a pokeball here, okay, what have I run? Oh for is this even used? This isn't used.

C

A

A

Okay, so I think what we can do is we can just have data flow source context, wrap.

A

A

A

Okay, we're gonna cut over we're gonna focus on a different use case.

A

A

Usage snippet, so I had to clone some repos the other day and I thought well. How would I do this um and.

A

A

A

C

A

A

A

A

All right we're checking out the manifest branch we're going back and it's dash pop okay, all right. So here's the overlay stuff, okay, so.

A

Let's go just make the flow.

A

Right so this is.

A

Class download.

A

Default overlay.

A

Okay, we'll teach merch.

A

A

That just goes in.

B

C

A

A

C

A

Yank, okay,.

A

A

um Let me just update, let me just grab this okay, echo.

A

All right, I think I showed this little http trick before.

A

Okay, I have to update something snippet.

A

We're gonna go through.

A

A

A

So what are we going to do? Well, we know that we.

A

A

Run our cv event tool on everything right. So um how are we going to do that.

A

Let's go ahead for a minute.

A

Okay, so overlay.

A

Okay, let's grab back from here.

A

We're going to create a new branch, we're going to say scanning.

A

Or we should say maintained all right, so this is some stuff that um this is basically so uh we presented some of the stuff we did in the the identifying security threats working group about a year and a half ago. I think it's in the weekly sync lecture, like the weekly sync.

A

B

A

Somewhere, I think it's in the notes um you can find the recording from there.

B

A

Yeah, so basically, what we're going to do is we're going to um just do a continuation of that based on should I just like we talked about, and so we're going to show how we, you know, maintain these upstreams over forks with the overlays um okay, so check out be maintained upstream.

A

A

A

So, okay, so how should we do this? Okay, repose, dot, sh.

A

A

C

A

So we had this shell demo we're making it a python demo um we're going to go ahead and create a data flow.

A

Okay, you know we're actually going to keep this separate, just because I personally like the cli and I move faster with cli, so.

A

A

Okay, so we've got a create command, we're going to create them.

A

All right, so here's, okay, so.

A

A

So we're creating a nodemon shell down here that says, run that repost.sh every time we save a bash or a python file, and we've got our different things here. Okay, so we've got that no branch given quarters okay.

A

So this is something that I just noticed. It seems to be creating a file that has git grip search. It has a bunch of other operations in it.

A

A

All right so we're going to fork http test to dffml.

A

Did that get changed.

A

Export that json.

A

See light airflow.

A

Okay, union story: data flow; okay. So that's where we change this.

A

Here, it's updated for objects as well as paths; okay,.

A

That's right so we're going to look in util, cli command.

A

And we're going to say.

A

We're gonna comment out these we're just gonna run this command here.

A

I don't know what's up with diagram, but we're just gonna try to run it and see what happens.

A

A

A

Is class field type.

A

A

So it's something here: okay, so get args field type, endless.

B

A

Let's see what happens.

A

A

Field, okay, so what is? This should be beautiful.

A

Okay, so it's not there.

A

Okay field name beautiful, um okay,.

A

C

A

Right, okay, so that all looks good to me. I don't see any types that are not error, argument invalid. So let's just try printing this down here.

A

Okay, field type, okay field, if name equals data flow, so pi people equals.

A

Type equals so field does not go pool, there's real time. So, let's see what is get origin on this.

A

Typing union, so if we have to open union.

A

Then we choose the first one, then it should. Let us use a string.

A

A

So dataflow not list.

A

All right, let's see what it shows.

A

A

A

Okay, dfml cli dataflow on 292.

A

Okay, so a weight.

A

Okay, so this is.

A

Let's see what's here.

A

Stop the data flow is a list that says export json.

A

Why is it a list.

A

All right, okay, arg, okay, because here said list double serve. If it's a union, then we want to do this.

A

So if it's a union, so if it's a list or tuple, we want to do nx plus, but if it's not, then we want to say just you know the type okay.

A

So, while resolving seed definition, mapping figures, definition, mapping traverse not found all right. Let's fix that.

A

Okay error: while the resolving seed definition push to github new repo inputs owner, not found inc, ticked keys data footprint, okay, so.

A

Push to github new repo inputs owner.

A

Oh because, that's probably using a.

A

Oh because it's named okay, so that's because that's the default name so.

A

So that means those need to change too so.

A

C

A

A

Create new name.

A

A

Result all right so and then false equals github.repo.public.

A

Okay, well, we'll make it true or well we'll make it false by default here all right right. So let's do if well that's that definition! Okay, let's see what happens.

A

Okay, we cloned http test.

A

Push to new github repo okay, so we could have this so we'll. We could split this out and have this be another operation we will so clone github repo.

A

A

I don't understand why this diagram command seems to be dumping. Everything.

A

Old name: url: okay, okay, so we also have to modify the flow to take that from so.

A

Repo okay, so we have to say push to github new repo inputs, old name. Okay, so we have to say that or no let's see so old name is already a url.

A

Repo is to get repository, checked out, okay, so but that's not true. It's a git repository.

A

And then we have github.repo.public github.owner and then the new name result.

A

Which, for some reason, was feeding to itself.

A

Okay got unexpected keyword, argument old name now we're getting somewhere.

A

Okay, okay, so it's forking the repo and let's go check and see if it did that.

A

Should be private, oh there we go. We have our private fork of http test, all right.

A

And the branch is the same as upstream.

C

A

Let us do another example here.

A

Let us feed in a git repository.

A

Which is local.

A

Okay, how about that active directory, verifiable credentials, python all right, so, let's say search records.

A

Okay, let's see it run.

A

Okay, remote origin already exists: okay,.

A

Gh repo create private.

C

A

Created the repo.

A

Okay, error: remote origin already exists, so we actually have one less thing to do here.

A

So we might say you know.

A

Add run this only on condition that origin does not exist right so.

A

And then we could say, you know, run this on an unconditioned that repo does not exist.

A

And we'll comment it out for now.

A

Branch is not defined okay, so oh, we wanted to reproduce default branch.

A

Right because we have to push up the default branch, so df now feature get triggered feature operations.

A

A

All right circuit branch, all right, so what do we have? We now have the branch right. All we had to do is add another operation, so we'll say.

A

Branch and we'll run black on that, okay, so.

A

Okay, so now it clone the repo push origin branch, but it's not going to run so it's not running the operation because we don't have the branch yet. So we need to add in that operation over here.

A

Get repo check default branch or no, it's get default branches. Now and now we should get the default branch and push it up or what happened.

A

Get repo default branch.

C

C

A

Get reboot default branch.

A

So it takes a git repository, oh and it needs that no get branch given okay,.

A

All right did it push it.

A

All right, let's see so, okay, push everything up to date.

A

Okay, because it's there is it just cloned it. Okay, so.

A

All right change.

A

Oops, oh god, that would have been bad.

C

A

Okay, get remote at origin, god damn it.

A

We're gonna have to delete this, though first.

A

Create push you origin, main okay, so okay, but it's going to be instead of origin, it's going to be owner.

A

All right, let's see what happens.

A

Dfml does not appear to be a git repository.

A

Create private push you.

A

I mean what what is it then.

A

Okay, so gh repo create okay, so create the repo.

A

A

This should be set url.

A

That was the problem.

A

All right, so it got mad because we didn't put github.com.

A

All right so now we're going to set the origin to github.com url we're going to push it up there and it should have.

A

Populated with stuff there we go, we did it okay, so now what we can do is we can automate the creation of branches right. So uh so this stuff.

A

Will come at this.

A

All right so we're gonna commit this so base. um Okay, arg fix type in unions.

A

To select first type.

C

A

Okay, so okay, so let's just put this repos repos.py dfw. um Let's do uh feature gate, df feature gate.

A

Util, let's see.

A

Yeah- let's just put it here.

A

A

Okay, so what did we just do? Well, we successfully.

A

Created a fork, what happened? Don't watch the name repos oh gifmel feature get dot, repos.

A

All right, so we don't know how to register these as entry points. At this point, okay branch given.

A

Here, no good branch, given definitions, we're in data flow.

C

A

No get branch kevin.

A

A

Get branch nuke branch given, so why give reposit clunky repo no get branch given.

A

Oh because we should also have how did that.

A

Oh because I went back, okay get repo default branch.

A

A

Already exists.

C

A

All right, we'll just write it in a little mark down here or a little restructure text.

A

A

All right, just formatting.

A

A

A

Okay, so feature gate, mirror repos for cv, bin tool, scans.

A

All right, I gotta go.