DFFML Rolling Alice: Engineering Logs: Volume 0: Architecting Alice, 24 May 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Architecting Alice: Volume 0: Context: Part 43: Nested Remap Operation for Overlay Output DataFlow

Description

https://github.com/intel/dffml/discussions/1369#discussioncomment-2813540

A

So we're going to continue with our overlays today.

A

Let's see where we're at and progress on, overlay rebasing, these changes, okay,.

A

Mirror repos, okay, great.

A

Great okay, I don't know if we need that source records.

A

Here's what we're going to do we're going to what is up what is running over here. So this is running this repos okay, so we don't care about this anymore.

A

So what were we doing when we were testing our overlays.

A

A

Let's just make it back up real bash history here.

A

What time is it okay,.

A

Okay, so looks like what were we up to overlay looking for a nodemon command in here.

A

So what were we doing.

A

Open architecture- okay,.

A

Okay, okay, we haven't run any commands yet.

A

A

All right, let's create an alice version command, as it is alice.

A

Alice cli, okay,.

A

House version.

A

All right- and so you know we're going to have the data flows as operations and all of this stuff and see all that commands combined, but for now we're just going to do it like this results in run.

A

A

A

Okay, so get single.

A

All right so we're going to pass a good single operation.

A

And we're going to say the definition is the input spec.

A

And the value is whatever the definition we want is right. So in this case uh we'll say, version.

A

I'll stop version; okay,.

A

House version all right so.

A

To do make this a typing dot: user type? Okay! So what would that look like.

A

All right, so this is what it's going to.

A

A

These are defined generic types.

A

So put some references.

A

And we'll put the example.

A

All right so type our.

A

T generic t logged for.

A

A

Okay down signals.

A

A

Locked bar and nice, this is almost like a c plus plus template.

A

B

A

Iterable log brand far set zero set: okay, new t, okay, generic t; okay, so it's just telling you that that's how you work with that variable. This is the values that it will accept. Okay, so we could have.

A

Okay, so basically the operations themselves turn into types, and so then you can accept.

A

I actually don't want user to find new type.

A

So possibly this one is maybe so these user-funded generic types may be useful for slash data structure shorthand for data flow definition.

A

Okay right so, let's see new type.

A

From typing import, new tab.

A

A

And it's a string.

A

A

Can we derive a type from a type? I think we can so we could say simber.

A

Semantic version is a string and alice version is a semantic version right, and so then we tie back right. So we'll input this information as lineage right so.

A

You know, let's just do it.

A

To do unify typing interface with python.

A

Typing definition.

A

So make definitions really inputs with parents for lineage.

A

Based off python's typing.

A

All right all right so semantic version, alice version, okay, alice version and then we'll say: here's the definition.

A

A

A

We're just going to leave all that in there.

A

Using four.

A

A

All right we're going to fix some imports.

A

Primitive base tab could not be determined.

A

Plug-Ins installed. Okay, entry point: okay, so let's go ahead and mess up this right now, so plugin base is of type entry point, so primitive could not be determined.

A

All right, the primitive of black cannot be determined.

A

All right raised application warning what happens so test.

A

Okay, so the primitive of name could not be determined.

A

The primitive of name cannot be determined, okay, so.

A

Let's just pop in a shell okay. So what do we got? Name default, param, annotation, entry, point? Okay! So let's just call it um primitive object.

A

Right anything with object, primitive, we basically say hey. This is memory local.

A

Hr must be a type gut.

A

Union arg type check two.

A

A

A

High level data flow okay, so this is mad about this load default, which should really be a type so we'll give it the load default type. So I can take an instance of that type.

A

A

A

So we're going to go and we're going to remove that commit.

A

A

So now we shouldn't have that: okay, great.

A

All right, we need to see what else is in that command.

A

I know that was just that, so why did I do that? Oh because.

A

All right, so what we're doing get check out dot dot. So basically uh we stash the changes. That's why the file wasn't there so we'll pop the changes? um Okay, okay, so.

A

Gif df mile overlay, diff mile okay, so this we're going to use read, get grab dash c five remap all right. Let's go look for remap, okay, here's some remap to response stuff.

A

We'll just grab that file.

A

Okay right, so this is our example. I think we actually started with this. So basically, let's just grab this body here and make this the data flow. Okay,.

A

Okay, so we're going to response okay, operation stream after response remap.op, okay, so within remap.op, so in the thread. Basically, what we decided is that we should do remap and then a remap next level down so I'll. Put that comment in here, but basically remap operation.

A

Okay, so these comments really should go in here on the common use cases. So what are they? Basically, you know in non-default, overlays.

A

A

You could apply, you could have like you and not do folder apply operationally uh over. Ladies, oh yeah, so basically yeah you could have.

A

You could you could install um yeah. You could have overlays that you have installed that uh are always applying the prioritizer if you want to, or you could specify, context-specific overlays, which always apply prioritization in a certain scheme. Okay, let's get rid of these comments.

A

Okay is youtube working youtube's working, great okay. So what are we going to do? Well, we're going to run remap to response uh we're mapping response as an output operation, so within that output, operation.

A

We're going to have.

A

Another output operation.

A

Which is remap to response again.

A

A

So, okay, okay, so keep in mind that I believe the remap to response executes the subflow within the parent flow context, so we're preserving and that's what all the output operations do, I believe is yeah. They execute within yeah remap to response executes the subflow within the parent flow context, so it doesn't create a new input output network context for the subflow right so that get formatted message gets single op can reach directly into the input network of the parent data flow.

A

A

Remap, output up and default overlay with dataflow, which just grabs all the data flow, will get multi. Okay,.

A

So, let's use get multi on our interflow, uh okay, so and we'll call this get data flows.

A

All right, let's see, get data flow. Okay, so wait. So the data flow is passed as an input.

A

We basically run this merge.

A

So we're going to run a merge and we're going to grab.

A

So when we run the overlay.

A

And overlay apply.

A

It will spit out a new data flow. The new data flow will be.

A

All right so, let's add this a weight here.

A

Diff mount overlay overlay um that is async great, so apply dataflow.definition.

A

Okay. So we want this to do different or, let's see, data flow, dot, def ignition dot, name all right so grab the data flow.

A

So merge data flows.

A

Upper, let's see overlay all right: okay,.

A

So here's what we have to work with.

A

All right, so this is the default overlay which should go through and.

A

It should grab okay, so it takes the data flow definition as an input, and then it needs to.

A

A

A

So convert so data flow to dict right.

A

Data flow to ticked, so actually can we just pass a lambda. I think we can. Let's see um data flow data flow.

A

Data flow can bypass.

A

Data flow data flow dot, export.

A

A

All right, so we can say data flow dict, so we can call this dataflow to dict operation, which will do this export right and then we'll call the merge operation. Actually we can move, maybe we'll just define them all here and that will just be simpler right, so merge goes and it takes a.

A

A

Right so uh data flow.

A

All right, so data flow stick.

A

So we'll just say definition.

A

Let's play with the typing stuff.

A

So from typing import, new type.

A

All right, so what are we doing? Data flow as dict, so new type data flow is ticked, ticked, right, okay, so.

A

Now we have this data flow.

A

So here's the dfml definition.

A

A

So here's the data flow sticked.

A

All right, so we imported definition. We imported up.

A

Up inputs inputs, okay, so merge takes data flow is.

A

And so we have one input definition: we output it as a data flow is ticked. uh We merge it.

A

A

A

Merged data flow.

A

Let's say flow to merge, okay, so ah this is.

A

Merge two merge, merge, merge, source right, so we're merging into this one.

A

Right and we're merging a regular data flow.

A

A

Okay, so where's the regular merge.

A

Okay where's, this merge.

A

Data flow merge, oh no, wait! That's the function.

A

Okay, util data.

A

Okay, one two list depend okay, so we can say 1 2 list append, and then we can just pass the merge function right. So we can say this is an operation right which describes the merge function from dfmalutil.data.

A

Right so I want the one input to come from. We want the one input to come from data flow as dict right and then we want the two input.

A

A

Here's the input data so flow, so maybe we have like a data like an overlay right. So maybe we have an overlay definition right, which derives from the data flow definition right.

A

All right so data flow.

A

Or let's see this derives from the data flow definition. The data flow definition will be the class itself. This would be a new type which says new type. Blank data flow. uh Let's see uh overlay data flow.

A

And actually this really belongs with the overlay stuff.

A

Right so from dot dot, overlay.

A

Or let's see we're in overlay from dot overlay import.

A

Overlay data flow right and we will call it.

A

Okay remark not defined.

A

B

Are not defined.

A

A

To do make this okay, so.

A

Let's just play around with this get type get origin.

A

A

All right, we'll import these from here and that way they'll have their stubs if they aren't around okay. So.

A

All right, this is quick hacks here, so.

A

All right so convert new type to definition,.

A

Right, so how would we do this? Well, let's take a look, so what kind of what was it that we wanted? It's this overlay data flow in this instance.

A

Okay, we already edited it there.

A

Okay, cannot import git type.

A

Okay from typing import or trend.

A

All right, let's see.

A

Okay, good args from util data.

A

A

All right just around wrong turn: okay, so, okay, so that was for okay. So let's try to inspect these types so.

A

And is not a user id okay.

A

A

So no information via get origin or get args. So let's look at new type itself.

A

Let's see what kind of information it provides, so um call name: name supertype, okay, so, let's see, if anything uses supertype nothing access is super type. Okay, so we're going to access super type.

A

How do we check.

A

A

B

A

All right, if it has after super type, okay.

A

A

Useless, okay, so there we go so it's super type.

A

Death find primitive.

A

A

A

A

So this will return us.

A

A

A

So what are we going to do we're going to say basically hey we're going to make this recursive function that goes in, tries to find the correct typing information.

A

So new type has november supertab. Are you sure it is a typing.new type.

A

A

All right, let's find the primitive of.

A

All right, if has super type, okay,.

A

A

Primitive types here we go.

A

So if it's in this list return the type otherwise.

A

Oh, we forgot to actually recurse in.

A

I have primitive fine, proto.

A

Okay, so if it's super type so.

A

So if in type return object.

A

Okay, so new type is.

A

Function locals.

A

A

A

Fine primitive over date overlay.

A

So passing the thing. Well, what is it it's, this overlay, dataflow? Okay, so what's the supertype? It's flow. Okay,.

B

B

A

All right, so, if you pass something, what we would like is to say overlay data flow for two.

A

All right, so the problem we had with these was that oh there's, no default values.

A

So we'd, like this thing to take overlay data flow.

A

And type 2 definition.

A

So we'll integrate it into the whole definition stuff later.

A

Okay, so we find proper call name and we do.

A

A

All right, so we're writing a little console example and we're saying that.

A

uh If you call this thing and you pass it a new type.

A

So from typing, import, new type right and then import this new type to definition and then call new type to definition and say you know, this thing is a string, its value is feed face and so then it'll say feed dot face that raised to squall name and which would sleep at his feed face so eventually so to do. uh Split on capital letters into lowercase, dot, separated.

A

And that'll just help us with name spacing and keeping track of things. Okay, so.

B

What will they do.

A

A

New type of definition, 77.

A

Okay, let's depend okay and in this case we're gonna do uh we're actually just not gonna pass. This depend.

A

Okay, so this overlay data flow.

A

A

Data flow all right: oh, what new type.

A

All right, great.

A

A

Okay, so data plus dict all right, so so here's the overlay data flow.

A

A

Let's put this stuff in here.

A

A

New type, to definition,.

A

A

A

Param annotation, okay,.

A

New type, to definition,.

A

All right so we'll say: if this thing has this super type try returning this.

A

A

There's subclass, okay, we have to you have to throw this is class check on. If you run.

A

A

Python will get mad at you for calling his subclass on something that you weren't sure was a subclass.

A

Okay, so but it's moved from being.

A

A function to a class, so let's go back to 3.8.

A

Right so here it was a class right.

A

Let's just check if it has attributes name and supertype.

A

A

Typing.Newtype support.

A

A

Plugin base cannot be determined.

A

Okay, we'll just call it an object.

A

So this is in create definition.

A

So this is where we're going through and creating.

A

So this is where we're going during creating in dfml df base. This is where we're creating the definitions on the fly based on the function. Signature.

A

Kwrx inputs.

A

A

So if we receive an operation in.

A

So dfrdf type, okay.

A

So here, if we come along in operation or actually within an operation.

A

Operation right now.

A

So if we come along in operation.

A

A

Data class frozen true.

A

Entry point: I think what we want is death post in it.

A

So effectively.

A

So for input and item in editor tools, dot chain, self.input, outputs, dot items.

A

A

So take types make them into definitions. If you see them in the inputs or outputs.

A

A

So, if not is instance.

A

A

um Definition- and we see that it's a type then.

A

A

A

Fine all right.

A

Let's select that input software, outputs.

A

uh Software conditions, okay,.

A

A

For definitions and if definitions, if.

A

Definition container in blank, so, if definition container.

A

Definition container so list: okay,.

A

A

Okay, so for key value.

A

A

All right, so what what did we have to do here?

A

So so? Basically, um we wrote a little post in it on the operation which will go through and after it's instantiated, it will look at the inputs, outputs and conditions if it finds any things that are.

A

You know, it'll it'll go through and it'll update, then it'll rewrite those definitions which are types into real definitions which will create, which will it will create from the new type to definition, uh mapping stuff, so uh we'll go ahead and take this out of there, put it back in types um or put it in types.

A

A

A

Oh yeah, this is definition rule okay, so this will do key value or it'll. Do.

A

Enumerate for the d conditions, which is the list, and that way we can have the same inner loop.

A

So we'll say: convert all typing dot: new type definitions, two definitions.

A

Okay, overlay data flow not defined.

A

All right, let's try it.

A

All right: well, it ran that's exciting, so prime adaptation is not defined.

B

Primitive types.

A

Oh, was it in base? Oh.

A

A

We're going to import them.

A

Now, from types.

A

This is what I get for, uh trying to auto format, um types from different utils data.

A

Formatter is not defined, yet this price should go in util data.

A

Okay, so um what do we got here? So we moved around the types we created default type or so we're creating types based on objects if they don't exist and we move the primitives into the other file.

A

Okay and we're converting the definitions.

A

A

All right so then we're gonna just add this stuff here, where we all we're going to do is make a commit where we remove the primitive types.

A

Right so we'll just commit that df types moved prototypes list right, that's what we did there. Okay.

A

Right, so that's what we just staged.

A

All right, so this is what we have left. Okay, so now, what else did we do here? So.

A

So this this is its own thing and what happens here so df so create.

A

A

Create definition.

A

um So create definition with object.

A

Object uh so for unknown ties uh set primitive to object instead of raise.

A

Okay, so now we've got what just our type conversions here right so um find primitive. Okay. So this is this new type stuff.

A

So, let's add this: all right get commit sm.

A

A

All right great see what we've got okay. So what did we do now, so we said.

A

So this is a commit where we said: df types operation, uh auto, convert, typing dot, new type to definition on post in it.

A

All right, what else did we do? Okay? So we have this thing that says create definition. So if we are running create definition and we're past an annotated parameter, so one was basically hey. If I see it so we're defining it.

A

A

Okay, so this says.

A

Right so when we're automatically creating a definition given a function.

A

I'll convert, I'm posting it so.

A

Okay, this is for typing dot, new type, auto, create definition; okay, wonderful, let's push that up! That was a lot of good stuff. Okay,.

A

A

Okay and progress on overlay.

A

A

Okay, so this shows us the difference between our current tree and the stash okay, so it looks like that. Get multi-data flows. Merge spec was in this tree.

A

I think we had the right stuff in our tree before we rebased in.

A

That looks like the same commit to me.

A

Okay, so we pulled in changes from the manifest branch. Log walk reflux.

A

Good log in progress just a stash.

A

A

Didn't we drop a commitment.

A

Okay, so this whatever this is, is after posting, it, okay, good.

A

Okay, so this is where we want to be.

A

So check out manifest.

A

A

God, damn it didn't, we drop that commit.

A

Where's my lock walk reflux is this: it.

A

Progress on over time.

A

So what about? I thought we dropped that commit. This is the only thing all right, so in person overlay.

A

Move primitive types: what happens if we look at this? Does this thing have that commit in there.

A

B

Move prototypes.

A

A

A

In progress on over there so for unknown type crisis,.

A

So what about this.

A

What about this.

A

That one has a two.

A

Okay, I'm confused.

A

Where are we what's going on.

A

Why is this here.

A

A

Okay, so back to our overlay, dfmo matter: okay, okay, so merge overlay data flow; okay, so the inputs is the data flow.

A

Right so the inputs I forget about formatter for now.

A

A

Overlaid so at the top level, the output should be overlaid. Okay and the definition that that's going to be.

A

And then we're also going to have a.

A

Operation in here so we're going to have an operation in here which converts this.

A

Which does the from dict on the resultant data flow right? So we can do data flow from dict and then exported.

A

A

And then data flows.

A

It's a data flow, two dig, so we merge. We merge everything.

A

So we merge everything.

A

So we run remap to response.

A

We take the output of remap to response.

A

A

Okay, formatter.

A

A

The output to be response, okay, so we're primarily we're leveraging remap just for the ability to so that's good signal, so overlay.

A

So overlaid with.

A

A

Name: okay, so this will create a dick that says, give formatted message and within that.

A

A

Response should be hello, world response will be hello world in this example, so get formatted message is then.

A

Give formatted message contains hello world and.

B

A

We say remap to response.

A

A

Overlay data flow.

A

Okay, so we combine all of these together, so this is the main package overlay.

A

So this will be merged with everything else.

A

Okay, so this will be merged with everything else, and then the result is that any data flow produced any so. Data flow goes in.

A

Things happen on data flow. Any resulting data flow objects go out, okay, so we need yeah. We need to to be specific about this being the input flow right, so this should be something like overlay. Data flow data flow to.

A

Their flow to be overlaid.

A

A

So now we're gonna have to come in here and say.

A

Okay, so input definition.

A

We'll add a post in it on this to also convert the definition on the input objects.

A

Now the input has an in it. Okay, so if.

A

So automatically convert non-definition.

A

Auto converting on definitions into definitions.

A

So auto convert new types into definitions; okay, so if not is instance definition, oh yeah.

A

That's right for those auto created definitions. It's a little bit confusing.

A

A

Okay, so we'll add that there too okay, so now, we should be able to pass input objects directly as these types, so definition, okay, so data flow to dict, okay, so input is this overlay data flow and data flow to be overlaid.

A

A

So all these data flows get run and then they may do something on run to.

A

Modify the data flow and when they do that, they're going to output a new data flow, okay, so grab all the modified data flows.

A

No grab any data flows right, so they will take the data flow to be overlaid as an input we'll provide that the default flow will grab all data flow objects and.

A

A

Overlay will grab any data flow objects produced by other installed, overlays.

A

Which consume the data flow to the overlaid, input.

A

A

It will output the overlaid, uh it will output a results, object with the overlaid key set to a merged.

A

To all of the returned, or all of the data flows generated via other over lays, which consumed the death.

B

A

A

Okay, what is this give formatted message all right? We can forget about that.

A

A

A

Data flow to be overloaded. We take data flow okay, so overall idea. Okay, so 2 is not overlay data flow. It is.

A

B

Okay, careful definition.

A

Okay, so let's create a new.

A

A new type or, let's see dataflow aztec.

B

I'll be right back.