DFFML Rolling Alice: Engineering Logs: Volume 0: Architecting Alice, 23 May 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Architecting Alice: Volume 0: Context: Part 41: DataFlow Merge Operation

Description

https://github.com/intel/dffml/discussions/1369#discussioncomment-2796678

A

A

Great works this time, all right so.

A

Let's jump back to our code here.

A

So, okay, so we have overlay this game, multi output. So let's think about this a little more. So what do we do here with this overlay default? Well, this is effectively okay, so this is wrong too remember. We said we're going to turn this into an operation. Well, it's time. It's already time.

A

A

Okay, so this is data flow, merge.

A

A

All right, so we're gonna make a couple operations here. So data flow merge um data flows list all right, so data flow.

A

A

A

A

Install overlay okay, so this is.

A

Before installed over there in class, so we load the data flows, so this would be, you know, default flows.

A

A

Maybe this is overlay.

A

Overlays installed.

A

A

Okay, so and then here's where sort of the data flows class stuff, like overlays, installed.

A

So this thing inputs, uh so this one doesn't take any inputs, but it does produce outputs.

A

Okay, produces output, subtype data flow.

A

A

Okay, so expand result.

A

Now this is plugins installed.

A

A

A

So this would be so config class equals.

A

Plugins installed.

A

So we could pass a plug-in name right, and this would return.

A

Plugin type, all right, so we are going to load a particular plugin.

A

A

How do we do that.

A

A

So we could potentially have so plugins resolve plugins installed, plug-in type.

A

So we could have some kind of plug-in resolver right, so we can have.

A

We can have some kind of plug-in resolver, because this plug-in is like a resource. Each plugin is like a tiny lib resource is like a did, has its own methods and its own event, streams.

A

um Okay, what time check okay.

A

Plugins installed.

A

I can base.

A

To do ah we'll just put it to do section here so.

A

A

A

uh Have a discover, plugin, a plugin which itself is used to discover plugin types.

A

We can then reference by string and load, so this would be like plugin dot, load, um diff, dot, service dot or stiff diff mel dot model right. So this would give you.

A

A

Load the base class for that plugin type.

A

A

Installed plugins.

A

A

So plugins installed data flow merged, so we list all of the installed overlays.

A

We merge them together. We create a data flow from them.

A

We apply the data flow.

A

This is the set of operations which makes up the overlay class right, so we've split it out almost line by line.

A

Yeah so logically, we split it out line by line so yeah, because this is data flow. So if we were to look at the line, we're currently on this merge command loaded, dot, export that really would get split out into its own. That's almost its own operation as well. um In a way, so um at least we can represent it. That way right, because we can represent this at arbitrary levels.

A

A

All right so plugins installed.

A

A

All right, uh so this is.

A

Plugins installed.

A

A

Okay, so it takes an entry point.

A

It doesn't really matter okay, so this is just a stub.

A

All right, so we grab all the plugins.

A

A

A

Okay, so this will merge okay, so and then.

A

So we direct plugins installed to data flow, merge and then the output of data flow merge to data flow from dict.

A

A

Only after all of the records run through where the records in this case are the entry points right. So we merge all the data flows.

A

We load all the overlays, we merge them all.

A

We execute now what we do within the overlay.

A

A

A little rice merge data. We execute.

A

Okay, so why were we in here? Why are we redoing this.

A

Because we got into something else and realized that it was the same thing. Oh, this get multi-data flows, merge, spec,.

A

Okay, get multi-data flows, merge, spec,.

A

A

Oh yeah, so this get multi data flows merge back.

A

A

Grabs the output from all of because we're grabbing every data flow that was seen in the output of running the overlay and then merging them all together.

A

Okay, so it's going to end up doing the same thing, but the source of the data flow is different right. So, instead of plugins installed.

A

So, instead of plugins installed.

A

So so to do for uh initial overlay load of top level system context.

A

We call plugins installed to get all the data flows which need to be merged.

A

um The we then execute the for the merged installed overlays as the uh if load default is given for overlay.

A

On run right, so basically, if we are given the load default parameter, then we go low default and.

A

If we are given the and then when we go low default, uh then when we run the only default.

A

Overlay added is one which.

A

Add it from the main package uh on install.

A

Is one which defines an output operation which.

A

Grabs all the data flows.

A

The input network and uh of the running context for the overlay merges them together.

A

And returns the 2b run data flow all right. So it's a little bit circular.

A

Plugins we load the dataflow.

A

What does this look like.

A

Well, we'll see it it'll show us, because we're gonna write, we're gonna, write, yeah it'll just show us okay, uh I have to go.