DFFML Rolling Alice: Engineering Logs: Volume 1: Coach Alice, 10 Jun 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Coach Alice: Volume 1: Chapter 1: Part 3: Execute Scanning From DataFlow

Description

https://github.com/intel/dffml/discussions/1369#discussioncomment-2663771

A

Hello, hello, hello, I'm right are, we live, see a black screen.

A

And I see a regular screen, fantastic we're back in wonderland, okay, so, let's get to it! This is this pen tabs. I don't think we need this okay. So where were we right? So we are going to see the bin tool oops.

A

And we have a dfmo window open here.

A

All right, let's see where we were at, I need to pull down some changes. Okay, wait hold up what happened here inspected some flow shuffle sure. What is this.

A

What the hell is going on in base right now.

A

Subflow, why were you doing that? Okay, ah switching computers, okay,.

A

A

This looks like good stuff.

A

Okay, all right all right: okay, oh fun, stuff.

A

Okay, we were messing with.

A

Atlas and overlays, and then we were putting subflow on the base class because we were going to make an operation and okay. So because we were going to put everything everything was going to be able to be overlaid because we're going to do um the ast. What we're going to do. Eventually we're going to do ast parsing on the function, then we're going to build the overlays out of the ast purchasing and the functions. And then you can have functions which you can programmatically override from any codebase.

A

All right, instrumentation, okay, so let's jump into df base.

A

Cover subflow, there's so flow okay. It looks like the recovered version of this is better not that much better but yield ctx, okay, subflow, okay, so basically all we did was just apply. The overlays.

A

And then return.

A

The orchestrator context with the overlay supplied okay.

A

Oh that's much much much better. Okay, that's the diff, we're looking for okay, okay um and then we need to uh let's go ahead and delete that now: def, subflow, okay. And so then we need to remember that, uh okay, so uh yeah, I believe yeah yeah overlay class equals overlay right. That's what we found on the last one.

A

Why is that? All staged.

A

A

Can we run something that uses a flow.

A

A

Surplus, so flawed differentiated from the operation data flow. I think, if we did, I think if we ran this, we should test it right. So we'll just run the test for my data flow and see if it works, see if we screwed up anything with that overlay application.

A

A

A

Okay, I didn't even read the error message. I'm now realizing oh dataflow, okay, yeah. Okay, now I did I did. um Where are we we're in df from types import?

A

Data flow seems like a natural thing to import here.

A

Subflow missing required positional argument, overlay.

A

All right we'll make it a required named argument.

A

Okay, this is defined where in system context.

A

Okay, partially initialized module due to circular import.

A

What is this imply installed? Overlays.

A

Oh, this is a this is, doesn't need to be here.

A

This can go in types.

A

We'll save us all some headache if we put it in types.

A

A

Okay, orchestra, just not fine.

A

I think we have an orchestrator already right. We should be able to use self.octx yeah.

A

A

Okay, we're missing something here.

A

Because the overlay.

A

The overlay is a data flow right, yeah.

A

All right, so we use whatever um orchestrated we have to apply the overlay.

A

That orchestrator could optionally have a policy that says um you know this is an overlay coming in I'm going to.

A

Yeah, I'm going to run in the memory strip, memory eric street or something.

A

Or maybe we could take an alternate argument to this.

A

A

A

A

Okay, so here's we might, we might use this.

A

A

Orchestrator equals.

A

The parent, if it's not in lunt, if it's not none, then it equals the one that we gave.

A

Okay, so yield octx, so yeah we gotta have that.

A

Okay, it looks like it worked all right, I'll accept it.

A

A

I forgot this: does nothing look at it? It does nothing.

A

So what do we do we're going to try to just run this.

A

uh We'll just say.

A

You know we'll create the dataflow on the fly and we'll pass the operation.

A

And we'll see what happens.

A

I'm very skeptical of this. I don't think this is running.

A

You're not running.

A

Because you would be failing right, there.

A

A

A

Oh wait we're not even trying to do that.

A

Okay, so run dataflow works, fine uh subflow.

A

A

Oh, I did it again.

A

A

Maybe I spelled it wrong.

A

Over that class is not defined. Now, I'm suspicious.

A

A

Because! That's something that we did on the other branch, all right.

A

A

A

A

Early class is not defined.

A

Oh in high level run.

A

A

Okay, because we have to get back to what we're supposed to be doing, which is we're fast approaching the timeline there so.

A

A

A

So what did we do here? We added to the operation implementation context, subflow uh allow enable application of overlays on sofas.

A

A

Okay: let's go fix that capital r.

A

A

Okay cannot import name.

A

Okay, we have a conflict here.

A

Updated upstream, okay.

A

That's not even used.

A

A

Okay, what happened here? This is important consistent kind of days.

A

A

All right, let's see where we're at.

A

Okay, so we got that still: we've got that still, we've corrected our location of apply installed, overlays.

A

All right level data flow. I had some import changes.

A

A

No default environment.

A

A

A

Active system contact six new arguments: okay,.

A

So active system context very much should take an argument.

A

uh That's the best memory that project this is okay,.

A

A

Make config parent.

A

Maybe we need to read base.

A

Does base make a suggestion based data for context, okay, yeah.

A

Config parent, okay,.

A

Active system contacts, parent system context: okay, config system contacts- config right, so this would be with the overlay applied. So if the parent, if the parent had um if the parent, if the parent would be upstream overlay, orchestrator and then the active system context would just be upstream, because we've already applied the the um the overlay so.

A

uh System, contacts config for active system, contacts should not have overlay, and only upstream, since overlay should have already been applied, resulting in the context config being used here.

A

Where applied, uh output of applied is now upstream and there is no more overlay to apply.

A

Okay, active system context, config, okay, so based it flow object context. So let's just go ahead and derive from that and then.

A

Okay- and you can begin parent.

A

And then we go down to call here, call returns off.

A

A

So when we do the a enter, um so when we do the call.

A

A

All right um god, damn we have to get back to cv mentor.

A

How did we get here? We need to get out of here. Okay,.

A

A

A

Yeah, this is still a little funky right. Okay, so deployment function.

A

Okay, so basically the deployment is this.

A

A

We'll pass the config right now and we'll say.

A

Because I will not remember this.

A

Okay, so how do we raise deprecation.

A

Where is that one.

A

A

What means duplication.

A

Morning, categories.

A

These are technically.

A

A

A

A

All right so config parent ifconfig is apparent, super dot. So basically.

A

A

Okay, so um active system context.

A

uh Dot config as.

A

System contacts.config.

A

Not support error.

A

As support will be deprecated.

A

Sap all right so because we don't want this laying around and I will forget, unless there's something yelling at me,.

A

And it should return, I'm not covering.

A

A

A

Soft parent orchestrator.

A

Missing one required positional argument: data flow; okay, so config.upstream right, yeah, okay, so that looks right.

A

A

A

So we have the system context when we enter the system context, we should apply the overlay.

A

So when we enter the system context, we should apply the overlay and then that way when we pass to the sub-context, we should have this upstream, and that should be all we need right. So.

A

Let's say to do um apply overlay.

A

Okay, so and we'll say, if uh self.config.overlay is not, none raise not implemented error.

A

And no support for application of overlays within system context uh process.

A

Application of overlays within system context, class entry not yet supported.

A

Okay, let's see what it has to say about that: okay, so, okay! So it's not happy with this.

A

Okay, celta config; okay, so it does have okay, so load default. Okay. So it's not.

A

A

Okay, so we just want it to be basically overlay load default deployment, environment.

A

So, what's the difference between load default deployment, environment and.

A

That doesn't seem right.

A

Okay, so yeah, so we don't want load default deployment environment there. We want.

A

Apply installed, overlays.

A

Okay- and we want this to be to do that check so, let's see if we end up in there, no data flow to dict operation, not instantiable.

A

Operation implementation, not instantiable data flow to dict. Let's see.

A

uh No caffeine, all right. Okay,.

A

uh Operation, not instantial initialize data flow.

A

Data flow to dict what the hell is using that.

A

Oh, I just screwed that up. Okay, I see.

A

A

A

A

A

Where the hell does this come from.

A

Okay, so these should have given the lambdas to the implementation.

A

A

uh Okay, so we need to stop this.

A

Oh god, this computer, blue screens, when it clones.

A

Okay, where is it is if a scanner right, okay, so here's this.

A

Okay, here's our docs.

A

Oh, my god, it's so much better on the 4k screen.

A

That's right: this is a computer where node one is alien or python's, alias, okay, the same issue: okay, so.

A

Okay um same issue, so, let's just let's just do it.

A

Okay, so where is it here? um Okay, so by uh definitions, implementation, so where's, implementations, okay, so implementations, so quotations, okay, update operations.

A

Is op not decorated? Okay, so is decorated, so breakpoint. So let's say, if instance, name equals equals data flow to er to dict. Was it let's break point?

A

Okay? Where are we okay right? Okay? So here we are. uh What are we looking at we're looking at um value.

A

Lambda instance name.

A

Okay, so get out your value, it adds her value amp, okay,.

A

Okay, so, and since the name data flow value is up and decorated, false there is nothing non-decorated.

A

Okay, so it has an implementation, it has an operation.

A

So we come in here, so it should say: implementation, operation.operation, name that set default.

A

A

Data flow to dict, op amp.

A

I don't understand why you can't find it operation implementation, not instantiable, oh because we're using a different memory, we're using a memory orchestrator within we're using overlay context, high level overlay system contacts, octx stack, enterprise, contacts async with overlay, that's our like context! Okay, so I think that our problem is just do this from the other one.

A

Okay, let's still see, we've been told.

A

A

Okay, we'll do this over here.

A

All right so now it's installed so now we can see it over here.

A

A

I think we can safely say it's being set in the implementations, so we're using the wrong. We must be using the wrong orchestrator somewhere, so 58, okay, octx self, config upstream breakpoint. Where are we.

A

Okay, so overlays over the contacts.

A

A

Oh no dear from high level, okay over the class orchestra has overlay synchro.

A

Okay, so dfm overlays installed, which is a system context, so.

A

So this is a system context and I don't know if we finish that unification there. So we see that we're trying to see the input set with the cv bin tool.

A

All right so we're running cbd, bend tool, we're using the fml and we're with all our in progress code that um lovely so lovely lily hit on this other machine. So we switch back and forth so.

A

Okay, so there's a memory orchestrator there's a data flow okay, so.

A

Let's check this thing out: okay, so this is our giant git repo analysis, so this is our whole. Should I- or this is our whole- um get features. Let's see, scan directory, okay, all right, so overlay.

A

Let's check out the the overlay.config.

A

Dot upstream export right, so there's our there's our yeah okay. So this is our default installed. Overlay.

A

And okay, so async with overlay is over the context. Okay, so if we step into this or if we, if we go, if we go past this, then we should see that error message that we were hitting where it says on the next line over here.

A

This line 213.

A

A

A

A

A

A

A

Okay, so overlays over the counters and really context apply okay, so.

A

We enter the overlay context. Let's just do it. Let's enter the overlay context. Okay, so.

A

Now wait a minute now, I'm confused it wears the problem.

A

We come through here once already so, what's going on two two two two one two.

A

My system, contacts.

A

Enter ctx so yeah, so.

A

hmm Will we get to here no okay? Well, that's good.

A

A

A

So this is the one in question.

A

Okay, so it says it has no implementation. So that's odd! Okay. So how did you end up with no implementations.

A

So, let's check out.

A

Okay, different licensed.

A

All right online package overlay.

A

Oh because we're doing different overlays installed, which is merging the exported, but not back with the okay, so it merges the export. So this is our bug so right here on line 174, we merged the exported main package overlay with this one that goes and grabs all the installed ones yeah, because the main package overlay is generic, and then we say this one is specifically for applying the installs.

A

Upstream over they apply.

A

Different religions, don't say different overlays and some different overlays installed, and then oh wait. So this is wait. No, this is none because.

A

This is not because we've already already applied the overlay.

A

A

Wait a minute okay! Now, this is what this is so dfmo overlay is installed.

A

A

A

None: okay! We don't need anything here because I think we're retrying somehow.

A

Okay, so let's just go through and do this: let's try some versions.

A

Okay, so for operation in.

A

Operation instance, name keys.

A

Implementations.

A

Instance, name.

A

A

A

So we'll do a little get at her.

A

So basically get it from here: dot, implementations.

A

Or get it from.

A

Here, dot, implementations.

A

But you better get it from somewhere.

A

Protect object.

A

Okay, so we can just do this dot get.

A

So we'll say: op amp, so the operation implementation equals this all right. Let's say if op-amp is none raise exception.

A

And then we set it.

A

A

And so populate implementations from lambdas.

A

Which were lost during above export.

A

Okay spelled implementations wrong.

A

Dataflow object has no attribute implementations.

A

I find that very hard to believe.

A

Another misspelling implementation, normalization.

A

Within okay, so is there merge? Okay, so there's merge okay, so it says it wasn't able to find this merge.

A

So why wasn't it able to find the merge? Well. So this is the phenomenal main package overlay here right. So oh yeah, because we get the instance name. Okay,.

A

A

Okay, so to merge.

A

Okay, name: dataflow merge; okay, so this is just named incorrectly.

A

Active system context object has no attribute apply, thank god, okay. So next error.

A

A

A

A

A

Okay, so this is not even needed.

A

Okay, so, okay, so yeah, I don't think we even needed that.

A

So basically we don't need that um because we're going we we already applied the the dfml overlays implied applied. Is we already merged it? um So?

A

Okay, so? And because, if we entered remember, we added a not raised not implemented error if we were to apply the overlay um in the in the entry. So we're going to do it right here right because we can't apply an overlay to an overlay right now, because we don't we it's not implemented earlier. So, okay, so overlay uh fix so.

A

Dip mount overlays installed.

A

Already overlaid no need to specify to load again.

A

All right, so what happened here so here we fix this.

A

So overlay do flow main package overlay.

A

Okay, so fix merge, op, name and consistency.

A

Okay and now it's a good command.

A

uh So overlay, um do you have no installed overlook overplays installed? um Oh, that's got a break point in it.

A

Okay, so uh carry through implementations.

A

Defined as uh in memory from merged flows,.

A

Okay, so where are we now, let's commit this.

A

So bass and k-arg.

A

If not is class field.

A

Instance of a class use the value from that instance.

A

A

Default data classes.

A

Okay, I see what we're doing there, so this is, like you know, support for applying.

A

Arc default value from instantiated.

A

Data class field.

A

What else we got? Okay, so.

A

A

A

So some context, so let's jump into the system context.

A

Okay, I should have just woken up at three. I was awake. She just got out of bed, god, damn it all right.

A

A

Okay, so 59.

A

So where are we here? So what happens if we move this breakpoint.

A

A

Active system context has no attribute apply, so what are we doing here so active system context says no attribute apply.

A

So an overlay, an overlay.

A

A

Maybe we'll just run the default.

A

A

A

So yeah so we'll just support we're going to get rid of the apply.

A

A

Df base and high level, okay.

A

Okay, so here's the application of the overlay.

A

A

Active system contact system, okay,.

A

So what does this even do? Okay, so this gives you a function which we'll then call the orchestrator okay, so after system context self as system context, okay,.

A

So this thing comes with an orchestrator ah yeah: it comes with an orchestrator. Doesn't it yeah? The overlay specifies its own orchestrator.

A

If we wanted to overlay the overlay or well, we would give it our. We would give it a different orchestrator if we wanted to do that. Okay, so.

A

A

What is the overlay? It's a class different overlays installed.

A

Okay, okay, so if inspect.

A

A

Okay, so this thing really should be instantiated by the time it gets to us.

A

But maybe not is class overlay overlay equals overlay.

A

And then we say overlay.deployment.

A

I'm always shocked when inspect is not already. There.

A

Okay, funk takes zero positional arguments for one was given.

A

And overlay that deployment okay wait a minute, so apply overlay equals overlaid that deployment, and that was it to do otherwise.

A

Right so we want to use the overlay, apply deployment of this system context and, let's say dataflow equals results.

A

Okay, so there we go so we got an output, okay, so so ctx.

A

Okay, so we got our output, so data flow equals results. Are we applying the overlay? We got the output okay. So let us.

A

So we're going to turn on logging, so we're just going to import login here.

A

And show some debug logging take a look at it, okay, so, okay, so let's take so. This is our overlay stuff right, so we instantiated all our operations within our overlay great. We had success there, so we ran it. We seated with the cd and data flow with inputs at blank okay, so- and I believe, is this still- this is just oh, so apply overlay right. So what is this so? This takes um arguments that are from the.

A

Because this should like do that short name thing right. So what do we have as inputs? So data flow being overlaid, so we should just be able to say data flow equals data flow.

A

What happens now? Okay, hey! Look at that! Oh look at that.

A

Data flow from dick- oh my god,.

A

We also have to remember to merge the implementations.

A

Thank you, john.

A

A

Okay have another hour.

A

Okay, I have another hour: okay, okay, okay, okay, fantastic! Yes! Thank you, john all, right! Okay! So let us not forget: um let's go actually! uh No okay! Let's not do that! Now, let's do um let's, let's use our friend the warnings, okay, so to angles. Result that items result dot, overlays merged right!

A

That's what we wanted.

A

Oh it's! A real dataflow object. Let's see if it runs okay operation, not instantiable repo to directory all right, so we ran into it right away. Oh, this is really exciting. Okay um holy! This is cool. Okay, um oh gives me chills. Okay, wow, all right! Oh! This is gonna, be great! Oh okay! I hope I hope it's gonna be great, all right, all right, all right, all right, all right!

A

Okay! So let's go.

A

A

So we're going to need, we obviously need some kind of little generic application of overlay helper, because we've I've done this a couple times now. So where is.

A

Overlay overlay, hello, hello, hello, overlays, okay, so, let's see apply, merge, merge data flow, merge, okay, here, so merge, okay, so critical piece after this, though right this part all right, so we're probably just gonna pull this out into its own thing. Its own function right. So let's just put it here, and maybe we should actually put it here.

A

A

A

And let's see uh you know, this is update, implementations or, let's see, uh preserve or let's see, uh um merge implementations and I'm not going to try to spell implementations as much as that's probably going to be good for me. Okay, so so we have the target.

A

Which is a data flow right and then we have the.

A

All the rest of the data flows right and what do we? Output the same data flow.

A

I yeah, I often go back and forth. Do I return you know, do. Does one return, the object that they received? I think it's helpful for chaining calls.

A

All right so yeah, so here's the target cell so add the implementation.

A

To the target so complain.

A

If we can't find the implementation anywhere, okay, so.

A

Let's do this now, so, basically we're going to say.

A

We're going to go through arg and we're going to try to get the instance name, so we're going to try to get it.

A

So we're going to go through all of the.

A

A

All right, so here's the operation implementations right, so we got a list of them and we can say: uh can we de-duplicate by the same ones? I think we can just say.

A

Deduplicate by.

A

Yeah, we'll just duplicate by id, okay, so basically the same objects in memory. We do not want to duplicate them right because we are in memory right now. Remember that so that's that's important right, so merge implementation, uh because, obviously we can't you know, you know you, you, don't you don't. If it's not a memory, it's not! It's not involved right. Now uh we're only dealing with in-memory stuff, so um yeah as we deal with the.

A

Yeah, so some implementation networks would load them right, but the implementation network would be responsible for loading in this case. uh We're just we're only concerned with maintaining in memory uh implementations it's because yeah.

A

A

Okay, uh can we do that.

A

So this is guaranteed to be unique among simultaneously existing objects, yeah. So.

A

All right, so, let's go through all the op-amps and we'll say.

A

All right, okay, so now.

A

So we create a list of all our operation implementations right from each uh from each argument. So let's see for op amp for arg and args, we look so we look for the operation instance name within each data flow which we're merging right and then we, you know you do this little uniqueness check. So if uh len op amp is.

A

Equals equals one or equals, or it's less than one right so within.

A

uh No implementation for blank right, so merge, implementations, okay, wait a minute! So this we should actually go through okay, so we should go through. We don't want to do it this way, because what? If what that resumes everything's in memory, so uh we need to say four organ args.

A

Okay: let's go through all the implementations; okay for.

A

Okay, this is better yeah. This is much cleaner, okay, so four yeah.

A

A

A

A

All right, so, let's go through all of them all right, so we use it or tools.

A

All right, so we hitter tools we chain, all of the. uh So basically you know flatten out that list so because we're going to have a list of lists right, so each items is going to be listed so flatten out with it or tools chain and then are we still on the internet.

A

We are still on the internet. Okay, oh I'm supposed to be tweeting. These links, oops, okay.

A

I don't have time for that right now.

A

A

Target implementations.

A

A

So yeah, so we don't need to complain about that. What we do need to do is say.

A

A

So we add the implementation to the target so check if we already have another conflicting version of this implementation,.

A

Okay, and and so, and so what, if somebody doesn't like this function? Well, they can override it right because everything is overlayable.

A

I really gotta fix this. How do I fix this? Could I just put it here: that'd probably be a lot better, just putting it here.

A

Oh, my god, that's infinitely better holy wow, okay, wow, all right wow! I should've done this last month, okay, so if target implementation, so if.

A

So if we already have it and.

A

A

A

Sound well merging so non-unique uh op-amp.

A

Blank convo merging and then we'll say you know, conflicts.

A

A

So basically, this one exists in target and attempted to add.

A

Right so currently exists in target.

A

Okay, so basically, what do we want to do? We want to see if it is the exact same or if it's not the exact same right, and so, if it's not the exact same, we're going to bail out so merge implementations so.

A

A

I don't want to screw up the stuff that I had so, let's make sure that that was committed at some.

A

Point: okay, so that does currently exist in there. We gotta commit the rest of this.

A

A

This is the only delta on this file right now. So that's okay, um not ideal, but it is what it is so.

A

Right so now we're calling merge, implementations and we'll merge this one into this one and here's our targets right so basically take these right and populate.

A

A

Okay, it looks like it's happy, it looks like it ran. It ran the previous flow right, so we did get these instantiated. So I think it worked our update here so now we just need to do merge, implementations.

A

So now we need to go here to merge.

A

And we need to say, merge, implementations.

A

So this does what okay data flow. Okay, so we need to overlay, therefore merge merge data flow, so therefore data flow from dict okay. So we need to do this here.

A

Merge implementations.

A

Okay, so stage I'll put data flow, stick merge.

A

A

Apply to data applying overlay today to be actually inherited.

A

Is that even being used.

A

I think this is not complete.

A

So we want overlaid, not overlays merged.

A

Data flow being overlaid.

A

A

Tearful being overlaid.

A

I'm gonna go get some food.