DFFML Rolling Alice: Engineering Logs: Volume 0: Architecting Alice, 26 May 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Architecting Alice: Volume 0: Context: Part 50: Enter System Context Stage Right

Description

https://github.com/intel/dffml/discussions/1369#discussioncomment-2824606

A

A

Okay, I realized something I had turned the stream off and I realized that.

B

A

Here, what we need to do is overlays installed. We don't care about results, overlaid because results. Overlaid won't exist.

A

We want results, merged and.

A

We can get results merged.

C

By wow to get single.

A

So we can get results merged.

A

By adding in another operation.

D

A

Okay, so this is, uh I get merge.

D

Data flow after.

C

Overlap merged.

A

Output stage same thing.

A

A

So that's it that should be all.

A

Didn't like that, okay, so.

A

A

Get merged okay, so stage output.

A

Okay, overlays installed get merged; okay, here merged.

A

C

See data flow with input set.

C

A

Overlay batch apply.

C

Batch apply, this should be.

C

D

A

A

Here merger ran that data flow. It said these don't match so.

A

Overlay apply batch.

A

Okay, we add these inputs.

E

A

Wait a minute: oh yeah! It never calls this so list.

A

A

Definition, data flow type, okay, so my guess is that these are not being converted into the same things: different df types.

A

So where's our input object.

A

My guess is that here this is wrong, uh create definition.

C

Data flow to apply as overlay.

A

So old definition equals definition.

C

So create definition, new.

A

C

Mouse version.

A

A

Does this thing have like a compare eq export.

C

A

A

All right, so we added a little debugging statement here.

A

That checks the equalities.

A

So what the hell is all that.

A

Flow being overlaid, so what we're going to do here is we're going to try to look for discrepancies within things that look the same, so we're going to say now if.

A

So if this thing is about to return false and they have the same name, print.

A

C

D

Kubernetes javascript job.

A

C

Because physician.

A

A

All right yeah, so my suspicion here is that we have two things at the same name: tape here.

A

If name equals, oh.

A

All right, yeah there's my suspicion, so er.

A

Two things with the same name: okay,.

A

Okay, so what's the problem.

A

We're not calling this thing.

D

D

A

Wait what does this look like in the future? Async def main, I think, with.

A

A

Okay, so what do we have here? So we have a system context.

B

A

A

ah Orchestrator.

A

A

So we have the inputs, we have the architecture, we have the orchestra.

A

Okay, that's the system context. Okay, so.

C

Overlay batch apply versus a relationship.

A

Use this data flow to merge these overlays into this data flow and apply merged overlay to this data flow. Okay,.

A

Okay, so use this dataflow.

A

Okay, so with system context.

A

D

A

Flow, so if we wanted to define a.

A

Overlay data flow to your main package.

A

So we could take the main package overlay.

A

So we could say different online package overlay.

A

If you want to just combine.

A

Okay, what was it we were gonna do here.

A

Dfml overlays installed.

A

So this dfm overlays installed.

A

So this overlays installed equals data flow dot from dict and then we just.

A

We say uh domain package overlay.

A

A

Combined with a different data flow, so data flow.

A

A relationship.

B

C

A

So I'll just do this underscore.

A

So this is the data flow that we're applying.

A

All right so effectively we're doing a merge right so.

A

All right, so you take your your main package. You merge in your dfml overlays installed like the underscore version of that right, and so what is the underscore version of that? Well, it's this.

A

Okay, so then, here we'll just run on the default. We can then skip all that and we can just run this different overlays installed.

A

A

A

Okay, so overlay batch apply.

A

Regular batch applies the same thing.

D

A

Match apply, it's like the same thing so.

A

Tiff mall overlays installed.

A

All right, let's depend defaults to true okay.

A

So we run this to release installed.

A

All right, let's do the update our float goes true. I think we need to do that on the output as well. So let's do that there.

C

A

That should update the that should re re-auto-generate the flow. um So uh because remember we have this inside the data flow. We have this by origin structure, which maps the input flow.

A

And you can modify that manually if you want to okay so but then update with autoflow attempts to figure out based on the operations inputs, what the flow structure should be right. So if you update the flow structure manually, then you don't want auto flow equals true.

A

A

If you're going to update the flow structure manually, you should do it after apply overlay to dataflow to be executed. If you were doing it in this case, so we'll do it there. Okay, so.

C

A

C

Okay, so dude overlay's installed so give merge. Okay, therefore,.

A

C

Basically, says.

A

So we can drop this operation, dot, export.

C

A

All right so we're going to replace this.

A

Okay, so we replaced apply, overlaid dataflow to be executed with get merged data flow.

A

Get merged overlay or get data flow overlay after merge.

A

Or after merged right so.

A

So basically, we replace this thing with the other one uh for this dfml overlays installed, um because this is going to go query just the installed overlays. um So we don't care about. uh You know we we do. We only want that result of that merged because, let's see.

A

A

Okay, dfml overlays installed okay, so this is going to be what are the default? Overlays, okay, so default default?

A

Is this let's see the default.

A

A

Overlay.Default.

A

Okay, so we have the default. The default is this thing.

A

I'm not sure what these comments say right now, but we'll put them up here.

A

A

All right, so this overlays installed.

A

If not overlays installed, okay, so basically.

A

So when we run dfml high level run or data flow.

A

Overlay, the default.

A

Okay, well, it's not load default anymore.

A

We can actually just grab it directly.

A

So if that overlays instead.

A

A

So this would be.

A

A weight data flows, class.

A

A

Airflow as class.

D

C

A

And quick overlay as overlay context.

A

Over the context.apply dot overlay batch apply.

A

All right so now we're following the double context: entry pattern.

A

Okay, so I said load default, then we grab dfml overlays installed.

A

Okay, load default or data flow, so we can just say it is dfmo overlays installed.

A

B

A

A

Okay, dude, oh there's installed. I think this should be because this will turn into this.

A

Run is the run system context.

A

So this is like the config class for the system context.

D

A

So you could apply the overlay before you get here and then just pass data flow with overlay equals none or you could, and how would you do that while you would pick an overlay or you grab this one which gives it a default install.

C

Overlay is installed.

C

C

I wonder if we have any input collisions here so definitely.

C

A

I'm just going to comment this out for now.

A

We're using this overlay batch apply anyway,.

A

Nicer forward to do data flow as class and pass the data flow.

A

You can't do data flow as class, but you can do system context. This class.

A

And why is that?

A

That is because.

A

That is because a data flow on its own is just it has no method of execution, so you can do system context as class, but you can't do data flows class. Let's write that down.

A

All right, so we can do system context as class.

A

So does that mean we can do create system context.

C

A

So we should really be doing okay, so, okay. This is some context, this class, because uh at diff, but not data flow as class, because uh data flows have no method of execution without an orchestrator.

A

um So we should switch the overlay infrastructure to support.

A

Taking a system context instead of an overlap or data flow.

A

To work with system context.

A

A

So yeah, so we should really when we should introduce system context infrastructure. uh We should introduce.

D

A

So we will introduce the system context, data structure which defines inputs, architecture and overlap and orchestration.

A

A

A

Turned into a data flow enter a class.

C

A

The previously called data flow as class methods where methods are send inputs into the flow with the running system context where methods send inputs into the running system, context with origins as labeled uh via system context to cla or as class uh keyword arguments.

A

So we won't have to define the origin of default to see so default defaults to seed seed, if not set okay, so this can be turned into class three. So previously called data qualities class under where methods send inputs to the running system. Context with origins, labeled via system keyword, arguments, uh input, objects, added, matching uh method, signature definitions, uh type hints.

A

C

A

Definitions: okay, dope- haha this is great okay um equals. So we can say dfmo overlays installed.

A

A

Diff installed overlays.

A

Equals system context as class.

A

A

Okay, so create a class for calling operations within the system context as methods, okay,.

A

All right so we're defining the system context. I thought we did this somewhere else.

C

Oh down here, I guess I swear, we did this. Someone else must be on another branch.

D

So our architecture is.

A

C

Open architecture.

F

A

Inputs is a list of inputs, okay,.

A

Okay, so system context says class class.

A

Alright, what is this this is an operation run system context, okay, run system context. Is this so.

A

All right so we're going to create a new meta class based configurable meta class. Let's take a look at how this worked.

A

So metaclass defines like how the class methods work on the real class, so this is, for example, we wrap a knit so that we can create config to config classes in the init. Actually, we need to fix up the init signatures and we'll do that here as well, uh now that we know how to do that, um so this configurable matter class.

A

Get the subclass just configurable as pass queued. Okay, so.

E

A

Space configurable, okay, so system context is class.

A

Okay context is class, metaclass, okay, so for uh create method methods for each operation, all right, so we're going to create methods for each operation so for operation instance, name comma operation in self.dataflow.

A

A

A

Okay, so if that config.

E

C

A

Okay, so create method for each operation: whoa, okay,.

C

A

System context, s-class meta class, we're like way deep in the stack here.

A

Pop back off, eventually, um okay, so system context, says.

D

Middle class, okay,.

A

So this is the init. It says: system contacts, this class matter class.

A

System context as class is an operational implementation, so let's check out operation implementation.

A

Operation, implementation.

A

Self context, config.

A

Okay, so I think it can be an operation implementation system context can be an operational implementation.

A

Can we have multiple meta classes because base data.

C

A

I don't think we can have multiple meta classes um because it changes the new okay, but we can try. Okay, so system context is class.

E

Meta class, okay, so system contact.

C

Says class, so if you set.

A

Data flow on this thing, so medical system contacts this class metaclass equals this.

A

System context is class.

A

We'll create a config object.

D

Okay, to configure.

A

Directory system contacts as well.

D

Where they can.

A

Is this system context.

A

So this system context, whatever the system is system context- is becomes the default, so we create a new class.

A

I think we need the name. I think we need to be past the name, so we create a new class.

F

A

A

A

A

System context says class.

A

We're going to leverage.

C

This run function.

A

We take the original system context.

A

And we create this config structure.

A

We don't put an operation.

A

Are you up at a class.

A

You output, an operation, implementation.

A

You output an operational implementation which, when double context, intrigued okay, yay.

B

A

Here we go so we take a system context. Where is our make config stuff.

A

ah That's right: we made this handy-dandy subclass method, okay, so dot subclass.

A

And this was for cli commands.

A

Okay, now, what did we do in here.

A

Make new config, so this should be its own thing.

A

A

C

A

A

Okay, let's see this is.

A

Replace config, okay, so.

E

A

C

Deaths subclass. Oh, I thought I took this out. Oh no, I just did a read. Okay.

A

A

So we're going to do this.

A

Subclass config, okay, so this is the general pattern.

A

D

A

All right system- oh.

C

A

Oh okay, wait a minute! So oh ouch, okay, um so wait a minute. So maybe we can just throw the meta class right on there.

A

Yeah! Maybe we'll just throw the meta class right on here system context.

A

A

Okay, validate.

A

All right so we're gonna set valid to false for now um wow. I really screwed up my back um ouch.

A

Okay, we're going to set valid to false for now, because we're not using it yet. But we need to remember to have a validity check um on the system context because it can't be executed if it's not valid and we can determine its validity in static fashion. So.

A

Okay, so when you enter the system context, it will start the data flow.

A

The orchestrator will run the data flow.

A

A

I think that yeah, open architecture.

F

A

All right, so we know that there's the data flow is probably a component of the open architecture.

A

There may be other metadata involved with the open architecture so with the architecture so.

A

A

Do we know all urgents for all inputs used and.

C

C

A

Oh, I o declared unused or explicitly.

D

A

Okay, so, okay, so we'll.

A

So when you instantiate a new instance of this thing,.

C

A

Context and then this is a running system context or active.

A

So it must be valid if it's active yeah it wouldn't have been able.

E

Active system down.

A

A

Okay, so an active system context the system context.

E

Active system context.

D

D

A

All right, so I'm going from the cli right now do read dfml data or read df model. Cli data flow um looking around to see if we could.

A

Leverage they enter methods there. um So where do we intern orchestrator.

A

Soft orchestrator on data flow.

A

Okay, I'm looking for a read, get grab async context uh enter within.

A

A

Or interracing context, okay,.

C

Enter async context.

A

Okay, so yeah here we go. I'm like. I know that we're doing this I know it. Okay, so looks like these all have an underscore underscore stack. Okay, so self stack. This has an underscore stack, um ace, jackie stack, a stack, a stack, okay stack, stack, stack.

C

A

Exit stack test, okay,.

C

A

A

C

A

Exit stack so now we're looking for previous usages of async exit stack.

A

This is what I want df memory.

A

Where is it oh wait?

A

So this is what I'm looking for right here is this exact enter await, enter that that flattens things out.

E

A

A

Exit I'm seeing if there's okay there, that's what we want.

A

Yeah, this is what we want.

A

A

So I think we can just call to say close, okay, so interesting context.

C

C

System context, config system context.

A

Okay, so system context.

A

Config system context, config, okay,.

A

C

Active system context.

D

A

A

Celticconfig.Orchestrator.

A

Okay- and this.

A

Kicks off the data flow.

A

C

Active system context.

A

Okay, so orchestra.

E

A

Okay, so parent config, this is very similar to.

A

This is very similar to the orchestrator itself.

C

C

A

C

A

Octx, okay, so um the okay, so parent active.

C

I should say.

A

D

Basically, facilitator object: okay,.

A

So this this system context thing it has an orchestrator.

A

It enters the orchestrator okay, when we want to create a new version of it right. So we want to create a new system context right. So here is.

C

A

Config as defaults for.

C

A

So we'll just call it subclass.

A

Okay, so subclass basically.

D

A

Okay, so okay, we can have this subclass method which says: okay, what's the new class so create a new system context replace the config with field modifications so effectively.

A

Just set the defaults.

A

Okay, so effectively set the defaults.

A

So go through go through the existing system. Context create a field. Modifications object, okay, so this is what our field notifications should look like: data classes, dot fields for field and field.name; okay, all right, so we want to go through and make the default factory for each field whatever it is in.

A

All right so this says.

A

Okay, so subclass, so we want to create a subclass of the system. Context. Okay, so well, what's the config? Well, the config is a system con. So by default the system context config contains no default values right. So this system context config up here. You must specify everything right so now, if I want to create a new system context right with the um with the values from a con from a populated config right. So I want to create a new class which represents a system context which can be executed right.

A

Then, I'm going to pass a system context config, so, okay, config.

A

Right- and that is the config to be used as a default value, and then we will do this yeah we'll do replace config right.

D

A

Oh yeah where'd that go went into base.

D

A

So that should give us okay field name, so that should do field name and then default. Okay, uh blank new class name, config, subclass, okay, so system context, dot, subclass,.

A

And then we say.

A

System context config.

A

System context config: maybe we do system context, config right now, system, context, config class for config or, let's see defaults config as defaults.

C

A

Okay and then we pass the config so.

A

Open architecture, data flow.

A

D

A

Right, okay, so we know that this thing works with the memory orchestrator.

A

Okay, so what do we got? What do we got?

A

I think this looks pretty good dfml installed overlays. So now, what do we do?.

A

We say okay, so now.

A

This functions as a plugin right, so dfl, dot, overlay or system context similar to the overlays.

C

A

Base entry point: we basically make another class.

A

We'll call I do not system context. Okay,.

B

A

Okay and then, when we create a subclass.

C

A

Go ahead and say.

A

We'll make it an entry point.

A

Okay, config is defaults for subclass, okay, just oh man, wow something's, really messed up. Okay,.

A

Great so dfml installed overlays that this effectively becomes a plotable plugin just register as an entry point or so do from uninstalled overlays. So we should also.

A

Okay, so what happens over here so overlay.

A

Equals system context.

D

Okay, overlay equals.

A

A

C

A

So this is a system context based base orchestrator system context; okay, so it overlays the system context.

C

We'll import the system.

A

Context from there, for now it overlays the system context. Okay, if it's load default, then we're going to say dfm, overlays d installed.

A

A

So any system contacts can go here.

A

Oh any system context.

E

E

A

So we have an orchestrator.

C

So we're going to use our orchestrator.

D

D

So if orchestrators systems so orchestra.

A

None if orchestrator is none use whatever orchestra you want.

A

C

E

Default factory.

A

Default factor doesn't really do what it's supposed to do anymore.

D

Okay, so now we're gonna.

E

Make this ergonomic okay.

A

Okay, so basically.

A

If overlay is not none.

D

D

A

Overlay is not none, then we want to do something specific. We want you to batch, apply.

A

We want you to apply.

A

Okay, so we say async with overlay class as overlay.

A

And then we say with overlays over the context and then we say overlay context.apply. So then we overwrite the data flow with the new data flow as a result of the application.

A

C

So, let's use the orchestrator.

A

E

Okay, orchestrator.

A

So that should reuse if it's already open, okay, so wait. Wait then quit.

D

Over that class orchestrated so right.

A

So then, over here we have orchestrated.

C

Equals asynchronously.

D

Orchestrator so the.

A

Orchestrator should the orchestrator just be re-entrant, then the orchestrator should probably be re-entrant to do.

E

A

And that means that you can do a enter on them multiple times that screen them up. So let's go take a look at.

A

A

So a enter so it will create a new.

A

B

Will create a new.

A

So it's not re-entrant.

A

C

A

If celta entered return, south.

D

A

Okay, so that'll make it re-entrant.

A

So, okay come at df memory, uh make uh memory or kistrator re.

A

I do not reset stack and networks.

A

Objects, and or do not reset stack if on.

A

Subsequent entry, okay, wait a minute.

A

Do not reset a second okay, uh wait to remove last item until exit.

E

A

Okay entered equals.

C

A

All right certain on exit, we say: minus equal one. If.

A

Less than one.

A

If entered is so self.entered plus equals one on entry all right and then on exit so close.

A

So if you're one then do the stack.

A

So, okay so enter to zero on first entry increment, if you're greater than one.

A

If it's not the first entry, just return on exit, if it is the first or if you finally get back to zero, then if you're less than one then close.

D

D

A

A

Config as defaults for subclass.

C

A

Yeah, I won't leave that note in case that patch that gets removed all right. So I guess that patch gets removed. Okay, so.

A

A

So if we do this, that means dfml installed overlays whenever it runs everything in the global namespace is going to access. It's going to use the same memory orchestrator. We don't want that.

A

We want configures default.

A

There, it is all right.

A

Okay: this is a little funky, so.

A

If not installed, overlays system context, config just install the release, config.

A

Architecture, orchestrated we're close, so this thing.

A

It's all like, I think this might turn back into a data flow.

A

B

A

And so with your so properly it's not none.

B

A

Install the relays config.

A

So overlay context.apply, okay, so this thing better have this apply operation.

A

The apply operation should take the data flow. Okay,.

E

A

Okay, the apply operation should take the date form.

E

Alright, contacts.

A

So overlay apply.

A

So overlay apply is a callable.

A

So we're actually going to make this call, though,.

B

A

Yeah, it's a co-routine.

A

E

Different installed, overlays config inputs, architecture, orchestrator.

E

Is default, class config is.

C

Default version class, okay, so default.

D

A

So what if we did it like this all right, so we async def apply overlays.

C

C

Df file installed.

A

All right, so if we treated it like a function.

A

A

I would say.

A

A

All right, so here's the caller here's the collie.

D

A

Overlays df monst.

A

E

Okay, where are we? I feel like from my valve track here.

D

Okay, so this overlays install so.

A

Default release install gives us the really beautiful aftermarket. So this is the merged set so running. This dfml overlays install gives you the the the overlay, which is the merge set.

A

Of all the other overlays so do overlays installed. You know diff.

C

Overlays install.

A

A

Definitely overlays installed. Okay, so.

C

A

So let's go down here so right now, let's go over here.

A

E

Over there applying it.

A

All right so async with orchestrator if overlay apply, is not none.

A

Okay. So if overlay, if overlay is load default, then we say.

A

It's this default overlay.

D

I'll be right back.

D

A

A

A

What what are we really trying to do here so we're trying to allow for people to do organizational overlays when they run the should I commands, which are on.

A

A

A

A

A

Oh, maybe it's because I am standing really weird, probably.

B

A

Yeah and that's right so we're doing this on the version command. Oh we're doing this on the version command right now, because we want to show the fact that it would default overlay. Okay,.

A

So we're doing this on the right. I don't know.

A

Default overlay, I think that there's like a lot of things. That should be the same thing and it's not clear right now, uh which ones are so.

A

A

A

Time to crack another high ball.

F

A

So what else did we learn about this.

F

A

I understand now thank you. Thank you, alice, okay, all right, fantastic.

A

Okay, so what did I just understand.

B

A

We have a handy-dandy comment somewhere here that says that possible, as class should be. The system context, says: class yada, yada, yada, yada yada yada, okay. So why is that important?.

A

Because of this, the parent, the parent object on the active system context- this is critical. This is absolutely critical. Why is this absolutely critical? Because this is how we maintain um this is this is this? Is the whole did situation? This is how we maintain.

B

A

This is how we maintain context: okay, so active system context.

A

All right, so the input value is of type system context, so parent input, parent parent parent.

A

Or so so the active system context takes parent which is system context system context takes parent, which is an input.

A

The input is where the system context came from. This was critical. We forgot about this okay, so then what can we do? So, let's try to define the cli command as.

A

All right, where are we over there? Dsmile overlay overlay overlay.

A

A

A

A

Version equals lambda.

A

Import lib metadata version now that's what I call a command line: interface, um okay,.

A

A

Okay, so we could auto wrap this.

A

Let's focus on: let's focus on what we're actually doing, okay, so this would be.

D

A

Should I cli so the cli command, so when the cli command runs, let's try to make sure it has a parent system context object. Okay, so when the cli command runs.

A

If my util cli command.

A

A

Okay, so when the cli command runs.

A

What does this thing? This thing has apparent input.

A

We're so close to something I can feel it.

A

Maybe not, I don't want to jinx it well, okay, so.

A

B

A

So parent value is system contacts config, okay,.

A

Apparent definition is system context, config, okay,.

D

A

So parent values is the instance. The definition is the system context config.

D

A

Turtles all the way down. Okay, so.

A

Let's try this so let's try.

A

The system context.

A

So when we execute this thing,.

A

Command do run.

A

A

Oh, this is the init okay. So here we say soft dot system context, off.top level.

A

uh The cli is the caller and, as such, must represent its environment as a top.

A

A system of context, okay,.

A

So top level system context, so self dot, subtitle system context equals.

A

A

System context config.

A

Okay, so system context, config.

E

A

So parent equals none uh inputs, equals input.

D

Value equal: I should.

A

Kind of kill every rx extra config.

C

A

Okay, ah interesting, okay, so.

A

A

So what is the data flow that describes this thing?.

B

A

Okay, so do you have types, okay,.

A

Okay, so data file definitions is all right, so addict.

A

So definition, definition for definition in.

A

Create definition.

A

All right so now we define all our new types, new type.

A

So top level system context data flow, so new type extra config predict.

A

Util.Cli.Command.Extra config.

B

A

A

A

A

So command keyword, dogs and then extra config, so extra config is used.

B

A

A

So we're going to define a couple new types, we're going to say extra config, it's optional and then keyword, arguments and.

A

We're going to add those as inputs.

A

All right, because this is our promised calling convention.

A

Okay, extra config, kwrx right.

A

So if you wanted to run the same thing here,.

A

Top level 67 contact inputs.

E

A

Equals open architecture data flow goes to file.

A

Okay, so how would you describe this top level system context? Well, so it has no parent. um Well, at least we don't know about the parent. Really I mean we could maybe create another. We could. We could potentially uh traverse up and if we wanted to, but we're not going to.

B

A

Okay, so open architecture.

A

A

A

All right so run cli command.

A

Output equals results all right. So that's this to run.

C

D

A

Okay, so extra config.

D

A

So this would be, I think,.

A

Seal our command.

A

A

A

All right, so the results might be like union list any or any. I'm not sure what the type is. Let's see um so run cli command.

A

There's a config twx, okay. So what does this do? Well, it says: command await return away, command, dot do run and then extra config equals extra config and then star star cadillacs right. So this will replicate. So if we ran this flow, we would end up with ncli command. So this would be.

A

Right but it'll decorate it for us so um so run. Cli command takes the command extra config, the qwr exam returns. The results, okay right, so that should be the same as await. Do run, wait to run perfect, okay right, so it does this part right so by the time we're running right. So we did do run okay, so.

E

C

A

All right, okay, so open architecture and luxury; okay, so top level system.

A

Okay, so maybe maybe we should just say some context: config.

A

Okay, maybe we should just say: system contacts, config, okay, so uh system contacts, config.

D

A

A

So we found the right cli command.

A

A

Orchestrator is not okay, so that's good! That's good! Okay! We're approaching noon here. Okay, so system contest, config parent non-input,.

E

A

A

Okay, so now the.

B

A

Okay, multiple things happened here.

A

All right so, let's get commit this first util cli command.

A

And where did that go df? No.

A

A

Replace config, yes,.

A

A

Config, okay, return a new instance: oh wait! This doesn't even have any arguments.

A

Your class config name.

A

Fill modifications.

A

Okay, I don't have time to go traverse all the possible type information.

A

Okay, this is a new instance of that's, not the right type information but whatever, technically speaking, so make config.

A

The modifications data classes, just data classes here inspect us here.

A

Turn a new instance or return a new class.

A

A

Has or new config.

A

Return, a new config class which has.

A

New field class properties.

A

A

Which has new field clas class scope, field properties just knew that which are distinct instances.

A

E

Okay class guys, okay.

C

A

Extinct as this is from there, let's go field properties.

A

Within the config passed in.

C

Into the traits similar to the parent class.

A

Okay, if we don't do it, make new field objects uh and only update uh within a subclass. It ends up updating the properties of.

C

A

Or of the base class as well.

D

A

Let's take a look at that.

C

Replace config.

E

A

That looks better yeah, replace config.

A

A

A

Build modifications.

A

All right, so this is moving replace config, so.

A

All right so we're moving this over here, get commit, dash m, commit uh util or, let's see, base, uh add new, replace config helper uh allows for creating a derived config.

A

A

Okay, we moved create definition.

A

This is moving okay, good comment. Oh wait! Wait! Let's remove this line where.

C

D

A

If types ah move create definition,.

A

All right, I don't know what I had there, but whatever.

A

A

So all right fill.

E

Modification: okay, active system context: okay,.

B

A

A

All right, okay, so.

A

A

Let's jump back in a high level.

A

Okay, so now, let's jump back in the house.

A

Okay, so we want to run this ass version, so.

D

A

I think when we do system context as class, we want to do a double entry.

C

A

Okay, so how would we apply an organizational overlay.

A

So we're going to run.

A

Defy till I okay, so maybe what we do.

A

Is we run that context we just created.

A

A

So system contact.

A

All right, none.

C

A

A

System context config.

A

Sanitize, our eggs.

A

What about extra config.

C

Mags.Extra config.

D

So extra config.

D

I said extra contact.

A

It's an extra configuration.

D

A

The value is ours, command, extra config, sanitized.

A

Okay, run cli command; okay, so this is.

C

So, link for run.

A

Orchestra equals.

A

A

A

A

Oh ctx don't run.

A

A

All right, so here's the system context config it's going to do this, run click command.

A

Okay default extra memory.

A

Okay, so this is the data flow for a cli command.

A

E

Let's see take.

B

A

Give now cli system context.

D

A

Do file cli system context.

A

Give my cli system context.

A

A

So if we run this, then.

A

I'll do this command to run.

C

Feel my command.

A

So, if okay so.

A

So, if we're a command, then a wait do run.

A

Otherwise, oh otherwise, this is our holdover all right. So this is our backwards compatibility backwards, compatibility with old style, cli commands.

A

And then this would be our.

A

So if this thing is a subclass of a system context.

A

So if inspect dot is class.

A

D

A

A

Where was that subflow.

A

All right, async, with self.subflow.

A

Command as octx so give me a new orchestrated context, so when I ask for a subflow, so if the command defined is a system context here, it is okay. Here it is here it is here it is. Finally, it all comes together. Okay, so um if the command defined is a system context, then we can oh man. Okay, then we can.

A

ah Then we can.

A

Execute it like a subflow.

A

A

Hey system context: okay,.

A

A

A

All right so system, content, okay, so df analysis, cli system, context, okay. So if we wanted to override this thing,.

A

Command object.

A

Okay, so by default.

A

Okay, so overlay keyword, argument.

A

Not given I use default, okay.

E

Yeah, I'm not sure.

E

E

I don't know if I can do this.

E

I might have to.

C

What is next week, oh wow,.

A

Holy may where's may, where did may go, may is gone now. It's the 26th, okay.

A

I should cancel this I'm in way too deep.

A

We are way too deep right now. Can we unroll this stack in 50 minutes?

A

Realistically, probably not, but could we could we could we could we could we? Could we no? No? No, no, no, no! No! No! No! No! No! Damn, there's no way this gets done in 50 minutes come on. I feel like we're close.

A

Okay: let's go for 30 more minutes.

A

Let's go till 12 30. woof.

E

Okay: let's go to.

C

E

E

Let's go challenge three all right.

C

D

A

Okay, so what do we got? So? Oh man, okay, so.

A

If alice cli is called okay, so async with something.

C

C

So far as octx sync for.

A

Ctx results in octx.run.

A

All right, so this acts as like a little translation layer over the old world in the new world. It's the new world. It's this fully system, context-based world and the old world. Is this one where we lose context right? We always maintain context.

A

A

Extra config category args and seal that command.

A

Okay, I think that sounds right. Oh of course,.

E

A

Okay, oh ctx, run okay.

F

A

Okay, what are we looking for? Where are we? Where are we started? Scrolling um lost track? Okay, so oct.

A

So here we're running from without self flaw.

A

And here we'll just say.

A

All right not sure what to do is command blank.

B

B

A

A

Okay, so this runs run subflow if subclass command system context. Okay, so.

A

Okay, so this is a system context.

A

So this dfm system contact just small cli system contacts, dfml cli.

A

System contacts you from lcli, okay, just in context, okay, so parent nine architecture is just run. Cli command, orchestrators, none, okay, orchestrator should be.

A

All right, your.

C

Serious memory.

C

A

I'm just going to cancel this there's no way. I can figure this out.

E

E

Oh man, this hurts.

A

Okay, so all right postpone the meeting.

B

A

System context.

A

A

Okay, so we preserve the context. Okay, now that we've preserved the context.

A

I'll get straight or run okay, so.

A

How do we preserve this overlay situation? How do we preserve the overlay situation.

A

How do we preserve the overlays.

D

A

Let's run the clock command, so this runs here like command.

C

We don't need to do an overlay that okay, so system.

A

Contacts: okay, so you can do we can do an overlay at the top level, this parent equals none. If system context do from lcli we can do. We can overlay this right, in which case you could do something other than this, and then, if you get in here and you decide to run a you know, a sub command right.

A

Which is reminding me, we went through something.

A

The other day.

A

Said, and here is subclass method command.

A

Okay, so we need a solution for this okay, so.

E

E

Command equals message.

C

A

C

So this is just for a shorthand.

A

Definition, so we can say: okay, so if inspect n or is sub class system context,.

C

A

Okay, so, basically, if we're sure it's a class check, if it's either a old thing or a new thing right, so either command or system context.

A

If it is a system contacts, yes.

A

A

Organizational overlays.

A

Okay, so on startup of the cli we're going to load all of the default organizational overlays so on startup. So here.

A

A

We do not explicitly set overlays.

A

Oh here, okay, here it is.

A

We need to treat it like we treat log. Oh that's right! Okay, that's right! Okay! So we need to treat this like. We treat log okay, command, config.

A

A

Required false default equals none.

B

A

So the overlays to apply before running the overlay data flow to run before running.

A

A

For the top level system context.

A

Okay, there we go. Finally, okay.

A

Okay, so the overlay for the top of system context, that's right, so this dash overlay all right. So let's go ahead and remove um overlay from here.

A

Okay, and what are we gonna do with it? Well, we're going to say overlay equals.

C

A

Okay, okay, so now we have our system context that we want to run.

A

All right, so here's our system context. We want to run.

A

Here's the things we pass into it.

A

Here's the overlay.

A

Okay, so now within.

A

So within this, so here we have the overlay, okay, args overlay, all right overlay.

A

uh Let's close system context.

A

So is the overlay of dataflow, or is this system context? Well, the overlay is a data flow.

A

Because it doesn't necessarily have a place, you have to run it. No is it a system context, it doesn't have system context would be what we create ah a system context would be what we create to run the overlay, because the system context is what gets executed.

A

A

ah Not necessarily, I don't know.

A

A

C

A

All right, so here's the overlay command.

A

D

A

A

I'll be laying, if it's not already added.

A

A

A

A

Okay, so we needed to fix this up to make it the same. So basically we're going to do a check. So you know first, first off the bat when we get into this run, cli command make sure we have a class if we do have a class check if it's a subclass of command, if it's a subtle house of command, do that do the legacy stuff, if it's a system context, do the new stuff right.

A

So uh if we're doing a system context, then we're going to create a subflow within which we'll run the system context, um and so we should be able to use our uh you know we should be able to um so it's just. If it's a system context, um then.

A

A

A

Mark's overlay, cli.

A

Alice version system, contacts, alice, cli version.

A

Okay, apparent is none.

A

Oh ouch, okay, so apparent is none.

A

Sure data file.

A

A

A

All right I'll just try to remember extra okay.

A

So yeah, so this one, for instance, uh yeah, so we by default say executed in the memory orchestrator right. So if you use the memory or extruder, I know you're going to be able to call import live metadata with this operation implementation. I'm providing you right. um If you use something else, this flow is not guaranteed to work right. Okay, you better overlay it with whatever the hell. You want to make sure that that implementation can be instantiated if you're going to pick another orchestrator right, but this one will work. So that's the point.

A

This is a valid system context, um so how we map that.

D

A

Sure we'll have to get into just dependency information, of course, so.

A

So in this case we would say: ask your version.

A

A

Or we could say op.

A

A

Multi-Output is files because we're doing auto-def or multi-output is false, because we're doing auto definite we'll set that for us, but.

D

What we care about is.

A

What do we care about here.

C

Oh output stage equals stage.output.

A

Right so this will say treat this treat this gate alice version as an output operation right. So this is just it's just a function, but when we instantiate it within a data flow within a system context, we say treat it as an output operation right, so our output operations, all of our output operations, are going to be callable as methods so.

E

A

Okay, so how does this thing know what to execute now? So, basically, uh what does it say default?

A

It doesn't need the deployment environment.

A

Does it need the deployment environment.

E

A

This is the calling convention, but we have another whole thing about the output.

A

What is the output is sort of the data flows, class methods, there's like a there's, almost like another calling convention within it to retrieve the outputs, so there's like the primary call and then there's a secondary call to get the outputs rather than a return value, because there's no return value for c, because all this is asynchronous right. So, okay, okay, maybe that's helpful. So, let's think like alice right, so we're thinking in parallel.

A

Okay, we start the flow uh we would like to know yeah, we start the flow, so we come in. We run alice version um and by default.

A

By default, okay actually wait a minute.

A

By default, we're going to say, load default.

A

Load installed.

A

Load installed overlay.

A

Default equals apply installed, overlays.

C

All right, so this would be.

A

It says dfml overlayed install is.

A

A

D

A

Installed default.

A

A

Default installed.

A

Apply installed, overlays.

A

Okay, apply installed, overlays all right.

A

A

We're thinking like alice think, like alice, we're coming to the cli we say default is apply installed, overlays, okay, so that's going to get.

A

Thrown into here with load default load default is.

A

A

Oops, it's this that's apply installed, overlays, it's not this different overlay installed is this. Is this truly is okay? The data flow, which has which grabs the overlays that are installed.

D

A

Apply and solder release, so this is gonna okay, so this will say: apply installed. Overlays.

A

So that when it comes in here and does.

A

The execution of the system context, it will say, oh really,.

E

A

To do add rest of sanitize.

A

Data into as inputs in case flow wants them.

A

Okay, so we run.

A

If, okay, here's the run overlay overlay, okay, util data, this one doesn't import cli. I hope none of this imports the util cli, because that would be an unfortunate circular dependency. Okay, so.

A

So this may be able to import run, which would be fantastic.

C

Not the top piece.

A

This is a little dangerous here.

A

A little run, okay, so.

C

A

Random system context.

A

A

A

Okay, so what I think happens here is.

A

No okay: we need to remove this comment.

A

A

C

A

System context.

A

We're so close, ah what is it it's so hard to know what you don't know?

A

I think it's the deployment. Quite honestly, I think this is what we're missing. I think what we need is. We need a input.

A

I think maybe we need like a.

A

Oh, maybe we should have what the output is. Maybe we add what the output is as an input.

A

Yeah, maybe we have something that's like if you want this output run this flow okay. So if you want this output run this flow. So if you want.

C

A

So if I want some output data.

A

I need to run any operations that produce that, where you want version as an output results to be us version, then we need to run.

A

All operations.

A

The operation which produces this version as an output, operation, okay,.

A

A

So in that case, what does it look like over here.

A

C

A

Outfits, okay, so if you want.

A

This output, if you want this output, run this thing, so we want ah so we want semantic version. Okay, we.

A

All right, this is just to help me as vim will highlight it.

A

All right, hopefully, that makes me see it.

A

A

A

So we want to grab, we want to grab semantic version: ding ding, ding, ding, ding, okay, so this version system context.

A

A

All right, so we have to remember to set that on run and after overlay. Okay, so we want to grab semantic version. Look for types.

A

Whose linear edge is derived from that, if there is no output or operation which outputs a derived or direct type raise invalid, um so we are going to overlay okay, so we.

A

Will overlay a system context or we will overlay.

A

So ask it version.

A

Version so maybe if we say cli see a result, so maybe these things are looking for cli result. So.

A

A

Okay, okay, so maybe okay, maybe we have uh maybe we.

A

Know that all cli commands.

A

Must accept an input and okay, so uh four, a system context to be used as a cli command. We will overlay with an output operation operation which we will overlay with an output operation which.

A

Returns a single result.

A

Within uh util, dot command so command.

A

Okay, I have to go. um Of course this uh we check if that.

A

This flow should produce a result uh for the cli of the of the of the cli result, data type. um We can say that any uh uh descend we can return any. We can so this flow. This flow should have an operation in it which uh this flow should have an operation in it which um returns the okay, which.

A

When cli result.

C

A

Reduces cli result and takes any.

A

Via taking any valid any data at a single piece of data derived from semantic.

A

Version, okay, so that should enable us to do that.

A