DFFML Rolling Alice: Engineering Logs: Volume 0: Architecting Alice, 25 May 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Architecting Alice: Volume 0: Context: Part 46: Thinking About Operations as CLI Commands Discovery

Description

https://github.com/intel/dffml/discussions/1369#discussioncomment-2813540

A

All right, this is oh we're on port part 46. Now.

B

B

Where's they stupid all right great.

A

A

So um what happened here? Oh yeah, I think I realized something when I looked at this um this required. So this was not here. So this is what we had before. I looked at this and I thought it didn't look quite right um and I realized okay. So it's requiring it's, it's complaining right. So what's the error message, the error message is okay. Well now it's different.

A

All right, so, what's the error message? Well, it's returning sysexit 2, because it's displaying the help, because it's requiring this data flow path to the diagram command.

A

A

The metadata is still set to required so.

A

We can just say metadata required false, and then it should stop bugging us or not.

A

Still bugging us.

C

B

A

Metadata required false.

D

Print the config, so yes.

A

Okay, so we print the config field. Data flow type is correct: default factory, okay,.

A

Okay, maybe we need to set default factory.

A

Maybe it's telling us, you know basically to go screw ourselves, because this is we're not seeing it because we shouldn't be setting.

A

So, let's take a look at required. False okay,.

A

Required false, so let's omit required, see what happens. My guess is: there's a problem with the merge.

A

Okay required false okay.

B

A

All right, I don't know how many times we've pretty printed this thing at this point, but.

C

A

Data flow config letter, simple display, looks good field name equals none. That's interesting name equals none.

A

Let's set the name.

A

Do the other ones have names.

A

D

New class config and class config.

D

E

Field takes you.

A

Seven keyword arguments were given.

A

uh Field got unexpected keyword, argument, name.

A

Name: yeah, okay,.

D

So this is what's getting created.

A

Data flow default missing type. So why is that um field modifications get field name.

A

Maybe this is not the same.

A

Right so maybe we're not getting the overrides or overlay whatever.

A

Default factory.

D

Data flow default.

D

E

Do you fall fire.

D

A

C

A

Compared to default factory missing type okay, so the merge is not doing what we want it to, because the lambda is not getting overlaid. So, what's merge doing.

D

A

Right, if key in one.

A

Why is it not getting set for key value in two items? Okay,.

A

C

A

Field name: data flow data flow; okay, that's the same default factory; okay, so we merged this thing. With this thing, so default factory should have ended up.

A

In one, why is it not in one if key in one, if instance,.

D

C

A

Okay, do we not okay.

A

I think we were missing a case there.

A

Oh there it is okay, so we were missing a case.

A

A

Okay, all right, so we found that we were missing a case from merge util.

A

Okay, missing, uh add missing else to update non-dict so basically uh or and non-list values. Okay, so basically what I mean? What happened here right so it looks like we were trying to. If okay, we merge. If the key already exists, we detect if it's a dictionary and we recursively merge the dictionary. If it's a list, then we append to the list. uh Otherwise we forgot to actually just update the key um so oops, uh but now we do that so because I think it was.

A

It looked like this falls through, but it does not um so.

C

A

B

Figure out where we were at here.

E

Where's ruffles.

E

He's right there.

C

A

And what do we do here? ah That's right! Oh yeah! We did that um testing stuff, oh great. We finally did that this will help with that.

A

D

I won't find it testing.

A

Move chord dog strings yeah. I think it is.

A

D

A

B

Okay, so this is related to this issue.

A

All right so tess dog strings, so we omit the prefix when there's always the same so test doc, strings refactor,.

A

um All right, so we refactored and we changed, which is not allowed. So what was the change this? This was the change, so let's commit just the refactor first right and you know you can see it's not actually running.

A

And then we'll put it back all right so now this is right. So now we're seeing it down here. This is not staged. This is what staged right. So just the refactor test, doc strings.

A

Okay, so test dot, dot, strings, um refactor.

A

uh Population of all objects into recursive, routine.

A

Seems descriptive? Okay,.

A

And then we can add this right. So now we can div stage now after we get add, so we do get add test test stock streams. Then we do get diff dash staged and we see that. Okay, here's the is method right, and so now we can make a commit where we can say test doc, strings.

A

Support for testing class methods.

A

All right, fantastic, okay! What else do we got so looks like we're? Getting a graph over. There looks completely useless. Okay, fantastic.

A

Maybe we'll actually have it go simple too.

C

A

Okay, how do we do.

A

C

A

What is it that we want.

A

Okay, maybe let's see dataflow and then simple.

D

Default is true.

A

A

Was our little trick.

A

uh Single quote: left arrow, comma single quote: right arrow, bang, python, mjson.tool.

A

Nope, oh python3, on the system.

D

Didn't like that.

A

Okay, I didn't like the pythonized values. I think we can actually pipe through black too. Let's try that.

A

A

Hey that was great, very nice. That's pretty convenient.

D

So I tried to say simple.

D

A

I thought there was a simple argument or oh, it's not simple, that we want uh it's stage.

B

A

A

Staged out processing.

A

Cut graph td yeah, it's like uh okay, there's nothing here,.

B

Perfect, okay,.

A

Dot config, why is it printing.

A

Oh because this.

B

D

Event, loop is closed, was.

E

B

A

Async test case that tear down never awaited.

A

One time oh what's happening here.

A

I think we hit a bug.

A

Diagram for my dude.

E

A

Test doc's turn okay, so it gets mad because it got nothing.

E

Expected nothing god, gravity. Okay,.

A

Is this one of those words running the same test case twice? This happens sometimes.

A

Self.Loop, okay,.

A

A

All right, so I think this is happening because of the way the loop, the loop variable gets past loop equals none and maybe we're creating a value somewhere. I'm not sure yeah, okay get event loop. We probably don't want to do git event loop.

A

We probably want async run it for us, so this is the thing that does with the event loop, so it just passes.

A

Okay. So if we just call underscore main.

A

Okay, if we just call underscore main we're good.

A

Great yippee, we generated a data flow, okay, a diagram, and we have an example.

A

Okay, till cli command.

B

Subclass with field overlay.

A

A

All right push push push.

C

A

So now, where were we okay so.

A

Why were we even doing this.

B

All right, okay,.

A

Oh, maybe we should run this command.

A

Or do all service dev source.

F

A

All right, great all right so now what happens over here.

A

A

I hate shells that delete history after 2000 lines.

A

I hate that, why is it the default config? I thought I overrode that.

A

A

Sick of this we're getting rid of this.

E

His depend uh history.

D

That's where you saw that history say because.

A

What is the maximum value? I think this overflows on some things, because I've set it to hot, too high of a value. I just want it to not go away.

A

A

This is so frustrating okay,.

E

Are you kidding me.

B

See that's probably too big.

A

Is it not being exported? Are you.

E

Makes me very frustrated: okay,.

A

A

There we go, this is the command we wanted.

C

D

Configure yeah.

A

What the hell we'll just put it in this directory.

A

All right, so what are we doing so we're saying: dump that remap to response data flow uh t it to this uh write the output via t to uh dfl main package overlay.yaml and then pipe that through dataflow diagram and and uh take the input from standard in there file not found hey, hey how's it going.

A

Can you hear me.

A

Hey sahil how's it going. Can you hear me.

C

I can't hear you.

C

A

I can't hear you.

G

A

Can you hear me.

C

A

Hello yeah, I'm not getting anything, says you're talking.

G

C

A

Discord no audio.

D

Oh, it's still gone.

A

Can I hear you now, let's see, oh, you can hear me: okay, yeah. I kissed okay,.

A

Okay, I don't know.

F

G

D

Hello: let's see what is this.

A

um Mute, no, I'm not muted. Oh wait! Yeah! I made it on the other computer, but I'm not made it here.

A

I still can't hear you, I can see your green, but it's not showing up in the volume mixer even.

D

uh Muted, you specifically, how did I do that user volume.

A

Mute, no, I can try meeting and meeting no user volume 200 no.

F

C

E

A

Permissions, not private.

D

Video speak okay. What is this speak allows members to talk in the channel if this permission is disabled. Okay, so careful.

A

Ufo, so what is this general.

D

Channel permissions disabling this and the connect.

A

D

Allows members to talk in the channel if this permission is disabled. Members are default muted until someone with the mute.

A

Member's permission on mute zone, okay,.

B

E

C

A

Did that work didn't you we were on here before, though,.

A

C

uh How do I.

A

Maintain your roles: maybe this will help.

D

uh Rolls maybe rolls maintainer.

F

A

Okay, I'll just keep going then.

A

um Okay, so we're just doing.

A

I'll try to check the chat periodic.

A

Okay: okay, oh yeah great, you can hear me: okay, okay, so yeah so I'll. Just uh you know I'll just keep doing what I was doing all right. So uh basically, what we're doing here is uh yeah we're exporting this overlay, um we're dumping it through and we had a problem with the. So basically, I introduced a bug with the diagram command.

A

Well, I don't know if I would argue that I introduced this quite honestly, yeah there's, let's see what this looks like: okay, good, single, okay, this looks good data flow to be overlaid. Okay, so we added some more auto. So basically, we did some auto definition where we can we're beginning to unify the typing stuff with python's type system. uh So basically you can take. uh You know new type from typing.newtype and you can um you can write uh these turn in definitions.

A

um Okay, so see get single, spec spec get single alpine orange okay, so this is going to go through and okay. So this basically yeah this grabs the data flow, and so this is data flow to be overlaid.

A

So this converts it into the thing. Okay. So then, any data flow here from src uh from seed will get merged in and then this merged one is the one that this we need a way to link this across across the output stage, um because to say that, like hey, this thing will go get this um well, maybe wait a minute!

A

A

Okay, I just realized this is something really ridiculous. uh These output operations didn't need to be like this at all. uh Okay, this is all right. Okay, that's really funny! Okay, all right! This didn't even have to be like this.

A

Oh wow, okay, um so this can actually just take uh directly from uh data flow, has um wow wow. That makes things a lot easier: wow, duh! Okay, um let's see what this looks like then,.

A

A

So this in hindsight seems extremely obvious um for the past three years not have not been doing this. Okay, wow, okay, wow.

A

Well, all right, then, okay, uh so basically, you know we had these output operations, but there was really no. There was really well. This simplifies a lot. This makes a lot of things a lot easier.

A

So we were talking about how we're going to make these data so basically we're going to support a data flow as a class right and then we're going to call different output operations based on uh you know what we want out of so basically, we set send different inputs to the data flow, to run it or like when the class is instantiated with the double context: entry.

A

The data flow starts running right every time we call a method, we add specific inputs to it under a given context, um and then we overlay uh specific output we base. Then we call specific output flow uh which will then grab. You know the resultant data right and so wow. I didn't realize how easy this was. um Okay, well, that's fantastic! um So once the processing stage is complete, then we move to the output stage.

A

So this will all of this will all of these have to run before we can go over here at which point this src.

A

So this dest is an object in memory and if we were to look at the definition for it in this df main package overlay yaml, we would see that yeah, the primitive is object right, and so what do we know about things that the primitive is object, which basically means that we're not sure if we can serialize them across the network? um That's really all! It means right now, so we'll we'll get more details on that as we flush it out so wow.

A

I feel I feel very dumb for not realizing this for years. I don't know how many of these I've written- um okay, wow, wow, okay, all right, then.

A

um Wow okay: mind blown.

A

All right so we're going to, let's make sure where's our nodemon commander, run.

A

Okay, so what is this service dev export? And what are we even doing? Oh we're running alice version. That's right! Okay, boom! All right! So now what's happening here. So uh when we run the alice, oh, I think I have to fix my um let's see yeah. I think I have to fix the overlay boop.

E

All right, let's see, transform that transform right bottom.

D

Let's do 300 add 300.

A

There, let's see.

A

400 there and add 400 there all right there we go so.

A

Oh okay or not, um okay, so.

E

B

Let's duplicate this.

A

Okay, all right! Well, that's interesting, okay, so we can basically just say okay, so what are we doing here? So basically, this is the default overlay um and when we run any data flow and for our example, data flow is going to be vim entities, alice, alice, cli.

A

So our example, data flow is just going to be running, alice's version command.

A

You know what version of alice are we on um and we talked about how you know eventually we're going to unify the the command, the command line, stuff and the operations, and it's going to be the same thing and under the hood, it's going to run a data flow and we talked about the top-level system context and basically, the top-level system context is this first data flow that we execute and then all within you know we have this calling convention, which is you know the system, context um which is the?

A

uh What do you want me to do the data flow or the open architecture which will then maybe have a data flow in it um and then the inputs and then the orchestration? So how do you want to get done? Where do you want it done so uh yeah? Where do you want it done? Is the orchestration?

A

uh How do you want it done? Is the data flow, and what do you want me to do it with? Is the inputs? Okay? So uh so we merged now we're running, run, dispatch.

A

Outputs here operations, so df memory run dispatch, operation, outputs key, so outputs operations.

A

Merge outputs operations; okay, so this is problem with do file.

A

Oh, this is multi-output equals false. That's what we need there! Okay,.

A

All right so we're doing a single output command.

A

And we can't type hint these lambdas, but we could. Basically, this is equivalent to.

A

Yeah, let's see, let's do.

B

Let me dip my df types.

A

Okay, so this is going to do the same thing right now, because uh the type the typing is very simplistic, so.

A

And we'll just call it data flow type so to do uh fix or to do uh resolve or unify this.

A

Data flow type used as an example to show auto typing dot, new type conversion to definition.

A

Which is definition.

A

Was are, or is our pre-input lineage based mechanism.

A

For identification.

A

Of a data type.

A

Okay, so in this case we could do this.

A

And we could say it: outputs, a oh: it doesn't output anything. Would you look at that? Oh yeah, it's just a raw update, so.

A

All right, so this is the same thing so.

A

Same thing, yeah.

B

B

A

Function has no object, replace merge, o.

A

Can we wrap it there? I thought it auto, wrapped it? Oh, no, it doesn't auto wrap it if you pass it. So if you pass it here, merge it'll, auto, wrap it, but it doesn't auto wrap it. If you pass it be the other one: okay, um because this is an op-amp and um or let's see new type, to definition, not to mine.

A

Oh, what's that where's that coming from vim df multi of base, we had a patch which did this the other day, but we must not have um carried along with us.

A

Where's that handy-dandy command here.

A

A

So we read in the output of get grab, it's in dfml, df types.

A

C

A

Merge in both blank and blank okay, so.

A

Data flow merge: let's try this all right, so the auto export of everything is complaining. Okay, um where are we at so data flow from dict? Okay, so it went through and it merged it. Okay, great!

A

Oh! So it didn't like the lambda wait, a minute. Why didn't you like the lambda? hmm So oh a cat.

A

All right so um well didn't like the lambo interesting. So um we have a data flow now looks like it. Output a real data flow, that's fantastic right, so this is the one with the overlays applied.

A

Are we still on default? We still are on default, so we're still so remember. Now. The overlay process is is twofold. So, first, uh if we're not provided with an overlay on uh run, then we're going to go in and load the default overlays as they are registered via the entry point system.

A

All right so here the internal points, so default.overlay dfmount equals dfml.overlay.overlay and then main package overlay right, and this is the main package overlay. What is that? Well, it's a dataflow which will execute and when we execute the dataflow.

A

So this is what okay so and it's getting. This is getting confusing and watch why so, because when we load the default, we execute the flow using.

D

Data flow to be overlaid: okay,.

A

So we execute the flow using class dot load definition. Data flow to be overlaid; okay, so, basically, this is data flow after public.

D

Supply deposit.

A

Merge merge is dataflow type, so we want to say basically add all these things. Okay,.

A

All right so now we can say.

A

So now we can say add you know the data flow. So basically we have a blank data flow which is to be overlaid on the default flow, so we're instantiating all of the default installed overlays. But what are we overlaying them? We're going to combine them all together into one data flow right and then we're going to execute them on the.

A

On the uh you know, when we so.

A

A

Where's the mermaid diagram, so this is blank when we're loading the default flow, and then these are all coming from uh the seed data flow.

A

All of these come from the the entry points right, so we're going to load over all the entry points right and so in this case we're providing this data flow type which is going to be passed to merge so which is going to be here, see data flow, so so we're going to provide everything from class.load, which means anything registered under dffmel.overlay, will be merged into this empty dataflow and then we'll be returned. This dataflow object.

B

A

Here we are so.

A

Merge processing merge, okay,.

A

A

Okay, so I think it picked up the so it looks like destination blank okay. So this was the blank here's the blank one.

A

So we merged it and the thing is we're merging it with itself. Remember because this is the only one that's installed, so we um so we manually grab it here right or we grab yeah. We hard code grab it.

A

We use it because this thing will take any data flows that are applied, merge them together and return the merge one right. So then we add all the data flows to to merge together um being the same data flow because it got loaded right and so now we're like okay. Well, what is the resulting? This should be this overlaid right, so outputs data flows to data flow from dict outputs data flow. I think this might be because we change okay, so data flow from direct outputs data flow, so adding a blank data flow.

A

Sure there are no overlays okay, so wait. We don't need to do this. Okay, so because we're adding a blanket over the data flow and seed. Okay,.

A

Input set value definition, data flow.

B

E

Very much response.

F

E

Data flow type.

A

So what seemed like? It was working.

A

When we were here, we seemed like we were working okay. This is a pretty good place to be. What happens if we remove this? Oh dataflow, 2b overlay. That's why.

E

D

I need a blank sure I need a blank flow.

A

Adding a blank flow if there are no other overlays, we end up just merging the input data flow, the blank overlay results and return of the original data flow.

B

Okay, so data flow to be overlaid.

A

Okay, that's what happened so this data flow to be overlaid is what we really want here on data flow, merge.

D

Okay, so data flow to be overlaid okay. So what about this data flow.

A

Type, so this is actually data flow to be overlaid or, let's see now this is the data flow okay. This is okay. We need to. We need a standardized terminology here. Okay, so, let's reset so data flow um used as overlay.

A

Okay, data flow. Being there we go. That's descriptive. Data flow being overlaid; okay, so this is the.

A

Data flow being overlaid as.

F

B

So where's this dataflow sticked.

A

Data flow being overlaid as ticked okay,.

A

Now this data flow stick just not used okay, so data flow being overlaid as um dataflow 2b overlay, okay. So that's a bad name.

A

Okay, so dataflow what time? Is it? Okay? Okay? You can meeting here. Okay, therefore, to be overlaid so.

B

This is really.

D

Data flow to be overlaid.

D

B

Data flow to be overlaid.

D

D

So inputs data flow to be overlaid.

A

Being overlaid.

C

Data flow being overlaid as ticked.

A

Data flow to apply as overlaid.

A

C

A

Okay, so uh we say run the main overlay. Okay, so we say dataflow to apply as overlay and we pass in all the ones which we want to apply as overlays. Okay, all the ones that are applied as overlays.

A

Okay, so all the ones to apply as overlays.

D

Data flow to applies overlay.

A

Data flow to applies overlay.

A

Okay and then we add a blank one just because if we don't have a blank one in there and there's none in the list, we're going to end up with nothing in the output. Okay,.

D

So do floater live early.

A

C

A

So then we need an input. Data flow, okay and that's where this, what this was doing already so.

B

The data flow to be overlaid.

B

The data flow to be overlaid.

A

Data flow being overlaid.

A

All right- and this is a blank data flow in this case- and this is something so this this needs to exist.

A

Okay, so this will be related to that. Allow list so.

A

B

A

A

uh Okay, so should be in okay, sure, okay, so- and this is where it's like. Okay, I think maybe maybe relates to data flow as class methods as if they were allowed list or arguments.

A

So maybe the date this, what we're seeing here is really this invocation of the data flows really invocation of a method which I believe it is. What we're really seeing here when we do this orchestrator.run.

A

Is that we're calling a dataflow as class method and we're passing these parameters, and so our input validation can be done on the method call and the method call can be done via in operation. So, okay, so.

A

Enters for this input objects, for this call are equivalent to the arguments of a data flow as class method.

A

This a method is simply.

A

An operation within the data flow.

A

A

As a place to add other inputs within context to the data flow, perhaps within their own context, even but by default now.

A

Okay, so there's a place to edit right now default same context uh as the operation for the method was executed in okay. So when we call a method, we're going to create a new context which will be it'll, have a unique identifier and it'll have yeah it'll have a unique identifier right and it will be related to basically um ok. So.

A

Method call create new context.

A

As input object, context maintains link to parent.

A

Context, okay, so this is how we're going to create you know this massive database of everything um of all of everything. So, okay.

E

If well, it's not defined.

A

Okay, all right there we go so I'll put data flow from dick looks like that was called twice. So why was that called twice? Okay, so data flow being overlaid.

A

And this output operation takes.

D

Different overlaid as dict.

D

Okay, so that's wrong.

D

So this should take.

A

Or no stick to apply as overlay. So that's wrong.

D

So this should just take data flow.

A

Being overlaid.

A

Data flow being overlaid.

A

Turns into data flow being overlaid as.

A

Outputs, this is just output operation, so just.

B

A

Wow, this is all so much easier.

A

I don't even need any of this remap stuff.

F

C

Wow, oh my god,.

D

Years, okay, well, all right, wow.

A

A

Oh, my god, it's so.

D

Much easier, okay! Well, this is fantastic news: okay,.

A

Dataflow has a no attribute apply.

A

A

This thing is a um okay, yes, so it loads the data flow and then it's not gonna yeah, so results, overlaid uh and we'll just say: overlay.

B

E

Overlay default.

B

A

Apply so yeah it overlays data flow really.

A

A

Let's maybe say.

D

Overlay default.

A

Okay, so it returns an instance of the class.

E

A

Okay, we'll just make a config.

A

And we'll say requires data flow.

B

The data flow to be.

A

Used for overlaying.

A

So there's basically, this translates into like an overlay underscore apply operation.

A

A

I wonder how the class method will work.

B

Okay, so we can call this a base. Data flow, facilitator, object.

A

Should it follow the double context, entry pattern, probably okay, overlay.

A

Okay, so maybe.

A

Okay, so maybe this thing is like.

D

Let's go look at the.

D

Okay, if overlay is low default, then we say overlay equal.

D

Hopefully that default.

D

A

Yeah, so this isn't a weight here.

D

A

C

A

A

I think we're close to something here.

A

I think we're close to something important.

A

We are close to something important.

A

I can feel it.

A

We're close to the ucan meeting in two minutes: okay,.

A

I think we're close to unifying the cli.

A

Okay, let's look for where subflow is.

B

A

Subflow could snow.

E

Contacts: okay.

D

C

E

Self-Confessed.

A

B

To be used for.

A

Data to be used for applying overlays.

A

Okay, it's 10.

A

I think this is turning into an operation.

A

hmm I think there's something happening here.

A

A

I think that might be it alright. I have to run.