DFFML Rolling Alice: Engineering Logs: Volume 0: Architecting Alice, 31 May 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Architecting Alice: Volume 0: System Context: Part 58:

Description

https://github.com/intel/dffml/discussions/1369#discussioncomment-2851131

A

Let's hit it okay, so we're going to get rid of primitive because it looks like basically, nothing is using primitive. We don't need to even care about this thing like let's stop wasting any time on it. um So.

A

Script- stocks yeah that um so scripts these are unused. They are well okay, so script stocks is failing. Okay, so that's going to be a problem. Okay, script stocks is going to be a problem, but I think we can just say it for now, because this is not critical.

A

What's critical is that we get this done, um everything else can be fixed at a different time. So does not equal string.

B

B

B

A

Inspect is class prime annotation name.

A

Links clean parentheses,.

A

A

A

We've got this.

A

Plug-In space, so it's an entry point.

A

Prime annotation entry point so basically.

A

A

A

Okay, I have to go actually links.

A

Why is definition.

A

Definition, oh.

A

A

A

Lengths primitive default lock: these are properties which can be populated on load, so primitive, primitive and lock are properties which can be prop populated on load based off of the links but reversal of the legs.

A

So you know because this really is.

A

So if we wanted to deserialize an opera or deserialize the definition within an input network, we wanted to do serialized definition.

A

We processed the links to find the primitive and the lock sounds good. Okay,.

A

Put this at the end to do remove validate.

A

Global code should be using operations instead, yeah overlays, okay,.

A

A

All right, so this is our links to their input, size, definitions.

A

A

Let's try to export it.

B

A

Spec equals parameter annotation. So let's stop setting spec.

A

Okay, it exported successfully so.

A

A

Okay, so name entry, point, dfml overlay, overlay, overlays, installed inputs plug and base alright. So we have our lineage links. Is entry point the links.

B

A

Actually, you know we need to modify export to print to do all the links, so I think we did it here.

A

B

A

Right so you know in the event event, the paramedician or the annotation is a class we create is a python class. We create definitions for both the.

A

Its usage within its usage within the context: uh okay, what name as well as the diff as a definition for the class or for the derive or for the uh parent.

A

Which is prayer meditation, as well as for the parents, module.

A

This will allow us to dynamically apply, serialization, slash, deserialization.

A

uh Sandbox scene.

A

To restrict uh uh aka allow list of modules and types which can be instantiated based off inputs and what operations they must go through before instantiation.

A

Okay avoid uh aka overlays, so I should apply vetting to input data to avoid uh yml dot, load situations, cve situations.

A

uh And I'm going to go ahead and throw in.

A

B

A

A

A

All right, let's see, documented in bold type yeah, so, even even if you document something in bold type, basically uh still a cve may be present. So um this is why um yeah. So this is why we're doing this uh at such level of detail right now, because we don't want cvs. um So, let's see I mean if we have them fine, we'll fix them right, let's just shoot for not having them, so we want to learn from other people's experience, and you know these.

A

These fine folks, over at the yaml library, may have documented things in bold text, yet still ended up with cbe. So we're going to avoid avoid the situation entirely, hopefully so, um uh by making it the responsibility of the downstream, uh which can be mitigated uh easily by plugins right, because we can apply, we can install overlays, laces plugins, so.

A

So basically we just want to mitigate we won't. We won't provide the functionality within the main package. We also won't mitigate within the main package right. So it's all um you know it's all a plug-in.

A

So if you want to yeah, let's see so is it okay? Well, maybe maybe we are about to screw something up. Let's see we'll see we'll see we'll see if we end up with the cv so.

A

Let's hope not fingers crossed now. um Okay could not determine their flow to applies overlay.

A

So it came through and it says it was not a class basically so now.

A

A

They come through and we say.

A

Okay, so could not perform a primitive of just all. The release result can never.

A

I gotta walk the dog. We've only been going for 18 minutes.

A

It's almost 7 a.m.

A

Okay. um I want this to be done. Okay,.

A

I want this to be done: okay, primitive of output. The results could not be determined type unless data flow to applies overlay, tick, tick, tick,.

A

A

Okay list: okay, good, not determined, primitive; okay, so we fell through to here.

A

A

Okay well never mind, then um sometimes, if you import redline, it will give you up arrow support. Sometimes not. I can't seem to tell why or when. Maybe I'm remembering that wrong, maybe it never did. um Okay.

A

Here's param annotation. What happens if we call.

A

So how did we get in here, though,.

A

So what was this thing typing list.

B

B

A

List list: okay! So what were you so mad about up here? Then.

A

Okay, here it is so it's not satisfied in here, so inner classes, primitive types so.

A

So it's not satisfied in here.

A

So what's happening so in our classroom.

A

P print port, p print.

B

A

Inner class new type.

A

How the hell did we get in your class? Good args, annotation, zero? Okay, new type?

A

Okay. So if we have a list okay, so if we have a list.

B

A

So in this case, we should actually call this recursively.

B

B

A

So this should be a recursive call.

A

Definition for internal type uh modify to declare its a list or map and then so subspec true spec inner class.

A

A

Okay, definition, okay, so basically we set spec will be already set because it'll be set to the right thing. Default will be set to the right thing. Primitive should be set to the right thing. Basically, if we just say subspec equals true.

A

It should do what we want.

A

Actual type of list.

A

Hey we hit a new bug.

A

This is fantastic news.

A

Let's go confirm that this thing is what it's supposed to be.

A

A

A

Link's none named inflative, that's a really primitive object.

A

I don't think that's what we wanted. Inner class.

B

B

A

A

Links beautiful object, hey I'm liking this so far, so without an object get in front.

A

A

Okay, so length goes to that.

B

A

Okay, so if we export, let's go to export and make sure that x4 expressed the links.

A

Okay, sorry flanks in kwrk.

B

A

Thanks, okay, so so here we're saying so we're writing the serialization and deserialization code for links so.

A

So we'll just call from dict, because it's the same thing so link data for link and.

A

A

A

A

Okay, so spec, so we're hung up on spec again.

A

Oh, so we screwed up that instantly.

A

In there, okay, hey, we got a different error, fantastic okay. So what is that link? So we want links to be words, export export, if not legs. Okay else,.

A

Equal export, okay, so export dict is in this context. Okay,.

A

um Export for link in links all right.

A

We export the links boom. Okay. Here we go: aha, fantastic, okay. So now we have dfml module, dfm df types as a module, and we know that this thing module primitive object, okay, yeah! So it's a module. The primitive is object. A primitive object means you best be pickling this thing. If you want to move it around, I'm going to move outside today, yeah it's been weeks since we've been outside um so very exciting, so uh name dataflow. This is yeah riveting. um Okay, so named.

A

Okay, now this is a little bit redundant here, but I think it's going to be okay. For now uh we can sort of uh check the redundancy later we'll deal with it later. um So the point is we have a serialization format here um and it's compatible with the ids and purity ids. So in the way that it works now um wow all right, we did it. Okay, no, we didn't do it.

A

Okay, we did not do it yet um uh we fixed one bug sort of we don't even know if it's fixed yet okay, so we did something. Something happened. Something happened, exciting, okay, so um let's go ahead and get rid of this.

B

So it's like, it goes true.

A

Actually wait a minute what was happening here: print d dot export. So why is subspec not equal? True.

A

Why does subspec not even not equal, true.

A

Export, I'm not seeing subspec in here.

A

It's set to true: why is it not in the export.

A

Oh because spec.

B

A

A

Okay, oh yeah, okay, so here we are here comes the problems: oh man. Okay, I'm supposed to rock walk ruffles.

B

A

Look at my calendar, okay,.

A

Okay, four o'clock.

A

That means we have how many hours.

A

I did not count that correctly. I counted to 15.

A

A

A

Nine nine hours nine hours now I know I have to clean this up.

A

A

It can't be done. Okay, I have at 7 10.

A

Let me block off my calendar.

A

This is a hell of a week.

A

It's always out of a week.

A

A

Where are we I'm going to forget this? When I come back, I have 10 minutes.

A

All right, I bid you all adieu.

A

Okay, I really want to finish this. I recognize that it cannot be finished in 10 minutes.

A

Or perhaps even 9 hours, but we will try.

B

A

Fine: okay, fine.

A

This spec is missing. Why is the spec missing, create definition for internal type.

B

B

A

Primitive: okay.

A

A

Equals primitive, obviously,.

B

A

Okay, so now it says primitive array, that's looking better.

A

I think we can leave out the subspec situation if.

A

So suspect, basically.

B

A

A

So determine if the object is something uh is a named tuple or data class which which we can populate.

A

The properties of.

A

Without and without triggering.

A

B

A

So the data's class effectively is not a safe, ah so name tuple is safe. That's why so name tuple is safe to run on deserialization, whereas data class is not because you might end up running post net, which means you might end up running code right, whereas with uh name tuple, uh if you d serialize, uh if you d series, okay with name tuple, okay, so uh uh with okay, so security, no note, okay, security uh named tuple is safe to use as spec, because it offers no mechanisms.

A

Around running code.

A

During it or post, immediately posting it or during it, okay, so data classes and everything else cannot be trusted.

A

Only a name tuple with typing information may be trusted not be trusted, they might run post init code.

A

A

We must must traverse the graph of uh links to uh sandbox instantiation of the correct type. Remember this is generic right, so provided we, this is not remember it's generic, um so, hopefully all right, um we should be able to d. We should be able to use yeah, okay, so false okay. So if um what was here previously.

A

Oh yeah- and this is there's that code right there now there's a little comment that says: don't it up, because I almost it up, I almost opened a big big hole there. Okay.

A

I'm going on and on about it, but I'm the one, the one screwing it up all right.

B

A

I have to go. Oh come on.

A

A

Okay, so we create a definition for the inner class.

A

A

Okay, this looks good, so let's change it to a named tuple, real, quick.

A

Data flow to applies overlay.

A

So testing nametable remove me and my definition, because I am going to forget by the time I get back here in two hours, so.

A

uh Two and a half hours so I'll be back in two and a half hours. Until then,.

A

New type, tribunam double.

A

Okay, goodbye all right.

A

Testing name table moving my definition later after uh fixed okay, I can always watch the recording.