DFFML DFFML Internals, 4 Mar 2021

Previous Meeting

Next Meeting

⏯

youtube image

►

From YouTube: Implementing Dataset Sources 2021-03-04

Description

Yash and John met and worked out how existing datasets could be integrated into DFFML as sources.

We came up with a way to leverage the existing source infrastructure. We wrote new WrapperSource(Context) which can be used to wrap an existing instantiated source.

We also came up with a decorator which allows us to wrap functions (context managers) and create sources out of them.

We implemented a source for the iris training dataset.

Design decisions behind existing source infrastructure were touched on, as well as config infrastructure and the importance of unification there. Entrypoints and plugins in general were also touched on.

A

Are you on a small monitor.

B

Yes, I'm on the 1080p.

B

Okay, let's see all right, so we are going to add a data source that or yeah we're going to add a data source. That's cached or a data set we're going to add a popular data set and as a source and cache it locally. um So, for example, we can just do do from all source.

B

um Let's see, let's do uh csv and then dfm source. Maybe we should make a directory here and do more source cached um or maybe this is like yeah. Let's just say what are these? These are. Data sets right source data set, so let's make a folder called dataset.

B

Okay and then your phone source, let me just grab the any source, fosters all right, I'll, go ahead and see this resource source csv if source data set up source data set and then we'll call this one yeah, iris, okay, so and then oops.

B

So let's take a look at that um tutorial that we have docs tutorials model, tensorflow or yeah. What is it iris is a good name for it all right.

B

So let's take a look at this and let's take a look at tests for it. Well, let's just write first, all right, so we're just going to look at the csv source and we're going to copy from that. So.

B

Iris data set source.

B

Okay and we're going to actually use the csv source within this.

B

Thing, let's see what's the best way for us to do this, do we subclass from cs resource now we want to. uh We want to be a source, and then we will sort of provide transparent access into the csv. So.

C

B

We need a new source to do some transparent access into an existing source to write a wrapper around this, um but we will.

B

Okay, um so yeah there's, so there's really no config for this thing, because it just is what it is um or I guess we could have a. We could have a config with the.

D

D

B

All right so yeah we could have um you know: url stir equals field to download data set from since there's like multiple now there's multiple places you could get this from um and I think we have it actually cached within.

B

I think we have it uploaded to github issues actually, so that we can avoid uh problems with that at download site. um Where did it go? uh It might be in the test tests.

B

No all right, I swear, we put it in there, but oh well! um It's in here.

B

Yeah, so we have this this guy, the train, oh yeah, okay, so we have the training and the test files, training, url and test.

D

B

um Yeah and then we just need, and then we just add the hashes, so training, hash and test hash.

B

Oops and then we should just be able to basically take the cash download on pack archive function or the cache download function and, uh let's see so, would be on a enter so load fd, temp fd. So this is memory based, so we just want to wrap. Then we have an existing source of graphs yeah. We do. We have data the data flow source, so df, that's actually a good one to use as a base. Here.

B

So we can just take this and we can say: okay, here's the context.

B

um We can sort of copy paste, yep.

B

Okay, so there's the update method, we'll use our same source there: okay, here's the record method, where's the usage of the parent source. Here there we go.

C

B

Yield all the records here.

B

Yeah, so we should probably create this wrapper source that just has these we'll, probably abstract this out into something. um You know when we.

A

When we hit that what I was thinking was can't, we have a config option that has the names of these data sets and then the source has the implementation on how to fetch that.

B

The source has the implementation on how to fetch that data set um okay config option. Well, that would be okay. Sorry, can you say that again.

A

So the source would be named, data set source and then you have a config option on what data set. Do you want to fetch like iris for iris, and then we have a dictionary of data sets or some something like that, where it checks? What data set? Are we fetching and then we fetch it whatever we want with python.

B

Well, I think that's the thing is you know the way that you implement fetching it would be with source right um like if we're doing, if we're thinking about. What's the dfml way of doing this, you know um we we that when you say like choose the correct implementation, choosing the correct implementation is, you know making a source. You know what I mean.

B

It's kind of like that this this ends up being just like the scikit, um just like the way that we implemented the scikit stuff.

A

B

Right, does that make sense, I see, do you I guess do. Am I not? Am I not seeing something about that way of implementing it.

A

um So like having a separate source for each and every data set like it it, it doesn't sound, not intuitive to me, but what I particularly thought was when I mentioned this was: if you have this data set source, and then you mention source data set and whatever source, and then what the data set name actually is, and you pass it iris and then, since iris is csv or uh I don't think it particularly matters, but we can just uh fetch iris in whatever way we want and then uh feed it through the source.

B

Okay, um I think you'll find that the implementation of that ends up being this.

B

If that, like I see what you're saying um you're saying basically key off of you know, look you have sort of like maybe a dictionary that says: okay like what do I do for iris, right, okay, download it and then I I I download it. I do the string transforms on it and I open a csv source right.

B

That's what you're saying yeah yeah. So the thing is okay, so I downloaded I do a csv source. I open them.

B

Let's see so you end up saying: okay, this source is an alias okay, so you have this essential source. So basically we would have a key here and you'd say you know, which one do you want me to do and then, within the a enter method here we would say: okay, if iris do blank um so yeah. Let's see, I think, I'm just thinking, because these may.

C

Not always be yeah.

A

So the way you are doing this like uh here, you meet training, url and test url as config options right, but shouldn't this be uh like training. Data set should be a separate source and then test should be a separate data source.

B

Oh yes, yes, it should yeah. That's a good point.

C

B

Okay, let me just let's see because yeah we would have iris train iris test, because I'm just thinking, I think part of the reason why this might look like look like it does is because we just we have. We need these base classes first. So now, if we have this data set source and data source context right, then we end up with.

B

This um so url and url.

B

Yeah I mean you could.

C

B

You could do it like that um yeah, you could do it like that yeah, it's just then how do you? Okay? So then we have. We have essentially end up with this giant dictionary of things that map iris training to iris. It's just. We we've ended up with a new way of doing a plug-in system. Now you know what I mean yeah and, and you can effectively get the same thing by doing it. This way you just might want to change the syntax right.

B

So, for example, like I see what you're saying like from an ease of use implementation perspective, it's you're saying it it's easier for me to implement a source. If I do it this way, right, like it's more straightforward but yeah, and what I'm saying is you can create the same thing by doing a wrapper around the sort, the existing source, implementation right and now you've now you've avoided creating a new plugin system right, because now you end up with multiple instances of our plugin system.

B

Right you have one instance: that's um you know the way that we do it and then you have this other way. That's like! Okay, now let me key off this thing and load this other thing, and, and you run into this thing where, if you have an edge case now, you need to so say you have an edge case wherein a data set is authenticated behind something right, say: you're doing kegel, and now you need some kind of authentication parameters right to download this data set.

B

Now you have to implement a way of passing parameters to this wrapper thing right.

C

B

And now you may as well have uses use the original plug-in system where you can pass parameters.

B

Does that make sense yeah, okay, so the other thing is that okay, so yeah, that's one thing: okay, so this and then so this is so. I think the thing is that that what we can make this in such a way that it doesn't it doesn't, it doesn't end up being as cumbersome as it is now right, so because currently we're we're making. So, let's, let's look at the way that this would ideally work right. So let's say def um def iris data set or data set.

B

Let's just say deaf iris, and you know we take optionally because you might host the url somewhere else so optionally. We take a url, a hash and yeah. We optionally take a url and a.

B

B

B

Okay, so and then we return a csv source right so.

B

And this is actually we need to make that damn cache download function do this without a wrapper, because right now you have to do.

B

B

Because, right now this is a decorator and we really need to make this not a decorator, so you can just call it um so. Cash download, url.

B

This would be, it would look more straightforward if it wasn't decorator, but you have to decorate it right. uh You know what I mean: yeah, yeah, okay, so and then we just say return func uh or no. We do return csv source.

B

Func cache download contents hash and then pathlab dot path, um home directory.

B

Dot cash um dfml, datasets, um iris.

D

D

B

And then the hash is where's. The hash.

D

B

Right, okay, so we download the file. We return to csv source um with the uh you know, with the appropriate call or yeah. We don't need that.

B

Yeah download so iris training right, so this is the uh with this. Is this kind of like what you would be looking? For so say, cash download was really just you know this function here right then this would be the kind of simplicity of implementation that that we're after right is that correct or is there a simpler implementation.

A

I don't think so. This looks like.

B

Okay, um yeah, so so this this yeah. So this is what it looks like here um and then I think you know we also have to do um uh what is this? We also have to make uh where's that scd command yeah. Okay, there's that scd command.

D

B

All right, so this is that find replace to make it look like re dot, sub.

B

And I had to do this regex sub thing the other day and I totally screwed up the order of the arguments and couldn't figure out what was going on. I was like: why is it not so like I can't find it. I totally screwed up the order, arguments, okay, uh replace and then string yeah that it's not intuitive to me. I don't understand that, but okay, I feel like it should be string and then replace, but I'm not gonna burn that into my brain anymore, okay, read.compile, um okay, now!

B

Here's the thing is that if that file exists, then, if that file exists, okay, we want to download the file. This is a trick yeah. If we want, we want to download the file and now the problem. The problem with the cache download function here is that um we're immediately going to end up here, let's just create a new file, we'll immediately end up with a uh original we're gonna we're gonna, modify it as soon as we download it have been. You know, that's no good! um So all right, let's just do redox.

C

B

And then path loop, dot path. Now, let's just do this.

B

So original path is this pathway.path object and then the okay. So now we'll we'll substitute and we'll write it back to.

B

um The replace I just oh man, we don't need to do that or yeah. We do. We need to say, okay, replace path, equals original path, um dot.

B

um And then we'll just put without the original.

B

All right and then we instantiate a csv source with this replace path.

B

And that should be that.

C

B

All right, okay, yeah, so get the original path download. It.

B

Where is our damn syntax right here? Oh there it oh is we have some texture somewhere or whatever?

B

Okay. So then this is the source, so we need so basically what we want is we want some kind of decorator that takes this iris training and converts it into a source.

B

So we could say you know, data set source and then iris, training or irish.train.

B

Yeah and now this okay, so this this this would effectively become. You know our function that gets called right here, so this data set source. So if we make this data set source decorator, we would say def, you know: data set source entry, point name.

D

B

Yeah so data set source entry point name, and then we create a new new class here so class.

B

Data set source.

B

Type entry point.

B

Name so class name, we create the class name.

B

Do you want to keep doing this or is this? um Should we take.

D

B

Because this is a lot of me coding right now so, and I don't I don't know if you want to you know if you want to if you want to do something.

A

I'm really like in a zen mode right now: okay,.

B

Cool all right, um so yeah entry by name replace, so we do replace dot dot title um and then we replace so we create we're creating a class name here.

A

B

Right um so now it should be formatted camel case um and we'll say you know, plus data set source. So now we have iris. So this should give us iris, train data set source.

D

D

D

B

So now we create the class we sub class from data set source, so this is the type, so type creates a new class with the class name, and these are the base classes, and these are the properties, so we can say um config and then so this is okay. So this is a part of the unified config problem.

B

Is that okay, so the unified config thing- and I was talking to saksham about this recently- is the ideal thing that we would have here is the ability to with the operations right when we wrap with op, and we auto create definitions.

B

I'd, like would be great if we could auto create a config um and because that would allow us to configure the operat that would allow us to take any function and create a config out of the arguments and uh that's powerful, because then we can obviously use the existing config infrastructure to instantiate or call in this case that that function right, and so this is something that we don't really have yet, but I guess we'll just have it right now: um where's, my stupid caps lock come on okay, so because we have this make config function.

B

Where's this make config function.

B

Make config class make config, where is that make config okay dammit function and create a data class? Okay? So what? Oh? Yes, this is just a wrapper around that and the other, the other one fields args, okay namespace. So this just looks just like data classes, dot make data class.

B

This is just a wrapper around this yeah. We need to document that thing. Okay, so what's the syntax here, okay, yeah! Okay, um just so we parse the arguments.

B

We grab the type value and the default value, and so we need to do this. We need pi, inspect dot, get signature.

B

All right, oh inspect.signature, so this gives us the arguments of a function and it'll give us their default values as well, so there's the annotation and it should give us the default somewhere. So, okay.

A

Does it give the default values? I.

B

B

um We have code for that, so.

A

What was the problem we faced with scikit there weren't, therefore, the default values defined.

B

I think yeah there weren't type hinting default values. So what we had to do is we had to parse the numpy docs string to grab the default value, so we have code to do this. May config numpy we'll do this? um Did we get, I think? Actually, where is oh, we may have code to do this actually model um dfml util.

B

Make config numpy make conflict tensorflow that parses a google style doc string, um but we don't have any. Oh, my god did we really not do the like most basic case. I think so. um I think that there should be one model pie torch. I think sakshan was going to implement this in model. Pi torch.

B

Because he has.

B

Oops yeah, alright make pie, torch, config, inspect pie, church brands. Okay, we just need to take this out of pi torch, because this is already what we want. um This is exactly what we want so because none of this is pi torch specific. So let's just take this out of pi torch, oh fantastic!

B

All right so and where's make config so make you tell config util config numpy. So um let's just do inspect or well. We don't want to name it inspect. It might get confused.

C

B

Yeah, oh my gosh, that's so funny we should have just not named this pie, torch.

B

Okay, make config inspect.

B

Inspect params.

C

B

I think this is what we want here. um So, let's see, config is dot, util, dot, dot, dfml, dot, dot, okay, sweet um yeah, wow, right, hey! That was easy, um okay, so now we just need to call that so we can just say, make config inspect and then pass it. The um okay entry point name and then we need a. We need a funk to decorate here.

B

So we need to. This is a decorator with an argument, so we need to do diff all right, so we get passenger point name and then we return a decorator wrapper which gets called with funk and then inspect dot. uh Or what is it funk?

B

B

So we wrap um yeah, we do we wrap funk and we turn wrapped. So this is a wrapped funk, and now these are the args and kw orgs.

B

So these are now you know these guys, so ours, that's this. um So then we call.

B

Okay, so now there's a slight variation here where so to do handle async case so return, async, def.

B

B

If inspect dot is co, routine.

B

um Yeah, if inspector is co routine.

A

B

You know: should we open, we should probably should we be doing a context manager, here's my question because then we can yield and that's usually helpful. We should actually be doing a context manager here. So let's yield csv source because you know there's always times when you want to open temporary resources and stuff, and we just do that now, then we won't need to worry about it later. You know what I'm saying.

A

B

Yeah, so when the source closes, we could clean up stuff right, um for example, we could just destroy this yeah. We could. We could keep this this modified file in temp if we wanted to um so. Let's just- and that simplifies this case, because when we're always doing is async gen, um because then because or else you have to handle the case, whether it's a routine or well, you you wouldn't because you're just looking for a return, is async generator.

D

B

D

B

So then we can just do yield um okay and so wrapper, wrapped funk tools wraps funk, um and we should also say um we should say you know, funk equals context, lib, dot, context, manager, funk okay, so we create a con. So we wrap funk. We we take the entry point, we create the class name, we wrap funk. We turn it into a context manager because, or else it's just going to be a generator turn it into a context.

C

B

We yield to it so our wrapper or wrap function yields the result. The wrapper returns the wrapped function, um and now we make a class, we make a dataset source class where config is, let's see the config is you know whatever the config for the function is and okay wrapped funk over funk? Okay, return, wrapped!

B

No, we don't actually return wrapped. We return okay, so we have two options here. We can take this function and we can so this is so. This is a thing that okay, one second, coffee, all right. Okay, we have two options here: we're writing this.

B

This wrapper we're writing this this this decorator here that takes the function and you know, creates a source out of it. So now, if you look at the op decorator, what the op decorator does is, it adds the implementation that it creates. So it creates an operation implementation and it sets it as a property on the function that it was wrapping and that way we don't actually change the function.

B

So if you return, for example, if we were to create a class and return the class now iris training that variable that, because the function is also a variable and the variable iris training is a class now- and I can't just call that class anymore right um so actually yeah. So we have a bit of a. We have a bit of a thing here so because we've wrapped it we've wrapped it as a context manager.

B

So right now I could say with iris training as whatever and I would get a csv source if I wanted it right now now the thing is okay. So let me just write a function here.

D

B

All right so right now I could do with iris training, as uh you know, iris, or with iris training as source and now there's no source equals csv source and- and you know I can go from there right.

B

Oops so sources the instantiated.

B

Csv source- and so we can do this right in this case, we return wrapped here right. So this is this is and we'll we'll just delete this for now. So this is this case right now we have the option of returning the class right, and in that case we will create this class, and when we you know, then, if we call iris training or iris training itself in this case is in instance, is this data set source subclass.

B

Data set yeah iris training data set source right in this case, so example, return class.

B

So the advantage of this is that when we list all the entry points um yeah when we list all the entry points- or the advantage of this is when you register the entry point, it's clear that you're pointing directly at this iris training function.

B

Now it's kind of not clear too, though, because it's like well, this is a function. It's not a date like I can decorate it and it becomes it, but then now I can't use this function anymore. If I wanted to right.

B

And so so and then the third option is, you know, set it as a property right. So now you can say you know, um and this is what we do with op.

B

So now we have so source is an instantiated csv source or you know so iristrain dot class um where iris training dot class is you know, iris trained data set source.

B

And in this case we say, you know, we return wrapped um and we say wrapped dot class equals class. Where rap.class is this right and now we have access to both things. So I think I think that this is um actually the this sort of answers. Our question here with the with regards to like what? What should we do in this case? um You preserve this, and this is why op does it the way it does? Is you preserve the ability to still use the thing right?

B

The function at its base level, if you set the thing as a you know, set this new thing: we're creating as a property right does it does that all make sense.

A

B

Yeah, okay, so so yeah! So now now we have this class. Now we have this yes, so now we have class, which is a trained data set source, and we have you know, training and we can just use that to create a csv source if we wanted to right- and that is nice from the perspective of creating unit tests right and it's just like also just because this is the code that's there like. If I want to be able to call a code, that's there.

B

You know that's nice to be able to do um from from an end user, like from from a random developer perspective. Looking at the code like hey what, if I just wanted to call this function and not have any of the wrapped stuff?

B

um Okay, so we create the config and then we create the um um we create the config, and then we create the uh let's see.

B

uh Oh, we need to set the function yeah, so we need to say wrapped so wrapped wrapped.

B

Well, it's yeah wrapped, let's just call wrapped for now all right, so we have this dataset source and let's just actually make this source a wrapper source.

B

And that way we can reuse this thing so.

B

Because we we, you know, this is something we'll probably do somewhere else too, so might as well just create a base class for it.

D

Let's get rid of this scrap here.

B

All right so wrapper search context. This is just a wrapper around you know, source context pass through complete pass-through, uh enter and exit whenever and here's a wrapper source- um and it is, you know, wrapper around whatever it's config sources, um let's see, and actually maybe we should now make that config source. We should probably make that source um because yeah, because we don't wanna, we don't know if this thing is going to have a config somehow um so now, basically, whatever we said is okay yeah.

B

So whatever we set as so data set source context, is source or wrapper source context, and this thing is a wrapper source.

B

And so what we can do here is say: yeah a enter super dot enter.

B

And we can say self.source.

B

B

So with self.wrapper.

B

Self.Config dot export.

B

As source bam, I think this is it: um hey, enter yeah data set source. Okay, so I think this is. I think this is it yeah, okay, cool, um so all right, yeah, so we've got this source here. That's the wrapper source right, and this basically just says: okay, you know my okay well yeah. So let's do source uh okay, so this would be an instantiated source, so self.source equals instantiated source. Okay, yeah, just a in our method, a or method should always for certain self. So that's not really necessary.

B

We can just await that um so yeah. So, basically the wrapper source enters the source. The wrapper source context enters the source context very straightforward, just pass through on everything here um and then the data set source basically says.

B

Okay, I have this wrapper property, um you know and that's or this wrapped wrapped property right, which is this function that that that we've wrapped we, the and- and I have some kind of config right- and so my config at the point of a enter the configs instantiated with all of its properties, um so um yeah the config will be instantiated with all of its properties. So we can just say export and that will return a dictionary and we can expand.

B

We expand the dictionary with kate, you know, keyword, argument, expansion and now so we're calling this wrapped function, which is this guy, wrapped and we'll do keyword, argument, expansion, which calls iris training, which does keyword, argument, expansion, which says you know, url equals url and contents. Hash equals contents hash which are already set from their default values. So they'll, probably just you, know, they'll get the same thing unless they're overwritten and then we call the function. So we do this cache download. We can optimize this or we can.

B

We can make this more user friendly later so cache download decorator. We decorate the funk and that way when we call funk so we need to call funk. um So we call funk that results in the cache download happening. um So this downloads this runs the download. We say the replace path is okay. This file in the cache is original. We resolved that path.

B

So now we have this replace path. We read, we read the contents, say great. Okay, we read the contents of the original file. We do a regex replacement here, saying, replace the headers and we write that back out to the replace pass and we return a csv source or we yield a csv source containing.

B

You know that will reference the contents of this replace path and it doesn't have that right, property set, so it should just end up with you know: file name equals replace path and yeah, and then that ends up here, so this gets yielded here as source source equals source super dot a enter. I think this is what we want. So, let's find out, let's find out if this explodes spectacularly um so bim tests test there. Let's see boom test source test data set.

A

So essentially, defining a new data set would be defining a new function.

B

Yes, exactly so now, all we have to do is do this right and actually I really don't like that: we're using train and training different places. So, let's see train training.

D

Training train.

B

So now we have sort of more like what what you wanted right, which is just sort of like a you know, a look up table right and so now now. The other thing is that we can do what saksham has done with the pi torch models and add a dynamic enumeration of possible plugins. So not only will you load from entry points right, so we can override the base source load class method to say that, because right now the base source load, class method, class method- I think, is just pass through to entry point load.

B

So we can sub like we can. We can create a a method in in this source. Let me just go in there stifle source source, so, okay, there's no. So this is a base. Data flow, facilitator object or, let's see, yeah the base. Data flow facilitator object.

B

Where is it? Yeah based data flow facilitator object, which is in base.

B

Where is that base configurable? Okay, which is base configurable, wait? Okay, now yeah, here's the base data, okay, yeah, and it's an entry point and different util.

B

Entry point, so this is entry point. So the entry point, class class method load method goes through all of the entry points right, which is basically uh yeah. That's set with this class entry point property, so this is dfml.source right. So when we look in.

B

um When we look in setup.py, that is all of this stuff right. So that's these keys here. So this this in question is dfml.source, so we can.

D

B

This method in source here we can override the load method to wherever yeah, wherever this source base source. So.

C

We can override.

B

That load class method, we're not going to do it now, but, um and we can say you know, source super dot load and then you know, there's there's handling here. Based on whether you did um you know, loading equals none right. So, basically, whether you did um one source or multiple sources and we can say: okay, if you fail to load it then look through.

B

We can enumerate all the sources that we created within this file dynamically, just like we did with the scikit classes and sort of say you know we can just sort of dump dump the contents of this file. Here, this uh source data set or everything within dfml source data set, and we can say, hey you know, let me load.

B

Let me let me go grab any thing: any source source classes that reside within dfml source data set and if the load fails from the entry point see if we can load, you know see if that exists in something that we've defined in that file right. um So that's something we can do um later uh and that way that way we sort of auto anything that's defined in here- gets auto exposed. As a source, um so this yeah import uh from different.util.testing., yeah or yeah async test case how we're doing on time.

B

Okay, cool import, uh async test case.

B

Iris test iris training data set.

B

B

All right, so I know we want to import that guy. So from.

B

Dot default.source.dataset.com import, iris training.

B

And this is how I run my tests, which I've showed you this before.

B

Oh yeah, we copy pasted all this crap.

B

B

Csv import csv source.

B

Okay, uh config.baseconfig field: I thought we got this somewhere, but maybe not oh. Did we get entry point? We didn't get the entry point yeah, that's a good thing to also get um context, manager, data classes. Okay, we don't need that async io! Don't need that editor tools. Okay, we need inspect and context slip.

B

Okay, so we need to inspect contact slope. We probably need typing donate. Any of that.

B

Okay, man we'll see it'll blow up on us, so we can just oh base source.

B

All right great, oh, and we need to do this entry point here. Oh we don't want this. We want make config inspect, which is from util config, inspect.

B

So yeah we got that guy context, manager, data set, sort, okay, context, lib, which oh entry point yo. We need to create this guy, so this is a new type and we need to wrap this with entry point and then entry point name right, and that is the so. This is the uh right hand. Side of.

D

B

All right, so the entry point decorator is the right hand. Side of this so or the left hand side. So this will be. You know, for example, if we do once we do at that at entry point, that's setting that that's setting the entry point so remember that entry point um within dfml, util entry point this entry point here. So this is, you know dfml.source, and that is grabbed from since this is so this the source class subclasses from entry point.

B

So the entry point property on anything subclassing for source is dfml.source entry point label is the like csv. So this when we do that at entry point decorator, it sets the entry point label right. So this is entry point label. So this is where we're going to be setting dataset.iris.

B

So entry point label is getting set to dataset.iris in this case. um Okay, where was I um and then type is not.

B

Callable, uh let's pray this dataset source: oh yeah, return.

D

Return wrapper funk.

B

Tools, okay, make config inspect. Now we need to. It's needs a name. Last.

B

B

No, it does not so search for iris.

B

Training so yeah there may be a method for this. Second, the unit test thing has so many so many helper methods- okay, yes, it has that class variable.

C

B

So now we test that the name is what it should be.

B

And that would be.

B

This great, that is also what it should be and then so a third class. Okay, let's see, um I think, we're now into assert class okay. What do we want to know now? uh I guess let's just like try to instantiate it. It hurts so like test, um but let's see test.

B

uh Okay, so now this is more of a different thing here,.

B

Test data set source.

D

B

So let's call this: let's have this decorator.

B

um Okay, so what do we want to test here? We want to test it like the wrapper that that this we're going to try to test this part here this data set source stuff. So um so I guess let's just enter the source, so async with async with yeah. Let's just let's just start here.

B

Let's just run it async with. Actually, I think we can just call. Let's just call await load.

B

We'll just load it and test that the records are correct.

B

B

Okay, so this should be the class and we'll just instantiate it with the default arguments and we'll load it and we'll check that that the record that there's there's data in the file.

B

B

Async generator can't be used outside the weight expression, so class.

B

Oh yeah, all of this was not async huh um here, wait a minute yeah weight load, wrapped, wrapped, okay with wrapped okay, so wait. This is still async generator can't be used in a weight expression load. Let's just make sure that this instantiates the right thing. First.

B

Okay, so this is an iris dataset, training, source, okay and there's the config. So that's again, it's instantiated class, so we should be able to pass it to load async with as source. So this should enter the source.

B

Okay, yeah config has no export. How about azdict.

B

Attribute error enter self.wrapped, okay, so func should be context. Lib.

B

There's training context manager funk yield wrap. Maybe we need to do it on this too.

B

I'm not familiar with.

B

Them I was trying to get multiple arguments for url.

B

B

Okay, maybe that was the trick. Maybe we needed to you'd think that funk tools wraps would preserve the context manager.

B

Okay, so url yeah, okay, so it looks like kwrx, is correct there and there's no or wait okay looks like args is getting oh itself. Is it being past self yeah there we go there. We go that. Should there you go so now we're not getting self. I always forget that all right, so our generator context manager has no attribute a enter so that source. Oh, we need to return itself.

B

All right returns first server source; oh wait: self.source.study enter generator context, manager, self.source, okay! What's this.

B

Context, loop generator. Oh, I think this is because we double wrapped it. Maybe it didn't need to be double wrapped like that.

B

Yeah, because this guy yields something going on here, attribute error, a enter self wrapped.

B

So we call wrapped contact slope context, manager funk.

B

B

Yield funk, oh, that's what it is with funk as result, yield, funk or road result, uh yeah, obviously, okay, and then we need that call to or we need that decorator.

B

Duh, okay path of not defined all right now we're making progress.

D

B

Download not find.

D

B

God I really miss. I need to get that vim plug into auto import stuff, no such file or directory. Okay. We need to create the parent, if not original,.

B

B

D

B

B

Okay, I accidentally made it a.

B

B

Message, file directory data sets, let's change original.

B

What is this wrapped read text.

B

This funk locals funk was never awaited. Okay, yeah that cache download function only works on uh this cache download function only works on.

C

B

Whatever, let's just do this url lib, it only works on code routines.

B

Okay, where is that my favorite url lib function? This is great. I love this thing. uh This thing is great url I have posted on there. They are like considering removing stuff. I was like. Please do not remove this thing. This is great um where'd. It go url, retrieve yeah check this out. um Where's.

B

Where's, no, where is it? Where is it where's? It hacks.

B

Come on yeah with progress download with progress. This is great.

B

D

Okay, I think your.

B

All of your arc retrieve, I think it does it do it to the temporary. That's the question or retrieve file name: okay,.

D

B

Yeah, this report hook is great um stir.

B

uh We don't need this replace path, all right, yeah, we do. Let's just do it like this. Let's just change as few things as possible right.

B

B

All right, we'll scrap that for now.

B

Let's just make sure they tell us the right place, test source test data set and then url, oh scissor, v1. haha um Don't do that.

D

B

Object has no attribute from the imposed a path that exists. Soft.Config.

B

Csv source, so it downloaded it to tr. We got training.

B

So it looks, looks pretty correct to me, um except for stir, object, has no attribute file name so to config. Oh, it's just yeah. Okay, maybe we did. Let's just set file name.

B

All right, hey, how's, that how about this now async generator can't be used in a weight expression.

D

D

B

Then type, let's know which paper.

D

B

Source source, what in records line 150.

B

None type we forgot to return something somewhere, something returned none. This.

A

Does your regular very game, look like this.

B

No, are you kidding me, I'm never so successful um this? Is this worked out really well, this went so smoothly, hey, we got all our records and I have to go to my next meeting. um That was like, I can't believe how well that went. um Anyways, um okay! Well, let's sink later, I'm gonna post this video. um I can't like this went really smoothly. I'm very excited about this um well.

A

B

Yeah, let me push this to a branch all right, hey. That was fun uh thanks for indulging me there. um I said it'll take two minutes, but I guess it took an hour and a half so but hey fun stuff. uh I hope I didn't bore you here. Oh no! No! No! All right! I will push this thanks. Josh have a good one.