DFFML Weekly Sync, 14 Apr 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Weekly Sync 2020-04-14

Description

Meeting Minutes: https://docs.google.com/document/d/16u9Tev3O0CcUDe2nfikHmrO3Xnd4ASJ45myFgQLpvzM/edit#heading=h.nf93uxx03qh6

A

Right trying to get this, you know how shame is having issues I'm, trying to sit up. My.

B

Okay, come on darker environments,.

B

Okay, let us what's going on there. Oh.

B

A

On No, oh geez, a cut pissed everything all at once.

A

A

So this will just start going through things or I wanted to tackle this first. This has shame on snow he's, not all right. Well, I! Guess we won't cycle this first, oh well! um Okay, just quick, tup date for you guys, so I was hoping to get this done for the meeting today, but didn't happen. Oh well, it'll get done after the meeting, but I'll show you right here. So basically, I was working. We've we've been talking like we've talked about how we wanted to get the HTTP API into the QuickStart documentation.

A

This pull request right here and so I'm almost done with it. I had to refactor some of the test code. This is now done, and so basically, what we're going to end up with is like at the end of the QuickStart documentation.

A

When you go and it's like it talks about how to do it from Python, then talks about how to do it from async Python. Then it will basically jump to something like this, where we start the HP server and I added a new syntax, so we can sort of just like pre configure the model. I, don't know if it guys have played with the HTTP server, but the idea would be it provides you all of the I mean it.

A

You can do anything from the HTTP server that you can do from the command line or from the python api and that's what's gonna. Let us write the web UI and so basically I just added the same same issue in tax. Basically, the syntax that you have for sources is now also the syntax yet for models, because with HTTP API you could have multiple models active at the same time. So you basically you load the models. You tag them with their their label.

A

So, like my model, equals the SLR model, and then you say you know all their options and then the HTTP server will have that model running on whatever. Given you are like it'll, be on my model. So, for example, you know it'll be in the quick start in a minute, but if you were to say slash, if you were to send this data to model slash, my model slash predict slash zero. Then you would end up with this as the response, which would have the prediction in it.

A

So that's that's just what's it's gonna, be there very soon I didn't get finished with it. Our mom will stone with it. But, alas, that's how things go. So, let's see a quick start with addition of HP service to do model prediction almost done.

A

Other news I have any other news for you guys, let's see I, don't think so. I think that's about it. On my end, we're still doing that the GSoC we have to go through and evaluate everybody's proposals and then we're we've got. You know we're all putting our mentor comments on the finalized proposals and then we have to make a decision eventually, but we don't make that decision yet. So we haven't made that decision yet and obviously you won't find out until the fourth anyways. So that's all the news on that front.

A

Alright! Well, let's see and Hashim is here all right. Great! Ok. Can you guys hear me by the way I assume you guys can hear me but go inside.

A

So first thing we're going to tackle today is the fact that.

A

Having issues Hashim you're, having issues with the entry point name stuff, and so this is so that I recently added so that we could have. Why did I? Add this oh for the labels basically or we'll run the script and I'll show you guys John.

C

Can you explain the ng pointing to I also don't understand.

A

Yes, alright, so this is why, right so, let's see, let's just go to setup touch you. Why I can you guys see my screen? One really obvious.

C

A

D

A

Okay, so yeah, if yeah that's what that was, that was my house. What I was wondering. So, let's see, let me just open it up in github, because that is then I can change the zoom easier. So this is probably something I need to document as well.

A

So did it alright, so entry points we'll just take it from the top, so the entry point system is this system that Python has is as a language or well yeah as a language that allows us to sort of like its. It allows us to give a path to some Python object within a Python file within our package and then label it as some label, and that way we can load it.

A

Other Python programs can can use this interface provided by the Python package, packaging library, the package resources library to discover what Python classes or functions or objects have been registered with that entry point. So the nice thing about this is what it lets us do is. Is we can we? It basically facilitates this plug-in system that we have so by registering a class or a function with the entry point. So the way that we do, that is we say, entry points, for example, console scripts is, is a set of tools, specific things.

A

A set of tools is the package that actually is in charge of installing Python packages and setup tools says: okay, if you see any, if you see anything in this array in console scripts, then I'm going to create a file within the path that calls this function, it loads this function and it calls it and obviously the function is. Is this main method within the CLI class, with India from LCLs CLI? And then the colon says the colon is the separator between?

A

This is a path to the file in Python notation with the dots and then this is the function on the other side. So that's this is. This is how it works for the standard, the stamp there. This is like what setup tools is doing when it sees this and we're doing a similar thing.

A

Basically, we say: okay, whenever you want to load a source, we go and say: okay, look through all of the entry points registered under the DFO, from l, dot, source entry point or all of the I guess labels they they're just called the syntax where this stuff is weird. When you look it up it's. This is called a entry point, I believe so. Each entry point under this DFO from l, dot, source label, I, guess I guess label is going to become an overloaded term real quick.

A

But basically, if we want to find all of the entry points which are a DFO from l dot source, we pass it to this function.

A

That, uh if util entry point, we pass it to this function rate here.

A

But package resources iterate entry points, so this function the so class that entry point is the DF of ml source and then, when we call this function, let's see.

A

Okay, come on that was a very unhelpful I favor where's, the packet resources API.

A

Come on now there we go.

A

Okay, this isn't really like the main documentation. The main documentation is kind of hard to find it's within set of tools. Yeah there we go I believe.

A

I'll try to hide it here. Alright, here it is yes, this is the main documentation for this thing, oh wow, let me but dogs, look all weird! Ok, yield entry point names from group matching name, oh well, look at that! You can do filtering. Ok, I, just learned something new. That would be very helpful because we have this whole thing that does filter in here.

A

So now we don't need that, but basically yeah it just it goes through all of the things so for every package it looks through every single package and it looks at the DFO from l dot source and that it looks at what's under there and if you actually want to see what that looks like I, think some of you have discovered that if you look in dot AG info- and you look in entry points, there's an any file in here and the any file is basically just when you install the package.

A

It creates this any file and that's why, if you modify the entry points within the setup here, why you have to reinstall the package because it's regenerating this any file, and so, unless you reinstall the package, this any file doesn't get read and this function, this Aitor entry points function, is just going through and reading the ami files. So it's not reading the set up top you wise.

A

So you have to reinstall the package when you modify the entry points, because it's regenerating the any file which is what's actually getting read all right and as you can see, this is just a mapping of you know the label to where we can find to that thing now this. This is all well and good right, but this doesn't tell the actual objects themselves what they should be called right, because this is just in this set up top here.

A

Why and now, when we're doing, testing and other things like when these things, so this works well, if we're going to load it by looking in like by calling this function right. So, if we're, if we specify CSV source on the command, then that's all fine and good, because when we load it with it our entry points, we know that it was CSV source. Now the thing is and we can set the entry point label equals.

A

So here we say: okay, when we load the thing the label equals whatever it was right, CSV source now this is all well and good, except for, if we're not loading this thing via the command line like what, if this thing is being loaded via some test case right well, in that case, we need to have this entry point label already set, which is why we have the base entry point and entry point decorators, and so you guys have probably seen these. So what they do is basically they say.

A

Okay, the entry point is the first argument and that's going to tell it like.

A

Okay, if you try to call dot load on some class right like if you call base source dot load, then it's going to call it's going to end up calling this load method, because space source derives from entry point and you have to put this base entry point decorator on anything that derives from entry point, because you need to set it's going to go ahead and set this entry point class property so that it knows what to iterate over with the package resources later entry points right and then it's also going to set the default entry point name and default entry.

A

Point name is the next argument here or the next series of arguments. If you wanted to, and so this says that DFM l dot entry point is going to be referenced as entry point right. So with that what this means is that when you're typing a command- and actually this is actually a great example of that, so you may have noticed that when you're typing a command on the command line right and you have like a model command or you're better, you have a source command right.

A

So you may have seen we've seen this syntax, where you say sources equal and you give it some. Some some label here equals the source type right and then you say, source my source file, name, training, dot, CSV! Well, you don't have to say my source, if you just say, source file, name, it's going to infer that you are talking about like the only source there right and well.

A

It's doing that because we've set this entry point name to be source for the case of sources and we'll just go to sources so that becomes more clear, so source sotius.

A

And then we're gonna move on soon, because this is getting a little long, but it is important because so much of this project has to do with configuration to make all of the plugins work together. Let's see yeah so here you see DFO from l, dot, source and then source right. So if you took off this my source, it would still work because, as sake amma seen with the code that that does the configuration it basically does, it does two checks. It says: okay, do I, see source my source file, name.

A

Ok, if I do see that I'm gonna use that as the file name for my source. If I don't see that do I see source file. Name. Ok, if I see that then I'm gonna use whatever that is as the file name for my my source right because sometimes like when you're. So you can specify multiple sources right to training commands and actually like to the machine learning base commands. But sometimes you won't like a lot of the time.

A

You're only specifying one source, in which case it's kind of just overkill, to write source, my source file name. Every single time, if you only have one source right so this lets us do, is it just lets us specify which source if we want to?

A

Otherwise it will apply to all of them right and and not just sources, basically whatever plug-in types, so it works for models and that's why that's why we have the entry point name and so what's going on here is like the error that we're currently talking about is, let me see if I can run it Octus.

A

Okay, so yeah there's something wrong, so there's something wrong with your environment. I.

A

Think, let's see so, the the error that hutch team is running into is saying that associate has no attribute entry point name and the reason why that is is because associate is an operation implementation if you mess with the operation, implementations, they're kind of weird right, they're in operation and they're, an implementation all-in-one they're like well there's the operation, implementations and then there's the things that you decorate with the OP decorator and those create like this sort of special concept, which is like this op-amp, which is the operation implementation, all-in-one, and so what happened here was because so the way this should work is that the operation implementation class will have the entry point name on it.

A

But operation implementations are weird or like the deification entry point is weird, because we have to make it sort of easy to use. We used to have this thing where it used to be that you had to register the operation implementation separate from the operations. Now you don't have to do that anymore.

A

Now you just put everything under DF imelda operation, and if it happens to be an operation implementation rather than just an operation, then it basically does the Association and maps both of them, depending on what you're trying to load, if you're, trying to load it through the through the operation. If you're calling operation, dot, load, it'll load, just the operation or if you're, calling operation implementation, dot load, it will load the operation implementation and because of that overloaded, behavior, that's why we ran into this situation where we have to.

A

Basically all we have to do. Is we in the OP decorator? If we decorate something with up, we.

A

Entry point and we just throw on the entry point name of operations so that it knows that this thing is in operation and that's for the purposes of generating.

A

Docks plugins- and this is what I was saying again like if you're loading these things, we have the package it or entry points. Usually you know what they are, but if it's for testing you don't I guess in this case we could have assigned it, but it's better that we keep it here, because if we need it within just the Python environment, where we didn't load it via the editor entry points like a lot of the test, cases will break.

A

Basically, if these decorators don't exist because they're not being loaded via the command line way of loading things. So here's the example all right. So this is why, and what is that? One in question.

A

Well, maybe it's not even there and maybe that's what's going on so you said, associate.

A

Thank you, let's see, oh, this could be why what's going on cuz.

A

Well, that's weird I'm not seeing associate, let's see or whoa. Oh it's associated our case yeah. Here it is okay. So what the reason why it's the right net errors, because it's trying to figure out what is the entry point name, because it's generating these labels that we can link across the documentation? This is because you can't link to a specific header on a page without doing one of these labels, and so the label has to be specific enough. So we need to make sure that we're saying it's a plug-in.

A

It's an operation and here's the package of the plug-in and here's the operation name and so, for example, like with all the get ones.

A

They are: where is it lines of code? Two common ratio, DF of or plug-in operation DFL feature get lines of code, two comments right. So that's a really long label and the reason why we have does so, let's see was because we needed to do docs usage IO. So when we added this the other day, we found out that well, we really need to be able to link specifically to this model plug-in right. So now the result of this is, you can say, ref reference, specifically this section within that file.

A

So you can say: I want a link to plug-in model, dff, MLS LR and when you're on the doc site and you're going through this tutorial- and you say: oh we're using the simple linear regression model. Well, what does that mean? Well, this is what it means and here's the docs for it right: okay- and that was a very long-winded explanation, but hopefully complete about what our entry points and what is all of that stuff. Is there any more questions on that? So I know I, probably missed something.

C

uh North right no I guess it was a really good explanation. Thank you. Okay,.

A

No problem, yeah I, thinks I, hope, I hope is I, mean I, think it was pretty complete, but I. Definitely there's there's lots of things there. The entry point, the utili entry point file can sort of clear up some of those stuff, but for this issue specifically ashame I would say: I. Just so. I just pulled master and I ran, I pulled master and I ran the doc tests and it looks like it worked. Fine and I did I. Did the fresh container install thing?

A

I did this right, so I ran this container and then I did this and it looks like it's working fine. So we can try it on your branch here.

A

A

Well, can you live or let's see so, what's your current setup you're got it in the container.

A

E

A

Let's see if I'll apply the patch and we'll see what happens.

A

Okay, it doesn't like adding mapping. Let's just do it really.

A

A

A

Looks like your test got at it, that's for sure, so nice and your test pass. So that's great sweet, sweet! This looks great yeah and it looks like it's all. Working, okay, so well. Good news is all as well I guess you do need to add to the change log to make this. Oh yeah. You need to do these guys and then you need to add to the change log and then I think this is good to go so I.

E

Was actually adding the DB operations to this one as well? Okay,.

A

Let's just do this as its own thing, then, because, since this is pretty close here, okay and then the other thing is okay, so why? Why is this I? Can you go ahead and try doing these commands again and let me just throw this in yeah.

E

I'll try to create a container and see you yeah.

A

I have a feeling that, if you do, even if you just did this in the existing product create a fresh container, but even if you just did this in the existing one, I have a feeling that it would, it would clear itself up. My guess is something got installed, not in development mode, and sometimes that happens. I, don't know why. But things are weird. Pip is weird but yeah, okay, yeah and let us know how that goes so, like.

B

A

News is it should it should work, um so is, there is hope? Okay, let's see all right. Let's just note about that. So I explained entry points or about that's see according.

A

A

All right, okay! So what's what's up next? Who wants to go next.

D

D

Notification, do we pull down the like the we pull down a report to a new folder or we want this working yeah.

A

So yeah, so this is a good question. Okay, so we have a couple ways we can do this and I was thinking about this. Some more and I was thinking, let's maybe just not mess with that at first I. Think what we want to do for now is kick off a data flow, because then we can make the behavior completely customizable right, so I think that we probably want to. We want to figure out like what is the, and maybe it's just like the whole web hook.

A

Structure that goes in is the data type as the input data to this flow, and then but yes, it will just kick off a data flow for now. So basically you make you make that web hook service, like you just make that handler. So let me just make these nuts where's that issue where's. That issue I.

A

Just clean up here, cuz I'll fake it all over the map. I know yeah I. Think.

A

Yeah from this perspective, of of what do we do, let's, where is.

D

The uncommitted changes.

A

Yeah yeah exactly so. This is yeah. This is what I'm thinking is so so my original the original plan with this was sort of like you would you would dump it on your server and then you you would start the web hook on your server and then anytime. It got a request. It would like pull down the new changes to the git repo that you were currently in, but that doesn't really make any sense. Actually so because, like in this, the the more common use case here is probably going to be.

A

Like I start this web hook on my server and every time we're gonna provide a data flow that every time it receives the web hook, it creates a tempter, it pulls down the git repo and we've already got operations for this Rae we've got to get operations, and then we need an operation for like build a container so basically see if they get.

A

If the git repo has a docker file, we built a container and then we redeploy the container, and this is all stuff- that's going to be like just operations in a data flow right. So if we just give the web hook the ability to run a data flow whenever it receives the honestly wait now the more I think about this I'm, not even sure we just need this, because we already have the HTTP service and if you just registered a path on the HTTP service, you could just register a data flow to that path.

A

Yeah all right! Sorry I was like I tried to think this through, but this was it started out as like a note that I jot it down, because I wanted to make sure not to forget.

A

Okay, you see what I'm saying.

D

A

Exactly like because the point of this was well, we need this deploy service so that when it hits the like, you can set this deploy service up and and then it will. You know it will receive it on able to receive it on some path, this Web hook, and then it will do something and then I just realized or just reinventing the HTTP service. So.

D

Most of the code was same from this.

A

Yeah, okay, well, sorry about that at least you're familiar with it! Now, let's see yeah, because.

D

The local police, in the it.

A

Does yes so I guess you know, this is one of the things that we need to that. We need to think about here and I think this is like. So let's just make some notes here and then we can transfer them there right later, so we realized that we probably don't need the deploy service.

A

What we probably need need is to create some operations that know how to build a docker container and as well as redeploy the container so and then, let's see, let's see, let's see, let's see, hmm okay, we don't need another service.

A

We can just give instructions on registering a data flow under I'm using the issue P service.

A

Hp service for some URL to be used by or as the webhook receiver.

A

Okay and let me just sort of pull this up for a second, so just to be clear. What we're talking about here is like this is I think this is the only example at the moment, but in this example, oh okay, here we go, this is gonna be this is this is this is this is what the deal is? Okay, so right now, right now, when we okay.

A

So this is the current example for how do you use the HTTP API with the data flow and we basically, we take the should I data flow, and we put it behind this. Slash should I URL on the on the HTTP service and it runs the data flow and it gives us the results now.

A

The input requires that it be in the same format as run data flow where you're saying: okay, what's what's the value and then what's the definition and you have to do like you know, you have to put it under the context and everything so I'm now thinking that our first step here really is to to modify the HTTP API to say to have like a couple ways that it can deal with with the inputs right, you could say: okay, the first way you can deal with the input. Is this way right?

A

I, don't know we're gonna call this, but we make a name for it. So it's just. Let's write this down, so modify new plan need to modify the HTTP API so that we can. We have multiple ways of dealing with the input so number one I can't change it, okay, whatever number one or it doesn't really matter, so we need a name for the existing okay and- and this is going to be some command line four or this is going to be an option.

A

What is this called now? The channel configure okay, it's going to be an option to the channel config, which is this thing right. So.

A

It will be an option to the channel channel, config and I believe this is the class name we'll just put channel config.

A

So we need a name for the existing way.

A

Which is to which is the run? Data flow style of value of specifying value and definition.

D

A

So you get the data from the coast post request right, but we might so we want to. We want to so okay. So the reason why we have two different ways here is because this so for the web hook all is well. If this, this all works, fine, okay, so for the way that it works right now we can specify okay I. Have this this I wonder there should be an example at the bottom. Yes, okay, there's an example.

A

This is the reason why it works the way it does now, because you, if you were to just take the whole input body and assume that it was one value for one definition, then you couldn't do things like this, where you're leveraging the whole run data flow thing where you can run different contexts with different input sets right.

A

In this example, we basically say we specify that for insecure of a package, here's the list of inputs and for DF FML, here's the list of inputs and the result is that it runs both of them right concurrently and does the analysis right and- and so we can only do that, because we can specify volts poll were able to specify multiple inputs for multiple contexts.

A

If we just took the input body as like, if we post it in secure package or something then we would have to like, we would only be able to do one thing with that and that's that's why it works. The way it does is just a pass through, because this was sort of like the easiest. The easiest way to get there to replicate the functionality.

A

The most transparent way to write- and so this is yeah, so this is the existing way, but we need a new way right and the new way could be I, see two two paths for this right and and we'll. We really only need to concern ourselves with one of them. At the moment we have I think we should be able to take the we should have.

A

We should have three input modes right and you'll use and you'll switch the input mode, the option to channel from config called input mode or something and the input mode it'll. Just you'll say, like you know this one. If we call this one default right, then in this file it would be like default. Calling the next line would be for input mode calling default, and in that case it would operate as it does right now. Now we need a new input mode. That's like.

A

mmm Let's see, maybe we say like input, we need a way of saying: okay in from a default input mode.

A

We okay, so we need two more one is basically we're. Gonna use a data flow to decide to transform the input into whatever it should be right. You could specify a data flow and we would let's not worry about this one for now, but just in the future, you could give a data flow as the data flow to process the input and output it into the format that we have currently as the default format right.

A

So, if I see some sort of you know arbitrary request body, it's an array of two values: right: okay, well, I'm, creating this data flow, that's basically going to export like the the versus return value of the data flow would be this structure when it sees an array that is, you know, 4 0, with index in secure package. Second index DF FML right, then this you know hypothetical data flow. If we, if we specified, use a data flow and then we specified that we should use this data flow.

A

This hypothetical data flow would turn that array of insecure packaged, gamma DF of ml into this value. Here, which is you know, the run, data flow style thing, so it would be some data flow that transforms whatever the body input is into the thing that we're actually going to pass to run data flow.

A

So that's that's like something that will need eventually, but not now so not now, but eventually, because that gives you complete control over what the hell are we doing with this thing and then we need the last mode, which is basically just take the whole input body and assign it to this definition, and so actually well, there's one more. We need okay, so for now have add a mode which says we should take. We should treat the whole input body as a certain definition.

A

Also provide- and this is like yeah- this is I- don't know if it's really it's sort of wrapped up in this, but also provide a way to have the input be treated as urn. Now, don't worry about this for now, so so, for now out of mode which says we should treat the whole input body as a certain definition and.

D

Specific, what mode or does it come with a mmmm.

A

Okay, so, basically with you for this case right we're seeing the web hook, post requests not come through right and well. I don't have an example of that right now, but it's this giant blob right of JSON and yeah, and so basically, what you're gonna say is in this file. You're gonna you're gonna create a new, a new kind of entry, and let me just pull this up real quick here.

D

A

Yeah, so you get this giant blob right all right. Yes, this this crap okay! So now this is just like. Okay, great, like I've, got this giant blob. What the hell am I gonna do with it. Well, what we want to do is we want to just say like okay, this is the this.

A

Is the thing that's going to get posted right so with this option, what we're doing is we're saying treat the thing that gets posted as the value corresponding, create a new input element assign the value to this whole thing and then have the have the definition be whatever I specify in this config file. Does that make sense.

C

A

That make sense sorry I can.

C

A

D

A

Yeah you're you're cutting off sorry.

D

Is it fine? No, yes,.

D

A

Great okay, so this- and this is the channel config. So basically, what we're seeing here is that.

A

A

Okay register: where do we use this? Damn thing? Oh yeah I think we just call register config I, wonder why there was some reason why.

A

A

Yeah because we're loading it like that I think, there's, oh I, know why okay yeah, so the thing is that there's this is like half the equation. The other half the equation here is in.

A

Do no smell where'd it go there, you go, should be someplace with the multi-column stuff. Multicom is just not the greatest word, but alright, that's what it is.

D

A

Are gonna now we're trying to come up with what the hell is this thing and we're like? Well, it's a thing that offers multiple channels of communication right. Each channel is like a path right. So if you're looking at a cheapy server, the channel of communication would be the URL path right, and so this thing you know whatever this thing is- is some it's some contract that offers multiple channels of communication. So that's end up being the name for the base class.

A

Here, it's a multi klom because it offers multiple channels of communication, and so, like the HTTP servers, a multi-column you could have something like like an IRC server could be a multi-column right if you had messages coming in on different channels, and you wanted to treat those channels like every every message to every channel. It gets treated with a different data flow. That would you know that would be a multi-column. I don't know where the hell it went, though beautiful it should be in here. Maybe oh, it's in DF! That's why multicom yeah!

A

It's not the greatest name at all, that's for sure, but it is descriptive, but it is what it is well I, don't know if it's descriptive, but it is what it is open to suggestions, so, let's see registered directory.

A

Basically, this is just is a mess, but it goes through, and this could probably be simplified now with some of the work that you've done recently, but it goes through and it looks this is what so you know where's that okay register could be the reason this register config exists is because this base class is going to call that register config method, to figure out what the hell that config class is and then it goes through, and it basically says the data flow. This is the way it looks.

A

So if you name the config file should I in MC ish P, so the config files before.

D

A

D

A

D

A

Right, great yeah, so you understand this, but yeah, basically all you're gonna. Do you need to modify that config? You need to modify the config and HP server side and then you need to modify the config here and then, if things get screwed up parsing it you just fix the parsing right, so I don't think it should blow up, but it might blow up on you anyway. Just add this new config mode.

A

Only only worry about this one for now eventually add the ability to specify a and you may want to also do you may also want to say, along with this, you may want to say right because right now it's just gonna take that info body and it's gonna be like well.

A

This is a giant string yeah and then it's gonna pass it on right and then your first operation is always gonna, be well parse, the JSON right and that's not going to be any fun, so you might want to add, probably probably want to add a method to or a option to say how the request body should be parsed before assigning it to the value.

A

All right so that the first operation.

D

Always be like our the validate which we currently have. Oh.

A

Yeah I guess it would I guess you could do that yeah yeah! You could do that you could just do validate and then you chase some downloads. But could you do that? Because this is the thing.

D

A

Can't export it right if you exported it, then a json dot load, it's no longer as a thing and that's dad that's why we have that problem. Yeah, so I think you need this for now and then we got to figure out that validation thing. Eventually, the validation thing is very cool, it's very excited about it, but now we gotta. Apparently it wasn't as easy as it seemed.

A

Nothing ever is right.

A

To deal to transform the.

A

Request body into.

A

Request body into.

A

Alright, is there anything else, so basically, I think this is. This is the new. The new plan here is make make make a demo where it does something like basically make an example where so you go make some make a documentation page where we show how to use a data flow with the HTTP service to receive a github webhook and do something with it for now just focus on getting a container built.

A

So so, and this this.

D

Is converting to GPX number so maybe like the documentation after the how first up will modify the operation to have maybe start time and end time, and this will just reveal that container. It's that point.

A

Wait sorry Sagan.

D

So currently we have a converter chip operation so for now it just takes the input path and maybe, after the first step like to have a push command. Oh yeah.

C

D

Add to take the start time and in time, and so the flow is modified, and now we push it on other pieces of command and yeah.

A

Yeah that sounds good, so you're, saying you're, saying you're gonna modify it you're gonna modify the start time and the end time within the operation. The code that runs the operation itself right.

D

A

Okay, that's that's! That's better! Okay, because yeah! Ideally the way this would work is the config or well they really the the IMP. You should have an input, an input that modifies the the start time and end time. Maybe we have something like that. Can there's a config that lets you do the resolution on it.

A

So, let's see so so that that what you have to code up, I'm, sorry, everyone! We went a little long, we're going along today.

A

Basically, okay, so let's just go with this: let's, let's say for ya, for the demo use the F MPEG operation have have the first commit the or have the yeah first commit slash initial state, the a data flow. So you have a you have like you, created the operations right and and then basically you're pushing an update to that deploy directory that it has a new serialized data flow, and the config now specifies a different resolution. Does that make sense?

A

Okay, cool having it have a first command initial state? Where big for operation is. One resolution then modify their resolution.

A

A

The operations config.

A

A

Like and show how it gets redeployed and you could even like yeah yeah. This sounds good. This sounds good, so basically, the flow here is going to be user. Creates this. You know we run them through DF ml service create operations, and then we say: okay, now that you've done that.

A

Here's, the you know, then literal include the ffmpeg file right within the examples directory and then then show them how to serialize the data create the data flow. And then you say: okay, here's how you create this.

A

Here's how you create another data flow, that's going to actually be watching on the web hook right. So you're gonna have two days: flows. You're gonna have to deployment directories. Basically, you're gonna have the one, that's the that's actually within the ffmpeg operations, and then you're gonna have the other. That's just somewhere else. That's actually in charge of doing the webhook and then in that data flow it pulls down.

A

It uses the get operations and it pulls down, it does the git clone and it has the cleanup and it does the container build and it does the well. Ideally, it does the deploy, it actually runs it with like pod man or docker, or something and yeah. You can sort of figure out how to do that or you can just take, take it as far as you want to take it and then we'll work it from there. Sorry what.

D

I saw the docker instant comment so that you can get all the.

D

D

A

Yeah that could be a good way of doing that, yeah. They are exactly that would be. That is exactly what I was envisioning for this initially so yeah. That would be the best way to do this. All right, perfect, perfect, yeah and then I mean that'll, be a very complete example. On okay, hey you have some code. You want to make sure that your server is always up to date with running that code.

A

Here you go and that's gonna be really cool, because then people can, you know always have their models and all their operations up to date. Sweet all right, thanks, Hagen, all right! Sorry, everybody that took a long time, I've been very talkative today, so who's next would also want to do.

F

A

F

A

Let's see who's gonna be shorter here either you guys have a quick thing.

F

Yeah I just had a good thing. All.

A

Right: okay, we'll go with Himanshu.

F

Okay, so I'm, just working on this model, predict example uses there's 536 I just wanted to tell you that okay.

A

You're, just working on that.

F

A

F

A

Right, cool I, just I, can't remember what this is. So, oh, yes,.

F

A

Great perfect thanks yeah.

F

And just one thing: I was just going through the dog and if we go on the plugins, then the Edit on github button is anything I.

A

Just noticed that okay yeah, we got to figure out what to do about that. Let's see yeah cuz, all of those don't work yeah, okay, maybe we just! We probably just need to make an issue and leave it as an open issue. So Doc's Logans, yeah I, just found that out when I went and did this I was under the HTTP API and this doesn't work either, because this is a symlink, and so this just goes nowhere.

A

We may have to yeah. Who knows what we'll do about that? You know hub all right: cool Thanks, yeah.

A

All right, yeah how's gone yeah.

D

It's going good, can you look at the PIA that I made about the Java? The permission error is still.

A

Let's see I don't know if I saw this sometimes I go to notifications and I click, every single one, with control and open like 20 tabs with all of them and then I forget, which ones I've actually looked at and I actually haven't. Let's see, okay, I updated, you can print sufficient. Nobody talked about the dot okay. We checked him at this okay and then, let's see what's up here, okay, dependency, jxh and we still are hitting the permissions issue, damn all right. Okay! Well, let's see what is the check say.

A

Permission denied that's dumb, okay, so this is the way that I've been debugging this.

A

Whenever you send a new thing, what I do is go here and I pull down these guys so.

A

Let's see what happens, we're doing in a container.

B

B

Oops, we don't want that plus I, don't know.

A

All right, okay, tell my Java down let this guy.

D

It's like that we have to in terminal. We have to do extra permission for executing it by using chmod and something oh.

A

Because it's a zip file is that why, let's see it probably is.

D

You know mission using yeah.

A

That's pretty you're, probably I haven't I, don't know if I looked at this yet but I would guess you're. Probably right here. Have you tried that? Did you try that or.

D

Yeah I was trying with that, should I create and use of process, giving them permission using chmod and self yeah.

A

That's probably will I mean so basically, what I'm gonna do here is: will pull it down and we'll see, if that's the case and god damn Java.

A

Let's see yeah we'll see, if that's the case, and if it is the case, then yeah the sub process just calling chmod well, you don't actually need to do that. You can actually just do it with Python, let's see so chmod, not so good. Thank you very much.

A

A

Yeah great sweet, so this works so yeah. You can actually just use this function here, but let's see, let's see what happens because we may or may not need that, because I feel like I saw that it was executable, in which case there's a different problem. Yeah Java, it's I, feel like you, should it's much faster than this such download simile?

A

Okay! So let's just let me just fix this something.

A

Backlog I'm, just gonna, put this right on the backlog. Cuz I have no idea what to do about this. One.

A

No one will try. Someone said wants to take a stab at this great, but I yeah this one. This will take some tricky editing with editing with the.

D

In us about oh.

A

Yeah well, yeah I mean we're not we're not the smartest and so they've like there's I mean yeah. mmm How do I describe this? Well like everybody's staying home, which is good? Well, not everybody. Staying home. The thing is like a lot of people. Whoever can work from home is working from home now and but there's still like a lot of people out and about and like all the restaurants are open for takeout and delivery. I, don't know where you guys are.

A

You like I would be curious to hear what you guys after I tell you guys about this, but yeah. All the restaurants are open to take out on delivery and I I kind of I'm, not so sure about that. That seems kind of like isn't one food be I'm, not I'm, no I'm, no scientist, but it seems like when food be kind of a way to transmit that. But who knows right? I, guess not me, and so yeah. That's interesting and then like.

A

If you go to the home, improvement stores are all like completely full for parking lots every day.

A

I recently accidentally broke the toilet, and so I've had to go to a home improvement store several times to try to fix it and every time I go it's completely packed and like they're doing this thing, where they space people out in line like six feet from each other and then when they let them in the store they're, all just like roaming around in the store and not everybody's wearing masks like it still seems like not many people are wearing masks like since a lot of people just aren't taking it seriously still and Mike.

A

Well, you see the numbers or the effect of that you like yeah, that's amazing, yeah yeah, that's a bummer, however, how how is it going for you guys over there yeah.

D

Our country is in a complete, lockdown situation, so we can't get out of the house and sits the the precautions are taken. Yeah well.

A

It's such good.

D

Let's see how yeah.

A

If I guess like it could be good right, who knows, let's see unzip dependency check all right so now Allister, so they have dependency check been okay, wait! Kappa! Well, okay! So it's got the executable bit set on it, which is funny so sorry.

A

I can't spell the day, okay, so let me just finish this up and then we'll add those binaries yeah and just like yeah. If anybody needs to drop just go ahead and drop because I know we've gone we're going over and we'll probably gonna continue to go over today.

A

Okay, so now what I'm doing is I want to make sure that we add the correct path. So basically the same thing that you added to the path here, so export path equals P. Do we do J to K 14, slash bin and.

A

Then it calling path and then let's see Java.

A

Which okay good job is there, and then we want to do path, equals dependency, check, slash, Bend, oops, oops whoa. What happened sometimes I accidentally find out about github keyboard shortcuts I recently got recently recently accidentally found out that okay- and this is now in the path alright. So weird, okay, so looks like this is running here and then I guess. The next thing to be checking for would be a spelling mistake.

B

A

Dependency check.

A

That is odd. Okay, mmm permission denied dependency, checked, Sh.

A

Okay, this is going to be a weird one, great.

A

Yeah well, let's pull down this patch see what happens.

B

All right, let's see.

A

Okay, we got to pull down Alaska to.

A

A

Just download this guy see what happens all right so.

A

I've got this again.

A

And we were scanning.

A

Oh, maybe that's what happened.

A

Our mission tonight, oh this, might be what happened. Okay,.

A

Let's we'll see for sure if it was just.

A

Okay, yeah, it's gonna do the whole thing where it craps the ability, database and everything. So this might take a little bit.

A

A

Okay, you know here's what I think it is I think that.

A

What the hell! Okay! So it says, permission denied dependency, checked out: SH Oh.

A

Yes, okay, this is what it is all right. Okay, so the reason why it's doing this is because we said CW d equals package and package right now is pointing to our X Java, slash this directory. Well, the thing is when we downloaded our X Java.

A

Okay, it looks like there's some kind of problem with the JSON, but let's see.

A

Damn: okay, whatever okay, so we downloaded when we did this yeah when we did this wait a minute. ah This is really the problem. I just run this test here, so python concerns.

A

Give me a different error, which is good.

G

Test dependence check: okay, let's see if I can spell okay.

A

Thanks, oh wait. Let me do.

A

I gotta fix that deprecated testing command.

A

Logging equals debug come on now. Wasn't her logging.

A

G

What is it doing about.

A

To test a fancy check.

B

The example sure.

A

Test just a paycheck.

A

Oh, maybe it's downloading Java yeah, that's probably what it started.

A

Yeah, that's very downloading Java.

A

Let's smell so good yeah, it's pretty down like Java, no great okay, we gotta wait for it to challenge other or no. It was just loading. Oh my gosh, it was just running Python. Oh my gosh, all right.

E

And so after Java, which language should I focus, it I.

A

Think there was some stuff for Otsego, there's Java, there's I mean I, don't think we have a static here. Yeah PHP would be good, but I think we could do more with Java. Let's see.

D

These checker or framework for java whoa, oh.

A

There is, let's see, there's wait, there's what check.

D

Out framework, by which we can check here, how you been good type.

A

O check type checking mmm mmm type checking. Let's see, we need something that we can basically just.

A

You know we need something that we can just.

A

We need something for like the code itself sort of like bandit. You know that would be nice if we had something like that. Let's see this looks promising now this looks expensive.

A

What is this kid? Wait? A minute kids got a football helmet and he's carrying a basketball. This I don't know if I trust. This thing, let's see, let's see.

A

No 2016 no way.

A

This could be good.

B

A

Yeah, this could be good.

A

That will build a low okay. Well that doesn't really help I'd like set it up. mmm That's a little maven. Can we just run it though command line? Yeah? Oh, you actually have to compile the Java Sea. This is. The thing is about. C code is like C and C++ anything you have to compile. You have to like download all the dependencies and it becomes a giant mess, and so this may not be it's. Why, like it may or may not be good right? It's like reminds a few pre-configured, so I can also say.

A

Yeah, it's it's just it. It becomes like a giant mess to get all the dependencies set up and and that it's gonna be like its whole own project. Let's see you put this.

A

Let's see, we need some logging on there. That's this new new issue.

A

Add some logging add some feature: requests until net.

A

Add some progress, logging downloads.

A

A

Oh wait like that. We got them all. Okay,.

A

Just downloads all right, great, all right, okay, I know that path is correct. Okay, okay, so this one is working, damn what the hell is up with that.

A

That's really weird! Okay, so it's running on on in the docker container here.

D

Issuing this steam additive and I don't even know the cord yeah.

A

The same air is this: the JSON decode yeah, okay! Well, that's good news! So should I.

A

So this is where I would recommend I think we should probably do this for all of them. We should do this. Try try except block around here.

B

A

Yourself here yeah well just that self. If we add self, then we could access the locker.

B

Let's see yourself down lager at debug.

A

Let's try this again and see what happens? Oh really,.

A

What wait a minute.

A

Oh, that's why all right, whatever.

A

Yeah, we don't have access to self, because we're just running the function is its own time.

A

It should dump the logs first now note that yeah. This is weird, why the hell? Isn't it weird?

A

This town loves.

A

Unable to download metadata file.

A

Unable to retrieve okay well, this I think is.

A

Let's try it download it.

A

Now it exists all right, let's see.

A

Well, let's see.

A

This one okay, this is probably miss if ik to my machine I, would suggest that you add the same line with the print statement. Let's see.

A

So I would suggest that you add this here.

G

B

A

Okay um and then, as for this.

B

A

D

What the download.

E

Cache right, yeah.

A

It does work with the files cuz we got, I mean it ran here, err well, I guess we did put it in a path, so we prayed or mmm yeah. Actually we we put it in the path with that other one. So we should probably undo that so echo path.

A

You know I'm saying we downloaded it and we added it, so we pushed it on at it before we run that test case.

A

Aha, permission denied.

C

A

Do check? Thank you. Yes, good call.

A

Uh-Huh, yes, you're right, you are 100% correct for some reason. Oh you know it is it's probably because the Python sh-shoot ill unpack archive does not preserve permissions, whereas unzip does the unzip utility, the command-line utility, so you were, you were definitely you were on the right track there. So use definitely use this function here. So.

A

A

Or actually you could do it? Let's see.

A

All right here we go, Catholic has chmod.

A

A

Right above here, I would do something like whoops.

A

If you do this,.

B

Let's see check to this age, this should do the trick for you.

B

A

B

A

B

All right there we go.

A

Yeah all right, so this should do the trick here.

A

Okay, I got this syntax correct, so yeah. Let's try that and then we'll see what happens and then, when you scanned it manually, did you were you able? Did you see that there were 99 total Seabees or you just? Did you just put that there for now.

D

Yeah they have less than 99, but I put it as uh just so.

A

D

After they did show after it should make a edit yeah.

A

Okay and then make sure to change this type here than two, because you won't get the really.

G

A

Sweet yeah all right, I'm glad we figured that out good job. You were on to the right right. You had the right right idea there, let's see.

A

And then, let's see, see check issue.

B

A

Being related to sit, util unpack archive not preserve.

B

Permission bits.

A

A

Shh, alright great sweet, nice job. Thank.

E

A

Right anybody else still wake here. What do we owe hello.

G

This time to hear a has.

A

G

I was working on the source for passing the dot ini file format. Yes,.

B

G

I made some changes to it and the tests are still failing. That's so, can you please check it.

A

Don't worry about my chances.

A

Metadata waiting should I, really, though this happens. Sometimes ok, let's not worry about that.

A

Weird weird mess with the NPM API.

A

Okay, let's see.

B

Testing source 195, let's.

A

A

A

Let's see yeah, you want to do.

A

A

This one is because we need to get rid of this probably long term, but so, let's see I, think just cause some sequel light. Has this correct? Okay, yeah? You need this to.

A

Skip tag: yes, okay, so tags were this thing that we came up with it's like you could have like different versions of a Davis. Oh.

G

A

You could have different versions of a dataset within the source so like you might have like v1 or v2 like within the source itself, and but we we haven't implemented that for this. Obviously, so we need to skip that test. So, let's, let's see, oh no.

A

Yes, we need to add this.

A

And then we need to let's see what the hell is going on here.

A

A

Lines, oh man I closed that tattoo. All right, that's it try to be better about cleaning on my tabs lately. This is what gets me. Let's see.

B

So testing source.

A

A

Prediction target name value.

A

Okay, oh that's! What's going on yeah we're, not saving prediction from this one? Okay, mmm we should probably do this kind of how the CSV source does it mmm should. We are sure we just this.

A

A

A

So the reason why this is happening is because it's expecting the prediction to get saved and loaded obviously, and we're and we're obviously only saving the features right. So so we just modify the test case.

A

We can just write the test case or so so you can just copy paste from this test case and modify it to be applicable to.

A

To to this there I'm gonna try a hide some of the complexity here, because it's a tutorial.

A

mmm No, this will be good because we don't need that. We don't care about this so much. We don't care about this. So much.

A

A

Yeah, why don't you write tests update? Why don't you actually write it yourself? There like.

G

By myself, yeah.

A

Just write I mean just write a few tests for this basically right, and so this is going to be used for, because that was this one I mean we're intentionally making this simple right and so I think we can just make the decision not to save predictions here and so yeah. Let's just go ahead and say, replace, don't worry or don't involve.

A

File source test just write a couple: little tests with comments that test the basic functionality.

A

Let's see who you know and what might be nice for this is I just added this function recently, and this could be good so for the this actually might be very helpful for the test cases and making them more simplified.

A

So I just added this recently to the high-level API, but I thought there was something with HTTP API that made me think wow, you know, wouldn't it be nice if we could just like save records real quickly like just open a source, update it with these records and then dump like and then be done close the source. So I made this save helper method. That's uh that's!

A

In a high high level, API, and- and here it is it's just- this is all I such as so say, sync with source source of source contact for record in args, updated right, so it effectively saves all the all the records to that source. So you could just have the the like. You know you could use this function and then what we can do is we could make another function called load, and this might be is.

G

The loading of the new tenant yeah.

A

Yeah so and I mean basically, what you could do here is is because the thing is that what weight having this save and load function does, is it it? It hides the fact that we do that double context. Entry pattern, which for new people is very confusing, I, think.

A

But, and so you probably want to implement the load method so in another PR, can you go ahead and implement the load method, which is basically just gonna, be like you know you could say a weight load and then, if okay, let's write this down and okay, let's see.

A

So you'll write a few a couple tests.

A

Maybe even just one.

A

But first just like the new save function in high level, okay, so the new save function in high level.

A

Top you I will make writing the tests cleaner for.

A

Therefore, we should implement a load function in high level top UI as a separate PR once that is complete.

A

We'll merge it into master master and use it within the tests for the any source. Does that sound like a good plan or no.

G

A

Okay, if you got any ideas, just stop me. This just seems like it'll, probably make the tests look, pretty clean, let's see and then so, for the load function, if No. So if args is an empty array.

A

Then so the load function should be, it should be in a sink iterator record. So this is sort of the return. Type is a sink iterator record, which means.

A

So, which means the syntax for calling it will be like this I'm, so a sink for a record and load source now. So if you wanted all the records.

A

Source, it would be.

A

A

Hey this is how you'd get all of them and then.

A

So, let's see did it uh that means means you'll, want to call sourced up record and yield each one.

A

Unless star args, which which is stir, is not an empty list, then call source dot record for each string and the lists and yield the record. Does that make sense? Yeah.

G

A

Yeah so basically their load all the records, if no arguments are given. If the the list of optional arguments is basically the record strings, record record keys of strings and if those are given then just grab each one from the source and yield that, and hopefully this will make that that tutorial a little simpler and then so can I. Ask you to please: please create an issue for this, so that the pr for load can close the issue yeah.

G

Yeah, yes, I'm doing great.

A

That's just just for tracking purposes, because I've found like if we go here and we see like oops issues.

A

If you look at like the milestones and the closed, it's very helpful because then we can see like all of the issues that were I've noticed this recently, just in general for everyone but yeah like yeah, then we can see everything that we did so yeah if anybody just that goes for everybody like if you're happen, to drop a PR, that you realized that there's no issue for please make an issue and then we can have the PR close the issue just that way. Then we know what we did but yeah cool.

A

Is there anything else on this one? Then? Oh.

G

No all right, that's all I'm sweet.

A

Great all right is.

G

A

Anything else for today, yeah, thank you.

C

John yeah, uh we haven't oh.

A

We haven't talked oh I'm. Sorry, yes, sir how's, the construction, it's.

C

Very confusing still, but the thing is that, from the function from dick and mmk are not very clear and our throwing errors every time, I try something so also the CM dot. Firefight is the main file where everything starts. I can't write everything and go with it. I have to do it one by one. So it's taking a lot of time. Okay,.

A

Okay, you want do you want like. Is there anything specifically that we do you want to show your screen or something we can work through something, or do you have anything specific that you wanted me to elaborate on.

C

Like the data classes, field thing right.

H

C

Using that you're making an argon from the from declined, m'kay odd functions, I'm, not a vet, hear about that and how do I go about it.

A

Alright yeah: let's, let's figure it out there.

A

Okay, so oh and everyone this might, this might take a while and I know it's late there. So if you want to drop, feel free to drop but yeah we're gonna go through this now. So if you want to learn more about this stuff than then feel free to stay on.

A

Okay, so we want to talk about alright, so you want to see how we can take a field and use m'kay argon it right. Oh.

C

A

C

So here, like I, created a new class of hot date, a class CMD config and then and then I made a variable called dog with typed it and then I give it is equal to field and then everything default factor is equal to our log. Then the everything in the our contents and everything alright.

A

So can I see what you're doing actually I think that will probably be more helpful to get started here.

C

So, like I created a log variable in the CMD config file, Oh get potato floss.

H

A

C

A

So, okay yeah, so here's the thing.

A

The okay, so let's, let's jump back on the keep, keep it on the on the BS code.

A

Alright, so the field itself is going to get so. Okay, the you you don't need to use arc. Basically.

A

Now one of the things we haven't figured out is things like parse logging action so that that hasn't been that's not something that we figured out yet, but I don't really know if we need to do that in this case, this is gonna, come up more okay, so the parsing, the parse actions right now I think are limited to logging. And what are they? Let's check it out? Can you go and parsers parse it up? He?

A

Why I think we were able to clean this up pretty much yeah okay, so we've basically got this list action and then we've got this parse input, jack, parse inputs, action?

A

Okay. How should we do this? um Okay, so okay, so let me just react. Splain, the the connection between the fields and m'kay argh, real, quick.

A

When we use m'kay argh, what we're doing is we're taking we yeah, okay, so we're looking at a field and we're converting it into an argument right. So we never want a field to be an argument. We never like you know one of the fields in the class to equal, an argh, because we're gonna actually convert it into the ark and so.

C

A

Shouldn't be a dictionary, basically go back to your, where you had that definition and we'll and we'll work through how we define arc or log. So log is going to be of type, let's see. Well, it's of type impt technically, so, let's see actually okay, this one's a tricky one. Actually this is this brings us right to some tricky stuff.

A

Let's just let's look at. Let's look at one that already exists as an example. How about one of the models like SLR or something? Oh.

C

A

That was yeah. This one's got plenty of.

A

Yeah, okay, so let's scroll on up here, so yeah there's a lot of fields right: okay, so and each one is like okay, it's got the data type and then it's got the help description and then it might have a default value and let's see.

A

A

Okay and it might do okay so and then, if you go to m'kay argh you'll, see you'll, see how basically it looks at field, so it looks at field default to try to so it makes the argument, and it says the type is whatever the fields type is right and then it goes through, and it says: okay grep. If there's default value now we need to set the arcs default right ends and fields, don't allow for default values that are mutable so like lists and things so.

A

But but we don't care with arcs because we're not going to modify them the an arc never gets modified. It's just it well.

C

I mean sources, it gets modified ones to copy it into another loss. Oh yeah,.

A

C

More defiant, yeah.

A

Dom modified yeah, that's actually creating a new instance of it, so it never actually gets modified. It's actually copying it and modifying the copy. That's why there's that method? So.

A

Okay, yeah, so then do basically we set the default and then well. What we're doing here is we're taking we're, look, we're inspecting this field and we're trying to figure out. How do we build the correct parameters to that that arg parse, add argument method now.

A

And so that's that's what action is so action is basically it we can use if it's a boolean value then then store true, we'll end up doing a switch on on you know if you just specify it like a dash, for example, if you have some sort of command on the command line, you say, like you know, read, you know, read only write or read right right. Well, we don't need to say, read right space. True, we can just say read right and it will assume that it's true.

A

So that's what the the action is and then we go through here and now we're inspecting the type and when we inspect the type this is like oK. We've already got list action. Actually, that's good! That's good news! So, okay, so okay, action list action is that okay, so I, can you flip back to command real quick where you had that log? Okay, an action equals parse logging action. Okay, this is good. Actually this is good. Okay, so we've already got away.

A

Let's see okay, so that the the reason why this is hard is because, at the top level you have all of these, you have various commanders right and you need those parameters for that command line class to run right and then, as you go down, the extra config like we talked about earlier with the entry points like that, builds this giant nested structure that you saw when you were renaming art to plugin and so because yeah because what'll happen is like okay.

A

Well, if I choose a plug-in like if I choose one plug-in that I end up with configuration options for that plug-in, rather than the other one right and so I won't always have all the arguments there, because I didn't specify them for that plug-in right, like I only have the I only have what was given, because I have I've, given the options applicable to the plugins that I chose I, don't end up with the options that aren't applicable to the other plugins that I didn't specify if that sort of makes sense, but so also things like required are gonna, basically we're okay, so you're gonna need to look at this and you're gonna need to you're.

A

Gonna need to figure out how do I take the things that exist as args and modify either field? How do I modify field to pass them down to, let's see, go to field, go to the definition of field? Okay, so see how we're setting metadata description here. Yeah.

C

A

You can put whatever you want in metadata and then you can access it later in in m'kay argh right, because you'll see an MK argh, we add the help from the metadata description. Let's see.

A

Yeah, if description then help right, so anything that you're, having trouble figuring out like you're like okay, how do I like what is the? How do I, how do I pass this down to MK argh right? How do I take this thing that I can describe for typing information? You mostly need to figure out like okay. You need an appropriate data type for this thing, and then you need to figure out how does MK argh like for these list actions right so for like sources?

A

We're gonna need source to have this, we're probably going to need source to have this sync we're going to need to add this singleton attribute to source right and the singleton is going to be it's going to be the base base source right because each entry, the list the list is sources, but each entry in the list is a base. Source is derived from base source right.

A

So for that type of thing, you're going to need to work for the typing information you're going to need to work with like how do I extract what I need to from that data type. So if I set a field and I say sources, : like, for example, if we're replacing self like argh underscore sources, then what we would do is we would say in the config class we would say sources, : sources with the capital s equals field. And then you know, like maybe the description for that field, right yeah.

A

So let's do this one right now. So let's make a new above that right class, a config class.

A

You know sources config, yes, perfect, and then you know sources like lowercase sources.

A

Yeah do lowercase sources.

A

And then close sources, I doesn't matter, I mean we're just we're just mocking this up right now, right! Alright! So and you don't even need field here, right, I mean you could do it or ya do field and say so, sources for loading and saving right, because we're just going to copy it right and then we're gonna do default.

A

There actually are going to have a default factory here.

A

Default factory equals lambda and then I think it's just lambda : and then you return yeah yeah. This.

H

Guy has it.

A

A

Think it's lambda : and then so don't do it in the array. Just copy paste from default equals sources down here, just copy paste until the next comma. There. Yes, oh, very well, just sources! You just want sources, so sources open, paren, yeah, so delete that action and type yet delete those two lines and then delete that delete that paren tour delete that yeah delete, Bacchae and then delete that bracket there and then delete default equals. So you just have sources.

F

A

Yeah and then delete that comma leave that guy yeah perfect. So this is what it looks like now right. So basically, what you're saying is sources is of type sources and it equals this field where the description is sources for saving and loading default. Factory is going to. Let you create this default, and the default value is sources where the first entry is this json source right.

A

Does that? Okay? Does that make sense, yeah.

C

So what about these guys yeah.

A

Okay, so now now, what's going on here is okay, and this is going to be. This is going to be now a little bit different too, so we're gonna I think we're gonna need a new thing that we're gonna passed a field that we're going to put in the metadata field or that we're gonna put in metadata. We're going to have we're gonna. Do let's say so right now, if you flip back over, so look first, look at type, you see type equals based, sort out, load, labeled, right and action equals list action sources right.

A

So if you flip back to MK argh.

A

So here we see okay, look when we see something, that's a that's. A list type were like okay, we're gonna, add this n are x, equals plus, which is what we saw on previously and we're gonna say the action equals list action whatever the type is that will create. That will give us that list action sources just like we had before, and then we need to set the type equals field, type, dot, singleton and so go to features real, quick and see. What is in features features differ from null search feature, slash feature.

A

No, not there, that's the plugins.

A

Yeah feature so I should get you okay here so scroll down. Damn we really still haven't gotten rid of that. Yet, okay.

B

A

Okay, just keep scrolling all the way to features.

A

Keep going keep going.

A

Features here we go singleton feature so collections that user list and says singleton equals feature alright. So what we're gonna need to do is we're gonna need to make it so that sources is kind of like features here now, okay, so the next question is so what we want to do is we want to go into sources and we want to say singleton equals base source so go into DL search, source, slash source.

A

And we want to get right next right under context, their context equals sources context singleton equals.

A

Base source right, so this is saying what we're doing is I, don't know if single single single tonight have been the right choice of words here, but it works. It's basically like what is one of these things. You know if the, if the plural is sources. What is the singular right? So? Well, it's a base source right. So if we have, we have multiple sources. But what is each thing in here well check the singleton property of the class okay.

A

Each thing in here is going to be a base source and that's what this property is telling us right. So now go back to where we've got m'kay, argh, okay and, let's see here so fill the type. That's singleton. Okay. So if has at her type load, then type dot load, okay. So this is how this is how okay? This is. How now this is a good question here.

A

So, okay, so we we are gonna need some way to say, load versus load, labelled right, and so this is probably a property that we're going to shove into metadata right because for some things like feature, we want to use dot, load right or like models. We want to use dot load most of the time right.

G

A

It just so happens that we haven't run into the case, yet we're gonna run into it. Now that I just added the model stuff to the CLI command for starting the HTTP server, I did modest with load labeled right, but right now, we've just done models without load and sources is the only thing that's had load labeled, which is created. You know this whole problem where we've got sources are different than everything else right now right, so we need some way to specify. Probably I would say you should add something to so this this.

A

If statement here, you want to add another one after that. That says, or maybe another one before that. That says, if has a sure, pipe load, labeled and.

A

Labeled in field metadata, then.

F

A

And labeled so kind of like if description and filled out metadata or you could do load labeled has a underscore in between load and labels. I believe yeah, ok and then you probably want to do if field metadata dot get.

A

Weight, sorry, if has at her argh type, so if the type has so you see the syntax for the top one. Basically, it's checking okay. What.

B

Is my value yeah.

A

I know you're good.

A

Yeah so then, and and then USA and.

A

Filled up metadata, dot get.

A

So get is right, metadata is a dict and if we use get then we can have it. We can specify default. So let's say get labeled.

A

So yeah in quotes labeled, comma, false.

A

Right, so what this will do is it'll basically say: okay, if the type has a load, labeled function and metadata, has the keyword labeled set to true, because it'll default to false. If it doesn't exist, then we know that we're supposed to be setting our type equals r dot, so we're the line above this line where we basically say: okay, if the type has the load method, then we're gonna use the load method and what that's going to do is just like when you specify a model on the command line. It's not!

A

Actually, you know if, if we said type equals model, it would then pass the string. That is the model name to the model right error, two to moded the class. That is the model or that is the base class for models right, because that's the way, if you look at our person type and the way it works, it basically says: okay, if I add an argument that, if I specify the type for this argument, I'm going to take the string value of whatever was given and I'm just gonna call the function right.

A

So if I say type? U equals int, it's just gonna pass the string value of two to the int built-in function and it's gonna convert. You know the string value of two to an int right, so we don't want that. We want it to call load or load labeled right because we have a string value and the string value isn't the thing that should get passed to base source or model. The string value is the name of the plugin, which is a model or a base source right. Does that make sense? Yeah.

C

A

And in the case where we have labeled equals true, then we have something where it's like. You know my source equals CSV right, in which case you know we're seen in ahead of time here right we're going to tell we're going to we're going to tell the.

A

We're telling MKR that we need the user to to load to load labeled sources here right like or we need the user to load labeled models here right if they specify labeled equals true within field right, so you're going to need to modify field two to accept a labeled keyword and if it does see the label keyword it it has it it pushes it down into the metadata property right and so, but here basically you'll just have copy of the line above and make it you know, are type equals, are type load and labeled, and actually this should go this.

A

If statement should go above the previous, if statement, because there else we're gonna end up, never seeing it, because if it has a property of load, then then it will yeah sweet all right so now, obviously this wasn't everything and for logging, you're gonna have to figure out I, don't know if there's a logging level type, let's see again.

A

A

Logging levels, yeah logging level, is actually.

A

It's an int value, basically, so what you're gonna need? Do you see? Sort of the Pat like this is. This is why this is not easy. So, but you see sort of the pattern here, hopefully like you basically are going.

B

A

You see the pattern right, so if you don't have access to something, add it to the metadata and then for action. I would say: action is another one of those things where okay right now we're just inferring that if this is a list type, we set the action well if action is set, if you pass action to field propagate action to field metadata and then use action within MK arc right, okay,.

C

Like if it's in the metadata, then you modifying them and go yeah.

A

So well, and- and this will be on a case-by-case basis right so, for example, if you see here right this, if statement the the if statement is saying the L: if inspect is class field type right and then we say we say: okay, this is a class, and if it's a list, then we know we're going to set action. Equals list action.

A

Well, you're gonna probably want something else around here to say, like you know, maybe at the end even like around, if description and field metadata, you could say something like if action in field metadata well, then Arg action equals field, metadata action right, so kind of just work with your you're gonna have to do a lot of a lot of a lot of massaging and figuring out like okay, like maybe okay, if I don't have.

A

If I don't know about this right now, how do I, let myself know about it right if I, if MKR, doesn't know about what I should do with this, then how do I add it to metadata or something right? And you basically just do this- you keep you keep knocking through. I would say: try to make these committees config classes for as many things as you can right, just keep going through and like making config classes for all of these command line, command line. Okay,.

C

A

All of them right, and so the thing is the reason why it works. The way it does is because the reason why works the way it does is because we had this like it gave the ability to subclass really easily and then all sudden, you know it's like okay. Well, this command takes a bunch of sources and a bunch of models and well then it's a machine learning commands, and now we just like create a what is I can't think of the goddamn word for it. Now it's like polymorphic or something I, don't know.

A

C

When you have like.

A

C

Can you repeat what you just said: you'll avoid oh okay,.

A

Sorry yeah so basically I mean the reason why we did it this way initially was because you could you could easily, like you know you could you could say okay. This is just like. If you look in commands right, you'll see that this is a sources command and this is a features command and and then all the sudden, you can start like having multiple parent classes. I think it's called polymorphism right.

A

It's been a long time, but when you have multiple base classes that you're inheriting from and so there you're gonna run into a lot of that which so and basically what I would recommend. Is you you just just like pull everything out at this point like like?

A

If you see these command classes with multiple base classes, because they're all they wanted to pull in arcs from just different places, just make one big at config class for it, because that'll give you more visibility and then you can start having the a config classes derive from each other later right, but basically just start making these a config classes like.

C

Like this, something like this whoa.

H

A

Did you pull out every single art just to check what they all were? Yes, nice yeah, so, but basically yeah I mean this is this is an extreme example right but but like, for example, go to go to ml, go to DF of ml. That's slash. Cli is / ml WI.

A

So ml command, right, checkout ml command.

A

Yeah, so ml command is model command and sources command right and so all of a sudden it got art model and argh sources right, and then you get self model and self that sources right. So here all of these are ml commands, so basically an ml command, it's model command and sources command. So what you're going to need to do is just this. This one actually ended up being easy right. What you're going to need to do is you're going to need to make an ad config. That's like at ml commands are big.

C

A

Gonna need to make a config, that's like at ml command, config, and what I'm talking about here is because ml, the one you're replacing is something that had model command and sources command where you're gonna need to do is you're gonna need to go. Look in model command. Grab that arg model pull it out, convert it to the new format. Then you need to go.

A

Look and sources command, pull out arc sources, convert it to new format, which is what we just did and then you're putting them now, both in this ml command, config, right and so all I'm telling you is that, like you're gonna run into these ones, these commands, where they have multiple base classes, and so you're gonna need to look in every base class and grab all their arguments and pull them into. You know their own config class. Yes,.

C

So I also wanted to us like use you're, saying that make a new config place for every command last yeah. So.

A

Not not for every one for every unique one right, whatever.

C

You need yeah, oh, that will not do decrease the code that much and will not look like the unifying stuff is happening well.

A

The unifying stuff is not about decreasing the the code base here. It's about the fact that were we're going to it's it's about conforming, the codebase all to the same standards right because right now we have one way of configuring. Every single other class and one way of configuring command line stuff so net on this whole thing is probably going to be like equal lines at it and deleted right.

A

Okay, so this is yeah. This is no I mean this is no small undertaking. This is like a good job doing this. It seems like you're figuring it out, so this I mean then this is why I like I've had so many things to do, and this is psycho Wow like this is going to be a big one, so I'm glad that you wanted to tackle this so yeah. So this basically, this is like this looks good for sources command. I mean this syntax.

A

Honestly I think that that sources might need to be able to on the same line as that :, but I'm, not sure, but I think do you get the do you get the picture? What needs to happen here, not not that sources, the second one under default factory I mean it's it'll, throw a syntaxerror you if it's wrong but yeah. So do you get you get what's going on here then.

C

A

Clear right, yeah, no, this is I mean this is a big big, big project. So.

C

Everything when Levin I can't do it on at once, yeah.

A

Exactly tackle everything one by one and make a commit right so so try something now make it like go. Go, find a class with very few arguments. Try to convert that one and then once you've got that one working move to the next one right and every time you've got one working, make a commit and get right or even when you don't have one working make commits just make lots of commits and get.

C

A

Yeah and the log one it's I mean it should be kind.

C

A

Log one should be well okay and here's. The problem is that everything takes log, so you have to sort of figure how log first yeah.

C

Because then we will know what is yeah.

A

So you need to do okay for log, we can work through log, so log is of type int. If you look at at the Python logging library, you'll see that it's like.

C

A

C

A

Yeah right and so yeah, let's just you- can grab that that link there.

A

So it is this is that the type is int right, but the default is, you know one of these predefined values and then the action like we talked about with action. The action is.

A

So parts logging action right, so action is parse logging action. So if you say like we talked about propagate action through field, allow setting of action equals whatever and then in def field within base. Say: okay, if you see action as one of the keyword arguments put it in metadata right and then you should be able to replicate log.

C

A

Basically, at the bottom, where you have, if description, equals field metadata, do the same thing for action.

A

So at the bottom of this function,.

C

Of setting it is equal to all like this one.

A

Yeah, don't do that yeah, because.

C

Some like few loops, like yeah.

F

C

Then yeah yeah.

A

C

The thing was working for it, it.

A

Was well that's good yeah, so that's that's good that you got it working, but we we don't want to set it equal to our right.

G

A

We're converting field into args, so what you want to do is you want to iterate over the fields and then just call MK argh right, and you want to modify MK argh Intel. It gives you the correct argument right that you would have specified there right. So you can put required, and stuff is probably something that you want to pass down it to metadata as well. Right I was.

C

Just very hesitant because I thought I wouldn't change code, much ya.

C

A

Yeah yeah, you have, you, have the go ahead. This is going to be a change things everywhere, sort of project. This is gonna, be a big one, so yeah yeah. This is gonna, be good. Stuff. I think you know monthly. Once you figure out like I, think we got sources should be pretty much figured out and then, once you add this action action part to metadata and field and stuff, and then you add required you're gonna be pretty much well on the way there now.

A

The other thing that you're gonna need to think about is that scroll up real quick here? So, let's scroll, yeah, add subs here, yeah so add argument, method, name, okay, so the thing is what's gonna happen now is. Is this is a for loop over write this for loop, you're gonna have another for loop at the same indentation level of that one and oh.

B

Yeah, that's where you already have it.

A

Yeah, perfect yeah, that's where you're going over the fields, yeah and so you're gonna do you're. Gonna do inspect get members yeah, all you actually actually, instead of inspected members, add from config just go just do for field in data classes. That fields add from da config.

C

A

Derick add from da config.

C

A

So this is, but this is so use this. Instead, this just says iterate over all the fields right and then the first thing you should do is Arg equals MK Arg field.

A

Actually, you you, you really don't even need to do this. What you can do is okay, and this is there's another part here too right. So our geek m'kay argh field.

A

Right, okay, so now you've got another problem. Well, more problems all right, so we also have to deal with positional arguments right. So positional arguments is something that you're also going to want to propagate into the metadata right.

A

So we had talked about this briefly at one point, and so the record, the recording is definitely going to be your friend here of this meeting.

C

A

Good I'm glad it's helpful, so let me make sure it's still recording. Yes thank God, okay, so you're gonna want to propagate position into here as well, and you're gonna want to go through and probably add all these arguments to some list and then you're, gonna, say sort by position and any ones that don't have a position you're going to.

A

So if, if position is specified right position will be some need to be some sort of integer value right, that you've passed a field and then it gets put in metadata and then when you're looping through in parser, you're gonna want to say, okay, add all these arguments to a list right now, sort out now create a new list.

A

That's all the arguments with a position field sorted by position now go through and do add argument for all of the ones with a or way go through an add argument for all of the ones without a position field and put a dash in front of their name right, and so you know like so anything that is not a positional argument will be like log is not a positional argument, so it won't. You won't set position when you call field you'll, just do like we've done right and or like sources is like sources.

A

Is another positional argument? It's an argument that has a dash in front of it. So basically, unless you see position in the metadata, then just hat on the first pass add all the arguments that don't have a position and put a dash in front of their name, because the first argument, too self dot, add arguments, II method, name, the it's going to be field, dot name but field. Dot name will need to have a like a prepended to it, because their else art purse is going to think it's a positional argument.

A

So your first loop, through you, add all the ones that don't have positional. Then you take all the ones that have a positional field in metadata. You sort them based on that integer value and then you add them appropriately in another loop and you don't put a hyphen in front of them so that art purse knows that they don't need a.

C

A

Basically, you're gonna look so R equals m'kay argh. If you don't see so look in field dot metadata and if you don't see position do self dot. Attica argument: if you do see position, don't do self dot. Add argument instead append this argument to a list and and maybe end a position as well or something okay, yeah yeah. Now you get it right or like a dict right and then it's zero through whatever just make sure you get you that first pass.

A

Add everything: that's not a positional and put a hyphen in front of it, because this argument.

C

A

Yeah exactly right, source file, name right, so so well that one actually is not that that's one of the ones that gets parsed in extra args, but like model and sources right they are they're required and, and they are not positional right- you can put them wherever you want. If you look at like the merge command or you know, the merge command I know is a good example of this, but that is that has a positional argument in it, so yeah Destin, SRC and right now this is flow around honestly.

A

I really wish they were the other way and now we're finally gonna be able to specify that so so, basically yeah here, if you converted this merged command, you would have guessed : source or a desk : base source. Let's convert this one right now.

A

So yeah give me your best shot here. Let me, let me see what you're thinking.

C

So, like the, what will be the kimono here over.

A

For log, it's int right, good, look at sources, real, quick and you remember where we did sources you'll. That's the that's a good example right now,.

C

A

Right so see we have the what we want to call it the datatype and then feel so base source equals field sources to merge records into, and then in this case. Currently this is it's really dumb and it's got dust is first and I hate that, and you don't need default factory on this one because there's no default, you want position, equals zero right or no position equals one in this case, and then you just copy-paste this and you do SRC and in that case position equals 0 right.

A

So then, what that means it's on the command line. If I say D, FML merge.

A

A

Like file equals CS b, DB or my sequel equals my sequel. That was that.

H

A

There should be an example for merge.

A

Yeah there you go now, you can see see this is this is what's going on, wait a minutes. Dataflow merge, no there's no command line, it syntax or way merge.

C

A

We can add right right now.

C

A

It is kind of confusing, so this I'll paste it in right now. What this in Texas.

A

Also one second.

A

All right, that's! Okay! There we go! That's what it looks like right now: I'm gonna get her chat.

C

A

Which is why it's annoying, because really destination should be after source, which is why it's not documented, because it's dumb right now, but now what we'll be able to do is say position under SRC will do yeah we'll do position equals zero, and when you add argument, you make sure you sort the list of arguments. So you what you went through this list. This is perfect yeah. You went through this list in in what is it command?

A

You go through the list of fields right and you see. Okay, both these have a position. Well then, I'm gonna add them to a list of things with the position right. Add everything: that's not that doesn't have a position just do partial to do self dot, Agra or that argument and put a hyphen in front of it. Everything that does have a position. Add it to a list, sort the list and then add it.

A

You add it to a list, while you're in this data classes, dot, field loop and then once you're done, you sort the list based on position and you.

A

You sort the list based on position and you, then you have another loop and you just add them all in the correct position or order.

C

Okay, I get it.

A

Cool all right and yeah so well. This was this was an adventure into the land of configuration person. Yes,.

C

I can do things like change field and prom date and and God for all.

A

Right, sweet, nice, stuff- and let me know I mean this- is like this is sort of like I, think I think that we covered everything you're going to need here, but we probably probably left something out right so and for example, on these labeled. It's going to be need to need to be set to true for this merge command.

C

A

So yep but yeah, so that's a that is that's the gist of this the big hour and some just of this yeah. That's it! So it's a lot to take in so don't.

C

Watch the meeting again in the morning.

A

Cool yeah thanks for staying up late.

C

Also, like I, wanted to talk. If there is time I want to talk about operations, what.

A

C

I want to add an operation to see how operations work like for the image scaling for Amnesty, like we are hard-coding into 28 by 28 right, yeah,.

A

C

When I had an operation image scaling so that we don't have to hard code, it yeah.

A

C

Can do it with the daughter, mr. PNG files, yeah.

A

So so, okay, so remember how we talked about the pre-processing source, though oh.

C

Yes, yes, okay,.

A

So part of that was, we were gonna, add some new syntax to be able to to specify like we had that's what we have. This whole plan laid out where we talked about like okay.

A

How are we gonna specify in the command line to call this operation and pass these arguments well, unifying this configs stuff is going to help us do that easier, because it's basically this is going to be the first step in that too, so in in getting rid of the amnesty mga and just making it PNG so because, or else what's gonna happen is we're. Gonna have multiple places where we're gonna need to change.

A

Config stuff, and once you finish this, then everything is going to be sent like in its everything's gonna, be one thing so I would say like you could go and make the operation. If you wanted to and write some test cases.

C

A

Yeah, so you could make the operation and write some test cases for it, and that would be good. That would be a good thing to do. I would say: go for that if you want to but but let hold off on doing pre-processing source and tell you finished, you don't find their config.

A

A

Alright, anybody got anything else.

E

Yeah, can you hear me yeah.

A

How's gone yeah.

E

So I tried the container thing and it worked alright.

A

Great, that's awesome. Yeah.

E

And I also want you to talk about something.

A

Yeah go for it.

E

Yeah I know it's been a long call. So if you suggest I should share an example- and we could do this over at the group.

A

Yeah sure thing.

E

Let me just preview on it: I'm using different two different ways to input stuff into the data flows actually I'm using multiple operations, so I'm not sure how exactly to give them input I mean they all require input. So it's kind of messy.

A

Can you post the code up yeah.

E

I'm posting, you agree.

A

So yeah, if you post the code, then we can probably review this offline because I think yeah, it's probably about time. We we all called it on this. Today's meeting.

C

A

Sorry to push this to the end, but maybe we should end up actually we might, we might consider I, don't know it. Would you be helpful to have two meetings during the week and cut the time because we keep going over? What are you guys? Do you guys think that that would be helpful or.

C

Speeding on Saturday would be Saturday.

F

Yeah yeah I think before we can really should have one before.

A

The weekend, okay, like okay like Friday morning,.

F

A

Wait and okay yeah: let's do Friday mornings, then, because I think I have availability.

C

Surgery Friday night photo say.

A

Yeah yeah: well, what works better? There.

C

Night olds, Meta Knight.

A

Works better so my Friday morning, yeah.

F

Yeah it movie your friend in the morning and our Friday night, okay,.

A

Okay, that works for me, yeah I, that that that will work all right cool. Well, let's set that up then, and we can start and we can.

A

We can just put that on the calendar and then people can just choose which meeting they want to go to and and we'll try to cut the meeting off at like fifteen after ten or ten for me so 15 after the whatever the end time is for you guys on Tuesdays and Fridays, so that that you know basically it's like okay, if you haven't, if you haven't gotten whatever then come to the next meeting, because I think we've been, we've been consistently running wrong long for a few months now.

A

So all right we'll do that, then all right well great! Thank you guys very much and hope you guys get some good sleep and I will talk to you next week or on Friday. Thank you. Thank you guys go on. Thank you.