DFFML Weekly Sync, 18 Feb 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: 2022-02-18: Weekly Sync Part 2

Description

Meeting Minutes: https://docs.google.com/document/d/1vKYEPtqKiwsFwhVKPmPub5ebMqN9HteBcbdFAuTXalM/edit#heading=h.bucu7l213uqd

A

All right so we're racing youtube because the hard drive is filling up so youtube has to upload. So I can delete the last part so that we can continue here. Let me leave that file window open, so I can delete it as soon as possible. All right.

A

Okay, got it selected ready to go. Okay, um I was gonna say remind me, but there's nobody here to remind me. So um we'll just hope. I remember all right: um let's bring this back up all right, so we're running the data flow and for some dumb reason it says we don't have the thing that we just added to the other file.

A

I will prove to you python. Do we added that.

A

A

Oh, that was asynch helper.

A

There, it is it's right there, it's right there. We must be on the wrong version of the package. How did that happen?

A

How did that happen? Dfml version.

A

So this command takes way too long to run everything takes way too long to run, probably because of this hard drive situation ah see. This is why we're doing this in version, because this is just unacceptable and you can also do are we are we recording right? You can also do damn it wow. This will take forever, though, to list the help, because of my hard drive.

A

Wow, okay and it's really slow all right, no errors.

A

All right and now this will just skip if it ever completes all right, so we installed the run. This is a very common problem, very common problem when you're working with package development and the reason why is here, this is the culprit. So all the plugins.

A

This makes testing very complicated, um so all the plugins um they all need the main package right. This is this is part of the second party stuff. This part I mean it's the the whole the whole situation right. Okay, so now you can see that it basically just said not insta or error, um so we need we need. We need okay.

A

This is this is part of why the second party plug-in thing is so complicated right. So in here it says I require dfml right now, but we installed the fml and now it doesn't work right so because we wanted to keep the development version installed right and so there's like I mean this splits out into several different use.

A

There's several different use cases, but mainly it's like I'm working on the package, I'm deploying the package right and then we split into our you know I'm testing the package um versus and, and that becomes that second party plug-in the second part of plug-in stuff is all about ci, basically and trying to make make all the ci systems work together right and the way that we do. That is we're going to define a bunch of data flows to do the ci and then we're going to execute them from from every uh plugins.

A

You know github actions right because they provide free compute, uh we're also going to look at like. Can we just read in a github actions? Workflow represent it as a data flow structure right and then that way we have interoperability there. You can run github actions workflows wherever you want um so part of this, when we do that, we need to be able to declare dependencies and where they're coming from right. So in this case we're declaring the dependency, but we have an alternate what it was it execution environment.

A

I think we called it the other day working on an example, same branch, execution environments. This is this is the commit that we just pulled down, and I was worried about the re, merge conflicts or rebased conflicts right. So this this commit right here.

A

So local yeah, so this is the example- is local versus production data flow orchestration.

A

So, basically- and this is I'm going to plop- I'm going to have an example up quickly here, where we're going to post process some results in pi test and submit for local or remote data flow execution to do um post-processing right, arbitrary process processing- um and you know basically the reason we have the split here is because we'll either say hey, you know kick off to the remote post processing, which will then show how we run the http service to do the same data flow as local post processing, and that way we can share the code base very easily.

A

Okay, um let's go ahead and commit that stuff. um Let's first, let's pick fix our packaging problem.

A

Okay, so look at this all right, so this is crap.

A

I can choose all right, okay, this is going to be great, so we're going to just grab all the ones that say site packages here right, because these are all the ones we don't want here. This one is nice. This is a perfect example here right. So this is what we want. We want it installed in development mode where it'll show us the get repo hash and then whether the gate repo is clean or dirty right.

A

So, if the gate repo is dirty, that means we have unstaged changes um right so get status, so we have modified files. So to get repo is dirty, um and we would like to know that right because we want to know if we're listing all the packages. What what am I using right? Okay, I'm using a development version, but um you know is: uh is it the is it is it you know? Is it just checked out or is it um is it? uh You know.

A

Checked out and edited okay, so pip python m, pip, uninstall.

A

A

Right, let me let me look so so we basically said you know grab all the packages show us their versions. uh We want to grab the site package or yeah. No, we do want to grab site packages. So basically tell me anyone that's installed in site packages by filtering out any any lines in this that don't involve site packages and then grab the first first delimited by spaces grab the first column that was output right, so this would be column one. This would be column two. This would be column two. This would be column three.

A

This would be column three. This would be column four, five, six, seven, okay, perfect! So then we ran that uninstall command. We should throw this somewhere. We should put this in the.

A

We should put this in the.

A

All right, so we should put this in the uh documentation.

A

Docs contributing.

A

Devon, bye.

A

Oh jesus, oh hey,.

A

Hey how's it going.

A

Hey all right, so how are you, how are you, what are you up to.

A

A

A

um Okay, great, um let's see, let me me make sure we get you on the recording here, so I've got a recording going. Oh, that reminds me. I have to go, cancel the other recording here. So um how do I move this over? Oh crop 700, all right, 2700, okay, I'm having to throw random numbers into the um video recorder software to try to get both of us on the recording, okay right, 500.

A

400: okay, perfect, okay and bottom, we'll say: 12.

A

Okay, there we go perfect all right, so I think we got and it's doing that live wow, that's cool! Okay, um so we you want to talk about the portland operations.

A

Okay, um I think we yeah we'll talk about that, so I'm just wrapping up something um here. Let me just save this, so what I was doing is, if you install the package in uh so if you go to do your like development setup right on the project um and you install uh you, you install a package in development mode.

A

um You need to uninstall the package if you've installed it installed it previously without the dash eflag right, so so we're just covering that in the documentation and covering how to find the installed plug-ins, as well as the main package here and just delete them all at once, to uninstall all packages of dfml not installed in development mode.

A

A

All right, we're gonna, add that dollars contributing.

A

Show and saw packages all right and okay, so and then the next thing we just did was we wrote this. um We wrote this, which is a little helper function for test cases that will allow us to quickly do unit tests, and this is going to be relevant for your portland data operations.

A

A

A

um And so what this will do is it takes a dictionary object. It takes a dataflow object and a dictionary object as the first and second parameters, and um let me think in this case add what is this called.

A

Okay, all right, so it takes a data flow and and a check object and we'll look at that right now.

A

Okay, oh, but for some reason it's not showing up okay, see if I'll watch the name too.

A

Okay, uh wait! That was wrong.

A

All right, so high level data flow run, test operations, don't watch an idea, oh man. I wonder if this thing doesn't work here: um okay,.

A

um All right, this might not work like this all right, we're just gonna put it I'll, just put it in that in the test, we're just in okay, um we'll drop that last comment.

A

Okay, so I tried to add this method to this class, but um we're just gonna have to have it as like a standalone thing, because.

A

Yeah, because it's not working so um because there it creates a circular dependency issue, trying to go into this high level um and high level is where we define the run function, which is responsible for for running the. uh You know, running the operations within the data flow. So um so, therefore, let's run checkouts. Okay,.

A

Oh, that's right. I forgot to reinstall.

A

All right so now we should have a working uh development setup again um I had a installed one of the plugins and it overwrote the main package. um So all right. So let's take a look at this issue here, while we wait for this.

A

A

It's not letting me type the stock. I think the stock has gotten too long all right. um So let's go find that issue.

A

Once this uploaded okay, great.

A

A

All right, so you gone over the um have you gone over the uh should I tutorial.

A

Yeah, so you did you uh play with doing the next one, which is no okay, um okay, so this is what we're.

A

um Yes, yeah so well: no, not actually, sorry, not a data source, we're going to create operations and we're going to use those operations.

A

So if we look at the architecture, so this this falls into the top block on the architecture diagram here, so we're gonna we're gonna write some operations which are going to go through the directed graph execution, the data flow execution to generate the data set which we'll then put in a data set storage abstraction which would be a source.

A

um So first, we basically just need to go write a bunch of functions to collect the data and a good example of this is the ice cream sales demo. um Let's see, I forgot, I should link this one, so we'll look at this one, um but this is a really good example too. So take a look at this and I'll put it in the um this is probably even closer to what this is. Is the ice.

A

It available it it downloads from this web, these websites, um the population data and the um you know what with the temperature data um and it then uh so. This actually.

A

Yeah, this actually uses csv files as a source. It will download them and then load from the csv file and then report that data out you know of of the individual operation. So it's just it's doing a lookup right and then you're you can choose now. Now that you have this approach, you can choose to feed that into you know it allows you to mix and match and combine your data sets very very quickly um and that's actually what we're doing right right now. So um what we're doing right now is.

A

We were looking at the automating classification demo um and we were looking at this uh combination of the should I demo, which is, in the top right hand, corner with the automated classification uh example and how we could take the data flow over here and combine it with a part of the data flow from the automating classification example, and so we're going to extend that concept to check git repos for presence of github actions, workflow files, and so what you're going to see here is we're going to, um let's see so we're going to run this uh we're going to run the data flow, which right now is just okay.

A

Yes, this has to be so we're going to run the data flow using this new little helper function that we created and all the helper function does is say um for these input. This doesn't even highlight. Does it no so for these inputs here right? So we create the input objects for each one.

A

um You know I want the value to be whatever we set the value to right and, and here we're taking we're, creating a object which is representative of a gate, repo and it just needs a directory um so that we can know where the files in the git repo are and then we go and we run um tests uh or no, we run search operations.

A

So then we go and we run this, which all this does is check. You know. Actually I don't need a bool on this because I was gonna do r glove and then it didn't end up doing our glob, because we can just check for the presence.

A

So so basically this you know this grabs, the repo that you pass it and it looks at the directory property which we set to the root of our repo for testing, um and then it just checks for the presence to see if this this workflows is a directory, um so this is analogous to um so.

A

The safety check operations right and this these are all operations right um and in the ice cream sales demo, you know looking up the population given the city and the state. That's an operation and looking at the temperatures in operation right, so we can take those operations and we can you know feed them a bunch of inputs and then by feeding them a bunch of inputs.

A

We we collect their outputs and we treat that as the data set um and that way our data set is dynamic um instead of just being static right and we can save it somewhere static if we want. um So what we're going to do is we're going to so right now we're just showing this and then we'll talk about the portland data, so right now we're just showing this because you'll you'll see what how how it gets gets run here. So this would be.

A

You know this is very close to what you would be doing um to uh work on that like this is. This is almost exactly what you would be doing when you're working on this issue right um so at test run on repos, okay, so.

A

Okay, so we want a new data flow here.

A

A

Or I think we can just do this yeah, so we're gonna say uh clone or wait. Did I copy it? No.

A

All right so and- and this part you're you may not get into as much um so we want to clone the get repo, we'll probably do the check, if valid, but the the idea behind how you're gonna you know, uh write the test cases and instantiate the data flow. That's all that's all the same here. um Let me just see gonna clear this.

A

Okay, great had to clear the recycle bin all right, or else we're gonna run out of disk space here, all right, so we're gonna, okay, so yeah, so uh we're gonna, download the gear. Put we're gonna, make sure it's a valid download link for a repo, we're gonna, download it and then we're gonna clean it up when we're done um before we exit um and then instead of feeding. You know directly.

A

This repo object um we're going to uh we're going to feed the url to the repo. um So or you can say you know clone getrepo.opt.inputs url.

A

A

Right so we're going to give it a url.

A

Oh, you know we're actually going to do a smaller repo because our repo is kind of large okay. So, let's see what happens all right so now, all we did was basically say check if I'll get.

A

Upper feature get dot, features.

A

Operations to the right path, so all we did was import. The operations add them to the data flow operation gives as a positional and in.

A

Oh because it's oh yeah, we don't want to do that. um Okay, we'll just import these and it'll. Do it for us all right! So there's a this op imps in using sys modules names. This module's name is how you get it's the same as doing globals. If you want all of the all of the um objects within the global namespace of a file, this tells you now.

A

I prefer this over globals, because I think globals isn't something you can modify. So if you need to exam, for example, dynamically add a variable to the global namespace of a file. You could have like a list of strings and you could iterate over the list of strings, and you could say so. You could do this.

A

Set at her and then you do a or I'll just call it. I and then true.

A

A

All right so now, if you were to say uh print.

A

A b c they would all it would print true true true, um so you could says, because this is saying you know this is equivalent to saying um a equals true, because it's almost like you're treating the global namespace as an object and you're saying globals dot a equals. True right, but you can't you can't do this, so you have to do this.

A

That's that's just a little trick for that, um and so you can do you know you can you can also look into? You can also inspect it and see what's in it right, and this is inspecting it to see if there are any functions that are operations, so anything that's been decorated with ad op right. So all we have to do is import these.

A

All we have to do is import them, and now they're gonna get picked up and we'll see that the get repo will get cloned and okay addict object has no attribute reuse, self-configuring use.

A

We will hopefully see that let's see what's going on.

A

Object read itself data flow. Oh, how did it? Let us do that. Oh we gotta. We actually have to pass the argument. Imagine that um oops run on repos. So now let's actually run this with logging.

A

So if you add this logging debug to any unit test that derives from async test case, then you'll get a bunch of log output. How am I doing one time.

A

All right so check it out where we cloned the git repo um and oh, it got mad, get rep, repo, ssh key definition, not in context.

A

Inputs, why is it grabbing repo ssh key okay repository.

A

Why is that doing that? Okay, so I have it down here. I have this thing: that's with shorthand to say: grab all the outputs um like and and make sure that there's there are all the outputs exist. Well, all the outputs did not exist right. So it's complaining now why it's complaining about git repo, it's ssh key, I'm not exactly sure so.

A

So- and this is the kind of debugging that you'll be doing, um you know working on that issue- uh you'll you'll want to figure out, you know how do I build? How do I and you'll look at things like you know, you'll write your operations, which are going to be very similar to.

A

To these operations here right because they're going to do a download of some some data right, let's see, let's look at the data yo. There are apis yeah. So then you know using that. That would be better um because that yeah so use you should use the aio http library or any other async capable http library similar to, as shown in the um in the metastatic analysis example, because everything runs- and this is part of that series of tutorials that'll- that I'll do um uh and if you have any other good async resources.

A

I think there's some async resources linked from the youtube channel. I think that there's a playlist so to learn more about async. It's not critical that you know all about async but learning about async io, but I think this should be public yeah. um So this is a good video. You can check that out. It's linked off to youtube. um uh The link to the youtube channel is under the contact us page and I also post the uh recordings each time so, um okay, so yeah. So let's take a look.

A

Okay, here's the open data api, okay, yeah, and we can- I mean you can do this for anything. I just happen to run into this website, and so you know this is this is the issue that got created um so you know you did you know you could try this, for you know, try it for something else too. It's not critical that you do it for this this website, particularly right, but the goal is to build up.

A

You know a uh the goal is to build up a large set of functions that we can just do exactly like what I'm doing with right now, right so public safety hazards, um flood hazards, okay,.

A

So public view data table okay. How do we get access to download.

A

Csv, okay, so here's one yeah. This is just a a csv file uh done approved, so this one you'll, probably call you can probably copy exactly now, I would say if they do have an api here um I mean you know. I would give this first stab at this using the csv files and then and then you know- maybe do a couple of these right. um Like you know, I don't know yeah this I mean they've done a really nice job.

A

I talked to these people at one point: they're, they're, doing good stuff, um they're, just sort of putting out all the data right. It's like you know trying to do this this. You know this open data approach and then see what people will do with it right and with dffml. We want to give people.

A

You know all the tools that they could just go start playing with some data right, so they're like oh, I want to you, know see if I can build a model, you know using tensorflow to uh you know, analyze the temperature data um you know versus uh you know the number of people walking like that. The number of traffic on that street right, if, like you, they'll, be able to just put those two things together.

A

Just like we're doing right now, all they have to do is import them in the same file and all of a sudden they'll get run together right um so and then they'll be able to do machine learning on on new new data. Sets that that that you don't even have pre-canned right, um it could be, could be interesting, um so, okay, so right now we're trying to figure out. You know why, so we used this one operation, we threw it in the data flow um and now we're trying to figure out well.

A

Why is this thing? Why is this thing? Oh it's because I wrote this in the wrong place, so this thing is saying I don't have a get repo ssh key. I must not have tested that. um I don't have a get repo ssh key, so I can't um you know I you I can grab because we asked it. Where did we do it? So you can see here we passed it. This um get get single says you know get it should be get multi.

A

um Get single says, grab me one value which matches the grab me any value, which has a which has a definition, any value that was generated so the way that this works is basically, it saves everything when you run right and then you query it afterwards to determine the outputs right. So each function just focuses on producing its stuff right and- and that's you know where you you'd go and you'd you'd say you know, hey, you know what what is, what does it mean to um you'd?

A

Probably grab one that downloads the data set, or maybe yeah? I don't know it depends right. You'll probably want to use that ice cream sales as an example, because you'll want to say what actually makes sense for me to extract you know. Should I provide this is just a pre-canned because there's a couple yeah there's a couple options. I think right now these are almost just data sets actually looking at this, um I thought there was more of a um oh. Oh I'm sorry.

A

I have to to uh take this call real, quick, um okay, um okay, wait all right, so I think these actually might be data sets. Let me I'll call her back so um so I think yeah, I think, you're right. I thought it was apis, um but if it turns out that it so if it is apis, we should write operations if it turns out that it's not apis and it is- um and it's all just csv files for the ones that are csv files. We should do data set source. We should use this.

A

A

Okay, great okay, so yeah, so so, whatever you think, you know basically use your best judgment if you think it's better fit as this data set source, do it as a source right. If you have a standard csv file, you can wrap it very easily using this existing method right and then we'll just create operations to pull out of that source, um but if the, if it repre, if it presents itself more of an api like hey, you know like this lookup thing: if they have something, that's look up temperature.

A

Given you know zip code and month, then that's more of a call, the api type of thing where we would write an operation all right. Okay, I'm sorry! I have to run. um I hope that that gave you some more clarity.

A

Good talking to you today have a good one. Bye.