DFFML Weekly Sync, 31 Aug 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Weekly Sync 2021 08 31

Description

Meeting Minutes: https://docs.google.com/document/d/16u9Tev3O0CcUDe2nfikHmrO3Xnd4ASJ45myFgQLpvzM/edit#heading=h.5j518uv7kahy

A

Now, let's see okay, good, all right! Okay, so let's see.

A

Okay, I wonder, should we go, should let do you want to do? Should we just should we play your video here and then we can all see it um or do you want to do? uh Do should.

B

We talk about perspective, I guess okay.

A

Sure I have covered everything.

B

A

Okay, great, should we talk? Is there anything that we should talk about? First today,.

C

I have one pr for like that. You said that you need that uh import linking stuff. I have implemented it with test and all so that pr number is. uh Let me check one two one: four, okay,.

A

All right so one two one, four.

A

Okay, okay, great the unused import stuff, okay, great and you added the ci job, all right, sweet: okay is this, and then it was the other pr.

A

Did we merge that wait? A minute.

A

Did we merge that other pr that had the commit linting or like yes.

B

A

All right we merged it and then now we have this new job. Okay, great, um okay, great, perfect, perfect.

A

A

Okay, sweet sweet. I think I I just I'm adding a helper around this as well. uh That will go on youtube.

C

That is, that is an issue for that, like adding the standard method to like do this a process exit part. There were some questions around it. I I will take a look at it and like let's make appear for that. Let's see why.

A

Because it is something we.

C

Do everywhere so.

A

Yeah, I think we had. Let's see where did I put, that I think I had a branch where I was doing that um yesterday. Actually um I'm wondering okay run command.

A

Looks like it didn't work.

A

A

That's identity and noun.

A

Okay, let's see.

A

Okay, yeah, I had a helper function that I added in this branch. um Okay, so I didn't. I forgot that there was an issue for that, so so this is what I did there.

A

um What is this where's the commit there? We go.

D

A

Issue number is 1078, 1070, sweet, okay, yeah. So basically I took the stuff out of I took this and I moved it into this run command which basically goes and it um it just executes with logging. So, um let's see log of w but yeah, I think it just oops it executes with logging, so it basically does that the output to the login. So what do we have here? I think it's similar so.

A

Where was this.

C

D

This was there already in.

C

This file uh in our uh yeah.

A

C

A

Here the comment link.

C

Program uh command, so I just moved it up right because.

B

It would be used twice so great, great okay,.

A

Yeah, that's perfect, I think uh yeah the main difference between this implementation and this implementation, I believe, is the proc.communicate. um So we'll see we'll see how that okay and this one, this one wants the output all right. So that's a different. It's essentially a different use case there. um So we'll look we'll have to look and see if there's.

C

We need to agree or not like agree on a standard interface that I have this. I want that so that a single function can do that.

A

Yeah exactly yeah, so we'll have to look and see where else create subprocess shell is used and uh and and standardize, exactly yeah, um okay, so all right, okay and then this. So we can merge this. It looks like.

C

So I tested it by deliberately failing the test once and yeah.

C

B

Perfect, all right, we still need to fix that.

A

Software portal thing: okay, so let's see changed.

B

B

C

One one question: I have about code coverage that is like: why does it show some files which have not been changed in certainty as well.

A

That's a really good question, so, um okay, so so so so the code coverage stuff is not perfect. um Let's see, let's pick an example, um because, like things, certain things don't make any sense, um certain things do make sense, uncertain things. Don't make sense, really.

C

You need to log into this.

C

So like we had no no, no changes in memory, but it's still like, showing and and in a lot of vr. It shows memory as a common thing. With this app.

A

C

Problem with memory.

A

Yep yep um where's, the diff.

B

A

Just what is the different coverage, though? Is it this the diff, or is this what is just not covered? Let's see, let's see the diff number of new lines missed yeah, I'm not exactly sure. What's going on with this right now, but 0.25 yeah there's one line in here. That means it's hard to see which line that would be.

A

Yeah, I'm not sure, I'm not sure what would be causing that, but we need to obviously we'll have to track that down eventually um so test dev yeah yeah I this is the coverage. Changing like this is is a is a it's a common thing.

B

B

A

Yeah it should obviously it should be the same, but I have noticed that that yeah, just like you, noticed, there's some stuff that seems to fluctuate that doesn't really make any sense um so yeah we do. We definitely need to figure that out at some point, okay, so the one thing here is: I'm going to squash this this time, just because the commit message, formatting you've used lowercase, someplace and uppercase other places.

A

So I believe what we have previously is all uppercase for the like the message without the qualifiers on where it is so, if you do uppercase next time uniformly, then I'll just rebase it, but for now I'm just going to squash it. I would also add.

B

The student commit an enhancement, cool.

A

Cool- um let's see yeah just just so that we keep it all consistent.

A

This this this as much as this is a machine learning project. uh This is also a project where we can figure out. um You know what are all the: what are, what are all the ci jobs that can be used for other projects to write code? You know, write cleaner code in general, um so you know a little bit of a we're doing a little bit of a meta experiment with the whole thing as well. um So, let's see fixes great, so the more ci jobs that we can.

A

We can introduce to make our writing code and review process easier. The better.

C

So what is that k thing called where everyone every first letter is capitalizes? There is some special word for that casing.

A

I mean you mean, like the first word of the commit message.

C

Yes, yes, what should we call like like? There is camel case, and there is something.

A

I mean that's just I mean that's just like you know a standard standard grammar right. We would we would capitalize the first word of our sentence right because really what we're looking at here is like these. Are um you know these? These define the scope of the message and then this is the message itself right.

A

um So this is sort of this is like the sentence, and then this is, you know, what's the scope right, so this is yeah. This is this. The message is as a regular english sentence and- and these are qualifiers on the scope- um is that the answer you were looking for or.

C

No, I was just trying to put a single word in and they okay uh commit linking. Oh.

A

Yeah, I see okay, um all right, okay, so we move we merged that great perfect thanks for doing that um so commit.

A

Sweet great that that is very nice to have okay, so.

A

Okay- and we don't know if we'll get sudanshi yet today or not okay, oh we did get sudanshi, but we lost hashem. Now all right. Okay, I was trying to wait. So we'd get everybody, and now we lost the shame all right. Okay, yeah! I know he was having internet issues so, um okay. Well, we will go ahead and uh we'll we'll roll so uh sudhanshu, let's see, did you get a chance to? Let's see, where did we do it? Did we merge that?

A

Okay? No, let's see, um okay, so there's some failing ci jobs. Did you get a chance to take a look at that.

D

uh Yes, so some of them is actually related. One of them is actually related to the commit message. So the way I have written commit messages like plain commentary message.

A

Okay, well, let's.

D

Do it yeah yeah.

A

Don't worry about it.

D

A

Don't worry about it.

D

Okay and another problem was with the bandit. I.

A

D

Are some issues with that.

A

Okay, so this is all bandit related, then, okay, all right. Okay, I wasn't sure whether we had introduced a vulnerability and to bandit or whether we just were grabbing a new um yeah. I wasn't sure what what whether it was a new vulnerability that that it was detecting or um or.

A

Or it just got updated somehow, let's see what what is it saying here.

A

Like because I'm having trouble figuring out, let's see well, I should look at it with a full screen.

A

Yeah I was having trouble figuring out why it was raising an error.

B

A

It should be right here.

A

Bend it expand it error exception raise exception all right. Let's go take a look.

A

A

All right, let's see.

A

Oh god, where are we? What are we doing.

B

C

One one more thing I was I wanted to bring up. This is not like development related, but this is definitely.

D

C

uh That is, that is we don't have an image like a logo or something that could.

C

I don't know how to put it like. It is like sort of yeah sort of uh it's important nice thing to have.

A

Yeah yeah, I agree so.

C

uh If we can like open an issue and just ask for the community to create some logos in this, yes.

A

And I think do we have to. We may not have an issue. um Let's see, logo.

C

So I'll just bring up some of my graphic designer friends.

A

Yeah great, that would be great um yeah. I got. I got commitment from the legal team to review um uh logo. um It's a community.

C

So is there any legal legal stuff that that needs to be with.

A

uh Yeah well intel's legal team is, you know like they? They they have all sorts of requirements, I'm not sure what they are. It's not transparent, um but they did say they were nice enough to say we will review a logo uh and they suggested you know. Looking at you know they said animals do do well. um You know, there's a lot of open source projects that have animal related logos, um but really and and yash, also shared with me um that he had had look started to look at um creating a logo.

A

um He wasn't he. He had one attempt, but he wasn't satisfied with it. So uh if you know we need, we need to build, you know a library of options right and then we can all ch. We can choose one right, um so that would be great.

A

Okay, so I'm gonna submit this and pin this um and then feel free to comment more details. um So let's say we can't pin more than three. So that's a good one to pin uh okay, so, okay! Well, all right! So.

C

Enough on the team, when I think we need something that has a left sidebar, because we are very much organized that way. Yeah.

A

We really are: aren't we yeah this one is not this one did not work out.

A

um Let's see the executable book project, I've been using this for an internal thing, and- and I very much liked that that's that's pretty slick looking, but we only have the two options, so we should see if there's any more out there before we, um you know before we make a choice. um Let's see so yeah. If anybody has any other suggestions there, let's put them there. So um are you commented on this? uh Let's see yeah, I wonder what the hell is going on with this thing. um We need to figure this out.

A

Actually, this price should be level with kind ci. Failing there we go yeah um yeah. This needs to be fixed. uh I had to.

C

Push that also, there are some some uh error messages that are warning messages that are thrown in uh while running those test talks testing the docs yeah. So if you like just take a look at once, I I don't know if it's an issue or not, let's see it could be like where, where it shows modeling properly contains reference to non-existing document.

A

Oh, it says that when building the docs.

C

uh Yes, for when I run the test, I'll just paste, the output and group one second.

A

Okay, yeah: let's keep an eye on that because I thought I just made the I thought: I'd turned on the thing that says basically treat all warnings as errors for the documentation builds, um but if not, then we definitely I mean either way like we need. We need to catch all those things.

A

We don't want to end up with a bunch of warnings now all right. Okay, so great, let's, let's record that so uh docs.

A

Logo, okay, great um yeah logo, like I said so yeah I should looked at something, but uh we could we could, if, if you have anything else or any other ideas or anybody who wants to contribute one uh that would be great um because yeah and the legal team is, is ready to review so.

A

We should make sure that with the logo.

A

Let's shoot for svg format, uh ideally with design files.

A

All right yeah, so that way we can change it as needed, um and svg is a good format. um Since it's scalable.

A

So, let's see it oops, I wanted to track the issue.

A

All right great, so, let's see do we have everybody all right. We do have everybody perfect. Okay, anything else for general business today.

A

So we needed to track down the should I failure and that okay, so let's do let's see.

A

So the should I failure.

A

A

A

Okay, so this guy.

A

This guy is failing and it's failing and it's not giving us the output, so we can't see why it's failing. So I think, and then you added the okay.

A

uh For preprocess.

A

Rename pr, oh yeah and I needed that for something. What did I need that for okay, yeah um yeah, I'm about to use your new source uh for something else? um Okay, so should I tell you, I've been at run for different hosts between names. Here um can't see. Log output so we'll merge, run command, um use, run command from bandit to c log output.

A

A

Master and then we'll use because this one, the point of adding this run command thing, is that it use the log output and then evaluate.

A

So analysis of async io dot create subprocess exec to see how we can combine.

A

Helper functions in uh service.

A

A

A

Okay, I think these are the ones I know about. There might be more. I know we have at least three health approaches now.

A

Okay, all right um all right: let's roll the demo done anything else um or we'll roll. The demo.

D

No nothing from my side.

C

A

Okay, great so.

E

Can you hear me fine, yes,.

A

You have anything for us to talk about before we do.

E

So uh I wanted to ask about the demo.

E

Yes, I want to ask if uh we are going to have some kind of intro and uh will we be creating a playlist like uh how will they go about the different notebooks.

A

Okay, well, you know, I think I think that let's watch uh sahil's demo and uh then we can sort of you know, use that as as as a guide and- and you know see what what works well with that and and what you know we might want to. You know what what we might want to change format wise um in case. You know, I haven't seen it yet, but you know I'm sure, we'll have some sort of comments.

A

So, let's, uh let's let's roll the demo and then we can do an analysis on on you know what what makes for a good demo. So here we go, you guys can see right.

A

Okay, great: let's roll your demo.

C

This in two ways.

C

Hello and welcome to another video from dfml today, my audio is not ordered.

B

Until you like specifically share that uh okay, let's see yeah, let's share the tab. Okay,.

B

C

We can do this in two ways we can fight for strangers as an archive. I would walk you through both of these methods and show you the differences between the two. So, let's get, let's get started.

C

First, we'll go ahead and import all the necessary modules.

C

Then we will build our data set for training.

C

In dfml, we have this very nice cache download function, which is idempotent in nature.

C

Then we'll go ahead and load our dataset into memory.

C

Now, let's split our data into train and test sets.

C

So now we are going to instantiate a model and define all our features.

C

Here I've set the location as a folder. If there's no extension in the location, it would be infer, doesn't folder.

C

Let's go ahead and train our mod, so let's go ahead and take a look at our model folder, which has all the necessary files to restore our model properly. So we have this job file which actually contains our model.

C

To test the reproducibility, we can go ahead and check the accuracy before and after loading. The model from the saved state on disk.

C

So currently we can see we have an accuracy of 0.4968 from the model in our memory and then we will restart the kernel to make sure that our results are reproducible.

C

And as we can see, we have the exact same accuracy from our looted model as well. Now, let's go ahead and change the model location to an archive.

C

Let's change it to scikit rc dot.

C

And we need to make the same change down below, where we would be verifying that our model is properly loaded.

C

So let's go ahead and restart the kernel.

C

All right, so, let's read in all of the cells and see how the model is saved as an arc.

C

As you can see, the cycle taxis zip, has been created.

C

And we get the accuracy of 0.4906 and let's go ahead and see if that is the case with the loaded model as well.

C

Yes, we get the same accuracy here as well from the zip file, so there is a slight difference between the two methods: let's go ahead and take a look. So if we go ahead and take a look at the psychic psychic folder one, we only have the psychic json and the job file, which is the safe model on our disk. If we go ahead and take a look at our zip file, we also have a config.json. So this is the same config that you pass while instantiating the model so for better reproducibility.

C

I would recommend you go ahead and use the archives because they also store the config. Now, let's go ahead and try another archive format. Now, I'm I'd be using dot dar.gz.

C

We need to make the same change down here so that we load the correct model and let's go ahead and train our model.

C

Instantiate and training, as we have started training- and we can see that the tartar gc has appeared here.

C

The accuracy from our in-memory model is 0.4562, let's restart the kernel and reload the model from our starter, gz archive as well.

C

And as we can see, we have achieved same accuracy after loading the model as well.

C

Now, let's go ahead and talk about config.js, okay, so let's go ahead and take a look at the differences between the two methods of saving and loading on the left. I have the folder in on the left. I have the folder and on the right side I have both the archives open. So, as you can see, the scikit features dot. Job live file is common among all the three. So this is the same. The only difference you will see is that we have a config.json in both of the archive file and it's not saved here.

C

So when we export a model in an archive, we save the config, which was used to instantiate the model.

C

This contains the hyper parameters and other features and all the information that was used to instantiate the model.

C

So this is a recommended method for better reproducibility. If we take a look at the config.json, we can see all the information that we passed to the model while instantiating it.

C

Same file is there in the side zip file as well.

C

I hope this helps and gives you a better understanding on how to save and load models in dfml. If you have any questions or issues, please feel free to open an issue on our github repository intel, dfm.

A

Very nice, very nice, great job, great job, that was great sweet. I I love it yeah. I love it great job.

C

um So so you can like download the full size file from the drive link. It is on your team, okay,.

B

Just upload it from.

C

There, because.

A

This is not published.

C

A

Yeah yeah yeah, perfect, okay, great um so, let's see so: okay yeah, that's great uh nice! Very nice work. Very nice demo.

C

So so hashem's code was like already very much polished. I didn't have to change anything, so I still just put that time in like making the video.

A

Sweet yeah, good, good, good, stuff, um okay yeah, that went great. So, let's see, let's think about um so for general demo format. uh What went really well here! So, let's see what went well, um so your window, the window manager layout very much facilitated understanding what the hell is going on here. With your you know your folder structure and the way that you showed the files.

A

So, let's see showing files and directories created as a result.

A

Explaining uh contents, so that that was very that was, you did a good job with that.

A

So let's see what else do we got here? um Let me resume. Sharing everything.

A

A

What else so, let's see.

B

A

I'm not sure, let's see, I'm not sure, does anybody else have any comments on things that we could make clear um or d or format like. I think I think, let's see what your your intro, what was your intro, your intro was nice. That's good! Let's see.

C

Hello and welcome to another video from dff today we will be talking about saving and loading models.

C

We can do this in two ways.

A

All right, okay, yeah- I like that intro, so you know welcome to the video. uh This is what we're gonna be talking about. Intro, that's good.

A

Little video uh we'll be talking about okay, um let's see, and then what did we have? Did you have a little wrap up at the end.

C

I hope this helps.

A

Gives you a better.

C

Understanding on how to save and load models in dffml, if you have any questions or issues, please feel free to open an issue on our github repository intel, slash, dfms,.

A

Open issue, if you have questions.

A

Help all right.

A

All right, great, okay, so anything else here.

A

So anything else for things that went well anything particularly stand out.

E

Oh well, since, since I'll be doing five or six demos, I was thinking of a separate video to cover the startup phase of the. I.

A

Think books, I think that's uh that's a good thing to think about yeah. I was thinking that as well um separate video to cover startup phase of notebooks or.

E

Like boilerplate, and I was thinking that in this way the tutorials could come below including science as well.

A

uh What do you say that again I didn't catch. What you said.

E

uh I said that uh in this way we could first uh explain the setup process of uh you know using cachet, download and importing stuff and all that, okay uh and after that, uh we could just uh go through the use cases of different notebooks yeah.

A

E

You know we could have a playlist like that.

A

Yeah, that's that's a good plan.

A

Okay, I was also thinking.

A

um I was also thinking we should really figure out how to move that nested uh stuff to um out of out of the uh into the main repo. Somehow, um where was that.

B

I think there's a branch.

A

Where's, this branch.

A

I swear I had a branch was messing with that.

A

I don't know oh well, um yeah, oh no, I was messing with it in a notebook, um okay, yeah, uh because there's there's got to be um well yeah. There's there's got to be a way that we can get rid of that nest. Async io apply stuff at the beginning of every notebook, because that's you know.

A

Ideally we wouldn't have to do that. um So yeah we'll cover cache downloads, hmm okay, so yeah separate video for boilerplate, um okay cool um and then we can have a playlist uh last.

A

Notebooks um video goes first.

A

Okay and then we'll just leave this one, as is um the only the other thing that this made me notice um is so config.json um is that you know yeah this uh right now we're only saving two um two archives.

A

um So, let's see what does our adr say? Do we have we even merged that adr, I don't know um no looks like we have not.

A

A

What does it say here? Okay,.

A

So we talked about immutable and immutable config.

A

We talked about.

A

Wool versus non-tuna wool.

B

A

We talked about okay, yeah because now I'm thinking, okay, so we're logging we're logging when we load if the things are different um and we sort of started this whole sorry, we started this whole discussion about objects, saving and loading off of that. um So we need to sort of flush out because I'm just thinking you know the config.json only being in the archive.

A

um Ideally, we should be able to put it on the directory as well. The question is, you know: what are we gonna do with it right now and we don't really have much to do with it right. um So I think we probably need to.

A

We probably need to take a take, a good look at the stuff that you've done, sahil and and figure out. How do we abstract it into this generic object, saving and loading um and then and then think about you know how to like: how do we instantiate figure out what object it is and then instantiate it?

A

um uh For example, like you know, given the base class and the plugin name, we're given the baseplat the base class and the plug-in name, you know instantiate using a given config and we may want to think about uh including the plug-in type. Let's see, where's the screenshot of that yeah. So we may want to think about change, because this is the config structure.

A

So we may want to think about exporting as well the plug-in type here um so- and this is not really demo- video related right, but we uh we know this so and now we're only saving uh the config.json to archive.

A

I need to look at.

A

Model saving and loading code to figure out how we could abstract it a generic version which would be related to this whole. This adr here.

A

And then right now we're only using convict saving convicted, justin archives. Okay, uh we should also save plugin.

A

Name to this on so plug in.

A

So, for example, you know plugins config.

A

All right um so to save the config, you know, we've been calling this. This is there's helper functions called config dict, um because it's config dictionary essentially is what this gets loaded in as and it has the plugin and then the config um we'd gone back and forth on including the plugin type um talking about this with such sean, I think such shaman I had talked about this and we'd gone back and forth on including the plug-in type, mainly because uh we don't want to open ourselves to the actually.

A

Maybe I said this in this meeting that that that yaml, the libyaml cve, um where they support arbitrary python objects um now. Well, we don't support arbitrary python objects with this, as as soon as we start getting into plug-in type, there's a risk of somebody instantiating the wrong plug-in type for something so controlling the plug-in type.

A

Is you know something that we might open up at some point, but for now, we'll probably be provided to this generic version here so we'll provide these functions with the plugin.

A

Type so model base source, etc.

A

Okay, uh okay, so anything else so well, we talked about okay, so we talked about the um the nest apply. We need to move out to its own repo.

A

Or move into the main repo, so that that isn't duplicated in every notebook and people you know when they import dfml in a notebook, get the right functionality. um I think that poses issues.

C

Hey hey, did I miss on something because my internet digest.

A

uh Just that we're gonna, just just the config.json, for we need to figure out the config.json, uh probably save the plugin name as well to there and then look at this stuff and start to abstract it into a generic version right which is related to this object, saving and loading adr and then probably continue from there before we do more model specific work.

A

uh I think getting getting this object, saving and loading down using data flows, um who will then allow us to you know effectively um decide what we want to do with the the config properties and stuff in terms of if they're not equal, mainly because we really need a way to so. Let's see this is an important next step, so this is an important next step, because we want, we would like to be able to load an object from disk without giving any config right. Currently, we must instantiate the model with a config, then load.

A

um So this is why this is sort of what I see as the next step here is because um yeah then we can we'd have this arbitrary mechanism for loading, something with a config, and then we can say okay, so models. How does that relate to models? It probably is the same as everything else with the mutable immutable configs, if you already specify um but yeah, let's get to a point where we can load something generic um okay. So anything else that we should think about.

A

With regards to uh you know the demo or notebooks in general,.

D

I had one question: do.

B

D

Have to create the demo like beforehand.

A

uh No, uh that was just because sahil had uh inter internet issues, uh so wanted to make sure the demo was was gonna play smoothly right.

A

E

uh Can we also go for that.

A

Yeah I mean yeah. If you want to record it, go for it, um then that that makes it easier, um because then we can just you know I'll, just repost. This video on the main main uh main channel.

D

I was also thinking that maybe we can, I can record the demo okay.

A

D

Give you the link.

A

B

A

Yeah, the only reason I was suggesting we can present the demos in here is just because I figured it would be two birds, one stone, um but if you guys want to do record it on your own that that's cool with me, um so let's see all right. Well, let's see we got this, uh the renamed pr. We were figuring out the should I issues we came up with the plan for that, so this I'll take care of this run, command, stuff and I'll.

A

Do the rebates and stuff and see what the output is and then, if the output is caused by something that that you know that was an addition from your pull request, then I'm going to punt that back to you and ask to see if you can dig more on that and then we'll do this analysis of subprocess exec when we finish up with that, so.

D

I will look into it.

A

Great, so um so yeah, so I'm just going to make that a note here. So is there anything else we need to talk about today? um Let's see, do we want to get? Does anybody have any any open issues that they'd like to highlight or any uh pull requests.

C

Okay, I want to ask back as the what is the question from here like it would be chasing the data release of.

A

uh Like what's our next step, sort of large large scale, architecturally here.

C

Yes, yes, that's what I'm saying: okay, like so.

B

C

To hit the beta sooner or later,.

A

Yeah yeah. Okay! So that's a good! That's a good! That's a good question! So, um let's see so what how do we get to beta here? So, let's see.

A

A

All right, so, let's put this.

B

A

A

Okay, so creating a shoe when.

A

Done with this pr, so we'll create an issue for this whole consolidation. When we're done with this pr. That way, we don't start doing it prematurely, um because we need to get that other helper in here. So we need to. We see issues.

A

Caused by so we see issues caused by changes and this pr will have x, okay,.

A

So I'll do this part and then, if we see changes or things issues being caused by this pr, then you'll fix it, and otherwise, um when once we're done with this, we'll create an issue for this, and so maybe you'll pick that up. Maybe you want whatever you feel like when we get done with that, so, okay, so let's see how are we getting to beta? um So this is where okay, we haven't. Okay, I haven't been doing a good job with the issue labels um or await the milestones. Okay, so essentially I haven't yeah.

A

I haven't gone through and done so this is what we had for for I've. Been I've been tagging things uh okay, so the way, the way that this this usually works is tag everything for the next like a a long shot um and okay. So, let's see so.

A

As far as beta is concerned, beta would be where we're hoping to declare like an api stability right, um so we really need to have like their accuracy. Scores were a big thing that that we needed to flush out, because that was a big api breaking change.

A

um So we really need to focus on this this right now, this milestone was was really a hodgepodge of of things that we uh that that we would like um and which I think included the the web, ui and and- and you know, sort of a lot of stretch goals um at this point. uh We need to take some of this stuff and and refocus it. Actually, the ice cream sales demo is done.

A

So yeah, so we need to refocus, because a lot of this stuff will need to be moved. Okay, here's what I'm going to do we're going to take all of this and we're going to make a new milestone.

A

Okay! So actually, let's look at this guy. So usually what I do is we write stuff? We have it for um you know. We know we know. Usually what happens is we know certain things that need need to get done like asap right and I try to tag those with the next whatever the next increment is right.

A

So we need to decide what things we we need to decide what things, um what things really matter here um and then we pick them off. We need to move. We need to move stuff from this current beta release, milestone probably into the 1.0 release and okay. Let's just move it all right now and then, let's see.

A

1.0 because I think we'll probably do a 4.1 for the work that you guys did um and and then we still need a few more things. There's there's definitely a few more things before we need before we can declare beta. uh I think we need to refactor the http api is a big one. um There needs to be. I was going to write some more documentation around data flows and expand that we, I was talking to some people internally at intel and they have a similar use case and a similar, similar yeah 100 complete.

A

uh They have a similar use case for an execution environment. However, theirs is like a linear like a sequential flow of operations, um so I'm going to expand that um that execution expand the data flow stuff to support different execution environments rather than just the parallel uh or like the permutations based one that we have. um We definitely need a bunch more see. The the ci related work is always um is always high priority right, because that's what helps us uh effectively take in new contributions.

A

um So, let's see uh oh yeah and then anything that's gonna get deprecated needs to needs to uh needs to get done. um Let's see so sphinx copy button, I don't think that is a huge priority here.

A

So, let's see oh, this probably definitely needs to be done, um so we can start taking these and putting them on beta and instead of using beta as sort of because beta was kind of like a uh beta was just more of a area to throw everything that we knew we needed. But now um it's looking more like that'll, be 1.0 um sort of as as wish list right.

A

So we know that we need to do this, exposing more information from the models um which, right now we have model type and uh model train, not trained. So essentially, let's think as far as thinking about planning- um and I wish you could give you guys permissions to just do this. But if you can ping me, if you think something should go in beta, then ping me and I'll I'll.

A

Try I'll add it right I'll, put the milestone on it right, because we really just need to decide what things, because, when the the main point is, if we declare a beta release like you each we need to be api, stable right and we need to we. We can start thinking about.

A

We could make api changes between beta and 1.0, but that type of thing is going to. uh We you'll lose users right when you make breaking changes, um so we really want this to be, as as as.

C

We don't want to be tensorflow.

A

Yeah, exactly precisely um and and working things in so, for example, okay and and and how do we avoid that right is really um things that are getting so so make make make sure that we have flushed out our usage of our high level apis as much as possible right. um So let me write this down so flush out usage of high level apis right, because these are what the high level ones are. Really what we're committing to right, we're committed to stability here um so make sure function. Names are correct.

A

Parameters are correct,.

A

uh You know flow flow works, so this is like accuracy to score, probably needs to happen. um um I don't I mean I'm sure, there's a few more here right um things, things like that, that are very user-facing. We need to figure out those right and we need to identify what those are um we need to.

A

C

Streamline dataflows.

A

Yeah we need to streamline data flows, that's for sure, um so we need to let's see document, so data flows there's a lot of stuff that needs to be done around data flows so abstract or support different.

A

Execution modes.

A

um When the sequential is the one that's been brought up and then support for different operation, implementation networks.

A

All right, um and so the operation- implementation networks, okay, so this is the main thing that I'm I'm held up on on writing the tutorials right now. So I wanted to write some data flow tutorials to to cover the various syntax and capabilities that we have right now.

A

The main thing that I wanted to do before that was.

A

Was to write another operation implementation network to showcase. You know why we're doing it this way, um because right now you know, there's the flow-based programming aspect and there's like a lot of benefits to that right. You can do um you know you can do analysis on this serialized format, for what the execution flow looks like um and that's very powerful, um and you can, you know, do a lot you can. You can do merges of data flows and and- and you know, expand expand upon things in that way.

A

um But the one of the main reasons is: is the ability to switch out the the implementations right underneath the operations right, um so getting that support- and I was going to target- is going to target open api based services, and so there's work to be done. Around definitions, um definitions and.

A

This work to be done around definitions and making and uh tying them closer to config infrastructure.

A

um So the the key with this is- and this is this- is also so uh unified unifying config, so we need to so so the config stuff right, a huge amount of this project has to do with configuration right. um So the data flow definitions are not as tightly coupled or is not are not as integrated with the definitions with the rest of the config infrastructure as they could be.

A

We need to get to the point where, um where the inputs, the inputs to a data flow, can be, can be a config, um a config config object, so inputs or sorry inputs for an operation should be able to be made into a config class instance.

A

Okay, um because this will so this will- this also enables us to run uh operations one as.

A

From command line, okay, the other thing that is happening here is that with this arbitrary, so that arbitrary saving and load loading thing, I think, is- is a big piece of this. um So uh object saving and loading.

A

uh So, okay, um so once we get the arbitrary object, saving and loading right, you can essentially define um so you could define a model, you could define a source right and- and you could say you you'd um like if you were defining a model um and you wanted to load the model right, you would. You would give the data flow to load and it would give you the instantiated model right in the data flow to load.

A

You know if it's a zip or a tar file, we would leverage, you know we'd be using the existing stuff that you you've implemented right.

A

So uh if it's a, um this also gives us the ability to say like okay, the you know: okay, tying tying the saving and loading in with the operations and making the operations, the the same like executing the the operations having their inputs be be able to turn their inputs into the config classes, allows us to reuse the command line, infrastructure around them, and it also allows or the command line infrastructure with them.

A

um So and this in conjunction with the saving and loading, you can imagine a world where you could have, uh for example, a run operation. So the the run data flow operation.

A

You could have a serialized description of the run data flow operation, along with um you know, along with the um the the data flow to run, and you would specify things like the orchestrator and all that stuff in there right, and so then you could have it as a self-contained description of you know a data flow to run along with the orchestration run in it and then anytime, you needed to do anything.

A

Let's see, how do I explain this? um If you can do, if you can fit these pieces together, then you can really use that run. Data flow operation like in in a lot of different places, because if you, if you begin to structure things around, you know, let's see so, for example, if for the object, saving and loading, um if, if it takes instead of taking a data flow right, so if it takes a data flow, it's still it will.

A

You still need to specify the orchestration environment right so if it takes like, if it takes in, if it takes.

A

If it takes this run data flow operation, then the run data flow operation we instantiate the operation like it would take an operation instead of a data flow and that data flow or and that operation would be the run. Data flow operation to load an arbitrarily, complex thing right or you could just have a single function, because an operation is a function and it would load.

A

You know your model, but if you have the run data flow operation, then you can specify the orchestration as well um and and what operation implementation networks to use and all of that um so really the point being we gotta. We have to to tighten up the config infrastructure here um because that's going to allow us to uh to to leverage things in more places.

A

So, ideally, we.

C

So, regarding regarding this object, saving the loading part. Do we have an issue to track this currently or we are just having it? We.

A

Have the discussion happening in the adr which is here so we need to sort of flush out. You know and write some write, some experimentation and then add to this document um as necessary.

C

A

C

Ideas I I sometimes feel like that. uh Why don't we leverage discussions.

A

Yeah, definitely, let's do it yeah, that's a good, that's a good way to do it. um We have because it ends up as a pr and.

C

And sometimes it's really difficult to find.

A

Yeah yeah cool: let's do it so, let's start a discussion so.

A

A

B

A

A

Okay, um let's see so. This is why this is tricky, so.

A

A

The the the goal here would be to take you know a data flow or.

A

C

So a data flow is like a set of operations, and an operation is also an operation, so can't be. A data flow uh consists of a single operation.

A

Yeah data flow consists of it can consist of a single operation, I'm pretty sure, there's like a run command.

C

A

Why do we differentiate between what.

C

You you just said right that takes the object and a data flow or location or an operation.

A

uh Yeah, because um we want to, we probably want to add some stuff around your, uh for example, we we want to. We want to use a default so, for example, if a location is given yeah okay, so if the location is given, then we run the code to create the data flow right so or then we we use the existing model, save load model archive.

A

Data flow uh stuff, uh yeah.

A

Okay, so this is all I was saying here- is that if we get a location, then we're going to call your code that you just wrote right and we're going to create a data flow from it, because it's likely, my guess, is that well we'll support, so that the point of this is to support saving and loading of arbitrary objects. Right um now, my guess is most of the time people will will just give a zip file location for whatever they're doing right.

A

So if I have a model, I'm going to save and load my my zip file model right or if I have a.

A

If I have a specific operation like that I've configured in python or something right so say say I take an operation and I I instantiate it with its config, and now I want to dump it out. um Dump it dump out the configured version to a to a place where I could. I could you know, move it somewhere else right and then execute it. uh Then um I might just specify the path right.

A

um So my guess is, my guess is most of the time the your location code is the location to archive code that that code will end up getting called here, but we would like the ability to to have it be arbitrary right, I'm kind of like the you know, the the at her for the the load and save right that you did um so so the goal is to create if there's a location.

A

um Otherwise, if there's a data flow, replot supplied execute the data flow using the default orchestration environment right. But if an operation is supplied, then, if an operation is supplied, then we have a single operation. Data flow right and that operation might be so if location is given. If operation is given execute single operation, data flow.

C

One one more question I have it might be: it might be like illogical, but but one question: how? How does this uh improve upon, like we are trying to generate save and load objects in a generic manner? And uh and how does that differ from pickling and python?.

A

Okay, so yeah, so the main reason, the main the main way that it differs from pickling is that pickling um pickling.

C

So the question is like: why do we do this, instead of like just pickling somebody.

A

Because you may not want to okay, so the the point of this is that maybe your maybe your data flow does pickle it, um but maybe it doesn't right the the point is you take an object and you produce a serialized representation or you do something to to save it right, and so so I could the something that I do could be pickle the object right, um but we don't want to enforce one particular way of doing it right.

A

uh The point of this is to to to make that open-ended right to provide the infrastructure by which you could say. Okay, I want to pickle it or I want to put it in. You know. I want to store it to json right, I'm just saying hey when I save it, I'm going to use this data flow to save it right and that data flow might do the pickling or it might. You know, you know, choose some other format right.

A

Does that does that answer your question.

B

A

Okay, yes, yes, okay, um point here is to uh allow user control over how their object.

A

A

uh So and then we could so uh if you had a single operation data flow. um So if you have a single operation data flow um and you executed, you know run data flow, then you would use that to configure uh configure different implementation or a different orchestrator.

A

Or networks: okay, uh okay,.

A

Let's see we'll put this here.

A

A

All right yeah, so we can flush out more stuff there, um so the saving and loading. I see that as a big thing, um mainly just because yeah it'll, so that'll extract abstract it um out to a higher level, the stuff that you did. That was the foundations of it right and now we abstract it out and we support other things and we support it in a more complete manner by loading instantiating. The object with the config we need to.

A

We need to do the unifying the config stuff, so this also includes you know like converting to correct data type uh and this so with new um mutable uh code.

A

A

We can now convert to correct data type uh if on access, if it is incorrect,.

A

uh So this includes including loading m config.

A

Instantiated object, um so we had. We have mini code paths um through the config stuff right now um and getting to a place and- and the result is sometimes you end up with okay. Here's an example of that where's that export.

A

Damn service http tests, this does this, have the export. No, it doesn't. Okay, so there's somewhere somewhere in here, there's a a a data flow with an export on it, because for some reason this see in this in this we're supplying config to this uh remap operation and it takes a data flow and we've passed an actual dataflow object here now, but so we've passed a dataflow object here.

A

This is a config dictionary, but not like the actual config object, and so we have lots of places where we mix real data types with the dictionary representations and so there's some places where that works and there's some places where it falls apart, and so, if we can consolidate all the config stuff and and and get it get the loading flow of objects to be uniform in the in the way that we're handling things in at config and all of that code, that's mostly in base that that is going to be important, because, right now we have lots of varying code paths, and sometimes it just blows up.

A

So we need to figure that out. um Okay. What else are we looking at for beta?

A

uh We need to refactor the http service, um so, let's see we need so yeah, so we need data plenty of data flow tutorials, um oh yeah, a factor http service.

C

So so one thing that I wanted to add about the documentation that is like there are some abstract concepts that probably you only understand the best and uh like, for example, operation implementation network. So so, for example, uh there could be like some some things which which actually tell what happens if you switch this, for example, if uh what is the need to switch operation implementation network and what could be another operation implementation network from uh how another different operation implementation network could be there, which is not the default?

C

One like something like that, you know you get to get my point.

A

Yeah, I get your point yeah and that's exactly what I'm uh trying to work on with the. uh I think correct me if I'm wrong here, maybe- but I uh this is when I'm talking about the data flows and the tutorials around that I want to go, uh I need to go implement one of these, so that we can show the example, because I think yeah that's important. So if there's other concepts, so what other concepts come to mind there so orchestrator.

A

Operation implementation network uh is there anything else, conceptually that comes to mind.

C

um These two are the ones, but I cannot really recollect what any other.

B

Thing right now: okay, okay,.

A

Okay, great um so, let's, let's and let's also track so if there's anything that needs to be documented, uh uh please open a missing documentation issue, because I know that yeah. Some of these things are things that I understand and I haven't written down so need to. I need to go, write those down and some of them I know, and some of them I don't know so uh yeah, please. Let me know if we have something that we don't know.

A

uh I know that I know that the data flow stuff is lacking, um mainly because there's been several gaps in there that that are always like. Oh, we need to fill this gap before we document it, but now I'm like okay, we need to go, fill those so okay and then we we talked about tensorflow and api stability.

A

So the main thing, I think the other main thing that we really need to figure out here and on on the vein of saving and loading, is file formats, so file, slash document formats, so uh versioning, uh metadata um and contents.

A

uh So so thinking about uh things uh thinking about our config dict structure, um which has a plug-in and config right now uh this uh when export, when exporting an object, uh we need more information to be backwards, to set us up to set ourselves.

A

Up for good backwards, slash forwards, compatibility.

A

um So things like versioning, so metadata uh versioning, uh let's see at and then so metadata black foams, yeah platforms.

A

Yeah so anything dependencies.

A

um So we need to figure out what this stuff is and the thing is we don't know what it is right now right, and so mainly it's not really about okay, so so just to double just to reset on this. Essentially, you know: we've got we have this way of specifying and we're just getting into it with the object, saving and loading right like what? What do we need?

A

What information defines an object right and we know we know it- takes the plug-in type right like model or source, and we know it takes which plug-in and then what the config is for that we know that right, but we also need to think about like things moving forward, which is like okay, what version is it right? um Because, as we, we increment our version numbers and we have um you- know different different implementations underneath um you know the the config this config object?

A

That's serialized may no longer apply right, so we need information to figure out, you know, does it does it apply when I'm trying to load it, you know, does the serialized representation uh apply to the in-memory representation?

A

Is there other information that we need and is there ways that we can we can support? um Are there ways that we can support addition of arbitrary data as well? You know, for example, with like a metadata section, um you know and are there required fields to the metadata?

A

um Is there maybe um you know uh so optional extensions, optional fields need to support optional fields um and then how like? What does what does interaction with optional fields? Look like.

A

For example, you know uh say, let's see so uh say: for example, we had an extension so um extension to saving and loading. So when so, when saving or wind loading.

A

Verify contents.

A

Or verify rest of document against h mac, so um your I'll just say: cryptoback.

A

Graphic, we verify rest of document.

A

Okay, so this is essentially like okay, so so say say: for instance, you know say we have an operation right and we need to um and we're going to use this operation in a data flow right. um How do we, you know?

A

There's a lot of there's. Just a let's see like say you have yeah say you have a data flow and you're rolling forward from one operation to the next operation. Right you need to. You need to make sure that the the.

A

I'm not really sure I'm not doing a good job of explaining this right now, I'm sorry, um but basically we just need to sort of to to play with the serialized format, which is why that saving and loading is important and make sure that we have a format that is one extensible and two. It sets us up for good backwards forward, compatibility right and and versioning, I think, is the main thing there and then an extensible metadata section.

A

um I think if we can nail down how we're gonna implement those two things, then, essentially as we roll forward right. Oh okay, versioning, so document, versioning document format, versioning and you know, object versioning.

A

um So, basically, if we can create a format that that tells us what version of the format we're on, then we can effectively do backwards for come forward compatibility really easily right, because we can just read the version number of the document, and now we can parse appropriately and we're going to set ourselves up for success going forwards because we aren't, you know, constantly trying and failing to parse something and then determining the version. You know, based on failure right to parse a given format right, so things like that right. That does that make sense.

A

Does anybody have any questions on that and then we can probably log off here, because I know we're over.

C

And it is like a lot of things: yeah.

A

There's a lot going.

B

A

Yeah but but really the point being, I think you know if we work on this serialization stuff, um we work on the config infrastructure and then we work on making the config stuff backwards forwards compatible.

A

um You know, then, as we go forward, we can save and load these objects and we can save and load them in a way that um you know we can do sustainably uh if we create the right formats, um that's at a high level. You know what I'm what I'm saying here right. I think these are the. These are the core concepts that will allow us to be successful going forward.

A

So these are really the things that we need to think about um as we're as we're on our road road to beta here, but before we hit that we're gonna, do the 4.1 release, which will have all the changes that you guys uh put in uh over the summer here so and.

C

We'll try to do that soon.

C

Where does the second party stuff fit in.

A

And where does the second party stuff fit in so the second party definitely has to has to happen before beta.

A

Ideally, sooner.

A

But must before beta- um and the main thing is this- is our guide uh for how to do third party, uh so so so the domain, the biggest the biggest thing around second party plugins, and why it's so important is because, right now our entire thing is set up around the mono repo right. But our plug-in. The point of the project is that we really we we want to be in a more distributed setting and we want people to be able to write their own plugins right and contribute them to an ecosystem right.

A

So we need to. We need to make sure that our development setup also follows that model right, and so, if we split this stuff out and do the second party um and second party is really an umbrella term to cover the taking the you know, the subdirectories out and putting them in their own repos right and if we can figure out that whole ci workflow and that git workflow and the way that contribution works and documentation works uh with that, then we should be.

A

We should be easily able to implement um what we should be easily able to handle. When somebody said. Hey, hey I've written this, you know plug-in for a model. How do we display that on the doc site and all of that workflow we figured out while we went through figuring out the second party stuff right and that's what you know?

A

We want to make sure that we have that that base understanding so that when people come and they they can see how they can easily create plugins and then share plugins right, because that's the that's one of the main goals right so does that does that does that? Is that a good answer there?

A

Yes, it is okay, cool all right, um all right, cool yeah and that's so these are our main things right. Documentation is always refactored. Http service objects. You can learn. I think I think this pretty much covers beta desires, um so obviously we've gotta, we'll use.

A

You know the four point x um to get there right and then shout out on the getter or on issues to say to say, uh if you think something should be tagged, a specific milestone, right and then I'll just go tag, it um so yeah cool and then, of course, always ci. Ci related things to catch stuff that that we catch during code review um for contribution, um because you know streamlining that workflow is key as well um and hopefully, when we move to second party, then we can take.

A

If you guys, you know as long as you guys still want to be involved, then we can start making. The other part of second party is we can we can take people and we can make them maintainers of given repos right. So so we can have a set of maintainers for each plug-in, and then we can have a set of maintainers that span all of the the plug-ins right, um and that way we can get better throughput on reviews. Yes,.

C

Yes, that is good, because currently you have all the responsibility of doing the reviews.

A

Yeah exactly yeah exactly it would be, and it would be great. You know if we can grow more. If I, if I'm the only one that has permissions to do reviews, then you know I'm really the only like. I'm the only one practicing the skills of reviewing right and we need to get to a place where other people can practice those reviewing skills as well, um so that they can, you know, become reviewers.

A

So hopefully we can get on that soon. I was talking to saksham about that um and we were brainstorming. But um oh anybody who yeah.

C

How it was thinking about us, but like uh what we can do is start start reviewing uh contributions from new contributors, yeah.

A

That would be great.

C

And, like you can come at the end and like see all the stuff at once and see like how how we should change our base. That's a great idea. Eventually, you would learn how to like. Probably.

A

Yeah, we could do uh actually, so we can start doing in in this. If we get, if we I've noticed, we've gotten a few contributions already, um you know that it seems to. There was a sudden burst. uh I assume, related to the end of gsoc.

C

A

So we could do that next week. So if you guys are going to record your demos, we can go through and we can do code review next week um so for next week.

C

A

Good yeah sweet all right. So let's do that next week and then we can practice that, and uh that would be great great suggestion.

C

So actually it boils down to one thing, one more thing that is like knowing the long term goals of the project you can. You can foresee stuff that we cannot sometimes so.

A

Yeah, it would come.

C

With time like understanding.

A

C

Things should be structured.

A

Yep yeah, yeah and and uh yeah we can. We can talk more about long-term goals.

C

All right, so so, if you have any issues like accessing the video file demo, video files.

A

C

I would right.

A

I will all right great thanks, guys and great job with the demo today, all right, we'll play you the, if you guys have your demos for for next time, we'll play them in the meeting. Otherwise, um you know we'll just whenever you're ready, so cool thanks. Everyone have a good one. Bye, bye.