DFFML Weekly Sync, 18 Feb 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Weekly Sync 2022-02-18 Part 1

Description

Meeting Minutes: https://docs.google.com/document/d/1vKYEPtqKiwsFwhVKPmPub5ebMqN9HteBcbdFAuTXalM/edit#heading=h.bucu7l213uqd

A

All right, it's just me today, I think so, unless anybody else joins.

A

So we are going to.

A

We're gonna do what I was gonna do anyways.

A

Which is we're gonna, look at the inner source use case, um and how do I make a new session.

A

So we're going to look at the inner source use case, which is as a reminder the bill tree organization has formed and they've gotten their governance going led by yash and saksham and other folks.

A

So we're trying to think about. You know how do we create automated tooling around the governance structures of you know: multi-project organization, of course, this maps very nicely onto our second party plug-in use case, um which is where we have the main package dfml, and then we have um you know all the other plugins live in their own repos and that's the second party stuff that we've been talking about forever. So um this would be okay, you've put them in their own repos. How do you make sure that they're doing all the right stuff like?

A

How do you make sure that they didn't stop doing linting right? Because we still want them to do linting? You know? Well, how do you make sure that? um How do you go verify? You know, maybe, as we get further into this, um that they are running the tests, the way they're supposed to be running them, because that's one of the things that we'll split out with the second party stuff is. uh This is good. This is what we're working on right now is basically like the auditing, um it's the auditing.

A

After we transition to the second party setup, uh you might say well, why are you doing that first, and I tell you because there's overlap with something else, all right? Okay, so um let's do it nope we're not here. Are we okay, wait now see operations?

A

What was it inner source aptly named all right where we at so I think I was trying to.

A

um I think I was trying to where were we here, oh and all that uh that's the remote uh ssh hsh based uh remote execution, um that was just random notes. All right here, we are.

A

So, okay, I created the package, um I think yeah. Is that all I did. I just created the package all right, let's just clean this up, real, quick, it's a dangerous thing to do um that deletes all the not tracked files all right. So here's our package um and we made the package by.

A

All right, so we made the package by. Let me open up the meeting notes.

A

So we made the package by following the development service, create command right, so we wanted operations. So if you do help on this, it'll tell you which kinds it's every kind of plug-in right. We should have a. We should have a skeleton project to facilitate the building of a new plugin um for every type of plugin. We have right. So if we have sources, we need a source, skeleton and all those exist under uh scale, and that is documented, um I believe, there's a working on scale section.

A

Go basically out of notes working on skill, okay, so this is documented here and it talks about how there are some simulinks that are needed, which I don't know how that will work on windows. But this is just for developing not for um not for usage. You don't have to be immediate. Sim links all right, so um so I I generated the package um reload to allow offline go away, so I created the edit the.

A

Don't do that don't do that alright, great.

A

It's gonna try to open teams on me. ah Don't do it don't do it windows all right, fantastic, okay,.

A

Wow slow today.

A

All right recap: 18th.

A

Oh, my god: okay.

A

All right february, 18th.

A

All right, so we are talking about um checking.

A

Interest: okay,.

A

So for the dfmo or big tree org, you know wherever we end up putting all these plug-ins. um We want to.

A

We want to ensure that um ci cd practices.

A

And other things maybe documentation test coverage um are being kept up to org standard levels.

A

uh So we need to write operations to gather the data.

A

Required to determine if project is healthy enough to meet org level standards.

A

And so the reason so this will. This will tie into our tier model, um with the tiered support levels so um and just as a recap, the tiered support levels. um So this is true right here, support mobile model, which can be found in the adr for the second party plugins, which is.

A

A

A

um Possible support levels, okay, main package, all right so support level. Zero is the main package.

A

We're not gonna break that if we break that we lose um it's a program right now, but you know not the release version, so, um uh okay, so sport level, one second party required to pass release okay. So this is it's been a long time since I looked at this, um so single pretty required faster. So this would be all the plugins which we consider to be part of the core set to make the project functional enough for end users like um sort of the out of the box.

A

um So this is things like you know, while you may so, if you were to do a pip, let's document this actually.

A

Installation, so if you were to do a pip, dffml install models right.

A

This corresponds to that so.

A

I don't think we said this before, so it's good we're going back over this gives some time to think it's excuses for procrastination. um This corresponds to model.

A

Okay, so this is and and hopefully that that helps clear things up, but this is basically saying if I want to install dfml and all everything related to models, then this set of plugins that gets installed should be considered to be our support level, one prod support level, one models.

A

So if you install all these these, these are the ones where we're going to guarantee the documentation work. So we don't ship unless the documentation works, so that would be spoiler one um all right um support level. Two is basically everything else. um You know that exists, probably within the main documentation website.

A

We talked about- and I think we have documented here and we're not going to go into the um various um ways of documenting the matrices of what versions, what version combination different things work with um so like, for example, if you were to go to, let me pick a a a plug-in that involves a tutorial that involves multiple plugins right. So I think this one.

A

A

A

Man, maybe let's go to the next branch tax.

A

Okay, now I'm going into it.

A

Oh, come on, there's got to be one of these things with the giant pip install command.

A

A

uh Not those using nlp operations there we go all right. Well, whatever all right. um This one only has two but it'll suffice so um model tensorflow is probably one of our tiers tier one support level plug-ins right, the nlp operations. um You know in a perfect world, they would be part of our support level, one plug-ins, but but the only person who really knows a lot about nlp is himanshu and, and uh so we need to. We need to get some more people who who know about nlp.

A

um Well, oh, we have uh sahil is also doing nlp, so we need to be beef off the amount of maintainers. We have in this area because yeah that that kind of is what makes these metrics drop is when we don't have enough people maintaining things so um so in this case, we would have something that uh you know. This would be a support level.

A

Two perhaps in this instance the nlp operations right um and if, for whatever reason, they stop working with model tensorflow one day, and we need to do a release that day, then we're going to just do the release and note on this page that the last known version of model tensorflow and dfml that that dfml operations nlp worked with was this set, and then these commands will be equals equals that set so that we can ensure the tutorials always work, even though we document that they don't work on the latest version, um and so that gets more complicated.

A

But this is just the general and then third party is just you know, everybody's plugins, um so um we want to make sure to to as a part of making sure that these. So this is the test surpassing right now. The stuff we're doing today is let's track things that are going to help us make sure that we have tests and the tests are passing right. So this is, like you know, are we running the tests in ci right so uh did like, even even before running the tests?

A

Have we or do we have something? That is set up to run the test all right. um So let's check this out. um I think what we're doing here yeah. So I was just parsing the workflow all right. So the approach that we're going to take with this one is test operations.

A

Okay, all right, so I went off in a little bit of a direction here, so um we're gonna we're gonna change this up a little bit.

A

A

All right come on all right. Just getting everything set up here, check.

A

Python just test.

A

Test one all right- and this is just watching for file changes- that's what notebond it's javascript utility. Nothing else. Doesn't I notify right for some reason um so and then we basically say we're going to clear the screen and rerun the test every time uh you guys have seen this for most of you so.

A

All right, so it should dump us out that doesn't have like.

A

Why is it still running.

A

Orchestrator, which can.

A

A

Why did it do that? Didn't do that? The other day.

A

It's in dfp process just instantiating the config, maybe I'm just killing it. While it's loading and I'm impatient. You never know.

A

I kind of don't think I'm killing it. While it's loading, I think nope. Okay, I wow. I am impatient all right.

A

All right, okay, perfect! So this is loading you're, seeing a search in error 42. So we said uh you know check that when we run this context right. So this says analyze dfml. uh When we run the the dfml context, uh the answer should be 42.. Obviously that doesn't make any sense uh we're in the middle of writing the test right here.

A

So um all right. Okay, so- and I started out with this parse github workflow, um but then I was given feedback that it doesn't really make any sense to um go straight into trying to analyze the deep internals of the workflow.

A

Until you know, um you know, until you just check for the presence of it or not right so start small. You know make sure that make sure that we know that it does exist right and then we can sort of grade it upwards. Based on the the presence, uh the presence is the first tick upwards, and then you know what the contents is. You know, what are we testing? What are we doing?

A

That would be the next. The next thing, all right, um so uh we're just gonna feed it a directory, probably uh yeah. I think we're gonna feed it. We should probably go look at the. um We should probably go. Look at the.

A

Let me just make the inputs here so we're going to look at the uh git um so surprise, surprise.

A

A

Feature operations right so surprise, surprise we're going to use the get repo operations right because they're, probably all this- is probably going to come from git repos. um So let's grab one get rid of release, get repository. I think we just do a git repository. We don't need to get repository checked out, um I mean technically we could do this on a directory right.

A

But I don't really want to put in an operation just to transform right now, because that's just going to complicate the examples.

A

So I'm just going to reuse the get repository operation our type and then we're just going to say you know, let's see.

A

Blah blah blah all right, um we want what is it yeah, repo dot directory? Okay, all right! So now we're just basically gonna say you know, presence github, workflow present.

A

A

I don't think this is a path object, all right. So now we just grab the directory and see, look, look at how easy this is, and I was making it so difficult.

A

All right, we look for the workflows directory if it's there we're good all right um get a word for a present. Oh my god,.

A

All right, so I gotta import that oh jesus remember the worst allergies.

A

Operations, import, uh okay repository. I turned on this auto complete stuff recently.

A

I can't decide if I like it or not: I've gone back and forth many times, okay and then we obviously need this pars github workflow. That's not defined all right, this one just imports star. So that's why it doesn't fail on import. I could hope workflow present.

A

All right we're just going to assume so we're assuming that, since git can only track directories with files in them. If there's any files in the github workflows directory, then, if the directory exists at all, then there's files in it. If there's files in it, then they should be workflow files, so all right so contents, all right, okay! So here's what we're going to do we're going to just take this root directory um where's this guy inputs.

A

Do you have a spec? I think you do.

A

Contents inputs, contents, spec, fantastic.

A

Okay, so what do we just do here? So we are changing the values, so what we just had.

A

Was this dfml router basically read the file right, the cut this is the the parse, the parse, the parsha action just took the raw. um Well, it took the string representation of the file right um so being pathlete. We can that's a nice one, liner, um so yeah, so we just read the file and- and you know we have this testing animal. So um that's that's what we did so now we're just going to create this github we're going to create this git repository.

A

um Oh it's in my vim history. So we're gonna create this git repository uh type, um we're gonna instantiate an instance of it as an input. Read definitions how we're done on time. Okay, we're doing good!

A

Oh okay, repo, spec, okay, all right the naming on this. We need to figure this out. I want we. We should make it so that these whole definitions and all this stuff is really just the same things that the python typing system.

A

I can't remember, I can't remember why it is that it's not that way. Already.

A

I really have no idea why we didn't just do that. That doesn't make any sense. um There must have been some, I might have been for the importing and exporting man. This thing is slow today, okay, so this is the gear repo spec object, so it takes a directory and url all right. Well, we don't have a url, so that's fine, so we're just going to since it'll default to none so we're just going to instantiate the spec. So basically we are so we grab.

A

You know we do an import star, um which is, you know, obviously not the best behavior, but this is a test um and we're just writing it as little lines as possible.

A

So the this this function, op-amp in you, can pass it something and it'll just go through it like and look to see if it has um any operation implementations in it um and of course, so anything you decorate with the op decorator is going to be an operation implementation um because it'll create the operation object uh with the arguments that you pass into op and then it'll create the implementation, class and implementation context class out of the function that it's wrapping. um Okay, so um then you know when we do when we create that op.

A

Let me just show that, so when we do that op right, um we set the inputs and here's the inputs contents, and so this op actually becomes a property on this. This also gets a property called dot, imp and dot. Imp is a implementation operation, implementation instance, and you can read about that in the um conceptual. So all of this is covered.

A

This is where so, I'm working on a series of tutorials about um async and event um like basically event loops and um like an event-based approach flowing into a data flow based approach to explain um you know what why why we're doing what we're doing? I guess I think that can get lost very easily.

A

How did this get nested so far? Okay, so it's on this page.

A

This is a very good page to read. If you want to know how the execution of a data flow works. Now this is so. This is one particular way we could execute a dataflow, but it doesn't it shouldn't it shouldn't be the only way. um So we need to figure out how to get that interface um going. I think right now, it's just implementing a new orchestrator, but I'm not sure if there should be something separate.

A

I'm not sure if there should be something separate to the orchestrator, because right now we have the operation implementation network in the input network and the two of those combined basically work together to do this. Permutation based execution, that's covered here.

A

And I'm not sure, I'm not sure we need to look into that, because we need to be able to support the use case. I'm thinking it was more of like a linear flow right. So basically you just put a bunch of operations in there and one goes to the next, um but I'm not sure if that's even different, I don't know it's a use case.

A

That's come out that I haven't had a chance to look at it so, but the point being think about how would you do this differently right and let me know so because there's got to be different ways to do this in different cases beyond this, this is like this is based on creating making sure every permutation of every function's inputs gets called, um and I'm not sure where that, where that breaks down, we need to find out where that breaks down, because it can't it must not work for all use cases right um they're.

A

We need to find a use case that we can't, because the one the one where it's just a linear sequence of things, I think that can be done by just chaining the inputs together.

A

So I'm pretty sure it breaks down somewhere, but we need to figure that out, okay, so um so that is the um so this this this is, and let me put it in the notes. So this is the explanation of where is the operation implementation over here operation implementation network um and then the operation, which is the operation implementation context, which would all be the dot imp. So this is dot imp and then this is the context that dot amp creates when you do the double context: entry.

A

Okay, um let me put that in the nets.

A

Okay- and we forgot to do that- okay- and we looked at this- and we looked at this- and we said this okay great- I think we're missing a few things along the way here, but whatever.

A

Come on, ah okay, whatever should be saved right. Google drive all right so now we um so then the spec. So this is the definition right and here's the definition, and we looked at the spec object earlier, um so we're instantiating that name tuple all right, so inputs contents all right. Well, we renamed that. So I better not have an input name context there. We.

A

Go repo great, so, let's see what happens, does it think there's any workflows.

A

Got an unexpected argument: repo! Oh! Why? Yes, it did.

A

Can we do this, maybe.

A

I haven't tried.

A

That'd be cool.

A

All right great, we can do that. That's interesting, I'm not sure. If that's that's, that's not pretty um see if they were all just regular definitions would be fine, but I I don't know if that's gonna work, we should try it all right. Hey, look! I'm seeing true at the end of that thing, so it looks like we do have.

A

We do have, let's print it.

A

All right look at that, so we determined that the workflow was present. Okay, so just for shits and google's now um we're gonna point it at arbitrary repos. um Did it try to print? Do you see this look at this.

A

All right, okay, god wow! That thing is really annoying um okay. So how should we do this? So, let's just go so we can follow this example.

A

So this is a this is just great, um no all right, okay! So let's just recap.

A

All right so should I should. I is a metastatic analysis tool that we built as a demo. um Basically it does very something very similar to this um goes in looks for presences of the presence of different files and then runs things you know based on the presence of those files. um So this is very very very similar to what we're going to do here and- and the goal of all of this is to basically be able to throw it all together right um so at a moment's notice, and so that's what we're gonna do.

A

um So uh we did a demo here. We make this tool, we make a bunch of operations.

A

And so this shows you the flow. So basically we get a python package. We go to pi pi, we grab the the metadata about it. We grab the download from the metadata we download it. We run this bandit utility once we have the version here from the package metadata, we run this safety check utility, that's the data flow right and that's making that's all these operations that we're implementing here. So then you'll remember the automating classification example.

A

We basically we go in, we clone gateway, clone gate repos, and we. um What do we do well? This is what we do. We count the authors we determine the um the diversity of authorship is the work metric metric and we have this data flow. Now we in this tutorial, we combine them, and so we look. Here's the get repository checked out operation, here's lines of code by language lines like code to comments um which I believe we use in that one.

A

We just looked at right and so now we're going to go and we're going to take. First of all, we deploy this. What do we do here? Yeah we create the the. What is that the overlay yeah they're override override overlay? I need to work on that wording. um So look now you can see the um you know. This is the the pi pipe um devs that should I utility and then this is. You know, part of that data flow that we just saw right so now we're just gonna.

A

Stick them together, um we're not going to write any more code right. So we wrote all these functions, we're not going to write any code, we're just going to stick them together now right. So now we're going to do the same thing here right so pi, pi package contents, um that is a directory, and so this is what I was talking about. I didn't want to do this, so I didn't want to go create this mapping, because it's just an extra step right.

A

So that's why I use the repository, because these guys accept a repository with the directory property, so you have to create a mapping object which then can gets converted, so you have to create. So if you do this create mapping create a mapping is like a more generic programming language, agnostic term.

A

For a dictionary right, so if you do this, you'll create a dictionary and its property of directory will be the directory where the package contents of the pi pi package were downloaded, then you set the key to be directory and now all of a sudden you have this sticked object with directory and then the path to the you know what is effectively your downloaded gear revo, but there's no gate dot get directory, um which is why it's not a real repository object and we're just faking the creation of it, and then we can pass it to anything that takes a repository right.

A

So that's it! There's like a little transform that happens there. um It transforms the data type one data type into another data type, um okay, so right we're going to download a bunch of key repos. So what do we do here when we do this? Okay? So we need to create the data flow that does the download and then we need to create the data flow. So I'm going to open the tutorials and we're just going to do this. So what is the tutorial tell us?

A

So we want to create a data flow with the git clone operation. This one does that so.

A

Just want to make sure we have all the operations, it should just be get clone clone get repo. Okay, oh, does it need check if get repository valid? um Okay, so we're gonna clone gear repo?

A

We don't need to do the checkout and we'll do the cleanup and then we want to grab the results still so we'll just build the data flow.

A

Okay, test operations test run. Okay. I think we can say that this is good, um so I'm just going to fix up this test here.

A

Let's see this just grabs all the outputs.

A

um Okay, so that just grabs all the outputs, um let's see um and here's the data flow, okay. So all right. So this we're going to move the data flow um to the to the to the global namespace in this file um and we are going to.

A

A

Can I do this, I can't remember if I can do this in python. I know you can do it in. Can you do it in javascript? I can never remember which one yeah you can do it in python all right, um not that name.

A

um And then it should be true right, so let's try this again. All right did we put that there it's going to want a data flow.

A

If you give a run a data flow, it's going to want some inputs.

A

All right, okay, am I all right all right, fine up outputs. This is why we need the more ergonomic api on this thing. Sahel is working on: okay, um god, damn it that name. I always forget that.

A

All right, so this we're just building building. You know what the output dictionary should be. So it's now 42 right so that before we continue we're not um we're not in a weird spot. Okay, so let's um let's commit that um all right. So this is operations in our source and how we're doing on time all right. We got eight minutes all right, we're gonna, we're gonna! Do this!

A

We're gonna run it against some some gear repos, um so operations intersource- uh and this would be you- know- fixed tests or switch to downloading or switch to checking for presence of workflow stir.

A

All right great look at that, so we got one test. Passing um oh come on. What did I do? Oh wait.

A

Oh, oh god. What did I do.

A

This is always scary: okay,.

A

Oh yeah, that's right! We're messing with that! Oh yes, that's the remote and local execution. I've had a breakthrough on that one. um Okay, so let's go write this in the tests. We can just look, not look. We don't need to look at that right now, all right so test operations um and we'll call this. You know maybe.

A

So, which we call this.

A

That's operations.

A

I'll just say: test data flow for now we're test on real reboot repos, all right great, so we're going to test it on some real repos all right. So you know I'm going to how we should have a basic test function, I'm realizing okay, wow. That would be really really a good. How would we not have this yet? Okay, we're going to write that um wow, really real real boneheaded move? um Okay. So if not.

A

Can you do that uh yeah? You can.

A

All right, I don't like to use a cert. Basically, uh if you python, by compile things um files um and then run from the python by compiled ones, then assert statements go away.

A

I'm I'm not fully versed on it, but this is what I hear so I just stay away from them at all times and then I never have to worry about it. So all right um so.

A

Ctxtr all right, so this is just going to be. Our basic test function uh was.

A

Because we have this code, a million places see texture wizard result should be.

A

A

I'm really kicking myself for not doing this earlier.

A

Oh, my god, this is really going to be how how nice this is. um Look at this we're just going to add this async test case. This is going to be fantastic. um No, that's a dumb idea! That's dumb idea! Let's not do that! um It should be staying alone. Should we stand on.

A

A

Let's just add it.

A

All right, so I think we do have this somewhere, so we're just going to make this a tuple right and so you're going to put your inputs for that context, and then your outputs for that context- and this should have been done a long time ago, um so where's the inputs, man, look at this yeah. We definitely have this somewhere why we didn't think to put it in a function. It's beyond me.

A

A

A

A

A

A

This is what the check object is going to look like, and then we need to take a data flow.

A

He went to the bathroom pretty recently yeah he actually went outside. I was surprised.

A

A

Therefore, inputs should we just do.

A

Sometimes this works, sometimes it doesn't haven't figured out when, when is what yet.

A

Yeah I like to do that. um I like to name the variable instead of just using the underscore. I think I think it's better. Okay, um I think it's. I think it's more clear just so you don't forget what it is. If you, you know, if you're analyzing the code later and you're like well, what why is this not working? Well, I'm not using the thing that you know I obviously should have been using right.

A

I'm not checking that value. It's more obvious! That way, all right, um so we're going to go through we're going to run every context.

A

So, basically, I dot items is going to give us the key and then value as a tuple and we're expanding the inner tuple, which is the list of inputs and the um the outputs man. Why did we not have this? This would have made life so much easier, um all right and then yeah all right. I think we're good to go and then we go through and we do our assertion error. You know if it doesn't match the uh what it should be.

A

Oh wait! No, we gotta do.

A

One because we're checking against the.

A

Look at that there we go.

A

So now we're going to move down here and we're going to paste.

A

I think we should be good to go.

A

A

Wow, that's great, and I love that that's fantastic. I can't believe we didn't do that before.

A

Okay, so let's add this: we're just going to add it right here right now,.

A

uh No we're just going to add it in async test case. Let's just put it there, I don't. I don't know why hesitating I'm putting it in here. I don't really like it. I don't like that. I don't like that. No, we can't do that because um no, we can't do that. Okay, all right! Fine, let's just put it in there all right.

A

A

All right yeah- and this is why so I think uh some very saintly person went and recombined the async test case and the integration cli test case, and now we have just this beautiful, beautiful class that really handles a lot of things for us.

A

So I can't remember exactly who did that, but thank you very much so you've made everyone's life so much easier. So we're just going to add it here, because we already have all those methods on it. So we can just add one more um and so because we're already using things like the make dynamic, temp file make temperature. I I I yeah anyways, we'll just do this. Okay, so let's run dataflow check results, okay, or what about the run?

A

Oh yeah? hmm How we're going to do that.

A

Okay, this thing should be, I think we can import run. I don't want to create a dependency circle or a circular dependency, whatever.

A

Okay, um where is that oh yeah, it's in high level.

A

And the reason why we do the relative imports with the from within internally is because, if we move a file around- and we don't have to change the usage everywhere right- we just change the import now we should be externally point from the top level. Okay, let's see, I hope that works.

A

Okay, so data file.

A

Let's serve around data flow async test case, but you're an async test case.

A

All right: well, this should be here anyways all right. I think I'm out of time.

A

Assert run dataflow async, def.

A

Async test case am I missing something here.

A

Is cfml not installed in development mode.

A

hmm Cool is going to kick me off my one person meeting um all right. Okay, we'll come back to this.